Despite the continued widespread use of the academic word list, there is a paucity of comprehensive and easily replicable tests which measure learners’ depth of knowledge of academic word list items. This paper outlines the development and initial validation of a test which measures multiple aspects of learners’ knowledge of academic vocabulary. The test comprises four sections – one productive knowledge section (sentence writing), and three receptive knowledge sections (collocations, synonyms, and L1 meaning). Each section includes 30 items selected from the academic word list, 10 of which are derived forms containing one or more affixes. The test was administered to 193 Japanese students studying at a public medical university in Japan. All four sections of the test showed a high level of internal consistency. In addition, the inter-rater reliability between the two different markers for the productive use (sentence writing) section was also moderately high. Furthermore, Rasch item analysis showed that the test items were well fitted to comprehensively measure learners’ knowledge of academic vocabulary. These encouraging initial results suggest that there may be merit in the test being used, evaluated, and possibly adapted in other EAP teaching and learning contexts.