Developing a Word Family-Based Depth of Academic Word Knowledge Test
Despite the continued widespread use of the academic word list, there is a paucity of comprehensive and easily replicable tests which measure learners.
- Pub. date: September 15, 2024
- Online Pub. date: September 10, 2024
- Pages: 137-154
- 57 Downloads
- 251 Views
- 0 Citations
Despite the continued widespread use of the academic word list, there is a paucity of comprehensive and easily replicable tests which measure learners’ depth of knowledge of academic word list items. This paper outlines the development and initial validation of a test which measures multiple aspects of learners’ knowledge of academic vocabulary. The test comprises four sections – one productive knowledge section (sentence writing), and three receptive knowledge sections (collocations, synonyms, and L1 meaning). Each section includes 30 items selected from the academic word list, 10 of which are derived forms containing one or more affixes. The test was administered to 193 Japanese students studying at a public medical university in Japan. All four sections of the test showed a high level of internal consistency. In addition, the inter-rater reliability between the two different markers for the productive use (sentence writing) section was also moderately high. Furthermore, Rasch item analysis showed that the test items were well fitted to comprehensively measure learners’ knowledge of academic vocabulary. These encouraging initial results suggest that there may be merit in the test being used, evaluated, and possibly adapted in other EAP teaching and learning contexts.
Keywords: Academic vocabulary, depth of vocabulary knowledge, diagnostic test.
0
References
Ackermann, K., & Chen, Y.-H. (2013). Developing the Academic Collocation List (ACL) - A corpus-driven and expert-judged approach. Journal of English for Academic Purposes, 12(4), 235-247. https://doi.org/10.1016/j.jeap.2013.08.002
Bauer, L., & Nation, P. (1993). Word families. International Journal of Lexicography, 6(4), 253-279. https://doi.org/10.1093/ijl/6.4.253
Beglar, D., & Nation, P. (2013). Assessing vocabulary. In A. J. Kunnan (Ed.) The companion to language assessment (pp. 172-184). Wiley-Blackwell. https://doi.org/10.1002/9781118411360.wbcla053
Biber, D., Johansson, S., Leech, G., Conrad, S., & Finegan, E. (1999). Longman grammar of spoken and written English (Vol. 2). Longman.
Boone, W. J., & Staver, J. R. (2020). Advances in Rasch analyses in the human sciences. Springer. https://doi.org/10.1007/978-3-030-43420-5
Brown, D., Stoeckel, T., McLean, S., & Stewart, J. (2022). The most appropriate lexical unit for L2 vocabulary research and pedagogy: A brief review of the evidence. Applied Linguistics, 43(3), 596-602. https://doi.org/10.1093/applin/amaa061
Browne, C., Culligan, B., & Phillips, J. (2013). The New Academic Word List (NAWL). New General Service List Project. https://www.newgeneralservicelist.com/new-general-service-list-1
Chui, A. S. Y. (2006). A study of the English vocabulary knowledge of university students in Hong Kong. Asian Journal of English Language Teaching, 16, 1-23.
Cobb, T., & Laufer, B. (2021). The nuclear word family list: A list of the most frequent family members, including base and affixed words. Language Learning, 71(3), 834-871. https://doi.org/10.1111/lang.12452
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46. https://doi.org/10.1177/001316446002000104
Cons, A. M. (2012). The use and misuse of academic words in writing: Analyzing the writing of secondary English learners and redesignated learners. TESOL Journal, 3(4), 610-638. https://doi.org/10.1002/tesj.36
Coxhead, A. (n.d.). Academic word list sublist families. Victoria University of Wellington School of Linguistics and Applied Language Studies/Te Herenga Waka Te Kura Tātari Reo. https://www.wgtn.ac.nz/lals/resources/academicwordlist/sublist
Coxhead, A. (2000). A new academic word list. TESOL Quarterly, 34(2), 213-238. https://doi.org/10.2307/3587951
Coxhead, A. (2016). Reflecting on Coxhead (2000),"A New Academic Word List". TESOL Quarterly, 50(1), 181-185. https://doi.org/10.1002/tesq.287
Daller, M. H., & Xue, H. (2009). Vocabulary knowledge and academic success: A study of Chinese students in UK higher education. In B. Richards, M. H. Daller, D. D. Malvern, P. Meara, J. Milton, & J. Treffers-Daller (Eds.). Vocabulary studies in first and second language acquisition (pp. 179-193). Palgrave Macmillan. https://doi.org/10.1057/9780230242258_11
Dang, T. N. Y., Coxhead, A., & Webb, S. (2017). The academic spoken word list. Language Learning, 67(4), 959-997. https://doi.org/10.1111/lang.12253
Durrant, P. (2014). Discipline and level-specificity in university students’ written vocabulary. Applied Linguistics, 35(3), 328-356. https://doi.org/10.1093/applin/amt016
Fernández, B. G., & Schmitt, N. (2015). How much collocation knowledge do L2 learners have? The effects of frequency and amount of exposure. ITL - International Journal of Applied Linguistics, 166(1), 94-126. https://doi.org/10.1075/itl.166.1.03fer
Gardner, D., & Davies, M. (2014). A new academic vocabulary list. Applied Linguistics, 35(3), 305-327. https://doi.org/10.1093/applin/amt015
Hunston, S. (2022). Corpora in applied linguistics. Cambridge University Press. https://doi.org/10.1017/9781108616218
Hyland, K. (2016). General and specific EAP. In K. Hyland & P. Shaw (Eds.), The Routledge handbook of English for academic purposes (pp. 17-29). Routledge. https://doi.org/10.4324/9781315657455
Hyland, K., & Tse, P. (2007). Is there an "academic vocabulary"? TESOL Quarterly, 41(2), 235-253. https://doi.org/10.1002/j.1545-7249.2007.tb00058.x
Ishii, T., & Schmitt, N. (2009). Developing an integrated diagnostic test of vocabulary size and depth. RELC Journal, 40(1), 5-22. https://doi.org/10.1177/0033688208101452
Kaneko, M. (2020). Lexical frequency profiling of high-stakes English tests: Text coverage of Cambridge first, EIKEN, GTEC, IELTS, TEAP, TOEFL, and TOEIC. JACET Journal, 64, 79-93.
Knoch, U., Rouhshad, A., Oon, S. P., & Storch, N. (2015). What happens to ESL students’ writing after three years of study at an English-medium university? Journal of Second Language Writing, 28, 39-52. https://doi.org/10.1016/j.jslw.2015.02.005
Kremmel, B., & Schmitt, N. (2016). Interpreting vocabulary test scores: What do various item formats tell us about learners’ ability to employ words? Language Assessment Quarterly, 13(4), 377-392. https://doi.org/10.1080/15434303.2016.1237516
Laufer, B. (2021). Lexical thresholds and alleged threats to validity: A storm in a teacup? Reading in a Foreign Language, 33, 238-246.
Laufer, B. (2023). Understanding L2-derived words in context: Is complete receptive morphological knowledge necessary? Studies in Second Language Acquisition, 46(1), 200-213. https://doi.org/10.1017/S0272263123000219
Laufer, B., & Cobb, T. (2020). How much knowledge of derived words is needed for reading? Applied Linguistics, 41(6), 971-998. https://doi.org/10.1093/applin/amz051
Laufer, B., Webb, S., Kim, S. K., & Yohanan, B. (2021). How well do learners know derived words in a second language? The effect of proficiency, word frequency and type of affix. ITL-International Journal of Applied Linguistics, 172(2), 229-258. https://doi.org/10.1075/itl.20020.lau
Linacre, J. M. (2022). WINSTEPS® Rasch measurement computer program (Version 5.3.2.0). [Computer software]. https://www.winsteps.com/index.htm
Linacre, J. M., & Tennant, A. (2009). More about critical eigenvalue sizes in standardized-residual principal components analysis (PCA). Rasch Measurement Transactions, 23(3), 1228. https://www.rasch.org/rmt/rmt233f.htm
Malmström, H., Pecorari, D., & Warnby, M. (2023). Teachers’ receptive and productive vocabulary sizes in English-medium instruction. Journal of Multilingual and Multicultural Development. Advance online publication. https://doi.org/10.1080/01434632.2023.2260781
Masrai, A., & Milton, J. (2018). Measuring the contribution of academic and general vocabulary knowledge to learners' academic achievement. Journal of English for Academic Purposes, 31, 44-57. https://doi.org/10.1016/j.jeap.2017.12.006
Masrai, A., Milton, J., El-Dakhs, D. A. S., & Elmenshawy, H. (2021). Measuring the contribution of specialist vocabulary knowledge to academic achievement: Disentangling effects of multiple types of word knowledge. Asian-Pacific Journal of Second and Foreign Language Education, 6, Article 8. https://doi.org/10.1186/s40862-021-00114-5
Mathieson, P., & Bolstad, F. (2020). A spoonful of humour helps the medicine go down: Enlivening academic English vocabulary learning in a medical English course. The Asian ESP Journal, 16(3), 64–87.
Mathieson, P., & Murray, C. (in press). Nurturing academic vocabulary knowledge growth with learner-inspired materials. [Manuscript submitted for publication]. Department of Clinical English, Nara Medical University.
McLean, S. (2018). Evidence for the adoption of the flemma as an appropriate word counting unit. Applied Linguistics, 39(6), 823-845. https://doi.org/10.1093/applin/amw050
McLean, S. (2021). The coverage comprehension model, its importance to pedagogy and research, and threats to the validity with which it is operationalized. Reading in a Foreign Language, 33(1), 126-140.
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-103). Macmillan.
Mochizuki, M., & Aizawa, K. (2000). An affix acquisition order for EFL learners: An exploratory study. System, 28(2), 291-304. https://doi.org/10.1016/S0346-251X(00)00013-0
Nadarajan, S. (2011). The challenges of getting L2 learners to use academic words in their writings. Electronic Journal of Foreign Language Teaching, 8(2), 184-200. https://e-flt.nus.edu.sg/v8n22011/nadarajan.pdf
Nagy, W., & Scott, J. A. (2000). Vocabulary processes. In M. L. Kamil, P. G. Mosenthal, P. D. Pearson, & R. Barr (Eds.), Handbook of reading research (Vol. III, pp. 269-284). Lawrence Erlbaum Associates.
Nagy, W., & Townsend, D. (2012). Words as tools: Learning academic vocabulary as language acquisition. Reading Research Quarterly, 47(1), 91-108. https://doi.org/10.1002/RRQ.011
Nagy, W. E., Herman, P. A., & Anderson, R. C. (1985). Learning words from context. Reading Research Quarterly, 20(2), 233-253. https://doi.org/10.2307/747758
Nation, I. S. P. (2011). Vocabulary research into practice. Language Teaching, 44(4), 529-539. https://doi.org/10.1017/S0261444811000267
Nation, I. S. P. (2022). Learning vocabulary in another language (3rd ed.). Cambridge University Press. https://doi.org/10.1017/9781009093873
Nation, I. S. P., & Webb, S. A. (2011). Researching and analyzing vocabulary. Heinle Cengage Learning.
Nation, P. (2021). Thoughts on word families. Studies in Second Language Acquisition, 43(5), 969-972. https://doi.org/10.1017/S027226312100067X
Nizonkiza, D. (2016). First-year university students’ receptive and productive use of academic vocabulary. Stellenbosch Papers in Linguistics, 45, 169-187. https://doi.org/10.5774/45-0-215
Phung, D. H., & Ha, H. T. (2022). Vocabulary demands of the IELTS Listening Test: An in-depth analysis. SAGE Open, 12(1), 1-13. https://doi.org/10.1177/21582440221079934
Porte, G., & McManus, K. (2019). Doing replication research in applied linguistics. Routledge. https://doi.org/10.4324/9781315621395
Qian, D. D., & Lin, L. H. F. (2020). The relationship between vocabulary knowledge and language proficiency. In S. Webb (Ed.), The Routledge handbook of vocabulary studies (pp. 81-96). Routledge. https://doi.org/10.4324/9780429291586-5
Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Danmarks Paedagogiske Institut.
Read, J. (2004). Plumbing the depths: How should the construct of vocabulary knowledge be defined? In P. Bogaards, & B. Laufer (Eds.), Vocabulary in a second language (pp. 209-227). John Benjamins. https://doi.org/10.1075/lllt.10.15rea
Read, J. (2023). Towards a new sophistication in vocabulary assessment. Language Testing, 40(1), 40-46. https://doi.org/10.1177/02655322221125698
Read, J., & Dang, T. N. Y. (2022). Measuring depth of academic vocabulary knowledge. Language Teaching Research, Advance online publication. https://doi.org/10.1177/13621688221105913
Schmitt, N., & Meara, P. (1997). Researching vocabulary through a word knowledge framework: Word associations and verbal suffixes. Studies in second language acquisition, 19(1), 17-36. https://doi.org/10.1017/S0272263197001022
Schmitt, N., Nation, P., & Kremmel, B. (2020). Moving the field of vocabulary assessment forward: The need for more rigorous test development and validation. Language Teaching, 53(1), 109-120. https://doi.org/10.1017/S0261444819000326
Schmitt, N., Schmitt, D., & Clapham, C. (2001). Developing and exploring the behaviour of two new versions of the Vocabulary Levels Test. Language Testing, 18(1), 55-88. https://doi.org/10.1177/026553220101800103
Snoder, P., & Laufer, B. (2022). EFL learners' receptive knowledge of derived words: The case of Swedish adolescents. TESOL Quarterly, 56(4), 1242-1265. https://doi.org/10.1002/tesq.3101
Stevens, J. P. (2002). Applied multivariate statistics for the social sciences (4th ed.). Psychology Press. https://doi.org/10.4324/9781410604491
Storch, N., & Tapper, J. (2009). The impact of an EAP course on postgraduate writing. Journal of English for Academic Purposes, 8(3), 207-223. https://doi.org/10.1016/j.jeap.2009.03.001
Therova, D. (2020). Review of academic word lists. TESL-EJ, 24(1). https://tesl-ej.org/wordpress/issues/volume24/ej93/ej93a5/
Warnby, M. (2024). Relating academic reading with academic vocabulary and general English proficiency to assess standards of students' university-preparedness - The case of IELTS and CEFR B2. Scandinavian Journal of Educational Research, Advance online publication.
https://doi.org/10.1080/00313831.2024.2318434
Webb, S. (2005). Receptive and productive vocabulary learning: The effects of reading and writing on word knowledge. Studies in Second Language Acquisition, 27(1), 33-52. https://doi.org/10.1017/S0272263105050023
Webb, S. (2007a). Learning word pairs and glossed sentences: The effects of a single context on vocabulary knowledge. Language Teaching Research, 11(1), 63-81. https://doi.org/10.1177/1362168806072463
Webb, S. (2007b). The effects of repetition on vocabulary knowledge. Applied Linguistics, 28(1), 46-65. https://doi.org/10.1093/applin/aml048
Webb, S. (2008). Receptive and productive vocabulary size. Studies in Second Language Acquisition, 30(1), 79-95. https://doi.org/10.1017/S0272263108080042
Webb, S. (2009). The effects of receptive and productive learning of word pairs on vocabulary knowledge. RELC Journal, 40(3), 360-376. https://doi.org/10.1177/0033688209343854
Webb, S. (2019). Incidental vocabulary learning. In S. Webb (Ed.), The Routledge handbook of vocabulary studies (pp. 225-239). Routledge. https://doi.org/10.4324/9780429291586-15
Webb, S., & Kagimoto, E. (2009). The effects of vocabulary learning on collocation and meaning. TESOL Quarterly, 43(1), 55-77. https://doi.org/10.1002/j.1545-7249.2009.tb00227.x
Webb, S., & Nation, P. (2017). How vocabulary is learned. Oxford University Press.
Webb, S., Sasao, Y., & Ballance, O. (2017). The updated Vocabulary Levels Test: Developing and validating two new forms of the VLT. ITL - International Journal of Applied Linguistics, 168(1), 33-69. https://doi.org/10.1075/itl.168.1.02web
West, M. (1953). A general service list of English words. Longman Green.
Wolfe, E. W., & Smith E. V., Jr. (2007). Instrument development tools and activities for measure validation using Rasch models: Part II - Validation activities. Journal of Applied Measurement, 8(2), 204-234.
Yanagisawa, A., & Webb, S. (2020). Measuring depth of vocabulary knowledge. In S. Webb (Ed.), The Routledge handbook of vocabulary studies (pp. 371-386). Routledge. https://doi.org/10.4324/9780429291586-24
Zhou, S. (2010). Comparing receptive and productive academic vocabulary knowledge of Chinese EFL learners. Asian Social Science, 6(10), 14-19. https://doi.org/10.5539/ass.v6n10p14