Using automated programmes & approaches for test development or assessing productive skills.

Workshop facilitators:

Scott Crossley, Georgia State University and Arizona State University, sacrossley(a)gmail.com
Danielle McNamara, Arizona State University, dsmcnamara1(a)gmail.com

Intended learning outcomes:

Participants will become familiar with the structure and applications of the Coh-Metrix tool, the Test Easability Assessor (TEA), the Writing Assessment Tool (WAT), and the Simple NLP (SiNLP) tool to include the number and variety of indices available in each tool, the interpretation of the indices, and common methods for analyzing assessment data using the indices.

Workshop content:

The workshop will cover the development and design of the Coh-Metrix tool, the Test Easability Assessor (TEA), the Writing Assessment Tool (WAT), and the Simple NLP (SiNLP) tool and the tools’ applications in test development and the assessment of productive skills (both reading and writing skills). Specifically, this workshop will demonstrate how these tools can be used to assess text difficulty during test development and how these tools can be used to automatically examine speaking and writing proficiency in testing situations.

Methods:

Participants will learn the basics of natural language processing (NLP) and how to code a simple NLP tool (SiNLP) using the Python programming language. The participants will then scale up to an introduction of the basic programming features underlying Coh-Metrix, TEA, and WAT. Once the participants have a full understanding of how these tools calculate linguistic features, they will be given practical exercises involving the use these tools in testing situations (to include assessing text readability and assessing students’ written and spoken responses). These exercises will include basic approaches for conducting statistical analyses with the tools (using the open-source software package R) and more advanced machine learning techniques such as discriminant analysis, naïve Bayes algorithms, and linear regressions (using the WEKA software package). The participants will also be introduced to techniques to increase the credibility and reliability of statistical analyses including the use of training, validation, and test sets as well as cross-validation.

Specific background or prior knowledge:

Basic statistical knowledge (the interpretation of means, standard deviations, p values) and familiarity with designing quantitative studies.

Please bring your laptop.

 

Pre-workshop activities:

Participants should read the following articles

Crossley, S. A., Varner, L., & McNamara, D. S. (under review). Analyzing discourse processing using a simple natural language processing tool (SiNLP). Discourse Processes.

Graesser, A.C., McNamara, D.S., & Kulikowich, J.M. (2011). Coh Metrix: Providing multilevel analyses of text characteristics. Educational Researcher, 40, 223-234.

McNamara, D. S., Crossley, S. A., & Roscoe, R. (2013). Natural Language Processing in an Intelligent Writing Strategy Tutoring System. Behavior Research Methods, 45 (2), 499-515.

McNamara, D., & Graesser, A. (2012). Coh-Metrix: An automated tool for theoretical and applied natural language processing. In P. M. McCarthy & C. Boonthum (Eds.), Applied natural language processing and content analysis: Identification, investigation, and resolution. Hershey, PA: IGI Global.

Bio-data of Workshop Facilitators:

Scott Crossley is an Associate Professor at Georgia State University and Research Fellow at the Learning Sciences Institute at Arizona State University. His research interests involve the development and use of natural language processing tools to investigate language acquisition, language use, and educational assessments. He has assisted with the development and testing of Coh-Metrix, TEA, WAT, and SiNLP for the past 10 years. He has published over 50 papers on using Coh-Metrix and other NLP tools in assessment situations. He has also presented three previous workshops on using Coh-Metrix and similar NLP tools to assess language development and proficiency. Finally, he has taught Ph.D. courses at Georgia State University on natural language processing, applied natural language processing, and quantitative methodology.

Danielle McNamara is a Professor in the Psychology Department and Senior Scientist in the Learning Sciences Institute at Arizona State University. Her research involves educational technologies that can be used to better understand cognitive processes involved in comprehension, knowledge and skill acquisition, comprehension strategies, and writing. She is also the primary investigator for the Coh-Metrix tool and the lead author on the in press book regarding the tool. In addition, she is the primary investigator for TEA and WAT. She has published over 100 articles on using Coh-Metrix and other NLP tools. She has also presented three previous workshops on using Coh-Metrix and similar NLP tools for language assessment.

 

 

 

Please bring your laptop.

Venue: 
Ramphals Building
Room R0.14

Online Registration:
Costs: £65
Registration is on a first-come first-serve basis, so please book early to avoid disappointment.
We will accept 30 participants.
To register click here.

Accomodation:
Warwick Conferences will hold rooms for you at a special price. A link for booking will be provided in due course.