Practical Considerations and Recommendations
Thus far, we have described state-of-the-art automated scoring capabilities and several case studies that are relevant to K—12 ELP assessment. Due to the potential benefits of the technology, it is clear that the field is moving forward with exploring the use of automated scoring for large-scale, K-12 ELP assessments. In this section, we discuss key considerations and recommendations for stakeholders who are considering the use of automated scoring in this context (see Madnani & Cahill, 2018, for a more in-depth discussion of the priorities of different stakeholders and how they relate to automated scoring system development). We also argue that an automated scoring system for a K-12 ELP assessment should meet the following three criteria:
- • It should support assessment of the construct in a reliable, valid, and fair manner;
- • It should work practically and meet the needs of students, administrators, and score users; and
- • It should make efficient use of available human, technological, and financial resources.
If a system meets these criteria for all of the tasks types included in an assessment, it can be labeled as “mature” and ready to be considered for operational use.
This section addresses a range of factors that contribute to these three criteria by discussing (1) assessment construct and task design considerations that affect the reliability, validity, and fairness of scores based on automated speech scoring; (2) scoring and score reporting considerations that influence how students, administrators, and score users interpret and use assessment results; and (3) automated scoring model development and test delivery considerations that have a practical impact on the use of available resources.
Assessment Construct and Task Design Considerations
The most important decisions are those about the aspects of the speaking or writing construct—that is, the test taker’s knowledge, skills, and abilities—for which the assessment should elicit evidence. For K-12 ELP assessments, these decisions are based on the applicable standards for ELP for EL students. If an automated scoring system is used for the assessment, the system should be able to extract evidence that is related to the construct targeted by the task. Automated scoring tends to be most effective on tasks that are more constrained in nature, in which the content of the student’s response is predictable from the content of the prompt. For open-ended tasks, automated scoring systems may not be able to extract features that cover the full range of the construct, such as whether a test taker has provided relevant details in a narrative writing task or whether a test taker has used appropriate argumentation strategies in support of a position in an argumentative writing task.
In the domain of automated speech scoring, mature automated scoring systems exist for task types that elicit constrained speech. One such task type is a read-aloud task, in which a test taker reads aloud a selection of printed text (word list, sentence, paragraph, etc.). Since the content is fixed, the main aspects of the speaking construct assessed by this task are pronunciation, fluency, and intonation. Mature systems can provide broad coverage of these aspects of the speaking construct and typically produce scores that correlate with human scores at or above human-human agreement rates. Automated scoring systems can therefore be used for tasks of this nature with a high degree of confidence in their reliability and validity. On the other hand, state-of-the-art automated speech scoring systems are not yet fully mature for most task types that elicit spontaneous speech, since the features they can extract related to vocabulary, grammar, content, and discourse do not typically assess all aspects of the targeted speaking construct.
Automated writing evaluation systems can also be considered on a similar continuum. Mature systems exist for evaluating writing quality, such as grammatical correctness and complexity, vocabulary use, discourse, and organization. Systems for automated evaluation of written content have been maturing rapidly in the recent decade as well, for item types in which the correct content can be pre-defined (e.g., science, math, reading comprehension). An example of a developing capability in automated writing evaluation is argumentation. ELP standards typically emphasize the importance of argumentative writing, in other words, the ability to persuade in written English. Research on argumentative features for automated writing evaluation systems has produced many features in recent years, and these features have been shown to improve the performance of automated scoring of argumentative writing (Nguyen & Litman, 2018). However, the rate of agreement with human raters and the coverage of the construct are not quite mature yet.
The strengths and limitations of automated scoring systems across the continuum create some interesting challenges for the design of speaking and writing tasks and the assessment of the related constructs. In order to increase the likelihood of success of an automated scoring system, it is advantageous to constrain the content of the student’s response in some way. However, any ELP assessment must appropriately sample the applicable domain as defined in ELP curriculum standards that the assessment is designed to assess. Current ELP standards often emphasize the aspects of language proficiency that are required to express higher-order skills involving critical thinking, and these aspects of language proficiency are best assessed via assessment tasks that elicit spontaneous speech or extended writing and that match the target language use of the classroom. However, if an automated assessment system is not able to score responses to these tasks based on features that fully cover the targeted construct, then the scores that it produces will not be fully valid. Therefore, it is crucial to find a balance between these constraints when designing tasks to be included in an ELP assessment that employs automated scoring. The overall goal must be to ensure that scores on the speaking or writing domain, on the whole, support the claims that the assessment wishes to make regarding the ability of students.
