Evidence Based on Internal Structure

As for validity evidence based on internal structure, a variety of evidence was provided in all available documentation from test developers. Researchers have also conducted validity studies especially regarding the factor structure or dimensionality of ELP assessments. In one published study, Cai and Hansen (2018) used ELPA21 score data to explore multivariate IRT approaches to analyzing test structure. In another, Romhild, Kenyon, and MacGregor (2011) utilized a factor analytic approach to examine whether domain-specific language knowledge could be said to constitute an ability factor on the ACCESS assessment. They found that domain-general knowledge was the prevalent dimension but that for higher proficiency levels domain-specific knowledge was a salient contributor.

For another example of how states have analyzed tests based on their internal structures, the New York State English as a Second Language Achievement Test (NYSESLAT) developers report intercorrelations of modality subscores as one type of validity evidence in this category (New York State Department of Education, 2019). The state claimed that the correlations supported the overall validity of the test as well as the validity of subscore reporting requirements. The state of Arizona performed a similar analysis (Arizona Department of Education, 2019) and came to the same conclusions about its assessments.

Evidence Based on Relations to Other Variables

The Standards support the use of traditional validation work that relates the measure under study or under construction to other measures of the same or different constructs. This has been called criterion-related validity and is further subdivided into predictive and concurrent validity evidence—respectively, evidence that current performance on the measurement of interest predicts future outcomes of interest, or that current performance on the measurement of interest predicts contemporaneous and related measures or scores.

Test developers and other researchers have made criterion-related validity a key component of validation work, and ample evidence supporting the current group of ELP assessments has been collected. For example, Kong, Powers, Starr, and Williams (2012) used ELP assessment and state reading assessment scores to predict subsequent student achievement. The researchers found that overall the test scores were highly predictive of future academic performance of EL students, lending support to the validity of the ELP assessment scores. In another example, Parker, Louie, and O’Dwyer (2009) conducted an analysis of ACCESS scores in fifth and eighth grades and compared students’ scores on this ELP assessment to scores on students’ state-mandated content-area assessments of reading and writing (English language arts) and mathematics. The authors found significant and substantial correlations between the ELP assessment and contentarea assessment scores. Correlations were highest in areas theorized to be most closely related (reading and writing domain scores from ELP and language arts assessment scores) and lower in areas theorized to be less closely related (ELP and mathematics assessments). The authors concluded this was evidence of construct validity for the ACCESS assessment for those grade bands.

Evidence for Validity and Consequences of Testing

The Standards require test developers to consider the intended and unintended consequences of testing. In addition, the Standards direct test developers to collect evidence of the overall beneficial impact of an assessment upon “components of the system that will go beyond the interpretation of test scores as a valid measure of student achievement” (AERA, APA, & NCME 2014, p. 19). The extent to which this work has been addressed by ELP test developers up to now is in question. Available evidence suggests that little has been done by test developers and states to validate the consequences, both intended and unintended, of ELP assessment use in schools. This is perhaps unsurprising given that this particular section of the Standards is not mentioned in the federal Peer Review guidance provided to states (U.S. Department of Education, 2018).

The Texas English Language Proficiency Assessment System (TELPAS) technical manual (Texas Department of Education, 2019a), alone among the publicly available technical reports, contains a section detailing the need for validation of consequences but does not cite any specific activity performed by developers to support consequential validity. Instead, a second document (Texas Department of Education, 2019b) is mentioned. This document describes a pattern of gradually increasing TELPAS scores, with a dip in average scores in 2013 when new standards were introduced, followed by another period of gradual increase, as evidence of beneficial consequential validity. Interestingly, this pattern—which is widespread in U.S. accountability testing—is precisely described by Koretz

(2002, 2017) as evidence of negative consequential validity in accountability testing. Koretz (2002) explains this pattern as a predictable outcome of teaching to the test and narrowing the curriculum—as educators leam about a new assessment, they adapt their teaching to the new test, resulting in modest average score increases for several years, until the “effectiveness” of this approach wanes. Then, a new assessment and new standards are created, and the cycle begins anew. Yet TELPAS cites this exact pattern of score increase as evidence that implementation of the test itself is promoting learning, with a premise that the new TELPAS assessment measures higher and more rigorous language skills. Increasing scores on an educational measurement, by themselves and without concurrent evidence of learning gains (such as that provided by an independently developed assessment), do not seem to be convincing evidence of beneficial consequential validity.

Little in the other available technical reports list any type of validity evidence that might fit into this category. Further, the federal Peer Review guidance documents (U.S. Department of Education, 2016, 2017, 2018) nowhere lists consequential validity evidence as a type of validity evidence that should be collected, nor note this as an area for states to work in the future. In reading the guidance documents, one would be forgiven for thinking that consequential validity evidence was not an explicit element of the Standards, elsewhere in the Federal guidance documents so carefolly held up as an accepted standard of best practice.