Validation of Generated Multilingual Test Items

We recommend that the same general strategies described in Chapter 7 be used to validate multilingual generated items. These strategies require substantive and statistical analyses. The substantive analyses focus on cognitive, item, and distractor model reviews. SMEs evaluate the quality of the content in the cognitive, item, and distractors models using the outcomes from the validation tables to ensure that the content in the models yields the intended results and meets the appropriate standards of quality across language groups. We also recommend substantive reviews using rating scales to produce independent evaluations of item quality, item comparability, and predictive accuracy across languages. SMEs can use rating scales to produce succinct and objective measures of quality and to provide evidence that the generated items meet the standards of quality required for operational use in a testing program.

The most important difference to consider when conducting the substantive evaluation with multilingual generated items is the composition and qualification of the SMEs. Validation of monolingual generated items requires qualified SMEs, whereas validation of multilingual generated items requires SMEs and translators. For monolingual item generation, the SME is responsible for identifying, organizing, and evaluating the content required to create the cognitive and item model, which relies on human judgement acquired through extensive training and practical experiences. SMEs are expected to possess a broad range of knowledge and skills that includes teaching experience and item development experience, as well as familiarity with the curriculum and the examinee cohort. SMEs participate in the creation of the AIG models, as well as the evaluation and revision of these models. For multilingual item generation, SMEs must possess this extensive list of qualifications. But in addition to the SMEs, translators are required. Test translators must possess a broad range of knowledge and skills. In the International Test Commission Guidelines for Translation and Adapting Tests (201 7), a qualified test translator is described as either a person or a team of people who possess knowledge of the languages, cultures, and test content. A qualified translator is also expected to have a general understanding of the principles in educational testing. SMEs can serve as qualified test translators but only when they possess all of these important skills. In practice, the creation and validation of multilingual generated items will likely require a team of SMEs and translators who, collectively, possess the broad range of knowledge and skills described in the International Test Commission Guidelines in order to produce and evaluate the models and content needed for multilingual AIG.

An example of a study that used substantive evaluation to assess item quality across language was conducted by Gierl et al. (2015). Seven different item models used to generate junior high school science, initially created in English, were translated into Spanish to generate equivalent

Spanish items. A sample of 22 generated items was selected for review. Two bilingual SMEs evaluated the sample of generated Spanish items. The bilingual SMEs were language experts who both received master of arts graduate degrees in Spanish and who worked for more than one year as item developers for a large testing company. The SMEs worked independently from one another and were blind to the item development process. They used a 4-point scale to evaluate the quality of 22 generated Spanish items in which the ratings included 1 (Acceptable), 2 (Minor Revisions), 3 (Major Revisions), and 4 (Reject). Across the 22 items, the two reviewers agreed on their rating for 20 of the 22 items. These 20 items were judged to be acceptable (i.e. rating of a 1 or 2), meaning that the items were judged as acceptable, requiring either no or minor revisions by both bilingual SMEs. The two SMEs disagreed on their rating for 2 of the 22 items. For both items, the first SME judged the items to be a 2 (Minor Revisions) while the second SME rated the items to be a 3 (Major Revisions). The comment used to justify the rating of a 3 for both items was that the stem was missing an indirect object pronoun and that one of the incorrect options was missing an article. The SME who rated these items as a 2 saw these changes as minor, requiring an in-house correction, whereas the SME who rated these items as a 3 saw the combined effect of these two errors as major, thereby indicating a more serious translation problem.

In addition to substantive analyses, statistical analyses should be conducted. Statistical analyses yield information about the quality of the generated items that, in a multilingual context, can be implemented within and between languages. The analyses we described in Chapter 7 are appropriate for evaluating item quality within each language. For example, an item analysis could be conducted using data from examinees who wrote a sample of the English medical generated items. A separate item analysis could be conducted using data from examinees who wrote a sample of the Korean medical generated items. The item analysis we presented in Chapter 7 can be focused on the correct option using the proportion correct value, the point-biserial correlation, the biserial correlation, and the index of discrimination in a classical test theory framework or the a- and b-parameter estimates in the item response theory framework. The item analysis can be focused on the incorrect options using the percentage selection value, the trace line plots with a chi-square test, the point-biserial correlation, the biserial correlation, and the index of discrimination in the classical test theory framework or the parameter estimates for the incorrect options using the nominal or graded-response models in the item response theory framework.

Statistical analyses should also be conducted between languages. The first between-group comparison that is often conducted in a multilingual testing situation is focused on evaluating whether the items measure the same construct using some form of latent trait statistical analysis, such as a factor analysis. Factor analysis is a scientific method for summarizing the interrelationships among variables as an aid in conceptualization (Gorsuch, 1983). In educational testing, the goal of factor analysis is to describe the association among a large number of observed variables (i.e., test items) using a small number of latent variables or factors (i.e., construct measured by the test items). The association is expressed as the loading in the factor analytic model. When exploratory factor analysis is used, the data for each language group is analyzed separately and the resulting loadings are reviewed to determine if the factor structure for one language group (e.g., English) is comparable in structure (i.e., items load on the same factor) and magnitude (i.e., do the same items yield the same loadings) to the other language group (e.g., Korean). Generated items that are equivalent will measure the same factor and produce similar loadings. When multiple-group confirmatory factor analysis is used, the data for each language group is analyzed together so that the comparability of the factor loadings and factor structure can be evaluated. Confirmatory, like exploratory, factor analysis yields factor loadings in which the structure and magnitude can be evaluated. In addition, confirmatory factor analysis provides measures for evaluating the comparability of the data and the models between languages using goodness-of-fit statistics. Generated items that are equivalent across languages will measure the same factor structure, produce similar factor loadings, and yield interpretable model-data fit statistics (Sireci, Patsula, & Hambleton, 2005).

The second between-group comparison that is often conducted in a multilingual testing situation is focused on evaluating whether the items elicit language group differences using differential item functioning (DIF) analyses. The results from a DIF analysis yield information about item equivalence. DIF occurs when an item functions differentially for members of one group relative to the other group after controllingfor overall ability in both groups.Three commonly used DIF statistical methods are the Mantel-Haenszel procedure, SIBTEST, and logistic regression (Clauser & Mazor, 2005). DIF is commonly used to identify item-level group differences for gender (i.e., male vs. female) and race (e.g., White vs. Asian). DIF can also be used to evaluate the equivalence of multilingual generated items (e.g., English vs. Korean). The presence of DIF indicates that the same generated item does not produce the same examinee performance across the two languages after controlling for ability differences between the groups. Hence DIF indicates an item is not equivalent between language groups. The purpose of conducting a DIF analysis is to identify the items that function differentially and then to review the DIF items in order to determine the reason for the group difference. In multilingual testing situations, researchers have used DIF detection to identify translation errors (e.g., Allalouf, Hambleton, & Sireci, 1999; Ercikan, 2002; Ercikan, Gierl, McCreith, Puhan, & Koh, 2004; Ercikan, Simon, & Oliveri, 2013) and cultural differences (e.g., Li, Cohen, & Ibarra, 2004; Scheuneman & Grima, 1997; van de Vijver & Tanzer, 1997). DIF can also be used to validate generated multilingual items. Generated items that are equivalent across language should be identified as non-DIF items.

References

Alberta Education (2019). Provincial Achievement Testing Program Grades 6 and 9 General Information Bulletin. |https://www.alberta.ca/education.aspx],

Allalouf, A., Hambleton, R. K., & Sireci, S. G. (1999). Identifying the causes of DIF in translated verbal items, lournal of Educational Measurement, 36(3), 185-198.

Brislin, R. W. (1986). The wording and translation of research instruments. In W. J. Lonner & ). W. Berry (Eds.), Field Methods in Cross-Cultural Psychology (pp. 137-164). Newbury Park, CA: Sage Publications.

Clauser, В. E., & Mazor, К. M. (2005). Using statistical procedures to identify differentially functioning test items. Educational Measurement: Issues and Practice, 17, 31-44.

Ercikan, K. (2002). Disentangling sources of differential item functioning in multilanguage assessments. International Journal of Testing, 2(3), 199-215.

Ercikan, K., Gierl, J. )., McCreith, T., Puhan, G., & Koh, K. (2004). Comparability of bilingual versions of assessments: Sources of incomparability of English and French versions of Canada's national achievement tests. Applied Measurement in Education, 17, 301-321.

Ercikan, K., Simon, M., & Oliveri, M. E. (2013). Score comparability of multiple language versions of assessments within jurisdictions. In M. Simon, K. Ercikan, & M. Rousseau (Eds.), An International Handbook for Large-Scale Assessments (pp. 110-124). New York: Springer.

Ganji, M., & Esfandiari, R. (2020). Attitudes of language teachers toward multiple-choice item writing guidelines: An exploratory factor analysis. Journal of Modern Research in English Language Studies, 7, 115-140.

Gierl, M. J., & Lai, H. (2015). Using automated processes to generate English and French test items simultaneously. Mesure et evaluation en education— Measurement and Evaluation in Education, 37, 39-61.

Gierl, M.J., Lai, H., Houston, L., Rich, C., & Boughton, K. (2015). Using automated processes to generate items in three or more languages. International Journal of e-Assessment, 6, 1-19.

Gierl, M. J., Lai, H., Fung, K., & Zheng, B. (2016). Using technology-enhanced processes to generate items in multiple languages. In F. Drasgow (Ed.), Technology and Testing: Improving Educational and Psychological Measurement (pp. 109- 127). New York: Routledge.

Gorsuch, R. L. (1983). Factor Analysis (2nd ed.J. Hillsdale, NJ: Erlbaum.

Hambleton, R. K. (2005). Issues, designs, and technical guidelines for adapting tests into multiple languages and cultures. In R. K. Hambleton, P. F. Merenda, & C. D. Spielberger (Eds.), Adapting Educational and Psychological Tests for Cross-Cultural Assessment (pp. 3-38). Mahwah, NJ: Erlbaum.

Hambleton, R. K., & Bollwark, J. (1991). Adapting tests for use in different cultures: Technical issues and methods. Bulletin of the International Test Commission, 18, 3-32.

Hambleton, R. K., Merenda, P. F., & Spielberger, C. D. (2005). Adapting Educational and Psychological Tests for Cross-Cultural Assessment. Mahwah, NJ: Erlbaum.

International Test Commission. (2017). The ITC Guidelines for Translating and Adapting Tests (2nd ed.). [www.lnTestCom.org]

Li, Y., Cohen, A. S., & Ibarra, R. A. (2004). Characteristics of mathematics items associated with gender DIF. International lournal of Testing, 4(2), 115-135.

OECD (2019). PISA 2018 Technical Report. Paris: PISA, OECD Publishing.

Scheuneman, J. D., & Grima, A. (1997). Characteristics of quantitative word items associated with differential performance for female and Black examinees. Applied Measurement in Education, 10(4), 299-319.

Sireci, S. G., Patsula, L., & Hambleton, R. K. (2005). Statistical methods for identifying flaws in the test adaption process. In R. K. Hambleton, P. F. Merenda, & C. D. Spielberger (Eds.), Adapting Educational and Psychological Tests for Cross-Cultural Assessment (pp. 93-116). Mahwah, NJ: Erlbaum.

van de Vijver, F. J. R., & Tanzer, N. K. (1997). Bias and equivalence in cross- cultural assessment: An overview. European Review of Applied Psychology, 47(4), 263-279.

van de Vijver, F. J. R., & Poortinga, Y. H. (2005). Conceptual and methodological issues in adapting tests. In R. K. Hambleton, P. F. Merenda, & C. D. Spielberger (Eds.), Adapting Educational and Psychological Tests for Cross-Cultural Assessment (pp. 39-64). Mahwah, NJ: Erlbaum.