Benefits of Using AIG for Item Development

AIG has at least five important benefits that help address the current need to produce large numbers of diverse, multilingual, high-quality test items in an efficient and economical manner. First, AIG permits the SME to create a single cognitive model that, in turn, yields many test items using the workflow presented in Figure 1.1. The ability to transform content from the initial state of a cognitive model into the final state of a test item is

Workflow and data transformation required to generate items made possible by item modelling

Figure 1.1 Workflow and data transformation required to generate items made possible by item modelling. Item modelling, therefore, provides the foundation for AIG. An item model is a template that highlights the parts of an assessment task that can be manipulated to produce new items. An item model can be developed to yield many test items from one cognitive model. Item models can also be written in different languages to permit multilingual AIG.

Second, AIG can lead to more cost-effective content production because the item model is continually re-used to yield many test items compared with developing each item individually. In the process, costly yet common errors in item writing (e.g., including or excluding words, phrases, or expressions, along with spelling, grammatical, punctuation, capitalization, typeface, and formatting problems) can be avoided because only specific parts in the stem and options are manipulated when producing large numbers of items (Schmeiser & Welch, 2006). The item model serves as a template for which the SME manipulates only specific, well-defined, parts. The remaining parts of the assessment task are not altered during the item development process, thereby avoiding potential errors that often arise during item writing. The view of an item model as a template with both fixed and variable parts contrasts with the traditional view where every single part of the test items is unique, both within and across items.

Third, AIG treats the item model as the fundamental unit of analysis where a single model is used to generate many items compared with a traditional approach where the item is treated as the unit of analysis (Drasgow et al., 2006). Hence, AIG is a scalable process because one item model can generate many test items. With a traditional item development approach, the item is the unit of analysis where each item is created individually. If, for instance, an SME working in the medical education context intends to have 12,480 items for her bank, then she would require 10 item models (Gierl et al., 2012, for example, generated 1,248 medical surgery items from 1 cognitive model). If a particularly ambitious SME aspired to have a very large inventory with over a half-million items, then she would require approximately 400 item models (i.e., if each item model generated, on average, 1,248 medical items, then 401 item models could be used to generate 500,448 items). Creating 400 item models within a year would be a significant but viable item development goal (i.e., about 33 models a month). By way of contrast, writing 500,448 individual items within a year would be a monumental and likely impossible item development goal (i.e., about 41,700 items a month). Because of this unit of analysis shift, the cost per item will decrease because SMEs are producing models that yield multiple items rather than producing single unique items (Kosh, Simpson, Bickel, Kellogg, & Sanford-Moore, 2019). Item models can be re-used, particularly when only a small number of the generated items are used on a specific test form, which, again, could yield economic benefits. Item models can also be adapted for different languages to produce items that can be used in different countries and cultures.

Fourth, AIG is a flexible approach to content production. Knowledge is fluid and dynamic (Nakakoji & Wilson, 2020; OECD, 2018). In the health sciences, for example, the creation of new drugs, the development of new clinical interventions, and the identification of new standards for best practice means that test content in the health sciences is constantly changing (Karthikeyan et al., 2019; Norman, Eva, Brooks, & Hamstra, 2006; Royal, Hedgpeth, Jeon, & Colford, 2017). These changes are difficult to accommodate in a flexible manner when the item is the unit of analysis because each part of the item is fixed. For example, if the standard of best practice shifted to conclude that a certain antibiotic is no longer effective for managing a specific presentation of fever after surgery, then all items directly or indirectly related to antibiotic treatment, fever, and surgery would need to be identified and modified or deleted from an item bank in order to reflect the most recent standard of best practice for antibiotic treatment. But when the model is the unit of analysis with fixed and variable parts, knowledge can be easily and readily updated to accommodate changes by modifying, updating, or eliminating content in the model. Even the task of identifying items that must be updated is made more manageable because only the small pool of item models rather than the large number of test items needs to be scrutinized and then updated to accommodate for the required changes.

Fifth, AIG can be used to enhance test security (Wollack & Fremer, 2013). Security benefits can be implemented by decreasing the item exposure rate through the use of larger numbers of items. In other words, when item volume increases, item exposure decreases because a large bank of operational items is available. Security benefits can also be found in the item assembly step of the AIG workflow because the content in an item model is constantly manipulated and, hence, varied.

This ability to modulate content makes it challenging for examinees to memorize and reproduce items because of the size, depth, and diversity of the bank.

< Prev   CONTENTS   Source   Next >