What Is an Item Model?

An item model is a template of the assessment task that specifies the parts and content in the task that will be manipulated to create new test items. An item model provides the foundation for template-based AIG because it offers the SME tremendous flexibility. Item modelling allows SMEs to create content for tests that are administered at different levels in the education system, for a wide range of purposes, and for different subject areas using a range of item formats and item types. Item modelling allows SMEs to create diverse, complex, and high-quality items using an approach that is both efficient and cost-effective. Item modelling allows SMEs to incorporate advanced AIG methods into their item development practices, thereby permitting the use of n-layering, content coding, alternative item types, rationale generation, and multilingual item generation. For these reasons, we contend that template-based, AIG-using item modelling is currently the preferred approach for generating items.

How Do You Ensure That the Generated Items Are Diverse?

AIG is capable of producing large numbers of items from a single cognitive model. This outcome is sometimes interpreted to mean that the generated items from one model can be used to produce many tests. The items produced by our AIG method are anchored to the content specified in the problem and scenario panel of the cognitive model. Large cognitive models are commonly used in operational AIG applications to generate diverse items. Operational cognitive models typically address one problem with 4-7 different scenarios (top panel), 5-7 different sources of information (middle panel), and 8-12 different features (bottom panel). Diversity is further enhanced using n-layer item modelling. An n-layer item model produces new items by manipulating a large number of elements at two or more levels in the model. Diversity in the generated items can be produced using constraint coding. Constraint coding is the method by which logical constraints are used to implement the decisions, judgements, and rules described by the SMEs. Moreover, with the use of constraints defined in bitmasks in step 3 of the AIG workflow, both plausible content combinations that would be anticipated by the SMEs and some unanticipated but viable combinations are produced during the item assembly process, which further increases the diversity of the generated items. Finally, diversity can be infused into the generation process by including more dis- tractors in the item model when the selected-response format is used. Increasing the list of distractors will always increase both the number and the diversity of the generated items. The level of item heterogeneity can be evaluated by calculating the CSI using a sample of the generated items.

At this point, we must return to our initial claim. The items produced by our AIG method are anchored to the content specified in the problem and scenario panel of the cognitive model. This outcome means that regardless of whether the strategies we suggest for generating diverse items (i.e., large cognitive models, n-layering, constraint coding, large distractors lists) are used to produce 500, 5,000, or even 500,000 items, these items will all be related to the problem and scenarios in the cognitive model. In other words, if 500,000 items are generated using a cognitive model with one problem and two scenarios, then those 500,000 items will all be related to the same single problem with two scenarios. Diversity runs against a diminishing return with effort. Increasing the complexity of a cognitive model will undoubtedly increase the diversity of the generated items. But a complex cognitive model will also require more content with additional constraints and a more comprehensive review. The alternative is to divide the complex cognitive model into smaller, more manageable models where the generated items at the end of the process can be consolidated in order to achieve the goal of producing diverse outcomes.

How Should Generated Items Be Scored?

In the first part of our book, we presented a method for generating items using the constructed- and selected-response formats. In the second part of the book, we presented alternative item formats that could be used for generating items. But item development—whether it be traditional or technology enhanced—is merely one small part of a much larger test development process. Lane, Raymond, Haladyna, and Downing (2016) described 12 separate components for test development in their introductory chapter to the Handbook of Test Development (2nd Edition). AIG uses an augmented intelligence workflow that combines the expertise of the SME with the power of modern computing to produce test items. AIG occurs in the item writing and review subcomponent, which is within the item development component, where item development is component 4 of 12. Therefore, it is important and humbling to note that our book only addresses one component out of 12 described by Lane et al. The other 11 components, which include important processes, such as test design and assembly, test production, reporting, and test security, are all outside the purview of item development and, therefore, beyond the scope of our book. Scoring is one of the important processes identified by Lane et al. (2016). They present it as component 8 of 12. This means that three more components are implemented (i.e., test design and assembly, test production, test administration) after item development but before scoring. All of these components are complex, and the outcome from each component must be coordinated and carefully integrated into the test development process before a generated item (i.e., component 4 of 12) can be scored (i.e., component 8 of 12). This outcome also means that issues related to scoring cannot be addressed until more information about the particular test design and assembly, test production, and test administration processes is available because many different scoring options could be implemented using the AIG items. In short, creating scoring solutions for the generated items does not deviate from the structured test development process, and these solutions should be designed accordingly.

< Prev   CONTENTS   Source   Next >