Item Generation: Approaches for Generating Test Items

Different approaches can be used to generate items. We adopt an item modelling approach, informed by a cognitive model, to leverage the benefits of binary computing to produce large numbers of test items in a controlled and flexible manner, as described in the previous chapter. But it is important to remember that different approaches can be used. We begin by reviewing three general approaches for generating items. Then we present our preferred approach by describing how plausible items are assembled with the use of logical constraints. This approach relies on a technique called bitmasking. Finally, we use bitmasking with the content from the mathematics and medical models developed in the previous chapters to demonstrate how logical constraints can be used to generate items.

The methods used to generate items can be categorized into three distinct areas: an instruction, an ontological, and a logical constraint approach. With the instruction-based approach (e.g., Arendasy & Sommer, 2012; Embretson & Kingston, 2018; Geerlings, Glas, & van der Linden, 2011; Geerlings, van der Linden, & Glas, 2012; Higgins, Futagi, & Deane, 2005; Singley & Bennett, 2002), a specific set of programming instructions are created to generate a specific set of items. This approach provides flexibility of expression for the SME to generate items in different content areas using different item formats. However, a limitation of this approach is the need to program each item model. The generation process requires the SME to provide instructions to the computer programmer for each item model, where the programmer's task is to express the problem in a generation format. This approach creates an extended workflow as the SME must express the problem in a language and format that the programmer can interpret and then implement. This approach also shifts the required time for item development from SMEs to programmers. An instruction-based approach can be used as a viable approach to generating items, but it is challenging to scale the item development process because each model must be programmed individually.

With the emergence of NLP techniques, non-template-based approaches can be used to generate items, as we described and illustrated in Chapter 3 (e.g., Danon & Last, 2017; Flor & Riordan, 2018; Gutl, Lankmayr, Weinhofer, & Hofler, 2011; von Davier, 2018; see also Leo, Kurdi, Matentzoglu, Parsi, Sattler, Forge, Donato, & Dowling, 2019; Mitkov & Ha, 2003). These ontological-based approaches generate test items by drawing on information that can be described in a corpus or knowledge base. As a result, they can be used to generate items without the use of templates and without intervention from the SME. These approaches are used to generate items based on representations described at the syntax, semantics, and sequence levels, meaning that novel items that may not have been considered by the SME can be generated from the structure of knowledge that exists in the corpus. While important developments in non-template-based AIG continue to emerge, this approach relies on the existence of corpora or knowledge bases that can reliably represent concepts and topics in the specific area of interest for the SME. Currently, knowledge bases that cover specific content areas and contain the depth of information needed to describe complex relationships that are suitable for the kinds of items used in modern testing programs are limited, at best.

A logical constraints approach provides a straightforward way of generating items with the use of iterations. Using this approach, the generated content is specified as an element ("element" is defined for a cognitive model in Chapter 2 and an item model in Chapter 3) in which each element contains all possible values to be displayed and substituted in each generated item. The presentation of the elements is organized in a cloze test format, where all values of an element can be displayed. Then all combinations of the element values are iteratively assembled. The total number of items that can be generated is a product of the maximum number of values in each element (Lai, Gierl, & Alves, 2010). But with logical constraints, not all combinations of the values will produce meaningful test items. To prevent implausible combinations, the constraints defined by the SME in the cognitive and item models are used to limit the generated outcomes to those combinations that are deemed to be meaningful. With the use constraints, the generation process can be described as an iterator that permutes through all combinations of elements and, in the process, eliminates combinations (i.e., meaningless generated items) that do not meet the constraint requirements. The logical constraints approach is flexible because it does not need specific computer programming for every item model as required with the instruction-based approaches. It can also be used to produce items using small amounts of content provided by the SME, thereby eliminating the need for corpora or knowledge bases as required with the ontological-based approaches.

< Prev   CONTENTS   Source   Next >