Item Generation With 1-Layer Models

One early attempt to address the problem of generating cloned items was described by Gierl, Zhou, and Alves (2008). They developed a taxonomy of 1 -layer item model types. The purpose of this taxonomy was to provide SMEs with design guidelines for creating item models that yield different and varied types of generated items. Gierl et al.'s (2008) strategy for producing diversity was to systematically combine and manipulate those elements in the stem and options typically used for item model development. Their taxonomy included three parts: the stem, options, and auxiliary information. The stem is the section of the model used to formulate context, content, and/or questions. It contains four categories, as shown in Figure 3.1. Independent indicates that the n. elements in the stem are independent or unrelated to one another. That is, a change in one element will have no effect on the other stem elements in the item model. Dependent indicates n_d element(s) in the stem are dependent or directly related to one another. Mixed independent/dependent include

Figure 3.1 Categories in the item model

Table 3.3 Plausible Stem-by-Option Combinations in the Item Model Taxonomy

both independent and dependent elements in the stem. Fixed represents a constant stem format with no variation or change. The options contain the alternatives for the item model when the selected-response format is used. The options contain three categories. Randomly selected options refer to the manner in which the incorrect options or distractors are selected from their corresponding content pools. The distractors are selected randomly. Constrained options mean that the correct option and the distractors are generated according to specific constraints, such as formulas or calculations. Fixed options occur when both the correct option and distractors are invariant or unchanged in the item model.

By crossing the stem and options categories, a matrix of item model types can be produced. The stem-by-options matrix is presented in Table 3.3. Ten functional combinations are designated with the word "yes". The two remaining combinations are labelled as "no" because a model with a fixed stem and constrained options is an infeasible item type, and a model with a fixed stem and fixed options produces a single selected-response item type (e.g., a single multiple-choice item). This taxonomy is useful because it provides strategies for designing diverse 1 -layer item models by outlining their structure, function, similarities, and differences. It can also be used to ensure that SMEs do not design item models where the same elements are constantly manipulated or where the same item model structure is frequently used. Gierl et al. (2008) provided 20 different examples of item models created by using different stem-by-option combinations in the taxonomy to illustrate how generated items would vary in content areas such as mathematics, science, social studies, language arts, and architecture.

Using the Gierl et al. (2008) taxonomy, the Table 3.1 mathematics item model would be described as an independent stem with a constrained correct option. The Table 3.2 medical item model would be described as a mixed stem with constrained options. The age integer values in the stem are independent because they can assume any value while the cough type, body aches, onset, and temperature string values will depend on the combination of content in the item model. As a result, both independent and dependent elements are included in this example, making it mixed. The correct options are constrained by the combination of integer and string values specified in the stem, regardless of the age element.

n-Layer Item Models

A generalized expression for a 1 -layer item model requires multiple layers or n-layering. Multiple-layer models are used to generate content in an item that varies at two or more levels if the model where n denotes a value of two or more. For example, a 3-layer item model contains three layers of information. The goal of AIG using the n-layer item model is to produce items by manipulating a relatively large number of elements at two or more layers in the model. Much like 1-layer item modelling, the starting point for the n-layer model is to use a parent item. But unlike the 1-layer model where the manipulations are constrained to a linear set of generative operations using a small number of elements at a single level, the n-layer model permits manipulations of a nonlinear set of generative operations using elements at multiple levels. As a result, the generative capacity of the n-layer model is high. The concept of n-layer item generation is adapted from the literature on syntactic structures of language (Higgins, Futagi, & Deane, 2005; Jurafsky & Martin, 2009; McCarthy & Boonthum- Denecke, 2012). Language is often structured hierarchically, meaning that the content is often embedded in layers. This hierarchical organization can also be used as a guiding principle to generate large numbers of meaningful test items. The use of an n-layer item model is, therefore, a flexible template-based method for expressing different hierarchical or layered structures, thereby permitting the development of many different but feasible combinations of embedded elements. The n-layer structure can be described as a model with multiple layers of elements where each element can be varied simultaneously at different layers to produce different items. In the computational linguistic literature, our n-layer structure could be characterized as a generalized form of template-based natural language generation described by Reiter (1995) and Reiter and Dale (1997). The main disadvantage of n-layer item modelling is that the structures are more complex to create and evaluate. Hence n-layering requires much more experience and practice than 1-layer modelling in order to ensure the cognitive model content is correctly specified and constrained in order to generate meaningful items.