Item Model Development: Template-Based AIG Using Item Modelling

With the content identified in step 1 as outlined in the previous chapter, the next step is to position this content in an item model. An item model is a template of the assessment task that specifies the parts and content in the task that will be manipulated to create new test items. Item models provide the foundation for template-based AIG. Item models (LaDuca, Staples, Templeton, & Holzman, 1986; see also Bejar, 1996, 2002; Bejar, Lawless, Morley, Wagner, Bennett, & Revuelta, 2003) have been described using different terms, including schemas (Singley & Bennett, 2002), blueprints (Embretson, 2002), templates (Mislevy & Riconscente, 2006), forms (Hively, Patterson, & Page, 1968), frames (Minsky, 1974), and shells (Haladyna & Shindoll, 1989). Item models are created by the SME. These models are cast as templates that specify where the content from the cognitive modelling step should be placed to create test items. While the cognitive model focuses on the organization of content in the knowledge domain, the item model focuses on the organization of how this content will be presented in a test item. In other words, item models are created by the SME to provide a specific structure that will be used to generate items. Item models identify the parts of an assessment task that can be manipulated for item generation. These parts include the stem, the options, and any other auxiliary information that will be presented with the item. The stem contains context, content, and/or questions the examinee is required to answer. The options include a set of alternative answers with one correct option and one or more incorrect options or distracters. Auxiliary information includes any additional content, in either the stem or option, required to generate an item, including text, images, tables, graphs, diagrams, audio, and/or video. The stem and correct option are generated for the constructed-response item format. The stem, correct option, and incorrect options are generated for the selected-response (e.g., multiple-choice) item format. The stem and options can be further divided into elements. Elements were first introduced in Chapter 2. Elements contain values for each feature in the bottom panel of the cognitive model that can be manipulated for item generation. Values are denoted as strings, which are non-numeric content or integers which are numeric content.

Layers in Item Models

Item models contain layers of information. Item models in AIG are often specified as either 1 or n-layer (Lai, 2013; see also Gierl & Lai, 2013). The goal of item generation using the 1-layer item model is to produce new items by manipulating a small number of elements at a single layer. The simplicity of this model makes it a popular choice for AIG because it is relatively easy to implement. We use element as the unit of analysis in our description because it is the most specific variable in the cognitive model that is manipulated to produce new items. Often, the starting point for 1-layer modelling is to return to the parent item used to create the cognitive model for AIG in Chapter 2. The parent item for the logical structures math cognitive model presented in Chapter 2 is shown at the top ofTable 3.1. The parent highlights the underlying structure of the item, thereby providing a point of reference for creating alternative items. Then an item model is created from the parent by identifying elements that can be manipulated to produce new items. The item model identifies each part of the model required for item generation. In this example, the item model contains the stem, which identifies each feature in a square bracket (i.e., Yesterday, a veterinarian treated [11] birds, [I2] cats, [I3] dogs. What was the ratio of the number of cats treated to the total number of animals treated by the veterinarian?) and the elements of the features (i.e., three integer values—11, I2, I3—beginning with the value 2 and ending with the value 8 using increments of 1). For item models based on the logical structures cognitive model, the solution for the correct option is presented as the key (i.e., [I2] to [[11] + [I2] + [I3] [).

Table 3.1 1 -Layer Logical Structures Mathematics Item Model

Parent Item:

Yesterday, a veterinarian treated 2 birds, 3 cats, 6 dogs. What was the ratio of the number of cats treated to the total number of animals treated by the veterinarian?

  • (A) 1 to 4
  • (B) 1 to 6
  • (C) 1 to 13 (Dj 3 to 8 (E) 3 to 11

Item Model:

Stem

Yesterday, a veterinarian treated [11] birds, [I2] cats, [I3] dogs. What was the ratio of the number of cats treated to the total number of animals treated by the veterinarian?

Element

[И ] Range: 2 to 8 by 1

[12] Range: 2 to 8 by 1

[13] Range: 2 to 8 by 1

Key

|I2] to | [11 ] + [I2] + [I3| J

Item models based on the key features cognitive model are formatted in the same manner, with one important difference: the key. The parent item for the key features medical cognitive model presented in Chapter 2 is shown at the top of Table 3.2. In this example, the item model contains the stem which identifies each feature in a square bracket (i.e., A [Age]-year-old female sees her doctor and reports that she's been experiencing a [Cough Type] cough and [Body Aches] that have developed [Onset], Upon examination, she presents with an oral temperature of [Temperature]. What is the most likely diagnosis) and the elements of the features (i.e., age with eight integer values [18 to 30 in increments of 1], cough type with three string values [mild, hacking, severe], body aches with four string values (slight body aches, slight body pains, severe body aches, severe body pains), onset with three string values [over a few days, within three to six hours, suddenly], temperature with four integer values [37°C, 37.8°C, 39°C, 39.5°C]). The correct option is presented as the key (i.e., common cold and seasonal flu). Notice, however, that the solution for the key is not presented in this model. The key helps differentiate the logical structures and key features cognitive models. Recall that the

Table 3.2 1-Layer Key Features Medical Item Model

Parent Item:

A 22-year-old female sees her doctor and reports that she's been experiencing a mild cough and slight body aches that have developed over a few days. Upon examination, she presents with an oral temperature of 37°C. What is the most likely diagnosis?

1: Hay Fever 2: Ear Infection 3: Common cold 4: Acute Sinusitis 5: Seasonal Influenza

Item Model:

Stem

A [Age]-year-old female sees her doctor and reports that she's been experiencing a [Cough Type] cough and [Body Aches] that have developed [Onset]. Upon examination, she presents with an oral temperature of [Temperature]. What is the most likely diagnosis?

Element

Age: 18-30, by 1

Cough Type: 1. mild, 2. hacking, 3. severe

Body Aches: 1. slight body aches, 2. slight body pains, 3. severe body aches, 4. severe body pains

Onset: 1. over a few days, 2. within 3-6 hours, 3. suddenly Temperature: 1. 37°C; 2. 37.8°C; 3. 39°C; 4. 39.5°C

Key

Common cold; Seasonal flu

logical structures cognitive model is used to measure the examinees' ability to apply a concept with different types of content (i.e., values in the elements). The concept is often used to implement a formula, algorithm, and/or logical outcome. This concept is presented as the key in the item model. The defining characteristic of this modelling approach is that the content for the item can vary, but the concept remains fixed across the generated items. The key features cognitive model, on the other hand, is most suitable for measuring the examinees' ability to assemble and apply key features. The defining characteristic of this modelling approach is that the content (i.e., values in the elements) can vary, as with the logical structures model, but also that the key concept varies across the generated items due to the meaningful combination of features, which is unlike the logical structures model. The systematic combination of permissible features is defined by the constraints specified in the feature panel of the cognitive model. Hence, the constraints needed to uniquely combine different values to produce a correct response—and there can be many combinations for one correct response—must be described in the cognitive model.

The main disadvantage of using a 1-layer item model for AIG is that relatively few elements can be manipulated. The manipulations are limited because the number of potential elements in a 1-layer item model is relatively small (i.e., the number of elements is fixed to the total number of words in the stem). By restricting the element manipulations to a small number, the generated items may have the undesirable quality of appearing too similar to one another. In our experience, generated items from 1-layer models are referred to pejoratively by many SMEs as "clones". Cloning, in a biological sense, refers to any process in which a population of identical units is derived from the same ancestral line. Cloning occurs in item modelling if we consider it to be a process where specific content (e.g., nuclear DNA) in a parent item (e.g., currently or previously existing animal) is manipulated to generate a new item (e.g., new animal). Through this process, instances are created that are very similar to the parent because the information is purposefully transferred from the parent to the offspring. Clones are perceived by SMEs to be generated items that are easy to produce. Clones are often seen as simplistic products from an overly simple item development process. Most importantly, clones are believed to be easily recognized by test preparation companies, which limits their usefulness in operational testing programs because items can then be studied before the test administration. In short, items generated from 1 -layer models are viewed by many SMEs as easily produced, overly simplistic, and clearly detectable. As a result, SMEs are often skeptical of items thought to be clones produced from the 1-layer item model.

 
Source
< Prev   CONTENTS   Source   Next >