Assembling Content Codes in Item Generation

Once the content codes are defined in the item model, they can be assembled in the generation step. Applying the content codes to the generated items adds an extra dimension of data to the item modelling step. A combinatorial approach is used to iteratively assemble permissible combinations of elements by varying each value. The outcome is a new set of items. To generate metadata for each generated item, the content codes that are used for each model, element/value, and/or option are added together to produce a list. This list, in turn, includes all of the codes that were used in the item model coding process. As a result, multiple content codes served as the metadata that defined each generated item. Because different elements and values contribute to different content codes, a unique list of content codes is compiled for each generated item. Different types of rules for coding can also be created and then implemented during the assembly process. For example, content codes can be structured hierarchically, which means that element and/or value-level codes can be nested within model-level codes. As an example, if a generated item assesses division where division belongs to mathematics, then both division and mathematics are added to the content coding list for this item by simply including the division.

Content Coding Examples

Content coding provides the SME with a method for adding metadata to the item model, which, in turn, produces a list of content codes for each generated item. This list can be used to organize and manage a bank. It can also be used to search for items, select items, and assemble items for test forms. To demonstrate how content codes can be applied to the logical structures and key features models, we return to our math and medical examples first introduced in Chapter 2.

Logical Structures Mathematics Model

To begin the content-coding process, an extra dimension of information is added as a column in the item model. The item model content is presented in the first column, and the content coding is presented in the second column. The 1-layer item model with content coding for the logical structures math example is shown in Table 8.1. The first code in our example is defined at the model level for the stem. This model will be used to generate items that measure range and ratio in mathematics. These codes will be

Table 8.1 Logical Structures Mathematics Item Model With Content Coding

Item Model

Content Code

Stem

Yesterday, a veterinarian treated [11) birds, [I2] cats, [I3] dogs. What was the ratio of the number of cats treated to the total number of animals treated by the veterinarian?

Range and Ratio, Mathematics

Element

[111 Range: 2 to 8 by 1 [I2| Range: 2 to 8 by 1 [I3| Range: 2 to 8 by 1

Single Digit Single Digit Single Digit

Key

[12] to [ [11] + [12] + [13] 1

Present a value among a set of values in a ratio

Distractor

[11] to[ [11] + [12] + [13] ] 1 to [12]

1 to[ [11] + [I2] + [I3] ]

[12] to[ [13]+ [11] ]

Incorrect selection of values

Incorrect use of value as a ratio

Incorrect use of 1 as a value Incorrect sum of all values

applied to all items generated from this model. The second set of codes in our example are defined at the element and/or value level. In this simple model, the only element code is a single digit. The third set of codes in our example are defined for the options. The correct response was coded as "present a value among a set of values in a ratio". The incorrect options were also coded to align with erroneous formulas based on misconceptions about how to solve this item. The misconceptions included "incorrect selection of values", "incorrect use of value as a ratio", "incorrect use of 1 as a value", and "incorrect sum of all values". This description can be used by the SME to describe the response option. It can also be used as a succinct rationale to provide examinees with feedback. Using this approach, codes can be added to each of the incorrect options. An example of the content code list for one generated item is presented in Figure 8.1.

A generated item with content coding using the logical structures

Figure 8.1 A generated item with content coding using the logical structures

mathematics model

Key Features Medical Model

Content coding for the key features medical model uses a similar approach. The item model content is presented in the first column, and the content coding is presented in the second column. The 1-layer item model with content coding for the key features medical example is shown in Table 8.2. The first code in our example is defined at the model level for the stem. This model will be used to generate items that measure a clinical presentation with female patients. These codes will be applied to all items generated from this model. The second set of codes in our example are defined at the element and/or value level. In this model, five different elements—age, cough type, body aches, onset, and fever—are coded. The age is adult, cough type can be mild or severe, body aches can be slight or severe, onset can be gradual or sudden, and fever can be low or high. The third set of codes in our example are defined for the options. Each option is described using the presenting symptom. The correct option, common cold, for example, is identified with a mild cough, mild body aches, and fever. Similarly, the incorrect option, bronchitis, as an example, is identified with a severe cough, mild body aches, bedridden, and fever. An example of the content code list for one generated item is presented in Figure 8.2.

In summary, the sheer number of generated items that can be produced from a single item model warrants a novel approach to banking. Content coding is an effective method for adding metadata to the generated items through the item model in order to produce a descriptive list of data that can be used to manage the content in a bank. By applying codes to the item model as an additional layer of information, content can be generated for each item during the generation step. Our examples demonstrate how content codes can be appended to each generated item as a list. The metadata can be used to describe each generated item at the model, element and/or value, and options levels. This added layer of information can be used to search for items, select items, and assemble items for test forms, as well as organize and manage the bank.

Table 8.2 Key Features Medical Item Model With Content Coding

Item Model

Content Code

Stem

A [Age]-year-old female sees her doctor and reports that she's been experiencing a [Cough Type] cough and [Body Aches) that have developed [Onset], Upon examination, she presents with an oral temperature of [Temperature], What is the most likely diagnosis?

Clinical Presentation, Female

Element

Age: 18 to 30, by 1 Adult

Cough Type: (1) mild, (2) hacking, (3) severe

Body Aches: (1) slight body aches, (2) slight body pains, (3) severe body aches, (4) severe body pains

Onset: (1) over a few days, (2) within 3-6 hours, (3) suddenly

Fever: (1) 37°C, (2) 37.8°C, (3) 39°C, (4) 39.5°C

Adult

Mild (1,2) Severe (3)

Slight (1,2) Severe (3,4)

Gradual (1) Suddenly (2,3)

Low (1,2)

High (3,4)

Key

Common Cold Seasonal Flu

Mild cough, mild body ache, fever

Severe cough, severe body ache, bedridden, fever

Distr actor

Bronchitis

Streptococcal Infection

Hay Fever Otitis Media Acute Sinusitis Bacterial Pneumonia

Severe cough, mild body aches, bedridden, fever

Mild cough, mild body aches, bedridden, fever

Mild cough Fever

Mild cough, fever

Severe cough, severe body aches, Bedridden, fever

A generated item with content coding related to diagnosing the common cold and the seasonal flu

Figure 8.2 A generated item with content coding related to diagnosing the common cold and the seasonal flu

References

Cole, B. S., Lima-Walton, E., Brunnert, K., Vesey, W. B., & Raha, K. (2020). Taming the firehose: Unsupervised machine learning for syntactic partitioning of large volumes of automatically generated items to assist automated test assembly. journal of Applied Testing Technology, 21, 1-11.

Frank,). R., Snell, L., & Sherbino, J. (2015). CanMEDS 2015 Physician Competency Framework. Ottawa, ON: Royal College of Physicians and Surgeons of Canada.

)ago, C. (2009). A history of NAEP assessment frameworks. Paper commissioned for the 20th Anniversary of the National Assessment Governing Board. Washington, DC: National Assessment Governing Board.

National Assessment of Educational Progress (2019). Science Framework for the 2019 National Assessment of Educational Progress. Washington, DC.: National Assessment Governing Board.

Generating Alternative Item Types Using Auxiliary Information

 
Source
< Prev   CONTENTS   Source   Next >