Putting It All Together to Generate Test Items: Overview
The purpose of this chapter is to draw on the ideas and concepts presented in the previous chapters and provide a succinct guide for SMEs and practitioners on how to generate test items. Each example is divided into four sections, which correspond to Chapters 25, respectively. Each section begins with a description of the purpose and is then followed by how the purpose can be implemented using our math and medical examples, which were first introduced in Chapter 2. This chapter also illustrates the AIG workflow, which is capable of serving as an item production system. We use the 1 layer example for our demonstration where we begin with the parent item and end with a set of generated items.
Mathematics Example Using the Logical Structures Model
Cognitive Model Development
In our math example, we covered several related math problems. The general problem is range and ratio, while solving this general problem requires the examinee, first, to recognize the requested value in a given set of values and, second, to calculate the ratios. To create a cognitive model, we will start by selecting a parent item, and we evaluate the possible variations that can be produced from the item:
Yesterday, a veterinarian treated 2 birds, 3 cats, 6 dogs. What was the ratio of the number of cats treated to the total number of animals treated by the veterinarian?
 (A) 1 to 4
 (B) 1 to 6
 (C) 1 to 13
 (D) 3 to 8
 (E) 3 to 11*
The content domain, which is described as the problem in the top panel of the cognitive model, is the range and ratio for this item. We can then identify the variables in the parent item that will allow us to vary ranges and ratios.
Yesterday, a veterinarian treated 2 birds, 3 cats, 6 dogs. What was the ratio of the number of cats treated to the total number of animals treated by the veterinarian? 3 to 11
The resulting item structure can be presented as follows:
Yesterday, a veterinarian treated
Next, we can create a list of possible scenarios for this model. These scenarios are specified using the problem element.
 1. number of birds
 2. number of birds and cats
 3. total number of animals
 4. ratio of the number of cats treated to the total number of animals
Each of the scenarios will have a matching answer value.
We also define the range of values that will be used for the generated items. In our example, we use a range of singledigit values from 2 to 8 in increments of 1 for the 11, I2, and I3 elements. We do not include the value 1 in order to avoid the need for singular wording in the stem.
 2 ... 8 by 1
 2 ... 8 by 1
 2 ... 8 by 1
Item Model Development
Once the cognitive model is specified and the content in the model is organized and coded, we can proceed to the item modelling step. Item modelling is the foundation of AIG using a templatebased method. The item model is used to format the content identified in the cognitive model. For the logical structures cognitive model, the format includes math expressions that will be calculated during the generation process. In addition to defining the contents of the item, our example requires an extra hidden variable called SCENARIO, which is used to simplify our content coding.
Stem 
Yesterday, a veterinarian treated [11] birds, [12] cats, [13] dogs. What was the [PROBLEM] treated by the veterinarian? 
Elements 
SCENARIO: (1) A, (2) В, (3) C, (4) D PROBLEM: (1) number of birds, (2) number of birds and cats, (3) total number of animals, (4) ratio of the number of cats treated to the total number of animals 11: 2 to 8, by 1 12: 2 to 8, by 1 13: 2 to 8, by 1 
Key 

Item Generation Using Constraint Coding
Our example is quite straightforward, and, as a result, the relations for the key are relatively simple.


1. 

Scenariol 
2 . 

Scenario2 
3 . 

Scenario3 
4 . 

Scenario4 
The final constraint matrix can be presented as follows:
Element 
Scenario 
13 
Problem 
11 
12 

Element 
Value 
A 
В 
C 
D 
0 
1 
2 
3 
4 
5 
6 
7 
8 
number of birds 
number of birds and cats 
total number of animals 
ratio of the number of cats treated to the total number of animals 
2 
3 
4 
5 
6 
7 
8 
2 
3 
4 
5 
6 
7 
8 
Scenario 
A 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
.0 
0 
0 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
В 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
0 
1 
0 
0 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 

C 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
0 
0 
1 
0 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 

D 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
0 
0 
0 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 

13 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 

2 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 

3 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 

4 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 

5 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 

6 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 

7 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 

8 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
Element 
Scenario 
13 
Problem 
11 
12 

Problem 
number of birds 
1 
0 
0 
0 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
number of birds and cats 
0 
1 
0 
0 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 

total number of animals 
0 
0 
1 
0 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 

ratio of the number of cats treated to the total number of animals 
0 
0 
0 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 

11 
2 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
3 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 

4 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 

5 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 

6 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 

7 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 

8 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 

I2 
2 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
3 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 

4 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 

5 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 

6 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 

7 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 

8 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
Systematic Distractor Generation
Our first example uses the logical structures model. The distractors are defined based on common errors, as described in Chapter 5, Table 5.1. Also, we have four scenarios in our example. As a result, we will need to define separate lists of distractors for each scenario. We require four distractors for each scenario. For clarity, we duplicate values in the following list to demonstrate that we can generate distractors for each scenario:


1 . 

Scenariol 
2 . 

Scenariol 
3 . 

Scenariol 
4 . 

Scenariol 
5 . 

Scenario2 
6 . 

Scenario2 
7 . 

Scenario2 
8 . 
<12>+<13> 
Scenario2 
9 . 

Scenario3 
10 . 

Scenarios 
11 . 
<12>+<13> 
Scenarios 
12 . 

Scenarios 
13 . 

Scenario4 
14 . 
1 to 
Scenario4 
15 . 
1 to 
Scenario4 
16 . 

Scenario4 
Once we have defined the distractors, we can produce the item model in its final form:
Stem 
Yesterday, a veterinarian treated [11] birds, [12] cats, [13] dogs, and no other animals. What was the [PROBLEM] treated by the veterinarian? 
Elements 
SCENARIO: (1) A, (2) В, (3) C, (4) D PROBLEM: (1) number of birds, (2) number of birds and cats, (3) total number of animals, (4) ratio of the number of cats treated to the total number of animals 11: 2 to 8, by 1 12: 2 to 8, by 1 13: 2 to 8, by 1 
Key 
(1) [11], (2) [[I1] + [I2]], (3) [ [Il] + [12] + [I3] ], (4) [12] to [ [11] + [I2] + [13] ] 
Distractors 
(I) [11], (2) [12], (3) [13], (4) [[I1] + [I2]], (5) [[I2] + [I3]], (6) [ [11] + [I3]], (7) [ [Il] + [12] + [I3] ], (8) [11] to [ [11] + [12] + [13] ], (9) 1 to [[I1] + [I2] + [I3]], (10) 1 to [12], (II) [12] to [ [11] + [13] ] 
A Sample of Generated Math Items
Using the method presented in this chapter, 876 math items were generated. When the second layer was added to the math example (see Chapter 5, Table 5.4), a total of 8,394 items were generated. A sample of the generated content from the 1layer math model created using the example in this chapter includes the following items:
1 Yesterday, a veterinarian treated 6 birds, 5 cats, and 2 dogs. What was the number of birds and cats treated by the veterinarian?
A. 5
B. 6
C. 7
D. 8
E. 11 *
2 Yesterday, a veterinarian treated 2 birds, 5 cats, and 7 dogs. What was the total number of animals treated by the veterinarian?
A. 2
B. 7
C. 9
D. 12
E. 14 *
3 Yesterday, a veterinarian treated 7 birds, 5 cats, and 4 dogs. What was the ratio of the number of cats treated to the total number of animals treated by the veterinarian?
A. 1 to 5
B. 1 to 16
C. 5 to 11
D. 5 to 16 *
E. 7 to 1 6
4 Yesterday, a veterinarian treated 6 birds, 4 cats, and 2 dogs. What was the number of birds treated by the veterinarian?
A. 2
B. 4
C. 6 *
D. 10
E. 12
5 Yesterday, a veterinarian treated 8 birds, 4 cats, and 6 dogs. What was the total number of animals treated by the veterinarian?
A. 8
B. 10
C. 12
D. 14
E. 18 *
6 Yesterday, a veterinarian treated 6 birds, 3 cats, and 8 dogs. What was the ratio of the number of cats treated to the total number of animals treated by the veterinarian?
A. 1 to 3
B. 1 to 17
C. 3 to 14
D. 3 to 1 7 *
E. 6 to 1 7