Putting It All Together to Generate Test Items: Overview

Table of Contents:

The purpose of this chapter is to draw on the ideas and concepts presented in the previous chapters and provide a succinct guide for SMEs and practitioners on how to generate test items. Each example is divided into four sections, which correspond to Chapters 2-5, respectively. Each section begins with a description of the purpose and is then followed by how the purpose can be implemented using our math and medical examples, which were first introduced in Chapter 2. This chapter also illustrates the AIG workflow, which is capable of serving as an item production system. We use the 1 -layer example for our demonstration where we begin with the parent item and end with a set of generated items.

Mathematics Example Using the Logical Structures Model

Cognitive Model Development

In our math example, we covered several related math problems. The general problem is range and ratio, while solving this general problem requires the examinee, first, to recognize the requested value in a given set of values and, second, to calculate the ratios. To create a cognitive model, we will start by selecting a parent item, and we evaluate the possible variations that can be produced from the item:

Yesterday, a veterinarian treated 2 birds, 3 cats, 6 dogs. What was the ratio of the number of cats treated to the total number of animals treated by the veterinarian?

  • (A) 1 to 4
  • (B) 1 to 6
  • (C) 1 to 13
  • (D) 3 to 8
  • (E) 3 to 11*

The content domain, which is described as the problem in the top panel of the cognitive model, is the range and ratio for this item. We can then identify the variables in the parent item that will allow us to vary ranges and ratios.

Yesterday, a veterinarian treated 2 birds, 3 cats, 6 dogs. What was the ratio of the number of cats treated to the total number of animals treated by the veterinarian? 3 to 11

The resulting item structure can be presented as follows:

Yesterday, a veterinarian treated birds, cats, dogs. What was the treated by the veterinarian?

Next, we can create a list of possible scenarios for this model. These scenarios are specified using the problem element.

  • 1. number of birds
  • 2. number of birds and cats
  • 3. total number of animals
  • 4. ratio of the number of cats treated to the total number of animals

Each of the scenarios will have a matching answer value.

+ <12 > +<12>+<13>

to +<12>+

We also define the range of values that will be used for the generated items. In our example, we use a range of single-digit values from 2 to 8 in increments of 1 for the 11, I2, and I3 elements. We do not include the value 1 in order to avoid the need for singular wording in the stem.

  • 2 ... 8 by 1
  • 2 ... 8 by 1
  • 2 ... 8 by 1

Item Model Development

Once the cognitive model is specified and the content in the model is organized and coded, we can proceed to the item modelling step. Item modelling is the foundation of AIG using a template-based method. The item model is used to format the content identified in the cognitive model. For the logical structures cognitive model, the format includes math expressions that will be calculated during the generation process. In addition to defining the contents of the item, our example requires an extra hidden variable called SCENARIO, which is used to simplify our content coding.

Stem

Yesterday, a veterinarian treated [11] birds, [12] cats, [13] dogs. What was the [PROBLEM] treated by the veterinarian?

Elements

SCENARIO: (1) A, (2) В, (3) C, (4) D PROBLEM: (1) number of birds, (2) number of birds and cats, (3) total number of animals, (4) ratio of the number of cats treated to the total number of animals

11: 2 to 8, by 1 12: 2 to 8, by 1 13: 2 to 8, by 1

Key
  • 1. [11]
  • 2. [ [11] + [12] ]
  • 3. [ [11] + [12] + [13] ]
  • 4. [12] to [[11]+[12]+[13]]

Item Generation Using Constraint Coding

Our example is quite straightforward, and, as a result, the relations for the key are relatively simple.

1.

Scenariol

2 .

+<12>

Scenario2

3 .

+<12>+<13>

Scenario3

4 .

to +<12>+<13>

Scenario4

The final constraint matrix can be presented as follows:

Element

Scenario

13

Problem

11

12

Element

Value

A

В

C

D

0

1

2

3

4

5

6

7

8

number of birds

number of birds and cats

total number of animals

ratio of the number of cats treated to the total number of animals

2

3

4

5

6

7

8

2

3

4

5

6

7

8

Scenario

A

1

1

1

1

1

1

1

1

1

1

1

1

1

1

.0

0

0

1

1

1

1

1

1

1

1

1

1

1

1

1

1

В

1

1

1

1

1

1

1

1

1

1

1

1

1

0

1

0

0

1

1

1

1

1

1

1

1

1

1

1

1

1

1

C

1

1

1

1

1

1

1

1

1

1

1

1

1

0

0

1

0

1

1

1

1

1

1

1

1

1

1

1

1

1

1

D

1

1

1

1

1

1

1

1

1

1

1

1

1

0

0

0

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

13

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

2

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

3

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

4

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

5

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

6

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

7

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

8

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

Element

Scenario

13

Problem

11

12

Problem

number of birds

1

0

0

0

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

number of birds and cats

0

1

0

0

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

total number of animals

0

0

1

0

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

ratio of the number of cats treated to the total number of animals

0

0

0

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

11

2

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

3

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

4

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

5

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

6

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

7

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

8

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

I2

2

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

3

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

4

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

5

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

6

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

7

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

8

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

Systematic Distractor Generation

Our first example uses the logical structures model. The distractors are defined based on common errors, as described in Chapter 5, Table 5.1. Also, we have four scenarios in our example. As a result, we will need to define separate lists of distractors for each scenario. We require four distractors for each scenario. For clarity, we duplicate values in the following list to demonstrate that we can generate distractors for each scenario:

1 .

Scenariol

2 .

Scenariol

3 .

+<12>+<13>

Scenariol

4 .

+<12>

Scenariol

5 .

Scenario2

6 .

Scenario2

7 .

+<13>

Scenario2

8 .

<12>+<13>

Scenario2

9 .

Scenario3

10 .

+<13>

Scenarios

11 .

<12>+<13>

Scenarios

12 .

+<12>

Scenarios

13 .

to +<12>+<13>

Scenario4

14 .

1 to +<12>+<13>

Scenario4

15 .

1 to

Scenario4

16 .

to +<13>

Scenario4

Once we have defined the distractors, we can produce the item model in its final form:

Stem

Yesterday, a veterinarian treated [11] birds, [12] cats, [13] dogs, and no other animals. What was the [PROBLEM] treated by the veterinarian?

Elements

SCENARIO: (1) A, (2) В, (3) C, (4) D PROBLEM: (1) number of birds, (2) number of birds and cats, (3) total number of animals, (4) ratio of the number of cats treated to the total number of animals

11: 2 to 8, by 1 12: 2 to 8, by 1 13: 2 to 8, by 1

Key

(1) [11], (2) [[I1] + [I2]], (3)

[ [Il] + [12] + [I3] ], (4) [12] to [ [11] + [I2] + [13] ]

Distractors

(I) [11], (2) [12], (3) [13], (4) [[I1] + [I2]], (5) [[I2] + [I3]], (6) [ [11] + [I3]], (7)

[ [Il] + [12] + [I3] ], (8) [11] to [ [11] + [12] + [13] ], (9) 1 to [[I1] + [I2] + [I3]], (10) 1 to [12],

(II) [12] to [ [11] + [13] ]

A Sample of Generated Math Items

Using the method presented in this chapter, 876 math items were generated. When the second layer was added to the math example (see Chapter 5, Table 5.4), a total of 8,394 items were generated. A sample of the generated content from the 1-layer math model created using the example in this chapter includes the following items:

1 Yesterday, a veterinarian treated 6 birds, 5 cats, and 2 dogs. What was the number of birds and cats treated by the veterinarian?

A. 5

B. 6

C. 7

D. 8

E. 11 *

2 Yesterday, a veterinarian treated 2 birds, 5 cats, and 7 dogs. What was the total number of animals treated by the veterinarian?

A. 2

B. 7

C. 9

D. 12

E. 14 *

3 Yesterday, a veterinarian treated 7 birds, 5 cats, and 4 dogs. What was the ratio of the number of cats treated to the total number of animals treated by the veterinarian?

A. 1 to 5

B. 1 to 16

C. 5 to 11

D. 5 to 16 *

E. 7 to 1 6

4 Yesterday, a veterinarian treated 6 birds, 4 cats, and 2 dogs. What was the number of birds treated by the veterinarian?

A. 2

B. 4

C. 6 *

D. 10

E. 12

5 Yesterday, a veterinarian treated 8 birds, 4 cats, and 6 dogs. What was the total number of animals treated by the veterinarian?

A. 8

B. 10

C. 12

D. 14

E. 18 *

6 Yesterday, a veterinarian treated 6 birds, 3 cats, and 8 dogs. What was the ratio of the number of cats treated to the total number of animals treated by the veterinarian?

A. 1 to 3

B. 1 to 17

C. 3 to 14

D. 3 to 1 7 *

E. 6 to 1 7
 
Source
< Prev   CONTENTS   Source   Next >