Distractor Pool Method With Random Selection

The second method for generating distractors is used when the elements used to create the correct option are first specified. Using this logic, a distractor can be created as an element that contains content that is related to the correct option because it shares some but not all of the features of the correct option. To identify the content for the element in the cognitive model, the SME identifies a list of plausible but incorrect options that are appropriate for all possible items generated for a given cognitive model. Then distractors are randomly selected from this pool of plausible but erroneous values and added to each generated item. Hence it is called the distractor pool method with random selection. It is based on the assumption that a pool of equally plausible distractors can be created. A sample of these plausible distractors is selected at random to complete the item generation process.

The distractor pool method with random selection requires that the SME identify elements that are related to the correct option. For this reason, this distractor generation method is most suitable for the key features cognitive model. To generate distractors for this cognitive model, information must first be added to the model that is related to the features that distinguish the correct option from the distractor. Chapter 2, Table 2.4, lists the symptoms of the common cold and seasonal flu. The symptoms associated with the flu include fever, severe cough, chest discomfort, severe body aches and pains, bedridden, severe headache, sore throat, and stuffy, runny nose. This set of features is associated with the correct option—flu— but these features can also be associated with incorrect options. The cold, for instance, is associated with the features of sore throat and stuffy, runny nose but is typically not associated with fever. Hence the common cold could serve as one plausible but erroneous distractor because its features are related to some but not all of the features of the seasonal flu. In other words, the common cold is a potentially attractive distractor for an examinee with partial knowledge who is trying to diagnose the flu because both diagnoses share some important features. A representation of the distractor model using key features is provided in Figure 5.1. Using this logic, a list of five possible distractors can be created. Table 5.5 provides a list of symptoms for bronchitis, streptococcal infection, hay fever, and otitis media, demonstrating how these incorrect options can be used as distractors because they share some but not all of the features associated with the cold and flu. Also, by listing the distractors, a complete 1-layer item model for the medical example which includes both the correct and incorrect options can be created (see Table 5.6). The complete n-layer medical model includes the distractors presented in Table 5.7. The feature set that related the symptoms to the management distractors used in the n-layer model is shown in Table 5.8.

To implement the distractor pool method with random selection, the SME identifies options that share some but not all of the features of the

Figure 5.1 A model for creating plausible distractors using the key features

cognitive model

Table 5.5 Distractors Sharing Symptoms With the Common Cold and Seasonal Flu

correct option (see Tables 5.5 and 5.8). The distractors are then randomly selected from this pool and added to each generated item. The strength of this method is its simplicity. It is a straightforward method for producing a unique distractor set for each generated item. The weakness of this method resides with the assumption required for its use. It must be assumed that all pooled distractors are equally plausible for all generated items. Equal plausibility is often a strong assumption. The distractor pool method with random selection is most appropriate for case-specific item generation, where a well-defined set of distractors are known to be plausible because of their consistent relationship with one correct option.

Systematic Distractor Generation

A third method for distractor generation is also available. It is called systematic distractor generation. It is closely linked to the cognitive modelling step described in Chapter 2 and builds on the logic presented for the distractor pool method with random selection. Specific information must be collected in the form of a cognitive model to generate an item stem and correct option (i.e., step 1 in the three-step AIG method). Systematic distractor generation also involves specifying information related to errors in the form of a cognitive model in order to generate plausible but incorrect options. The process of modelling incorrect options is more complex

Table 5.6 1-Layer Key Features Medical Item Model With Distractors

compared to modelling the correct option because the item stem is constantly changing in a generative item development system. As a result, the small number of constraints required to ensure that information presented in the stem yields a correct response must be counterbalanced with a much larger number of constraints required to ensure the information presented in the incorrect options is erroneous but plausible. Also, more than

Table 5.7 2-Layer Key Features Medical Item Model With Distractors

Table 5.8 Symptoms Related to Management Distractors

A, Send the patient home to rest; B, Prescribe an oral antiviral medication; C, Prescribe a corticosteroid inhaler; D, Prescribe an oral antibiotic medication; E, Suggest an over-the- counter allergy medication; F, Suggest an over-the-counter cough suppressant, C, Suggest the over-the-counter supplement Echinacea, H, Send the patient for a chest x-ray.

one incorrect option is required for a multiple-choice item. Two, three, or four incorrect options must be produced for each generated item.

Systematic distractor generation requires the SME to identify elements that are related to the correct option. These elements are specified as a cognitive model because each element is also related to every distractor. For this reason, this distractor generation method is most suitable for the key features cognitive model. To generate distractors for this cognitive model, information is added to the model that is related to the features that distinguish the correct option from each distractor. Table 5.9 provides a list of symptoms for acute sinusitis and bacterial pneumonia. In this example, acute sinusitis was selected because it is a good distractor for the cold but not the flu because of its key features. Similarly, bacterial pneumonia was selected because it is a good distractor for the flu but not the cold. If the distractor pool method with random selection were used, acute sinusitis and bacterial pneumonia would have an equally likely chance of being selected as a distractor for the cold and flu. But using the logic of systematic distractor generation, acute sinusitis will only appear when the cold is the correct option and bacterial pneumonia will only appear when the flu is the correct option based on the key features of the correct and incorrect options. With systematic distractor generation,

Table 5.9 Distractors Sharing Symptoms With Common Cold or Seasonal Flu

the features of each distractor as they relate to the correct option must be coded, resulting in a larger list of distractors. Some of these distractors may be equally plausible for all of the correct options because of their key features while other distractors may only be plausible for one of the correct options. The use of this approach allows us to overcome the limitation of the distractor pool method with random assignment because we no longer need to assume that the distractors are equally plausible. As a result, we can create a larger list of distractors, thereby increasing the generation capacity and diversity of the model. Because the distractors are also linked to the correct response, it means that some of the distractors will only appear with one correct option while other distractors may appear with all of the correct options, thereby producing a more heterogeneous set of generated items. For these reasons, systematic distractor generation provides better control over how incorrect options will be presented in a generated item and is the method of choice for creating incorrect options when using the key features cognitive model. It provides the most customizable approach for distractor generation using key features.

Systematic distractor generation has many strengths. For example, it can be used in diverse testing situations and across many content areas because of its generality. It is an adaptive method because distractors are selected conditionally using the associate information in the stem and the correct option. The features that lead to the correct and incorrect options are also explicitly identified. Therefore, SMEs can scrutinize the features that produce distractors as part of the validation process. Systematic distractor generation also permits the SME to control the presentation of the options. Constraints can be added in order to control the appearance of options in an item. For example, the correct option can be constrained so that it is not consistently presented as the largest or longest option. The option list can also be constrained so that the alternatives are ordered from the smallest to largest value or the shortest to longest text. This method also has weaknesses. Systematic distractor generation requires that specific information be collected for each distractor (seeTables 5.5 and 5.8). Because selected-response items contain two or more incorrect options, this specific information is required for each item in order to model the distractors. The collection of this additional information requires both time and effort on the part of the SME.