The Contribution of Distractors in the Selected-Response Item Format

Thissen, Steinberg, and Fitzpatrick published a paper in 1989 titled "Multiple-Choice Models: The Distractors Are Also Part of the Item" to draw attention to the importance of the incorrect option for the selected-response item format. They claimed that most SMEs believe that the stem and the correct option serve as the most important part of the item. The AIG method we have presented up to this point in the book permit SMEs to generate the stem and the correct option. But to produce selected-response items, the incorrect options or distractors must also be created. The distractors are an equally important part of this item format. The distractors may even require more time and resources to create compared to the stem and the correct option. For each selected-response item, one stem and one correct option are required, along with two (i.e., three-option item), three (i.e., four-option item), or four (i.e., five-option item) plausible but incorrect options. When many items are needed, distractor development becomes a formidable task. For example, when 100, five-option, selected-response items are created, the SME is required to create 100 stems, 100 correct options, and 400 distractors. In addition, distractors form an important part of the context required to solve a selected-response item and, as a result, distractors affect the psychometric properties (e.g., difficulty, discrimination) of each item. A complex and unpredictable relationship exists between the correct and incorrect options because examinees are required to make a distinction among response options in order to select the correct response (Hambleton & Jirka, 2006). This complex relationship is fostered by the effects of partial knowledge in response performance that, in turn, interacts with the plausibility of each distrac- tor and thus influences the psychometric properties of both the correct option and each of the distractors (e.g., Bock, 1972; Haladyna, 2016; Haladyna & Rodriguez, 2013; Penfield, 2008, 2010; Shin, Guo, & Gierl, 2019; Thissen et al., 1989).

Traditional Approach for Writing Distractors

With the overwhelming popularity and prevalence of the selected-response format, testing organizations are required to create large numbers of diverse, high-quality, selected-response items both quickly and efficiently. The most common approach that has consistently been described and advocated for distractor development focuses on creating a list of plausible but incorrect alternatives linked to common misconceptions or errors (Collins, 2006; de la Torre, 2009; Haladyna & Downing, 1989; Haladyna & Rodriguez, 2013; Moreno, Martfnez, & Muniz, 2006, 2015; Paniagua & Swygert, 2016; Rodriguez, 2011, 2016; Tarrant, Ware, & Mohammed, 2009; Vacc, Loesch, & Lubik, 2001). Misconceptions can be identified by looking at examinees' answers from constructed-response or open- ended items (e.g., Briggs, Alonzo, Schwab, & Wilson, 2006) or from studying examinee response processes using verbal reports (e.g., Haladyna & Rodriguez, 2013). If outcomes from these two procedures are not available, then SMEs can create a list of distractors based on their perception of the common errors or misconceptions related to the concepts in the item (Collins, 2006).

Unfortunately, this approach to distractor development has important weaknesses. For instance, the feasibility of collecting empirical data in the form of construct responses or verbal reports is limited, particularly when large numbers of diverse items must be created quickly. Misconceptions can also be anchored to judgmental results by asking the SME to identify common errors in thinking, reasoning, and problem solving. This judgmental approach is typically used in practice. But to successfully implement this approach to distractor development, three assumptions must be satisfied. First, lists of answers, algorithms, or rules can be identified by the SME for each test item. Second, plausible but incorrect distractors can be produced using these lists of answers, algorithms, or rules. Third, the misconceptions identified by the SME are, in fact, the same misconceptions held by the examinees (Gierl, 1997). Proper alignment of these assumptions is critical for creating distractors that elicit plausible misconceptions. Moreover, the alignment must occur for each distractor across every multiple-choice item. For example, if the task is to create an item bank, with 500,448 five-option, multiple-choice items, which was the goal for our ambitious SME in Chapter 1, then she needs to write 2,001,792 distractors that satisfy the three assumptions to yield plausible distractors for the items in her bank. Because of these limitations, writing distractors that measure plausible misconceptions and replicating this outcome consistently is an infeasible approach to distractor development for AIG.

Distractor development in AIG is also complicated by the fact that large numbers of plausible distractors are needed. Content in a multiple-choice item is constantly changing in a generative item development system. While a relatively small number of constraints are needed to ensure that information presented in the stem yields a correct response, this requirement must be counterbalanced with a much larger number of constraints that are needed to ensure the information presented in the distractors is plausible yet erroneous. This outcome helps explain why we presented the constructed- and selected-response item format separately in our book. The three-step method described in Chapter 2, Chapter 3, Chapter 4 guides the production of a set of constructed-response items with one correct option. But a substantial amount of additional modelling is needed to constrain and produce two, three, or four incorrect options that each item must present in the selected-response format. Moreover, the complexity and magnitude of the generative task also mean that the recommendations commonly presented for distractor development using a traditional approach cannot be used for generating distractors. Hence new methods are required.1 In short, generating selected-response items is complex. The method used for creating constructed-response items, as described in Chapter 2, Chapter 3, Chapter 4, must be used to produce selected-response items. But additional modelling for the distractors, as described in the next section, is also required.

< Prev   CONTENTS   Source   Next >