Rationale Generation: Creating Rationales as Part of the Generation Process

We noted in Chapter 1 that educational testing is in a noteworthy period of transition. CBT is replacing paper-based testing, thereby creating the foundation for the wide-spread use of technology-based systems. Technology-based systems allow educators to implement sophisticated test designs that support both formative and summative inferences (e.g., Choi & Zhang, 2019). The use of designs that permit educational tests to yield both formative and summative inferences is a noteworthy shift in practice. For much of the 20th century, educational measurement was synonymous with large-scale testing intended to satisfy demands for accountability. These tests were taken by all students within a specific context; they were administered under standardized conditions; they produced results that guided decisions. Summative assessments are administered at the end of a unit or sequence of instruction to provide a succinct measure of educational attainment at one point in time for the purpose of supporting decision makers. But in the 1990s, a shift began to occur in educational testing in which designs were used that yielded explicit evidence to help teachers monitor their instruction and to help students improve how they learn (e.g., Black & Wiliam, 1998; Sadler, 1989). Formative assessment is a process used by teachers and students during instruction to produce the feedback required to adjust teaching and improve learning so that students can better achieve the intended outcomes of instruction. Feedback has maximum value when it yields specific information in a timely manner that can direct instructional decisions designed to help each student acquire knowledge and skills more effectively.

Feedback has noteworthy characteristics (Bennett, 2011; Kluger & DeNisi, 1996; see also Black & Wiliam, 1998, 2010; Hattie & Timperley, 2007; Hwang & Chang, 2011; Popham, 2011). For instance, Shute (2008) published a comprehensive review in the journal Review of Educational Research in which she analyzed more than 100 articles related to formative feedback. The purpose of the review was to identify the features, functions, interactions, and links between formative feedback and learning, as well as to consolidate these findings into a set of guidelines for using formative feedback. Shute (2008) defined formative feedback as information communicated to learners so that they can modify their behavior to improve learning. She also provided recommendations on how to use formative feedback. Based on Shute's recommendations, we can identify the characteristics of feedback that are important in educational testing. Feedback should be focused on a specific task. When the feedback is elaborated (i.e., focused on what, how, why), it should include a rationale. The rationale should be presented as a small, manageable piece of text. This text should be specific, clear, and simple.

Feedback can be created as part of the generative process because the logic used to assemble the correct and incorrect options can also be used to identify the solution and to generate the rationale for the solution. Hence AIG can be used to produce the items, as well as the corresponding solution and rationale, required to solve each item. Rationale generation occurs by expanding the item modelling process in step 2.

Methods for Generating Rationales

Solutions and rationales for items generated with the logical structures and key features cognitive models can be added to the generation process in the item modelling step (Gierl & Lai, 2018). Three different types of solutions and rationales can be generated. We will illustrate how this method can be implemented using the logical structures and key features cognitive models in this chapter.

Correct Option

The first method is focused only on the correct option where the solution for each item is presented to the examinee. It is called the correct option method. Examinees are presented with their answers. They are also presented with the correct answer. This is the most straightforward method for providing examinees with feedback. It includes the solution, but it does not include a rationale for the solution. The 1 -layer logical structures math model first presented in Chapter 5 is shown in Table 10.1. Method 1 is at the top of the "Method" panel in Table 10.2. The examinee is presented with the correct option. It states, "Your selected [Option], The correct option is [Correct Option]". The 1-layer key features medical model presented in Chapter 5 is shown in Table 10.3. Method 1 is at the top of the "Method" panel in Table 10.4. The examinee is presented with the correct option. It states, "Your selected [Option]. The correct option is [Correct Option]".

Table 10.1 Logical Structures Mathematics Item Model

Parent Item:

Yesterday, a veterinarian treated 2 birds, 3 cats, 6 dogs. What was the ratio of the number of cats treated to the total number of animals treated by the veterinarian?

  • (A) 1 to 4
  • (B) 1 to 6
  • (C) 1 to 13 (Dj 3 to 8 (E) 3 to 11

Item Model:

Stem

Yesterday, a veterinarian treated [11] birds, [I2] cats, [I3] dogs. What was the ratio of the number of cats treated to the total number of animals treated by the veterinarian?

Element

[11 ] Range: 2 to 8 by 1

[12] Range: 2 to 8 by 1

[13] Range: 2 to 8 by 1

Key

Correct Option: [I2] to [ [11] + [I2] + [I3] ]

Distractor

[11] to[ [11] + [12] + [13] ] 1 to [12]

1 to [ [11] + [I2] + [I3] ]

[12] to [ [13] + [11 ] ]

Table W.2 Three Methods for Providing Feedback Using the Logical Structures

Mathematics Example

Method

Method 1 (Correct Option): You selected [Option]. The correct option is [Correct Option].

Method 2 (Correct Option With Rationale): You selected [Option]. The correct option is [Correct Option] because it is |I2] to [111 + |I2) + [13].

Method 3 (Correct Option and Distractor Rationale): The correct option is [Correct Option] because it is [12] to [11 ] + |I2) + [13]. The option you selected is not correct because [Distractor Rationale],

Distractor Rationale

  • 111] to [ |I1] + |I2] + [I3] ]—You missed the fact that the ratio of the number of cats is required to solve the problem and instead uses the count for the first animal on the list. As a result, the total value is the same as in the correct answer.
  • 1 to [I2]—You appear to misunderstand ratios. As a result, you selected a 1 to cats ratio because cats were mentioned in the stem.
  • 1 to [ [11 ] + [I2] + |I3[ ]—You appear to misunderstand ratios. As a result, you ignored the cats count for the ratio.

|I2] to [ |I3[ + |I1] ]—You appear to misunderstand the calculation for the total number in the ratio.

 
Source
< Prev   CONTENTS   Source   Next >