A Cautionary Note on Generating Solutions and Rationales

Creating solutions and rationales is challenging because this modelling approach needs to account for different types of feedback that can be provided to examinees. Also, when the rationales for models 2 and 3 are included, a clear explanation must be created for the generated statement. It is a skill that SMEs acquire over time and with practice because it requires a clear explanation. When creating solutions and rationales, we suggest that the SME follow Shute's recommendations. Feedback should be focused on a specific task. When the feedback includes a rationale, it should be presented as a small, manageable piece of text that is specific, clear, and simple. The solutions and rationales produced as part of an AIG workflow are intended to provide examinees with immediate feedback. This feedback should be used as a quick and concise review of test performance. Solutions and rationales are not intended as a replacement for teaching content, and they do not replace instruction.

Benefits and Drawbacks of Rationale Generation

Solution and rationale generation has many benefits. For example, it can be integrated into the three-step AIG method by adding a solution and rationale item model in step 2. The elements that lead to the correct and incorrect options can be explicitly identified and modelled and then used as feedback. Item generation relies on explicit dependencies that exist between the stem, the correct option, and the incorrect options through specific elements. These dependencies can be used to produce the correct response. Solution and rationale generation is also adaptive because the content for rationales is selected conditionally using relationships among the elements specified in the stem, the correct option, and the incorrect options. Consequently, the solutions and rationales created using this method will be plausible and appropriate for every generated case within an item model.

Solution and rationale generation also has drawbacks. Solution and rationale modelling requires that specific information be collected for all of the correct and incorrect options. This is a time-consuming process because data must be collected from SMEs, the data must be coded to identify the shared elements, and then computer programs must be written to assemble the content in the form of a solution and rationale using constraint logic programming. Moreover, when the key features cognitive model is used, the rationale may include a key features list or key features set. The rationale can focus on the correct option or it can include both the correct option and the distractors. Because of the large number of different outcomes that are possible, solution and rationale generation will add extra time to the three-step AIG method.


Bennett, R. E., (2011). Formative assessment: A critical review. Assessment in Education: Principles, Policy, & Practice, 18, 5-25.

Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education: Principles, Policy, & Practice, 5, 7-74.

Black, P., & Wiliam, D. (2010). Inside the black box: Raising standards through classroom assessment. Phi Delta Kappan, 92, 81-90.

Choi, J., & Zhang, X. (2019). Computerized item modeling practices using computer adaptive formative assessment automatic item generation system. The Quantitative Methods for Psychology, 15, 214-225.

Gierl, M. J., & Lai, H. (2018). Using automatic item generation to create solutions and rationales for computerized formative testing. Applied Psychological Measurement, 42, 42-57.

Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77, 81-112.

Hwang, G. J., & Chang, H. F. (2011). A formative assessment-based mobile learning approach to improving the learning attitudes and achievements of students. Computers & Education, 56, 1023-1031.

Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and preliminary feedback intervention theory. Psychological Bulletin, 119, 254-284.

Popham, J. (2011). Transformative Assessment in Action: An Inside Look at Applying the Process. Alexandria, VA: Association for Supervision & Curriculum Development.

Sadler, R. (1989). Formative assessment and the design of instructional systems. Instructional Science, 18, 119-144.

Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78, 153-189.

Multilingual Item Generation

< Prev   CONTENTS   Source   Next >