Purpose of This Book
The purpose of this book is to describe and illustrate a practical method for generating test items. Different methods can be used, but in this book, we will focus on the logic required for generating items using an item modelling approach. By item modelling, we mean methods that use templates to guide item generation. Our book is intended for two types of readers. The first audience is researchers from a broad range of disciplines who are interested in understanding the theory and the current applications of AIG. The second audience is practitioners and, in particular, SMEs who are interested in adopting our AIG methodology. Taken together, our presentation of the theory and practice of AIG will allow researchers and practitioners to understand, evaluate, and implement our AIG methodology.
Ten different topics related to template-based AIG will be covered in this book. These topics will be presented in two major sections. The first section focuses on the basic concepts related to generating constructed- and selected-response items. Constructed-response items present examinees with a prompt in the stem of the item. Examinees are then expected to construct or create their own responses. Selected-response items present examinees with a prompt in the stem of the item, as well as a list of possible responses. Examinees are expected to select the best response from the list of alternatives. Chapter 2 focuses on cognitive model development, which is appropriate for both the constructed- and selected-response item format. A cognitive model is a formal structured representation of how examinees think about and solve tasks on tests. We will begin by providing a definition and description of cognitive modelling and then highlight why these models are needed to generate test items. Chapter 3 addresses the topic of item model development. Item modelling is also required for generating constructed- and selected-response item formats. Item models are needed to structure the content that has been specified in the cognitive model. Chapter 4 provides an overview of the item generation process. Item generation relies on constraint coding in which the content from the cognitive model is placed into the template specified in the item model using specific instructions and assembly rules. The rules we present are applicable for generating constructed- and selected-response item formats. Chapter 5 contains a summary of distractor generation. Distractors are the incorrect options needed for creating the selected-response (e.g., multiple-choice) item format. We will describe how to create distractors for AIG. Chapter 6 is framed as a succinct guide that summarizes the information presented in Chapter 2 to Chapter 5 into a practical description of how to generate test items. Chapter 7, which is the last chapter in the first section of our book, presents different methods for evaluating the quality of the generated items. These methods focus on the substantive evaluation of the content in the cognitive and item models, as well as the quality of the content in the generated items. It also includes the statistical evaluation of the generated items using item analysis for quantifying item difficulty, discrimination, and similarity.
The second section focuses on advanced topics in item generation. Chapter 8 highlights the importance of content coding. Once the items are generated, they must be organized. Coding is used to tag the content for the generated items so that they can be structured and accessed for different purposes. Having described an AIG method applicable for the most common item formats of constructed and selected response, Chapter 9 focuses on alternative item formats. In this chapter, we describe three alternative models that can be used to generate content in different item formats. The examples in this chapter also help demonstrate how AIG can be used to go beyond the standard constructed- and selected-response item formats. Chapter 10 addresses the topic of rationale generation. AIG is a method for generating new test items, but it can also be used to create the corresponding rationale or solution for each of these items. Three methods for rationale generation are introduced and illustrated in this chapter to support formative feedback systems. Chapter 11 covers the topic of multilingual AIG. We present and illustrate a method that can be used for generating items in two or more languages. Multilingual AIG serves as a generalization of the three-step method described in Chapters 2 to 4. Finally, in Chapter 12, we summarize some of the key issues raised throughout the book by structuring this chapter in a ques- tion-and-answer format, and we describe the topics that we anticipate will shape the future of research and practice in AIG.