How Do We Define the Word “Item” in AIG?

Our experience working with SMEs and other testing specialists has demonstrated that many implicit definitions and conceptualizations surround the word "item". Also, one important discovery we made while writing this book is that the term "item" is rarely defined in the research literature, which suggests that some universal and well-understood definition exists. We have yet to find this definition. In those rare situations when a definition is offered, it tends to be a black box description, meaning that an input and output are presented but no description of the internal mechanism for transforming the input to output is provided. For this reason, these descriptions offer the definition of a concept that is impossible to either replicate or evaluate. The Standards for Educational and Psychological Testing (2014) serves as the most comprehensive statement on best practices in educational and psychological testing currently available. The Standards define the term "item" as "a statement, question, exercise, or task on a test for which the test taker is to select or construct a response, or perform a task" (2014, p. 220). The authors of the Standards also direct the reader to the term "prompt", which is described as "the question, stimulus, or instruction that elicits a test taker's response" (p. 222). From this description, we can infer that an item contains an input in the form of a statement, question, exercise, task, stimulus, or instruction that produces an output which is the examinee's response or performance. But no description of the internal workings or characteristics of the item is included.

To promote the science of AIG, we offered a new definition of the term "item" that we used to guide AIG in our book. An item is an explicit set of properties that include the parameters, constraints, and instructions used to elicit a response from the examinee. Our definition specifies the contents in the black box as a set of parameters, constraints, and instructions. The input must also contain relevant and explicit information that helps address the specific purpose for creating the item, and it should be represented in a way that can be replicated and evaluated. The output is the response produced from the input by the examinee. This response can be interpreted using the knowledge and skills specified in the cognitive model. Our definition does not include a format requirement, and it does not specify the representation for the parameters, constraints, and instructions. It only requires that this representation be detailed so that it can be replicated and that its specific purposes be explicit so that it can be evaluated.

How Do You Generate Items?

Items can be generated using non-template and template-based AIG. Non-template AIG is guided by the syntactic, semantic, or sequential structure of a text. Non-template AIG, which relies on NLP techniques and knowledge bases, can be used to directly generate statements, questions, and options from inputs, such as texts, databases, and corpora of existing information without the use of item models or templates. Our book is focused on template-based AIG using item modelling. We argue that, at least for now, template-based AIG is preferable for operational test development because it provides the SMEs with the most control and flexibility for generating testing content. Template-based AIG using item modelling is based on a three-step workflow (see Chapter 1, Figure 1.1). These three steps are required because the data in each step is transformed from one state to another. We describe the AIG workflow as an item production system. The system is used to create a product that is consistent with our definition of an item. In step 1, the content required for item generation is identified. The content is specified as a cognitive model. A cognitive model for AIG contains specifications that include the content, parameters, constraints, and/or instructions that will be used to control the behaviour of the model during item generation. In addition to task creation, the cognitive model can be used to describe test performance by identifying the knowledge and skills required to elicit a correct response from the examinee, which, in turn, can be used to make inferences about how examinees are expected to solve tasks generated by the system. In step 2, the content is positioned in the item model. An item model is a template of the assessment task that specifies the parts and content in the task that will be manipulated to create new test items. The content is extracted from the cognitive model and placed as individual values in an item model. In step 3, the instructions for assembling the content are implemented. The instructions are specified as constraints on the content in the cognitive and item models. Constraint coding is considered to be the source of intelligence in AIG because it is the method by which logical constraints are used to capture the specific decisions, judgements, and rules described by the SMEs in the cognitive and item models to produce diverse, multilingual, high-quality test items. Because each step in this workflow is explicit, the input and output from the system can be replicated and evaluated. The purpose of this item development workflow is to produce hundreds or thousands of new test items from a single cognitive model.

 
Source
< Prev   CONTENTS   Source   Next >