# A Word of Caution When Creating Cognitive Models

Creating cognitive models for AIG is challenging. As a result, it is a skill that SMEs can only acquire over time. We noted in Chapter 1 that the SME is responsible for the creative activities associated with identifying, organizing, and evaluating the content and cognitive skills needed to generate test items. Many different tasks are required to produce high-quality cognitive models. For example, SMEs identify the knowledge and skills required to solve different types of tasks; they organize this information into a cognitive model targeted to a specific content area; they create the detailed instructions required to coordinate the content within each model; they are expected to design many different cognitive models, often from scratch; they are also expected to evaluate and provide detailed feedback on models created by other SMEs. These responsibilities require judgement and expertise. In our experience, cognitive model development also requires a lot of practice.

# Two Types of Cognitive Models for AIG

Two types of cognitive models are commonly used for AIG: the logical structures and the key features cognitive models (Gierl & Lai, 2017). These models differ in how they organize information for item generation.

## Logical Structures Cognitive Model

The first type of cognitive model is called logical structures. This model is used when a specific concept or a set of closely related concepts is operationalized as part of the generative process. The logical structures cognitive model is most suitable for measuring the examinees' ability to use a concept across a variety of different content representations. The concept is often used to implement a formula, algorithm, and/or logical outcome. The defining characteristic of this cognitive modelling approach is that the content for the item can vary, but the concept remains fixed across the generated items. To illustrate this model, we present a simple math

Table 2.1 Parent Item Related to Ratio, Proportion, and Rate

 Yesterday, a veterinarian treated 2 birds, 3 cats, 6 dogs. What was the ratio of the number of cats treated to the total number of animals treated by the veterinarian? (A) 1 to 4 (B) 1 to 6 (C) 1 to 13 (D) 3 to 8 (E) 3 to 1Г

* correct option

example using the three-step AIG method. This example will be used throughout the book to illustrate principles and applications using logical structures. The example, presented in Table 2.1, is adapted from a parent item used to measure concepts related to ratio, proportion, and rate. It was selected as a straightforward example so that readers can focus on the logic of our method without being overburdened by the content within the model. Figure 2.1 contains a cognitive model based on the parent item in Table 2.1 that can be used to solve word problems that measure range and ratio. Because this task is straightforward, the associated model is relatively simple. To organize information with the cognitive model, content is presented in three different panels. These panels structure and organize the information within the model. The panel structure is helpful for the SME when creating the model. The top panel identifies the general problem and its associated scenarios. The SME begins by identifying the general problem specific to the parent item. The middle panel specifies the relevant sources of information. Sources of information can be specific to a particular problem or generic, thereby applying to many problems. The bottom panel highlights the salient features. Each feature also specifies two nested components. The first nested component for a feature is the element. Elements contain content specific to each feature that can be manipulated for item generation. The content in each element is stored as values. These values can be denoted either as string values, which are non-numeric content, or integer values, which are numeric content. The second nested component for a feature is the constraint. Constraints serve as restrictions that must be applied to the elements during the assembly task to ensure that content in the elements are combined in a meaningful

Figure 2.1 A logical structures cognitive model for range and ratio

way so that useful items can be generated. Alternatively, constraints can be described as the problem-solving logic which serves as the instructions required to assemble the content in the elements. This logic serves as the cognitive component of the problem-solving task. A generic representation which outlines the required components for any cognitive model— not just those models described in our book—is provided in Figure 2.2.

Figure 2.2 A generalized structure for a cognitive model for AIG

The cognitive model in Figure 2.1 focuses on a range and ratio word problem. The specific scenario for this problem is based on a set of values in a ratio. Because this model is simple, there is only one source of information: the range. Each source of information contains at least one feature. In our example, the element for the range source of information is an integer value (11 to I3). The range for the integers is identical, 2-8 in increments of 1. Each feature also contains constraints. There are no constraints for the integer elements in our example, meaning that values

Figure 2.3 A logical structures cognitive model with four different scenarios

for range and ratio

in the range 2-8 can be used to generate test items. Models can quickly expand and become more complex. For example, Figure 2.3 contains a cognitive model with the same problem but an expanded list of scenarios for range and ratio word problems. Table 2.2 contains four different stems that are represented in the cognitive model presented in Figure 2.3. The general problem is still focused on range and ratio, but it includes four

Table 2.2 Four Different Stems for Range and Ratio

 Yesterday a veterinarian treated 2 birds, 3 cats, 6 dogs. What was the ratio of the number of cats treated to the total number of animals treated by the veterinarian? (A) 1 to 4 (B) 1 to 6 (C) 1 to 13 (D) 3 to 8 (E) 3 to 11 *

different scenarios: recognition of a value in a given set of values, sum of selected values among a set of values, sum of all values among a set of values, and presentation of a value among a set of values in a ratio. For each of the problems and scenarios presented in Figure 2.3, the same source of the information (range) and features list (integers) is used for our example.

Large cognitive models which are common in operational AIG applications often address a single problem with four to seven different scenarios (top panel in Figures 2.3 and 2.4) and five to seven different sources of information (middle panel). The sources of information, in turn, typically contain 8-12 different features (bottom panel). This type of large cognitive model contains many variables capable of generating millions of items prior to the application of the constraints. Then when the constraints outlined in the cognitive models are applied, the majority of the generated items are eliminated because they result in infeasible combinations that would produce meaningless or inaccurate items. The combinations that remain are meaningful items produced by assembling specific combinations of elements within a feature for well-defined sources of information that can be used by the SME to measure different scenarios for a specific type of problem. Every aspect of the model—problem, scenario, sources of information, features, elements, constraints—is created by the SME. Examples of logical structures of cognitive models for AIG can be found in the content areas of science (Gierl & Lai, 2017; Gierl, Latifi, Lai, Matovinovic, & Boughton, 2016), mathematics (Gierl & Lai, 2016b; Gierl, Lai, Hogan, & Matovinovic, 2015), and classroom testing (Gierl, Bulut, & Zhang, 2018a).

Figure 2.4 A key features cognitive model for cold versus flu