AIG Three-Step Method
Step 1: Cognitive Model Development
To begin, test developers must identify the content that will be used to produce new items. This content is identified using design principles and guidelines that highlight the knowledge, skills and abilities required to solve problems in a specific domain. This content must also be organized and structured in a manner that can promote item generation. A strong body of literature exists on how medical knowledge is conceptualized, organized and structured. Norman, Eva, Brooks and Hamstra (2006), for instance, characterized the organization of medical knowledge as causal (normal human functioning and disease processes), analytic (relationship of specific symptoms and features with specific conditions) and experiential (prior case experiences). Leighton and Gierl (2011) provided a detailed account for how knowledge is organized and structured to account for mathematical reasoning, reading comprehension and scientific reasoning.
Just as frameworks are needed to study the structure and application of knowledge in medicine, mathematics, reading and science, frameworks are also needed to generate test items. Figure 21.1 contains a framework that specifies the knowledge required to make a therapeutic (i.e., drug intervention) decision to address infection during pregnancy Gierl, Lai and Turner (2012) called the framework in Figure 21.1 a cognitive model for AIG. A cognitive model for AIG is intended to highlight the knowledge, skills and abilities required to solve a problem in a specific domain. This model also organizes the cognitive- and content-specific information into a coherent whole, thereby presenting a succinct yet structured representation of how examinees think about and solve problems.
To create the cognitive model in Figure 21.1, two content specialists, who were experienced medical item writers and practicing physicians, described the knowledge, content and clinical-reasoning skills required to solve different problems using therapeutic interventions. The knowledge and skills for the Figure 21.1 cognitive model were identified in an inductive manner by asking the content specialists to review a parent multiple-choice item (see Figure 21.2) and then to identify and describe key information that would be used by an examinee to solve the item. Three types of key information required to solve the parent item in this example can be described. They include the problem and associated scenarios, sources of information, and features (see Figure 21.3).
These three types of key information are specified as separate panels in Figure 21.3. The top panel identifies the problem and its associated scenarios. The content specialists first began by identifying
Figure 21.1 Cognitive model for AIG using infection and pregnancy example.
Figure 21.2 Parent item used for infection and pregnancy example.
Figure 21.3 A general cognitive model structure for AIG.
the problem (i.e., infection and pregnancy) specific to the existing test item. Then they identified different drug types that could be prescribed (i.e., penicillin [P], cephalosporin [C], macrolides [M], sulfa [S], furantoin [F]) to treat infection during pregnancy, along with the associated noncommercial drug names (e.g., penicillin G, amoxicillin, ampicillin).
The middle panel specifies the relevant sources of information required to create variables than can be manipulate in the item model. Sources of information can be case-specific (e.g., type of infection) or generic (e.g., patient characteristics). We selected a relatively simple example for illustrative purposes in this chapter, where only two sources of information were identified from a universe of all possible sources of information. But many different sources of information related to the problem and its associated scenarios (e.g., symptomatic presentation, laboratory results, patient history) could be included in the cognitive model, thereby increasing its generative capacity. That is, the cognitive model is developed, in part, to reflect the knowledge, skills and abilities required to solve the problem. But the model can also be developed with the more pragmatic goal of reaching a generative target given the developer’s item banking requirements.
The bottom panel highlights the salient features, which include the elements and constraints, within each source of information. For Figure 21.1, six features (i.e., urinary tract infection, pneumonia, cellulitis, gestation period, allergy, age) were identified across two sources of information. Each feature also specifies two nested components. The first nested component for a feature is the element. Elements contain content specific to each feature that can be manipulated for item generation. As one example, the cellulitis feature in the bottom left corner of the cognitive model contains the element “present” (i.e., the pregnant patient with infection has cellulitis). The second nested component for a feature is the constraint. Each element is constrained by the scenarios specific to this problem. For instance, cephalosporin (C) and macrolides (M) are the drugs “very likely” to be used to treat infection and pregnancy when the type of infection is cellulitis. A generalized cognitive model structure that can be used to produce items in different content areas and in diverse knowledge domains is presented in Figure 21.3.
The content presented in the cognitive model for AIG serves two purposes. The first purpose is practical. The cognitive model guides the computer-based algorithms described in step 3 so that new items can be assembled. Therefore, one important purpose of the cognitive model is to link the problem (infection and pregnancy) and the associated drug types and noncommercial drug names (e.g., penicillin G, amoxicillin and ampicillin are drug names for the drug type penicillin) to the features (urinary tract infection, pneumonia, cellulitis, gestation period, allergy, age) through the sources of information (type of infection, patient characteristics). These prescriptive links are used for item generation, as the features can be inserted in their appropriate information sources, as outlined in Figure 21.1, subject to the elements and their constraints to yield new test items.
The second purpose of the cognitive model is more abstract. A cognitive model for AIG highlights the knowledge, skills and abilities required to solve a problem in a specific domain. It also organizes the cognitive- and content-specific information to provide a structured representation of how examinees think about and solve problems. Hence, the cognitive model could be considered a construct representation that guides item development. More than 30 years ago, Embretson (1983) suggested that cognitive theory could enhance psychometric practice by illuminating the construct representation of a test. The construct that underlies test performance is represented by the cognitive processes, strategies, knowledge and content used by an examinee to respond to a set of test items. Once these cognitive requirements are sufficiently described, Embretson also claimed they could be assembled into cognitive models to develop items that elicit specific knowledge structures and cognitive processing skills. Test scores anchored to a cognitive model should be more interpretable and, perhaps, more meaningful to a diverse group of users because performance is described not only using a specific set of cognitive skills in a well-defined content area but also using items developed to directly measure these skills. Norman, Eva, Brooks and Hamstra (2006) provided a similar line of reasoning by stating that problem representation was an important way to organize and study the content and processes required for expert medical reasoning and problem solving. The method described in this chapter provides an operational example of how Embretson’s construct representation and Norman et al.’s problem representation can be used to generate test items. The cognitive model for AIG was created by medical content specialists, thereby serving as a representation of how these experts think about and solve problems related to infection and pregnancy. This representation was documented in the form of an explicit cognitive model and then used to guide the detailed assembly process needed for item generation. The item model rendering and computer- based assembly are described next.