Generating Items With Images

The second alternative item type we demonstrate requires the generation of images. The structural requirement is that the item be image based and, therefore, the auxiliary information must contain images. The images must also be generated as part of the three-step AIG method. A radiograph is an example of a specialized image that is used extensively by

An image-based item using a panoramic radiograph

Figure 9.4 An image-based item using a panoramic radiograph

health-care professionals to identify and diagnose problems. The ability to understand and interpret radiographs is an important skill for many health-care providers. An example of an item that contains a radiograph is shown in Figure 9.4. The examinee is presented with the image that contains an identifier. In this example, the image is a dental panoramic radiograph where the identifier is an arrow that points to a specific tooth. To solve this item, examinees are required to interpret the identifier in the radiograph in order to identify the correct tooth.

To generate image-based items with identifiers, an item model is created to capture and coordinate the images needed to generate the items (Lai, Gierl, Byrne, Spielman, & Waldschmidt, 2016). The item model contains images that have identifiers pointing to different teeth. Teeth serve as the sole feature in this example. This example could easily be expanded to include other oral anatomical structures and locations which would serve as new features in the item model. Feature 1, tooth location, has 29 identifiers (see Figure 9.5). After one image is selected and the 29 anatomical locations are identified, constraints are specified. The most important constraint in this example is the selection of the distractors.

An anchor image using a panoramic radiograph

Figure 9.5 An anchor image using a panoramic radiograph

Distractors for this sample can vary depending on the other teeth located in the vicinity (i.e., teeth to the left, right, top, and bottom) of the tooth specified by the identifier (i.e., the correct option). After constraints are defined, the model is used for item generation and a random sample of these teeth is selected as the distractors (see Table 9.1). Using one feature with a distractor constraint, we can generate items with images. A sample of six generated items is presented in Figure 9.6.

Table 9.1 Six Anatomical Features From an Oral Radiograph With Plausible Distractors

Correct Answer

Distractors

Tooth 4

Two of: 13,29,20 Two of: 5,12,21,28

Tooth 6

Two of: 11,22,27 Two of: 7,10,23,26

Tooth 13

Two of: 4 20,29 Two of: 5,12,21,28

Tooth 15

Two of: 2,18,30 Two of: 3,14,19,29

Tooth 21

Two of: 5,12,28 Two of: 4,11,22,27

Tooth 26

Two of: 7,10,23 Two of: 6,11,22,27

A sample of six generated image-based item using a panoramic radiograph

Figure 9.6 A sample of six generated image-based item using a panoramic radiograph

Generating Items With Shapes

The third alternative item type we demonstrate requires the generation of geometric shapes. The structural requirement is that the item must contain a geometric shape, therefore, the auxiliary information must contain shapes with different labels and properties. The shape must also be generated as part of the three-step AIG method. Geometry is an important topic in mathematics in which examinees are presented with shapes and, using the appropriate formulas, required to compute different properties for these shapes. An example of a geometry item that requires a volume computation is presented in Figure 9.7. The examinee is presented with a cylinder. The cylinder in this example is presented as a three-dimensional shape with two congruent circles joined by a curved surface. The radius of the circular base is given by r and the height is given by H. To solve this item, examinees must use the appropriate formula (i.e., яг2 x H) to compute the volume.

To generate geometric shapes, an item model is created to capture and coordinate the shapes needed to generate the items (Gierl, Lai, Hogan, & Matovinovic, 2015a). The item model contains geometric shapes that have labels pointing to specific variables, such as the height

and radius within the shape. The proportions of the geometric shape are varied relative to the measures provided to examinees in the stem of the item. The labels are also varied so that they match the description in the stem. In our example, three features are used to generate items. The type of cylinder computation is the first feature. The tank height, the volume of liquid in the tank, and the volume of that liquid that remains in the tank can be computed (see Figure 9.8). Height is the second feature. Height can be defined using the tank height (H) or water level height (h). Radius is the third feature. The diameter of a circle is computed as кг2. Items can be generated by manipulating values across these three features. For example, base radius r, tank height H(1), and water level h( 1) are used to compute the volume left in the tank. Base radius r, tank height H(1), and distance from water level to the top of the tank h(2) are used to compute water volume in the tank. Base radius r, water level H(2), and distance from water level to the top of the tank h(2) are used to compute the volume of the tank. Other combinations of features can also be used with this item model to create simple volume problems, such as finding the volume of the tank given H(1) and r, the volume of the water in the tank given H(2) and r, and the volume left in the tank given Y(2) and r. Generating items with shapes first requires the assembly of the appropriate stem, and then the auxiliary information with the appropriate shape is rendered using the values presented in the item. A sample of six generated items using geometric shapes is provided in Figure 9.9.

Type, height, and radius variables in a cylinder that can be manipulated for item generation

Figure 9.8 Type, height, and radius variables in a cylinder that can be manipulated for item generation

Challenges With Generating Items Using Auxiliary Information

As demonstrated in our examples, auxiliary information can dramatically vary the presentation of the generated items. The approach we described for assembling auxiliary information fits well into our three-step AIG method. Moreover, the dimensions and descriptions of the auxiliary information can be drawn electronically through scripting, which further enhances the generation process. However, there are three challenges with generating items using auxiliary information. First, the process for incorporating auxiliary information into the item development process is often cumbersome. The specification and output formats used by designers and illustrators often guide the requirements for how images can be presented with an item. Hence different image and file formats must be integrated to produce the generated item. In our experience, aligning these formats is often an awkward process. Second, working with images requires a functioning asset management system to seamlessly create the images, handle rights and permissions for the images, and display the images in a secure manner. While traditional item development often uses a process when the image is stored individually as part of the item through the item banking system, AIG needs to produce and display the correct image on a batch-production basis. The workflow needed for batch production using auxiliary information requires careful planning in the item development infrastructure for any testing organization that uses auxiliary information. Third, when an item is presented in an image-based format, SMEs will need to focus on modelling the auxiliary information in the same manner that they model text. The ability to organize and model auxiliary information may require additional training because SMEs may not have previously considered how auxiliary information can be modelled as part of the three-step AIG method. Extra attention may also be needed in the review process to ensure that the auxiliary information produces the intended outcomes.

References

Anastasi, A., & Urbina, S. (1997). Psychological Testing (7th ed.). Upper Saddle River, NJ: Prentice-Hall.

Arendasy, M., & Sommer, M. (2005). The effect of different types of perceptual manipulations on the dimensionality of automatically generated figural matrices. Intelligence, 33, 307-324.

Arendasy, M., & Sommer, M. (2010). Evaluating the contribution of different item features to the effect size of the gender difference in three-dimensional mental rotation using automatic item generation. Intelligence, 38, 547-581.

Gierl, M. J., Lai, H., Hogan,J., & Matovinovic, D. (2015a). A method for generating test items that are aligned to the Common Core State Standards, lournal of Applied Testing Technology, 16, 1-18.

Gierl, M. J., MacMahon-Ball, M., Vele, V, & Lai, H. (2015b). Method for generating nonverbal reasoning items using n-layer modeling. In E. Ras & D. Joosten- ten Brinke (Eds.), Proceedings from the 2015 International Computer Assisted Assessment Conference, Communications in Computer and Information Science (pp. 1-10). New York: Springer.

Lai, H., Gierl, M. J., Byrne, В. E., Spielman, A., & Waldschmidt, D. (2016). Three modelling applications to promote automatic item generation for examinations in dentistry, lournal of Dental Education, 80, 339-347.

Rationale

Generation

 
Source
< Prev   CONTENTS   Source   Next >