Research That Can Inform Practices and Policies

Mehrens and Kaminski (1989) and Koretz (2008) provided guidelines for ethical test preparation, summarized in Table 28.3. Application of acceptable practices likely will result in expanded knowledge in the target domain and increased score gains. Mehrens and Lehmann (1991) contended that the line between ethical and unethical practices is often ill defined. In such cases, context usually will dictate how ethical or unethical the test preparation actions are. They note that this is certainly the case with the practices labeled as “sometimes acceptable” in Table 28.3. “Instruction on general objectives” should be okay more often than not, while “instruction on specific objectives in the format of the test” will almost never be okay.

It is clear that certain practices cross the line of acceptable test preparation. Any preparation that attempts to increase test scores without increasing knowledge and skills in the larger targeted domain raises ethical questions. Doing so limits the range of inferences one can make from a larger domain based on test performance. Mehrens and Kaminski (1989) declared, “If one wishes to infer to a broader domain from a sample of items (or objectives), then teaching directly to the items or the specific sample of objectives lowers the validity of the inference” (p. 14).

There is ample opportunity for educators to focus test preparation on activities that increase knowledge and skills in the target domain. For example, Crocker (2005) introduced the idea of teaching for assessment, which includes four components: content, instruction, testing skills and ethics. Ideally, teachers would focus almost exclusively on teaching the curriculum. If test developers carefully create tests that properly sample from the same curricula, then the efforts of both teachers and test developers will be aligned to the same instructional and test goals.

Table 28.3 Guidelines for Ethical Test Preparation From Mehrens and Kaminski (1989) and Koretz (2008)

Acceptability

Mehrens and Kaminski (1989)

Koretz (2008)

Examples

Always acceptable

General instruction

Working more effectively Teaching more Working harder

Instruction in content/ construct aligned to test content; using better instructional methods; more homework

Usually acceptable

Instruction in test-taking skills

Guidance on how to read and respond to various types of test items; use testing time efficiently; and engage with test interface

Sometimes acceptable

Instruction on general objectives

Instruction on specific objectives from the test Instruction on specific objectives in the format of the test

Reallocation

Alignment

Coaching

Test preparation guides addressing test objectives, but only if all objectives are covered; unacceptable if objectives are limited to only those on a test

Never acceptable

Practice on or instruction in a parallel form of an actual test Practice on or instruction in actual test items

Cheating

Providing examinees with access to operational test content

Encouraging memorization of operational test content

Researchers have closely studied teachers’ test preparation practices. Moore (1994) explored educators’ perceptions of the appropriateness of some commonly used test preparation activities and found that the views of classroom teachers and measurement specialists differed. Indeed, teachers consistently regarded many test preparation activities (e.g., motivational activities, same-format preparation, pretest interventions, previous form preparations, current form preparations and interventions during testing) to be more appropriate than testing specialists did. Of course, in accordance with Mehrens and Kaminski (1989) and Koretz (2008), activities such as preparing students with current test forms and providing interventions during testing are clearly unethical. Firestone, Monfils and Schorr (2004) found that teachers who felt significant pressure related to new state tests integrated more test preparation into regular teaching, particularly as the test date neared.

Pedulla et al. (2003) investigated the effects of test stakes on teachers’ test preparation practices. Several differences appeared among teachers, depending on whether their students took high-stakes tests or low-stakes tests. These differences are noted in Table 28.4. For example, even for low-stakes tests, two thirds of teachers (67%) taught test-taking skills. Not surprisingly, the percentage was higher for teachers giving high-stakes tests (85%). Few teachers in low-stakes situations used commercial test preparation programs or provided students with released items from the state test.

Lai and Waltman (2008) found no significant differences in teacher test-preparation practices among schools with low, moderate and high achievement levels. The authors reported that this conclusion contradicts previous research; they suspected this discrepancy may be caused by methodological differences or by specific contextual factors related to the testing program studied. Consistent with earlier research, Lai and Waltman found test preparation practices differed across elementary, middle and high school grade levels. For example, elementary school teachers used the greatest variety of procedures, including teaching test-taking skills and using practice tests, and used them more frequently than teachers in other grades. Some teachers reported preparing students using test items that were still operational. Teachers, however, generally objected to the use of operational test items because it violates professional ethics. Interestingly, teachers more often objected to other questionable test preparation practices because they did not facilitate learning and they had a negative effect on score meaning.

Popham (2001) suggested that many educators simply do not reflect on the appropriateness of their test preparation practices. In response to this concern, Mehrens and Lehmann (1991) recommended

Table 28.4 Percentage of Teachers Engaging in High- and Low-Stakes Test Preparation Activities

Test preparation activity

High stakes

Low stakes

Teach test-taking skills

85%

67%

Teach the standards known to be on the test*

75%

54%

Provide students with items similar to those on test*

75%

54%

Provide commercial or state-developed test preparation materials

63%

19%

Provide students with released items from state test*

44%

19%

Note. Adapted from Perceived Effects of State-Mandated Testing Programs on Teaching and Learning: Findings From a National Survey of Teachers, by J. J. Pedulla et al., 2003, retrieved from http://www.bc.edu/research/nbetpp/ statements/nbr2.pdf.

*In some cases these activities can be ethically wrong and will limit the meaning and inferences one can draw from test scores. In the first case, focused instruction that excludes standards that are part of the curricula but not included on the test would be inappropriate. In the third case, providing examples of constructed response tasks along with the associated scoring rubrics for the purpose of letting students what is expected of them is acceptable as long as other instructional activities are utilized as well.

educating teachers about appropriate test preparation and their ethical obligations as educators. Ideally, such training would begin early in the teachers’ educator preparation programs.

Mehrens, Popham and Ryan (1998) addressed the topic of preparation for performance assessments. They indicated that the sample/domain issue described earlier looms just as large, if not larger, for performance assessments because fewer tasks are sampled from the broader domain. The authors provided six general guidelines for designing test preparation for performance assessments.

  • 1. Determine if inferences about student performance are related only to a specific task or to a broader domain of performance tasks.
  • 2. When the intended inference about student performance is to a broader domain, teachers should not narrow instruction in a fashion that would minimize the accuracy of the inference to the broader domain. For example, when teaching editing skills, a teacher should not provide editing instructions using the same writing sample that students will later see on their editing performance assessment.
  • 3. To avoid student surprise and confusion, ensure students are familiar with the performance assessment’s format.
  • 4. Before planning instruction, identify the evaluative criteria and communicate them to students.
  • 5. During instruction, teachers should stress transferability of the skills and knowledge assessed by performance tests (e.g., informing students about how the skills and knowledge required of a given performance test may or may not apply to similar types of tasks).
  • 6. Foster students’ self-evaluation skills. For example, have students judge their own efforts using the evaluative criteria that would have been shared with them (see guideline 4).
 
Source
< Prev   CONTENTS   Source   Next >