Efficacy of Test Preparation
Considerable research has examined the effectiveness of test preparation. This is an area in which researchers can dive deeper, remain longer and, just perhaps, reach fewer conclusions than in any other topic addressed in this handbook. Few clear main effects are found and interactions abound. In other words, the results evaluating the benefit of test preparation depend on the outcome measured, the preparation method used and the population tested. Research designs are susceptible to various internal and external validity threats as well. Still, test developers should become familiar with efficacy research for measures in their area of expertise and should share substantiated results with prospective examinees.
Mehrens and Lehmann (1991) made several helpful observations regarding research focused on evaluating the efficacy of test preparation, noting that efficacy studies are difficult to conduct. True experimental design with random assignment is often impossible. Subjects usually self-select into either the treatment group or the control group. Further, those in treatment groups also may be engaged in other test preparation activities beyond those under study, a common occurrence for examinees who receive coaching, according to Camara (personal communication, May 4, 1999). Consequently, score gains cannot be attributed unequivocally to the study treatment. Mehrens and Lehmann (1991) also indicated that the operational definition of coaching has been applied ambiguously across research studies, variously meaning (a) test-taking orientation, (b) a short-term cram session or (c) long-term instruction. Research results can be difficult to interpret as a whole because of the different ways test preparation terms are defined and used.
Expected gain-score effect sizes usually are small. According to Mehrens and Lehmann (1991), a large effect size is about one fifth of a standard deviation unit (SD). Such results are the exception, however. Most gain-score effect sizes will be lower, about one twelfth of an SD. According to Messick
- (1980) , mediating and moderating factors influence the effectiveness of coaching. He suggested that gains may increase with time invested in training, but with diminishing return, and that vocabulary may be less responsive to coaching than mathematics skills. Furthermore, Messick and Jungeblut
- (1981) suggested that students must invest nearly as much time in coaching as they do in full-time school to significantly raise their SAT scores.
When examinees retake tests, the retest score gain often equals or exceeds that seen after test preparation, with score increases typically falling between one fourth and one half of an SD (Hausknecht et al., 2007; Kulik et al., 1984). Mehrens and Lehmann (1991) noted that, while retake gains are not equivalent to coaching gains, retake gains provide an interesting point of comparison regarding the efficacy of test preparation that is different from “no-treatment” control groups.
According to Mehrens and Lehmann (1991), research suggests test-wiseness (a) can be taught, (b) results in small score increases on many tests, (c) helps students more on multiple-choice items than free-response items, (d) increases with age and test-taking experience and (e) moderately correlates with achievement measures. They also observed that such skills are more helpful on poorly constructed tests than on the well-constructed tests that are typical of most standardized exams.
Authorities have issued a caveat regarding the limits of focusing preparation efforts on test-taking skills alone. Mehrens and Lehmann (1991) noted that test-wiseness is not a substitute for thorough knowledge of a subject. Similarly, Bond (1993) cautioned that instruction solely in test-taking skills does not affect test performance on standardized tests to a meaningful degree. Crocker (2005) echoed these assertions, noting that the most effective test preparation is comprehensive instruction in the subject matter, such as quality high school instruction that prepares students for college admission tests.