Alignment Evidence: Content and Cognitive Processes
In many cases, particularly with current educational assessments and credentialing or licensure tests, scores are interpreted as indicating the extent to which carefully specified content or processes have been mastered. Test developers should provide evidence of the degree to which test items and test forms are aligned to the content specifications used to support this type of interpretation, and also document the qualifications and experience of the experts who provided these judgments. Standards 1.11 and 1.9 describe the requisite documentation.
When the rationale for test score interpretation for a given use rests in part on the appropriateness of test content, the procedures followed in specifying and generating test content should be described and justified with reference to the intended population to be tested and the construct the test is intended to measure or the domain it is intended to represent. If the definition of the content sampled incorporates criteria such as importance, frequency, or criticality, these criteria should also be clearly explained and justified.
Standard 1.9
When a validation rests in part on the opinions or decisions of expert judges, observers, or raters, procedures for selecting such experts and for eliciting judgments or ratings should be fully described. The qualifications and experience of the judges should be presented. The description of procedures should include any training and instructions provided, should indicate whether participants reached their decisions independently, and should report the level of agreement reached. If participants interacted with one another or exchanged information, the procedures through which they may have influenced one another should be set forth.
Sometimes evidence is based on the processes used in responding to test questions. Often, cognitive laboratories are used to elicit evidence on response processes. Standard 1.12 calls for empirical and/or theoretical evidence when response processes are part of the validity argument.
Standard 1.12
If the rationale for score interpretation for a given use depends on premises about the psychological processes or cognitive operations of test takers, then theoretical or empirical evidence in support of those premises should be provided. When statements about the processes employed by observers or scorers are part of the argument for validity, similar information should be provided.
