APPENDIX A. Key Considerations in Evaluating the Selection Process

Validity and Reliability

It is always in the best interest of an employer to ensure a reliable and valid selection process. This affords three benefits: (1) it assures the employer that those selected are more likely to perform well on the job than those not selected; (2) it allows for a legally defensible process; and (3) it is fair to applicants. Best practices in personnel selection require amassing evidence that each selection tool used provides reliable and valid measurements of job applicants.[1]

Reliable selection tools are those that are likely to produce the same scores about applicants regardless of where or when[2] the measurement took place, or who scored or who conducted the test. Valid selection tools are tools that distinguish the applicants that are more likely to be successful on the job (or achieve some other important job outcome) from the ones less likely to be successful. The more valid a tool, the fewer mistakes it makes when making that distinction. The more valid a tool, the better it is at identifying who would succeed on the job.

There is no silver bullet that ensures reliability or validity. However, there are key features that can be put in place to help make both more likely.

Methods for increasing reliability include establishing highly structured processes for all steps and conducting reliability checks. To establish structure, training on how to conduct each part of the selection process should be provided and documented. Clear instructions, rules, and guidelines for how to provide any subjective ratings should be established. Anchoring ratings with behavioral examples and holding training sessions where raters practice applying the rating scale and receive corrective feedback on the accuracy of their ratings is also important. Additionally, checks on reliability help ensure the structure is serving its purpose. To check interviewer reliability, for example, interviewers can be asked to independently rate the same person without discussing their ratings. This can be done for several candidates and scores can be compared. To check whether performance by the same person varies drastically depending on the interview question, candidates can be asked multiple questions and their answers to each can be scored independently. Then their scores on each question can be compared.

Methods for increasing validity of a selection tool start with alignment of the selection process with a well-designed job analysis. KSAOs identified in the job analysis that cannot be easily trained and are either important or frequently needed in entry-level assignments on the job should be considered for inclusion in the personnel selection process. Once the key factors to be assessed are identified, tools supported in the existing research literature should be identified to address each factor. Tools with good existing support should be analyzed to examine ideally at least two forms of validation evidence: predictive (how well it predicts outcomes considered relevant to the job, such as training success, job performance, or injuries) and content (how well independent expert judges agree about the tool's content alignment with the factors identified in the job analysis). Examination of predictive validation evidence is ideally conducted as a pilot study where the tools are administered to applicants and those applicants are followed over time. In addition to examining both types of evidence, disparate impact, possible differential prediction, and the extent to which observed relationships are underestimates (because low performers have already been eliminated) are examples of key factors that should be examined before the tool is implemented and used for employment decisions.

  • [1] Any point in a selection process where some people are allowed to continue and some are not is a point where a selection decision is made. Hence, we refer to any judgments, criteria, tests, or scores used to make those decisions as selection tools. The written test, the interviews, the panel reviews, etc., are all being used as selection tools in this context.
  • [2] Measurements should be stable across time, assuming someone's underlying capabilities have not changed. However, training, acquisition of new knowledge, or gains in experience could be expected to lead to changes in candidates' underlying capabilities across time.
 
< Prev   CONTENTS   Next >