Align the Content in Each Selection Step with the Job Analysis and Deconflict It with Other Elements in the Process

Each selection method should be designed so as to not be inadvertently duplicative of other methods. Instead, any duplication that is occurring should be by design, to either check the information, increase the reliability of the information, or measure the information in a way that increases the comprehensiveness of the dimension being measured. For example, the interview should focus on factors that are difficult to otherwise measure or to judge in other ways, whereas the background checks could be designed to verify some of the information obtained through the interview or other sources. Interviews should be designed to supplement rather than overlap with the information obtained during the background investigation steps.

At present, it appears that there is significant overlap in several steps of the process. Interview specialists interview candidates about the information in their PIQ and PHF (Step 6) after different interviewers interview the candidate in Step 5. In Step 7, new raters evaluate the PIQ and PHF information collected by the interviewers in Step 6. In Step 8, a background investigator compiles information on the candidate. In Step 9, an entirely new set of raters rate the information collected by the background investigator.

Consolidating some of these steps could speed up the review process and reduce the reliance on firefighter and Personnel Department staff. If firefighters still desire involvement in the decision process, some could be trained in the rating processes and asked to doublecode a random subset of the investigations to determine whether their ratings agree with the investigators'. If they provide similar ratings to the investigators, there is no need to have them participate for all candidates. If they do not provide similar ratings, then until they do, the rating process and/or the training should be revised using existing job analysis information.

Tie Minimums on All Selection Factors to Acceptable and Unacceptable Performance in Training or on the Job

For example, in the interviews, although a score of 70 is considered passing, candidates must score above 95 to move forward, unless there are too few high-scoring candidates. Similarly, on the final panel reviews, candidates move forward in the process based on their scores and the number of candidates needed. Although it is possible that candidates rated “Satisfactory” meet all of the qualifications necessary to be a successful and dependable firefighter, overall ratings of “Good” theoretically represent the minimum cutoff, and only candidates rated “Outstanding” by both raters were allowed to move forward in the last cycle. This top-down approach to selection can be justified if there is evidence suggesting that the applicants receiving the highest scores are in fact more likely to be successful firefighters.

Meaningful minimums for ratings should be set by tying scores to key job outcomes and job requirements. The minimums should be set such that candidates at that rating or above are indeed likely to perform well and meet other needs. If candidates are being held to a higher standard (such as by the use of top-down selection where only the highest scorers are offered employment), those higher standards should be checked for validity and disparate impact. We recommend that the city establish minimums that are tied to job requirements.

< Prev   CONTENTS   Next >