Explore Options for Reducing the Applicant Pool to a Manageable Size

In 2013, the number of firefighter applicants to the city dwarfed the number of available positions. When the number of applicants becomes excessive the Personnel Department cannot allow everyone to move through the selection process. Instead, the department uses a multiple- hurdle system in which only those who pass a given hurdle are allowed to continue on to the next step in the screening process. In 2013, this approach resulted in disparate impact for key demographic groups.^[1] Therefore, the department should reevaluate the processes it uses to winnow down the number of applicants to a manageable size. In doing so, the city should pay particular attention to disparate impact (i.e., whether these processes affect the diversity of the applicant pool) and the validity of the tool and the minimums used in that winnowing process.

There is no obvious answer for how best to winnow the applicant pool. We discuss two options, each of which has advantages and disadvantages:^[2]

• Top-down selection on the written test. If the Personnel Department had evidence that scores on the test relate to higher performance in job-related situations, it could select people in order of their scores to narrow the applicant pool to a manageable level. This is a merit-based approach known as top-down selection. However, the use of such a test could result in disparate impact against some demographic groups. If this approach is used and the test shows disparate impact, we recommend that the Personnel Department consider adding a non-cognitive-personality measure to the written test and use a combined score for top- down selection. This can help maintain greater diversity within the applicant pool (as personality tests often do not show disparate impact) and provide a more well-rounded assessment of the applicants. We also would recommend targeted recruiting aimed at finding and attracting minority applicants who are likely to be highly competitive on the aptitude test. Doing so would serve to lower the observed disparate impact of the test for those groups.

• Random sampling. To help increase the probability that minority representation is maintained in the applicant pool, the city might consider using random sampling (either simple random sampling or stratified random sampling) rather than aptitude testing to narrow the initially very large applicant pool to a manageable level. This approach would help preserve the diversity of the initial applicant pool, and it could be used in combination with a careful, rigorous, and valid applicant screening process to ensure that only highly qualified candidates receive offers, consistent with the fact that public safety is the primary desired outcome of firefighter hiring. However, we are aware of only one large metropolitan fire department currently using random selection as part of its hiring process. Although that organization plans to continue to use random selection in future hiring cycles, two other departments that have tried it in the past reported that it was not viewed favorably by many of the applicants.

The approaches are not mutually exclusive, and an effective selection process might involve a combination of both approaches.

Whichever approach or combination of approaches is used, the Personnel Department should take care in determining the size of the pool that continues in the selection process. Over-reducing the number of applicants early on in the process could result in too few applicants later on, in which case standards and expectations might have to be lowered in order to have enough selectees at the end of the selection process. To prevent that from occurring, we suggest that the Personnel Department seek to maximize the size of the applicant pool deemed “manageable.” We also recommend that checks be in place to make sure that the resulting pool stays highly competitive in the later hurdles.

Set Aside Funding and Resources for a Robust Appeals Process for Applicants Who Believe That They Have Been Wrongly Passed Over

Having robust appeals procedures for applicants who believe they have been treated unfairly during the selection process or believe they were not selected because of some bias in the system will build trust in the firefighter hiring process and minimize chances for costly litigations. The firefighter recruiting website should post information about the appeals process.^[3] This should include the necessary information to submit to initiate appeals and clear guidelines on how individual applicants can better meet selection criteria.

Improve the Firefighter Selection Process Through a Variety of Specific Revisions

The city's current personnel selection practices are generally consistent with best practices in personnel selection. However, the process could be improved. In Chapter Five, we offer specific suggestions for improving the selection process, such as conducting electronic background checks earlier in the process, identifying overlaps in content among selection processes, outsourcing the written test to a private vendor, and further standardizing interview and background investigation processes. In general, our suggestions are intended to promote transparency in the selection process, manage applicants' expectations, and help to identify viable and competitive applicants early in the process so as to improve efficiency and save resources for both the city and its applicants. The suggestions can also provide additional theoretical and empirical support for the reliability and validity of selection practices, two key elements in ensuring legal defensibility.

• [1] See Chapter Four.
• [2] Legal issues should be factored into the decision of which method is most appropriate (see Appendix A for more discussion on this). Those legal issues are constantly evolving. Recent court cases involving aptitude tests that show disparate impact have new complexities. Moreover, it is not clear how random selection practices would be received by the courts. We therefore recommend that the city's legal counsel advise them on which, if any, of these options is advisable in the current legal environment. See Appendix A for more discussion on this.
• [3] Note that there is an appeals process for candidates who fail parts of the application process. The only part of the process not appealable is the panel review processes, because instead of a “fail” result or disqualification, candidates are non-selected if they are not the most competitive. The city has noted that resources would be a big concern if appeals for panel review were implemented.