Study Approach

To address these four objectives, a multidisciplinary team of RAND researchers executed a research plan that included qualitative as well as quantitative analysis of the firefighter hiring process. We explored some of the LAFD's firefighter recruiting and outreach efforts, identified key duties and KSAOs of members of the fire service, reviewed the fire- fighter selection process, and estimated the impact of selection stages on the demographic diversity of the applicant pool.

Drawing on Existing Expertise and Past Research

As noted above, RAND was asked to conduct a review of the city's existing selection practices and suggest changes. Typically, changes to selection processes are made after data and evidence supporting the validity of the selection procedures have been amassed. That is, changes to selection processes should not be implemented until they have been validated. Validation includes research showing that the selection prac- tices are effective at predicting important outcomes. Validation may also include systematically documenting the links between the selection tools and requirements of the job. However, collection of such data would have required more time than was available for this study. The city instead asked RAND to provide answers in the near term using our existing knowledge of well-established recommended practices.

As a result, in addition to data collected directly from the city's firefighters and firefighter applicants, the recommendations we make in this report draw on findings and knowledge from our past work in personnel selection and workforce diversity issues. For example, our recommendations regarding recruiting practices and initiatives for improving diversity draw from our past and ongoing work in recruiting and diversity initiatives in the military, police, and civilian sectors. And our recommendations regarding changes to the selection practices draw from our past work in personnel selection. This includes work in test design and validation, measurement and prediction of performance, psychological measurement, evaluation of bias, standard setting, and ongoing work for the Department of Defense on establishing gender-neutral entry standards for physically demanding occupations.[1]

Our recommendations are also informed by the vast literature of applied and scientific research that has been conducted on topics such as the validity of tests in employment contexts, successful approaches to increasing diversity, recruiting best-practices, and firefighter selection practices. There are many comprehensive reviews of research in personnel selection (for just a few examples, see Schmidt and Hunter, 1998; Hough and Oswald, 2000; Campion, 1983; Schmitt and Chan, 1998; Salgado, Viswesvaran, and Ones, 2001; Triandis, Kurowski, and Gelfand, 1994).

We also rely on a well-established set of professional practice guidelines in the field of personnel selection to inform our recommendations. These professional practice guidelines are viewed as the authoritative source on the proper development and use of tests and measures in employment contexts. An overview of these guidelines can be found in two published resources:

Principles for the Validation and Use of Personnel Selection Procedures (Society for Industrial and Organizational Psychology, Inc., 2003). This source (referred to as the Principles) was produced by the Society for Industrial and Organizational Psychology to “specify established scientific findings and generally accepted professional practice in the field of personnel selection psychology in the choice, development, evaluation, and use of personnel selection procedures designed to measure constructs related to work behavior with a focus on the accuracy of the inferences that underlie employment decisions” (p. 1).

Standards for Educational and Psychological Testing (American Educational Research Association et al., 2014). This source (referred to as the Standards) was developed jointly by the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education. It summarizes professional standards for the development and use of tests in educational, psychological, and employment settings.

This report examines the firefighting selection practices the city currently uses and the existing evidence amassed by the city to support their use. We rely on our past work in these areas, our knowledge of existing work in these areas, and professional practice guidelines to evaluate and recommend changes to the selection practices. We note that research derived from other, albeit similar, selection contexts cannot wholly serve as a substitute for research on the tools used by the city. As such, we also make recommendations for undertaking further data collection to establish validation evidence to support the selection practices. As already stated, the time frame of the study prohibited this additional undertaking.

Although we relied on existing general expertise and knowledge on personnel selection to inform many of our recommendations, we tailored the recommendations specifically to the city's selection processes and selection needs. We therefore reviewed the city's existing processes very closely. In our review we sought to identify not only the issues that the city viewed as its biggest challenges in the selection process, but also the city's goals for improving the selection process. We also carefully reviewed their existing selection processes and supporting materials and met with city personnel to fill in any gaps in the materials the city provided. We supplemented those materials by conducting a quick analysis of disparate impact of the selection process and by interviewing firefighters as a check of the existing job analysis information. The methods for both of these additional efforts are described in the following sections.

  • [1] For examples of the past work of some of our authors see: Lim et al., 2009; Lim, Haddad, and Daugherty, 2013; Matthies, Lim, and Keller, 2012; Marquis et al., 2007; Haddad et al., 2012; Hardison et al., 2009; Burkhauser, Hanser, and Hardison, 2014; Manacapilli et al., 2012; Hardison, Sims, and Wong, 2010; Hardison and Vilamovska, 2009; Hardison, 2007; Cullen, Hardison, and Sackett, 2004; Sackett, Hardison, and Cullen, 2004; Hardison and Sackett, 2007; Sims et al., 2014).
< Prev   CONTENTS   Next >