In addition to investigating and understanding traditional assessment issues such as psychometric properties (i.e., validity and subgroup differences), construct measurement, cross-cultural considerations and applicant reactions related to simulations used for personnel selection, there is a range of other considerations that must be taken into account when considering the use of simulations. What follows is a brief summary of further topics in need of consideration.
Maximum versus typical performance and validity degradation
The changing nature of the predictor-criterion relationship has long been a focus of investigation (Ghiselli, 1956; Humphreys, 1960). A consistent finding of that work has been that the correlation between predictor and criterion measures decays over time (e.g., Alvares & Hulin, 1972; Hulin, Henry & Noon, 1990). Multiple models have been proposed to explain validity degradation (e.g., changing-person model, changing-task model, task consistency, skill acquisition models, dynamic criteria). A number of studies have been conducted to test these models with varying results, demonstrating the complexity of these relationships (e.g., Deadrick & Madigan, 1990; Keil & Cortina, 2001). These studies have mostly focused on cognitive and other ability measures (e.g., psychomotor ability, perceptual speed).
Relatively few studies have specifically examined the relationship between simulation measures and performance over time. While simulations have exhibited high criterion- related validities and have been found to be among the best predictors of job performance available, they likely maximize prediction at the point of selection. As a result, when used alone they may be deficient. Simulations are maximum as opposed to typical performance measure and assess ‘can do’ and not ‘will do’ performance over time (Borman, Bryant & Dorio, 2010; Callinan & Robertson, 2000). As such, they may have limited value for predicting long-term performance, which is a desired part of a selection system for most jobs. The few studies that have been conducted have found that, to a greater extent than other measures such as cognitive ability, the validity of simulations attenuate over time (Robertson & Downs, 1989; Robertson & Kandola, 1982; Siegel & Bergman, 1975).
The potential for validity degradation and its causes when using simulations merits additional investigation, especially in comparison to other measures which may cost less to develop and administer or lead to more or less adverse impact. Much in the same way that multiple models have been proposed to explain validity degradation for cognitive ability, similar research is needed that focuses on simulations. For example, the attenuation may be related to the specificity of skills that are sometimes measured (Callinan & Robertson, 2000). Alternatively, job performance models have described the interaction among personality, motivation and ability, as well as their importance for the prediction of job performance over time (Helmerich, Swain & Carsrud, 1986; Hollenbeck & Whitemer, 1988; Kanfer & Ackerman, 1989). Goldstein, Zedeck and Goldstein (2002) found that non-cognitive predictors become more important when the criterion data are collected later. The extent to which a simulation is cognitively loaded or correlated with personality variables may impact these relationships and the attenuation of validity. Cognitive ability may be more important than the competences assessed by a simulation at various points in the lifecycle of the job, such as during transitional stages when additional learning is required (e.g., when the job is new, when major duties or responsibilities change, when past experience cannot be relied on for performance) (Murphy, 1989). To the extent that the lifespan utility of a predictor can be estimated, a better understanding of its initial utility can be acquired (Keil & Cortina, 2001) and more informed decisions can be made about what predictors to use for specific hiring goals and their potential organizational return on investment.