Validity of references
How valid are letters of recommendation in predicting relevant job outcomes? Again, research into the validity of references has been scant, especially in comparison to the frequent use of references in personnel selection. This is no doubt partly because it is unclear what the criterion variable is. Most of this research has focused on structured references, not least because it is easier to quantify the validity of these references (particularly compared to the highly variable and, by definition, hard to standardize, unstructured letters of recommendation). For example, studies on the ERQ showed that reference checks correlated in the range of 0.00 and 0.30 with subsequent performance. In a meta-analysis, Reilly and Chao (1982) reported a mean correlation of 0.18 with supervisory ratings, 0.08 with turnover and 0.14 with a global criterion. A more generous estimate of 0.26 (corrected for unreliability and restriction of range) was provided by Hunter and Hunter’s (1984) meta-analysis, and one of the largest validities was (again, corrected) 0.36 for head teachers' references and training success in the Navy (Jones & Harrison, 1982). Jones and Harrison (1982) pointed out that teachers' (or, for that matter, professors') references tend to be more accurate because they are more motivated than former employers to maintain credibility as they are likely to write more references in the future.
It would be incongruent to expect higher validities from the reference letter if it is not reliable in the first place. Yet there are several other converging factors that threaten the validity of this assessment and selection method. References tend to be very lenient, which produces highly skewed data. This effect, often referred to as the Pollyanna effect, reduces the true variance between candidates (producing more heterogeneous outcomes than predictors) and means that ‘most applicants are characterised as somewhat desirable' (Paunonen, Jackson & Oberman, 1987, p. 97). This is hardly surprising since referees are nominated by the candidates themselves and referees' ‘primary interest is not with the organization but with the applicant' (Colarelli, Hechanova-Alampay & Canali, 2002, p. 316). Recent research shows that even in academic settings (e.g., for grant proposals) applicant-nominated assessors provide biased and inflated reviews of the candidates (Marsh, Bond & Jayasinghe, 2007). Clearly, referees who are asked to provide a reference have no incentives to be harsh and may indeed fear being too harsh as they may be sued by the candidates. Moreover, given that harsh comments are rare and seen as a ‘kiss of death' (typically, negative points are given more weight than positive ones) referees are even more sensitive about making them, though research suggests that when both negative and positive comments are included references are perceived as more genuine and the result may consequently be a positive hiring decisions (Knouse, 1983). It is also likely that referees abstain from providing a reference if they cannot be positive about the applicant, which would explain the poor response rates found (Schneider & Schmitt, 1986).
Referees tend to write similar references for all candidates. In fact, it has been pointed out that references - particularly unstructured ones - provide more information about the referee than the candidate (Baxter, Brock, Hill & Rozelle, 1981). Moreover, dispositional traits (personality factors) and affective states (mood) distort references significantly (Judge & Higgins, 1998). This leads not only to low reliability but also to lower criterion-related validities.
Referees (often acting in benefit of their own organization) may wish to retain good employees and know that a positive reference may have the opposite effect. Moreover, for the same reasons they may choose to write very positive references for staff they are eager to see off. These ‘hidden agendas' are hard to evidence but indicate that employers' motivations can have a profound effect on the type of reference provided.
There are now many serious legal issues associated with references, so much so that some organizations refuse to give them. People are directed only to say that the candidate was employed for the specified time they worked there and nothing else. Litigation has followed where a person has been hired partly on the basis of a reference only to discover the person was extremely poor at the job. In this instance it appears references have been unrealistically positive to get rid of the employee. However, what has more recently occurred is that people and organizations have been sued if they refuse to give a reference knowing the candidate is in some sense problematic (e.g., has criminal or anti-social tendencies). In this sense some employers claim with respect to references you are ‘damned if you do and damned if you don't'.