Biases and Assessment Error
There are a number of well-known biases that can impact on the reliability of rating the non-technical performance of individuals and teams. The first group of biases that can negatively impact on rating non-technical skills all relate to situations in which one aspect of performance becomes the main focus of assessment, and in turn, that aspect of performance then influences other aspects of assessment. A second set of biases relate to general patterns in the way in which an assessor makes their ratings. The final set of biases relate to the influence of normative assessment instead of the criterion-referenced assessment at the heart of a competency-based training approach to non-technical skills development.
Perhaps the most common of the biases that can influence raters’ scores is the ‘halo’ effect, in which a general conception of the individual or team performance is made, and that subsequently influences all ratings. This is especially the case when a particularly good, or poor, outcome is achieved, and this outcome ‘masks’ the areas of suboptimal performance.
EXAMPLE FROM PRACTICE: HALO EFFECT
During a simulator exercise, a control room team are able to quickly resolve a complex malfunction impacting on a large part of the electricity network. It was one of the quickest restorations of power the instructor had seen, and the impact on service delivery to customers was minimal. The instructor therefore gives good scores across all the categories, even though the coordination of the team, the evaluation of risk and the identification of alternative solutions were lacking. These failures in non-technical skills led to a solution that while timely, actually compromised safety. However, due to the positive outcome, consistently high scores were given by the instructor.
The halo effect is best detected by examining the amount of variability between the individual categories rated by an individual. If a low degree of variance, or a high degree of correlation, is seen consistently between ratings on all the different categories assessed, the ratings might be influenced by the ‘halo’ effect.15 This is especially the case if ratings are consistently high. Further training in the use of the assessment and rating tool is the most effective mechanism to address instances of the halo effect. Similarly, the presence of the halo effect may indicate deficiencies in the assessment tool, such that the sub-elements are inadequately specified, and raters can’t discriminate between areas of good and poor performance.