Reliability statistics

This section describes the statistics used to assess the different types of reliability defined earlier.

Measures of internal consistency: Tests developed in the last few years tend to quote computer-generated reliability statistics. The default option (i.e. what it does automatically unless you tell it to do something else) is Cronbach’s alpha (the alpha coefficient of reliability). The authors have always found this coefficient delightfully metaphysical, as it is defined as the estimated correlation of the test with any other test of the same length with similar items (i.e. items from the same item universe) (Kline, 2015). If you want to understand how this coefficient is derived and calculated, then you could consult Urbina (2014) or Kline (1999). The reliability facility on your statistics package will do the computation for you. Alpha should normally be at least 0.70 for reliability to be regarded as satisfactory. As previously mentioned, you can also ask for more information about how each item relates to the rest of the scale: inter-item correlations, itemtotal correlations, and the effects on alpha of removing each item will be displayed, or can be computed. This information is helpful if a scale or measure has poor internal consistency, because it can be used to identify which items could be removed to improve consistency. This procedure is described in Chapter 5.

There are interesting differences between Cronbach’s alpha, and other correlation-type statistics. You probably know that the acceptability of a conventional correlation coefficient is usually judged by its statistical significance, which is related to sample size. The acceptability of Cronbach’s coefficient alpha is not determined by its statistical significance. An absolute value of 0.7 (or sometimes 0.8 or 0.6) is normally taken as the criterion of acceptability. Another difference is that while alpha cannot exceed 1.0, it is possible for it to have a value of less than —1.0, indicating poor cohesiveness.

Kuder-Richardson 20 (K-R 20): This is a coefficient that can be interpreted in the same way as Cronbach’s alpha. Your statistics package may calculate it for you automatically in cases where your data are dichotomous and therefore not suitable for ordinary correlation coefficients. Ordinary correlation coefficients can be calculated when people answer each item on a rating scale (say from 0 to 7) or get a score (say from 0 to 20). If they just said ‘yes’ or ‘no’, ‘agree’ or ‘disagree’, then answers would be entered on a database as 0 or 1 (or 1 or 2); the data are said to be dichotomous, and the statistics package will calculate K-R 20 instead of Cronbach’s alpha. As mentioned, it can be interpreted in the same way as Cronbach’s alpha.

Test-retest reliability: This would be calculated by calculating the correlation coefficient between scores on the test on two different occasions. Normally, the Pearson (bivariate) correlation would be calculated in the usual way. Split-half reliability is also expressed by a correlation coefficient, showing the strength of the relationship between the two halves of the test.

Guttman’s coefficients: These and other reliability statistics may be offered by your statistics package. Refer to your statistics package handbook, or to Kline (1999), Tabachnick and Fidell (2013) or Urbina (2014), if you wish to discover more about them. It is probably premature for the beginner at test construction to worry about whether they are needed.

Factor and principal components analyses: The literature on test construction frequently refers to factor analysis (FA). Factor analysis is a way of discovering statistical ‘factors’ among a lot of test items (variables); it is a way of analyzing relations between correlations, so that one can look at relations between all variables rather than just pairs. The first step is the construction of a correlation matrix showing the correlation between each variable and every other variable. Factor analysis involves examining the way variables relate to each other, and identifying factors underlying the clusters of inter-correlations. The final output of an FA includes a list of factors - which are listed in the order in which they contribute to the variance — with ‘factor loadings’ for each variable. These factor loadings show how much every variable contributes to the factor, and they can range from +1.0 to —1.0. Approximately, they correspond to the item-total correlations for each factor. The researcher names each factor by examining which variables are most heavily loaded on it.

You might consider using FA if you want to develop a test

• • That is assessing something complicated, in which you suspect more than one underlying factor. A variety ofitems are needed for assessment, they would not be well associated with each other, and a conventional reliability analysis yields poor results.
• • For which you are unsure what the underlying factors are. When you know what the different factors are, you can write a subscale to assess each, and then examine the reliabilities of each subscale. When you do not know, then factor analysis may tell you. An example of a case for FA was said to be intelligence. Researchers were unsure about the factors involved, and so intelligence tests were constructed by writing many items testing many abilities, and then subjecting performance scores to FA. There has been a certain amount of controversy surrounding the applications of FA to the construction of tests of intelligence, personality and the like. FA used to be a time-consuming process in the pre-electronic era - now of course it can be done in a few seconds. Days, even weeks of work would be involved. That may have contributed to the mystique, in which proponents would seem to be claiming a special psychological status for the factors they had laboured to discover, while critics would claim that these were mere statistical artefacts without any inherent psychological reality.

Principal components analysis (PCA) yields similar results to FA but gets there by a slightly different route. Instead of looking at the relations between variables, the PCA starts off by looking for factors that explain as much of the variance as possible (Field, 2017). PCA is often described as a form of FA - strictly speaking this is inaccurate. PCA may be more popular, but the solutions it offers are similar to those offered by FA, and Pallant (2016) points out that sometimes the terms are used interchangeably. It is not wise to be careless about the use of these terms: the computational processes involved are different, though it is true that the two processes give similar results. It would be best not to do both FA and PCA on the same data. See Pallant’s discussion on the different merits of the two techniques.

FA and PCA would be appropriate in the circumstances we have described, and further (conventional) reliability statistics to establish internal consistency would normally be redundant.

More detailed descriptions of the uses of FA and PCA in test construction appear for example in Field (2017), Kline (1999), Tabachnick and Fidell (2013) and Urbina (2014).

IBM SPSS offers FA and PCA under Hnalyse/Dimension Reduction/Factor. MYSTAT offers FA and PCA under Analyze: Factor Analysis, but with PCA as the default option here. Tabachnick and Fidell (2013) include a comparison of the FA and PCA facilities on different statistics packages. A brief outline of FA appears in Chapter 5.

Inter-rater reliability: This is usually assessed either by Cohen’s kappa, or sometimes more simply by calculating the percentage of times two observers’ ratings or categorisations agreed with each other. If there are more than two people making the ratings, Siegel and Castellan (1988) and Field (2017) describe a useful non-parametric statistic, Kendall’s coefficient of concordance (W), which is offered in IBM SPSS.