Concurrent validity; comparison with existing tests

This means looking at the performance of your participants on one or more other methods of assessing what your test is assessing.

This is potentially quite an embarrassing situation because if there is another test assessing what your scale is assessing, you have to demonstrate why your scale is necessary. This can be done by establishing that, by comparison with the standard (comparison) test, your scale is superior on one or more of the following features:

• • Meets reliability criteria as well or better.
• • Is (on other criteria of validity) as valid or has better validity.
• • Is as quick or quicker to administer.
• • Can be used on as wide or wider range of people.
• • Is preferable on some other attribute: for example your test might be more interesting or less threatening than the existing standard test, and this is associated with higher completion rates.

Your test should be no worse than the standard test on any of the earlier criteria, and it should be an improvement in at least one respect.

If there is a standard test measuring something similar to yours, and it is reliable and valid, then you can examine correlations between scores on the two tests. If there is a significant association, then your test has concurrent validity. The two tests obviously have to be completed by the same people -preferably on the same occasion (otherwise any lack of association between the two might be due to different circumstances at the time of testing).

If your participants did the standard test on a later occasion, you can add the results to your database.

You then correlate scores on the two tests. A method for doing this in SPSS is described in Appendix 6. The correlation® should of course be in the expected direction, and statistically significant.

In presenting the results of concurrent validity testing, you would say something like the following:

One form of concurrent validity was tested and found to be satisfactory.

One hundred British undergraduates (50 female and 50 male, aged 19-46, mean age 22.5) completed the prayer scale, and X’s (1932) attitudes to prayer scale. The (Pearson) correlation was r= 0.554, one-tailed p < 0.01. The new prayer scale is suggested as a rapid and convenient alternative to X’s scale, which is much longer (50 items), some of which appear archaic.

Content and face validity

This should be done when you are writing the scale, by seeing if one or more independent judges agree that your items appear to be or are about what you are trying to measure. Items that judges have reasonable doubts about, should be discarded. Obviously, if you have more than one judge, and both or all agree that an item is unclear, this strengthens the case for scrapping it.

In the case of content validity, judges should have some expertise in the topic of the scale, so that they are qualified to judge whether the items really are about what the scale is measuring. For example, in the case of measuring attitudes to prayer, judges should know something about prayer and its functions - perhaps ministers or practitioners of religion. In the case of liking for chocolate, judges should have some knowledge or experience of eating chocolate. If assessing depression, mental health professionals such as clinical psychologists, psychiatrists or psychiatric nurses would be appropriate. Clearly, judges who know nothing about prayer, or have never tasted chocolate, or who have no dealings with depressed people, will not be appropriate.

In the case of face validity, judges should be drawn from the same population of those you propose to assess. So if you wish to assess the attitudes of soldiers with experience of battle conditions to the efficacy of prayer, or young Europeans’ liking for chocolate, or levels of depression in a Canadian urban community sample, then judges should be drawn from the same populations, namely, soldiers with battle experience, young Europeans, and Canadian city dwellers. Their opinion about whether your items appear to be about the efficacy of prayer, liking for chocolate, and depression (respectively) indicates face validity.

Predictive validity

The other methods of determining validity are often more difficult to deal with.

Predictive validity is achieved if your test predicts behaviour occurring after testing. The main problem is the worry and time involved in following up the people who did your test, as described in Chapter 1 Nevertheless, you will have to consider getting such data if you are trying to create a test that will only be useful if it can predict performance. Sometimes follow-up data may come your way quite easily. Such data can be added to your database.

• • If the data consist of scores on some measure, you can correlate scores on your test with these performance scores. See Appendix 6.3 for a method of carrying out a correlation using SPSS.
• • If the performance measure is binary (e.g. did the people succeed or fail on something, do or not do something?), the simplest way to look at associations is to do a t-test using the binary performance measure as a grouping variable, and scores on your test as the dependent variable (see Appendix 6.2).
• • If the performance measure is categorical, but there are more than two categories, then a one-way ANOVA could be done, using the performance measure as a grouping variable, and scores on your measure as the dependent variable. Appendix 6.4 describes how to do this in SPSS.

To present the results of an ANOVA, you can present an ANOVA table if there was a fairly complex research design involving two or more grouping variables, and/or a mixed ANOVA involving within- and between-subjects effects. In a case with only one grouping variable, an ANOVA table is not necessary; you only need to describe on one F ratio, and the results can be presented by quoting the F ratio, the two relevant figures for degrees of freedom, and the probability. For example,

Significant differences in prayer scores went with later religious-group-joining: F₂₉₇ = 431.176, p < 0.001. Participants were 100 British undergraduates (50 female and 50 male, aged 19-46, mean age 22.5). Those who joined a campus religious group scored significantly higher than those who did not. Post hoc comparisons using Scheffe’s test showed significant differences (p < 0.05) between those joining the two religious groups (Anglicans mean = 20.5; New Christians mean = 23.0); both of these groups scored significantly higher than those not joining a religious groups (mean = 13.52).

Construct validity

This is obtained when what you are measuring relates to other factors in ways that were predicted by the theory underlying the development of your test. If such a possibility arises, you should be able to carry out tests of associations between variables using correlation, unrelated t-test, ANOVA or methods of partialling out spurious effects described in this chapter, and in Appendix 6.

Confounded variables

Sometimes two or more variables may be confounded with each other (they co-vary), and this can affect the results of validity testing. ‘True’ relationships can be obscured.

For example, if we found that students who believed in the efficacy of prayer were more likely to join religious groups on campus, we might begin to worry about whether this was a true effect if we noticed an age difference between those joining religious groups and those not joining. Perhaps it is really age that is related to beliefs about prayer, and not religious group membership as such? Or perhaps both factors are related to beliefs about prayer?

If you suspect that there are confounded variables in your data, select the appropriate analysis from Table 6.2.

All of these tests will express the ‘true’ association between the dependent (outcome) variable and, in turn, each of the independent (predictor) variables that have been entered into the analysis, while taking out the ‘spurious’ contribution of the other independent variables in the analysis.

Appendix 6.5 provides some further details of how to compute these, but if you wish to use any of these methods of analysis, you may need to consult a text such as that by Tabachnick and Fidell (2013), a guide such as that by Field (2017) or Pallant (2016), and/or your statistics software manual.

Table 6.2 Selecting the right analysis when there are confounded variables

a For logistic regression, independent variables may be a mixture of continuous and categorical (dichotomous).

b For ANCOVA, the covariate(s) should preferably be continuous.

c For multiple regression analysis, independent variables may be a mixture of continuous and categorical (dichotomous).