Reddaway on Statistical and Economic Significance
In the tradition of the Cambridge Keynesians, Reddaway maintained a healthy scepticism about the use of econometrics. This attitude was based on the familiar premise that relevant variables and their possible real world interactions far exceed the capabilities of the architecture of econometrics. One of the most distinguishing features of Reddaway’s applied economics was his insistence on the fundamental distinction between statistical significance and economic significance. At an intuitive and elementary level, the issue is quite simple. A difference between two variables can be permanent and tangible without being ‘significant’ in statistical terms. On a similar note, a difference in variables can be highly statistically significant and yet not be valuable in practical economic analysis and therefore not have any economic significance. It is unfortunately common in applied economics not to recognise this essential distinction between economic and statistical significance, and is invariably ignored by many researchers, even in the top-rated journals.
Reddaway was convinced rather by simple and straightforward economic arguments than convoluted ones. An example of Reddaway’s criticism of the improper use of statistics is his disparaging review of Rayner and Little’s Higgledy Piggledy Growth. While conceding that ‘the power of computer-aided research batteries to pulverise untenable arguments is certainly impressive’ (Reddaway 1967: 595), Reddaway is unconvinced by Rayner’s strength of final statement in his analysis and describes it as ‘an almost classic case of a confusion between failure to prove concisely that there. [exists] .a tendency’ and ‘coming to a conclusion that there is not’ (ibid.: 595). Choosing to remain ‘rough and relevant’ only where need be, Reddaway valued easy readability, good scholarship, and detail in his research and openly criticised where he found these qualities lacking. The use of statistical descriptors was highly relevant to Reddaway, who was always a critic of indirect indicators without precise definition—such as the description of ‘performance’ versus ‘growth’ in discussing corporate strength over time (Reddaway 1967).
Reddaway’s argument was founded upon the work of Jerzy Neyman and Egon Pearson, who were two highly distinguished mathematical statisticians working during the 1930s. They identified two types of errors: Type 1 and Type 2, writing:
Is it more serious to convict an innocent man or to acquit a guilty? That will depend upon the consequences of the error. Is the punishment death or a fine; what is the danger to the community of released criminals; what are the current ethical views on punishment? From the point of view of mathematical theory all that we can do is to show how the risk of errors may be controlled and minimised. The use of these statistical tools in any given case... must be left to the investigator (Neyman and Pearson 1933: 296; italics added).
Wald (1939) developed this argument further and suggested that
[t]he question as to how the form of the weight [loss] function.. .should be determined is not a mathematical or statistical one. The statistician who wants to test certain hypotheses must first determine the relative importance of all possible errors which will depend on the special purposes of his investigation (ibid.: 302).
Reddaway’s important insistence on differentiating between economic and statistical significance in the practice of research was not unfounded. McCloskey and Ziliak (1996) carried out a survey to discover how economists use the concepts of statistical and economic significance and how consistent they are in the use of these concepts. The results indicated a considerable amount of ambiguity. Indeed, in their influential article, ‘The Standard Error of Regressions’, they found that out of 182 full-length papers published in the American Economic Review in the 1980s, 70% did not distinguish between economic and statistical significance. Other principal findings include the following (ibid.: 106-107):
- • At the first use of statistical significance, typically in the “Estimation” or “Results” section, 53% did not consider anything but the size of t- and T-statistics. About one third used only the size of t- and T-test statistics as a criterion for the inclusion of variables in future work.
- • 72% did not ask “How large is large?” That is, after settling on an estimate of a coefficient, 72% did not consider what other authors had found; they did not ask what standards other authors have used to determine ‘importance’; they did not provide an argument one way or another whether the estimate P = 0.999 is economically close to 1.0 and economically important even though ‘statistically different from one.’
- • 59% used the word ‘significance’ in ambiguous ways, at one point meaning ‘statistically significantly different from the null,’ at another ‘practically important’ or ‘greatly changing our scientific opinions,’ with no distinction.
- • Despite the advice proffered in theoretical statistics, only 4% considered the power of their tests. 1% examined the power function.
- • 69% did not report descriptive statistics...that would allow the reader to make a judgment about the economic significance of the results.
- • 32% admitted openly to using statistical significance to drop variables ... One would have to have more evidence than explicit admissions to know how prevalent the practice is in fact.
- • Multiple-author papers.more often spoke of ‘significance’ in ambiguous ways, used sign econometrics, did not discuss the size of estimated coefficients, and found nothing more than the size of test statistics to be of importance at the first use of statistical significance (Table 2 in ibid.: 107).
Despite the hope that following such a paper in a leading journal the prevailing situation might improve, McCloskey and Ziliak (2004) found in a follow-up article that this was not the case. Using a test of statistical significance, they found that out of 137 papers published in the American Economic Review, a high 82% ‘mistook statistically significant coefficients for economically significant coefficients’ (ibid.: 333). The paper indicated that a large majority—81%—considered it acceptable to look at the sign of a coefficient rather than its magnitude, thus ignoring the distinction between statistical and economic significance. McCloskey and Ziliak suggest that this state of affairs may be due to the chaotic rat race in getting papers published by academic journals.
These findings indicate the difficulties of correctly specifying the nature and meaning of statistical significance, which Reddaway always emphasised in both his teaching and his research. Studies find that even research published by economists attached to top-tier universities (e.g. Chicago, Harvard, MIT, Princeton, Stanford, and Yale) do not fare much better in this respect. As such and somewhat unfortunately, the ordinary usage of econometrics in economics, as Reddaway had always maintained throughout his career, more often than not incorrectly takes statistical significance to be the same as economic significance. In later years, the distinguished statistician, the late Morris DeGroot (1975: 496-497) observed:
It is extremely important...to distinguish between an observed value of U that is statistically significant and an actual value of the parameter ... In a given problem, the tail area corresponding to the observed value of U might be very small; and yet the actual value.. .might be so close to [the null] that, for practical purposes, the experimenter would not regard [it] as being [substantively] different from the [the null].
[I]t is very likely that the t-test based on the sample of 20,000 will lead to a statistically significant value U ... [The experimenter] knows in advance that there is a high probability of rejecting [the null] even when the true value.. .differs [arithmetically] only slightly from [the null].
Had Reddaway taken the time to develop his argument and formalise it, he might have had the kind of influence that McCloskey and Ziliak have achieved.
