This is a hot topic in social science. Fifty years ago, researchers began arguing that statistical tests of significance are virtually useless (Rozeboom 1960) and the debate has raged on ever since (see Ziliak and McCloskey [2008] for a review) (Further Reading: significance tests).

I wouldn’t go that far. Consider the hypothesis that the universe is expanding. As Wainer (1999) points out, being able to reject the hull hypothesis at p < .05 would be quite a contribution. Will Dr. X be denied tenure a year from now? Lots of people, says Wainer, would be happy to know that they could reject the null hypothesis at, say, p < .001.

It’s true that if you don’t have a representative sample, then a test of statistical significance doesn’t allow you to generalize beyond your particular sample of data. On the other hand, if you get significant results on a nonrandom sample, at least you can rule out the operation of random properties in your sample (Blalock 1979:238-39).

Use tests of significance but remember that they aren’t magical and that the .01 and .05 levels of significance, although tribal customs, are not sacred. They are simply conventions that have developed for convenience over the years. Greenwald et al. (1996:181-82) offer some useful advice about reporting p values.

1. In many situations, it’s enough to use simple asterisks in your prose to indicate statistical significance. But if you need to report p values, then do so with an = sign, not with a < or > sign. If a p value is .042, don’t report it as p <.05 (‘‘the probability is less than .05’’). Just report it as p = .042 and be done with it. Probability, just like the confidence we have in probabilistic results, is a continuous variable. Why cut out all that information with arbitrary cutoffs?

2. A single p value of .05 is as an indicator of a relation, but is not convincing support for a hypothesis. By tradition, researchers almost never report probabilities that are greater than .05. Five repeated results of p = .06, or even .10, are more convincing than a single result of p = .05 that something’s really going on.