There is one more piece to the logic of hypothesis testing. The choice of an alpha level lays us open to making one of two kinds of error—called, conveniently, Type I errors and Type II errors.

If we reject the null hypothesis when it’s really true, that’s a Type I error. If we fail to reject the null hypothesis when it is, in fact, false, that’s a Type II error.

Suppose the government of a small Caribbean island wants to increase tourism. To do this, somebody suggests implementing a program to teach restaurant workers to wash their hands after going to the bathroom and before returning to work. The idea is to lower the incidence of food-related disease and instill confidence among potential tourists. The program is implemented in a small number of restaurants as a pilot study and you get to evaluate the results.

The null hypothesis is that the average amount of food-borne disease will be the same in restaurants that don’t implement the program and in restaurants that do. The data show that H_{0} is false at the .05 level of significance. This is great news. The program works. But suppose you reject the null hypothesis when it’s really true—the program really is useless. Your Type I error (accepting H_{0} when it’s false) sets off a flurry of activity: The Ministry of Health requires restaurant owners to shell out for the program. And for what?

The obvious way to guard against this Type I error is to raise the bar and set alpha at, say, .01. That way, a Type I error would be made once in 100 tries, not 5 times in 100. But you see immediately the cost of this little ploy: It increases dramatically the probability of making a Type II error—not rejecting H_{0} when we should do exactly that. In this case— where H_{0} being false means that the program works but you don’t detect that—the result is that you place people at greater risk of food-borne disease.

In a probabilistic science, we are always in danger of making one or the other of these errors. Do we try to avoid one kind more than the other? It depends on what’s at stake. A Type I error at the .01 level for an HIV test means that 1 person out of 100 is declared HIV-free when they are really HIV-positive. How big a sample do you need to get that error level down to one out of a thousand? This is the kind of question answered by power analysis, which we’ll look at in chapter 21.