PLANNED COMPARISONS VERSUS POST HOC COMPARISONS
Generally speaking, experiments (and particularly tests of behavioral interventions in a Phase II pilot randomized trial, Phase III efficacy trial, or Phase IV effectiveness trial) are usually conducted with a specific hypothesis in mind. These planned analyses are derived from theory and are pivotal and essential tests. Even so, the more comparisons or analyses conducted, the more Type I error will be made when the null hypothesis is true. Although everyone agrees that planned comparisons should be limited in number, there is no real agreement on what this number should be. One suggestion is to restrict the number of comparisons to the number of degrees of freedom associated with the treatment source of variances. Other researchers suggest a special correction such as the Bonferroni or Dunn test to compensate for the Type I error. Researchers who pursue post hoc comparisons often use Tukey’s tests, for example, which will help generate planned comparisons for subsequent experiments.