Evaluation Design: Controlling for Biases to Internal Validity
School-based evaluation studies need to be especially concerned about two major sources of bias: (1) selection biases caused by E versus C school/class/ student differences and participation rates at baseline; and (2) selection biases caused by non-random E versus C school/class/student attrition rates at follow-up. In an evaluation where the number of units of treatment and analysis, for example, schools or clinics, available to randomize to the E group is < 10 or C group is < 10, the need to balance independent predictors of student behavior is compelling. Potential, large selection biases in a multi-school evaluation may be addressed by applying a stratified, matched GRCT.
If school-student heterogeneity within and between districts is large, as was apparent in this evaluation, simple random assignment of schools would not achieve E group and C group baseline equivalence. Stratification within each district, matching of dyads, and randomization of a large number of schools could significantly increase control over the large number of known and especially unknown independent characteristics that predict student behavior. Stratification and matching before randomization, if a sufficient number of schools are available, should substantially increase the probability of E group and C group equivalence and should increase statistical power.
Conversely, a quasi-experimental design, for example, matching only (Design #2), would introduce multiple, serious methodological problems. Matching, without randomization, particularly if the total number of units is 8-10, and only 4 or 5 E and C sites are available, will not have sufficient statistical power and will not provide sufficient control for large selection biases in participation and attrition rates. It will not create baseline equivalence for school/teachers/students. The stratification and matching methods and group randomized design used to evaluate Healthy and Alive had not been previously used in school-based HIV/AIDS evaluation studies.