Quasi-Experimental Designs and Bias

A quasi-experimental design, by definition, will exert variable control over the major biases to internal validity. Thus, caution is advised in interpreting results from a quasi-experimental study. Selection and historical bias will always be the initial, major biases to examine when a quasi-experimental design is used. After matching on one or two baseline variables, it usually becomes impossible, in most cases, to match on a third major population baseline characteristic. The initial, pre-treatment differences from known, and especially numerous unknown, selection biases make all E and (C) group adjusted post-test comparisons challenging.

There is a lack of consensus in the literature about what analytical technique is the most appropriate for results produced by non-randomized comparison group designs. This issue has been discussed in the social and behavioral science literature for 35 years, for example, see Cook and Campbell (1979) and Kenny (1979). Statistical adjustment methods cannot fully adjust for known, and cannot adjust for multiple unknown, selection characteristics of participants or matched groups/sites. Grossman and Tierney (1993) in “The Fallibility of Comparison Groups,” provided an excellent methodological and analytical discussions about the use of a (C) group and quasi-experimental designs. They noted: “despite using a comparison group explicitly designed to overcome many self-selection issues endemic to quasi-experimental methods and using a variety of statistical methods to control for selection bias, quasi-experimental designs are still subject to the threat that the comparison group did not adequately represent a non-treatment state.”