Regression to the Mean

A particular threat to causal inference with this study design is regression toward (or to) the mean. This common phenomenon suggests that if a variable value for a case is extreme on its first measurement, it will tend to be closer to the average on its second measurement. Regression to the mean is why it is so important to have matching treatment and control groups before the treatment. In the example of traffic calming in New York City, presented in Chapter 10, streets selected for traffic calming treatment are likely to have higher crash rates than average before the treatment, and thus they would likely regress to mean in any event in a second measurement, with or without traffic calming.

Propensity Score Matching

Propensity score matching (PSM) is a type of pretest-posttest design with a control group (i.e., two group pretest-posttest design). PSM has been widely used to overcome non-random assignment of treatment in the evaluation of social programs (Oakes & Johnson, 2006). In evaluation studies, individuals in the treatment group are likely to differ systematically from those in the control group. For example, households living in suburban regions could be more affluent than their counterparts in downtown, a result of residential self-selection. Therefore, the observed difference in behavioral outcomes between the groups is confounded by residential self-selection. Statistically, it generates a biased estimate of the treatment effect.

The propensity score is defined as the conditional probability of assignment to a particular treatment given a set of observed covariates (Rosenbaum & Rubin, 1983). The propensity score for matching purpose can be best calculated when we include as many variables as possible that serve as confounders for the causal path between independent and dependent variables.

The first step is to develop a binary logit model to estimate a propensity score using the subsample of treatment group and control group. Then you can match each subject in the treatment group with those in the control group based on their propensity scores. Nearest neighbor matching is the most common match approach and Caliper length of 0.03 is often used for matching, meaning that for a treatment observation, you search matching control observations whose propensity scores are within 0.03 of the score of the treatment observation (Austin, 2009). The choice of caliper width is a trade-off between bias and variance, because a too wide width will select control cases too different from treated ones and a narrower width will reduce sample size, leading to more variance in the treatment effect (Austin, 2011). Third, to check the quality of the matching, you can evaluate whether the matched subjects in the treatment group are systematically different from those in the control group, using a /-test.

The final goal of PSM is to compute the true impact of treatment on an outcome variable. Once the matching is complete, you can compute the average treatment

Quasi-Experimental Research 311 effect (ATE), which is computed as a difference in the outcome variable between treatment and control groups (see Figure 14.2). Sometimes, the ratio of ATE over the observed influence is calculated as the proportion of the observed difference that can be attributed to treatment itself. A planning example of propensity score matching is presented in the next section.

The validity of propensity score matching relies on the assumption that unobserved characteristics do not influence treatment participation. The propensity scores are only as unbiased as the predictors included in their calculation (Beal & Kupzyk, 2014). As a result, failing to include an important confounder in the calculation of a propensity score will lead to biased results (Stürmer, Schneeweiss, Avorn, & Glynn, 2005). Another assumption is that when properly modeled, the distributions of propensity scores are close to equal between two groups. When this assumption is not met, propensity scores will also lead to biased results (Rosenbaum & Rubin, 1983).

Regression Discontinuity (RD)

Regression discontinuity (RD) has been rarely used in planning research, but has potential for evaluation research of a specific public program. Assignment to the treatment or the control group is made based on a cutoff value, or threshold (e.g., test score, income, density). Although assignment is not random, assignment to, and differences between, the control and treatment group near the cutoff value are assumed to be random. Trochim and Donnelly (2008, p. 216) define RD as “a pretest-posttest program-comparison group quasi-experimental design in which a cutoff criterion on the preprogram measure is the method of assignment to group."

An advantage of RD design is that it is appropriate when a researcher wants to target a program to those who most need or deserve it (Trochim & Donnelly, 2008). For example, school children who obtain scores that fall below a cutoff value on an achievement test can be assigned to remedial training and compared to those who

Figure 14.2 The Relationship Between Observed Effect and Treatment Effect

Source: Cao, Xu. & Fan (2010) obtain higher scores. Because of the cutoff criterion, treated and control groups are deliberately and maximally different on pre-program characteristics in RD designs. Contradictory to the assumption in experimental or other quasi-experimental designs that two groups are equivalent prior to the program, the RD design assumes that in the absence of the program, the pre-post relationship would be equivalent for the two groups (Trochim & Donnelly, 2008).

In RD designs, you draw a regression line for each group, cut at the cutoff value. A discontinuity in regression lines indicates a program effect. In other words, a program effect in the RD design is not calculated as a difference between the posttest measures of two groups, but by a change in the pre-post relationship at the cutoff point. Trochim and Donnelly (2008) point out that while the RD design is strong in internal validity, it needs as much as 2.75 times the participants as a randomized experiment, to achieve the same level of statistical accuracy.

Take a hypothetical example used by Lan Deng and Lance Freeman in the Summer 2011 edition ofJPER (Deng & Freeman, 2011). In their example, neighborhoods with 15 percent or less homeownership rates are offered a public program to increase their homeownership rate. If the intervention has an impact, then neighborhoods barely qualifying for the program should be significantly different than neighborhoods barely missing the qualification after the intervention. Figure 14.3 demonstrates two possible research results. In the first regression, the public program has no impact. The regression line is exactly the same for both the treatment group and the control group. The homeownership rate increases at the same rate for both the treatment group and the control group. In contrast, the second regression clearly shows an interruption caused by the treatment; homeownership rises from 15 percent to 60 percent in the treatment group after the intervention. Neighborhoods just missing the cutoff see no change after the policy is implemented. The authors argue that the RD design can be especially useful in evaluating targeted place-based programs.