Experimental Methods for Impact Evaluation of CDFI Product Lines

Experimental methods are generally considered the first-best option in program evaluation. In such evaluations, "treatment" and “control” groups are randomly selected from some broader population. While experimental designs are more often associated with household- or firm-level data, it is possible to conduct experiments at the neighborhood or community level.

When applied to CDFI activity, experimental impact evaluation might look something like this: A randomly chosen subset of applicants would be provided with CDFI products or services, while the remaining applicants would not. Thus, the CDFI services are actually withheld from a control group. The experimental group cannot consist of all households or firms that express a desire for CDFI products. Otherwise, a selection bias problem would exist, because applicants are likely to possess some traits (e.g., motivational characteristics) that make them more likely to do better (or worse) on the chosen outcome measures. Given sufficient numbers of applicants in each group, randomization will insure that the two groups are “mean equivalent” on both measured and unmeasured characteristics. Given adequate randomization, outcome data can be collected on each group, and differences in mean outcomes are attributed to the program.

There are a number of challenges to implementing experimental studies for CDFls. First, gathering outcome data from members of the experimental and, especially, the control groups can be difficult. Acquiring data on the firms or individuals receiving assistance is conceptually straightforward, albeit not without substantial challenges and expense. It is even more challenging to gather data on firms or households that do not receive loans or assistance. What incentives do they have to provide data and comply with research requirements? A second problem is the difficulty of denying services to control group firms. Some CDFIs may view such denial as unfair, although such experiments may be easily justified in a context of scarce resources. Another problem is that denying services to otherwise eligible firms may be politically difficult. Finally, for some programs, the deal flow may be so weak that there are insufficient firms or households to constitute a control group.

Experiments Using Geographic Areas

In addition to experiments at the individual level, researchers can conduct experiments structured by geographic clusters such as neighborhoods. The classical experiment can be conducted by identifying the universe of eligible geographies – based on programmatic criteria – and then randomly assigning areas to treatment and control groups. As in the case of the individual-level experiment, services would be withheld from the control group areas and differences in outcomes between the two groups would be tracked. One advantage of geographic experiments is that it is often easier to withhold the offering of a product line from some set of otherwise eligible geographic areas rather than attempting to withhold services from different individuals or Anns. A CDFI could randomly choose, from a larger set of eligible neighborhoods, a group of neighborhoods in which its products or services would be made available. This might not be done for ah products but for a specific product line. Politically, such an approach might prove more feasible in the case of new products that are not already offered in a region.

Another advantage of geographic analyses is that outcome data, while perhaps not plentiful, are not being collected from individuals. Rather, census or other, more frequently collected small-area data may be sufficient to provide quantitative indicators for key outcomes. This is advantageous in the case of control groups, from which data may be hard to gather.

One challenge that immediately arises with group or geographic randomization is the requirement to choose large enough numbers of geographies in both experimental and control groups that sufficient randomization is ensured. Bartik (2002) suggests a rule of thumb of at least twenty areas in each of the treatment and control groups, though he provides no justification. Bloom (2005) suggests that, depending on the statistical power desired, groups of twenty geographies may be less than sufficient for randomization.

< Prev   CONTENTS   Next >