Research Design Issues for Measuring CDFI Performance and Impact

Dan Immergluck

Community development financial institutions (CDFIs) represent a key innovation in community development. CDFIs are loan funds, venture capital funds, microloan funds, credit unions, and banks that include community development at the core of their mission. Federal, state, and local governments, as well as private foundations and financial institutions, have invested billions of dollars in CDFIs over the last twenty years to revitalize communities through improved access to affordable capital and credit. Both private social investors as well as governments are increasingly asking for evidence of CDFIs' impacts.

While recognizing the substantial limitations and constraints of program evaluation in this field, I address major research design questions for CDFIs:

1. What is the appropriate level at which to examine the impacts of CDFIs? Should we measure impacts at the level of the entire subsector, a product line, or a specific product? Can we identify product lines that are relatively better candidates for impact evaluation?

2. What are some of the fundamental methodological approaches that might be used to conduct summative impact evaluation for specific CDFI product lines?

CDFI Borrower-Investee Types, Product Groups, and Product Lines

It is important to develop a typology of CDFI products and services, critical for understanding the context in which to measure impacts and to prevent an overaggregation of different products. Literature on CDFIs is widely available, much focused on distinguishing types of CDFIs first and then, within each CDFI class, examining projects – business, housing, nonprofit facilities, for example – financed by that category of CDFI (CDFI Data Project 2004; Tholin 1994). While the literature sometimes focuses on specific products, little effort has been made to disaggregate them into predominant products.

Approaching questions of CDFI outcomes by first thinking about the different types of institutions (e.g., community development bank versus community development loan fund) encourages a focus on financial performance measures (PMs), such as financial health. This may not be a bad thing to do, especially if financial sustainability is of interest. If questions concern the mission-oriented outcomes of the CDFI sector, however, it may be more productive to begin with outcomes, not delivery systems and their financial strength.

Some researchers think about the types of borrowers and investors that CDFIs target (Benjamin, Rubin, and Zielenbach 2004). This focus is closer to the approach taken in this chapter. I first consider the sorts of borrowers or investees that CDFIs target, then I examine the products offered.

Table 6.1 provides a typology of CDFI product lines by financial product group and borrower/investee (“customer") and financial product: households, microentrepreneurs, low-to-moderate growth firms, emerging or high-growth small firms, nonprofits (excluding housing development organizations), and real estate developers (both for-profit and nonprofit). The table does not include every product or service and technical assistance or educational services. It also does not consider any broader community development or community building activities not involving financing. These services could be added to the typology, but they were omitted to focus on financial products and services.

Financial product groups include bank accounts and related products, unsecured consumer loans and car loans, single-family mortgages, multifamily real estate loans, commercial real estate loans, business equity financing, and business term loans and lines of credit. These groups are defined by the structural characteristics of the products (e.g., mortgages versus term loans). Some CDFI product groups focus on one customer (e.g., low-cost bank accounts), while others serve multiple customers (e.g., commercial real estate loans).

The table lists specific product lines. For example, for the household customer category, the single-family mortgage group is disaggregated to include home purchase, home improvement, and refinance loans. Even within these categories, we can disaggregate further. For example, home purchase loans might include junior as well as senior mortgages.

The versatility of some product groups across multiple customer types is indicated by the vertical arrows, which signify that a group covers more than one customer type. For example, CDFIs may extend term loans to microenterprises, small Anns (from low- to high-growth), or nonprofit organizations. While the rates and terms of these loans may vary by customer type, the basic loan structures are likely to be relatively similar.

Another important component of Table 6.1 is the set of acronyms at the bottom of the table, identifying types of CDFIs offering products. Identification of product lines and their organization into customer and product groups will assist with thinking about measuring outcomes and gathering data on comparable non- CDFI products.

Table 6.1. A Typology of Major CDFI Financial Product Lines, Excluding Technical Assistance and Education

Financial Product Groups

Type of Borrower/ Investee



Unsecured Loans & Auto Loans

Multifamily Single-Family Real Estate Mortgages Loans

Commercial Real Estate Loans




Business Term Loans & Lines of Credit





checking accts CU, CB

Payday alternatives Overdraft products Student loans Auto loans CU, CB


• senior/junior Improvement

• senior/junior Refinances

• rate/term

• antipredatory LF, CB, CU


Low- to Moderate- Growth Small Firms

Emerging, High- Growth Small Firms



Land financing Purchase

• senior/junior Refinance

• senior/junior Predevelopment

grants/ recoverables LF, CB, CU

Equity; unsecured debt; royalty financing VC

Term loans • senior/junior Lines of credit LF, CB, CU ML for microentrepreneurs

Nonprofits/Charter Schools/Public Facilities (Excluding Housing Developers)

Real Estate Developers (For-Profit and Nonprofit)




• senior/junior Refinance

• senior/junior

Key to institution types: LF-loan funds; CB-community development banks; ML-microlenders; CU-CDCUs; VC-CDV

Figure 6.1 A Simple Logic Model for a CDFI Home-Improvement Loan Product Line

A Simple Logic Model for a CDFI Home-Improvement Loan Product Line

Performance Measurement Versus Summative Impact Evaluation

When people talk about CDFI “impacts," what do they mean? Some may mean “outcomes" in PM (Hatry 2007). Outcomes are the purported, effects of completed CDFI activities – called outputs. They are the concrete, desirable conditions a program hopes to bring about for individuals and communities. PM distinguishes between end outcomes, intermediate outcomes, and outputs, with the former being defined by the CDFI mission and objectives. End outcomes include access to quality, affordable housing, decreased unemployment or underemployment, and higher wages. Intermediate outcomes are recognized as good in and of themselves but are most important in their critical role in bringing about desired end outcomes. An example is the expansion of firms in a region expected to lead to reduced unemployment or underemployment in a community. Outputs are completed program activities, which in themselves have little intrinsic value. They are desirable only in that they lead to intermediate or end outcomes.

Outputs and outcomes are tied to logic models, describing and analyzing program theory, operation and performance. A logic model for a hypothetical CDFI with a home improvement loan product line is presented in Figure 6.1.

Those adopting the PM perspective focus on defining and distinguishing outputs, intermediate outcomes, and end outcomes and in obtaining data needed to constitute sound, quantitative indicators. Distinguishing outputs and intermediate outcomes, for example, can be a key to PM.

In discussions of CDFI outcomes, impact frequently appears to be used when what is really meant is end outcome. This confusion occurs perhaps because an outcome is indeed a purported effect; that is, PM does not require that evidence support the notion that the program caused the outcome.

Strictly speaking, impact is best left to the domain of summative impact evaluation (SIE). SIE attaches a meaning to the term impact that is linked to causality. Measuring impact requires measuring the difference between CDFI outcomes and whatever would have occurred without CDFI intervention – the “counterfactual.” SIE identifies causality between the program and the outcome and involves measuring the extent of that causality, or how much of the measured outcome can be attributed to the program.

To SIE practitioners, the challenge becomes identifying the counterfactual or developing a reasonable estimate of it. Although the evaluator would prefer measuring impacts on end outcomes, she may be willing to settle for measures of impact on intermediate outcomes or outputs if she can be confident of establishing a counterfactual for them. This is true if there is a strong, well-established link between these more proximate effects – outputs or intermediate outcomes – and end outcomes. For example, if evidence demonstrates that access to capital is a key driver of a desired outcome, an evaluator's concerns might lie chiefly in whether, in the absence of CDFI activity, borrowers would have access to capital from elsewhere and, if so, at what costs and conditions compared to the CDFI products that they received.

Some SIE practitioners may look upon PM with skepticism. They may view efforts to measure outcomes without identifying the extent to which such outcomes can be attributed to CDFIs as a hollow task. PM practitioners are more pragmatic, arguing that, together with qualitative and interpretive knowledge of an initiative and its context, PM can support a reasonable argument that impact is likely or unlikely and perhaps tell us something about the scale of any likely impact. PM can be used as a heuristic tool that, when combined with other, often less formally or quantitatively acquired knowledge, can contribute to assessing impact.

Regardless of whether an analytical exercise provides a strong measure of impact in a given product line, comparing CDFI outcomes to contextual data regarding needs or similar market activity can be useful. It can give us some sense of how much aggregate difference a program might be making at its current scale. For example, assume we know that all of the microenterprise lenders in Big City A made a total of ten microloans in a recent year and the lenders in Big City B made a total of 200 microloans in the same year. Let us also assume that the cities were similar in size, economic demography, and bank lending. This information would be useful in comparing microlending across cities. Moreover, it seems fair to say that microlenders in Big City B were likely to be having a greater overall impact on access to capital for very small firms than those in Big City A. Of course, the SIE practitioner might question whether microlenders in either city were actually having any impact, and that is a reasonable question. However, the point here is simply to consider outcomes and their context and not to quantify or prove impact. It seems that in many cases, even this limited information would provide us with a good deal more information than we currently have. Of course, it is important not to draw definitive conclusions from such analyses that microlending programs in Big City A “don't work." We have not been presented sufficient information to determine impact.

Depending on the intermediate and end outcomes of the CDFIs in a particular subsector, we might look for data to place these outcomes in an appropriate context. If the microlenders, for example, are looking to influence the behavior of conventional banks toward microenterprises (e.g., encourage them to make more loans to very small Anns) – what might be called an institutional or structural impact – it is important to understand the scale of outcomes relative to the activity of the industry that is the target of influence.

Relative scale is just one context. Others include the relative challenges of working in different places. It may be that in one place, a certain outcome would be quite impressive, while in another, it would not be so remarkable. To complement quantitative measures of performance or outcomes, qualitative data, including information from informant interviews, could prove helpful.

In the realm of institutional impacts, scale matters for reasons beyond just the subsector's ability to deliver more capital to firms or households. Scale matters because the subsector becomes substantial enough to affect lender behavior. Mainstream lenders' (e.g., banks, thrifts, finance companies) actions are affected by a complex set of forces derived not only from traditional competitive pressures but also from regulatory obligations, reputational concerns, policy developments, and their involvements in the corporate and civic life of a community. The presence of an active microlender may encourage bankers to view microenterprise as an important activity, to value becoming more involved with smaller firms, and to partner with a CDFI they see as bringing reputational or Community Reinvestment Act (CRA) benefits.

My intent is not to take sides on the PM versus SIE perspective but rather to assess what may be achievable from both. PM is less ambitious and more feasible than SIE for most CDFI applications. It will be sufficient in gaining confidence that a CDFI is headed in the right direction and is likely – or not – to be having some meaningful impact. Systematic, quantitative evidence of impact is not always required or appropriate.

PM can provide information that, together with a broad and perhaps less than systematic set of other, often less quantitative knowledge, gives decision makers comfort that a program is making a difference. PM provides just one heuristic tool that, as part of what Schorr (2003) calls "multiple ways of knowing,” can contribute to a much less systematic but more pragmatic and perhaps achievable way to discern – if not precisely measure – impact.

Although SIE studies may promise more systematic and quantified impact measures, their greater ambition brings with them greater risks for error. As in any social science research, studies should be rigorously reviewed and never be considered conclusive in and of themselves.

At the same time, to dismiss the potential for research that may give us better information than we currently have on CDFI impacts may condemn the field to a sort of purgatory of “we can never convince skeptics that we are making a real difference.” In the end, in a world of many programs and policies competing for scarce resources, merely resigning to an inability to measure impacts may not be good enough to sustain the field. It is important to recognize the limits of impact research, especially as it exists today, but it is also important to invest in improved data and methods so that the CDFI field has an opportunity to document the differences it makes in lower-income and disadvantaged communities.

< Prev   CONTENTS   Next >