Approaches to Performance Measurement Social Performance Measurement

There is a wide, if not yet deep, literature on social performance measurement for financial institutions, a review of which yielded several conclusions. First, need for credible social performance measurement of financial institutions is growing as a result of increasing demands by investors for social investment opportunities and because of pressures on mainstream financial institutions to report on a double or triple bottom line, which means that “we are all double or triple bottom line now.” Specifically, new and existing investors need simple, credible measures to distinguish the social performance of financial institutions more broadly.

Second, true impact measurement is increasingly regarded as too expensive and difficult to achieve because of problems in adequately defining a control group outside of ideal experimental conditions, including randomized assignment of cases. Rather, the focus has moved towards measuring forms of outcome (for example, see Chapter 6, “Research Design Issues for Measuring CDFI Performance and Impact”). An outcome is a desired change resulting from an output or series of outputs. Output and outcome need to be linked by a theory of change that explains causality. While outputs are easy to measure – for lending banks, volumes of loans granted constitute output – outcomes are much harder. If the causal chain between output and outcome were demonstrated for a product, one could rely on collecting output measures alone.

The evidence that increased financial intermediation at a local microlevel – such as a census tract – leads to positive social outcomes at that level has not yet been demonstrated. This is in part because of the limitations of outcome-related data at tract level and, more so, because of the perennial questions about spillover effects across boundaries that may dilute the evidence, though not the reality, of impact within an area. Recent empirical work by Galster, Hayes, and Johnson (2006) on parsimonious indicators of neighborhood vitality may lead to the definition of tractable indicators that accurately measure changes in neighborhood characteristics over time.

While there is little or no strong evidence yet of positive outcome effects of intermediation activity at a tract level, there is finance literature that has demonstrated a causal link between volume of intermediation and economic growth at the national and international levels (summarized in Levine 2005). Only in recent years has the direction of causality been definitively isolated: while the effect is clearly bidirectional, it is now accepted that financial intermediation has a “first-order positive causal impact on economic growth." However, this relationship is not simple or linear. In fact, in one of the few empirical studies done at the state level, Dehejia and Llevas-Muney (2003) found evidence of positive causality in distant history but suggest that this effect exists only within bounds. For example, overlending as the result of a credit bubble will usually have a negative outcome on subsequent growth. World Bank researchers Beck, Demirguc-Kunt and Levine (2004) extended this theory of change further than impact on economic growth alone. Using a cross section of fifty-two developed and developing countries from 1960 to 1999, they showed that increased intermediation is related causally to other socially desirable outcomes – reduced poverty and income inequality.

Macrolevel findings give more credibility to the claim that “output" measures of intermediation volumes are linked to positive outcomes, although this is not definitive. However, neither can output-related measures be dismissed as irrelevant to the search for parsimonious performance indicators for CDFIs that operate on a local or regional level.

In the literature, there are a variety of approaches that capture direct outcomes, but as yet, none provides a widely accepted way of comparing social performance of financial institutions across a broad spectrum (Clark, Rosenzweig, Long, and Olsen 2003; Kramer and Cooch 2006).

As methods have proliferated, even financial institutions committed to social impact are increasingly sensitive to the cost in time and resources of complying with reporting regimes for measuring performance and impact (e.g., Coastal Enterprises 2006). This underlines the need to use existing data sources as much as possible, accepting the likely trade-offs between precision and the cost of data collection. An example from a related, although very different, sector – mutual funds – shows how widely used performance measurement tools can be developed on the basis of publicly available data alone.

Mutual Fund Rating: An Example

Morningstar started its mutual fund rating services in 1986, using publicly available information on fund performance to create Morningstar ratings used by retail investors to navigate the increasingly complex range of products and offerings. Morningstar's methodology is relatively simple (corporate.morningstar. com/US/asp/detail.aspx?xmlfile=279.xml):

Step 1: Create consistent categories within which meaningful peer-group comparison can be undertaken. Morningstar currently has some sixty-two categories of funds, based on characteristics affecting performance such as the size and focus of fund, and has a methodology to control for movement between categories.

Step 2: Measure risk-adjusted, return within each category. Morningstar uses moving averages of risk-adjusted return, which for mutual funds are easily available and relate to the investment experience of retail investors.

Step 3: Rank funds within each category using this criterion, in fact, for ease of investor use, Morningstar assigns stars on a bell curve so that a few top performers get five stars, most get three stars, and so on.

For Morningstar and others, the hardest parts of the process are not the latter steps but the first: creating a credible and robust means of peer classification. This is a challenge for CDBIs, which differ in size, focus, and approach.

< Prev   CONTENTS   Next >