POSITIVE OUTCOMES DO NOT GUARANTEE POSITIVE PROGRAM IMPACTS
Using outcomes instead of impacts is based on the presumption that the outcomes identified in a program’s theory of change are valid and reliable indicators of positive or desired impacts. That is what makes their use an acceptable and potentially important element of modern performance measurement. But, in order to be used to assess program effectiveness, their validity must be supported by well-established theory and relevant supportive research. That provides the grounds for a reasonable presumption that a measured outcome is likely to have the desired impact—always remembering that the presumption is being used in performance management, not policymaking. That is to say, the advantages of doing so outweigh the lack of certainty, especially if limitations of doing so are acknowledged.
Programs that seem to produce the desired outcomes can be considered effective, at least unless or until a long-term impact evaluation indicates otherwise. Assuming that the program’s theory of change is correct, achieving the program’s intended outcomes should, in turn, lead to the intended impacts. Almost invariably, statistical analyses show that, all things being equal, each year of additional school leads to an 8 percent to 13 percent increase in earnings (Kolesnikova 2010).
Thus, based on theory and past research, we presume that certain skills (such as the ability to read) are essential in modern labor markets, and so, teaching children (as well as illiterate adults) to read is presumed to be a productive outcome—leading to positive long-term impacts. K-12 education rightly uses gains in reading and math ability as performance measures (sometimes controlling for contextual and other value-added factors).
On a more micro level, a 1997 to 2001 International Child Support Africa deworming program in Western Kenyan schools sought to reduce absenteeism from school, based on theory and prior research that reduced absenteeism would lead to positive employment and earnings impacts. Harvard researchers randomly assigned children to be dewormed (or not) and found that the program significantly increased children’s attendance compared with a control group. A long-term impact evaluation later found that children in the program group had higher wages and better health (Glennerster and Tavarasha 2013).
If the logic model’s theory of change is incorrect, however, then achieving the specified outcomes does not indicate an effective program. Thus, achieving the intended changes in an individual or organization is not guaranteed to lead to the desired impact. (Consider the provision of job-related skills, such as how to use a specific and complicated piece of equipment or the skills of a specific profession, such as registered nursing). Greater job skills may not translate into higher earnings if they are the wrong skills for available jobs. It does little good to learn information technology (IT) skills on an obsolete computer that was last used by business firms in the 1980s. And, of course, there may be no job openings, at all, regardless of skill levels.
For example, the theory behind the Drug Abuse and Resistance Education (DARE) program is that instructing middle school and high school students about the dangers of illegal drugs and alcohol abuse would make them afraid to use them and, hence, reduce their substance abuse. Multiple evaluations of the program, however, found that, although the program succeeded in increasing students’ knowledge about the risks of drug and alcohol use, it had no effect on students’ drug or alcohol use (and, in some evaluations, increased use) (Lillenfeld 2007). Many observers think the explanation for this is that, as the students learned about drugs and alcohol, they, mistakenly or not, felt better able to use them safely (Werch and Owen 2002). (This is a well-known phenomenon. Studies of the implementation of seat belt laws found that the increased use of seat belts also increased careless driving behavior as drivers felt safer behind the wheel; Adams 1994).7
Similarly, in 2007 to 2009 MDRC researchers evaluated the effect of the Opportunity NYC-Family Rewards program. The program provided cash transfers to low-income families on the condition that the children in the family met school attendance benchmarks, scored well on tests, and graduated from high school. The researchers found that although the program was effective in increasing school attendance, it did not have an effect on test scores or graduation rates—suggesting that the theory of change was only partially correct: incentives did lead to increased attendance, but increased attendance did not lead to increased learning (Riccio et al. 2010).
The examples abound: A sex education program that teaches students about contraception may increase their knowledge about safe sex and, instead of discouraging sexual activity, may increase it because the students feel (rightly or wrongly) that they know how to avoid pregnancy or sexually transmitted diseases. Actually, this is quite a common problem.
Ultimately (and ideally), therefore, the correctness of the logic model’s theory of change should be validated—through a scientifically rigorous impact evaluation, strongly supported by theory, past research and, best of all, one or more additional impact studies of the same or similar program. In the meantime, though, outcomes are probably the best timely indicator of program effectiveness.