Evaluation as a Business

Is There a Market for Program Evaluation?

The immediate answer to this question is "Definitely yes." Both individuals and organizations that sponsor a program are very interested in knowing whether that program "performed." They are interested in determining whether the program did what it was supposed to do, did it in an efficient manner, and if it made any difference. These are questions important to sponsors regardless of whether they are part of the public sector (that is, government) or the private sector (corporations, foundations, or private individuals). Based on the answers to any or all of these questions, the sponsor can make a decision to continue funding, expand funding, or to market the program. If you need further convincing, the above topics are elaborated on in Chapter Two, Why Evaluate?

Public Sector

In an effort to provide examples supporting this assertion, let's first look at the opportunities that abound in the public sector. The federal government's myriad funded programs have one common denominator: evaluation requirements. Often, depending on the current climate in the government, these evaluation requirements range from being strongly recommended to being strictly required. Again, this pendulum swings regularly, depending on the need for proof from the executive, legislative, and judicial branches of the government. Currendy, there is a culture permeating all funding that requires programs to be scientifically based or data rich, or that decision making be data based. What this means is that funding proposals must have data supporting the assertions made by the proposal author for the need of certain fundable services or activities. It also means that once funded, the recipients will be expected to provide data indicating the performance of program.

For example, the No Child Left Behind Act, which funds the lion's share of educational activities in prekindergarten through secondary education in this country, has specific language that

puts special emphasis on determining what educational programs and practices have been proven effective through rigorous scientific research. Federal funding is targeted to support these programs and teaching methods that work to improve student learning and achievement. (ED Gov., 2003)

A good example of this trend is that the federal government makes funds available to teachers so that they can strengthen current skills or gain new skills in techniques of effective reading instruction. Therefore, all reading programs funded under No Child Left Behind must be supported by scientifically based reading instruction programs for teachers, such as the Reading First program in the early grades, and the new Early Reading First program in preschools. These are tested, evaluated, and validated programs that have shown their effectiveness and transportability.

Specifically stated in the language of all federal funding streams is this requirement of the use or development of data-supported techniques, methods, and activities. For example, the U.S. Department of Education (USDOE) spends upwards of $ 100 million per year on program evaluation and data collection. Key decision makers in that department, in the Office of Management and Budget, and congressional appropriators on Capitol Hill continually request the information they need to make decisions on maintaining a federal initiative. An interesting confession of the USDOE was that

Evaluation studies are not as helpful as they could be to practitioners at the local level, nor can they answer questions about causation. . . . We propose a significant shift in program evaluation, away from a compliance model and towards a system of research and evaluation focused on results and the effectiveness of specific educational interventions. (ED Gov., 2002)

In 2002 the Office of Management and Business introduced a new performance-based data management process, sporting an electronic system of data collection focused on outcomes. A major goal of the system was to collect higher-quality data on program results:

These data will tell us whether the education system and its components are performing well, and they might help us understand which of our programs are having the greatest impact, but they will not tell us why. How can we supplement this data management system with program evaluations that give decision makers the information they need to allocate funds, make policy changes, and consider new directions? How can we build on the knowledge base so that practitioners know "what works" and can spend their federal dollars wisely? (ED Gov., 2002)

Does all of this sound familiar? It should, as this rationale for evaluation echoes much of what was presented in the Introduction of this book. If you are not convinced of the federal government's commitment to evaluation yet, then consider the following. The Electronic Data Gathering, Analysis, and Retrieval system (EDGAR) process— used to report to the federal government the results of all activities receiving federal money—has specific language relating to the requirements for evaluation. In subpart C (sec. 80.40) of the regulations it stipulates that as part of the monitoring and reporting of program performance:

Grantees must monitor grant and subgrant supported activities to assure compliance with applicable Federal requirements and that performance goals are being achieved. Grantee monitoring must cover each program, function or activity. (ED Gov., 2008)

The regulations also require that performance reports be submitted at least annually and must compare actual accomplishments to the objectives established for the period.

Again, these regulations remain in place continuously, but the extent to which they are enforced varies from administration to administration.

Private Sector

Lest you think that evaluation is required only in the world of public dollar-supported programs, let's briefly discuss the evaluation requirements of the private sector. Private sector support for programming might come from such sources as foundations, corporations, or corporate foundations. As stated earlier, evaluation in the private sector might take the form of quality control, employee appraisal, or market analysis activities. However, a great number of programs are evaluated as well. Most foundations, as part of their application requirements, identify program evaluation activities as one of their required activities. In fact, many foundations employ individuals whose sole or main activity is the monitoring of evaluations. A good example of this requisite by a foundation is found in the materials offered to grantees by The Health Foundation of Greater Cincinnati. They indicate that both process and outcome evaluations are important so that funded projects can

• Analyze practices and procedures to uncover what worked and what failed

• Gain insight into effective strategies for solving problems

• Investigate what made collaborations and relationships successful

• Learn what impact the project had on community health

• Collect information that may be useful in dealing with the funding sources

• Provide information for future Foundation grant making decisions, program planning efforts, and new project development

• Assist Foundation staff in monitoring the progress of grant-funded activities (The Health Foundation of Greater Cincinnati, 2010)

A search of the database of The Foundation Center (fdncenter .org), which is the central source of information and the leading authority on philanthropy for thousands of foundations, corporations, public charities, and nonprofit organizations, identified over fifty such organizations that publish informational packets on preparing evaluation designs for proposed projects, and over 3,700 reports or papers on third-party evaluations that were completed through foundation-sponsored activities. Each private sector sponsor will have some form of monitoring or evaluation requirement. Some are quite explicit, like that of The Health Foundation of Greater Cincinnati. Others might be less restrictive, allowing evaluations to be performed by "in-house" personnel. However, they all require some form of evaluation activity— whether process, outcome, or both.

Companies both large and small require evaluations of new and existing programs to determine whether funds should continue to be expended on these activities. Usually the purpose of such evaluations is to assess the value of these programs to the company's bottom line, the extent to which there was value added by these programs, or the return on investment provided by these programs.

We hope readers are convinced that there is a need for third-party or outside evaluations both in the public and private sectors, and that this need will continue over time. In addition, we have identified a number of potential customers for this program evaluation work within public agencies providing educational, human service, health, transportation, correctional, and workforce training services to the general public. Also, a similar need for program evaluation exists in the private sector for programs sponsored by foundations, corporations, public charities, and nonprofit organizations. Now it is time to explore how you find these potential customers.