Decision Making: Whom to Involve, How, and Why?
Program and Project Cycles
Every program, no matter what its origin, has a life cycle through which evaluation activities flow. Everyone involved in the program, from those who fund it, to those who lead it, to those who receive its services or products, has a stake in the decision making. These are the people, the stakeholders, who need to be a part of the evaluation process. In a perfect world, the evaluator—whether internal or external—would be involved from the outset and would contribute to every phase of the life cycle, making sure that all the appropriate people were involved and receiving pertinent information along the way.
In the real world, unfortunately, sometimes one or more groups are left out. Sometimes project staff members are left out because the evaluation is viewed as an activity being performed for someone else—usually the sponsor—so there is no need to bother staff with more duties, as they will not be recipients of the report. Usually the recipients (clients) are left out because "they don't really understand the program. After all, they are the clients."
Sometimes the evaluator is left out until the last minute because the project gets funded, the activities begin, and only then does the project director remember that an evaluation component was written into the proposal. Thus the evaluator begins work after the project activities have begun and in some cases when they are almost completed. Even an evaluator who is internal (one of the program or agency staff) is not necessarily included in, or attuned to, all phases of program planning.
To understand how, when, and why stakeholders are involved in an evaluation, you need to understand what evaluators call the program's life cycle. In one sense, the cycle of a program is unique to each and every program. In the public sector—education, human services, and community-based organizations—the program cycle usually refers to the funding year, which is generally the fiscal year. Normally a program is funded for one fiscal year, so the evaluation looks at the program's progress over that period. Where funding is granted for two or three years, the evaluation may be able to examine long-term effects and the impact that the program might have had on the clients.
In the private sector—business and industry—program cycles may vary from several weeks to several years. The funding of the program may or may not come into play in defining the cycle. A decision maker may need the evaluation results of a two-week training program that introduces a new computer-based client data system immediately after each two-week session is completed, not after all employees have been trained. The information allows the manager to determine the extent to which the training activities are effective and the number of employees who need to repeat the training.
Because all programs and projects differ in purpose, size, time frame, and other factors, the evaluator must clarify the sponsor's mandate (Wholey, Hatry, and Newcomer, 1994).
The traditional program cycle starts with an organization's mission and goals, moves to needs analysis, through program planning (with the evaluation designed into the program plan), and to program implementation, where the formative and summative evaluations take place. (See Figure 3.1.) During each of these phases, the evaluator can perform the functions that lead to evaluation findings and ultimately to the evaluation report. At each phase, evaluation data can assist decision makers, bringing us right back to our main topic: Whom do you involve?
If the evaluator looks at the organization's mission and goals, decisions regarding the appropriateness of programs or program activities can be examined before the program begins. People do this every day when they are asked to make some hard decisions: Should the company continue to conduct certain activities? Are these activities truly necessary? Can the company afford them? This scrutiny,
Figure 3.1 Program Cycle.
however, is usually viewed as a fiscal process rather than a program evaluation process.
If the evaluator looks at the needs analysis, she can determine the actual need for or marketability of the program to the clients, internal or external, before any resources are committed. Even though the "gut feeling" of some decision maker or even the "literature" seems to dictate that this is the program to pursue, the data collected from clients may convince you otherwise.
If the evaluator looks at the program planning processes, he can determine the probability that the proposed activities will indeed result in the needs being met. Furthermore, an efficiency evaluation at this juncture might help in making the decision to develop new activities rather than adopt or adapt someone else's similar activities. If for no other reason, involving an evaluator at this point will give the program staff help in formulating evaluation questions that will be meaningful while they are conducting or improving the program. Also, the evaluation design can be created now and integrated into the overall program plan.
Usually the evaluator is brought into the process during the program implementation phase. The plan has been designed, ideally in response to real needs and in concert with the organization's mission, and now the evaluator is expected to collect data and, when the cycle is complete, make a report.
Later in this chapter, Table 3.1 outlines the information needs, decisions to be made, and contributions of an evaluator at the early stages of needs assessment and program planning and in the implementation stage when formative or summative evaluations may be conducted.