Definitions of Evaluation
People do not always agree on one definition of evaluation. Following are two different definitions:
• Evaluation is the systematic process of collecting and analyzing data in order to determine whether and to what degree objectives have been or are being achieved.
• Evaluation is the systematic process of collecting and analyzing data in order to make a decision.
Notice that the first ten words in each definition are the same. However, the reasons—the why—for collecting and analyzing the data reflect a notable difference in the philosophies behind each definition. The first reflects a philosophy that as an evaluator, you are interested in knowing only if something worked, that is, whether it was effective in doing what it was supposed to do. The second statement reflects the philosophy that evaluation makes claims on the value of something in relation to the overall operation of a program, project, or event. Indeed, many experts agree that an evaluation should not only assess program results but also identify ways to improve the program evaluated (Wholey, Hatry, and Newcomer, 1994). A program may be effective but of limited value to the client or sponsor. One can imagine, however, using an evaluation to make a decision (the second definition) even if a program has reached its objectives (the first definition). Federal grants are based on the first statement, that is, whether the program has achieved its objectives, but the harder decision to downsize or change may be a consequence of the second definition of evaluation.
Evaluating Efficiency, Effectiveness, and Impact
We can define evaluation even more precisely as a process that is guided by the reason for doing the evaluation in the first place. An evaluation might be a process of examining a training program in light of values or standards for the purpose of making certain decisions about the efficiency, effectiveness, or impact of the program. To carry out this task, you need to understand the concepts of efficiency, effectiveness, and impact. These three terms will be referred to from this point on as the levels of program evaluation. (See Table 1.1 later in this chapter.)
Efficiency relates to an analysis of the costs—dollars, people, time, facilities, materials, and so forth—that are expended as part of a program in comparison to either their benefits or effectiveness. How is efficiency, or the competence with which a program is carried out, measured in a program? The term itself gives clues to what this is about.
Program monitors look at the efficiency with which details are carried out in a program. Programs often begin with recruiting, gathering materials, providing for space, setting up fiscal procedures, and so forth. Thus the relationship between the costs and end products becomes the focus of an efficiency evaluation. Although very important, these aspects of efficiency may have no bearing on the program's effectiveness. If the investment in the program or project exceeds the returns, there may be little or no efficiency.
For example, let's consider a nuclear power facility that houses a rather substantial training and staff development enterprise. As part of this enterprise, ten instructors are responsible for ensuring that five hundred employees are cycled through training every six months, for a minimum of twenty hours of training each cycle. The training revisits the employees' basic knowledge of their jobs and introduces new concepts developed since the last training. The staff development enterprise might work very efficiently by making sure that all employees cycle through in a timely fashion, in small enough groups to utilize the best of what we know about how adults learn. The students' time on task is often not enough, however, and many of them do not retain much of what was covered in the training. Thus the program is not effective.
The enterprise may be efficient in that it fully utilizes the time of each of the available trainers, it stays within the parameters of the staff development budget, it keeps employee down time to a minimum, it uses materials and equipment that are available, and it completes the training agenda for the company. Yet there may be an increase in accidents or hazardous incidents because employees are making simple, basic mistakes. The enterprise's training has been efficient but not necessarily effective.
When you look at the effectiveness of your program, you are asking this question: "Did the activities do what they were supposed to do?" Simply put, a program's effectiveness is measured in terms of substantive changes in knowledge, attitudes, or skills on the part of a program's clients. Although the right number of participants may have been recruited and the best possible site may have been secured, the effectiveness test is this: Did the activities provide the skills to run the new equipment? Did the participants gain the knowledge they need to sell the new mortgage or other banking product?
In another example, the same nuclear power plant's staff development program may conduct a training session on a new procedure to decontaminate after entering a containment area. The trainer may pretest all the employees as they begin their training session. Upon completion, the employees are posttested and the results compared to determine whether their knowledge increased, decreased, or stayed the same. An increase in their knowledge would be an indication that the training was effective—it did what it was supposed to do. Yet two weeks after the training, when one of the employees was back at her job post, a situation arose in which she exited a hazardous area after spending some time checking water flow. She used the older, more comfortable procedure for decontamination and caused a problem that put her and her coworkers at risk. Here is an example of training that was effective—the worker passed all the posttests—but had little impact on changing the behavior of the employee.
Thus the impact that the program has had on the people or organization for which it was planned becomes an important evaluation consideration. Impact evaluation examines whether and to what extent long-term and sustained changes occur in a target population. Has the program or project brought about these desired changes? Are employees using the new procedures? And in other scenarios: Are more people off welfare? Has the program changed a family's life? Do your employees have more job satisfaction?
Evaluators frequently pay too little attention to assessing impact. One reason is that impacts often manifest themselves over time, and program managers have already turned their attention elsewhere before computing this aspect of the evaluation. The actual impact that training in new procedures might have in people's everyday life often takes time to percolate and evolve. An attempt to collect impact data after allowing for this delay may run into a number of blocks, such as learner turnover (you cannot find them), job or circumstance change (they no longer need to use the skills), or lack of time or resources for the evaluator to conduct these follow-up activities.
Still, program and project sponsors are most interested in impacts. Whether a learner feels satisfied with the training, or the training results in knowledge gain means little to a sponsor or employer if the learning doesn't help the organization.