EVALUATION RESEARCH AND IMPACT STUDIES
The evaluation of enacted policy is as old as policy itself. Policy-makers always have made judgments regarding the benefits, costs, or effects of particular policies, programs, and projects (Royse and Thyer, 2016). Many of these judgments have been impressionistic, often influenced by ideological, partisan self-interest, or subjective criteria. For example, a tax cut may be considered desirable because it enhances the electoral chances of the evaluator’s political party, or unemployment compensation may be deemed “bad” because the evaluator “knows a lot of people” who improperly receive benefits. Undoubtedly, much conflict may result from this sort of evaluation because different evaluators, employing different value criteria, reach different conclusions concerning the merits of the same policy.
Another type of evaluation has centered on the operation of specific policies or programs, such as a juvenile correctional reform, boot camp, various police programs, or specific crime prevention programs. Questions asked may include: Is the program honestly run? What are its financial costs? Who receives benefits (payments or services) and in what amounts? Is there any overlap or duplication with other programs? What is the level of community reintegration of participants? What is the degree of staff commitment? Were legal standards and procedures followed? This kind of evaluation may provide information about the honesty or efficiency in the conduct of a program, but like the impressionistic kind of evaluation, it will probably yield little if anything in the way of hard information on the societal effects of a program.
Since the late 1960s, a third type of policy evaluation has received increased attention from policy-makers. It is the systematic objective evaluation of programs to measure their societal impact and the extent to which they are achieving stated objectives. In 1967 and 1968, Congress altered some of the central pieces of President Johnson’s Great Society legislation so that mandatory evaluation would be included in all programs, such as the Economic Opportunity Act of 1964. The aim was to monitor the progress of programs and to terminate those that did not seem to yield the desired level of results. There were also political benefits to be obtained by emphasizing evaluation. Low-cost experiments on social problems and rigid evaluation requirements could be used to subvert attempts to solve social problems through (expensive) direct social change or action programs.
For many sociologists, evaluation research quickly became a proper use of sociology in policy-related work (Babbie, 2017). An entire field of specialization has developed about methods and procedures for conducting evaluation research. Technically speaking, however, there are no formal methodological differences between evaluation and nonevaluation research. They have in common the same techniques and the same basic steps that must be followed in the research process. The difference lies in the following:
- • Evaluation research uses deliberate planned intervention of some independent variable
- • The programs assessed by evaluation research assume that some objective or goal is desirable
- • Evaluation research attempts to determine the extent to which this desired goal has been reached.
As Edward A. Suchman (1967:15) once put it, “evaluative research asks about the kind of change the program views as desirable, the means by which this change is to be brought about, and the signs according to which such change can be recognized" Thus, the greatest distinction between evaluation and nonevaluation research is one of objectives.
Carol Weiss (1998:6-8) proposed several additional criteria that distinguish evaluation research from other types of research:
- 1. Evaluation research is generally conducted for a client who intends to use the research as a basis for decision making.
- 2. The investigator deals with his or her clients questions as to whether the client's program is accomplishing what the client wishes it to accomplish.
- 3. The objective of evaluation research is to ascertain whether the program goals are being reached.
- 4. The investigator works in a situation where priority goes to the program as opposed to the evaluation.
- 5. There is always a possibility of conflicts between the researcher and the program staff because of the divergences of loyalties and objectives.
- 6. In evaluation research, there is an emphasis on results that are useful for policy decisions.
Social policy evaluation is essentially concerned with attempts to determine the impact of policy on real-life conditions. As a minimum, policy evaluation requires a specification of policy objectives (what we want to accomplish with a given policy), the means of realizing it (programs), and what has been accomplished toward the attainment of the objectives (impacts or outcomes). In measuring objectives, there is a need to determine that not only some change in real life conditions has occurred, such as a reduction in the unemployment rate, but also that it was due to policy actions and not to other factors, such as private economic decisions.