Policy Evaluation

The reform policy evaluation concerns the assessments of the content, process and particularly the effects of the reform policy. Various criteria can be used and may relate to the different stages of the policy process. Obviously goal achievement is the key focus of attention in the evaluation process (Fischer, 1995; Patton, 1978; Pawson and Tilly, 1997). Assessing goal achievement however is not without problems (Kraft and Furlong, 2007; Dunn, 2004). First, policy objectives are not always clearly stated, making it difficult to assess their achievement. Second, policymakers may anticipate responses to reform policies, for instance, resistance and scepticism, and therefore adapt their ambitions in advance. Also, goals on paper and the real goals can differ. Rhetoric or consciously under- or overstating the goals complicates the policy analyst’s work. Third, structural reforms may have more than one goal, possibly with some degree of conflict among the goals, which complicates judgements on effectiveness. Fourth, various groups of actors may hold different goals, which affect one or more of the policy stages to different extents: Which are then the goals against which the policy should be evaluated?

Also, many other criteria than goal achievement can be part of evaluation (Yeh, 2010; Bryk, 1983; DeGroff and Cargo, 2009; Linder and Peters, 1989; Salamon, 2002; Birkland, 2001; Kraft and Furlong, 2007; Dunn, 2004). We list these criteria in Table 1.

Table 1 Evaluation criteria in public policy analyses

The aims of the evaluation can also vary. On the one hand, evaluations may be carried out to take stock (summative). On the other hand, they may serve to draw lessons from and to improve the reform process (formative) (van der Knaap, 2004). Through feedback (information on content, process and effects), part of the reform policies can be readjusted, ultimately resulting in a fully effective structural reform (as shown already in Fig. 1). Therefore, a careful analysis of how reform processes are evaluated is needed, particularly because the final outcome of such an evaluation is supposed to feed back into the reform process.

Associated with evaluation of goal achievement is the question how important it is for the policy to be successful whether implementation is ‘high fidelity’, or whether there is tolerance for ‘low-fidelity’ implementation (Land and Gordon, 2013). A ‘low-fidelity’ policy would allow for a large degree of local variation in the ways in which actors might wish to approach the policy and finally achieve specific elements of its goals, and yet achieve the originally intended aims of central policymakers. Low-fidelity policies leave room for adaptation to context and require policymakers to trust shop-floor implementers of policy.

Time is a crucial factor in evaluating content, process and effects of a structural reform. A structural reform means that actors have to learn new rules and abandon old ones. Apart from the fact that based on their interests or capabilities some actors may be unwilling to do so, time is needed for such learning processes. Structural reforms have a time lag - only after a while lasting effects are likely to sink in. It may also take some time for emotions to ebb away and to make a ‘fair judgement’ possible. Finally, dissatisfaction during or just after the reform may shift to satisfaction when for instance the effects are more positive or less disruptive then initially thought.