Policy Evaluation

The reform policy evaluation concerns the assessments of the content, process and particularly the effects of the reform policy. Various criteria can be used and may relate to the different stages of the policy process. Obviously goal achievement is the key focus of attention in the evaluation process (Fischer, 1995; Patton, 1978; Pawson and Tilly, 1997). Assessing goal achievement however is not without problems (Kraft and Furlong, 2007; Dunn, 2004). First, policy objectives are not always clearly stated, making it difficult to assess their achievement. Second, policymakers may anticipate responses to reform policies, for instance, resistance and scepticism, and therefore adapt their ambitions in advance. Also, goals on paper and the real goals can differ. Rhetoric or consciously under- or overstating the goals complicates the policy analyst’s work. Third, structural reforms may have more than one goal, possibly with some degree of conflict among the goals, which complicates judgements on effectiveness. Fourth, various groups of actors may hold different goals, which affect one or more of the policy stages to different extents: Which are then the goals against which the policy should be evaluated?

Also, many other criteria than goal achievement can be part of evaluation (Yeh, 2010; Bryk, 1983; DeGroff and Cargo, 2009; Linder and Peters, 1989; Salamon, 2002; Birkland, 2001; Kraft and Furlong, 2007; Dunn, 2004). We list these criteria in Table 1.

Table 1 Evaluation criteria in public policy analyses

Goal achievement

Certainty (administrative

Equity (fairness or justice in


capacity and agent compliance)

the distribution of benefits, costs and risks among actors)

Administrative intensiveness (administrative costs, operational simplicity, flexibility)

Timeliness (extent to which instruments work quickly)

Social and political acceptability and support

Political risk (nature of support and resistance, public visibility, chances of failure)

Costs (of developing, implementing and monitoring)

Technical feasibility (availability and reliability of technology needed)

Constraints on state activity (difficulty with coerciveness and ideological principles)

Efficiency (outputs related to inputs; goal achievement in relation to costs)

Targeting (precision and selectivity among actors)

Choice and agent autonomy (degree of choice and restrictions offered by the policy)

Accountability (extent to which implementers account for their actions)

Responsiveness (extent to which outcomes satisfy needs and preferences of particular groups)

The aims of the evaluation can also vary. On the one hand, evaluations may be carried out to take stock (summative). On the other hand, they may serve to draw lessons from and to improve the reform process (formative) (van der Knaap, 2004). Through feedback (information on content, process and effects), part of the reform policies can be readjusted, ultimately resulting in a fully effective structural reform (as shown already in Fig. 1). Therefore, a careful analysis of how reform processes are evaluated is needed, particularly because the final outcome of such an evaluation is supposed to feed back into the reform process.

Associated with evaluation of goal achievement is the question how important it is for the policy to be successful whether implementation is ‘high fidelity’, or whether there is tolerance for ‘low-fidelity’ implementation (Land and Gordon, 2013). A ‘low-fidelity’ policy would allow for a large degree of local variation in the ways in which actors might wish to approach the policy and finally achieve specific elements of its goals, and yet achieve the originally intended aims of central policymakers. Low-fidelity policies leave room for adaptation to context and require policymakers to trust shop-floor implementers of policy.

Time is a crucial factor in evaluating content, process and effects of a structural reform. A structural reform means that actors have to learn new rules and abandon old ones. Apart from the fact that based on their interests or capabilities some actors may be unwilling to do so, time is needed for such learning processes. Structural reforms have a time lag - only after a while lasting effects are likely to sink in. It may also take some time for emotions to ebb away and to make a ‘fair judgement’ possible. Finally, dissatisfaction during or just after the reform may shift to satisfaction when for instance the effects are more positive or less disruptive then initially thought.

< Prev   CONTENTS   Source   Next >