The evaluation literature has provided plenty of useful lessons concerning the broader impacts of performance measurement - of which indicators constitute an essential element. For example, lessons from the literature concerning the impact of public sector performance measurement on decision making are far from conclusive (Hezri 2006, pp. 156-157). Norman (2002) characterizes the debate as a battle between three groups: the 'true believers' who highlight benefits such as new investment in data capture, harmonization of measurement methods across institutions, and behavioural changes; the 'critics and doubters' who stress problems in the use, interpretation and societal relevance, lack of political will, bureaucratic inertia, and use of indicators for propaganda purposes; and the 'pragmatic sceptics' (for example, van der Knaap 2006) who see active contestation as a sign of an evolution towards better theory and practice. As a counterpoint to the promise that indicators would provide greater accountability, efficiency and citizen control over policymakers, there is a considerable body of literature highlighting numerous negative features of performance measurement. These can be summarized as follows (for example, Perrin 1998; 2002; Blalock 1999; Davies 1999; van der Knaap 2006; Hood 2007; Jackson 2011; Le Gales 2011):

• complexity and opacity, which reduce potential for dialogue and deliberation;

• disincentives to responsibility, innovation, creativity and achievement;

• goal-shifting and 'gaming';

• dissimulation and distortion of data or even lying and cheating;

• reductionism and the suppression of the plurality of values and points of view;

• a management rhetoric inappropriate in sectors with a 'non-managerial' tradition;

• legitimization and reinforcement of prevailing power structures;

• 'misuse' and misunderstanding resulting from ignorance of the sources, definitions and methods underlying the indicators;

• potential systemic effects: loss of public trust, risk of a system collapse (Hood 2007, p. 102).

The problems of performance measurement can be seen as a subset of the more generic paradoxes and dilemmas involved in indicator work. Hence, it is precisely the widespread use and institutionalization of performance indicators - policy 'success' - that accentuates their risks and downsides.

The absence of a linear connection between use and influence represents an example of the many paradoxes, dilemmas and trade-offs involved in indicator work. These include tensions between:

• deductive and inductive approaches (whether indicators should serve to test theory and hypotheses, or whether the inquiry should progress from data gathering towards theory-building);

• use of indicators as inputs for the design and implementation of public policies versus as tools for monitoring and evaluation;

• international comparability and national/regional/local relevance;

• description and prescription;

• objectivity and normativity; and

• academic and practitioner emphasis, in other words, whether the quality of an indicator should be defined by the scientific quality or practical usefulness of the indicator (Cobb and Rixford 1998, pp. 3^4; Rosenstrom and Lyytimaki 2006).

Four further tensions merit particular attention:

• The 'paradox of conservatism'.2 The factors enhancing instrumental use - institutionalization, consensus on data, policy and conceptual frameworks - are often in conflict with the challenge function of indicators, in other words, their capacity to destabilize prevailing practices, frameworks of thought, and 'hegemonic discourses' (Driscoll Derickson 2009, p. 904). For instance, the paradigmatic consensus underpinning the GDP as a proxy measure for wellbeing has guaranteed its resistance against pervasive criticism (Morse and Bell 2011).

• Matching supply with demand. The objective of better matching supply with demand emphasizes the instrumental role of indicators and single-loop learning, while the more complex types of learning entail shaping demand rather than merely responding to the existing demand.

• Process versus product. Indicator research and practice tends to overwhelmingly concentrate on the quality of the indicator as the 'final product', despite the growing evidence of the importance of indicator production processes as a crucial source of especially conceptual influence (for example, Mickwitz and Melanen 2009; Bell et al. 2011; Lehtonen 2013).

• Aggregation, quantification, scientific rigour and policy relevance. Aggregate and composite indicators can be powerful tools for communication, comparison and peer pressure (for example, Pagani 2003), yet aggregation can feed reductionism, over-simplification and disregard for contextual differences. For instance, in the area of Social Impact Assessment, strong disagreements prevail between the defenders (for example, European Commission 2009) and critics (for example, Esteves et al. 2012, p. 40) of quantitative indicators.

< Prev   CONTENTS   Next >