The Quality and Legitimacy of Multi-criteria Analysis in Practice
Opening up the policy formulation process can increase or undermine the legitimacy of a MCA study
Among the advantages of policy formulation with MCA is its opening to different value systems, as mentioned above, which is particularly fostered by a transparent and inclusive participatory process (see Chapter 2, this volume). Although a generic MCA cannot be considered as a participatory tool in itself, stakeholders can participate in some or all stages of the policy formulation process when MCA is used. Some MCA tools specifically designed for opening up the decision making process to participation, such as multi-criteria mapping (Stirling and Mayer 2001), are listed among the current tools for participatory policy assessment (see Chapter 2).
In practice, the extent to which stakeholder inclusion in MCA takes place largely depends on the people steering the process and brings with it challenges inherent to any participatory assessment exercise. In reported case studies (see Table 6.1) its inclusion has sometimes involved fairly narrow approaches, such as when only experts and/or authorities are included (Bana e Costa and Oliveira 2002; van Gennip et al. 1997; Petras 1997; Brouwer and van Ek 2004). Broader participation entails sharing the involvement and responsibilities in the policy formulation process more widely, as shown by Borges and Villavicencio (2004) or Marttunen and Hamalainen (1995). In these cases, experts built the scenarios, as well as the criteria, while the evaluation and the weighting process were performed by wider stakeholder groups. Or, as in the case described by Bombaerts et al. (2007), the options can be identified in a participatory manner, such as was done for low-level radioactive waste disposal options elaborated in a dialogue between a radioactive waste agency, local communities and local individuals. Examples where stakeholders participated in all phases of the policy formulation analysis, including the definition of alternatives and criteria, are provided for example by Joubert et al. (1997) and Messner et al. (2006).
Opening up the appraisal processes to a wider stakeholder group has other clear benefits, not only in making the outcome of a policy formulation process with MCA more legitimate, but also in terms of clarifying the problem, both between the public and experts, as well as between experts of different fields (Kontic et al. 2006). In a larger participative context, including the general public, Rennet al. (1993) propose a three step procedure relying on MCA, but making a division of decision making tasks between three levels of society: evaluation criteria are to be constructed by involvement of all relevant stakeholders; identification and impact assessment for the decision options are mainly carried out by experts; and weighting should be done by citizens' panels.
Early involvement of stakeholders (Banville et al. 1998) can give a more pragmatic dimension to MCA and contribute to an increased acceptance of the final result. Stakeholder processes are, however, costly and time consuming and, in terms of legitimacy, may on some occasions not contribute in the manner suggested by advocates (see Chapter 2, this volume). In some cases they may even lead to a stalling of the decision process. Therefore, difficult questions remain over which stakeholders should get involved, at what time and through which processes.
In addition, politics may sometimes constrain wide stakeholder involvement. Political actors may not wish to openly express their priorities, or may have their own hidden agendas. Bana e Costa et al. (2001) describe a case where direct participation was replaced - at the request of one of the actors - by an analyst simulating the viewpoints of all relevant stakeholders. Similarly, Brouwer and van Ek (2004) report on a stakeholder analysis, where experts judged the effect different strategies might have on the elicited stakeholder groups, but the MCA was then performed without them.
Another challenge relates to knowledge and information sharing, which means on the one hand making technical information understandable to all stakeholders, and on the other making technical specialists aware of the social and political dimensions of the problem they face (Bardos et al. 2002, p. 19). This brings with it a necessary reduction of complexity, but carries the risk that participants consider that the evaluation criteria employed in the final analysis oversimplify the underlying problem (Marttunen and Hamalainen 1995).
MCA is sometimes regarded as challenging the expert's prerogatives since it may be interpreted as making specialist knowledge subject to non-expert evaluation. In a study on the use of decision aid tools including MCA in environmental management, Joliveau et al. (2000) show that experts may oppose such tools due to several factors, including inter alia hesitation in changing the usual procedures, fear that the model will collide with their recommendations or reluctance to share their power of decision. By contrast, Belton and Stewart (2002, p. 160) emphasize that an important role of MCA is to provide a 'sounding board' against which experts and decision makers can test their intuitions. They illustrate a good number of MCAs for which analysis and intuition were 'successfully reconciled' (Belton and Stewart 2002, p. 288), for example due to creation of new decision alternatives, or to the reconsideration of preferences.