Application of the Theory of Change approach to evaluating normative environmental projects and programmes

This section gives a brief overview of the use ofTheory of Change/causal pathway analysis in the evaluation of different types of environmental interventions at UNEP. Examples are chosen to highlight trends and lessons.

The ‘Synergies Decisions’ of the Stockholm, Basel and Rotterdam conventions

The Decisions (BC.Ex-1/1, RC.Ex-1/1 and SC.Ex-1/1; the... ‘omnibus decisions’), were adopted by the Conferences of the Parties to the Basel, Rotterdam and Stockholm Conventions, respectively, at their simultaneous extraordinary meetings in February 2010. Parties decided that they would review the arrangements adopted pursuant to the ‘Synergies Decisions’,⁴ in particular actions taken by the Secretariats of the Convention to merge and conduct joint activities, joint managerial functions and joint services in order to determine how far they had contributed to achieving the following objectives:

• 1. Strengthening of the implementation of the three Conventions at the national, regional and global levels;
• 2. Promoting coherent policy guidance;
• 3. Reducing administrative burdens;
• 4. Maximizing the effective and efficient use of resources at all levels; and
• 5. Protecting human health and the environment for the promotion of sustainable development.

The decisions further called upon the Executive Director of UNEP in consultations with the Director General of Food and Agriculture Organization of the United Nations to undertake an independent evaluation and to prepare reports of the evaluation for consideration by the Conferences of the Parties. The evaluation was to focus primarily on the extent to which the synergies process among the chemicals and waste-related conventions has contributed to improving cooperation and coordination at the global, regional and national levels. A Theory of Change (Figure 9.3) was developed prior to initiating the main data-gathering phase of the evaluation from a desk review of convention documents that outlined the aims of the ‘Synergies Decisions’. The Theory of Change sets out the causal intent of the synergies decisions and provides a conceptual framework for the evaluation’s implementation phase, guiding the data collection necessary for assessing performance indicators. The Theory of Change framework was used to structure interview questions with key stakeholders (e.g. Secretariat Staff and Parties to the Convention), which aided in identifying assumptions and threats that were key to assessing the likelihood of contributions being made to higher level objectives.

Assumptions were identified as the following:

• 1. Leadership is actively committed to the synergies process: The synergies process involves merging and restructuring the Convention Secretariats into one cohesive team serving all three Conventions, particularly on substan- tive/technical activities. This requires leadership commitment to the process at the highest level (Executive Secretary) and mid-tier management buy-in to the process of change management and joint-delivery of sendees to Parties;
• 2. Accountability and incentive structures are in place to support synergies: Responsibilities and roles are structurally and individually aligned to carry out synergies process and to support teamwork within the restructured secretariat (e.g. job descriptions, operating procedures, functions and processes for teams and clear reporting lines). Appropriate incentives (rewards and sanctions) are in place to underpin delivery of services internally and to Parties;
• 3. Synergies restructuring is accepted and supported by all staff: The synergies process and actions need to be accepted and supported by all staff and at all levels of the secretariat organization to be successful;

FIGURE 9.3 ‘Synergies Decisions’Theory of Change (Shown in Part)

• 4. Synergies restructuring is cost-neutral and will lead to improved efficiency in implementing the Conventions: The restructuring and merging of the Secretariats (while preserving the legal autonomy of each Convention) and delivery of joint services and substantive activities to Parties will lead to improved efficiency; and
• 5. Parties are supportive of synergies and carry out the necessary institutional and organizational changes at the national level to promote cooperation and coordination: For the strengthened implementation of the three Conventions and progress towards protecting human health and the environment, it is assumed that Parties will support the synergies process and pursuant actions and take actions at the national level (supported by the Secretariat/UN agencies and other relevant bodies).This will include, inter alia, improved systemic and institutional capacities to improve enforcement of the three Conventions, inter-ministerial coordination (including cooperation between focal points)³ and bringing responsibilities for implementation under one ministry (e.g. one focal point for all chemicals Conventions).

Threats to the achievement of outcomes were identified as the following:

• 1. Cost of synergistic actions rise, leading to decreases in party interest and support: If Assumption 4 (cost neutrality and improved efficiencies) does not hold, then Parties may decide to abandon or limit the synergies process;
• 2. Potential conflicts between managers, leading to poor delivery of services: Managers may have conflicts over human and financial resources if managerial controls, responsibilities and roles lack clarity within the new matrix structure;⁶
• 3. No increases in funding for implementation of activities to support the strengthening of the Conventions: A lack of financial support is likely to be a significant threat to the realization of synergies in terms of delivery of substantive activities by the secretariat and implementation of synergies at the national and regional levels; and
• 4. National cooperation and coordination fails: Parties are unable to forge operational synergies at the national level due to lack of enabling policy, capacity or mandate disagreements between ministries (e.g. lack of communication and institutional structures that can facilitate regularized coordination). Implementation of synergies actions fails.

In this case, Goal (e) of‘protecting human health and the environment for the promotion of sustainable development’ is intended to result from‘strengthening the implementation of the three conventions at national, regional and global level’, that is, changes in third-party behaviours within countries acting in compliance with national regulatory requirements that stem from Convention provisions. Although reducing threats to human health through the convention mechanisms features in the overall goal, it would be unrealistic to attempt to measure the downstream effects of changed management arrangements in Convention Secretariats at that level.The locus for measuring performance in the normative work, in this situation, is not feasible at the level of changes in environmental status/condition, because the causal linkages are indirect and weak and so any observed effects are difficult to attribute to the upstream change in question. This implies that evaluation of the effects of the synergies decisions are best captured at the outcome level and viewed as contributions to higher-level intended objectives.

The Global Environment Facility joint geothermal imaging project

The evaluation of the Joint Geothermal Imaging project and a follow-up study undertaken by the Global Environment Facility (GEF) Evaluation Office made use ofTheory of Change and provided another example of normative research work - the development of ways to combine data from different imaging methods for site identification of geothermal wells. The potential to establish linkages between site identification and environmental benefits in terms of C0₂ emissions avoided was identified during the Terminal Evaluation of the project, and UNEP developed the Theory of Change as a contribution to the follow-up study undertaken by the GEF Evaluation Office. The Theory of Change shows the pathways from imaging method development, scientific capacity-building and training events (often considered as normative activities) through to the resulting benefits expressed as C0₂ equivalents avoided from the use of geothermal electricity generation (rather than the ‘business as usual’ scenario, which would have resulted in fossil fuel CO, emissions) (see Figure 9.4).

The points to note in this example are that many development projects contain normative aspects, and it is not unusual for the causal pathways from project activities to the desired development objectives to be somewhat indirect. In this case, tracing the actual results of the project along the causal pathways leading towards the intended objectives highlighted the fact that the application of the imaging technology' could be linked to quantifiable improvements in energy-generating efficiency of geothermal wells, which can be expressed in terms of CO, emissions avoided. Routine application ofTheory of Change approaches can aid in identifying projects that have potential for more detailed impact evaluations, because causal pathways that may have high ‘attributive ease’ can be identified.