Monitoring, Evaluating, and Managing Adaptively

Evaluation of collaborative governance is fraught with challenges. One such challenge is the often blurry distinction between process performance and outcome performance (Emerson and Nabatchi 184). We separate monitoring and evaluation of collaborative governance into three separate activities: (1) monitoring of each party’s individual performance with regard to their commitments and agreements; (2) evaluation of group performance in addressing the public problem or issue; and (3) monitoring and evaluation of the collaborative process itself, including relationship impacts.

Monitoring Individual Performance

Accountability in a collaborative governance process, as noted in chapter 4, is inherently horizontal; the parties are accountable to each other for following through on their agreements and commitments. For this to work, Susskind and Cruikshank point out, performance monitoring mechanisms need to be in place (150). Did each of the parties follow through with their commitments and the spirit, if not the letter, of the collaborative agreement?

Even when individual performance falls short of commitments, accountability mechanisms in these horizontal relationships are more indirect and informal. It is the threats of social embarrassment and reciprocity (others not cooperating in turn) that are the primary consequences of noncompliance. In an environment of horizontal relationships, says Ostrom, reciprocity is the glue that holds these collaborative agreements together. If one fails to honor commitments to others, they may do the same in the future (10). Martin Nowak argues that reputation (what he calls “indirect reciprocity”) can have a similar effect. If you have a reputation as someone who cooperates for the good of the whole, Nowak writes, it brings you a certain power to negotiate future agreements, even with people you have not dealt with before. Word gets around, Nowak writes, and it is your reputation that counts, whether good or bad. If people see you as someone who does not follow through with your cooperative commitments, people will be less willing to engage in cooperative arrangements with you. Your social credit rating, in effect, will suffer (60).

The importance of reputation that Nowak describes is one reason we now regularly include a date for reconvening in the written agreements for many of the collaborative processes we facilitate. We commonly observe a striking increase in stakeholder activity in the weeks and months leading up to the reconvening date, suggesting that stakeholders are paying attention to that date and wanting to uphold their reputations for following through.

Evaluating Croup Performance

We stated in chapter 1 that the intention of collaborative governance is ultimately to advance some public purpose. Therefore, some evaluation mechanism is needed to determine if the collaboration succeeded and, if it did not, what course corrections may be in order.

Collaborative groups should be thinking about accountability and evaluation from the very early stages of their work together. Evaluation of group performance is aided by a clear framing of the goal and purpose of the collaboration itself (see chapter 6). Emerson and Nabatchi emphasize this point by recommending that collaborative groups have explicit discussions up front about what will constitute success (222).

At the end of the process, the agreement should reflect who will monitor and evaluate progress, whether it is a participant in the original collaborative governance group or an agreed-upon third party. We recommend that, at least at a general level, the method of performance evaluation be included in the group’s initial agreement, describing what will be monitored, who will monitor, and how it will be reported to the group (and, potentially, the public). A number of collaborative groups we have worked with have created websites that provide transparency on current activities, monitoring, and progress.

Reconvening the group, we find, is also a way to help the group evaluate progress. The value of this evaluation, however, is directly linked to its use in adaptive management. Circumstances frequently change, beyond the control of those signing a collaborative agreement. A local government may have a change in political leadership and priorities, or new grant funding may become available. The agreed-upon strategy to address the public problem may simply not work, or it may work better than expected. Such changes are to be expected, say Carpenter and Kennedy, and parties should not be alarmed when adjustments are needed. “Something will go wrong” (149). Monitoring progress toward the public policy goal can help the group evaluate and adapt its approach.

Emerson and Nabatchi recommend that collaborative groups evaluate their work from a variety of perspectives, including those of the participants, the groups represented by the participants, and the intended beneficiaries of the public policy or program (84). One of the most effective monitoring and adaptive management programs we have seen, one that incorporates the multiple perspectives recommended by Emerson and Nabatchi, is that used by the Lower Columbia Solutions Group project, the collaborative system that we discussed in chapter 7. This group established an elaborate monitoring program as part of its initial agreement, with roles assigned to specific parties. Part of that program was a commitment to hold periodic adaptive management meetings. The group now conducts an annual workshop open to all stakeholders to review the year’s activities, evaluate the results, and plan for needed changes or adaptations for the next year. Their annual workshop keeps everyone accountable and focused on their goals. It is one of the reasons, we believe, for the group’s successful performance.

Monitoring of the Process

In addition to performance monitoring, Emerson and Nabatchi emphasize the need for evaluating the process as well (185). One of the challenges of evaluating collaborative processes is that their context is not uniform. When no two collaborative groups are alike, how does one determine which has been more effective? How does one compare a group process involving fifty stakeholders, for example, with a group of nine stakeholders? How does one compare a process addressing water supply issues in the western United States with one addressing child health in Malaysia?

Some of the challenges related to evaluating and comparing the effectiveness of collaborative processes derive from the frequent attempt to measure collective action and agreement-seeking in the same manner, when, in fact, they are different types of processes, solving fundamentally different types of problems. Those differences should be kept in mind as a group evaluates the effectiveness of its collaboration.

Our work over the last two decades with several hundred collaborative groups tells us that a rigorous approach to the definitional norms, fundamental dynamics, and process design leads to better outcomes in collaborative governance. For that reason, our process monitoring and evaluation has focused on those elements. Common questions, therefore, include the following:

  • • Was the process seen as fair and inclusive?
  • • Did the parties work toward mutual gain?
  • • Was there a strong sense of interdependence among the parties?
  • • How high was the trust level in the collaborative relationships?
  • • How high was the trust level in the process?
  • • Was the framing of the group goals clear and agreed upon?
  • • Was there shared ownership of the problem and process?
  • • Did the convener promote a fair environment for discussion?
  • • Did the parties feel like their voices were heard and appropriately considered by others?
  • • Was there broad participation in the discussion among those at the table?
  • • Was there a willingness to deal with conflicts as they emerge?
  • • Were communities of color and other historically underrepresented communities meaningfully involved in the process?
  • • Was the process equitable and inclusive?
  • • Was there attention to results and accountability?

We have used several methods for evaluating our collaborative processes, some more elaborate than others. For agreement-seeking processes, we used to administer a pages-long, comprehensive survey of all participants, but the lengthy survey was off-putting for some participants. We have found we get a better response with a more concise format that is administered soon after the collaborative process is concluded. We have also conducted process evaluations, both formally and informally, during a process in order to determine what midcourse corrections may be needed. For example, we ask: Is the process transparent? Is it fair and equitable? Is the decision-making seen as effective? Is the convener or facilitator performing to expectations? We then adapt the process in response to the feedback.

In other instances, we have enlisted research teams to perform more detailed process evaluations on a few projects, including the Oregon Sage-Grouse Conservation Partnership. These types of evaluations have generally been conducted as academic research, usually months or years after the agreement is reached, and include detailed interviews of participants.

Conclusion

The unifying message of these last four chapters might be: process matters. An assessment will provide indicators as to whether a collaborative governance approach can be helpful, and if so, provide input about how best to organize and design the collaborative governance group. How a group comes together, how the process is designed and convened, and how decisions are made can all make a difference in the performance of the collaborative governance group. But even after all of that, it is the attention to implementation, the last step in the process, that ultimately determines if the intended purpose is actually met. Without implementation, there can be no public benefit.

References

Carlson, Christine. A Practical Guide to Collaborative Governance. Policy Consensus Initiative, 2007.

Carpenter, Susan L., and W. J. D. Kennedy. Managing Public Disputes. Jossey-Bass Publishers, 1988.

Emerson, Kirk, and Tina Nabatchi. Collaborative Governance Regimes. Georgetown UP, 2015. Nowak, Martin, and Koger Highfield. Super Cooperators: Altruism, Evolution, and Why We

Need Each Other to Succeed. Free Press, 2012.

Oregon Solutions. “Jade Greening Declaration of Cooperation.” Portland, Oregon, 2017.

Oregon Solutions. “Salmonberry Trail Declaration of Cooperation.” Tillamook, Oregon, 2015.

Ostrom, Elinor. “A Behavioral Approach to the Rational Choice Theory of Collective Action: Presidential Address, American Political Science 1997.” American Political Review, vol. 92, no. 1, March 1998, pp. 1-22.

Susskind, Lawrence E., and Jeffrey L. Cruikshank. Breaking Robert’s Rules. Oxford UP, 2006.

SECTION 3

Skills to Improve

 
Source
< Prev   CONTENTS   Source   Next >