Measuring Effectiveness

When do PPPs attain their goals, and when are they effective? In order to determine and explain effectiveness, we first must define this concept. As Raus- tiala and Slaughter observe, “Effectiveness is a concept defined in varying ways: for example, as the degree to which a rule induces changes in behavior that further the rule’s goals; improves the state of the underlying problem; or achieves its policy objective” (Raustiala and Slaughter 2002, 539). Drawing on Easton (1965) and Young (1994, i40ff.; 1999) and the compliance literature, we define effectiveness along three dimensions: output, outcome, and impact. In doing so, we consistently relate effectiveness to the goals set by the PPP itself.

“Output” signifies the immediate activities of a PPP, whether it is setting up institutional structures, convening meetings, setting rules, disseminating policy papers, or delivering medication, counseling, or treatment. These activities might cause changes in the behavior of the involved actors or have other effects during the further implementation of measures, that is, what we call “outcome.” Outcome can be observed when members or target groups of a PPP alter their behavior, “either by doing things they would not otherwise have done or by terminating or redirecting prior patterns of behavior” (Young 1994, 145). Hence, a PPP as an institution is effective to “the extent that its operation impels actors to behave different than they would if the institution did not exist or if some other institutional arrangement were put in its place” (Young 1992, 161). However, we are not only talking about a narrow understanding of “compliance”: for example, after a health PPP has delivered training for medical personnel (output), more cases of disease might be correctly managed at health facilities (outcome). Outcome is, moreover, not limited to behavioral effectiveness alone. Other indicators may include case detection rates or treatment success rates in cases of health PPP delivering medication and therapy. There is a direct and immediate link between this outcome of treatment and the potentially reduced prevalence, mortality, or morbidity rates—the “impact” Impact refers to the broader results of PPP activities, in particular to their contribution to problem solving. We furthermore have to discount for negative side effects. The impact of PPP activities is hard to measure because reliable data are rarely available and there are many attribution problems in a dense web of complex causal relations. Therefore, we chose to collect data on the solution of problems, such as mortality rates, and then evaluated whether this success may be attributed to PPP activities.

As stated earlier, we are mostly interested in the effectiveness of PPPs in relation to their goal attainment (Young 1994, 144). PPP goals may focus on the output, outcome, and/or impact dimensions. Some health PPPs, for example, either want first and foremost to hand out medication (output), alter behavior that causes the spread of infectious diseases (outcome), or eradicate an entire disease (impact). An HIV/AIDS initiative might be extremely successful in distributing condoms (output), but may nevertheless contribute little toward solving the problem of HIV/AIDS (impact) because of a lack of willingness or knowledge (outcome). For this very reason, most PPPs try to be active in all three dimensions. We can nonetheless identify a primary focus in their work.3

In order to rate the effectiveness of our sample of transnational PPPs, we use a three-point ordinal scale (see table 5.1), which is mostly based on qualitative indicators—the nature of a PPP’s work is hard to compare or evaluate otherwise. Our data are based on evaluations, documents, and nearly 150 interviews with PPP members and experts. Our comparison of twenty-one transnational PPPs in the areas of environmental, health, and social policy reveals that the effectiveness of PPPs varies to a great extent. We can group them into three categories: (a) PPPs with high effectiveness, which achieve most of their goals and provide output, outcome, and impact; (b) PPPs with medium effectiveness, which have reached some goals but failed on other dimensions; and (c) PPPs with low effectiveness (sometimes near ineffectiveness), which

table 5.1 Qualitative Indicators for Effectiveness




High (3)

Provision or adoption of knowledge, standards, services as envisioned in the stated goals

Substantial change in behavior of targets, extensive application/ implementation of knowledge, standards, services

Substantial contribution to solution of problem

Medium (2)

Substantial policy papers and some provision of knowledge, standards, services, but failure to achieve all stated goals

Some change in behavior of targets, some application or implementation of knowledge, standards, services


contribution to solution of problem

Low (1)

Mere paperwork and meetings with no or few results

No or low change in behavior of targets, hardly any application/ implementation of knowledge, standards, services

No or low contribution to solution of problem

struggle to provide the desired output and fail with regard to most of their goals, and the outcome and impact dimension (for an overview see table 5.2). Within each dimension of effectiveness, we rank each PPP as compared to the others, thereby identifying the relative effectiveness of each. We will now present one case for each category to demonstrate how we applied our indicators in the coding process.





GAVI Alliance

Extend the reach and quality of immunization coverage



Global Fund to Fight AIDS, Tuberculosis and Malaria (GF)

Attract, manage, and disburse resources to fight the three diseases



Water and Sanitation for the Urban Poor (WSUP)

Provide access to water and sanitation in urban slums



Global Alliance for the Elimination of Leprosy (GAEL)

Attain elimination of leprosy at a national level in all endemic countries



International AIDS Vaccine Initiative (IAVI)

Accelerate the research and development of an HIV vaccine



Common Code for the Coffee Community (4C)

Develop a voluntary code of conduct comprising basic social, environmental, and economic practices in mainstream coffee production, processing, and trading



Social Accountability 8000 (SA 8000)

Establish an auditable certification standard based on international workplace norms



Global Network on Energy for Sustainable Development (GNESD)

Carry out policy analysis on energy issues that can facilitate attaining the Millennium Development Goals



Global Alliance for Improved Nutrition (GAIN)

Fight vitamin and mineral deficiencies in developing countries






World Commission on Dams (WCD)

Develop internationally accepted standards for the planning and construction of large dams



United Nations Global Compact (GC)

Mainstream ten principles in business activities around the world and catalyze actions in support of broader U.N. goals



Building Partnerships for Development in Water and Sanitation (BPD)

Promote the effective delivery of safe water and sanitation services to poor communities in developing countries



Global Public Private Partnership for Handwashing with Soap (PPPHW)

Spread awareness about the importance of handwashing with soap to prevent diseases



Renewable Energy and Energy Efficiency Partnership (REEEP)

Promote renewables and energy efficiency in developing countries



Global Water Partnership (GWP)

Support countries in the sustainable management of their water resources



Global Alliance for Workers and Communities (GAWC)

Improve workplace experience and future prospects of workers in developing countries



Global Village Energy Partnership (GVEP)

Increase access to sustainable modern energy in rural areas in developing countries



Roll Back Malaria Partnership (RBM)

Halve the global burden of malaria by 2010






International Alliance against Hunger (IAAH)

Increase the public awareness of hunger, mobilize public campaigns, and facilitate local and national initiatives against hunger



World Committee on Tourism Ethics (WCTE)

Promote the Global Code on Ethics in Tourism and reconcile disputes about code compliance



Children’s Vaccine Initiative (CVI)

Develop new and improved vaccines


< Prev   CONTENTS   Source   Next >