Qualitative and quantitative methods
The choice between qualitative and survey-based (quantitative) statistical approaches involves trade-offs; combining both is generally recommended. In the assessment of sustainable livelihoods, the combination of qualitative and quantitative methods needs to be comprehensive and carefully planned (La Rovere and Dixon 2007; Walker et al. 2008). Combining approaches makes it possible to estimate the indirect consequences of agricultural research and to establish a detailed and contextual assessment while simultaneously identifying the quantitative patterns of impact. Most recent guidelines recommend CGIAR use combined research impact tools.
Baseline data
A weakness in many studies is the absence ofbaseline data.This is evident in natural resource management cases, which often rely on the “best” available data and indirect indicators. One explanation for this is that impact assessments were not planned for when projects were designed and launched.To mitigate this, an obvious recommendation is proper planning for evaluation processes in the initial phase of a project. Recently, donors have provided substantial funding to CGIAR, which has enabled the establishment of extensive baseline databases that will permit rigorous tracking of projects or programme impacts over time.
Counterfactuals
In the absence ofbaseline data, impact assessment is methodologically challenging, not only due to poor benchmarks but also because baseline information is necessary to establish counterfactuals. The history of CGIAR ex post impact assessments at the project and programme levels is significant, but the use of proper before and after counterfactuals (as well as with and without counterfactuals) is less frequent due to the noted scarcity - until recently - of properly designed baselines and comparison groups.
Valuation methods
The majority of impact assessments and evaluations have neglected to incorporate non-market environmental impacts (impacts that do not translate into monetary costs) because estimating research impacts outside market effects is complex (Bennett 2009). With evaluations being established largely within the conceptual rigour of traditional cost-benefit analysis, the inclusion of non-marketed effects requires the estimation of benefits and costs in monetary terms. Hence, for evaluations to include non-market benefits and costs requires not only the already challenging processes of forecasting the impacts of research outputs on various goods and services (marketed and non-marketed) but also the estimation of society’s values for all the forecast impacts.
CGIAR’s evaluations mostly draw attention to the omission of environmental impacts in their scope. For example, Hazell (2008: XV in a study of agricultural research in South Asia concludes “there are few impact studies from South Asia that estimate a return to a research investment corrected for environmental benefits and costs”. He also points to the use of indicators as a means of ranking research investments in terms of both their environmental and poverty impacts. There is also no real consensus as to which indicators should be used. Without a consensus, the use of indicators remains arbitrary and limited. Empirical studies are recommended to link research investments to environmental outcomes in order to assess past investments and design more effective research initiatives.
The 2011 Independent Science and Partnership Council book on CGIAR sheds light on the state-of-the-art methodological progress on valuation methods (choice modelling and contingent valuation) and ongoing work on measurement, modelling, and data collection for environmental impact assessment of agricultural research.
