Practical Lessons Learned in the Matching Process of a Large Computer Modelling Framework
The integrated project SEAMLESS, funded by the European Commission, aimed at developing an integrated framework of models that can be employed to better inform ex ante impact assessments of EU agricultural and environmental policies (van Ittersum et al. 2008). It was funded by DG Research (the European Commission's Directorate-General responsible for funding and implementing European research programmes) as one of a series of integrated projects aimed at developing research tools to underpin ex ante impact assessment. In the case of SEAMLESS, DG Research perceived that the European Commission's Directorate-General (DG) for Agriculture (and perhaps other DGs) would have need for this type of model-based framework, to be used by or to provide information to the policy analysts and policy support units in the DGs. In the course of the SEAMLESS project, around 20 meetings took place in Brussels with DG Research and/or DG Agriculture and representatives of various other DGs to define the potential role of the project. DG Research and the research consortium defined the role as being essentially heuristic; symbolic and relational roles were never demanded nor discussed. Concrete topics on model development and contextualization -which were discussed in the course of the many interactions in Brussels - as well as the responses of the project's modellers, are summarized in Table 5.1.
Next to the 'extrinsic' factors (for example, making a policy impact) that will be further discussed below, there are of course 'intrinsic' methodological and technical requirements of models that must be satisfied. Peer review and publication of all model components - and their integration - in international journals are a necessity to build credibility. Model
Table 5.1 The Integrated Framework: a comparison of potential user requirements* and the responses from the SEAMLESS project
Note: *As defined and discussed in a series of workshops in Brussels.
documentation is a second obvious requirement, but is far from trivial in practice. Third, the models should preferably be freely available, that is, open source, such that those interested in the model and its code can, in principle, themselves evaluate or use the model. In a recent overview article, Britz et al. (2012) present a number of other intrinsic qualities of integrated assessment models in agriculture. These include consistent linkages between different organization levels, often the micro and macro level (in other words, farm to regional or market level), model calibration and validation and uncertainty analysis. The model description and documentation must explicate underlying assumptions. In an uncertainty analysis, consequences of model assumptions and all sorts of uncertainties as to processes and data can be investigated by the modellers. The challenges of doing this in a scientifically sound yet meaningful manner for users are far from trivial. Gabbert et al. (2010) explored a user-oriented approach, but uncertainty analysis is clearly an intrinsic model quality that requires more attention to avoid 'black box' syndromes of research models and their application. This is a quality contributing to a successful contextualization of computer models for policy assessment.
As to the extrinsic factors, a number of lessons learned became apparent to the project coordinator (the lead author of this chapter) while reflecting on the process of science-policy interaction. First, research project formulation and execution require careful attention to expectation management. Project proposals (for Framework Programmes of the EU and other funding agencies alike) must be ambitious and promise well-defined outputs to win funding. In the case of SEAMLESS it was not possible - it was strongly discouraged by DG Research - to interact with potential users during the definition of the project. Yet the proposal had to be precise in its deliverables, and the complexity of the consortium of 30 research institutions (with over 150 scientists) required a precise work allocation and plan of work. Once the project had been approved and started, interactions with foreseen users were initiated and both the funder (DG Research) and foreseen users (mainly from DG Agriculture) strongly encouraged the project to raise its ambitions (Table 5.1) and sometimes to deviate from the original project proposal. The latter requires a level of flexibility which is sometimes difficult to attain in a research consortium in which the partners and individual scientists have their own specific roles. Also, although the project was funded primarily to achieve methodological advances, there was a continuous push to analyse 'hot' political topics. The project had to manage expectations in terms of what could be delivered in that respect, that is, a tension exists between methodology development and application. The methodology-application tension is a particular issue when the work is carried out by universities and institutes primarily motivated by research rather than commercial/policy applications.
Already at an early stage in the policy formulation interactions in Brussels, the issue of maintenance and continuity of the research tools was brought up by the foreseen users. While originally DG Research had suggested that it would take responsibility for continuity in the event of a successful project, it subsequently became clear that continuity was to be first and foremost a responsibility of the research consortium, despite various intermediate project reviews being very positive. As no single consortium member (university or institute) was able to maintain and apply all the computer models of the framework, it was essential to identify the key partners needed to maintain, further develop and apply the core components of the framework. Just before completion of the project, the SEAMLESS Association was established with around 10 core members from the consortium. The budget of the Association was modest and composed of membership fees from each partner. Though DG Research favoured the establishment of an association, neither it nor DG Agriculture felt responsible for providing financial support. The establishment of the Association is precisely the type of institutional mechanism that the knowledge utilization literature (Nutley et al. 2007) argues is required to institutionalize knowledge use over the longer term.
Finally, two important overarching lessons were learned from the science-policy interface during the SEAMLESS project. First, a stepping stone must be created in Brussels to network and contextualize the models and their representation of systems. It seems indispensable to post an intermediate person (cf. knowledge broker - Ward et al. 2009) in Brussels, to work on the science-policy interface on a daily basis. Working on this issue remotely, in the case of the SEAMLESS project from Lund and Wageningen, is not sufficient, whatever the level of personal commitment. A second lesson learned is the crucial role of the funder, as well as the agency responsible for drafting the research call, in this case DG Research. Much can and should be expected from efforts of the research consortium to contextualize the research models and to ensure a proper matching of methodologies to the politically relevant questions and processes. However, the donor(s) can play the crucial role of stepping stone in a networking process which potentially greatly facilitates the contextualization and uptake of the developed models.