Influencing the Supply of and Demand for Results Information in Government: Can Outsiders Help? A Case Study from Ireland
In this chapter, the author’s examine the evolution of evaluation in public investment programs in the Republic of Ireland, highlighting the seeming inverse relationship between the demand and supply of results information, and the strength of the local economy. The late 1980s and early 1990s represented the "golden age” for evaluation in Ireland when EU structural funds supported a significant portion of public expenditures but carried with them requirements for evaluating those programs. Subsequent improvements in public finances in the 1990s and 2000s relaxed the demand for evaluation and it played a decreasing role in informing public policy. Until 2008, that is, when the financial crisis caused a resurgence of interest in demonstrating that government policies generated "value for money.” A private philanthropic organization, Atlantic Philanthropies, stepped in to rebuild the public sector’s evaluation infrastructure within a time-bound framework and met with some success. But, the chapter questions, how sustainable will these efforts be now that the philanthropy's mandate is running out?
Background: Factors Influencing the Supply and Demand of Evaluation Information in Ireland
As in many countries, governments in Ireland have invested significant time and effort into producing better results information about government policy and programs. Demand for evaluation to support the production of results information in Ireland has varied over the years, both in intensity and in where the demand comes from. Similarly, the supply of evaluation has fluctuated, both in terms of the quality of evaluative information being produced and with regard to the capacity to conduct evaluation (Boyle, 2014).
Historically, a “golden age” for evaluation in Ireland was when EU structural funds support was a significant element of expenditure in the late 1980s and early 1990s. There was an associated requirement from the European Commission for evaluation of the programs supported by the structural funds. Results information became more prominent, and there was some evidence of enhanced supply of and demand for evaluations and use of findings to influence resource allocation decisions (Hegarty, 2003).
As the Irish economy grew rapidly in the 1990s and early 2000s, the demand for evaluation reduced as the pressure to engage in detailed scrutiny of public expenditure decreased. As structural funds support dwindled, the external pressure to evaluate coming from the European Commission also lost influence. While some results information was still being produced, there was limited evidence of it being influential with regard to the reform of spending programs (Department of Finance, 2004).
In response to the global fiscal crisis from 2008, and the major economic downturn in Ireland, the control and management of public expenditure has been a topic of much debate and concern. This has prompted developments aimed at re-growing policy evaluation capability within government, in the context of a wider public service reform initiative which includes a commitment to apply evaluation techniques to deliver “value for money” (as it is colloquially known) in public expenditure with more emphasis on results information (Department of Public Expenditure and Reform, 2011). These new measures include:
- • The use of evaluations in periodic comprehensive reviews of expenditure (CRE), including both departmental-based and thematic evaluation reports;
- • The introduction of an Irish Government Economic and Evaluation Service (IGEES) being developed as a cross-govermnent sendee to enhance the role of economics and value for money analysis in public policymaking;
- • Changes to the program review process, known as the Value for Money and Policy Review (VFMPR) process, including more targeted reviews, alignment with the expenditure allocation process, and greater involvement for parliament in the selection and assessment of reviews;
- • The introduction of focused policy assessments (FPA), more narrowly focused assessments designed to answer specific issues of policy configuration and delivery and complement the VFMPR process;
- • The introduction of a Public Spending Code to bring together good practice guidance on evaluation and appraisal;
- • The creation of an Irish Government Economic and Evaluation Service Network to help build capacity.
Another interesting development has been an enhanced role for philanthropies in promoting evaluation of policies and programs they jointly fund with government. The Atlantic Philanthropies and the One Foundation, for example, are two “giving while living” limited-life philanthropies that have placed building evaluation and organizational learning capacity as a central element in their grant giving.
This latter point provides the main focus of this chapter, examining in particular the role of The Atlantic Philanthropies (a major philanthropic organization (www.atlanticphilanthropies.org) - hereinafter referred to as Atlantic) in prompting government engagement with results information in a more systematic way, and their influence on both the demand for and the supply of results information in government.1 The Atlantic Philanthropies were founded by entrepreneur Chuck Feeney, who decided in 1982 to establish the philanthropy and become a champion of "giving while living," maintaining that people of wealth should use it to better the world during then- lifetimes. In 2002 it was decided to limit Atlantic's life to a fixed term and to close their portfolio in 2020. By then Atlantic will have invested nearly S8 billion in programs and people, in the USA, Bermuda, Cuba, South Africa, Vietnam, Australia, Northern Ireland, and the Republic of Ireland (referred to as Ireland in this chapter).
Atlantic has worked closely with government in Ireland in co-funding a number of programs and projects based around alternative models of service delivery. For example, since 2012 Atlantic has supported 19 co-investments with government in Ireland: Atlantic’s SllOrn investment in the areas of children and youth, dementia, and disability has been matched by $290m in public funding. Atlantic has collaborated with government at all stages, on a programmatic basis, from project inception to evaluation. They have been one of the pioneers internationally of philanthropies working closely with government to achieve their goals.
As part of this partnership, they have put particular emphasis on supporting the strengthening of both the demand and supply sides of evaluation in government to promote the production of results information and enhance the evidence base for policymaking. As well as supporting service delivery, Atlantic funding has the intention of foregrounding the evidence or research base to provide better information for decision-makers on policy selection, design, and implementation.
This chapter, based on research work commissioned by Atlantic, examines how successful this approach has been. It is based on a documentary analysis of government policy and evaluation papers, combined with a small number of case studies and a range of extensive, semi-structured interviews including members of Atlantic, selected grantees and, primarily, a range of officials from selected government departments and agencies. A total of 26 interviews were carried out (of which 16 were with senior policymakers). As part of this process, a number of interviews were conducted with "bellwethers" - influential people in the public sector who are thought leaders whose opinions about policy issues cany weight and predictive value (UNICEF, 2010, p. 34). The emphasis is on taking a critical look at whether or not the emphasis placed on the generation and use of results information has impacted on government practice, and the prospects for longer-term evaluative capacity building in government.
These approaches were used to provide evidence to support or refute a theory of change established for the study (see Figure 4.1). This theory of change was designed to show how the interventions of Atlantic were intended to influence government policy with regard to public service reform.
Atlantic has put significant effort into supporting evidence generation and gathering with a view to enhancing the evidence base used to inform policy development. With regards to the program for children, for example, which covers over 120 grants and an investment of approximately SI70 million aimed at improving the health and development of children, Little
Figure 4.1 Theory of change for The Atlantic Philanthropies influence on government policy regarding public service reform.
and Abunimah (2007, p. 61) note in relation to both the Republic of Ireland and Northern Ireland:
In designing the programme, many influences were brought to bear but two stood out strongly. First, it was clear that there was potential to bring a new perspective to ideas around community engagement. Second, it was considered that government expenditure would not be swayed by community engagement alone - that there was a need for high-quality evidence. The hypothesis was that governments on the island would take notice if rigorous and independent evidence showed that innovations for children and young people were having an impact on health and development. The marrying of community engagement and serious evaluation was never going to be easy, especially on an island where, as far as can be ascertained, there had previously only been a single experimental evaluation of a programme intended to improve children's health and development.”
With regard to the program addressing ageing issues, covering approximately 110 grants and a total investment of around $200 million aimed at enhancing the rights of older people to health and economic security, Cochrane, McGilloway, Furlong and Donnelly (2013, p. 14) indicate the effect of limitations in the evidence base on the development of policy for the ageing population in the early 2000s:
... it has been argued that overall, there has been a low level of awareness of ageing as a core government policy issue (O’Neill, Twomey, & O’Shea, 2009, p. 283). The subsequent lack of planning and provision for an ageing population has been further hampered by a lack of national datasets, with a dearth of information about the experience of ageing in Ireland.
(O'Shea & Conboy, 2005, p. 4)
The following sections of the chapter examine the supply side and demand-side implications of Atlantic’s engagement with government on the production and use of results information, the influence on policy development and implementation, and summarize the main findings. On the supply side, significant investment has been made with regard to the monitoring and evaluation of programs; the development of longitudinal studies and robust indicator sets, the building of academic capacity in research and evaluation, and the creation of institutional mechanisms to connect evidence with practice. These initiatives have been undertaken with a view to developing an enhanced evidence base to inform government policy and practice. On the demand side, an important aspiration behind the investment in evidence was that policymakers would make use of this evidence base to inform thinking about policy development and investment decisions. And that policymakers would subsequently demand similar results-oriented information in the development and assessment of future policies and programs. Atlantic hoped to build capacity amongst policymakers with regard to the production and use of results information. This was part of their goal of developing leaders and champions in the public service who would mainstream the learning from the co-funded programs after Atlantic’s exit.
