Evaluation and Return On Investment (ROI)

Bethia Mcneil


What matters more, the pleasure or the measure? Or is it in fact both? In working with young people, particularly in out of school or informal settings, measurement is often considered to come at the expense of something else; it is burdensome, and gets in the way of developing relationships. It can be felt to be at odds with a more intuitive perspective, which values individual experiences and stories, and the quality of interaction.

Services that focus on the personal and social development of young people have long collected and shared qualitative evidence about the difference they make to the lives of those they work with—through stories, journeys, and anecdotes. This evidence can be very powerful, but the youth sector has historically struggled to draw together quantitative evidence of value and impact. An important part of making the case for such services is showing that they can offer savings to the public purse, preventing the outcomes we know can represent high costs to society and the individual. Articulating value clearly is critical, particularly when resources are stretched. It can help build a case for investing in the most effective services which secure the best long-term outcomes for young people. It can also help services evidence their role and impact.

But measurement is often seen to be someone else's agenda, usually imposed. It is not felt to be something which sits comfortably alongside the practice and processes of our work with young people, which would prioritise their voice and their individual journeys travelled.

Measurement is difficult, and resource intensive. Many organisations feel isolated when they think through the practical implications, and are struggling as they consider the path that lies ahead.

But measurement is not an agenda that can be ignored. More to the point, we should not want to ignore it.

This chapter focuses on the role of measurement in work with young people, and why it should be a shared agenda. It explores the wider context and the growing emphasis on evidencing impact. It looks at some of the challenges of “measuring” personal and social development, and offers some guidance on translating theory into practice.

Measurement in context: focusing on mental toughness

As a society, we tend to value what we measure, so the first step is to make sure that we are measuring what we value—the things that matter.

Young people are living, learning, and negotiating transitions to adulthood and independence in an increasingly complex and challenging world, in which they face greater levels of choice and opportunity, but also unprecedented uncertainty and risk. This calls for empowered, resilient young people, who play an active role in navigating these paths.

There is substantial and growing evidence that developing social and emotional capabilities supports the achievement of positive life outcomes, including in education, work, and health. Employers increasingly talk of capabilities such as resilience, communication, and negotiation as being the foundations of employability. These capabilities are important for their own merit and for their significance in achieving other outcomes. In other words, the services that support young people to develop social and emotional capabilities help them to achieve “personal change” in their lives, which itself can lead to “positional change” in their circumstances.

Figure 1. The factors that can affect young people's outcomes.1

Evidence suggests that approaches focusing on building social and emotional capabilities can have a greater long-term impact than ones that focus on directly seeking to reduce the “symptoms” of poor outcomes for young people. Yet, at the same time the very services that most explicitly focus on supporting young people to develop these capabilities are under unprecedented financial pressure. At a time of financial austerity, demonstrating how services improve outcomes, and reduce costs to the public purse, will be attractive to providers, funders, and commissioners alike.

Many services play a vital role in this picture, but while they have powerful examples of lives transformed, over time they have struggled to provide “harder” quantitative evidence of the difference that they make, and to articulate the value that they produce for young people and for society more broadly.

Historically, the evidence base for the significance of social and emotional capabilities has been hard to draw together. There has been a lack of consensus around language and definitions, and it has been widely assumed that the development of these capabilities is too difficult to measure or evidence. Providers have tended to depict the value of their work through the individual journeys of young people, and by measuring or monitoring the activities that are easiest to quantify. Often these are the tangible aspects of their work: “indicators” such as number of qualifications achieved, number of hours of services provided, or attendance, for example. These are activities where it is possible to capture externally verifiable and recognised inputs and outputs relatively easily. This focus on the “harder” outcomes often comes at the expense
of so-called “softer” social and emotional capabilities. Self-esteem, resilience, and thinking skills, for instance, all underpin young people's progress but can be hard to assess. It can be difficult to make the case for such “softer” outcomes, even where young people need to develop these capabilities before they can go on to achieve “success” in other areas of their lives.

But just focusing on the aspects of services that are easier to measure has major weaknesses, and does not reflect their value. It fails to reflect a cornerstone of the value added by services for young people: the attainment of social and emotional capabilities.

Moreover, it has never been more important to reflect this value. In times of financial austerity, all public spending is under scrutiny. Every service funded with public money needs to be able to demonstrate the difference it makes, and its long-term value—its case for investment. As a consequence, there is increasing pressure to assess and articulate the value that services produce, both for the young people who use them and for society as a whole. Individuals and organisations involved in funding and/or commissioning, organising and delivering such services need to know the outcomes that matter and the difference services are making to the lives of young people.

However, measuring and isolating the impact of a particular service on the development of young people's social and emotional capabilities is not straightforward. The link between the impact of services and the achievement of these outcomes is hard to assess or demonstrate, because for many young people, these outcomes are some way ahead in the future. Part of the difficulty lies in the complexity of young people's lives: there are a huge range of influences on their development—from school and youth clubs to health professionals, alongside family and friends—and many services will touch on multiple aspects of a young person's life, across home, family, and community. The process of measuring outcomes is hard too—we do not always share the same language or terminology, and there are hundreds of different approaches in use.

< Prev   CONTENTS   Next >