Facing the challenges—putting measurement into practice

Attribution and contribution

Making a judgement about whether or not a programme “works”, and achieves its intended impact, involves making causal links between
what we do and the outcomes for young people—that is, that the outcomes young people achieve are because of their involvement in our programme, and are not likely to have happened anyway. This is particularly challenging in work with young people, for two main reasons: first there is the problem of attribution—young people lead complex lives, and it is difficult to isolate the impact of one service or programme in amongst a range of other influences. Second, for many young people, the outcomes we want for them (and that they want for themselves) lie some way ahead in the future. This may be because of their age (for example, a ten-year-old will not find fulfilling and sustainable employment before he/she is at least sixteen) or because of their stage (a young person leaving custody, for example, will need intensive and long-term support to build new relationships, find somewhere safe and secure to live, and to re-enter learning or work).

So, how can a service or programme working “upstream” with young people demonstrate its impact? How can it show it is making a difference even when it cannot directly measure the numbers of young people achieving longer-term or “harder” outcomes like finding and sustaining a job, getting a degree, or settling into a healthy, happy relationship? Most services work to develop young people's social and emotional capabilities precisely because of their importance in achieving these longer-term outcomes, and much effort has gone into understanding the relationship between social and emotional capabilities and the achievement of such outcomes—for although important, this link is not straightforward.

There is a growing consensus around the role that social and emotional capabilities play in the achievement of “extrinsic” outcomes— those that can be measured and valued by other people. The evidence base which supports these connections is substantial and evolving, but more work is needed to strengthen our understanding.

One way to reflect on the challenge is to focus on contribution— where it is too difficult to prove cause and effect, contribution analysis2 can help to make reasonable conclusions about the impact a programme is having on outcomes. Contribution analysis means taking into consideration all other influencing factors (for example, a young person's family, peer group, and teachers) and clearly articulating why you think the programme will play a part—what is its contribution and why might we logically expect it to make a difference?

Articulating return on investment

Alongside demonstrating the impact of the work they do with young people, many services are increasingly keen to show return on investment—that is, the positive “returns” that are generated, after the service has been paid for. This is usually calculated as savings—costs that are avoided as a result of a service. For example, this may be a service which aims to divert young people from prison, thus avoiding the high costs associated with custodial sentences. A key challenge for services for young people is that potential savings are spread among a variety of agencies: programmes that support young people to return to employment, education or training, for example, may provide savings for the local council, but also the Department of Work and Pensions in the future; similarly, work with young people to prevent drug and alcohol misuse may accrue savings for the Department of Health too. When savings are spread around like this, and sometimes over time, it can be difficult to make a strong case for investment into programmes—who should make the investment if the returns are shared? It is also important to understand the range of fixed costs and economies of scale to take into account: diverting a young person from custody, for example, will not close a unit in a Young Offenders Institution. Building positive mental health and wellbeing will not make the local CAMHS team redundant—in the short term at least, many more young people will be on the waiting list. Whilst we can hypothesise savings, we need to translate this into available hard cash in order to make the strongest case. Furthermore, the issue is made much more complex when focusing on social and emotional capabilities. The challenge here is to connect the development of these capabilities with outcomes where we can ascribe costs savings. The diagram below shows how we might begin to do this, but more work is needed to strengthen these connections.