Evaluating difference and what makes a difference
Being able to confidently say that a programme or service makes a difference to the lives of young people is important—but can we also say why? Knowing why a programme or service has the impact it has is vital in achieving the same positive outcomes consistently, spreading this learning to others, and replicating success more widely. It also helps to demonstrate what is unique about a service or programme: there may be many others who aim to—and do—develop young people's social and emotional capabilities. But approaches will differ. Why should your particular service or programme be continued, scaled up or invested in?
A key step in understanding why a programme or service has the impact it does is developing a theory of change. A theory of change is a causal pathway, which links what you do with your outcomes and impact. It explains the “how”, by setting out the steps needed to make change happen. It also helps to check assumptions (for example, are you assuming that all young people respond to IT, which is why they engage with your programme, or is it really the lunch you provide?) and the evidence behind a programme (for example, how do you know that young people learn effectively using IT? Is there evidence to suggest that this is the case?). This can assist in focusing measurement— making sure what is being measured is what matters, and that the methodology is robust and appropriate. Developing a theory of change can be a positive transformative experience for those involved, and is also very effective at bringing together different stakeholders—face to face practitioners, managers, funders, parents, and young people themselves.
This chapter has looked at the importance of measurement in work with young people. Measuring our impact is critical in demonstrating not only the difference we make directly to longer-term outcomes around learning, work, and health for example, but also how we
contribute indirectly through the development of social and emotional capabilities—the outcomes we know that matter to young people as they navigate risk and opportunity.
We know that there are no easy answers, and the size of the challenge we face: we have thousands of individuals and organisations working with young people, who provide vital services that change lives and offer good value for money—but who are struggling to prove it. Even when we can begin to show where we are making a difference, we cannot always show how.
We also have funders, investors and commissioners who are struggling to identify effective programmes, services and approaches, and risk making uninformed decisions about where best to put their money.
At sector level, the evidence base for social and emotional capabilities is not growing, the difference in language and terminology remains, and there are no common platforms that allow for knowledge sharing or comparison.
Ultimately, perhaps the biggest and as yet unspoken problem of all—young people lose out. The best services are not always sustained, decision-making processes are not transparent, leading to a loss of trust, and young people—as service users—cannot always explain the value they take from services.
So what is the solution? A useful way to think about this is in the context of a change agenda for the whole sector—a journey we are travelling on together, rather than an individual path.
In previous work, New Philanthropy Capital NPC4 has set out a series of steps that we might follow:
1. What is the outcome to be measured? Do organisations in the sector agree on a single outcome or set of outcome measures?
2. How is that outcome defined? Has it been defined by a measurement tool or set of criteria?
3. How should the outcome be captured? Are the right systems in place to enable services to capture it?
4. How can the outcome be attributed to an intervention? Can services explain what would have happened to young people without their intervention?
How can the outcome be valued? Are there good financial proxies that can be used to estimate value?
A clearer focus on the outcomes that matter, and why, alongside greater consensus and consistency in use of language, is the first vital step. This will allow a clearer focus on measuring these outcomes, and understanding how our work makes a difference. More robust measurement will also enable us to grow more confident in showing where we contribute to cost savings.
We also need to remind ourselves of the benefits to our sector:
• The development of a common language—transparent, comparable, and consistent
• A growing evidence base, which testifies to the role and contribution of services for young people
• A virtuous circle where providers grow in confidence as do investors, commissioners, and funders
• Clarity about your role to communities, schools, and business, forging links and creating partnership opportunities
• A better understanding of value, and parameters for assessment
• Support for reflective practice, professionalisation, and growth
• Better service design and hence better outcomes
• A stronger case for the most effective services.