Evaluate the pilot
As part of the experimentation stage and the undertaking of a pilot, it is also critical to gather evaluation data. There are many different types of evaluation and ways of carrying them out. However, there is no single optimum way of designing or undertaking an evaluation: the most effective evaluations are those that identify what information is most needed to test the potential of the intervention and support its future development and involve a bespoke evaluation design. This can be split into three core questions:
- 1. Is the intervention well designed and suitable for its participants?
- 2. Does the intervention run smoothly and fit in with the care process?
- 3. Does the intervention show signs that it will, in its full form, be able to achieve its aims?
To answer these three questions, there are three types of evaluation that can be carried out:
- 1. Formative evaluations—Formative evaluations have the aim of improving a project’s design and performance. A formative evaluation may assess whether the project is suitable for the target participants, ascertain whether it has integrity as an intervention, and identify what changes are needed to improve its design
- 2. Process evaluations—Process evaluations explore the successes and challenges related to how an intervention runs. This may include identifying not just what worked well but also what the potential obstacles were and how these could be overcome; assessing how well the intervention fits in with other processes involving participants, staff, or institutions; and confirming whether the logic model from Step 4 appears to hold in practice as well as in theory and whether the inputs-intervention-impacts flow holds strong
- 3. Outcome evaluations—Outcome evaluations ascertain whether an intervention had the desired effect on participants. This may include confirming whether the main outcomes show signs of being achieved, exploring whether there are other subsidiary effects, and identifying how the intervention could be adapted to improve its impact
Evaluating a pilot is, of course, not going to give a complete picture of how the actual intervention will run, as the pilot is, by nature, a simplified or smaller- scale version. However, a project that does not produce strong results for the three types of evaluation is unlikely to find success on a larger scale. A key issue is how to go about running these evaluations, as different qualitative and quantitative methods can be involved in an evaluation.