Our approach to understanding success

To understand whether we are fulfilling our objectives, there are two key questions we ask about our pilots:

• Are people better off now than when we started?
• Is the initiative financially sustainable?

To support this and to ensure that we continue to develop best practices in the approach we take to the operation of our pilots, the third question we ask is:

• What are the characteristics of a successful initiative?

It is important to capture the essence of a successful pilot so that we do not have to re-invent the wheel each time we start a fresh initiative: What internal or external factors have affected the pilot, and what the relationships are between pilot inputs (time and resources), outputs (results) and outcomes (changes realised).

Answering these key questions

Our assessment approach is threefold:

• Benchmarking key socio-economic metric data before the pilots are launched.
• Regular metric collection during the pilot to assess progress (or otherwise) in two areas: Programme deployment and socio-economic benefit. Financial metrics are also be collected to understand unit cost.
• In-depth qualitative analysis to understand what has driven change in the number and quality of the pilot outputs and outcomes, and how these changes have been experienced by pilot participants, and by users of pilot services.

Reporting Results

• Programme management status reports - key performance indicators related to the programme management of the pilot, and capture lessons learned during implementation.
• Pilot results evaluation framework - will allow regular snapshots of the progress of the pilot in terms out outputs and outcomes, and a final view of results achieved at the point of handover to the operator.
• Qualitative evaluation - In-depth interviews and surveys, the use of focus groups where appropriate.

Continuing to learn

Our assessment approach will evolve with each pilot launched. The final test of a pilot is whether it is scalable. The business case for scaling up a pilot can be built on the output, outcome and financial metrics collected during pilot deployment.