Purpose
This report is a process evaluation of the Social Bond Pilot
Oranga Tamariki was directed by Cabinet to evaluate the effectiveness of the Social Bond Pilot. Because of the early stage of the pilot, this evaluation two-years after the start of the pilot is a process evaluation with two main areas of interest:
- How the Social Bonds contractual arrangement is operating
- How the Genesis Youth Trust programme is operating.
Methodology
A logic model provided the theoretical foundation for the evaluation
We developed a logic model to outline the different activities, outputs and outcomes the Social Bond Pilot aims to achieve (Appendix 3). Logic models are important because they provide a theoretical foundation for evaluation by setting out the ‘building blocks’ for change. The evaluation questions, data collection tools and analysis are aligned with the steps in the logic model.
An evaluation framework was developed to align with the logic model and set out the indicators or measures for each evaluation question (Appendix 3). Indicators were generally based on qualitative data such as information from interviews and measures based on quantitative date such as Genesis administrative data.
The evaluation included qualitative and quantitative data
We collected qualitative and quantitative data for the evaluation from:
- In-depth interviews with key stakeholders
- A case study with a rangatahi (aged 12-18) and whānau
- Review of a report by Synergia of interviews with Police frontline staff
- Review of a report commissioned by Genesis of consultation with local iwi and Māori organisations
- Analysis of Genesis administrative data.
A detailed table of these participants is in Appendix 4 of this report.
The Oranga Tamariki Ethics Advisory Committee reviewed and approved the evaluation approach. Our evaluation team included evaluators with different ethnic and cultural backgrounds ensuring the perspectives of senior Māori and Pacific evaluators contributed to all aspects of the evaluation.
We analysed interviews and case study data thematically using a framework grounded in the evaluation questions. We identified similarities and differences in emerging themes. We workshopped emerging themes as a group to ensure consistency of coding across researchers. Quotes are included in the report with generic labels such as ‘Genesis’, ‘Investor’ to provide confidentiality to those we interviewed
Administrative data were provided by Genesis to the end of March 2020 Genesis provided administrative data for the evaluation. An Oranga Tamariki Evidence Centre team member analysed administrative data descriptively for this
stage of the evaluation. Additional analyses to identify factors contributing to outcomes will be a focus of the later outcome evaluation.
Key Results
How the Social Bonds contractual arrangement is operating
- Interviewed stakeholders had differing views on the benefits and disadvantages of social investment, influenced by their views on the ethics and morality of ‘private investment in the public good.’
- The procurement process was complex and took several years.
- The resulting Social Bond Pilot agreement is complex but working. A challenge has been the lack of ability to review the measures in the agreement that are the foundation for payments. Rapid organisational change and pressure to meet enrolment targets also put pressure on staff.
- However, the outcomes focus of the contract has provided a foundation for innovation in the use of evidence-based tools to guide practice, data to monitor progress, longer duration of support, and development of multi-disciplinary teams.
- Consultation with Māori and Strategic Partners is required to explore whether and how a Social Bond model and contracting approach would work in a kaupapa Māori context, and if there is any interest in pursuing this approach.
How the Genesis Youth Trust programme is operating
- Genesis support young people for two-years: an initial intensive intervention phase; and a less intensive follow-up phase. The Genesis team provides support to rangatahi around education, development of skills that will help with employment, one to one mentoring, counselling, and referral to other organisations and specialist services such as addiction services.
- Rangatahi Māori were 71% of those enrolled with Genesis.
- Genesis currently rely on staff personal networks and informal links to iwi and providers to connect rangatahi to their cultural identity.
- Lack of access to rangatahi and whānau limited the evaluation.
- Overall, the Social Bond Pilot is supporting young people as intended although referrals and enrolments are lower than the contracted maximums.
- Participants’ likelihood of reoffending, number of offences (if any), and offence severity all declined following enrolment. This was true for both Māori and non-Māori participants.
- However, it is too soon to draw quantitative conclusions about the effectiveness of the programme on offending related outcomes. This will be explored in a forthcoming outcomes evaluation.