Purpose
In February 2020, Oranga Tamariki commissioned a collaborative team – Judy Paulin, Michele Lennan, Hector Kaiwai and Dr Sue Carswell – through Artemis Research NZ Ltd to conduct a process and outcomes evaluation of the SB pilot.
The key evaluation questions are:
- To what extent did the SB pilot enable rangatahi to remain out of custodial remand during the bail period and improve their wellbeing and the wellbeing of their whānau? What has helped or hindered the achievement of any changes in compliance and wellbeing?
- How well was the SB pilot implemented? What helped or hindered SDP to implement the pilot as planned? How could the SB service specifications be improved, if at all?
Methodology
Overview of the evaluation design
All five supported bail pilot providers agreed to participate in the evaluation: Reconnect Family Services, Auckland; Te Ikaroa Rangatahi Social Services, Hawkes Bay; Highbury Whānau Centre, Palmerston North; Youth & Cultural Development, Christchurch; and Te Hou Ora Whānau Services, Dunedin.
The intended evaluation design comprised:
- five focus groups with rangatahi who have experienced the Supported Bail pilot (one focus group/provider)
- five focus groups with SDP staff in the Supported Bail pilot (one focus group/provider)
- 10 case studies centred around individual rangatahi, each with interviews with the rangatahi, their whānau, youth worker, whānau support worker (where applicable) and social worker (two cases/provider)
- 10-15 interviews with key Oranga Tamariki staff – Partnering for Outcomes (PfO) advisors, Youth Justice (YJ) Services staff and other Wellington-based staff
- analysis of structured data from SDPs about completion or early exit from supported bail
- an analysis of key documents.
On 4 August 2020, the Oranga Tamariki Evidence Centre granted ethics approval for Artemis Research to proceed with the data collection based on the intended design.
The providers gave a good deal of assistance to the evaluators for the main fieldwork. As much as possible, the data collection matched the intended design. However, at three providers – Te Ikaroa Rangatahi Social Service, Highbury Whānau Centre and Te Hou Ora Otepoti – it was not possible to conduct a focus group with rangatahi due to the small numbers of potential participants living locally who had started on the supported bail programme. And at two sites only one case study was possible, and at a third none was possible in the timeframe for the fieldwork.
Data collection with rangatahi
Eighteen rangatahi who had started supported bail with one of the pilot providers were interviewed for the purposes of the evaluation in a focus group or one-on-one setting.All rangatahi gave their informed signed consent, and were given koha for their time and the sharing of their experiences and views. Six case studies were undertaken centred around six rangatahi.
Data collection with whānau members
All whānau members gave their informed signed consent, and were given koha for their time and the sharing of their experiences and views. The tools we used to collect the data are included in Appendix 4. At least one whānau member contributed to each case study. In total, 10 whānau members contributed to the six case studies (five fathers, two mothers, a grandmother, an aunt, and a sister). In addition, one whānau member (a father) was interviewed with his child.
Data collection with others for the case studies
Six kaimahi and two whānau support workers (one family liaison, one Whānau Ora worker) also contributed to the case studies, as did six Oranga Tamariki social workers. All first gave their informed consent to be interviewed. The tools we used to collect the data are included in Appendix 4.
Data collection with supported bail providers
The data collection methods we used with the pilot providers and numbers we interviewed are set out in Table A3.2. All first gave their informed consent to be interviewed.
Data collection with other key stakeholders from Oranga Tamariki
Seven interviews (by Zoom or by phone) were undertaken with Oranga Tamariki staff: four youth justice site managers, two Partnering for Outcomes advisors and the programme manager. All interview participants first gave their informed consent to be interviewed. The tools we used to collect the data are included in Appendix 4.
Limitations
The main limitation of the methodology is that information was collected from providers in the SB pilot only and not from ‘standard’ supported bail providers. Structured quantitative data on the SB pilot was not sufficient to be able to do further outcomes analysis. Most other key stakeholders from Oranga Tamariki had little direct contact with their local SB provider so their feedback was mostly second-hand.
Analysis
Audio recordings from the focus groups and interviews were transcribed for analysis. Qualitative material (from the case study interviews, focus groups, other interviews) were coded for key themes and used to answer the key evaluation questions. The identification of themes was undertaken in an iterative way and through a combination of an individual evaluator’s analysis and collectively as a team. We have included some quotes in the evaluation report where they represent a commonly held view or a unique perspective. Quantitative administrative data (sourced from Oranga Tamariki) about completion or early exit from supported bail is presented in tabular form in section 3.1.
Key Results
The evaluation, completed in 2021, found that the Supported Bail pilot enabled a good proportion of rangatahi to remain out of custodial remand during their bail period. Of the 168 rangatahi on the pilot in 2019 and 2020, 106 (or 63%) completed the programme. The completion rate for rangatahi Māori was on a par with that for rangatahi of all ethnicities.
The programme served to improve wellbeing of rangatahi and their whānau, based on self-reports from those participating in the evaluation. Rangatahi spoke of positive changes they had made or were making towards improving their lives. Most rangatahi said they would recommend the programme to other young people in similar circumstances.
The pilot was found to be well implemented, largely due to the skilled and experienced kaimahi (youth workers) and whānau support workers employed by the pilot service delivery partners. Kaimahi were instrumental in bringing about change in rangatahi, who attributed some of their achievements to the level of support from kaimahi. Kaimahi also supported rangatahi and their whānau at their Family Group Conferences (FGCs) and Youth Court appearances.
The presence of whānau support workers alongside kaimahi helped parents/caregivers to support their rangatahi even after they completed the programme. They were often able to exert a positive influence on the wider whānau.
Key recommendations:
- Stronger messaging about the availability of a supported bail provider on evenings and weekends, if needed
- Using longer-term contracts to provide greater stability and continuity for supported bail providers
- Work to enhance the recognition and the place of kaimahi at FGCs and court appearances
- Taking care when transitioning rangatahi onto other programmes given they may have become dependent on their kaimahi
- Any expansion in the number of supported bail providers should include more iwi providers, given the over-representation of Māori in the youth justice system
- Broadening the eligibility criteria for supported bail to include rangatahi at lower risk of reoffending, but who would still benefit from support.