Partners for Change Outcomes Management System

Partners for Change Outcomes Management System
30 Sep 2019


The report evaluates the Partners for Change Outcome Management System (PCOMS), a behavioural health outcomes management system that was designed for counselling and other therapeutic services.


Evaluation design

The evaluation used a mixed methods sequential explanatory design. As understanding the experiences of trial participants was always going to be central to the evaluation, qualitative data was the priority; the quantitative and qualitative analysis and findings were integrated during the interpretation stage.

Data collection and analysis methods

Literature scan

A literature scan, including books, journal articles, websites and grey literature, was undertaken. In addition to On Becoming a Better Therapist: Evidence-based Practice One Client at a Time, by Barry Duncan (2014), and the material on the website, other key texts included:

  • PCOMS Document (SAMHSA, 2017);
  • Report (Partnering for Outcomes Foundation Aotearoa, n.d.);
  • Report (Connect+Co, 2016) (unpublished version of above with appendices); and
  • Various book chapters and articles (for example, Anker, Duncan, & Sparks, 2009; Bohanske & Franczak, 2010; Bringhurst et al., 2006; Cooper, 2013; Duncan et al., 2003, 2006; Duncan & Sparks, 2016; Lambert & Shimokawa, 2011; Low, 2012; Manthei, 2015; Miller & Duncan, 2004; Miller et al., 2003; Reece, Norsworthy, & Rowlands, 2009; She et al., 2018).

Based on recommendations by Superu the scan also reviewed evidence based and systematic review websites relevant to statutory social work. These included:

  • Campbell Collaboration;
  • Cochrane Library;
  • California Evidence Based Clearing House for Child Welfare;
  • Crime Solutions;
  • Office of Juvenile Justice and Delinquency Prevention;
  • Blueprints; and
  • Investing in Children Programme materials.


The evaluation also reviewed the following Oranga Tamariki documents:

  • Oranga Tamariki PCOMS Pilot Plan July 2017 (11 August 2017, v.1_2_1);
  • Oranga Tamariki PCOMS Logic Model (10 October 2017, v.3);
  • Oranga Tamariki PCOMS 2-pager guidance (7 February 2018);
  • Oranga Tamariki PCOMS PowerPoint presentation for Managers and Practice Leaders (undated);
  • Oranga Tamariki PCOMS Training day PowerPoint presentation (19 Feb 2018, modified
  • Oranga Tamariki PCOMS 23 May 2018 workshop presentation slides from Barry Duncan; and
  • Oranga Tamariki Core Practice Standards (undated).


The PCOMS online survey of Oranga Tamariki staff attending PCOMS training (n = 52), was designed by the Oranga Tamariki Evidence Centre, and administered before the commencement of the evaluation. The survey was run twice, (November 2017, and April/June 2018) and achieved a response rate of 42% (noting that some of those who had attended the training were no longer using PCOMS and so on that basis may have chosen not to respond). Material was provided for the evaluation in the form of Survey Monkey output presentational data, as well as a consolidated Excel spreadsheet. Analysis was limited to presentational graphics.

Administrative data

For Oranga Tamariki PCOMS administrative data, access was provided to the PCOMS BON website. BON does not allow for raw data to be downloaded and externally analysed, and so the use of descriptive statistics was not practical. Outcomes Rating Scale (ORS) summary report screens in relation to each client, social worker, team, or Oranga Tamariki overall, were available.


Face-to-face semi-structured interviews were held with 15 Oranga Tamariki social workers, supervisors and managers from across all four sites. In association with Oranga Tamariki, individuals were selected on the basis that they were best placed to help the evaluators address the key evaluation questions. Therefore, those who had made little use of PCOMS (or had stopped using it altogether) were under-represented.

The Oranga Tamariki Evidence Centre undertook an ethics review of the developed Participant Information Sheet and consent form. The consent form ensured that it was clear that the participation of social workers in the evaluation was entirely voluntary, and that what they said in their interviews would be treated confidentially, and any information that might individually identify them would not be included in the report without their agreement. They were also told that it could be possible for an informed person to deduce who had said a particular thing.

Oranga Tamariki also approved the developed interview guide. When approached, all 15 prospective participants agreed to take part and all made themselves available for interview on their respective agreed date and time. All interviewees signed a consent form.

Evaluation limitations

Some of the limitations identified in relation to this study are listed as follows:

  1. This was a small-scale evaluation, which limits somewhat the ability to generalise.
  2. The questionnaire includes some responses from Tauranga social workers; The Tauranga site subsequently ceased their involvement with the PCOMS trial. This reduced the overall sample size.
  3. Several of the social workers appear to have had little experience of actually using PCOMS at the time that they were surveyed, and this may have impacted upon their responses.
  4. While a large proportion of the social workers who were involved in the PCOMS trial were interviewed as part of the evaluation, as with all qualitative interviews the experiences and views of those interviewed are not necessarily representative of all others.

Key Results

The evaluation’s overall summary findings were that:

  • Almost all interviewed social workers participating in the trial reported struggling with PCOMS. For most, any early enthusiasm waned.
  • PCOMS usage appeared to be well below expected levels. Almost no-one on the trial consistently used PCOMS as an engagement, measurement, and feedback tool as intended.
  • For intake and assessment social workers in particular, PCOMS was deemed by them to be inappropriate for use with parents; in part because they were focussed on the immediate safety of the child.
  • However, some staff did find PCOMS valuable as an engagement tool with children and young people, and a reasonably large proportion of young people reported liking it.
  • The trial generated little evidence to suggest that adopting PCOMS across Oranga Tamariki operations would meaningfully achieve any of the outcomes that some of the randomised-controlled trials have found in other settings.
  • There may still be value in using PCOMS as part of a broader practice development initiative on working directly with children and young people.
Page last modified: 10 Nov 2023