This is an executive summary of the final evaluation report prepared by the New Zealand Institute for Economic Research (NZIER).
The substantive evaluation of the CDRP was agreed between the Foundation and the Ministry of Research Science and Technology (MoRST) as part of the full transfer of the scheme to the Foundation for 2004/5. The terms of reference for an outcome evaluation were prepared by the Foundation in consultation with MoRST, and approved by the Foundation’s Science Committee (April 2005).
The aims of the outcome evaluation were to:
a. Assess the achievement of the CDRP objectives.
b. Assess the effectiveness of CDRP processes.
The evaluation sought information from Government departments and agencies on CDRP processes, outcomes and benefits attributable to completed CDRP projects.
A Steering Committee was convened for the project. The committee selected NZIER to complete the evaluation. The Steering Committee were also involved in finalising the evaluation instruments and the draft and final reports.
Purpose
The aims of the evaluation were to:
- assess the achievement of the CDRP objectives (a, b, and c in section 1.1 above); and
- assess the effectiveness of CDRP processes.
In particular, the objectives were to examine:
- CDRP processes in terms of their effectiveness, including quality of funding applications and standards of applications received;
- Uptake of CDRP research information, findings or evaluations by lead and supporting Government departments and agencies;
- Impact of the research on lead and supporting Government departments’ development and implementation of strategic policies and specific programmes, and outcomes;
- Cross-departmental strategies, policies and impacts;
- Achievements and effectiveness in ‘making a difference’ in lead and supporting Government departments and agencies;
- Impact on departments’ R&D capabilities and capability development, including investment in longer-term high quality research; and
- Departments’ linkages with other Government departments and research organisations (New Zealand and overseas) influenced through CDRP.
Methodology
In response to these aims and objectives, a 3-phase methodology (which closely followed that suggested in the evaluation request for proposals) was applied:
- Review of successful and unsuccessful applications: This involved testing for good practice in terms of the quality standards of selected CDRP applications. We reviewed all the final bid assessments for 2002/03 and 2003/04 (26 and 14 respectively). Application and assessment documents were supplied by MoRST for this work.
- Generic views on CDRP: Government departments and agencies were informed of the evaluation and invited to submit their comments on the scheme and the achievement of its objectives. This phase was intended to enable all departments to express their views, especially those that did not participate in the qualitative interviews in phase 3.
- Qualitative Interviews: The aim was to complete a set of structured interviews examining CDRP processes and outcomes. These interviews investigated quality, and outcomes, including pathways to policy development and implementation, cross-departmental interactions, benefits, constraints, and issues. The intention was to interview representatives of lead, collaborating and supporting departments / agencies involved in 10 selected CDRP projects.
In sampling projects for detailed examination, we were guided by the need to avoid projects that were: a) completed too long ago for there to remain institutional memory of the outputs and outcomes, and b) started too recently for the full effects to emerge. We, therefore, sample a set of projects that were awarded funding between the years 2000 and 2003, and that were due to have been completed before this evaluation got underway. Further details of the sample characteristics are included in section 3.