Review: The evaluation of a home-based paediatric nursing service: concept and design development using The Kirkpatrick Model

The authors of the reviewed paper have chosen to use Kirkpatrick’s model (2016) as the structure for evaluation of this programme. This is laudable; however, for all application of models of evaluation, it needs to be clear why this choice was made, and that the implementation of the evaluation is in fidelity with the model. In this instance, the authors have chosen to use the upgraded ‘New World’ Kirkpatrick model (2016). For the purposes of this review, an analysis of the application of the ‘new world’ Kirkpatrick model by Jones et al., 2018 is presented, so that others can judge the fulfilment of the new model’s potential.

Many scholars will be aware of the original Kirkpatrick’s model (1996) being widely used to evaluate education programmes for health professionals. However, as Jones et al. indicate, it has been criticised in the literature for lack of acknowledgement of the difficulties associated with evaluating behaviour (level 3) and reporting results (level 4). The development of the modified Kirkpatrick model has led the authors to describe the model further (Kirkpatrick and Kirkpatrick, 2007) and then develop the ‘New World’ Kirkpatrick model (Kirkpatrick and Kirkpatrick, 2016). The new model has clarified the importance of level 3 by acknowledging that behaviour change is the culmination of a range of factors including learning and experiences. Training may lead to changes in practice or improved clinical outcomes; however, other contributing indicators such as quality, cost, efficiency and compliance need to be acknowledged.

Moreau (2017), Frye and Hemmer (2012), Bates (2004) and Holton (1996) have addressed Kirkpatrick’s original model (1996) and its failure to address specific indicators, such as motivation and baseline knowledge of the health professionals being trained. In contrast, the ‘new world’ version (2016) accepts the context of the nature of the learner, the resources of the organisation, and that both of these can impact upon the success or failure of the programme. There were changes at each of Kirkpatrick’s four levels, particularly at the first three levels of reaction, learning and behaviour. For the first level (reaction), Kirkpatrick and Kirkpatrick (2016) were clearer that evaluation should be about engagement and relevance. For the second level (learning) that the authors indicated confidence to apply skills and commitment to do so impacted on learning. For (behaviour) level 3, required drivers such as reward and monitoring, to deliver behaviour change. The authors acknowldged there was ‘on the job learning and support’ in the form of mentorship available to support learning and change, as well as the sense of responsibility to change practice (Moreau, 2017).

The authors describe the development of the design of their evaluation framework using the Kirkpatrick model for a home-based paediatric nursing service. Table 1 summarises the evaluation methods in diminishing order, used by previous authors. Table 2 (Jones et al., 2018) shows the research methods used to answer the evaluation questions in each stage of this project using the Kirkpatrick model. As previously noted by the authors in the paper, levels 1 (reaction) and 2 (learning) can be more easily measured than the latter two levels. Through the research questions asked, the authors have responded to the ‘new world’ Kirkpatrick model for evaluation of the home-based paediatric nursing service. Level 3 (behaviour) is an integrated question that can be used to explore which other indicators may impact on any behaviour change. Although this project acknowledged the benefits of collecting data through observation, they recognised the difficulty in doing so. The other levels use quantitative methods to collect data; however, level 3 (behaviour) relies on qualitative data collection from nursing staff and their managers. There is no attempt to measure the nurses’ behaviour from the perspective of the families of the children. There are merits and drawbacks from utilising subjective data collection, especially as the nursing participants will know a positive evaluation could lead to further funding. Additionally, the clinical nurse specialists interviewed may perceive positive performance as a reflection of their capability. Methods for collecting data for level 4 seek both objective and subjective measures for assessing impact or changes in clinical outcomes. Acknowledgement of the potential for bias will be imperative when reporting the outcomes of this evaluation. The use of semi-structured interviews (level 3) and family satisfaction surveys to measure level 4 may create opportunities for respondent bias. In level 4, the authors have included incident data to offset this potential for introducing bias. Triangulation of data sources will enable an appropriate use of the ‘new world’ Kirkpatrick model providing the other indicators that emerge and discussion of the limitations are acknowledged.

Jones et al. have included data collection strategies for collecting level 3 and level 4 data within the ‘new world’ Kirkpatrick model (2016) and, as they mention, it is useful to provide a demonstration of a comprehensive use of the model to show others that it is possible. Complete use also affords robustness to the evaluation that a modified form does not. Report of the completed evaluation is awaited with interest.

Biography

Steven Campbell received his nursing degree from Manchester University in 1981 and his PhD from Northumbria University in 1996. He has been a Professor of Nursing for over 17 years, with honorary and substantive appointments at Northumbria University, Bournemouth University, the University of New England, the University of Tasmania, Griffith University and Southern Cross University. He is currently Professor of Clinical Redesign, Nursing, at the University of Tasmania, Australia.

Carey Mather has worked in the health sector for over 30 years in various capacities and settings including the acute, palliative, health promotion and community environments. From 2001, she was periodically employed by the University of Tasmania as a clinical teacher, tutor and lecturer. Carey currently works full time within the School of Health Sciences (Nursing and Midwifery) as a lecturer.