Effectiveness and meaningful use of paediatric surgical safety checklists and their implementation strategies: a systematic review with narrative synthesis

Abstract

Objective

To examine the effectiveness and meaningful use of paediatric surgical safety checklists (SSCs) and their implementation strategies through a systematic review with narrative synthesis.

Summary background data

Since the launch of the WHO SSC, checklists have been integrated into surgical systems worldwide. Information is sparse on how SSCs have been integrated into the paediatric surgical environment.

Methods

A broad search strategy was created using Pubmed, Embase, CINAHL, Cochrane Central, Web of Science, Science Citation Index and Conference Proceedings Citation Index. Abstracts and full texts were screened independently, in duplicate for inclusion. Extracted study characteristic and outcomes generated themes explored through subgroup analyses and idea webbing.

Results

1826 of 1921 studies were excluded after title and abstract review (kappa 0.77) and 47 after full-text review (kappa 0.86). 20 studies were of sufficient quality for narrative synthesis. Clinical outcomes were not affected by SSC introduction in studies without implementation strategies. A comprehensive SSC implementation strategy in developing countries demonstrated improved outcomes in high-risk surgeries. Narrative synthesis suggests that meaningful compliance is inconsistently measured and rarely achieved. Strategies involving feedback improved compliance. Stakeholder-developed implementation strategies, including team-based education, achieved greater acceptance. Three studies suggest that parental involvement in the SSC is valued by parents, nurses and physicians and may improve patient safety.

Conclusions

A SSC implementation strategy focused on paediatric patients and their families can achieve high acceptability and good compliance. SSCs’ role in improving measures of paediatric surgical outcome is not well established, but they may be effective when used within a comprehensive implementation strategy especially for high-risk patients in low-resource settings.

Strengths and limitations of this study.

• There are few studies and no randomised controlled trials.

• There was high variability in study quality, study designs and study populations.

• There is insufficient support to make unambiguous and evidence-based recommendations regarding checklist implementation.

• There are no other studies that quantitatively and qualitatively assess the critical factors for successful implementation of the surgical safety checklist (SSC) in the paediatric surgery population.

• This systematic review highlights the role that parents play in a paediatric SSC.

Introduction

Checklists have been used in aviation, nuclear power and construction to manage complex tasks associated with the risk of significant harm.¹ Medicine is a similar high-stakes industry that has adopted the checklist. In 2007, a group of surgeons, anaesthesiologists and public health advocates working with WHO created a checklist to improve adherence to practices critical for safe surgery. The checklist was implemented in eight pilot hospitals internationally and reduced mortality at these sites significantly.² Since its development, the surgical safety checklist (SSC) has been integrated into surgical systems worldwide.^3–6 As of 2011, the WHO SSC had been adopted by more than 3900 hospitals in 122 countries.⁷

Paediatric surgical systems internationally have integrated SSCs into policy and accreditation standards.3 5 6 8 The SSC was developed primarily for adult patients, and the initial trial and most subsequent large-scale trials have been performed in adults.^{9 10} Evaluating the impact of this quality improvement tool in paediatric surgery requires a shift in perspective. The vast majority of children undergoing surgery are healthy and experience low mortality rates when compared with their adult counterparts.^{11 12} Nevertheless, children still suffer from preventable surgical complications.¹³ What constitutes an acceptable complication rate is influenced by our knowledge of the rates accompanying adult surgery. Lower rates of complications in paediatric patients should not be reassuring.

Paediatric patients have unique surgical risks. Physiologic challenges include the transitional circulation of the neonate and their high propensity to suffer from fluid losses and hypothermia during surgery. A child’s airway anatomy is different than that of adults. They have smaller blood volumes, immature immune systems and many other physiologic differences that challenge the surgical team.

In addition to the patients’ physiologic differences, there are complex social issues that impact the structure and function of paediatric surgical systems. Communication strategies have evolved to anticipate a short preoperative period accompanied by significant stress when the patient enters the operating room prior to induction.¹³ The patient often has minimal understanding of the nature of their operation. Parents are called on to serve as advocates and decision-makers. As such, they have both a responsibility and an expectation to take an active role in the care of their paediatric charges.

Information is sparse on how paediatric checklists are used, how they work and what they have achieved. These concerns are acknowledged by the Paediatric Surgical Chiefs of Canada (PSCC) who published their consensus opinion that a paediatric SSC is important for patient safety but also expressed their concern that information on the implementation and impact of paediatric SSCs is limited.⁸

The aim of our study is to investigate the effectiveness and meaningful use of paediatric SSCs and their implementation strategies within a systematic review of literature.

Methods

We have performed a comprehensive systematic review to synthesise published studies related to use of surgical checklists in children. Through this review, we collate evidence regarding the multiple measures of effectiveness of paediatric SSCs, to describe the strategies and attitudes related to implementation of paediatric SSCs, to evaluate the risk of bias in the evidence used to evaluate SSC use and to identify knowledge gaps in regard to the content or implementation of the SSC in children.

This systematic review follows Preferred Reporting Items for Systematic Reviews and Meta-Analyses and the Meta-analysis of Observational Studies in Epidemiology standards.^{14 15} Clinical outcomes, process measures and attitudes are explored, and elements of implementation are synthesised using a narrative synthesis approach, with exploration of themes and contents through subgroup analyses. A priori, we established clinical outcomes, measures of compliance and assessment of barriers and facilitators as measures of interest. However, we also allowed for the identification of additional measures through the literature search.

Our review presents a comprehensive synthesis of multiple study designs and objectives. We framed our question using the Population, Intervention, Controls, Outcome, Setting/Study (PICOS) type format. The population focus is paediatric patients, parents, caregivers and paediatric health systems. For the purposes of our work, we defined paediatric patients as those less than 18 years of age. The intervention is performance of a perioperative checklist defined as a list that exists in a physical and/or electronic form used prior to an operation to ensure patient safety. We did not specify how the checklist was used. Controls were not required for inclusion. Any study involving an interventional procedure undertaken with a general anaesthetic was considered. All study types except case reports and non-original research were considered. Studies had to measure at least one outcome related to the checklist including clinical outcomes, compliance, attitudes and process measure outcomes. Elements of implementation were examined in included studies. The analysis takes the form of a narrative synthesis, exploring themes and contents through subgroup analyses. Idea webbing is used to further elucidate the components of implementation strategy that were successful and those that failed.

Search strategy

A broad search of electronic databases including Pubmed, Embase, CINAHL, Cochrane Central, Web of Science, Science Citation Index Expanded and Conference Proceedings Citation Index-Science was performed in partnership with a research librarian (PB) on 23 March 2016 (see online supplementary appendix 1).

Inclusion and exclusion criteria were based on the PICOS elements described above. There was no language or date restriction. Data specific to paediatric patients needed to be available for separate analysis. Inclusion and exclusion criteria were agreed on a priori through team consensus. Non-English articles were translated by native speakers. Authors were contacted for missing data.

Titles, abstracts and full texts were reviewed independently in duplicate (MB and JL) to generate the final list of articles for data abstraction. Discrepancies were resolved through consensus. Inter-rater agreement was measured. Citation searching was performed, searching both the references and citations of included papers. Conferences identified through the Conference Preceding Citation Index were screened in duplicate to identify relevant abstracts.

The Data Extraction tool was generated through an iterative process. A pilot extraction created by the senior investigator (MB) was reviewed and modified by the research team (see online supplementary appendix 2). Data extraction was performed independently by three authors (MB, JL, SL) to ensure complete collection of relevant information. During the synthesis phase, additional information was reabstracted to develop key themes.

Quality assessment

The quality of each study was assessed using risk-of-bias tools appropriate to study design including pretudies/poststudies of compliance, cohort and cross-sectional studies as well as surveys. We developed this tool using the National Institutes of Health, Newcastle-Ottawa and CASP quality assessment tools (see online supplementary appendix 3). Using the tool as a guide, each study was given a rank of ‘good’, ‘fair’ or ‘poor’ by two reviewers. When two separate study elements were present (eg, clinical outcomes and compliance), both elements were evaluated separately. Overall quality assessment was obtained through discussion of ranking and final consensus of all three reviewers.

We have summarised the risks of bias across studies describing limitations of methods, the quality and generalisability of the evidence as well as discrepant findings. Subgroup analysis was restricted to studies rated as ‘good’ or ‘fair’. Implementation information was abstracted if one of the subgroup analyses in a study was ‘fair’ or ‘good’.

Synthesis

A narrative synthesis was conducted as described by the Economic and Social Research Counsel Methods Programme¹⁶ using subgroup analyses in the form of tables and narrative description. Subgroup themes related to outcomes and meaningful checklist use were identified in an iterative fashion through examination of extracted data. Implementation strategies were explored in a descriptive fashion to identify the elements used within different strategies. A further subgroup analysis was used to describe educational components of implementation.

We examined implementation strategies used across studies and the outcomes associated with various approaches. Relationships were explored visually using idea webbing to describe conceptual relationships between implementation strategies and outcomes.¹⁶

Results

The electronic search identified 1921 citations. After screening, 84 papers were identified for full-text review with a Cohen’s kappa of 0.77. After full-text review, 37 articles were included with a kappa of 0.86. Disagreements were resolved through discussion between the two raters with a final decision by a third rater in situations of persistent disagreement. Thirteen studies were removed after data extraction and contacting the authors for data. Two additional papers were identified through citation searching. Twenty-six papers were included in the final systematic review (figure 1). The majority of studies were cohort studies with and without controls and cross-sectional studies. Ten studies had a predesign and postdesign related to implementation of the checklist or a strategy to improve checklist use (online supplementary table 1 describes included studies). Outcomes assessed within selected studies included attitudes, barriers/facilitators and effectiveness in terms of clinical and process measure outcomes.

Figure 1.

PRISMA diagram. PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

Study quality

Study quality ranged from ‘poor’ to ‘good’. Six papers were judged as ‘poor’ for reasons including insufficient information provided about study design or an insufficient number of paediatric patients.^17–19 ‘Good’ studies shared the following traits: strong survey quality, robust definitions of compliance and reliable measurements and robust definitions of intervention and reliable measurements.^{20 21} ‘Fair’ studies were those that had sufficient numbers of patients and sufficient information about study design but had minor flaws in design such as the use of survey tools that were not validated or the use of compliance measures prone to bias. For studies with two different elements evaluated (eg, compliance and survey), both elements were separately evaluated for quality and each element was considered for inclusion separately (online supplementary table 1 provides quality ranking scores)

Parental involvement in the checklist

Four studies explored the unique role of patients and parents in the performance of the SSC.8 19 22 23 Pires et al’s study involved a SSC designed specifically for patients and parents,²³ while Corbally and Tierney, Skarsgard and Avansino et al explored the role of parents within a standard SSC.^{8 19 22} Pires et al reported that parents believed their involvement in a SSC improved patient safety and reduced their child’s anxiety.²³ Parents in Corbally and Tierney’s study believed their participation improved patient safety and reassured them that the appropriate procedure would be performed. All parents felt that the parent role should be mandatory.²² Surgical team members surveyed in Corbally and Tierney’s and Avansino et al’s study agreed that parental involvement in the SSC improves patient safety.^{19 22} Skarsgard’s survey of Canadian chiefs of surgery indicated that these surgical leaders were more divided on their opinion of the role of parents with 6 of 12 surgical chiefs indicating that parental presence added little or no value.⁸

Study of checklist use and clinical outcomes

The effectiveness of checklists at improving clinical outcomes was measured in three studies. These studies were population based and examined rates of morbidity and mortality before and after institution of an SSC. Two studies (Urbach et al and O’Leary et al) were performed in a similar population over a similar time frame leading to the likelihood of duplicate data.^{11 21} Urbach et al’s study demonstrated no change in mortality or infection rates after checklist introduction.¹¹ Complication rates in both studies remained unaffected by checklist introduction.²¹ In addition, Urbach et al found that the checklist made no difference in length of stay or reoperation and O’Leary et al found that the checklist did not change the rate of emergency department visits or unplanned trips back to the operating room.^{11 21} A very different study by Jenkins et al demonstrated a dramatic decrease in mortality and morbidity in paediatric patients undergoing congenital heart surgery in a low-resource setting after implementation of a broad surgical safety implementation strategy including a checklist. In Jenkins et al’s study, the odds of mortality 1 year after checklist introduction was 0.71 of the baseline odds and it remained low at 0.76 in the second year. The decreased odds of infection were 0.65 that at baseline in the first year and 0.53 in the second year of the study.²⁰ Urbach et al’s and O’Leary et al’s studies were retrospective, involving primarily low-risk surgeries with no standardised implementation process, while Jenkins et al’s prospective study examined high-risk surgeries in developing countries and employed a comprehensive implementation strategy, (table 1).

Table 1.

Impact of checklist introduction on mortality outcomes

Study	Study number	Study context	Study description	Preintervention cohort	Postintervention cohort	Mortality change after checklist introduction	Morbidity change after checklist introduction
Jenkins et al20	15 049 operations	Surgery for congenital heart disease 17 developing nations	Prospective cohort study Complex prospective Intervention including checklist	Reference year* 2010 or 2011	Year 1 2011 or 2012 Year 2 2012 only	Mortality† Year 1 OR 0.71 (95% CI 0.62 to 0.81) Year 2 OR 0.76 (95% CI 0.69 to 0.83)	Infection† Year 1 OR 0.65 (95% CI 0.58 to 0.73) Year 2 OR 0.53 (95% CI 0.48 to 0.58)
O’Leary et al21	28 772 operations	Common paediatric surgery excluding neonatal surgery, cardiac and transplant ON, Canada	Retrospective database (CIHI) analysis Prior and posthospital mandate of checklist	Year prior to checklist adoption September 2008 to October 2009	Year after all sites adopted checklist Sept 2010 to October 2011	Only one death recorded in the entire study (prechecklist)	Total complications† OR 1.01 (95% CI 0.90 to 1.14) Emergency visit† OR 1.06 (95% CI 0.93 to 1.21) Unplanned OR† OR 0.88 (95% CI 0.55 to 1.39)
Urbach et al11	15 495 operations	All surgery ON, Canada	Retrospective database (CIHI) analysis Prior and posthospital mandate of checklist	3-month period of time 3 months prior to checklist adoption (site specific)	3-month period of time 3 months after checklist adoption (site specific)	Mortality† OR 1.55 (95% CI 0.50 to 4.82)‡	Total complications† OR 1.07 (95% CI 0.84 to 1.36) RR length of stay† 1.01 (95% CI 0.98 to 1.04) Readmission within 30 days† OR 1.12 (95% CI 0.92 to 1.37)

*Enrolment was staggered; reference year was chosen as the first year prior to checklist introduction.

†Risk adjusted, reference population prechecklist cohort.

‡Actual OR and CI obtained from study authors. If Entrolment was staggered, reference year was chosen as the first year prior to checklist introduction.

CIHI, Canadian Institute for Health Information, RR, relative risk.

Compliance with checklists

Nine studies of ‘good’ or ‘fair’ quality evaluated compliance. The majority of studies (6/9) studied an intervention to improve compliance. Some studies (5/9) had baseline comparison populations either before implementation or immediately after implementation, while others had no comparison populations or used data from piloting the checklist prior to full implementation.

Improved compliance was noted in all six studies involving an intervention to improve compliance. The authors of these studies reported that feedback, education and incorporating stakeholder solutions improved compliance. Two studies were notable for poor compliance. Levy et al’s study from 2012 noted little correlation between the excellent compliance reported by the hospital and the actual compliance with the SSC measured by a study auditor.²⁴ Only 4 of 142 audits of SSCs in this study demonstrated completion of more than 50% of SSC elements. A survey of the surgical team members at Levy et al’s institution identified possible causes for poor compliance including a lack of understanding of both the roles and responsibilities of team members and the purpose and the timing of SSC. In addition, the authors ascribe poor compliance to the perceived failure of an adult checklist to address the needs of the paediatric population.²⁴ In Ride et al’s study, agitation of the paediatric patient in the operating room was cited as the primary reason for skipping the checklist with coordinating the care team presenting an additional challenge (table 2).²⁵

Table 2.

Studies of checklist compliance

Article	Study description	Comparison population	Method of measuring compliance	Compliance measure	Checklist compliance	Comments
Avansino et al19	Study of intervention to improve compliance Compliance measured over 12 months after checklist introduction Monthly feedback of compliance provided to surgical team	Baseline at introduction of checklist	Audit Submitted paper checklists	Compliance defined as any aspect of checklist completed	Improved compliance Baseline compliance: Overall 88% Weekends 64% Weekdays 90% 12-month compliance: Overall 97% Weekends 100% Weekdays 94%	Feedback may increase compliance Lax definition of compliance Steady increase in compliance over 12 months
Gottumukkala et al31	Study of intervention to improve compliance All cases videotaped Compliance measured over 3 years through review of videos Feedback provided to team with ‘corrective action’ to improve compliance	Baseline when recording system in place and evaluation started	Audit 13% of videos of checklists performed	Completion of each element of the checklist assessed by multiple raters	Improved compliance (time series analysis) (crude values not available) Decreased variability in performance over time	Feedback and incorporating stakeholder solutions to compliance failures may improve compliance Possible selection bias and generalisability may be limited to interventional radiology but reliable measure of compliance Steady increase in compliance
Khoshbin et al26	Compliance measured 1 year after introduction of a surgical ‘time out’ and 2 years after introduction of a preoperative ‘huddle’	None	Audit Direct observation of ‘huddles’ and ‘time outs’	Compliance defined as any part of ‘huddle’ or ‘time out’ completed Completion measured for 4 elements of the huddle and 9 elements of the ‘time out’	‘Huddles’: Compliance 64% Mean completion 3.2/4 ‘Time out’: Compliance 99.1% Mean completion 6.2/9	Meaningful completion of checklist can be marred by the perception of the checklist as a ‘task’ Unblinded audit
Levy et al24	Compliance measured over 7-week period	None	Audit Direct observation of cases	Compliance defined as completion of all preincision components of the surgical checklist Completion measured for the 13 elements in the checklist	Hospital-reported compliance 100% True compliance (full checklist completed) 0% Time out compliance 97% Completion The average number of checklist items performed 4/13 4/142 had more than 50% of elements completed	Hospital-recorded compliance may not reflect implementation fidelity Meaningful completion of checklist can be marred by the perception of the checklist as a ‘task’ Factors commonly completed were patient identification and procedure Factors least completed were sterility confirmation, essential imaging, team member concerns and anticipated blood loss
Norton27	Study of intervention to improve compliance Universal Protocol introduced Education provided before implementation and feedback on effective use provided during audit	Baseline at introduction of checklist	Audit Direct observation and documentation by surgical team	Compliance defined as time out and site verification completion	Improved compliance Site initialled: Observed 97%–100%. (increased from 88% first month) Documentation 97%–100% Time out compliance: Observed 97%–100% Documentation 97%–100%	Education and feedback may maintain compliance Compliance does not always reflect meaningful completion Limited to Universal Protocol
Norton and Rangel29	Study of intervention to improve compliance Two separate pilots used to develop and modify intervention Ongoing education during implementation	Data from piloting of checklist before implementation	Audit Direct observation of cases	Compliance defined as ‘usage of the checklist’	Improved compliance First pilot, overall compliance 80%–90% 6 months after introduction Sign in 90%–100% Sign out 95%–100%	Addressing barriers identified through piloting the checklist and education may improve compliance The Hawthorne effect may lead to improved compliance Ongoing improvement in compliance
Ride et al25	Compliance measured for 48 cases	None	Audit Direct observation of cases	Measures include team member attendance Compliance defined as verbal verification of all checklist items Documentation of checklist completion Correct use of checklist	Attendance for sign-in surgeon 10% Anaesthesiologist 95% Verbal compliance: Sign ins 95% Time outs 69% Sign outs 44% Documentation: Sign ins 79% Time outs 71% Sign outs 31% Correct use: Sign in and time out 46% Sign out 23%	The sign out is poorly done and correct use of the checklist is not common Correct use is not defined in this study Challenges to compliance reported by authors included patient agitation and difficulty coordinating members of the care team
Montgomery et al30	Study of intervention to improve compliance Intervention to improve compliance using feedback provided to teams	Baseline prior to intervention	Audit Direct observation of surgical pause	Compliance defined as performance of the surgical pause Completion measured for the 4 elements of the surgical pause	Compliance improved Preintervention: Compliance 100% Completion of all 4 steps 51% Postintervention: Compliance 100% Completion of all 4 steps 77%	Feedback may improve completion Improvement of 4 elements: Introduction 61%–87% Identity and procedure 90%–100% Checklist performed 98%–100% Concerns voiced 71%–100%
Putnam et al28	Study of intervention to improve compliance Staged strategy to improve compliance First stage: safety council and modification of checklist Second stage: workshops stakeholder audits/feedback	Baseline prior to interventions	Audit Direct observation of surgical pause	Compliance defined as completion of entire checklist Completion measured for the 14 checkpoint elements.	Compliance improved Baseline: Compliance 0% Completion 30% After first stage: Compliance 19% Completion 76% After second stage: Compliance 61% Completion 96%	Feedback, education and incorporating stakeholder solutions to compliance failures may improve compliance

Implementation/operationalisation characteristics and educational strategies

Eleven studies described the implementation and operationalisation of SSCs. Baseline data were collected in all studies. The majority of checklists were modified from the WHO SSC. Others were developed before the WHO SSC was published.^{26 27} Modifications of the WHO SSCs were generally based on feedback from pilot data or feedback from stakeholders. Stakeholder involvement was noted in eight studies. Putnam et al’s 2014 study noted that stakeholder focus on paediatric-specific modifications to the SSC improved compliance beyond that obtained after a more general modification of the SSC for the parent institute (table 3).²⁸

Table 3.

Implementation/operationalisation characteristics

Study	Checklist creation process (adoption or modification of existing checklist)	Expected outcomes of implementation process	Key attributes of checklist implementation process					Outcomes
			Stakeholder engagement	Targeted education	Preimplementation ‘change management’	Use of ‘pilot’	Use of CQI methods (eg, PDSA cycles or simple feedback) to improve implementation	Outcomes	Documented sustainability
Avansino et al19	Use of baseline data and paediatric needs to revise WHO SSC	Compliance with checklist	Unclear	Yes	Education marketing and notifications: Newsletters, bulletins, posters in operating room Use of feedback	Yes	Compliance rates posted monthly during pilot	Improved compliance	Sustained compliance over 1 year
Gottumukkala et al31	Institution-modified WHO SSC and Universal Protocol	Adherence to Good Practice	Yes	None	No educational strategy outlined Use of feedback	No	Monthly audit of videotaped procedures Frontline teams use feedback to target lapses and create strategies to improve performance (PDSA)*	Improved compliance with process measures and decreased variability	Continual improvement sustained across multiple years
Jenkins et al20	Site-specific stakeholder involvement to modify WHO SSC	Decreased mortality Infection reduction Improved perioperative practice Improved teamwork and communication	Yes	Yes	Comprehensive education strategy Global aims: central organisation, support from NGOs Local implementation: Local checklist modification and implementation Translated materials Standardised ‘Practice Bundles’ and communication scripts Use of feedback	No	Annual benchmarking	Decreased mortality and decreased infections	Sustained benefits in mortality and morbidity reduction at 2 years
Khoshbin et al26	Institution developed Non-WHO SSC	Compliance with ‘huddles”and ‘time-outs’	Unclear	None	No educational strategy outlined Marketing and notifications: Announcements at departmental meetings Posters in operating rooms	No	No	Compliance excellent with time outs and good with huddles Nurses view SSC as important for safety Physicians view SSC as a tool for increasing efficiency	Sustainability not assessed
Levy et al24	Site-specific stakeholder involvement to modify WHO SSC	Compliance with checklist	Yes	Yes	Limited educational strategy Notifications: Posters in operating rooms	No	No	Poor compliance Poor implementation fidelity	Sustainability not assessed
Montgomery et al30	WHO ‘surgical pause’ Unclear if SSC modified	Compliance with checklist	Yes	None	No educational strategy outlined Use of feedback	Yes	Feedback presented at surgical team department meetings Feedback used to create posters to improve compliance	Improved Compliance	Sustainability not assessed
Norton27	Universal-Protocol-based checklist created by local champion and stakeholders	Compliance with checklist	Yes	Yes	Comprehensive education strategy Marketing and notifications: Marketing team CEO letter to practitioners Logo, flyers, posters, wearable button Stickers on charts Mandated education	No	Results posted	Excellent compliance throughout implementation	Sustained compliance over 14 months
Norton and Rangel29	Site-specific stakeholder involvement to modify WHO SSC	Compliance with checklist Process measure compliance Teamwork and communication measures	Yes	Yes	Education Marketing and notifications: Marketing team CEO letter to practitioners Posters in operating rooms Flyer in operating room sterile pack Screen savers on computers with SSC reminder, newsletter Mandated checklist use Local leadership Use of feedback	Yes	Feedback to staff monthly Feedback used to revise implementation strategy	Improved compliance	Continual improvement in compliance over 7 months
Putnam et al 201428	Use of baseline data and site-specific stakeholder involvement to modify WHO SSC	Compliance with checklist	Yes	Yes	Comprehensive education strategy Marketing and notifications: Distribution of web-based media Posters in operating room Local leadership Use of feedback	Yes	Feedback to staff Feedback from stakeholders and pilot testing used to revise implementation strategy	Improved compliance	None
Putnam et al 201528	Institutional checklist based on WHO SSC modified by stakeholders	Compliance with appropriate antibiotic administration	Yes	None	No educational strategy outlined Marketing and notifications: Notices on appropriate antibiotic use placed on anaesthetic carts Emails sent to staff regarding guideline changes Computer order entry system to facilitate correct antibiotic use	Yes	Unclear	No improvement in compliance with appropriate antibiotic use	None
Wyrick et al 2015 39	Stakeholder (surgeon) involvement to modify WHO SSC	Compliance with accurate SWC	Yes	Yes	Education Marketing and notifications: Bulletin boards near operating theatres	Yes	No	Improved accuracy of documented SWC	Sustainability not assessed

*Gottumukkala et al’s study did not specifically describe their implementation strategy as using the PDSA framework. However, given the fact that the elicited feedback from those using the checklists, collected and analysed this feedback and then used the feedback for checklist improvement, we have categorised this within the PDSA column.

CEO, chief executive officer; CQI, continuous quality improvement; NGO, non-governmental oranisation; PDSA, Plan-Do-Study-Act; SSC, safe surgery checklist; SWC, surgical wound classification.

Checklists were implemented using a number of measures including marketing and raising awareness, institutional mandates, feedback as well as educational strategies. The use of feedback to improve compliance was described in a number of studies. Piloting was performed to provide feedback on checklist performance in 6 of the 11 studies. Norton and Rangel’s 2010 study addressed barriers noted during the piloting of the checklist to improve compliance.²⁹ Similar success using feedback to improve compliance was described by Norton and Rangel²⁹ and Montgomery et al.³⁰ Identifying and addressing barriers based on feedback was particularly effective at improving compliance in the time series study by Gottumukkala et al.³¹ This study involved videotaping all checklist performances and regularly auditing a sample of these on an ongoing basis over years with continuous adaptation of the implementation process. The integration of these audits into regular practice resulted in an effective strategy of continuous quality improvement.

Educational strategies within implementation models ranged from limited and focused educational interventions to expansive, sustained programme. Educational strategies were described in detail in seven studies (online supplementary table 2). Studies with a strong implementation plan demonstrated a positive impact of the checklist, while studies with limited implementation strategies demonstrated poor compliance, unchanged clinical outcomes and negative attitudes regarding institutional safety. Comprehensive strategies for patient safety with an integrated checklist appear particularly effective but the contribution of the checklist itself becomes difficult to discern. Conceptual but not necessarily causative relationships between implementation strategies and outcomes are explored through ideas webbing (figure 2).

Figure 2.

Exploration of implementation approaches and outcomes using idea webbing. SSC, safe surgery checklist.

Attitudes related to the surgical checklist

Eight studies explored attitudes related to checklist use and the impact of the SSC on the culture of surgical safety. These results were generally gleaned from surveys. Several positive attitudes related to the perceived value of the checklist were noted, but there were often conflicting views presented within studies. For example, Avansino et al’s study and Norton et al’s study from 2016 showed that there was strong agreement that the SSC is important for patient safety; however, only 59% of practitioners in Avansino et al’s study and 36% in Norton et al’s study felt that the SSC had actually identified patient safety issues.^{19 32} Positive attitudes to the checklist are highlighted in Norton’s publications from 2010 and 2016. Specifically, these studies conclude that nurses and physicians generally believe that checklists: improve patient safety, improve efficiency, prevent communication errors and work well in high-volume centres with significant distractions.^{29 32} Although the attitude towards surgical safety and the role of the SSC was generally positive, attitudes of nurses differed from physicians in some consistent ways. Nurses were more likely to perceive problems with communication and attitudes towards safety in their institution and more likely to view the checklist as playing an important role in improving patient safety (online supplementary table 3).^{19 26 32}

Discussion

This study is the first synthesis of peer-reviewed, published data concerning the impact of SSCs on paediatric patients. It is not a traditional systematic review in that it does not focus on a single area of inquiry. Our study tackles the scanty literature on paediatric SSCs by combining the broader research aims of a coping review with synthesis methods of a systematic review.

Since its creation, the SSC has been integrated into paediatric surgical systems alongside adult surgical systems worldwide.^3–6 Studies in adults have demonstrated that SSCs may be associated with reductions in postoperative morbidity when part of a strong implementation strategy that is tailored to the patient population.^{10 33} Paediatric patients and paediatric surgical systems have unique needs. Data on checklist implementation in the paediatric population remain limited. Understanding the utility, the barriers and facilitators of paediatric checklist use is crucial when developing policies and implementation strategies. Our review of the literature regarding SSCs in the paediatric population reveals several key themes regarding compliance, implementation and impact on outcomes.

More than half of the studies in our review explored compliance.⁸ We identified challenges to compliance specific to the paediatric surgical setting. Many studies identified difficulty in performing a preinduction check in the operating room with an agitated child^{34 35} and the lack of buy-in without stakeholder-generated, paediatric-specific adaptations to the SSC.^{24 26} The definition of compliance in studies is inconsistent. Institutional compliance data often overestimate checklist use.²⁴ If compliance is defined as completion of any aspect of the checklist, attaining high levels of compliance is not difficult.¹⁹ However, true compliance with the entire checklist is rarely obtained.²⁴ Unblinded audits of compliance may be influenced by the Hawthorne effect.^{26 33} A further difficulty, and one that is much harder to evaluate, is determining meaningful compliance. An informal audit in one study identified that meaningful use did not always accompany good compliance.²⁷ The concerns with the definition and measures of compliance were identified in a survey of PSCC who reflected on the dangers of using compliance as an administrative tool and the importance of developing strategies for meaningful use.⁸

Strategies to improve meaningful compliance in adult studies include stakeholder involvement in checklist development and SSC as well as implementation tailoring to meet the needs of the patient and the surgical team.^{36 37} Similar findings were apparent in our review of paediatric SSC studies. Adapting the implementation strategy to the culture and workflow of the paediatric surgical service, as described by Putnam et al, resulted in improved compliance.²⁸ Feedback plays an important role in achieving compliance. A single episode of feedback can improve compliance.³⁰ Frequent feedback over a longer duration demonstrates continued improvement of compliance to high levels sustained for the duration of the study.^{19 29 31} The incorporation of feedback into an ongoing strategy of continuous quality improvement is demonstrated by Gottumukkala et al’s study where institutional practice involved continuous auditing and iterative development of improvement strategies over multiple years. This entrenched strategy was associated with high compliance and minimal variability in performance.³¹

As in the adult surgery, meaningful checklist implementation enhances successful teamwork and communication in paediatric surgery. Some aspects, such as parental involvement, are unique to the paediatric population. Including the parent as an active participant in the SSC offers an opportunity for improved patient safety and patient/family satisfaction in the paediatric setting. Surveys of surgical team members directly involved in strategies to improve parental participation reflected a positive view of the parental role.^{19 22} In these studies, parental satisfaction was high and both parents and surgical team members believed that parental involvement improved patient safety.^{22 23} Corbally and Tierney points to the crucial role of the parent in situations where the members of the healthcare team change. Children are often removed from the process of consent and even mature children may not feel comfortable challenging a surgical plan. A parent, however, frequently assumes the role of protector and advocate and is more likely to challenge a different or unclear plan. Unlike other surgical team members, the parent of the patient has been consistently involved in the care of the child, has not suffered checklist fatigue and is not likely to regard participation in a surgical checklist as a routine activity. The parent assumes a unique role as a patient advocate in the preoperative setting and the development of a role for parental participation within a surgical checklist respects the philosophy of family-centred care and may positively influence the culture of patient safety. Skarsgard’s study demonstrates that the views of the surgical team leadership on parental involvement in the SSC may be divided, and these divergent views might act as a barrier to developing a strategy of parental participation.⁸ Though there is still a paucity of literature exploring the role of parents within a checklist, current literature does highlight a parent’s importance in serving as patient advocates. Moreover, the involvement of parents in a SSC further promotes the principles of patient and family-centred care.

The impact of SSC use on patient outcomes has been established in the adult literature. A few studies have started to shed light on the impact of the SSC on paediatric patient outcomes. Jenkins et al’s study demonstrates that a comprehensive strategy aimed at improving the quality and safety of surgery can improve patient outcomes.²⁰ In this study, the SSC was just one of many interventions and the value of the tool cannot be discerned separately from the complex strategy of surgical safety in which it is implemented. The comprehensive undertaking over the 2 years of the study would require a substantial commitment by health systems hoping to replicate these findings. Although the gains possible within developing nations may not be achievable in other settings, similar high-risk vulnerable populations exist in high-income countries. O’Leary et al’s and Urbach et al’s studies remind us that a strategy that is primarily focused on mandating checklist use to improve outcomes in paediatric surgical patients is unlikely to provide much benefit. This is particularly true when dealing with low-risk patients in high-income settings.^{11 21} The disappointing findings of these database studies reflect those of adult studies that examine the impact of mandatory checklist use without a clearly defined implementation strategy. The checklist must be regarded as a tool within an integrated strategy for safety improvement with a strong focus on meaningful use. Our findings suggest that engagement of paediatric surgical stakeholders in the process of checklist adoption along with education of team members and parental participation in the SSC may improve team communication and meaningful checklist use. Evidence from our review suggests that SSC compliance may be improved using feedback. A comprehensive strategy such as that undertaken by Jenkins et al could improve clinical outcomes for children even in high-income countries.

These results emphasise that the checklist may not achieve results when mandated as a stand-alone tick-box exercise, but it may achieve great success as part of a comprehensive quality improvement strategy.

Our systematic review has some notable limitations and strengths. There are few studies and no randomised controlled trials that examine the performance of checklists in children. Only three studies specifically measured checklist impact on clinical outcomes. Reporting bias may have resulted in suppression of negative results. Some WHO SSC published studies that include paediatric patients within studies of patients of all ages might not have been identified through our search strategy. Changes in outcomes seen in studies could be secondary to secular trends and compliance measures could be influenced by the Hawthorne effect. There is high variability in study quality, study designs and study populations. Thus, summarising and interpreting results is challenging. There is insufficient support to make unambiguous and evidence-based recommendations regarding checklist implementation.

The strengths of this review are also notable. This study is the first synthesis that explores paediatric SSCs. We provide a broad quantitative and qualitative assessment of the critical factors for successful compliance with and implementation of the SSC in the paediatric surgery population. Our study is unique in its ability to highlight key attitudes held by members of the paediatric surgical team towards the checklist, its use in paediatric patients and its role in contributing to the culture of safety. Additionally, this systematic review highlights the role that parents play in a paediatric SSC and how their involvement can be optimised to improve patient safety. The findings of this review provide an evidence base that can guide paediatric surgical teams looking to implement strategies that will improve the positive impact of checklists.

In conclusion, this review reinforces the findings from the adult literature that the SSC is not a stand-alone tool. Currently, the SSC often fails to respond to the unique needs of paediatric patients and the systems that care for them. Understanding contextual influences and addressing implementation fidelity are crucial in achieving meaningful outcomes.^{33 38} The results of this knowledge synthesis echo some of the key findings of the realist review of SSC implementation published by Gillespie and Marshall in 2015.³⁷ In SSC implementation, similar strategies may achieve very different outcomes in different environments. SSCs can form a key part of initiatives to improve surgical safety for children. SSCs, however, require adaptation for the paediatric patient and environment both to improve patient safety and encourage acceptance. Parents may assume a unique role in the paediatric SSCs, which may improve the patient and family experience and ultimately improve surgical safety.