Transforming Team Performance Through Reimplementation of the Surgical Safety Checklist

This study assesses the ability of a comprehensive system of safety checklist reimplementation to change behavior, enhance safety culture, and improve outcomes for surgical patients.

Key Points

Question

Can a systematic approach to reimplementation of an underperforming surgical safety checklist improve team behavior and performance?

Findings

In this implementation-effectiveness hybrid study, reimplementation of the surgical safety checklist showed significant improvements in fidelity to the safety process. Team performance, as measured by the Oxford Non-Technical Skills system, also showed improvement after reimplementation.

Meaning

This approach to comprehensive reimplementation of the surgical safety checklist intervention may improve team behavior, safety culture, patient safety, and patient outcomes.

Abstract

Importance

Patient safety interventions, like the World Health Organization Surgical Safety Checklist, require effective implementation strategies to achieve meaningful results. Institutions with underperforming checklists require evidence-based guidance for reimplementing these practices to maximize their impact on patient safety.

Objective

To assess the ability of a comprehensive system of safety checklist reimplementation to change behavior, enhance safety culture, and improve outcomes for surgical patients.

Design, Setting, and Participants

This prospective type 2 hybrid implementation-effectiveness study took place at 2 large academic referral centers in Singapore. All operations performed at either hospital were eligible for observation. Surveys were distributed to all operating room staff.

Intervention

The study team developed a comprehensive surgical safety checklist reimplementation package based on the Exploration, Preparation, Implementation, Sustainment framework. Best practices from implementation science and human factors engineering were combined to redesign the checklist. The revised instrument was reimplemented in November 2021.

Main Outcomes and Measures

Implementation outcomes included penetration and fidelity. The primary effectiveness outcome was team performance, assessed by trained observers using the Oxford Non-Technical Skills (NOTECH) system before and after reimplementation. The Agency for Healthcare Research and Quality Hospital Survey on Patient Safety Culture was used to assess safety culture and observers tracked device-related interruptions (DRIs). Patient safety events, near-miss events, 30-day mortality, and serious complications were tracked for exploratory analyses.

Results

Observers captured 252 cases (161 baseline and 91 end point). Penetration of the checklist was excellent at both time points, but there were significant improvements in all measures of fidelity after reimplementation. Mean NOTECHS scores increased from 37.1 to 42.4 points (4.3 point adjusted increase; 95% CI, 2.9-5.7; P < .001). DRIs decreased by 86.5% (95% CI, −22.1% to −97.8%; P = .03). Significant improvements were noted in 9 of 12 composite areas on culture of safety surveys. Exploratory analyses suggested reductions in patient safety events, mortality, and serious complications.

Conclusions and Relevance

Comprehensive reimplementation of an established checklist intervention can meaningfully improve team behavior, safety culture, patient safety, and patient outcomes. Future efforts will expand the reach of this system by testing a structured guidebook coupled with light-touch implementation guidance in a variety of settings.

Introduction

Surgical complications account for 8% of all deaths worldwide, by some measures, the third leading cause of death.^1,2 Thirty percent of these deaths and nearly half of all postoperative complications are potentially preventable.^3,4 Technical errors are responsible for fewer than half of preventable deaths; nontechnical errors, such as breakdowns in communication and teamwork, are increasingly recognized as a major contributor to surgical morbidity and mortality.^3,5,6

The World Health Organization (WHO) Surgical Safety Checklist (SSC)⁷ was designed to prevent errors and adverse events by enhancing teamwork and communication among operating room (OR) teams and promoting a strong culture of safety. It includes 19 items divided into 3 pause points: a preinduction sign-in, a preincision time-out, and a postoperative sign-out. In a landmark 2009 study,⁸ the SSC was shown to reduce surgical complications by one-third and mortality by nearly half. Multiple subsequent studies yielded similar results and the SSC is now the standard of care for surgical safety.^9,10,11,12

Unfortunately, the mere presence of a checklist does not guarantee a patient safety benefit. Several large-scale implementation efforts failed to produce positive results, most notably a mandate-driven initiative in Ontario, Canada.^13,14 These contradictory outcomes highlight a crucial point: meaningful improvements in surgical safety require effective SSC implementation. Indeed, there is a clear relationship between implementation intensity and clinical effectiveness of the SSC.^{12,15,16,17,18,19}

The need for institutions to examine the performance of their checklists became increasingly clear as the SSC surpassed its 10th anniversary.²⁰ Evaluation of the SSC use may reveal early implementation failures, resulting in improper or inconsistent checklist use or sustainment failures, whereby staff support and enthusiasm have waned. The checklist itself should be assessed as accumulated revisions may degrade its effectiveness and contextual changes may necessitate modifications for it to remain relevant. Reevaluation also presents an opportunity to address gaps that have emerged since the development of the original checklist. For example, surgical device use—which accounts for nearly 25% of intraoperative errors—is not addressed in the original SSC.^21,22

Despite widespread recognition of the need for monitoring and periodic reevaluation in the implementation science and quality improvement communities, such activities require significant investments of time and resources. To date, to our knowledge, there is no evidence that SSC reimplementation provides meaningful safety benefits and there is limited guidance for centers interested in doing so. To address this gap, we designed and tested a comprehensive system of reimplementation that we hypothesized would measurably change behavior, enhance safety culture, and ultimately improve outcomes for surgical patients.

Methods

Study Design and Setting

We conducted a prospective multiphase study using a type 2 hybrid implementation-effectiveness design.²³ This design tests the effectiveness of both the clinical intervention and the implementation strategy used. The study protocol was registered with ClinicalTrials.gov (NCT05123495). In the first phase, we piloted the Device Briefing Tool, a communication instrument designed to enhance surgical device safety, details of which have been published previously.²⁴ In the second phase, we developed and tested a comprehensive approach to SSC reimplementation based on the Exploration, Preparation, Implementation, Sustainment framework.²⁵ We aimed to evaluate the ability of our approach to drive meaningful use of the SSC and assess the effect of reimplementation on team performance and patient safety.

Ethical approval was obtained from the Harvard T.H. Chan School of Public Health Institutional Review Board and waived by the SingHealth Centralized Institutional Review Board. Data are reported in accordance with Standards for Reporting Implementation Studies and American Association for Public Opinion Research guidelines.^26,27

All study activities were completed at Singapore General Hospital (SGH) and National Heart Centre Singapore (NHCS). SGH is the largest academic referral center in Singapore with 2000 beds and over 92 000 operations performed annually.²⁸ NHCS includes 186 beds and performs 9000 operations annually.²⁹ Both SGH and NHCS are governed under the SingHealth cluster, which explicitly named surgical safety as an area of focus in 2018.³⁰ Both institutions adopted the SSC in 2009, with minimal adjustments to the checklist since that time.

Approach to Reimplementation

The reimplementation package we designed is represented in Figure 1. We formed a multidisciplinary implementation team, including an executive sponsor (H.K.T.), a senior surgeon as implementation lead (T.T.Y.), 2 senior OR nurses, and 4 quality improvement specialists. A project manager was allocated 20 hours per week; all other team members received no protected time or funding.

Figure 1. Approach to Surgical Safety Checklist Reimplementation.

In the Explore phase, a combination of surveys, direct observations, and administrative data were used to assess checklist use and the Atlas Foundations Survey provided data on implementation readiness. Engagement with frontline staff began in this phase and continued longitudinally. The Prepare phase used a structured approach to modify the checklist and refine the revised version. During the Implementation phase, multiple approaches were combined to build support and familiarity with the new safety process. Sustainment will rely on periodic reevaluation, as well as ongoing monitoring of checklist use.

Reimplementation began with an assessment of surgical care quality, safety culture, and SSC performance; the findings of this assessment have been previously published.³¹ The Exploration phase also included engagement with frontline stakeholders, solicitation of recommendations for improvement, and recruitment of surgeon and nurse champions. In the Preparation phase, the institutional version of the SSC was compared against the original WHO version, an options document for modifications was created, and a best-bet version was prepared (eFigure in Supplement 1). The Device Briefing Tool was included based on results from phase 1. Pilot testing and rapid-cycle improvement were conducted by departmental champions and other volunteers in July 2021. OR staff were trained through a combination of in-services, demonstration videos, and online modules. The revised SSC was formally implemented in November 2021.

Implementation Outcomes

Measures of implementation success were selected based on Proctor’s conceptual framework.³² Feasibility was assessed using administrative records to calculate the percentage of trained OR staff members. We evaluated penetration and fidelity using direct observation of OR cases by trained observers. Initiation of each pause point was used to determine penetration. Fidelity was assessed based on the number of checklist items completed during each pause point, cessation of activity by the OR team, and, as an indicator of engagement, eye contact among team members during SSC performance.³³

Effectiveness Outcomes

The primary effectiveness outcome was OR team performance, assessed by nurse observers using the Oxford Non-Technical Skills (NOTECHS) system (eTable 1 in Supplement 1). NOTECHS is a well-validated system for assessing leadership, teamwork, problem solving, and situational awareness among OR teams.³⁴ Our training methods and interrater reliability have been previously published.³⁵ Secondary outcomes included safety culture, assessed using the Agency for Healthcare Research and Quality (AHRQ) Hospital Survey on Patient Safety Culture (HSOPS) and device-related interruptions (DRIs), which were tabulated by observers using a standardized classification system.^36,37 Four exploratory outcomes were selected to evaluate the impact of reimplementation on patient safety. Patient safety events and near-misses were captured from an institutional event reporting system. The institutional National Surgical Quality Improvement Program (NSQIP) database was used to calculate 30-day mortality and serious complication rates as defined by NSQIP.³⁸

Sample Size and Sampling Methodology

Sample size calculations indicated that 80 observations per time point would be required to detect a 20% rise in NOTECHS scores. OR cases were selected for observation via purposive-criterion sampling. In this approach, observers maintained a running tally of case characteristics and were directed to cases by senior OR nurses to maximize variety of cases. Colorectal surgery was intentionally oversampled due to routine use of devices required for the first phase of the study. Surveys were electronically distributed to all surgeons, anesthesiologists, and OR nurses using FormSG (GovTech, Singapore).

Observers collected baseline data from November 2019 to February 2021 with an 11-month pause beginning in February 2020 due to the COVID-19 pandemic. Equivalence between baseline periods was established using Schuirmann 2 1-Sided Test procedure.³⁹ End point observations occurred from February to May 2022. Baseline HSOPS data were collected in November and December 2019, with end point surveys collected from April to July 2022.

Analysis

Descriptive statistics were calculated for implementation outcomes and compared using Fischer exact tests for categorical data and Wilcoxon rank sum tests for continuous variables. NOTECHS data were analyzed using generalized linear models on a pseudolinear time scale allowing adjustment for pre– and post–COVID-19 time trends with clustering by department and adjustment for case complexity. DRIs were analyzed similarly using Poisson regression and adjustment for surgical device type. HSOPS surveys were analyzed by calculating average percent positive responses, as recommended by AHRQ.³⁶ Exploratory patient-safety analyses used run charts with standard definitions of statistical shifts as 6 or more consecutive points above or below the median and trends as 5 consecutive points moving in the same direction.⁴⁰ Medians for run-chart analyses were set at the fourth quarter of 2020, immediately before the first phase of this study. Statistical analyses were conducted using SAS version 9.4 (SAS Institute). Run charts were constructed using the Institute for Healthcare Improvement macro for Microsoft Excel 2016 (Microsoft Corporation).⁴¹

Results

A total of 252 cases were observed, including 161 baseline and 91 end point operations. Case characteristics are shown in the Table. Overall missingness was 0.8%. There were 593 survey responses at baseline and 259 postintervention, equating to response rates of 53.4% and 23.3%, respectively. There were 15 dropoffs and survey data were 96.2% complete.

Table. Characteristics of Operations Evaluated by Observers.

Characteristic	Baseline (n = 161), %	End point (n = 91), %
Surgical department
Colorectal	58 (36.0)	30 (33.0)
Cardiothoracic	19 (11.8)	10 (11.0)
Upper gastrointestinal	15 (9.3)	11 (12.1)
Hepatopancreatobiliary	21 (13.0)	10 (11.0)
Head and neck	10 (6.2)	5 (5.5)
Obstetrics and gynecology	18 (11.2)	10 (11.0)
Urology	14 (8.7)	10 (11.0)
Acute care	6 (3.7)	5 (5.5)
Case complexity
Very complex	15 (9.3)	1 (1.1)
Somewhat complex	105 (65.2)	56 (61.5)
Not complex	41 (25.5)	34 (37.4)
Surgical device use
No device recorded	63 (39.1)	1 (1.1)
Tissue sealer	34 (21.1)	21 (23.1)
Linear stapler	30 (18.6)	43 (47.3)
Circular stapler	24 (14.9)	23 (25.3)
Other	10 (6.2)	3 (3.3)

Implementation Outcomes

Feasibility

Thirty-minute in-services were held with all surgical departments, as well as anesthesia and OR nursing. The electronic training module was completed by 96.9% of OR team members (100% of nurses and 92.6% of physicians). Twelve surgical departments achieved 100% compliance with training requirements; no department attained a completion rate below 88%. Posters of the revised SSC were printed and displayed in all ORs and the electronic health record was updated systemwide to reflect the new checklist.

Penetration and Fidelity

Data on penetration and fidelity are displayed in eTable 2 in Supplement 1. Baseline data on the preinduction sign-in were not collected as this pause point was used only sporadically prior to reimplementation. After reimplementation, the sign-in was initiated in 99.0% of observed cases with an average of 91.6% of checklist items completed. All members of the OR team ceased other activities in 98.0% of cases and made eye contact with one another in 87.2% of cases. The time-out was performed in 100% of cases both before and after SSC reimplementation but mean item completion increased from 62.7% to 97.3% (P < .001). Activity suspension increased from 42.5% to 88.1% (P < .001) and eye contact increased from 31.6% to 92.7% (P < .001). Initiation of the sign-out increased from 94.9% to 100% after SSC reimplementation (P = .02). Concomitant improvements were observed for mean item completion (61.7% to 88.6%), activity suspension (4.4% to 47.7%), and eye contact (3.0% to 49.0%; all P < .001).

Effectiveness Outcomes

Team Performance and Device-Related Interruptions

Reimplementation of the SSC was associated with an increase in mean NOTECHS scores from 37.1 to 42.4 of 48 possible points (Figure 2). This equated to a 4.3-point increase after adjustment for preexisting time trends and case complexity (95% CI, 2.9-5.7; P < .001). Improvements were noted across all OR subteams and NOTECHS subscales (eTable 3 in Supplement 1). DRIs decreased by 86.5% after reimplementation (21.7 vs 2.2 per 100 cases; 95% CI, −22.1% to −97.8%; P = .03). Similar estimates were obtained on unadjusted models and sensitivity analyses.

Figure 2. Non-Technical Skills (NOTECHS) Scores and Device-Related Interruptions (DRIs) Before and After Reimplementation of the Surgical Safety Checklist.

The Oxford NOTECHS system assesses surgical, anesthetic, and nursing subteams across 4 dimensions of nontechnical skills, yielding a total score between 12 and 48. DRIs were assessed as a count variable using a previously detailed classification system.

Culture of Safety

Significant improvements after SSC reimplementation were observed in 9 of 12 HSOPS patient safety composites (Figure 3). Overall positive perceptions of patient safety increased from 59.1% to 70.8% (P < .001), comparing favorably with the AHRQ benchmark of 66%. Improvements were also noted in managerial support for patient safety (70.7% to 77.0%; P = .01), interdisciplinary teamwork (62.7% to 68.8%; P = .02), and communication openness (44.9% to 50.7%; P = .05), though communication openness remained below the benchmark level.

Figure 3. Results of the Hospital Survey on Patient Safety.

Average percent positive responses for each safety culture composite were calculated before and after reimplementation of the surgical safety checklist. These were referenced against Agency for Healthcare Research and Quality (AHRQ) 2018 benchmark levels.

Patient Safety

Run charts are shown in Figure 4. A shift demonstrating reduced patient safety events coincided with pilot testing of the revised SSC and persisted through the end of the study period. A similar reduction in near misses was noted at the same time but failed to meet requirements for statistical significance. Mortality and serious complications declined after SSC reimplementation with shifts beginning at the time of implementation for both variables and a downward trend for serious complications beginning 1 month thereafter.

Figure 4. Run Charts .

Charts are annotated to indicate the initiation of study phase 1, pilot testing of the revised surgical safety checklist, and formal implementation of the new checklist. The horizontal dashed line in each chart represents the median.

Discussion

To our knowledge, this is the first study of its kind to demonstrate that comprehensive reimplementation of a checklist intervention can meaningfully improve team behavior, safety culture, patient safety, and patient outcomes. Despite considerable progress over the past decades, surgical safety remains a pressing concern worldwide. Complication rates after surgery approach 20%, while mortality is estimated at 1% to 6%.^42,43,44,45 The SSC is a simple scalable solution to reduce the burden of surgical morbidity and mortality worldwide. Unfortunately, ineffectual implementation frequently hampers its effectiveness. Where thoughtful implementation supports widespread use and meaningful engagement, the clinical benefits of the SSC are realized; where SSC performance is driven solely by mandates or regulatory requirements, the checklist may become a mere tick-box exercise, losing its crucial mediating effects on team performance. Moreover, while the WHO promotes adaptation of the SSC to meet local needs, injudicious modification may have detrimental results.^46,47 In short, maximizing the public health impact of the SSC requires both an effective checklist instrument and an effective implementation strategy to ensure uptake and meaningful use. Thus, institutions with ineffective checklists require evidence-based strategies for reimplementation.

In this study, our reimplementation package produced statistically significant changes in behavior, reflecting a shift toward more meaningful engagement during performance of the SSC. This change translated into clear improvements in OR team performance. NOTECHS scores increased by an impressive 4.3 points, even after controlling for changes in case complexity, distribution of surgical departments, and preexisting time trends. For comparison, 1 subteam moving from “standard” to “excellent” in all 4 categories only equates to a 4-point increase. Prior studies targeting OR team nontechnical skills have achieved improvements of less than 2 points.^48,49,50,51 Safety culture more generally also improved after reimplementation. SGH and NHCS now exceed AHRQ benchmark levels in 5 domains and international benchmarks from the Organisation for Economic Co-operation and Development in 10 domains.^36,52 Although communication openness remains below benchmark levels, the average percent positive response increased substantially.

The increased focus on surgical device safety was highly effective. DRIs decreased by nearly 90% after reimplementation; this finding persisted after adjustment for surgical device type, complexity, and baseline time trends. Interruptions in surgery are known to contribute to intraoperative errors.²¹ Surgical devices in particular are implicated in up to 15% of interruptions and 25% of errors.^5,22 As such, the reduction in DRIs observed here is an important indicator of the degree of improvement in patient safety.

Exploratory data suggest that these improvements have already impacted patient outcomes. Reductions in patient safety events, serious complications, and even mortality clearly coincided with the reimplementation process, despite the relatively short duration of follow up. Although the present study is underpowered for a formal analysis of morbidity and mortality, future efforts should confirm these findings using established practices for risk adjustment and rigorous quasi-experimental methods for longitudinal data.⁵³ As time accrues to support such an analysis, we expect that the observed improvements will continue to strengthen with increasing institutionalization of the reimplemented SSC.

Most impressively, these benefits accrued in an already high-functioning academic center with an institutionally-mandated SSC used—at least nominally—in nearly 100% of cases. The success of our reimplementation package is attributable to 2 factors. First, sound implementation science principles were embedded in every aspect of our process. Interventions relying on frameworks like Exploration, Preparation, Implementation, Sustainment are known to be more effective than those without a theoretical basis.⁵⁴ Our implementation team was carefully selected, including a leader with a participatory management style and an executive sponsor with strong social capital.^55,56,57,58 We engaged frontline users and opinion leaders early and often to assess needs, gain insight on the local context, and build strategic support.^59,60,61 Champions were recruited from all departments to test and promote the revised SSC.⁶² A project manager was specifically allocated time to support implementation and regular coaching provided opportunities for guidance and troubleshooting as reimplementation efforts progressed.^63,64

Equally importantly, our approach produced an evidence-based version of the SSC tailored to the needs of end users. The original WHO SSC included a mixture of process checks and conversation prompts.⁶⁵ Conversation prompts are crucial to fostering teamwork and communication, but studies have shown a tendency to omit them in favor of an expansive range of process checks, as was the case with the preexisting institutional checklist.^20,46,66 This promotes a conception of the SSC as a tedious rather than valuable process.^67,68,69 We were able to remove unnecessary process checks and reinsert all conversation prompts from the original WHO version. Additionally, responsibility for pause points was divided between the anesthesiologist, surgeon, and nurse, as migrated leadership encourages consistent engagement as a multidisciplinary team.^65,70 Moving the checklist from the electronic health record to a wall-mounted poster further promoted team engagement.⁷⁰ Lastly, we added the surgeon statement, an appeal from the lead surgeon to call out safety risks, which has been shown to empower team members.^65,71,72

Limitations

Limitations of this study relate predominantly to the single-system pre-post design. Both hospitals involved are well-resourced academic referral centers in a high-income context. Additionally, the effect of cultural factors unique to this context cannot be ignored. However, in line with best practices in implementation science, our approach is designed to allow flexibility for variable contexts. Additionally, data on patient outcomes are limited. Further research is necessary to delineate the impact of reimplementation on morbidity and mortality. Our data are, nonetheless, adequate to indicate a statistically significant shift in morbidity and mortality in addition to upstream mediators, such as OR team performance. OR observations were subject to the Hawthrone effect, though the presence of observers would presumably have similar effects both before and after reimplementation. Lastly, limited conclusions should be drawn from our HSOPS data. Although these results are consistent with other positive changes observed after reimplementation, they are subject to nonresponse bias, particularly in the postintervention time frame.

Conclusions

The WHO SSC is a powerful tool with the potential to drastically reduce the burden of surgical morbidity and mortality, but ineffective implementation frequently precludes these benefits. In this study, we demonstrated the ability of a comprehensive system of SSC reimplementation to effect behavior change, improve OR team performance, safety culture, and surgical device safety, and, most importantly, prevent harm to patients. Thus, it provides the strongest and most direct evidence to date that effective SSC use requires careful attention to the principles of implementation science. Future efforts will expand the reach of this system by testing a structured guidebook in a variety of settings.

Supplement 1.

eTable 1. The Oxford NOTECHS system

eTable 2. Implementation outcomes by pause point

eTable 3. Change in NOTECHS scores and device-related interruptions (DRIs) after Surgical Safety Checklist reimplementation

eFigure. Surgical Safety Checklist

Supplement 2.

Data sharing statement