Addressing knowledge gaps in Surgical Safety Checklist use: statistical process control analysis of a surgical quality improvement programme in Ethiopia

Abstract

Background

The WHO Surgical Safety Checklist reduces morbidity and mortality after surgery, but uptake remains challenging. In particular, low-income countries have been found to have lower rates of checklist use compared with high-income countries. The aim of this study was to determine the impact of educational workshops on Surgical Safety Checklist use implemented as part of a quality improvement initiative in five hospitals in Ethiopia that had variable experience with the Surgical Safety Checklist.

Methods

From April 2019 to September 2020, each hospital implemented a 6-month surgical quality improvement programme, which included a Surgical Safety Checklist workshop. Statistical process control methodology was used to understand the variation in Surgical Safety Checklist compliance before and after workshops and a time-series analysis was performed using population-averaged generalized estimating equation Poisson regression. Checklist compliance was defined as correctly completing a sign in, timeout, and sign out. Incidence rate ratios of correct checklist use pre- and post-intervention were calculated and the change in mean weekly compliance was predicted.

Results

Checklist compliance data were obtained from 2767 operations (1940 (70 per cent) pre-intervention and 827 (30 per cent) post-intervention). Mean weekly checklist compliance improved from 27.3 to 41.2 per cent (mean difference 13.9 per cent, P = 0.001; incidence rate ratio 1.51, P = 0.001). Hospitals with higher checklist compliance at baseline had the greatest overall improvements in compliance, more than 50 per cent over pre-intervention, while low-performing hospitals showed no improvement.

Conclusion

Surgical Safety Checklist workshops improved checklist compliance in hospitals with some experience with its use. Workshops had little effect in hospitals unfamiliar with the Surgical Safety Checklist, emphasizing the importance of multifactorial interventions and culture-change approaches. In receptive facilities, short workshops can accelerate behaviour change.

The WHO Surgical Safety Checklist reduces morbidity and mortality after surgery, but uptake is particularly challenging in low-income countries. The impact of educational workshops on Surgical Safety Checklist use in five Ethiopian hospitals was assessed. Workshops can accelerate uptake of the Surgical Safety Checklist in facilities with some experience with its use, but had little effect in hospitals with no experience, emphasizing the importance of multifactorial interventions and culture-change approaches.

Introduction

In its pilot study, the WHO Surgical Safety Checklist (SSC) was shown to reduce surgical morbidity and mortality^1,2. The original SSC pilot study was conducted in a variety of locations globally as the SSC is intended to be adaptable to all perioperative settings, regardless of income setting¹. However, differences in implementation affect both the uptake and efficacy of the checklist³. When implemented via government mandates without formal team training, it has not produced the same improvements in perioperative morbidity and mortality, and may be viewed as a box-checking exercise^4,5, while approaches that focus on the checklist’s ability to improve team dynamics have been most effective⁶. Further studies on SCC use found that morbidity and mortality improvements were most pronounced where all components of the checklist were done completely and with high fidelity, while the SSC was less effective in settings where it was done incompletely^5,7. A quality improvement study in the US state of South Carolina showed that teamwork and buy-in from hospital leadership and physicians were critical for implementation, as were in-person meetings and training regarding teamwork skills^8,9.

Additional barriers to SSC uptake exist in low-income settings, including limitations regarding human resources, knowledge gaps, and cultural differences^10,11. Compared with high-income countries, hospitals in low-income settings have less-robust processes in place for the implementation, auditing, and maintaining accountability of new safety standards¹². Nonetheless, encouraging examples, such as a countrywide educational programme in Madagascar, demonstrate that checklist implementation is possible in low-income settings^13,14.

The Clean Cut programme is a surgical quality improvement programme that integrates the use of the SSC as one critical tool in the prevention of complications from surgery. The 6-month adaptive programme, designed and implemented by the Lifebox Foundation, local clinical collaborators, and hospital leaders, aims to reduce surgical infections by improving perioperative infection-prevention practices. In its pilot study, Clean Cut increased SSC use¹⁵, with persistent improvements 6–18 months after implementation was completed¹⁶.

Clean Cut’s adaptive design uses monthly action-planning meetings over the course of the programme to address gaps in perioperative processes, including gaps in SSC use¹⁵. As the Clean Cut programme developed, pilot programme data were analysed for areas of improvement and knowledge gaps in how to use the SSC were identified. In response, educational workshops were developed as an additional intervention aimed at addressing those knowledge gaps. Because SSC usage does not rely on hospital purchasing decisions or available material resources such as antibiotics or functional autoclaves that determine feasibility of other practices important for surgical safety, addressing knowledge gaps has the capability to improve staff engagement and increase SSC adoption. In this analysis, it was hypothesized that educational workshops may have significant benefit with regard to improving SSC compliance. The aim was to understand if the behaviour change in SSC usage was related to educational sessions and workshops that occurred as part of the larger Clean Cut quality improvement programme.

Methods

This study was an analysis of educational workshops delivered as a component of the Clean Cut programme, the design and results of which are described elsewhere¹⁵. The five hospitals selected to be included were given the formal SSC workshop training as an interventional package. These hospitals included four tertiary centres and one district hospital. Three hospitals were located in Addis Ababa, while the other two were located in smaller cities in the Amhara region of Ethiopia. The SSC workshops were scheduled during the 6 months of Clean Cut programme implementation and were delivered to members of operating-room teams who would be involved in completing the checklist for each operation. Time-series data were analysed to compare improvements in checklist compliance during operations before the workshop (‘pre-intervention’) and operations after the workshop (‘post-intervention’).

Intervention

Each of the hospitals received a day-long workshop on the use of the SSC. On the day of the workshop, participants were given a pre-workshop multiple-choice quiz that allowed the trainers to tailor the workshop to gaps in knowledge. At the end of the day, participants were given a post-workshop quiz and an opportunity to evaluate the workshop. The workshop was designed in collaboration with clinicians with experience implementing the SSC in Ethiopian hospitals. Its content emphasized areas of improvement identified by those clinicians who were familiar with local operating-room culture and practices. The workshop included a review of the history and significance of the SSC, followed by demonstration and group practice of the three components of the checklist: a sign in before induction of anaesthesia; a timeout before incision; and a sign out at the end of the case. Because the SSC was intended to be adapted to the local context¹⁷, the workshop also included a session on modifying the checklist to capture the key safety concerns in each hospital. For example, at one facility where electrical outages were common, the team included checking that there was fuel in the backup electrical generator as a point on their checklist. It also focused on building skills in teamwork and communication, and auditing checklist use over time.

Data collection and ethics approval

Ethics approval was obtained from the Ethiopian Federal Ministry of Health through the Armauer Hansen Research Institute. Data were collected using direct observation of operating-room teams before, during, and after the operation by operating-room nurses on a standardized form. Nursing staff were engaged as data collectors as they were present in the operating rooms throughout the day and familiar with the surgical practices reported in the data-collection tool. Before the start of the programme, these nurses were trained on the data-collection process and definitions to accurately identify and document behaviours pertaining to SSC use. Data collectors were instructed to only include observable behaviours; in cases where the operating-room team filled out a paper checklist, but did not correctly perform the associated behaviours, surgical teams were not considered to have completed the checklist and data collectors were instructed to record what they saw, not whether the checklist was filled out on paper alone.

Outcome measures

The primary outcome was weekly overall SSC compliance, defined as the number of operations in which the checklist was performed divided by the total number of operations that occurred in operating rooms where Clean Cut implementation was ongoing. Checklist performance was required to include key behaviours in its three sections: first, a sign in, which was completed aloud and before anaesthesia induction; second, a timeout that was announced aloud and included the anticipated blood loss announced aloud; and third, a sign out at the end of the case, which was completed aloud and completed in the operating room. Operations in which only part of the SSC was completed were not considered compliant. Secondary outcomes included weekly checklist component compliance counts or the number of operations in which behaviours were completed for each of three checklist sections: sign in, timeout, and sign out.

Data analysis

A statistical process control methodology was used to compare rates of checklist compliance before the workshops with checklist compliance after the workshops. Statistical process control is a set of time-series analytical methods to detect non-random variation over time by comparing continuous changes in an outcome with outcomes in the past¹⁸. While this methodology originated in manufacturing, more recently, these approaches have been used to detect changes in clinical performance after a quality improvement intervention by comparing performance before and after the intervention¹⁹. This concept was applied to the authors’ quality improvement efforts, focusing on variations in performance before and after the checklist workshop by comparing uptake of checklist use before and after delivery of the workshop.

Statistical process control methodology is an analytical method that requires continuous data over time. Because this was a 6–9-month intervention, weekly compliance counts were chosen to be used and included all consecutive weeks before and after the intervention. Occasionally, breaks in data collection occurred as a result of a drop-off in operations due to the coronavirus pandemic or sociopolitical factors. In cases where a break in consecutive data collection occurred for 7 days or longer at a given hospital, a weekly checklist compliance count could not be calculated for that 7 day interval at that hospital. In these cases, data were excluded either before or after the break in data collection to maintain the continuity of the time series. If the break was in the pre-implementation interval for that hospital, any data collected before the break were excluded. If the break was in the post-implementation interval, data collected after the break were excluded. After excluding these weeks, 94 per cent of checklist compliance data were included (supplemental Appendix, Table S1).

Separate models were constructed for overall checklist compliance (the primary outcome), as well as three secondary outcomes: sign-in, timeout, and sign-out components of checklist compliance (using a single modelling strategy). Time-series data were modelled using Poisson regression. To account for hospital clustering effects and in-hospital autocorrelation, these models were fit with a generalized estimating equation using first-order autocorrelation structure. Weekly checklist compliance counts were used as the dependent variable and total weekly operation counts were used as the exposure variable. The independent variable of interest was the post-intervention indicator (versus pre-intervention) and fixed effects for each hospital were included.

From these models, the incidence rate ratios (IRRs) were estimated in the post-intervention interval (versus pre-intervention) for each outcome of interest. However, because IRRs are more difficult to interpret than compliance rates, the mean SSC compliance rates before and after the checklist workshop using the same models were also estimated. Data analysis was performed using Stata v16.1/SE (College Station, TX, USA).

Results

A total of 2767 operations were included (1940 (70 per cent) pre-intervention and 827 (30 per cent) post-intervention). Patient characteristics were similar between the pre- and post-intervention operations (Table 1).

Table 1.

Patient and operation characteristics

	Total (n = 2767)	Pre-workshop (n = 1940)	Post-workshop (n = 827)	P
Patient characteristics
Age (years), mean(s.d.)	30(14)	30(14)	30(13)	0.164
Sex ratio (M : F)	856 : 1911	594 : 1346	262 : 565	0.580
Diabetes	56 (2)	41 (2)	15 (2)	0.777
Hypertension	145 (5)	99 (5)	46 (6)	0.377
Hospital				<0.001
1	349 (13)	172 (9)	177 (21)
2	144 (5)	56 (3)	88 (11)
3	441 (16)	410 (21)	31 (4)
4	1344 (49)	902 (46)	442 (53)
5	489 (18)	400 (21)	89 (11)
Type of case				<0.001
Elective	929 (34)	698 (36)	231 (28)
Urgent/emergency	1585 (57)	1068 (55)	517 (63)
Operation type				<0.001
Obstetrics/gynaecology	1254 (45)	897 (46)	357 (43)
General surgery	755 (28)	563 (29)	192 (23)
Other sub-specialty	474 (17)	302 (16)	172 (21)

Values are n (%) unless otherwise indicated.

Weekly checklist compliance demonstrated an increasing trend in compliance after workshop delivery, although variation in compliance was still noted week to week (Fig. 1). Overall checklist compliance improved by 52 per cent from 27 per cent compliance in the pre-intervention interval to 41 per cent in the post-intervention interval (P = 0.001) with an IRR of 1.51 (P = 0.001) (Fig. 2). Similarly, each component of checklist compliance improved, with the most substantial improvement in compliance with sign in from 39 per cent compliance pre-intervention to 52 per cent compliance post-intervention (IRR 1.34, P = 0.008; rate difference 33 per cent, P = 0.010). Improvements in the use of timeout and sign out were also observed, but were not statistically significant (supplemental Appendix, Fig. s1).

Fig. 1.

Mean weekly checklist compliance

Fig. 2.

Change in mean compliance rate

IRR, incidence rate ratio.

When analysed separately by hospital, three of the five hospitals showed consistent increases over 50 per cent in SSC compliance after the checklist workshop; however, two hospitals failed to improve (Table 2). Of the three hospitals that improved, two were tertiary centres and one was a district hospital. Both hospitals that failed to improve were tertiary centres. The two hospitals without overall post-workshop improvement in checklist compliance demonstrated improvements in one or two checklist components, but these improvements were not statistically significant. Notably, the best-performing hospitals had higher than mean rates of compliance before the checklist workshop and larger improvements after the workshop, while hospitals that performed poorly before the workshop did not improve (Fig. 3).

Table 2.

Mean compliance rate for overall checklist compliance, sign in, timeout, and sign out by hospital

Hospital		Mean compliance rate (95% c.i.)		Rate difference (95% c.i.)	Percentage increase in compliance	P
		Pre-intervention	Post-intervention
1	Overall SSC compliance	0.13 (0.08,0.19)	0.20 (0.12,0.28)	0.07 (0.02,0.12)	54	0.005
	Sign in	0.32 (0.23,0.41)	0.43 (0.31,0.55)	0.11 (0.02,0.19)	34	0.013
	Timeout	0.87 (0.70,1.05)	0.95 (0.75,1.15)	0.08 (−0.06,0.21)	9	0.25
	Sign out	0.23 (0.17,0.31)	0.29 (0.20,0.37)	0.05 (−0.01,0.10)	22	0.088
2	Overall SSC compliance	0.53 (0.37,0.70)	0.80 (0.58,1.02)	0.27 (0.11,0.43)	51	0.001
	Sign in	0.73 (0.53,0.94)	0.98 (0.73,1.22)	0.25 (0.06,0.43)	34	0.009
	Timeout	0.78 (0.54,1.00)	0.85 (0.60,1.09)	0.07 (−0.05,0.19)	9	0.246
	Sign out	0.83 (0.63,1.03)	1.00 (0.78,1.21)	0.17 (0.02,0.35)	20	0.073
3	Overall SSC compliance	0.02 (0.0002,0.03)	0.02 (−0.0001–0.05)	0.00 (−0.002,0.02)	0	0.102
	Sign in	0.17 (0.12,0.22)	0.23 (14,31)	0.06 (0.01,0.11)	35	0.027
	Timeout	0.04 (0.01,0.07)	0.04 (0.01,0.07)	0.00 (−0.003,0.01)	0	0.303
	Sign out	0.11 (0.07,0.15)	0.14 (0.08,0.19)	0.02 (−0.01,0.05)	18	0.112
4	Overall SSC compliance	0.001 (−0.001,0.004)	0.002 (0.00,0.01)	0.00 (−0.0007,0.002)	0	0.333
	Sign in	0.01 (0.00–0.02)	0.01 (0.00,0.02)	0.00 (−0.0004,0.006)	0	0.089
	Timeout	0.78 (0.70,0.87)	0.85 (0.72,0.98)	0.07 (−0.05,0.19)	9	0.355
	Sign out	0.00 (0.00,0.01)	0.00 (0.00,0.01)	0.00 (−0.0004,0.002)	0	0.235
5	Overall SSC compliance	0.66 (0.56,0.76)	1.00 (0.78,1.22)	0.34 (0.12,0.56)	51	0.003
	Sign in	0.74 (0.63,0.85)	0.99 (0.78,1.20)	0.25 (0.05,0.45)	34	0.015
	Timeout	0.90 (0.76,1.03)	0.98 (0.80,1.15)	0.08 (−0.06,0.22)	9	0.255
	Sign out	0.85 (0.75,0.95)	1.02 (0.82,1.22)	0.17 (−0.028,0.37)	20	0.093

Fig. 3.

Overall checklist compliance by hospital

Discussion

This study shows that addressing knowledge and practice gaps in SSC use through checklist workshops improved the rates of correct use, but improvements were limited to hospitals that were already familiar with the checklist. Hospitals that were already familiar with the SSC were the most likely to benefit from further training, while the lowest-performing hospitals showed almost no benefit from the workshop. This suggests that both pre-existing factors, such as hospital staff buy-in and preconceived perceptions of the value of the SSC, and ongoing engagement with quality improvement efforts remain crucial to the success of programmes aiming to implement surgical safety checklists in similar settings³. Although the SSC has been part of the Ethiopian Ministry of Health’s national strategy since 2016, a national cross-sectional audit revealed that it was completed less than half the time²⁰. This may reflect a lack of understanding of how to use the SSC or its relevance to clinical practice. In hospital environments that are ready to engage with quality improvement initiatives, short workshops can serve as a catalyst to improve understanding of how the SSC can be most effective and accelerate behaviour change, as seen with the SSC workshops in higher-performing hospitals.

Not every portion of the checklist improved equally in this cohort. The most substantial improvements were in sign in. This could reflect that the workshop gives particular emphasis to this part of the process, rather than timeout or sign out. The timeout had the highest rate of compliance before the workshop, so participants may have directed their energy elsewhere. These differences may be used to improve workshop content and delivery in the future.

All five hospitals were participating in the larger Clean Cut quality improvement programme at the time of workshop delivery. While statistical process control methodology specifically targets changes made at the time of the workshop, it is unknown if additional process changes in checklist use were instituted around the same time. One would expect changes such as these to work synergistically with the education provided through the workshops and expect that the surrounding context of the quality improvement efforts through Clean Cut play a role in creating a critical environment for the uptake of practice changes. The effects of the workshops on SSC uptake shown in this study are consistent with the broader published data on the efficacy and sustainability of the Clean Cut programme, which supports the role of multifactorial interventions that allow quality improvement skills to be applied best with an improved knowledge base^15,16.

Because the workshops were delivered within the context of Clean Cut implementation, they were delivered at various time points throughout the programme. The interventional aspects of Clean Cut implementation commence with a meeting of all stakeholders after baseline data collection and process mapping to identify the first targets of process improvements to increase compliance with critical infection-prevention standards. The workshop was introduced as part of this effort, but usually several weeks after initial interventional efforts. As such, the pre-workshop group was proportionally larger than the post-workshop group, even though generally the post-Clean Cut intervention group comprises more patients in the overall programme.

This study has several limitations. First, the statistical process control analysis requires a very rigorous time-series analysis, without any gaps in continuous data collection. To meet these requirements, data were excluded that were separated by any intervening week without any operations. Due to the coronavirus pandemic and other sociopolitical factors, gaps in data collection did occur at the very beginning or very end of the study interval, despite hospital teams’ best efforts to keep the programme going continuously. However, even without the excluded data, the study included 94 per cent of collected checklist compliance data. Second, data collection itself poses a challenge in a resource-limited setting like Ethiopia, because direct observation is time intensive and adds to the workload of operating-room nursing staff. Studies in resource-constrained environments often rely on hospital staff who work in the relevant clinical areas to participate in data collection as third-party observers are not readily available or accessible. There is a risk that including operating-room nurses who are familiar with the surgical teams being studied could have introduced bias in their observations. Operating-room nurses also participate in SSC use in their clinical roles when not acting as data collectors and may be more likely to use the checklist when they were also engaged as data collectors as part of this study. Third, it is also possible that over the course of the study, data collectors may have become fatigued and misinterpreted or extrapolated data. To minimize these limitations only specific and concrete observable behaviours were recorded. The training of nurses in data collection could improve uptake of the SSC. However, training occurred at the very beginning of the study, often weeks or months before the SSC workshop, which is less likely to affect improvements made in response to the workshop. Finally, surgical teams were aware they were being observed. The Hawthorne effect, an improvement in performance due to a subject’s awareness that they are being observed, could drive behaviour²¹. For this reason, ongoing research via a stepped-wedge cluster-randomized design is being pursued.

The study found that the utility of an SSC workshop for future use of the SSC was highly dependent on the environment in which it was implemented. A thoughtful implementation strategy is a critical component in improving uptake of tools such as the SSC and must go beyond just the introduction of the tool itself. In the Ethiopian context, familiarity with the SSC was key to improving uptake after educational workshops, suggesting that sensitization before workshop delivery may improve workshop efficacy. Key areas of future research include further investigation into what factors predict success for some hospitals and identifying barriers in less-successful hospitals early. For example, because only a single district hospital was included, further research is needed to understand how hospital size or the presence of trainees may affect how workshops are received. While hospital environments may vary, adapting surgical safety initiatives to address these barriers is critical as access to surgical care increases globally²². Understanding factors that will predict success or challenges would open opportunities to adapt SSC workshops to hospital-specific factors to create receptive environments for implementation.

Funding

This study was funded in part by a grant from the Global Surgery Foundation, Royal College of Surgeons of Edinburgh and the Lifebox Foundation. M.R.N. and N.S. were supported by the Stanford University Center for Innovation in Global Health via the NIH Fogarty International Center Global Health Equity Scholars award number, NIH/FIC D43 TW010540.

Author contributions

Maia R. Nofal (Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Visualization, Writing—original draft, Writing—review & editing), Nichole Starr (Conceptualization, Data curation, Funding acquisition, Investigation, Methodology, Project administration, Visualization, Writing—original draft, Writing—review & editing), Tihitena Negussie Mammo (Conceptualization, Funding acquisition, Investigation, Project administration, Writing—review & editing), Amber W. Trickey (Data curation, Formal analysis, Methodology, Visualization, Writing—original draft, Writing—review & editing), Natnael Gebeyehu (Investigation, Project administration, Writing—review & editing), Luca Koritsanszky (Investigation, Writing—review & editing), Mechale Alemu (Investigation, Writing—review & editing), Mansi Tara (Investigation, Writing—review & editing), Senait Bitew Alemu (Investigation, Project administration, Resources, Writing—review & editing), Faye Evans (Investigation, Writing—review & editing), Selam Kahsay (Investigation, Writing—review & editing), and Thomas G. Weiser (Conceptualization, Funding acquisition, Methodology, Supervision, Visualization, Writing—original draft, Writing—review & editing).

Disclosure

M.R.N., N.S., and N.G. are Lifebox Safe Surgery Fellows. T.N.M. is the Global Clinical Director at Lifebox. M.T. is a former programme manager at Lifebox. S.B.A. is Lifebox’s Head of Programs for East and Southern Africa. F.E. serves on the Lifebox Global Governance Council. S.K. is a former consultant for Lifebox. T.W. is the Consulting Medical Officer at Lifebox.

Supplementary material

Supplementary material is available at BJS online.

Data availability

De-identified data with anonymized hospital facilities can be made available upon reasonable request to the authors.