Non-technical skills in surgery during the COVID-19 pandemic: An observational study

Abstract

Background

Non-technical skills are critical to surgical safety. We examined the impact of the COVID-19 pandemic on non-technical skills of operating room (OR) teams in Singapore.

Materials and methods

Observers rated live operations using the Oxford NOTECHS system. Pre- and post-COVID observations were captured from November 2019 to January 2020 and from January 2021 to February 2021, respectively. Scores were compared using Schuirmann's Two One-Sided Test procedure. Multivariable linear regression was used to adjust for case mix. A 10% margin of equivalence was set a priori.

Results

Observers rated 159 cases: 75 pre-COVID and 84 post-COVID. There were significant differences between groups in surgical department and surgeon-reported case complexity (both P < 0.001). Total NOTECHS scores increased post-COVID on raw analysis (36.1 vs 38.0, P < 0.001) but remained within the margin of equivalence (90% CI 1.3 to 2.6, P < 0.001). Multivariable analysis demonstrated a similar increase within the margin of equivalence (2.0, 90% CI 1.3 to 2.7). Teamwork and cooperation scores increased by 1.0 post-COVID (90% CI 0.8 to 1.3); all other subcomponent scores were equivalent.

Conclusion

Non-technical skills before and after the peak of the COVID-19 pandemic were equivalent but not equal. A small but statistically significant improvement post-COVID was driven by an increase in teamwork and cooperation skills. These findings may reflect an improvement in team cohesion, which has been observed in teams under duress in other settings such as the military. Future work should explore the effect of the pandemic on OR culture, team cohesion, and resilience.

1. Introduction

Non-technical skills are increasingly recognized as critical to surgical safety. Although technical skills have traditionally been emphasized, technical deficits are responsible for only a small fraction of adverse events [1]. Meanwhile, deficits in non-technical skills, such as communication, teamwork, decision-making, and situational awareness, are responsible for up to 60% of surgical errors [2,3].

The operating room (OR) is a complex, high-risk environment which requires careful coordination among all surgical team members. Although teams have always been subject to the demands of this dynamic high-pressure setting, the COVID-19 pandemic introduced a number of unanticipated stressors. Risks to personal health in the OR are not novel, but the widespread nature of COVID-19 represented a pervasive and previously unseen degree of exposure for surgical teams [4]. Resulting personal protective equipment requirements provided a measurable sense of protection, but may also have degraded OR communications and situational awareness while inducing fatigue and impairing decision-making processes [5]. Resource shortages, staff redeployment, and financial concerns added to the strain on surgical teams [6,7]. Burnout among healthcare workers is at a record high, which can further degrade OR team performance and contribute to errors [8,9].

Although the safety implications of COVID-19 in surgical practice have been studied, little is known about its effects on non-technical skills. We began a surgical safety initiative targeting non-technical skills in an academic tertiary referral center in Singapore in 2019. As cases mounted in early 2020, we were forced to terminate baseline observations prematurely. When COVID-19 restrictions eased, OR observations were allowed to resume. We therefore took this opportunity to assess the impact of the COVID-19 pandemic on non-technical skills among OR teams.

2. Materials and methods

2.1. Study design

We used observations conducted before the COVID-19 pandemic (pre-COVID) and after easing of COVID-19 restrictions (post-COVID) for a natural experiment of surgical team behaviors. Pre-COVID observations were captured from November 2019 to January 2020. Observations resumed in January 2021 and continued through February 2021. All data were collected during the baseline phase of a surgical device safety initiative. No interventions directed towards non-technical skills took place between observation periods. However, engineering and administrative controls to reduce transmission of COVID-19 were implemented, such as dedicated COVID-19 ORs, personal protective equipment, and COVID-19 testing requirements [10].

2.2. Setting and participants

All observations were collected from a large academic referral center in Singapore. Four general surgery departments (Acute Care Surgery, Colorectal Surgery, Hepato-Pancreato-Biliary Surgery, and Upper Gastrointestinal Surgery) and four non-general surgery departments (Cardiothoracic Surgery, Head and Neck Surgery, Obstetrics and Gynecology, and Urology) were included. All elective, urgent, and emergent cases during regular business hours were eligible for inclusion unless precluded for safety reasons, such as active COVID-19 infection or inability to maintain adequate social distancing in the OR. Purposive criterion sampling was used to maintain balance between specialties as much as possible given variability in observer availability, personnel restrictions, and operative schedules. Colorectal surgery was intentionally oversampled due to their use of a specific surgical device of interest.

2.3. Data collection and variables

We utilized the Oxford NOTECHS rating system, a well-validated behavioral marker system for OR teams [11]. The NOTECHS system includes four subscales: leadership and management, teamwork and cooperation, problem solving and decision-making, and situational awareness. Each OR subteam (surgery, anesthesia, and nursing) is rated on a 1–4 scale for each of these dimensions, yielding a total score between 12 and 48. Observers underwent formal training on the NOTECHS system. Interrater reliability was excellent across all subscales, as has been previously described [12]. Standardized data collection sheets were used to record observations and NOTECHS scores. Total NOTECHS scores were the primary outcome of interest. Subscale scores were analyzed as secondary outcomes. Covariates captured by observers included surgical department and surgeon-reported case complexity, rated as high, moderate, or low. Names or other potential identifiers of individual OR team members were not collected to protect anonymity.

2.4. Analysis

Pre- and post-COVID case characteristics were compared using Fisher's exact test. NOTECHS scores were compared using Schuirmann's Two One-Sided Test (TOST) procedure [13]. This method allows testing for both equality and equivalence of two means. Equality is tested using a standard Student's t-test. Equivalence is tested within a preselected margin using two one-sided t-tests. The left-sided t-test establishes whether the difference in means is greater than the preselected lower bound; the right-sided test establishes whether it is less than the preselected upper bound. Thus, if the results of both tests are statistically significant with α < 0.05, the null hypothesis is rejected and the difference in means is said to lie within the margins of equivalence.

A secondary analysis using multivariable linear regression was used to adjust for differences in surgical specialties and case complexity between periods. By convention, the 90% confidence interval (CI) of the parameter estimate for the pre/post-COVID variable was taken and compared against the preselected margin of equivalence [14,15]. This approximates the use of two one-sided tests with α set to 0.05. Based on power calculations, we set a 10% margin of equivalence a priori. Data were reported in accordance with Strengthening the Reporting of Cohort Studies in Surgery (STROCSS) guidelines [16]. This study is registered with ClinicalTrials.gov, identifier #NCT05123495. Ethical approval was obtained from the Harvard T.H. Chan School of Public Health Institutional Review Board and waived by the SingHealth Centralized Institutional Review Board.

3. Results

Observers rated 159 cases: 75 pre-COVID and 84 post-COVID. No surgical teams refused participation during either period. There were significant differences in both surgical department and case complexity between periods (Table 1 ). Of note, colorectal cases were particularly oversampled in the post-COVID period, while the proportion of cardiothoracic, hepato-pancreato-biliary, obstetrics and gynecology, and urology cases declined. The proportions of both high- and low-complexity cases also increased post-COVID, possibly reflecting case prioritization strategies.

Table 1.

Characteristics of operations before and after the peak of the COVID-19 pandemic.

	Pre-COVID (n = 75)	Post-COVID (n = 84)	P-value
Department			<0.0001
Colorectal	11 (14.7%)	47 (56.0%)
Cardiothoracic	14 (18.7%)	5 (6.0%)
Upper GI	7 (9.3%)	8 (9.5%)
HPB	13 (17.3%)	8 (9.5%)
Head & Neck	5 (6.7%)	5 (6.0%)
Ob/Gyn	12 (16.0%)	6 (7.1%)
Urology	13 (17.3%)	1 (1.2%)
Acute Care Surgery	0 (0%)	4 (4.8%)
Surgeon-reported complexity			0.0004
High	4 (5.4%)	11 (13.1%)
Moderate	60 (81.1%)	43 (51.2%)
Low	10 (13.5%)	30 (35.7%)

As shown in Table 2 , mean NOTECHS scores increased from 36.1 pre-COVID to 38.0 post-COVID (p < 0.0001). This change remained within the preselected margins of equivalence (p < 0.0001). Subscale scores for leadership and management and teamwork and cooperation also increased, while problem solving and decision-making and situational awareness scores remained unchanged. Only teamwork and cooperation exceeded the upper bound of equivalence (p = 0.5531).

Table 2.

NOTECHS scores before and after COVID-19.

Score	Pre-COVID (n = 75)	Post-COVID (n = 84)	P-value for difference in means	Left-sided P-value	Right-sided P-value
Total NOTECHS	36.1 (1.3)	38.0 (3.3)	<0.0001	<0.0001	<0.0001
Leadership & management	9.0 (0.5)	9.7 (1.1)	<0.0001	0.0120	<0.0001
Teamwork & cooperation	9.0 (0.3)	10.0 (1.3)	<0.0001	<0.0001	0.5531
Problem solving & decision-making	9.1 (0.3)	9.0 (0.6)	0.2681	<0.0001	<0.0001
Situational awareness	9.0 (0.4)	9.3 (1.3)	0.0569	<0.0001	<0.0001

Results of multivariable linear regression analyses are shown in Table 3 and Fig. 1 . Total NOTECHS scores remained unequal but statistically equivalent between periods (5.1% increase, 90% CI 3.3%–6.8%). Teamwork and cooperation was not equivalent (10.2% increase, 90% CI 7.8%–12.6%). All other subscale scores remained equivalent.

Table 3.

Adjusted differences in NOTECHS scores before and after COVID-19.

Score	Pre- to post-COVID difference	90% CI
Total NOTECHS	5.1%	3.3%–6.8%
Leadership & management	7.3%	5.1%–9.5%
Teamwork & cooperation	10.2%	7.8%–12.6%
Problem solving & decision-making	−1.0%	−2.5%–0.5%
Situational awareness	3.1%	0.3%–5.9%

Fig. 1.

Change in NOTECHS after COVID-19 pandemic, adjusted for surgical department and surgeon-rated complexity.

4. Discussion

The COVID-19 pandemic has had a profound impact on the practice of surgery worldwide. As healthcare resources were strained, a backlog of millions of elective cases developed [17,18]. For operations that could not safely be delayed, OR teams found themselves grappling with new risks to their own health as well as challenges to health system operations and patient safety [19]. In addition, many surgical team members were redeployed to unfamiliar work environments to meet the clinical demands imposed by the pandemic [6]. The effects of these changes on OR teams remain poorly understood.

We used observations of surgical teams pre- and post-COVID using split baseline data from a surgical safety initiative as a natural experiment to assess changes in non-technical skills. Overall, post-COVID NOTECHS scores were found to be slightly higher but statistically equivalent compared to pre-COVID scores. The observed difference of 1.9 is equivalent to a single team member moving from “basic standard” to “excellent” in one category. Given the number of unmeasured confounders with the potential to impact scores, we place greater emphasis on the equivalence of scores than on this small but statistically significant increase.

Nonetheless, there was a clear increase in teamwork and cooperation in the post-COVID period on both unadjusted and adjusted analyses. Improvements in team cohesion have been described in other teams under duress, including high-risk settings such as the military, the fire service, and space flight [[20], [21], [22], [23]]. These improvements can translate into improved team performance and safety, but they require strong leadership and a culture of error management rather than error aversion to ensure that stressors do not degrade performance [22]. Surgeons and surgical leaders should employ collaborative leadership to identify and mitigate stressors imposed by the COVID-19 pandemic as we enter yet another wave to ensure that improved teamwork translates into improved team performance and, ultimately, improved patient safety [24].

From a research standpoint, our results show that studies of surgical team's social and cognitive skills may be able to proceed, albeit cautiously, even if they are interrupted by the COVID-19 pandemic. The methods described herein can serve as a model for such studies. The TOST procedure, complemented by multiple linear regression to control for confounding, is a powerful tool to establish equivalence in an interrupted dataset. However, equivalence margins should be selected carefully, taking into account both the clinical significance of the chosen equivalence level and the statistical power afforded by the available sample size.

The present study is limited by its pre/post observational design. Although no interventions targeting OR teams’ non-technical skills took place during the COVID-19 pandemic, unmeasured secular trends may have contributed to the observed differences in NOTECHS scores. Given the nature of the pandemic, it is not possible to enroll a comparator group that was unaffected by COVID-19. External validity is also limited by our single center design. The reactions of teams in an academic tertiary referral center in Singapore may differ significantly from those in other settings. Singapore was able to mount a swift response to the pandemic, with country-wide public health mandates and implementation of robust safety measures in the OR [25]. Ranked among the most COVID-resilient countries, Singapore has low infection rates and high levels of vaccination [26]. The effects of COVID-19 in countries with less coordinated responses to the pandemic may therefore vary greatly. Longitudinal multicenter data would have significantly strengthened our findings. Such data would also indicate whether other changes in non-technical skills occurred early in the pandemic as providers adapted to rapidly changing conditions and whether the changes we observed were transient. Unfortunately, local conditions did not allow observations to be made during the first wave of the pandemic, and our dataset has since been contaminated by the introduction of our planned intervention.

Our ability to capture NOTECHS scores immediately before the pandemic and one year thereafter is a unique opportunity that cannot be recreated. This is an opportunistic analysis and is therefore restricted by the design of the parent study. It is nonetheless possible to corroborate our evidence using qualitative data such as interviews with surgical staff. Future efforts will engage OR teams to assess their perspectives on the impact of the pandemic on communication, teamwork, and other non-technical skills.

5. Conclusions

Non-technical skills before and after the peak of the COVID-19 pandemic were equivalent but not equal. Increases in teamwork and cooperation exceeded the margin of equivalence, suggesting a positive effect of the pandemic on team cohesion. The statistical methods used to assess these changes can serve as a model for other studies with observation periods interrupted by COVID-19.

Funding

This work was supported by a grant from the Office of the Chief Medical Officer, Johnson and Johnson Medical Devices.

Provenance and peer review

Not commissioned, externally peer-reviewed.

Ethical approval

Ethical approval was obtained from the Harvard T.H. Chan School of Public Health Institutional Review Board (IRB19-0389) and waived by the SingHealth Centralized Institutional Review Board.

Sources of funding

This work was supported by a grant from the Office of the Chief Medical Officer, Johnson and Johnson Medical Devices.

Author contribution

Study conception and design: JCE, RMS, YS, MEB, CL, JMH.

Data collection: TTY, HKT.

Data analysis: JCE.

Drafting of manuscript: JCE, RMS.

Critical revision: YS, MEB, TTY, HKT, CL, JMH.

Research Registration Unique Identifying Number (UIN)

Name of the registry: Clinical Trials.gov

Unique Identifying number or registration ID: NCT05123495

Hyperlink to your specific registration (must be publicly accessible and will be checked): https://www.clinicaltrials.gov/ct2/show/NCT05123495?term=etheridge&draw=2&rank=1.

Guarantor

James Etheridge.

Data statement

Research data from this study are not available for sharing.

Declaration of competing interest

Christine Lim is the Head of International Safety and Policy, Johnson and Johnson Medical Devices.

Appendix A. Supplementary data

The following is the Supplementary data to this article: