Virtual non-technical skills assessment training is an effective, scalable approach for novice raters

Abstract

OBJECTIVES

The COVID-19 pandemic has forced a creative transition to virtual platforms due to physical distancing and travel restrictions. We designed and tested a highly scalable virtual training curriculum for novice raters using the Oxford NOTECHS non-technical skills rating system.

DESIGN

A three-day training course comprising virtual didactics, virtually facilitated simulations, and independent live observations was implemented. NOTECHS scores were submitted for eleven standardized video simulations and four live operations. Intraclass correlation coefficients (ICCs) were calculated for total NOTECHS scores and subcomponent scores. Raters previously trained in-person with the same standardized videos served as a comparator group for equivalence testing.

SETTING

All study activities were conducted in a large academic tertiary referral center in Singapore as part of an ongoing surgical safety initiative.

PARTICIPANTS

Seven staff members underwent training (three virtually and four in-person).  None had prior surgical experience or non-technical skills assessment training.

RESULTS

ICCs for total NOTECHS scores were 0.85 (95% CI, 0.73-0.98) for virtually trained raters and 0.83 for those trained in-person (95% CI, 0.68-0.99).  Scores were equivalent between groups within a 10% margin.

CONCLUSIONS

Non-technical skills assessment can be reliably taught in a highly scalable virtual format. Virtual NOTECHS training is a valuable tool for educational and quality improvement initiatives during the COVID-19 pandemic and for centers that lack ready access to onsite non-technical skills training expertise.

Introduction

Non-technical skills development is increasingly recognized as critical to patient safety.1, 2, 3, 4 Many quality improvement initiatives and surgical training curricula have now embraced non-technical skills education.^{5 , 6} The Accreditation Council for Graduate Medical Education includes Interpersonal and Communication Skills as one of its six Core Competencies for resident education, and non-technical skills training is specifically identified as a curriculum resource in the Surgery Milestones Supplemental Guide.^{7 , 8}

Evaluating interventions to improve non-technical skills requires high-fidelity assessment tools. The most common behavioral marker system for surgical teams is the Oxford NOTECHS system.⁹ The NOTECHS rating system, which assesses leadership, teamwork, problem solving, and situational awareness for each subteam in the operating room (OR), is well-validated with excellent inter-rater and test-retest reliability.^{10 , 11} However, NOTECHS generally requires expert raters; the majority of studies relied on attending surgeons.¹²

Previous work from our group established the feasibility of a NOTECHS training program for novice raters, the need for on-site expert trainers limits scalability.¹³ The COVID-19 pandemic has underscored the need for robust virtual approaches to education, which can also benefit lower-resourced institutions outside of pandemic conditions. We therefore developed and tested a virtual NOTECHS training program for novice raters.

Materials and Methods

Setting and participants

The virtual NOTECHS training curriculum was created as part of a surgical safety initiative in Singapore. Ethical approval was obtained from the Harvard T.H. Chan School of Public Health Institutional Review Board and waived by the SingHealth Centralized Institutional Review Board. All training activities took place at an academic tertiary referral center in Singapore. Virtual training sessions were conducted in January 2021. In-person training conducted at the same center in May 2019 provided a comparator group.¹³ All participants were female, and none had prior OR experience or non-technical skills assessment training (Table 1 ).

Table 1.

Participant Characteristics

	Virtual training (n = 3)	In-person training (n = 4)
Female	3 (100%)	4 (100%)
Primary role
Nurse	2 (67%)	2 (50%)
Research associate	1 (33%)	1 (25%)
Speech therapist	0	1 (25%)
Years of clinical experience
None	1 (33%)	1 (25%)
1-5 years	0	0
6-10 years	2 (67%)	2 (50%)
>10 years	0	1 (25%)
Prior OR experience	0	0

OR, operating room

Curriculum

Adapting our training program to a virtual format presented several challenges. The original training program relied on expert facilitators accompanying participants for live observations. We were concerned that video simulations alone would not adequately prepare participants. As with any virtual program, we also worried about maintaining engagement without a physical presence. Finally, due to staffing restrictions, we were forced to compress the program from five days into three.

Our adapted curriculum is shown in Figure 1 . Two experienced facilitators led each virtual session. We first introduced non-technical skills, NOTECHS categories, and instructions for scoring OR teams. We cut all extraneous background material to prioritize practice time and added prompts for participants to share their experiences to make the session as interactive as possible.

FIGURE 1.

Virtual NOTECHS training curriculum.

Simulated observations provided opportunities for practice and iterative improvement. Video simulations were drawn from the widely available eNOTSS platform, which provided a common basis to compare ratings (http://enotss.herokuapp.com/).¹⁴ Participants began with four training videos, each representing one NOTECHS category. Group discussions after each video focused on behaviors that positively or negatively impacted NOTECHS scores. Facilitators provided feedback on scores and identified additional behaviors not recognized by participants. Training videos also provided an opportunity to set the expectation that participants provide thoughtful scores with specific examples. A total of eleven formally scored simulations were conducted over the course of two sessions. Scores were recorded and group debriefings were held after each video simulation.

Live OR observations were the most challenging to convert to the virtual format. Previously, facilitators had escorted participants to the OR, introduced them, and observed operations together, debriefing immediately thereafter. We added a brief session during didactics to practice introductions, and a senior OR nurse guided participants to appropriate cases. However, there was no way for a facilitator to accompany participants to the OR. Participants instead conducted observations as a group, recording their responses independently. Each participant also took detailed notes. Because of the time difference between facilitators and participants, debriefing sessions for live observations could not be held until the subsequent morning. However, this delay did not appreciably affect recall, particularly given the quality of notes taken by participants.

Analysis

Inter-rater reliability was assessed with intraclass correlation coefficients (ICCs) for each pair of raters for all simulated or live OR encounters. Generalized estimating equations were used to derive a common point estimate and confidence interval for NOTECHS scores.¹⁵ Equivalence between training methods was assessed using the 90% confidence interval of the mean difference between scores. A 10% margin of equivalence was set a priori. All analyses were conducted using SAS v9.4 (SAS Institute Inc, Cary, NC).

Results

Virtually trained raters achieved excellent inter-rater reliability. The ICC for total NOTECHS ratings was 0.85 (95% CI, 0.73–0.98), comparing favorably to that of the in-person training group (0.83; 95% CI, 0.68–0.99). Scores for all subcomponents also demonstrated excellent ICCs (Fig. 2 ). Total and subcomponent ICCs did not differ significantly between virtual and in-person training groups. NOTECHS scores were equivalent between groups (90% CI, -8.2%–5.1%).

FIGURE 2.

Inter-rater reliability for NOTECHS scores.

Conclusions

The virtual NOTECHS training curriculum for novice raters was equivalent to in-person training, yielding excellent inter-rater reliability. Measuring non-technical skills with robust rating systems like NOTECHS is crucial to many educational and quality improvement initiatives. Given the effects of the COVID-19 pandemic on medical education and healthcare quality, these initiatives are more crucial than ever.^{16 , 17} However, these same conditions make it difficult to identify and train reliable OR observers. Travel bans and capacity restrictions make live training impossible for many centers. Existing virtual platforms such as eNOTSS are useful, but they require baseline knowledge and experience in the OR that novice raters lack. With millions of backlogged surgical cases, few centers can dedicate experienced surgical personnel to such resource intensive work.¹⁸ The ability to train novice raters using a virtual format is therefore critical.

It is tempting to further streamline the curriculum described here. Groups may consider eliminating the need for expert facilitators or live OR observations. We would caution against such steps. Novice raters require calibration of expectations and perceptions through debriefing with experienced facilitators to provide reliable data. They are prone to misperception of actions in the OR which were easily corrected in debriefing. In the absence of prior OR experience, raters were also somewhat apprehensive about working in this environment; group observations provided an easy transition in addition to a valuable training experience.

This study was limited by its small sample size and single center design. Our participants were highly engaged and interactive. As the COVID-19 pandemic abates and virtual interaction becomes less commonplace, achieving this level of engagement may become more difficult. Furthermore, the academic center in which this curriculum was tested is poorly representative of the institutions most likely to benefit from virtual NOTECHS training. Nonetheless, this study provides proof of concept for a highly scalable and effective means of training novice raters to assess OR team non-technical skills.

Disclosures

This work was supported by Medical Devices and Diagnostic Global Services, LLC.