Pilot One-Hour Multidisciplinary Team Training Simulation Intervention in the Operating Room Improves Team Nontechnical Skills

Abstract

Background:

The primary objective of this study was to determine if a 1-hour simulation-based training with interdisciplinary operating room (OR) teams could improve nontechnical skills of the providers as assessed by the nontechnical skills tool (NOTECHS II).

Methods:

Interprofessional otolaryngology OR teams consisting of surgery faculty and resident, anesthesiology faculty and resident, and OR nurses, scrub technician, and perioperative technician underwent a 1-hour simulation-based intervention in the OR. The teams were rated on their nontechnical skills during the intervention and throughout the clinical day following. They also completed self-reflection surveys (SRS) before the intervention and in 3 intervals after the intervention (immediately following the intervention, at the end of their shift on the day of the intervention, and again 2 weeks later).

Results:

Four interprofessional teams with a total of 26 unique participants participated in this pilot program. Team nontechnical skills, assessed using NOTECHS II, improved from the first simulation to the second simulation during the intervention. Team NOTECHS II scores remained higher throughout the clinical day. Individual self-reflection scores (SRS) followed the same trend.

Conclusions:

On-site interprofessional OR team training simulation can take place in a brief time period that is dedicated for education. A brief intervention resulted in improved team nontechnical scores when assessed following intervention. In addition, participants found the intervention to be effective and beneficial to their learning.

Introduction

Errors in communication are the cause of an estimated 70% of adverse events during the perioperative period.^1,2 Interprofessional team training has been shown to vastly improve communication and even reduce the incidence and severity of adverse events during this high-risk period.^3,4 Improving teamwork and communication is necessary and highly effective for all personnel involved in the perioperative care of patients.⁵ While interprofessional training in communication in the perioperative environment is important, how to make such training effective—but logistically feasible—is still a challenge.

There is substantial evidence supporting the benefits of incorporating team training. Many studies have been able to show the effect of interprofessional simulation-based team training on patient safety, quality improvement, and even patient outcomes.^3,6–10 Team training on specific learning points, such as World Health Organization checklists, has been shown to increase adherence to guidelines.^9,11 However, most of these studies have incorporated simulation-based trainings that last more than 1 hour, require operating rooms to close for an extended period or are located off-site, making it difficult to achieve a high-fidelity experience since participants were not in the environment in which the team typically performs together. In 2017, Gillespie et al¹² were able to show that when surgical teams viewed a 1-hour video, their non-technical performances improved. While this study did show some early evidence that shorter trainings may have benefit, it did not have the potential added benefits of a truly interprofessional team training intervention.

Few institutions have requirements for training or assessment regarding team interactions during crisis situations. At most institutions, it is difficult to find the time and resources to get all interprofessional members out of the operating room (OR) for training or assessment. Also, it may not feasible to close the ORs to implement team training sessions.

The purpose of this study was to develop a pilot program evaluating the feasibility and efficacy of a 1-hour interprofessional simulation training in crisis resource management that takes place in the OR, called OR Team Training (ORTT), during regularly scheduled education time. The primary hypothesis was that team members who underwent the ORTT intervention would improve their nontechnical skills during the intervention and throughout the day of the intervention.

Materials and Methods

Simulation Intervention

The intervention occurred on site before the first case started in the OR in which the participants were scheduled to work that day. Every Thursday morning, the first cases started 1 hour later to allow each department to hold its own educational conferences. The intervention was scheduled during that time. There was a SimMan 3G (Laerdal Medical, Stavanger, Norway) in the OR for the simulation. The OR was prepared by the research team to resemble an actual case.

Participation in this study was intended for members of the interprofessional OR team of head and neck surgery in an adult hospital at a Level I Trauma Center. All were eligible: otolaryngology faculty (OtoF), otolaryngology residents (OtoR), anesthesiology faculty (AnesF), anesthesiology residents (AnesR), nurses (RN), surgical scrub technicians (ST), and perioperative technicians (PT) were assigned to the head/neck team during the study period. Providers were excluded from participating if they were unable to attend the intervention due to scheduling conflicts or were previously involved in facilitating simulation education for crisis resource management. Each provider participated in 1 intervention. All participants had the opportunity to decline to participate at any point in the study. All eligible participants received a verbal introduction to the study as well as written communication via direct email regarding the details of the study. For each intervention session, at least 6 participants were present. Because this was a pilot study, it was determined by the study team that analyzing 4 completely unique OR teams would serve as an appropriate and convenient study sample.

The leaders of the simulation were faculty members from anesthesiology and otolaryngology (LRR and KMM). The intervention started with the anesthesiology faculty member introducing the simulation exercise and allowing all participants to introduce themselves. After 10 minutes for introductions and briefing, Simulation 1 started with no training or further instruction on the nontechnical skills needed to handle crisis situations. Simulation 1 concluded after 10 minutes. Debriefing was then facilitated by anesthesiology and otolaryngology faculty, each with over 10 years' experience in leading team debriefings. Debrief 1 focused mostly on methods to enhance communication, such as use of direct and closed-loop communication. The participants were then asked to go through the simulation again (Simulation 2) using the techniques of enhanced communication covered in Debrief 1. Simulation 2, which also lasted 10 minutes, had another debrief, Debrief 2, where the group was asked to discuss how they felt the scenario went and where they saw improvements in their communication skills. The debriefs were both 10 minutes, leaving a remaining 5 to 10 minutes at the conclusion for a summary of the debriefing points and any additional questions from the participants.

The clinical content of the simulation scenario was a common adverse event wherein the patient desaturates and has a difficult airway, making intubation challenging. A scalp laceration closure under local anesthesia with sedation was used so that the surgeons would be working at the head of the bed, with anesthesiology and surgery discussing the inevitable: determining who would be actively managing the airway. Available literature and team member expertise were used to develop this high-yield simulation scenario with clear objectives for each discipline involved.¹⁰ Published curricula on this topic were reviewed for assistance in development.^7,13 Clinical content was agreed upon by the planning leads representing all 3 departments (anesthesiology, otolaryngology, and nursing; LRR, KMM, and HC).

Full scenario details are available in Appendix A. To summarize, the patient was a 76-year-old man undergoing closure of scalp lacerations under monitored anesthesia care. There were 2 incisions to close: 1 anteriorly on the forehead, 1 posteriorly on the scalp. The OtoF and OtoR teams were asked to physically suture a simulated wound on the mannequin while the RN and ST counted instruments before closure was completed. The AnesR was in the room independently with the AnesF waiting outside of the OR to be called in if necessary. Leading up to the beginning of the scenario, the patient was doing well with sedation, propofol infusion, and supplemental oxygen provided by nasal cannula. Approximately 1 minute into the scenario, the patient began to desaturate. The patient was not responsive to any initial maneuvers by the team and continued to desaturate to 60%. The difficult airway settings (tongue and pharyngeal swelling) were in place on the model, making it more challenging for the anesthesia team to intubate.

Assessment

There were 2 types of assessment used to determine the effectiveness of the intervention. First, there were trained observers who assessed participants' nontechnical skills using the Non-Technical Skills II (NOTECHS II) tool shown in Tables 1 and 2. The NOTECHS II was selected because one of the authors (LRR) received extensive rater training for NOTECHS II and was able to use that experience to train the other raters. The second form used was a Self-Reflections Survey (SRS) completed by participants of the intervention. This SRS was chosen for its simplicity and ease of using both pre- and postintervention. Figure 1 describes the assessments and their timing.

Table 1.

Non-technical Skills (NOTECHS) assessment tool

Domain	Behavioral Markers
Leadership and management	Leadership Maintenance of Standards Planning and preparation Workload management Authority and assertiveness
Teamwork and cooperation	Team building/maintaining Support of others Understanding team needs Conflict solving
Problem-solving and decision-making	Definition and diagnosis Option generation Risk assessment Outcome review
Situation awareness	Notice Understand Think ahead

Table 2.

Non-technical Skills (NOTECHS) II behavioral parameters

Behavior	Frequency	NOTECHS II Score
Compromises patient safety and effective team work	Consistently	1
	Inconsistently	2
Could directly compromise patient safety and effective team work	Consistently	3
	Inconsistently	4
Maintains an effective level of patient safety and teamwork	Inconsistently	5
	Consistently	6
Enhances patient safety and effective teamwork	Inconsistently	7
	Consistently	8

Figure 1.

Timeline of NOTECHS II assessment and self-reflections.

Objective Assessment Using NOTECHS II

The primary hypothesis of this study was that following simulation-based training, the teams would improve their nontechnical skills as assessed by trained observers using NOTECHS II. The categories in NOTECHS II include leadership and management, teamwork and cooperation, problem-solving and decision-making, and situation awareness.^14–17 Each intervention group received one NOTECHS II score for Simulation 1 and a separate score for Simulation 2. The same observer then remained with the intervention group through the day during their regularly scheduled cases.

NOTECHS II assessments were made at various points during the case when communication was necessary, such as during the preinduction verification, preincision time out, hand-offs of care, and any clinically significant event or point in the procedure when communication across teams was necessary.

Rater Training

Two senior AnesR underwent training to use the NOTECHS II to score observed communication in the OR. Raters underwent training with authors (LRR, LZ) in how to use the NOTECHS II tool by watching videos of intraoperative simulations previously recorded for another educational program. The authors and raters discussed what constituted acceptable and unacceptable responses and worked until there was appropriate agreement.¹⁸

Self-reflection Survey (SRS)

The secondary hypothesis for this project was that each participant would improve their SRS scores over the course of the intervention and 2-week follow-up. Each participant completed the SRS prior to the intervention and at 3 separate points postintervention. This survey was developed based on a previously validated measure used in similar settings and was designed to assess the participants' views on the communication and teamwork in the OR.²⁰ The changes to the survey consisted of adding questions to evaluate the simulation session and did not change any of the existing questions from the original validated measure. The day preceding the intervention, each participant was asked to complete the SRS online via Qualtrics. If the participant did not complete the day prior, they were given a paper version to complete when they arrived for the intervention. The survey is available in Appendix B. Postintervention surveys were completed immediately following the conclusion of the intervention, at the end of their shift on the day of the intervention, and again 2 weeks later. The postintervention SRS is available in Appendix C. Questions were added to the postintervention SRS to allow the participants to provide feedback regarding the intervention and identify any personal commitments to change their practice. Once the final SRS was completed, participants were provided with $5 to use at the hospital cafeteria.

This study was determined to be exempt from full IRB review by University of Michigan IRB (HUM00133930) as it involved the evaluation of normal educational practices in an educational setting.

Statistical Analysis

To explore how the NOTECHS II and SRS scores change across evaluation time points, linear models using Generalized Estimating Equations (GEE) were fit to account for the clustering within each team across the varying time points, for each assessment tool. The NOTECHS II was used to assess objective improvement in the teamwork skills after training simulations. The SRS assessed the extent to which the training simulations improved the perception of teamwork by individual participants. Using GEE models, a stratified analysis examined how profession (anesthesia, surgery, or other) affected changes in the SRS scores. Summary statistics of mean and standard deviation were also calculated as appropriate. A significance level of 0.05 was used throughout. All analyses were performed using SAS software (version 9.4; SAS Institute Inc, Cary, NC, USA).

Results

Four intervention sessions were conducted on 4 separate days. Each intervention had unique team members from each subgroup. Three of the 4 interventions included a PT. The total number of participants was 31.

The average NOTECHS II score during Simulation 1 was 21.8 (± 2.1), increased to 24.8 (± 2.2) during Simulation 2, and slightly decreased from Simulation 2 to 23.8 (± 5.1) postintervention (Figure 2). All teams earned higher scores during postintervention than during Simulation 1. However, these changes (assessed by slope estimates) were not statistically significantly different from zero, as estimated by the GEE model. The approximate change in score at Simulation 2 and corresponding 95% confidence interval, relative to Simulation 1, was 3 (−3.16 to 9.16), P value = .3650. The approximate change in score postintervention and corresponding 95% CI, relative to Simulation 1, was 1.7 (−3.82 to 7.23), P value = .5607.

Figure 2.

NOTECHS II scores.

Breaking down the NOTECHS II tool by question, the average scores for the team-work and cooperation, problem-solving and decision-making, and situation awareness questions followed a trend similar to that of the total NOTECHS II overall scores: increase in score during Simulation 2 followed by a decrease postintervention that is still higher than the score during Simulation 1. The average leadership and management scores had increasing trend at both time points following Simulation 1 with scores of 5.3 (± 1.0), 5.8 (± 0.5), and 6.0 (± 1.3) during Simulation 1, during Simulation 2 and postintervention, respectively.

Twenty-seven participants completed 4 SRS (1 participant had 1 missing survey) for a total of 107 surveys. The range of scores at each evaluation point can be seen in Figure 3. The observed mean (± standard deviation) for each time point were preintervention: 4.11 (± 0.84); first follow-up right after the intervention: 4.68 (± 0.64); second follow-up at the end of day: 4.94 (± 0.11) and the 2-week follow-up: 4.60 (± 0.62). Using the GEE model fit to account for clustering within team, the average survey score increased right after intervention by an average of 0.57 (95% CI: 0.25 to 0.89, P value = .0175) above preintervention, peaked at the end of the day with a mean increase of 0.81 (95% CI: 0.36 to 1.25, P value = .0150) above preintervention, and then slightly decreased at 2-week follow-up with an average increase in 0.49 (95% CI: 0.13 to 0.84, P value = .0426) relative to preintervention scores. All comparisons results were statistically significant in comparison to the pre-intervention scores at alpha = 0.05.

Figure 3.

Self-reflection scores.

The type of profession affected SRS scores. AnesF and AnesR had the following average changes in scores, relative to preintervention: right after intervention: 0.97 (95% CI: 0.39 to 1.54); end of day of intervention: 1.09 (95% CI: 0.37 to 1.82); and 2-week follow-up: 1.01 (95% CI: 0.38 to 1.64). OtoF and OtoR had the following average changes in scores, relative to preintervention: right after intervention 0.32 (95% CI: −0.26 to 0.9), end of day of intervention 0.62 (95% CI: −0.13 to 1.36); and two-week follow-up 0.49 (95% CI: −0.14 to 1.12). Lastly, the other profession group (RN, ST, PT) had the following average changes in scores, relative to preintervention: right after intervention 0.52 (95% CI: 0.07 to 0.98); end of day of intervention 0.31 (95% CI: −0.62 to 1.23); and 2-week follow-up 0.03 (95% CI: −0.48 to 0.53).

Qualitative comments from the participants were also reviewed. Many participants stated that they felt the training was helpful and would like to participate in future interventions. A few participants also mentioned that they enjoyed working in a fixed team for the day after working as a team during the intervention. One participant noted they would be less afraid to speak up in the future.

Discussion

In this study, we developed and tested a 1-hour on-site interprofessional team training simulation intervention. The primary objective of this study was to show that team members who participate in an interdisciplinary ORTT intervention would improve their nontechnical skills following the intervention. The results from this project supported the theory that the simulation-based team training improved overall team nontechnical skills after a single 1-hour intervention. The intervention was on-site in the OR with minimal effect on the regular function of the OR. The intervention was successfully completed during a time each department already allocates to weekly education, thereby increasing likelihood of future support from department leaders and OR management. Keeping this intervention short and during a time already set aside for educational meetings, this intervention provides an educational experience without further burdening a population already suffering from burnout.

While many previous studies have been able to show a positive effect on teamwork and communication following team training, much of this work has been focused on training occurring outside of the ORs and over multiple educational sessions.^21–23 However, recent surveys suggest that the largest barriers to implementing more comprehensive simulation-based training include time and resources.^24,25 This work adds to the current literature by demonstrating that even a brief intervention may be beneficial and can be completed within the OR during time already set aside for education. Having the training happen in the actual setting of regular practice increases the authenticity for the participants, making it easier to transfer skills to real emergencies and possibly to identify latent systems issues.^26,27

Despite the power limitations from the small sample size for this project, we note that the overall NOTECHS II scores for the groups improved from Simulation 1 to Simulation 2, and then remained higher throughout the working day. When broken down by individual NOTECHS II categories, the highest improvement was seen in leadership and management. This finding may be due to the focus on communication skills during the debriefing sessions. Enhanced communication skills may translate into better leadership. The other NOTECHS II categories showed slight improvement as well. This suggests that even with a focused debriefing, collateral effects of training and practice may improve other aspects of team nontechnical skills.

The raters used were senior anesthesiology residents. The study team felt it was important to have raters with the appropriate clinical knowledge and the ability to identify critical events as they happened throughout the day in the OR. The same rater was present during the intervention and throughout the clinical day following. Therefore, they were not blinded to the participants' performance in the intervention. Along with the Hawthorne effect, this presents some limitations to the raters' observations.

Regarding the SRS scores, there was significant improvement in total scores from the preintervention to postintervention. One of the goals of this project was to develop a curriculum that was not only effective but also perceived by the participants as valuable. These subjective SRS results suggest that participants felt the intervention was valuable and retained this belief through to the 2-week follow-up. When analyzing the SRS scores by subgroup, the anesthesiology group had the greatest increase in mean scores at each time point. This finding may be due to perceived benefit from the training that the anesthesiology team members interpreted when compared with the other subgroups, but further research needs to be done to determine why there was a difference in the SRS scores for different groups.

In addition, while the data could have been substratified based on role, the sample size was not large enough to power that type of analysis.

Comments from the SRS emphasized comfort level and communication with team members with whom they are familiar and have worked previously. Studies have shown that familiarity with team members not only improves team performance but also can reduce morbidity.²⁸ With the continuously changing healthcare team, especially in large institutions, there are many situations where providers find themselves working with others they have never met before. One participant noted that they would be less afraid to speak up with any concerns in the future. This may indicate increased comfort with providers through the intervention. The concept of fear in communication is an interesting one that warrants further investigation.

The limitations of this study may be attributed to the fact that this was performed at a single center with a single surgical department and additional participants specialized in that surgical area. These teams are already somewhat familiar with one another, which may have resulted in false positive correlations. Also, the study days were limited to 4 interventions. Team members were randomly assigned to intervention based only on OR scheduling availability. Participant demographics and simulation experience were not controlled for when assigned to a team. Because this was a pilot of an intervention, the sample size was small and it was difficult to assess significant changes. NOTECHS II data were captured for each team by a single rater, and the rater was not blinded to which groups received the intervention and which were the control. This could have biased the raters' results. In addition, participants were not blinded to the fact that they were being observed. Since they knew they were being evaluated, they may have been paying closer attention to their teamwork and communication. Due to the nature of the OR schedule, there were periods when team members who did not participate in the intervention were in the OR during the clinical observation day.

In this project, we were able to develop an interprofessional team training intervention that took place in the OR. The research team has a goal to replicate this team training and expand to additional interprofessional teams. The hope is to accomplish larger goals such as increasing participation from other departments while maintaining the high authenticity of the experience. With additional interventions, we hope to have more data to examine the benefit of this training as well as retention of skills learned.

Appendix

Appendix 1

ORTT Simulated Intraoperative Scenario

Patient description

• 76yo M undergoing excision Moh's procedure under MAC

• 2 incisions to close, one anteriorly on the left forehead and one on the left clavicle for cutaneous malignancy

• Bed is turned 90 degrees

• Oto resident and faculty are working on closing 1^st incision

• OR RN and ST are counting before closure is complete and it is OK to move on to 2^nd incision

• Anesthesia team is discussing add-on case

• Patient has been doing well under sedation with propofol infusion 80mcg/kg/min, did not receive midazolam, 50 mcg fentanyl, NC in place; anesthesia provider has not made any changes to sedation

• HTN on lisinopril; atrial fibrillation on Coumadin, held for 1 week; OSA, uses bipap at home; Weight= 120kg; 20g IV in the left hand

• Labs normal, vitals stable in preop

• No previous airway history available; thick neck, MP3, good extension, teeth intact

Progression of scenario

• Baseline vitals: HR 88, afib, BP 117/78, RR 12, Spo2 92, Sim Man obstructing, snoring, swollen tongue +EtCO2 on monitor, BP cycles q3 minutes; nasal cannula in place

• Time 2:00 – Spo2 decreases to 85 over next 30 seconds

• Time 3:30 – Spo2 decreases to 80 over next 30 seconds; difficult airway settings on sim man to include laryngeal changes, EtCO2 begins to decrease

• Further progression based on participants' management; All difficult airway settings in place

• Scenario stops at 10 minutes

Role of RN/Scrub:

• Counting instruments for closure of 1^st incision before moving on to 2^nd procedure

• To call out for staff stat if requested

Role of surgery:

• Continue closing first incision before preparing for 2^nd part of the procedure

Role of anesthesia:

• Resident and attending notified of an add-on case to follow at the end of the day

• Patient has been stable; there is one more incision during this case so need to maintain current level of anesthesia

OR Team Training Self-reflection

Q1 Date:

________________________________________________________________

Q2 Pager #

________________________________________________________________

Q3 Team:

• ○ Anesthesia

• ○ Nursing

• ○ Scrub

• ○ Surgery

• ○ Other ________________________________________________

Q4 Please indicate your agreement with the following about your work in the OR today

	Strongly agree (1)	Somewhat agree (2)	Neither agree nor disagree (3)	Somewhat disagree (4)	Strongly disagree (5)
Our interdisciplinary OR team discussed procedures before starting today	○	○	○	○	○
I felt everyone on the team was aware of what was happening	○	○	○	○	○
I felt comfortable giving feedback to other team members	○	○	○	○	○
I felt heard and understood in all important communications	○	○	○	○	○

Q1 Evaluation Time

• ○ Post-intervention (Right after intervention)

• ○ Post-intervention (End of the day of intervention)

• ○ Post-intervention (2-week follow-up)

--------------------------------------------------------------------------------------------------

Q2 Date:

________________________________________________________________

--------------------------------------------------------------------------------------------------

Q3 Pager #

________________________________________________________________

--------------------------------------------------------------------------------------------------

Q4 Team:

• ○ Anesthesia

• ○ Nursing

• ○ Scrub

• ○ Surgery

• ○ Other ________________________________________________

Q5 Please indicate your agreement with the following about your work in the OR today

	Strongly disagree	Somewhat disagree	Neither agree nor disagree	Somewhat agree	Strongly agree
Our interdisciplinary OR team discussed procedures before starting today	○	○	○	○	○
I felt everyone on the team was aware of what was happening	○	○	○	○	○
I felt comfortable giving feedback to other team members	○	○	○	○	○
I felt heard and understood in all important communications	○	○	○	○	○

Q6 How much do you agree with the following?

	Strongly disagree	Somewhat disagree	Neither agree nor disagree	Somewhat agree	Strongly agree
I thought this training was excellent	○	○	○	○	○
I believe this training will help me respond better in a crisis situation	○	○	○	○	○
I believe this training will help me communicate more clearly in the OR everyday	○	○	○	○	○
I plan to change my practice as a result of participating in this training	○	○	○	○	○

Q7 How much do you agree with the following?

	Strongly disagree	Somewhat disagree	Neither agree nor disagree	Somewhat agree	Strongly agree
I believe the training helped me communicate more clearly in the OR everyday	○	○	○	○	○
I was able to change my practice as a result of participating in the training	○	○	○	○	○

Q8 Please describe the changes you were able to make as a result of participating in this training.