Identifying Non-Technical Skill Deficits in Trainees through Interdisciplinary Trauma Simulation

Abstract

Objective

The goal of this study was to investigate non-technical skills in a simulated trauma setting both before and after a debriefing session in order to better understand areas to target for the development of educational interventions.

Design

Wilcoxon Signed Ranks Tests were used to compare scores on the five domains of the T-NOTECHS pre and post debriefings. A qualitative analysis utilizing the PEARLS debriefing framework was performed to provide a rich description of the strategies used by the debriefing facilitators.

Setting

The Joint Trauma Simulation Program is an interdisciplinary project designed to improve the quality of trauma care through simulation exercises emphasizing non-technical skills development.

Participants

Thirteen teams of five trauma trainees participated in trauma resuscitation simulations: a surgical chief resident, a surgical junior resident, an emergency medicine resident, and two emergency medicine nurses.

Results

Teams significantly improved on Communication and Interaction skills in the simulation scenarios from pre to post debriefing. The debrief facilitators spent the majority of their time engaged in Directive Performance Feedback (56.13%).

Conclusions

Interprofessional team simulation in trauma resuscitation scenarios followed by debriefing differently impacted individual non-technical skills domains. Additional facilitation strategies, such as Focused Facilitation and encouraging Learner Self-Assessment, may target other non-technical skills in different ways.

Introduction

The majority of errors in trauma resuscitation occur during the initial assessment.^1,2 Trauma is the leading cause of death in patients younger than 44, and prior work has estimated approximately a tenth of these deaths to be related to preventable errors.¹ Because of the complex, time-critical, and high-risk nature of trauma initial assessment, errors of non-technical skill, predominate over errors of technical skill.² Development of non-technical skills during trauma education is essential to improve trauma outcomes,³ and previous studies have found evidence that training in non-technical skills can improve team performance.^4,5,6,7

One widely accepted instrument used to assess non-technical skills in trauma settings is the modified non-technical skills scale for trauma (T-NOTECHS). The T-NOTECHS assesses performance in five domains: 1) Leadership, 2) Cooperation and Resource Management, 3) Communication and Interaction, 4) Assessment and Decision Making, and 5) Situational Awareness and Stress Coping, and scores have been found to be correlated with improvements in clinical practice.⁸ However, there is a paucity of research investigating how educational interventions may impact improvement in the individual domains of the T-NOTECHS and the effect on trauma team performance.

The interdisciplinary nature of trauma resuscitation can present challenges for the implementation of non-technical skills, especially given that each team member has a specialized role to play in the process. The cognition of the team as a unit, as reflected in their communication and other teamwork skills, is important for mitigating non-technical errors and providing optimal patient care. In an attempt to address this issue, interprofessional interventions have been found to positively impact nontechnical skills.⁹

Along with interprofessional education, simulation training has also been found to confer benefits for non-technical skills,^10,11,12 and even brief interventions can help to facilitate non-technical skills performance.¹³ Given that there is rarely time for reflection during a trauma resuscitation, timely debriefing following an event supports learners in critically reflecting on their performance and areas for improvement.^14,15 Simulation-based teamwork training combined with debriefing has been found to improve both teamwork and clinical performance of interdisciplinary trauma teams.¹⁶

As discussed, interprofessional interventions and simulation-based training can improve non-technical skills. However, although non-technical skills are often discussed as a single construct, they actually encompass a variety of abilities related to such things as communication, decision-making, and interpersonal relationships. As such, debriefing may facilitate these skills in different ways. Consequently, the goal of this study was to investigate non-technical skills in a simulated trauma setting both before and after a debriefing session in order to better understand areas to target for the development of educational interventions. Specifically, the questions we sought to address were:

1. On which non-technical skills do teams perform the strongest (e.g., decision-making, communication, leadership, cooperation or stress management) during a trauma resuscitation simulation?

2. Are improvements in non-technical skills observed immediately following a debriefing session? Does the debriefing appear to improve some skills more than others?

Material and methods

The Institutional Review Board at the University of Wisconsin-Madison determined this study to be an exempt program evaluation initiative. The University of Wisconsin Joint Trauma Simulation Program is an interdisciplinary project designed to improve the quality of trauma care through simulation exercises emphasizing non-technical skills development. The monthly simulations take place in a trauma resuscitation mock-up room, equipped with a high-fidelity manikin (Laerdal, SimMan 3G), advanced audio-visual streaming, capture and playback systems, and direct observation one-way mirrors. Thirteen teams of five trauma trainees participated in the simulations: a surgical chief resident (PGY 4), a surgical junior resident (PGY 2), emergency medicine resident, and two emergency medicine nurses. Eight standardized trauma scenarios have been developed (see Appendix). Some residents and nurses participate in more than one scenario, but the scenarios used are changed so that no team member performs the same scenario twice. The sessions are audio and video recorded. Three faculty facilitators from trauma surgery, emergency medicine and emergency medicine nursing conduct the scenarios, the prebrief and debriefings, and evaluate the teams’ performance using the T-NOTECHS.⁸

A formal prebrief is conducted with the whole team of trauma trainees prior to the simulation in order to orient them to the simulation environment and the goals of the simulation. Two standardized trauma simulations are performed at each hour long session, with a debriefing between scenarios. This debriefing session is held within a separate, private room to allow learners a chance to decompress and remove themselves from the scenario. Trainees leave the debriefings with areas to focus on for improvement of their team non-technical skills, established in conversation between the trainees and debrief facilitators during the debriefing session. After the debriefing, the team has a chance to practice the areas they identified as needing improvement in a second simulated scenario. Following the second scenario, a subsequent short debriefing is facilitated in order to reflect on any changes in performance.

After completion of each scenario, a T-NOTECHS score was used to evaluate overall simulation performance of the trainee team. Each of the five domains is graded on a 5-point Likert scale, with one indicating that the team did not demonstrate the behavior and five indicating faultless performance on this dimension of teamwork. Anchoring example behaviors are provided.

Analyses

Quantitative analyses were performed in order to better understand the teams’ performances on non-technical skills. T-NOTECHS ratings for 13 sessions were analyzed, for a total of 26 simulated scenarios, including both pre and post debriefing simulations. Descriptive statistics of the data were examined, and due to the non-normal distribution of the data, non-parametric tests were employed to examine the data for statistically significant differences. Wilcoxon Signed Ranks Tests were used to compare scores on the five domains of the T-NOTECHS pre and post debriefings.

A qualitative analysis was performed to provide a rich description of the strategies used by the debriefing facilitators. Videos of seven debriefings were available for this analysis and were transcribed verbatim. The PEARLS framework was used as a framework to better understand and describe the approach to debriefing use by the facilitators.¹⁷ The videos of the debriefing sessions were transcribed and the dialogue of the faculty debriefers was analyzed according to the teaching/learning strategy utilized in each turn of talk. The facilitators could use three broad categories of these educational strategies, 1) learner self-assessment; 2) focused facilitation; or 3) directive performance feedback (see Table 1). A research assistant coded each of the facilitators’ turns of talk in the seven transcripts, and then a senior member of the research team reviewed the transcripts for accuracy of coding.

Table 1.

Education strategies for teaching and learning in debriefing

Education Strategy	Definition
Learner Self-Assessment	The learner focuses on what he or she did well, poorly, right, wrong, etc.
Focused Facilitation	Promotes critical reflection and deeper understanding of events.
Directive Performance Feedback	Focused teaching that points out rights and wrongs directly.

Results

The three faculty raters demonstrated acceptable interrater reliability on the T-NOTECHS, with an intraclass correlation (ICC) = 0.63. Scores were then averaged across the three raters. The first quantitative analysis allowed for examination of differences and changes in non-technical skills across the simulation sessions. In order to control for Type 1 error, we used Holm’s sequential Bonferroni approach,^18,19 with alpha = .05. Pre versus post debriefing scores were compared for all five T-NOTECHS domain scores. There was significant improvement in the domain score of Communication and Interaction between the pre and post debriefing scenarios (Z = 12.83, p < .05, See Table 2 for means and standard deviations).

Table 2.

Mean (SD) of scores in the T-NOTECHS domains

T-NOTECHS Domains
	Leadership	Cooperation/Resource Management	Communication and Interaction	Assessment and Decision Making	Situation Awareness and Stress Coping
Scenario 1 (N = 13)	3.88 (.71)	4.17 (.66)	3.62 (.72)*	3.94 (.76)	4.03 (.70)
Scenario 2 (N = 13)	4.21 (.45)	4.44 (.28)	4.19 (.40)*	4.00 (.54)	4.14 (.54)

^*Significant difference pre to post debriefing after Holm-Bonferroni correction, p < .05

Based on the results of our first analysis, in order to better understand the differences in performance on Communication and Interaction in relation to the other T-NOTECHS domains, two families of follow-up comparisons were conducted comparing this domain to the other four domains on the T-NOTECHS scale both pre and post debriefing: Leadership, Cooperation/Resource Management, Assessment and Decision Making, and Situation Awareness and Stress Coping. These comparisons were done using Wilcoxon Signed Ranks Tests and used the same Holm’s sequential Bonferroni approach,^18,19 with alpha = .05 for each family of four comparisons. For the pre debriefing scenarios, Communication and Interaction was significantly lower than the other four domains. For the post debriefing scenarios, there were no significant differences between the domains. See Table 3 for means and standard deviations and Table 2 for Z test statistics and p-values.

Table 3.

Z test statistics (p-values) for comparison families pre and post debriefing

Comparisons of “Communication and Interaction” to other T-NOTECHS Domains
	Leadership	Cooperation/Resource Management	Assessment and Decision Making	Situation Awareness and Stress Coping
Pre Debriefing: Scenario 1 (N = 13)	−2.351 (.019)+	−2.680 (.007)+	−2.405 (.016)+	−2.699 (.007)+
Post Debriefing: Scenario 2 (N = 13)	−.070 (.943)	−2.111 (.035)	−.770 (.442)	−.154 (.878)

⁺Significant difference after Holm-Bonferroni correction

The codes for the facilitators’ turns of talk were averaged across the seven transcripts. The qualitative analysis revealed that the debrief facilitators spent the majority of their time engaged in Directive Performance Feedback (56.13%) with about a third of their turns of talk (36.64%) engaged in Focused Facilitation, and less than a tenth of their statements (7.23%) focused on eliciting Learner Self-Assessment (see Figure 1).

Figure 1.

Average Percentage of Facilitation Strategies Used by Debriefing Faculty Leaders

Discussion

Non-technical errors during the trauma initial assessment contribute to trauma-related deaths and training in these non-technical skills is fundamental to improving trauma outcomes.^1,2,3 This study used simulation training combined with debriefing to further investigate teams of trauma trainees’ non-technical skills performance and their improvement. Our study found that teams significantly improved on Communication and Interaction skills in the scenarios from pre to post debriefing. Although this was the weakest domain in the first scenario, teams did just as well in this area as they did in the other four domains post debriefing. Teams performed relatively strongly in the other four domains of Leadership, Cooperation/Resource Management, Assessment and Decision Making, and Situation Awareness and Stress Coping, with average scores close to four on the Likert scale even pre debriefing. One possible explanation for this finding is that since Communication and Interaction was the weakest domain in the first simulation, debriefers targeted this as a primary area for improvement in the goals that they set and performance feedback they gave.

Despite overall good performance on the four domains of non-technical skills discussed above, there was still opportunity for improvement in teams’ utilization of the skills. Although performance in all domains increased during the second simulation, the only area with significant improvement was Communication and Interaction, in which performance increased from 3.62 to 4.19. Essentially, teams changed their communication patterns from more unorganized and not always going through the team leader or being relayed to the scribe towards communication that was driven by and directed towards the team leader and relayed to the scribe. Differences in T-NOTECHS have been previously correlated with improvement in trauma care outcomes.¹⁷ However, we have not compared the T-NOTECHS differences found in our study to other outcomes. This will be a subject of future studies.

Teams did not consistently score fives in any of the non-technical skills domains. The best performance was in Cooperation and Resource Management, with an average score of 4.44 in the simulation after the first debriefing. Leadership also increased to the second highest score of 4.21 during the second simulation. Perhaps, the team leader was trying to facilitate communication by instigating a closer monitoring of the actions of team members and the ways in which the team distributed resources and tasks while caring for the patient. This may help to address errors due to non-technical skills by better distributing the resuscitation process among team members, allowing for increased cooperation and improved interaction.

The debriefing allowed learners to gain insight on other team members’ performance, attitudes, works styles, and roles and responsibilities within a trauma in order to better understand their point of view in providing care for the patient.¹⁶ Discussions from the debriefing seem to support this idea, particularly from an interprofessional context. A frequent topic of discussion in the debriefings was offering up critical information about a patient, even if it did not fit within the typical role or duties of the team member. The interdisciplinary make-up of the team as well as the facilitation of the team leader were cited as important factors in allowing team members to step out of their typical roles in service of patient care and safety. As such, as found in prior studies reporting that timely debriefing is an effective method for facilitating non-technical skill improvement,¹⁴ the debriefing allowed learners a space to reflect on and discuss their roles and responsibilities as an interdisciplinary team whose members may not be familiar with each other.

The results of our study lend support to the findings of previous work that has found benefits of non-technical skills training for improving team performance in trauma situations.^3,4,5 Moreover, although not all improvements were significant, our findings of improvement in non-technical skills confirm prior studies that have found that simulation training improves non-technical skills, even with brief interventions.^7,8,9,12 The goal of this study was to investigate non-technical skills in a simulated trauma setting in order to better understand areas to target for the development of educational interventions. We are able to make some recommendations based on our findings. First, it is essential that educational programs designed to target non-technical skills improvement create a space for discussion of the tasks and responsibilities distributed among interdisciplinary team members. Second, although learner-focused debriefing allows learners to reflect deeply on their performance and set goals for improvement that are meaningful based on their experience, it is also important that simulation instructors and facilitators structure the learning environment so that goals come out around all aspects of non-technical skills. As our results indicate, not all of the teamwork domains targeted in the T-NOTECHS improved across simulations, and this may be due to a lack of focus on these skills during reflection and goal setting during the debriefing. Lastly, the facilitation strategies used by the debriefers shape the tone, content, and focus of the debriefing. Using more learner-focused debriefing strategies may change the ways in which non-technical skills are targeted for improvement during the debriefing sessions. For example, we have learned from this analysis that the ways in which our debriefings are conducted seem to target communication without as much emphasis on the other non-technical skills. We can now work on adopting facilitation styles that allow for more learner reflection on a breadth of non-technical skills.

Limitations and Future Work

Scores on the T-NOTECHS were near the high end of the scale for many of the domains, approximately four or greater, which may have caused ceiling effects to interfere with the goal of better understanding areas to target for non-technical skill improvement. Subsequent studies will employ additional raters to score the videos of the trauma simulations. Further, our debriefings were more facilitator directed than we would have liked. To address this issue, we have been working to incorporate a tool to guide the debriefing of the trauma teams into the debriefing sessions. Future work will include the piloting of this tool with other interdisciplinary faculty in surgery, emergency medicine, and nursing to better understand how to support both novice and more experienced debriefers in the trauma simulation context. Finally, future research will focus on identifying specific elements of discourse that impact non-technical skills in trauma resuscitation and how these elements are used, both by team members in different roles as well as by the trauma team as a whole, in order to facilitate team interactions and dynamics that optimize patient care. As non-technical skills were only measured in a simulation environment to which learners were exposed for a relatively short time, we will also work to investigate whether changes in these skills are maintained in an actual trauma resuscitation setting.

Conclusions

Interprofessional team simulation in trauma resuscitation scenarios followed by debriefing differently impacted individual non-technical skills domains. The only domain to significantly improve was Communication and Interaction. The debriefings were primarily focused on directive performance feedback. Additional facilitation strategies may target other non-technical skills in different ways.