Non-technical skills assessment in acute care trauma simulations: A mixed methods approach using eye tracking and behavioral marker systems

Structured Summary

Introduction

The condition of trauma patients and the urgent need for timely resuscitation presents unique challenges to trauma teams. These difficulties are exacerbated for military trauma teams in combat environments. Consequently, there is a need for continued improvement of non-technical skills (NTS) training for trauma teams. However, current approaches to NTS assessment rely on subjective ratings, which can introduce bias. Consequently, there is a need to identify objective methods of NTS evaluation. Eye-tracking (ET) methods have been applied to studying communication, situation awareness, and leadership in many healthcare settings, and could be applied to studying physicians’ NTS during trauma situations. In this study, we aimed to assess the relationship between trauma team leaders’ objective gaze patterns and subjective expert NTS ratings during patient care simulations.

Materials and Methods

After Institutional Review Board approval, nine trauma teams from first year post-graduate general surgery and emergency medicine residents were recruited to participate in one of two trauma simulations (a difficult airway case and a multi-patient trauma). Each scenario lasted approximately 15 minutes. All team leaders wore a mobile ET system (Tobii Pro Glasses 2, Tobii AM, Danderyd, Sweden) to evaluate gaze metrics – time to first fixation (TTFF), average fixation duration (AFD), and total percentage of the scenario (TPS) focused on Areas of Interest (AOI), which included patient, care team, diagnostic equipment, and patient care equipment. Trained faculty raters completed the Non-Technical Skills for Surgeons (NOTSS) assessment tool and the Trauma Non-Technical Skills (T-NOTECHS) scale. One way analysis of variance, Kruskal-Wallis, and appropriate post-hoc pairwise comparison tests were run to assess differences between ET metrics across AOI groups. Spearman’s Rho tests were used to assess correlations between ET and subjective NTS ratings.

Results

Compared to other NTS domains, trauma teams scored relatively poorly on communication across both T-NOTECHS (3.29±0.61, maximum=5) and NOTSS (2.87±0.66, maximum=4). We found significant differences in trauma team leaders’ TTFF between teammates and the patient (Team: 1.56 vs. Patient: 29.82 seconds, p<0.001). TTFF on the diagnostic equipment was negatively correlated (p<0.05) to multiple measures of subjective NTS assessments. There were no significant differences in AFD between AOIs, and AFD on teammates was positively correlated (p<0.05) to communication and teamwork. There were significant differences in TPS across most AOI pairs (p<0.05), and the average TPS fixated was highest on the patient (32%). Finally, there were several significant correlations between ET and NTS metrics.

Conclusions

This study utilized a mixed methods approach to assess trauma team leaders’ NTS in simulated acute care trauma simulations. Our results provide several objective insights into trauma team leaders’ NTS behaviors during patient care simulations. Such objective insights provide a more nuanced understanding of NTS behaviors and can be leveraged to guide NTS training of trauma physicians in the future. More studies are needed to apply these methods to capture NTS from a larger sample of teams in both simulated and real trauma environments.

1. Introduction

Timely and effective diagnosis and resuscitation of patients in trauma situations is imperative for positive patient outcomes.^1,2 This is particularly salient to military trauma teams, as the patient acuity and environmental demands (e.g., balancing multiple casualties, limited patient extraction opportunities) may be exacerbated compared to civilian settings.³ In order to achieve accurate diagnoses and impart effective interventions in the management of trauma patients, particularly in combat environments, effective teamwork is critical.⁴ Non-technical skills (NTS), which include constructs such as communication, situation awareness, leadership, and decision-making, are needed for effective surgical performance during all phases of care.⁵ In trauma care environments, poor communication has been identified as a significant risk factor for errors, and errors in NTS occur more often than technical errors in this environment.^6,7 Moreover, NTS failures are linked directly to technical mistakes and decreased patient safety.⁸ Given the importance of physicians’ NTS to patient care in trauma environments, it is necessary that effective assessment methods are adopted.

Multiple tools have been developed to evaluate surgeons’ and surgical teams’ NTS in surgical environments, including trauma situations. The Non-Technical Skills for Surgeons (NOTSS) assessment tool is an observer-based measure that evaluates surgeons’ leadership, decision-making, situation awareness, and communication.⁹ The NOTSS system has established sound and valid evidence in its ability to be used for accurate and reliable measurement of surgeons’ NTS intraoperatively.¹⁰ The Trauma Non-Technical Skills (T-NOTECHS) instrument is an observer-based NTS measure that is designed specifically to evaluate NTS in the unique context of the trauma bay.¹¹ Evaluating five domains of NTS (i.e., cooperation and resource management, leadership, communication, situation awareness/coping with pressure, and decision-making), the T-NOTECHS has displayed adequate validity and reliability evidence to measure NTS during trauma situations accurately.¹² However, despite the validity evidence suggesting the utility of these measures to evaluate surgeons’ NTS, their subjectivity can inevitably introduce bias into the evaluation process.^13,14 It is possible, though, that physiological sensing measures can offer more objective insights into surgeons’ NTS than traditional methods.

Eye-tracking (ET) methods, which utilize infrared cameras to detect pupillary movement, can enable researchers to objectively detect visual attention (i.e., frequency and duration of fixations) directed toward key areas of interest (AOI).¹⁵ Regarding NTS, ET methods have been used to effectively measure nurses’ decision-making in simulations where patients suffered the rapid onset of a stroke.¹⁶ Furthermore, in a study of medical students’ NTS during a simulated patient care scenario, our team found that increased duration of visual attention on the patient was negatively correlated with leadership and communication.¹⁷ Additionally, increased attention on non-critical AOIs was associated with poorer decision-making and situation awareness. Given these previous findings, ET methods could be effective at objectively measuring surgeons’ NTS in other healthcare settings like trauma situations.

The purpose of this study, then, was to:

1. Assess trauma team leaders’ objective gaze behavior during patient care simulations.

2. Evaluate the relationship between team leaders’ objective gaze metrics and subjective NTS measures when leading trauma teams during patient care simulations.

2. Methods

Following Institutional Review Board approval, first-year post-graduate surgery and emergency medicine residents were recruited from large academic training programs to participate in one of two possible trauma simulation scenarios. This convenience sample was recruited during their designated interdisciplinary trauma team training education day, which is designed to enhance residents’ performance during acute patient care events. All residents had obtained Advanced Trauma Life Support certification prior to study participation. The designated trauma team leaders were targeted for the study participation when ad-hoc teams were formed prior to the simulation. The team leaders wore a mobile eye tracker to capture gaze patterns during the simulation, and their NTS performance was evaluated by trained surgery and emergency medicine faculty using the NOTSS and T-NOTECHS.

2.1. Simulated Scenarios

All resident participants in this study received guidance on the equipment available in the simulated trauma bays and the capabilities of the patient manikins (SimMan 3G, Laerdal Medical, Wappingers Falls, NY). Patients were controlled by experienced simulation technologists who were educated on the case evolution and patient vital sign changes ahead of the simulation date. The simulation technologists also had experienced trauma surgeons available in the control room to provide guidance on patient vital sign changes if needed (i.e., in response to provided interventions).

Each simulated scenario lasted approximately 15 minutes, and residents were instructed to diagnose and treat the presenting trauma patients. The first simulation represented a multi-patient trauma event, where two patients were involved in a motor-vehicle accident. Residents were initially asked to attend to an unresponsive patient who experienced a pelvic fracture. After the initial assessment of the first patient, a second patient suffering from a traumatic lower limb amputation arrived in the trauma bay. Residents were expected to diagnose and place a pelvic binder on the first patient and place a tourniquet on the injured leg of the second patient.

The second simulation represented a patient presenting with severe facial trauma resulting from a motorcycle crash. The patient presented with declining respiratory status requiring a cricothyroidotomy, and as the scenario progressed, residents were expected to diagnose and treat the patient’s pneumothorax with the placement of a thoracostomy tube.

2.2. Subjective NTS Assessments

Trained faculty raters observed each trauma team’s performance during the simulations. Immediately after the scenarios ended, they completed the T-NOTECHS to evaluate the NTS of the trauma team, and the NOTSS to evaluate NTS of the team leader. Two or three faculty raters completed both assessments for each participating trauma team. The T-NOTECHS evaluates the NTS of the entire trauma team using five constructs (i.e., situation awareness, decision-making, communication, cooperation, and leadership), each on a five-point Likert scale. Higher scores reflect better NTS and total T-NOTECHS scores range from 5 to 25. The NOTSS, specifically designed to assess a primary surgeon’s NTS, evaluates four NTS domains (i.e., communication and teamwork, leadership, situation awareness, and decision making) on four-point scales (i.e., ranging from 1 to 4). Each domain in NOTSS consists of three elements, and mean scores were calculated for each domain based on their respective elements. Each domain was treated independently and a total NOTSS score was not calculated. Since multiple raters evaluated each group, mean scores were derived across all raters for each NTS domain across T-NOTECHS and NOTSS.

2.3. Eye-Tracking Metrics

All team leaders wore the Tobii Pro Glasses 2.0 ET system (Tobii AB, Danderyd, Sweden). The Tobii Glasses 2.0 features a mobile eye tracker and high-definition external camera to map each gaze point on external video from the wearer’s perspective. Utilizing the Tobii Pro Lab software, each gaze point was mapped to patient (head, neck, torso, arms, and legs), teammate (other members of the trauma care team and a representative from the Emergency Medical Service (EMS)), diagnostic (vital sign monitor), and patient management equipment (laryngoscope, endotracheal tube, breathing mask, thoracostomy tube, tourniquet, pelvic bunder) AOI groups. For each AOI group, three ET metrics were calculated: time to first fixation on AOI group, average fixation duration on AOI group, and total percent of the scenario fixated on AOI group. Time to first fixation refers to the amount of time (in seconds) individuals take to visually attend to an AOI and reflects which AOI attracts visual attention first (i.e., or which AOI is most relevant to solving the problem at hand).¹⁸ Average fixation duration or sustained visual attention on a stimulus without looking at another stimulus, reflects cognitive processing of visual cues.¹⁸ Finally, the total percentage of the scenario fixated on each AOI may reflect the importance of particular AOI groups to the problem at hand (i.e., more important AOI have a higher percentage of fixations).

2.4. Statistical Analysis

Statistical analyses were conducted in R¹⁹ within the RStudio integrated development environment,²⁰ and Tidyverse,²¹ ggpubr,²² and rstatix²³ packages were used. To assess the distribution of our datasets, Shapiro-Wilk tests were performed. For ET metrics that were normally distributed, one-way analysis of variance (ANOVA) and Tukey’s honest significant difference (HSD) post-hoc tests were performed to evaluate differences in metrics across AOI groups. Differences in non-normal ET metrics across AOIs were evaluated using Kruskal-Wallis test. Pairwise comparisons across such AOIs were done with Dunn’s test and Dunn-Bonferroni corrections were applied. Correlations between objective gaze and subjective NTS variables were evaluated using non-parametric Spearman’s Rho tests. p-values < 0.05 were considered statistically significant.

3. Results

Nine trauma teams participated in this study, with each team comprised of four or five trainees. Six trauma teams participated in the motor vehicle accident scenario, and three in the motorcycle crash scenario. ET metrics from nine trauma team leaders (56% females) were extracted. Most participants in this study were residents from general surgery (89%).

3.1. Subjective NTS Scores

T-NOTECHS scores of the trauma team and the NOTSS scores of the team leaders were assessed. The following were the mean and standard deviation (SD) scores of T-NOTECHS constructs: leadership (3.29±1.02), cooperation and resource management (3.73±0.76), communication and interaction (3.29±0.61), assessment and decision making (3.66±0.86), and situation awareness/coping with stress (3.71±0.60). Individual constructs of T-NOTECHS were assessed on a scale of 1-5. Mean and SD of the total T-NOTECHS score was 17.68±3.32.

The following were the mean and SD scores of NOTSS constructs: situation awareness (3.05±0.42), decision making (2.89±0.72), communication and teamwork (2.87±0.66), and leadership (2.87±0.77).

3.2. Objective Eye Tracking Metrics

3.2.1. Time to first fixation (TTFF) on AOI groups

Figure 1 is a boxplot of the time to first fixation (TTFF) on AOI groups. Kruskal-Wallis test showed that there were significant differences in TTFF across AOI groups (χ2(4) = 20.77, p < 0.001). Pairwise Wilcoxon test with Dunn’s test showed significant differences in TTFF between the patient and teammate AOI group (median patient TTFF: 29.82 seconds vs median teammate TTFF: 1.56 seconds (p<0.001), and the patient and others AOI group (median patient TTFF: 29.82 seconds vs median “others”: 1.20 seconds (p<0.01). The median time spent before first fixation was shortest for the teammate (1.56 seconds) and longest for diagnostic (11.76 seconds) and patient (29.82 seconds) AOI groups.

Figure 1:

Box plot for TTFF on AOI Groups

*: p ≤ 0.05; **: p ≤ 0.01; ***: p ≤ 0.001

3.2.2. Average fixation duration (AFD) on AOI

Figure 2 is a boxplot of the average fixation duration (AFD) on AOI groups. One-way ANOVA test showed that there were no significant differences in AFD across AOI groups [F(4,40) = 1.76, p = 0.157]. Mean AFD was shortest for “other” (0.25 seconds) and longest for diagnostic (0.41 seconds) AOI groups. Mean AFD on teammate AOI group was 0.32 seconds.

Figure 2:

Box plot for AFD on AOI Groups

3.2.3. Total percent of scenario (TPS) fixated on AOI

Figure 3 is a boxplot of the total percent of scenario (TPS) fixated on AOI groups. One-way ANOVA test showed that there were significant differences in TPS across AOI groups [F(4,40) = 18.50, p = 1.12⁻⁸. Tukey’s HSD test showed significant differences in TPS across all but three AOI group pairs – patient (32%) and teammate (24.82%), diagnostic (22.84%) and teammate (24.82%), and “others” (9.36%) and equipment (11%) AOI groups. Average TPS was smallest for “others” (9.36%) and highest for patient (32.00%) AOI groups.

Figure 3:

Box plot for TPS fixated on each AOI

*: p ≤ 0.05; **: p ≤ 0.01; ***: p ≤ 0.001

3.3. Correlations between subjective NTS measures and objective ET metrics

Table 1 shows the spearman’s Rho correlation coefficients between the subjective NTS measures and the ET metrics on AOIs. Statistically significant correlates are also shown. There were statistically significant negative correlations between TTFF on the diagnostic AOI group and subjective NTS measures. AFD on the teammate AOI group correlated with subjective NTS measure of communication (p<0.05). There were no statistically significant correlates between subjective NTS measures and TPS for any AOI group.

Table 1:

Spearman’s Rho correlation coefficients (R) between subjective NTS and (a) TTFF, (b) AFD, and (c) TPS for AOI groups

a. Total Time to First Fixation
AOI Group	T-NOTECHS						NOTSS
	L	C	CI	SA	ADM	Total	SA	DM	CT	L
Patient	0.22	0.15	−0.02	0.10	0.12	0.17	−0.07	−0.10	0.16	0.00
Teammate	0.16	0.03	−0.03	−0.28	−0.19	0.00	0.16	0.02	−0.07	−0.17
Diagnostic	−0.89**	−0.38	−0.70*	−0.90***	−0.86**	−0.83**	−0.64	−0.90***	−0.54	−0.68*
Equipment	−0.63	0.06	−0.22	−0.48	−0.48	−0.48	−0.54	−0.54	−0.22	−0.39
Others	−0.09	−0.55	0.24	−0.14	−0.30	−0.28	−0.24	0.04	−0.27	−0.27
b. Average Fixation Duration
Patient	−0.13	0.02	−0.03	0.16	0.16	0.00	−0.44	−0.24	0.02	−0.49
Teammate	0.31	0.35	0.51	0.51	0.58	0.52	0.18	0.29	0.75*	0.34
Diagnostic	0.12	−0.03	0.30	0.31	0.32	0.22	−0.07	0.08	0.38	−0.08
Equipment	−0.13	0.15	−0.08	0.31	0.35	0.12	−0.26	−0.13	0.13	−0.31
Others	−0.45	−0.08	−0.16	−0.08	−0.08	−0.25	−0.62	−0.45	−0.09	−0.51
c. Total Percent of Scenario Fixated
Patient	0.00	−0.29	0.14	−0.48	−0.50	−0.25	0.07	−0.02	−0.11	−0.05
Teammate	−0.24	0.23	−0.13	0.26	0.20	−0.02	−0.38	−0.16	−0.10	−0.10
Diagnostic	0.43	0.17	0.46	0.18	0.13	0.28	0.35	0.34	0.40	0.27
Equipment	0.23	0.18	0.06	0.38	0.53	0.42	0.40	0.32	0.30	0.36
Others	−0.41	0.09	0.00	−0.15	−0.19	−0.27	−0.64	−0.34	−0.20	−0.20

Statistical significance - *: p ≤ 0.05; **: p ≤ 0.01; ***: p ≤ 0.001

R: Spearman’s Rho’s correlation coefficient; L: Leadership; C: Cooperation and resource management; CI: Communication and interaction, SA: Situation awareness; ADM: Assessment and decision making; Total: Total T-NOTECHS; DM: Decision making; CT: Communication and teamwork

4. Discussion

In this study, we have demonstrated the feasibility of ET to objectively measure team leaders’ and trauma teams’ NTS in simulated scenarios. We found that ET metrics were sensitive to several aspects of surgery and emergency medicine residents’ NTS during trauma scenarios presented in this study. Additionally, we found strong correlations between these metrics and subjective NTS scores from trained emergency medicine and surgery faculty raters. These results suggest that ET metrics may provide important insights into NTS. NTS are particularly important in the military environment, as trauma teams often face staff and resource constraints. Given the importance of NTS to trauma team performance, finding objective assessment methods is critical to expand researchers’ abilities to categorize NTS in such austere environments.

While technical skills of trauma teams are needed for effective patient care, they alone are not sufficient to ensure positive patient outcomes. Hence, NTS training for trauma teams is critical to improve patient care and medical management.¹¹ The literature suggests that simulation training can be used to effectively train NTS during trauma scenarios in a safe educational environment that does not put patients at risk.²⁴ However, an important component of simulation training is the need for objective assessments to guide the feedback provided to trainees to effectively enhance their performance.²⁵ Current NTS assessment tools used in surgery rely on subjective, observer-based evaluations of performance, which can introduce bias in the assessment process and negatively impact the feedback provided to trainees.^13,14 Given the validity evidence obtained in the current study on the association between ET and subjective NTS measures, future educators could utilize ET metrics to provide more objective and structured feedback to trainees on their gaze patterns and association with NTS. For instance, ET videos could be used to determine the distribution of trainees’ gaze patterns during a critical phase of the scenario to assess whether they are attending to all crucial stimuli, and this can help to corroborate subjective situation awareness ratings.

During the scenarios presented in this work, trauma teams scored worst on communication across both T-NOTECHS (3.29±0.61) and NOTSS (2.87±0.66). Research has shown that 43% of errors in surgical care are related to communication breakdowns.²⁶ Due to the ad-hoc nature of trauma team formation, there might be reduced time to deliberately communicate care plans. This problem can be exacerbated in military trauma environments, which are naturally chaotic due to the high patient acuity.²⁷ Furthermore, international teams could be formed with limited prior experience working together leading to language barriers under heightened stress. Additionally, resource-limited teams could be formed such that team members need to perform multiple duties during resuscitations, necessitating clear communication from the team leader on what actions are needed per time. Hence, it is important that trauma care members receive formal communication training that ensures effective team performance. One study found that the use of closed-loop communication (CLC) among pediatric trauma teams reduced medical errors and improved the speed and efficiency of task completion.²⁸ Hence, education on the use of standardized and reliable communication systems like CLC should be incorporated into trauma team leaders’ training. Sensor-based recordings like ET videos can facilitate these training sessions because they allow for a transparent assessment of trauma care members’ specific communication-based training needs.

The use of ET metrics derived from AOIs is increasingly used in surgical education for skill assessment and training purposes.²⁹ This approach can also be utilized in highly realistic and dynamic environments like the trauma bay to understand behaviors that promote best practices. In this study, we extracted three AOI based metrics to better understand trauma team leaders’ visual attention during simulated scenarios. We found that median time elapsed before trauma team leaders fixated on the presenting patient was about 20 and 25 times more than the time elapsed before fixating on team members and “other” AOIs, respectively (p < 0.001). Although the difference was not statistically significant, the median TTFF on patient AOI was also 2.5 times more than the diagnostic AOI (vital sign monitor). Fixating on the trauma team members and other sources of auxiliary information once a patient presents to the trauma bay might be indicative of thoughtful and deliberate planning. However, we found that the TTFF on the diagnostic AOI was negatively correlated (p < 0.05) to multiple measures of subjective NTS assessments (Table 1). This means that higher TTFF on the vital sign monitor is associated with poorer NTS including communication, leadership, and situation awareness. While this AOI is a critical source of diagnostic information used for decision-making that needs to be attended to quickly, shorter TTFF on the patient may be necessary to better understand the presenting situation. Further study on the gaze patterns of experienced physicians participating in similar scenarios may provide further insights on ideal TTFF on various AOIs upon entering the trauma bay.

TTFF is a surrogate measure of the team leader’s attention around the time a patient first presents. A good and balanced distribution of fixation across the trauma bay around the time a patient first presents might be beneficial for the team leader to achieve high levels of situation awareness. Faculty members can access such objective metrics to guide team leaders on optimal distribution of attention especially in the first few minutes of a patient presenting to the trauma bay.

While TTFF is an objective measure of attention distribution in the first few minutes of trauma care delivery, AFD offers a broader perspective of sustained attention on visual cues throughout the entire caregiving period. In this work, we found no statistically significant differences in AFDs across the AOI groups. This means that on average, team leaders evenly fixated on different AOIs during the simulation. We also found that AFD on teammates AOI was positively correlated (p<0.05) to communication and teamwork (Table 1). This means that higher AFD on other trauma team members and the EMS representative is associated with better communication and teamwork. This is in line with psychology literature that showed that visual attention modulates communication patterns in group settings.^30-32 Communication and teamwork are crucial constructs of NTS that have been linked to improved patient safety.³³ Trauma team members have reported that communication training is important and were likely to attend such trainings.³⁴ ET-based trainings can further bolster the acceptance of communication training among trauma team members.

TPS provides a measure of the fraction of the scenario the team leader fixated on a specific AOI, denoting visual cues that were deemed most important. In this study, there were statistically significant differences in TPS across most AOI pairs (p<0.05), alluding to the fact that team leaders deemed some AOIs more important that others during the caregiving period. Specifically, there were no statistically significant differences in TPS between three AOI pairs: patient and teammate; diagnostic and teammate; and others and equipment. This implies that on average, team leaders evenly fixated on the patient, teammate, and diagnostic AOI during the scenario compared to other visual cues. Fixations on these AOIs is critical in ensuring optimal team and patient outcomes. Finally, there were no correlates between TPS on AOIs and subjective NTS scores.

In the context of high-acuity traumas in combat environments, NTS are particularly important.^3,4 Healthcare providers must rapidly diagnose and provide treatments to patients with limited support compared to regular hospital settings.³ Thus, this setting would benefit significantly from simulation-based training to enhance NTS performance. Indeed, research has shown that simulation training for military traumas can enhance military healthcare team performance.³⁵ Our study approach could be leveraged to objectively assess military trauma team leaders’ NTS during simulations and provide even more targeted education to enhance their NTS in that environment. The literature has argued that deployed surgical teams should rely heavily on ongoing training and continuous improvement of care processes to perform optimally in harsh and austere environments.³⁵ Hence, utilizing ET-based objective measures associated with NTS during high-fidelity trauma simulations could offer data that supports process improvement education for providers in military trauma teams.

There were limitations with this study. Due to equipment constraints on the number of ET devices available for use in this study, our team was forced to use the ET with just a single member of each team. Accordingly, there was a small sample of team leaders in this study, which could have led to type II errors in our data analysis. Despite our small sample, we found statistically significant differences in ET metrics between AOIs and significant correlations with NTS metrics (i.e., with several p-values less than 0.01). These findings highlight the magnitude of our findings and suggest that further work extending this research with a larger sample of is needed. Another limitation is our global comparison of ET metrics and subjective NTS ratings. We did not consider trauma team leaders’ discrete NTS related to specific critical events, as we intended to study the relationship of ET metrics and commonly used subjective NTS measures. In the future, we plan to identify critical NTS events in a larger data set and perform a more comprehensive analysis on how ET metrics in these events are related to NTS.

Also, we cannot establish the missing connections between visual attention, cognition, and NTS. While we did identify associations between ET metrics and NTS, we cannot be certain that simply attending to an AOI indicates effortful cognition required for comprehension and projection of future states during trauma resuscitations. Accordingly, it is necessary to consider additional methods to identify the connection between visual attention, cognition, and NTS. The Situation Awareness Global Assessment Technique (SAGAT) tool is a method with validity evidence to evaluate situation awareness in real time.³⁶ The SAGAT approach involves pausing operators during simulations and asking them questions related to their perceptions about the environment, which gives insights into operators’ cognitive states during the simulation. Our team could potentially leverage this approach to pause trauma simulations and ask the trauma team leaders questions about their differential diagnosis to ascertain their understanding of events related to the ET metrics obtained in the moments preceding the pause. We plan to explore this approach further in future studies. The use of subjective rating tools to capture NTS may have introduced bias into the ratings, which is another limitation of our study. There are numerous biases detailed in the literature that can negatively affect subjective assessments from observers.³⁷ However, multiple faculty raters evaluated each group of participating residents, which led to a single mean rating for each group. Thus, potential biases from raters can be minimized using this approach. Furthermore, the biases inherent in subjective measures of NTS motivated this study, as we aimed to utilize objective ET data to further add insights to subjective ratings.

Another limitation was the inconsistency in the number of faculty raters available to complete NTS evaluations, which prevented us from evaluating inter-rater reliability. Unfortunately, our team was forced to conduct this study over several years with different faculty raters, and occasionally a different number of raters. However, our team carefully trained raters ahead of every simulation day to ensure that faculty were consistent in their evaluations. Furthermore, almost all SDs for individual NTS constructs across both T-NOTECHS and NOTSS were less than one, indicating little variability in ratings. Finally, while it may be possible to assess the clinical transfer of physicians’ acquired NTS in civilian clinical trauma environments using ET methods, their application in military trauma clinical settings may be less feasible. The ET system requires a dedicated researcher to calibrate the system to each individual user and given the limited number of ancillary staff that may be available in these environments, studying the skill transfer of military surgeons’ simulation acquired NTS may be impractical.

5. Conclusion

This study utilized a mixed methods approach to assess trauma team leaders’ NTS in trauma simulations. The results suggest that objective ET metrics are aligned with subjective ratings of residents’ NTS when caring for trauma patients. Educators could obtain a more holistic and nuanced understanding of behaviors through ET metrics, and these can be used to provide more objective feedback to trainees to enhance their NTS. Importantly, more studies are needed with a larger sample of physicians to continue exploring the relationship between ET metrics and subjective NTS ratings in simulation, and expansion of this work to studying surgeons’ NTS in the clinical environment is warranted.

Data availability statement:

Data will be made available to researchers upon reasonable requests.