Team Size Impact on Assessment of Teamwork in Simulation-based Trauma Team Training

Abstract

Non-technical skills (teamwork) assessment is used to improve competence during training for interprofessional trauma teams. We hypothesized non-technical skills assessment is less reliable for large size teams, and evaluated team size effects during teamwork training. Small-teams (n = 5; 5–7 members) and Large-teams (n = 6; 8–9 members) participated in three simulation-based trauma team training scenarios. Following each scenario, teamwork was scored by participating trauma attending physicians (TA), non-participating critical care trauma nurses (CRN), and two expert teamwork debriefers (E), using the Trauma Nontechnical Skills Assessment tool (T-NOTECHS). Large-team scores by TA and CRN were higher than E scores (P < .003); small-team scores did not differ by rater. Small-team inter-observer agreement was substantial (ICC = 0.60); large-team agreement was low (ICC = 0.29). E and TA scores showed no concordance, whereas E and CRN scores showed poor concordance for large teams (ICC = 0.41, r = 0.53, P = .02). By contrast, correlation between E and TA (ICC = 0.52, r = 0.80, P < .001) as well as E and CRN (ICC = 0.57, and r = 0.65, P < .01) for small teams was high. Team size should be considered in team-training design, and when using teamwork rating instruments such as T-NOTECHS for assessment of simulated or actual trauma teams. Modified rating scales and enhanced training for raters of large groups versus small groups may be warranted.

Introduction

Lapses in communication and teamwork failures have been identified as key common contributing factors in patient safety incidents.^1–3 Trauma resuscitation is a common, complex and time-critical endeavor performed by interprofessional, ad-hoc clinical teams. As such, trauma resuscitation provides a valuable platform for training and evaluation of healthcare professionals in critical non-technical (teamwork) skills. High-fidelity simulation-based training has been used effectively in a variety of clinical disciplines to modify behaviors and affect patient outcomes.^4–6 Recent studies have reported that simulation-based training can improve teamwork, clinical efficacy and trauma process.^6–9 Assessing clinical competency during real-life trauma resuscitation is challenging;⁴ there is however a growing body of evidence supporting the use of simulation for assessment and training of trauma teams.^10–13

Accurate evaluation of non-technical skills is central to optimizing team-based training, assessing competency and gauging the impact of educational interventions. A variety of tools have been developed to assess non-technical skills in the operating room and trauma bay.^7,11,14 Considerable focus has been appropriately placed upon rater qualifications, training and structured debriefing.^15–17 Less information is available regarding the impact of team composition on the ability to execute requisite patient care tasks in an optimally safe environment characterized by excellent teamwork, cooperation, and communication. Team training effectiveness is moderated by team size, in addition to other recognized factors such as training content.¹⁸ Teams are thought to be most effective when composed of a sufficient, but not greater than sufficient, number of members to perform team tasks;¹⁹ Definitions of “sufficient” however remain unclear. Larger teams have been noted to suffer from loss of coordination and process, yet have been shown to be more effective in some studies—perhaps dependent on the nature of the tasks and team environment.²⁰ In healthcare studies optimal team size effects are not widely reported. One report documents decreasing clinical performance in pediatric trauma resuscitation when optimal number of team members is exceeded.²¹ Another recent study suggests that increasing team size and variability prolongs operative times.²² Intuitively, team size is a critical component of team function, requiring a minimum number of members to accomplish tasks and prevent individual task overload, balanced by limiting total membership to reduce barriers to communication and cooperation. Generalizable factors which influence optimal team size in healthcare domains have not been well defined to the author's knowledge.

Teamwork training and feedback depends on an accurate assessment of teamwork, by observers, in both training settings and actual clinical practice. We hypothesized that the total number of team members would influence accuracy of assessment of trauma team non-technical skills. We investigated the effect of team size on trauma teamwork skills as assessed by trauma attending physicians (TA), critical care trauma nurses (CRNs), and teamwork subject matter experts (E) using the modified Trauma Nontechnical Skills Assessment tool (T-NOTECHS).²³ Secondarily, we sought to evaluate the correlation of T-NOTECHS when administered by three types of raters.

Methods

Subjects

A retrospective review was conducted using data from a previously published simulation-based trauma team training project conducted “in situ,” in the emergency department of a Level II trauma center.²³ The research was institutional review committee approved, and participants gave informed consent. Trauma simulation scenario participants were clinical trauma team members, including surgical residents, trauma surgery and emergency department attending physicians, critical care trauma specialist nurses who are designated as Crisis Resource Nurses (CRN), emergency department nurses, respiratory therapists, trauma physicians' assistants, and emergency department technicians. Participants were assigned to multidisciplinary training teams for trauma simulation exercises based on professional experience and availability. Each team participated in three randomly sequenced 15-minute trauma resuscitation scenarios, conducted with a high-fidelity human patient simulator (SimMan - Laerdal Medical, Wappingers Falls, NY) in the emergency department trauma bay. Scenario design required the completion of eight key “clinical tasks” for an optimal simulated trauma patient outcome for each scenario. Simulated resuscitation sessions were captured using a digital video recorder and a synchronized simulator event log. Teamwork scores were assigned independently by scenario participants and by non-participants. Scenario participants who completed teamwork assessment were Trauma Attending Physicians (TA) and Crisis Resource Nurses (CRN). The CRN routinely functions as a “scribe” during trauma resuscitations, recording times and clinical interventions, with minimal hands-on patient contact. Only one TA and one CRN were assigned to each resuscitation team. Non-participants who completed teamwork assessment and conducted debriefings for simulation scenarios were “Expert Debriefers” (E). Two authors, critical care (BB) and trauma (SS) physicians with training and multi-year experience in multidisciplinary teamwork training and clinical trauma care served as E. Expert ratings were used as the “reference standard” for comparison. Teamwork was scored on a T-NOTECHS scale from 5 (lower performance) to 25 (higher performance), which evaluates five primary teamwork domains: Leadership, Cooperation and Resource Management, Communication and Interaction, Assessment and Decision Making, and Situation Awareness/Coping with Stress. T-NOTECHS scores were self-scored and collected from participants (TA and CRN) immediately following each simulated resuscitation, prior to videotape review and debriefing, using an automated audience response system (ARS; Turning Point Technologies).

After teamwork scores were assigned, a 30-minute debriefing was conducted focused on T-NOTECHS domains. The a priori identified key clinical tasks for each scenario were not alluded to or included in the debriefing. After each day of training, the E performed an independent video review of each scenario and recorded the time-to-completion for each of the clinical tasks.

Statistical Analysis

Preliminary analysis of group size effects on teamwork assessment outcomes was conducted on three convenience groups considering team sizes of 5–6, 7–8, and ≥ 9, using ANOVA. Teams with ≥ 9 members had lower teamwork scores especially on cooperation and resource management and communication and interaction items than other groups. Teams were subsequently grouped and sequentially analyzed according the number of members, detecting significantly different teamwork scores between “small” teams with 5–7 members, and “large” teams with 8–9 members. Demographics of study groups (small versus large) were analyzed by Mann-Whitney U test. T-NOTECHS rating scores were reported as mean ± standard deviation. Comparisons between three raters (TA, CRN, E) were assessed with a two-way, mixed design analysis of variance (ANOVA). Inter-rater reliabilities were evaluated using intraclass correlation coefficients (ICC) based on absolute agreement for single raters in various settings, as follows: (1) 3 raters: E, CRN and TA, (2) 2 raters: E and CRN as well as E and TA. The correlations of T-NOTECHS ratings were calculated using the Pearson correlation coefficient. The data was analyzed using nonparametric methods when appropriate. All analyses were conducted using SPSS version 17.0 (SPSS, Inc, Chicago, IL). Significance was determined at P < .05.

Results

Eighty-one trauma team members completed training and had ARS T-NOTECHS (teamwork ratings) available. These included 6 attending surgeons, 12 emergency department attending physicians, 16 residents, 29 nurses and 20 other special healthcare providers (10 respiratory therapists, 3 physician assistants, and 7 emergency department technicians). All participating members during each training session were assigned to a single team. Team size varied according to availability of participants in the convenience sample. Five “small” (n = 5–7) and six “large” (n = 8–9) teams resulted. There were a significantly greater number of resident physicians in the large (median 2.5, P < .05) versus the small (no resident physicians) teams. Otherwise, there was no significant difference in team composition between the two groups. Each team completed three simulated resuscitations; thus, 33 scenarios were available for analysis.

There was no difference between small and large team time to completion or total number of completed key clinical tasks. T-NOTECHS scores showed no significant difference in E ratings between small and large groups. However, when compared to E ratings and clinical task execution, T-NOTECHS scores assigned by both TAs and the CRNs were higher in the large groups (Table 1). Similarly, the three raters showed substantial inter-observer agreement when scoring small (ICC = 0.60) but not large (ICC = 0.29) groups (Table 2). The difference between TA's and E's assessments was particularly prominent in the large groups, where ICC was 0.18 with no statistically significant correlation between the two scores. Inter-observer agreement between E and CRNs also exhibited poor concordance in the large groups (ICC = 0.41). In contrast, when evaluating the small trauma teams, there was improved and significant correlation between the E and TA (ICC = 0.52, r = 0.80, P < .001) as well as the E and CRN scores (ICC = 0.57, and r = 0.65, P < .01).

Table 1.

Comparison of mean T-NOTECHS score among raters for each group.

Raters	Expert	Trauma Attending	CRN	P-value
Small Teams# (n=15)	16.23 ± 2.38	18.93 ± 3.50	17.53 ± 3.07	.054
Large Teams‡ (n=18)	16.44 ± 2.79	20.26 ± 2.64*	18.83 ± 3.36*	.001

Values are expressed as the mean (± SD).

^#Small Teams comprised 5–7 members.

^‡Large teams comprised 8–9 members.

^*Significant difference versus Expert.

Table 2.

Intraclass and Pearson correlation coefficients between raters' T-NOTECHS scores according to team size.

Raters	Small Teams (n=15)				Large Teams (n=18)
	ICC	P	CC	P	ICC	P	CC	P
E/CRN/TA	0.60	<.001			0.29	.003
E/CRN	0.57	.004	0.65	.009	0.41	.011	0.53	.023
E/TA	0.52	.001	0.80	<.001	0.18	.082	0.34	.177
TA/CRN	0.84	.001	0.78	.001	0.36	.085	0.37	.144

ICC: intraclass correlation coefficient, CC: Pearson correlation coefficient. E: expert, TA: trauma attending physician, CRN: critical care trauma nurse.

Discussion

The conduct of healthcare team training using simulation utilizes a structured instructional design incorporating the experience of simulated case management involving a technologically advanced human patient simulator. Following participation in a clinical simulation, structured debriefing is conducted for participants by a facilitator, who incorporates elements of the team's simulated clinical performance and specific teamwork behaviors during the interactive post-simulation session. In order to conduct a meaningful debriefing a facilitator uses rating tools to perform team assessment during simulation, and provides debriefing and feedback based on the assessment. Simulation based performance assessment can also be used for formative assessment of both individuals and teams. Thus both formative and summative assessments require accurate assessment of performance to reach educational goals and objectives. Development and application of accurate, reliable, and valid assessment tools remains a challenge for healthcare educators.²⁴

There is growing awareness of the importance of non-technical skills among surgeons, and the necessity to critically examine non-technical training and evaluation methods. T-NOTECHS has been used for the assessment and debriefing of teams in both simulated and actual trauma resuscitations In both settings, T-NOTECHS has documented high inter rater reliability when utilized by expert evaluators of trauma team performance. T-NOTECHS has also demonstrates a high correlation of teamwork scores with clinical performance parameters. ^14,23 The results reported in this study unveil several considerations relevant to designing team-training curricula, and for real-life assessment and reinforcement of non-technical skills.

First, high-fidelity human patient simulators are a commonly utilized for training but require substantial equipment and trainer. Efficient use of resources with higher in-situ fidelity might favor providing resident training in larger inter-professional groups with clinicians, such as trauma attending physicians and nurses, as instructor-evaluators. Findings showed inaccuracy and decreased reliability of specialty specific trauma clinician teamwork (TA and CRN) scoring using T-NOTECHS, compared to expert teamwork scoring for larger groups (≥ 8 participants). This finding suggests that constructing smaller training groups allows for more accurate and reliable teamwork scoring. There is scant information about precise team size cutoffs for either clinical effectiveness, or accuracy and reliability of teamwork assessment during training. Our results serve as a pilot to inform the construct of further rigorously designed studies of team size effects. Second, this study reports that despite higher workload for individuals in smaller teams, there appear to be no differences in overall teamwork scores or time to completion of clinical tasks when compared to larger teams. Moreover, the smaller teams were able to accomplish a greater number of tasks per clinical team member - this speaks to efficiency and potentially reduced healthcare costs if the same job can be done, and done better, by smaller teams. Optimal team size is well described in domains outside of medicine^18,25,26 and our findings generate consideration of a hypothesis that there are identifiable critical team sizes or limits which may be optimized for healthcare teamwork and task performance.

Others have corroborated the findings reported here of the inaccuracy of physician self-assessment.²⁷ Two studies in surgical fields demonstrated a strong correlation between self-assessment and expert assessment scores for technical skills but not for non-technical skills.^28,29 These reports indicate that, regardless of experience, surgeons are capable of assessing their own technical skills but not their non-technical skills. Together with our data, these studies provide strong rationale for the employment of non-technical skills experts for evaluation and training in this realm.

Assessment and debriefing of non-technical skills in an inter-professional clinical situation is an essential adjunct to simulated training and may be more effective in changing behavior.³⁰ Data from this study suggests that the traditional model of attending physician-led debriefing following real-life resuscitation events may be suboptimal. The noted high self-ratings and very poor concordance with teamwork experts by the trauma physician participants implies that attending physicians are likely to miss opportunities for formative feedback from larger interprofessional teams. The findings suggest that including nurses as teamwork debriefers may improve accuracy, since the CRN T-NOTECHS scores were more concordant with expert raters. However, lower concordance between both clinician groups and the expert debriefers suggests that novel methods for clinical event debriefing, and incorporating external assessors and/or video review, warrant consideration in the development of interprofessional teamwork training and debriefing paradigms.

There are inherent limitations in this single-institution study with a relatively small number of participants, in a single clinical domain. Multiple factors unaccounted for in our study may influence reliability and accuracy of non-technical skills assessment and debriefing.^14,17,28 In addition, findings that large versus small groups are more difficult to rate consistently by different rater groups may be a function of the T-NOTECHS rating instrument, and should be confirmed using other validated non-technical skills/teamwork rating instruments.

Conclusion

Findings of this study indicate that trauma resuscitation teamwork assessment and rating using the T-NOTECHS instrument is less reliable for larger teams compared to smaller teams. In trauma resuscitation there is evidence that larger teams perform less well than smaller teams.²¹ This study confirms that teamwork training paradigms should consider team size as a potential confounder during instructional design and application of teamwork assessment tools and processes, and supports the need for further investigations in the role of healthcare team constructs and size in teaching and assessment of non-technical skills. Self-assessment by healthcare professionals remains problematic, particularly for physicians. The impact of team size as a potential confounder should be considered in studies seeking to examine teamwork self-assessment by participants in clinical teamwork settings or training exercises.

Table of Abbreviations

Abbreviations	Definitions
E	Expert teamwork debriefers
TA	Trauma attending physicians
CRN	Critical care trauma nurses
T-NOTECHS	Trauma Nontechnical Skills Assessment tool
ARS	Audience response system
ICC	Intraclass correlation coefficients
SPSS	Statistical Product and Service Solutions - statistical analysis software
ANOVA	Analysis of Variance

Conflict of Interest

None of the authors identify a conflict of interest.