Abstract
Introduction
Trauma resuscitations require a coordinated response from a diverse group of health care providers. Currently, there are no widely accepted methods of assessing team effectiveness in this setting. Simulation affords a method to assess team effectiveness. The purpose of this study was to use a simulation setting to develop a specialized assessment instrument for team response in trauma resuscitation.
Methods
We developed our assessment instrument using clinical simulation. Four teams of 3 postgraduate year–2 surgical trainees in conjunction with scripted confederates were videotaped enacting 6 separate trauma resuscitation scenarios that mirrored clinical conditions encountered at our level 1 trauma center. Ten of the resulting videotaped scenarios represented a spectrum of team behavior (ineffective to effective) and were scored by 8 experienced clinicians using the Mayo High Performance Teamwork Scale.
Results
Based in part on the Mayo High Performance Teamwork Scale, we created a prototype trauma team assessment tool consisting of 7 attributes that we scored in binary fashion (present/absent). We validated this prototype by assigning a normalized ranking score to each of the 10 scenarios based on the score supplied by each rater. The presence/absence of the 7 attributes varied significantly among scenarios (52.5% to 93.8%; P < .001). Median scores differed significantly comparing the 5 lowest-ranking scenarios with the 5 highest-ranking scenarios (P < .001).
Conclusion
Our prototype instrument may be effective at assessing team effectiveness during trauma resuscitations. This instrument may prove useful for assessing team competency skills, providing timely feedback to teams, and examining the relationship between effective team function and clinically important outcomes. Further, it may be applicable to other high-acuity, time-sensitive clinical situations.
Introduction
Successful resuscitation of the severely injured patient requires that a diverse group of health care providers function as a highly integrated team. Several factors may detract from the effectiveness of this team. In many academic medical centers, there is substantial turnover in resuscitation team personnel as a consequence of “shift work,” resident rotation schedules, and gain and loss of established employees. Further, trauma teams by design are multidisciplinary, composed of individuals from a variety of specialties and backgrounds (eg, emergency medicine, surgery, anesthesiology, nursing, respiratory therapy). While intended to achieve synergy, this diversity might pose challenges to effective team function, particularly in stressful situations. Such factors may limit the ability of the trauma team to harmonize—that is, team members might not be optimally attuned to the personal habits, communication styles, and idiosyncrasies of their coworkers. The extent to which lack of team cohesiveness both occurs and detracts from patient care is largely unstudied.
In standard surgical and emergency medicine curricula, little attention is devoted to team-training exercises or to critique of effective team function in the context of trauma care delivery. Similarly, Advanced Trauma Life Support is a standardized approach to teaching algorithms for trauma management on an individualized basis, but it does not provide structure or guidance for coordinating team response.1,2 A challenge of assessing team function during trauma resuscitations is the inherent variability these situations present. Clinical simulation technology, in contrast, is a platform in which team skills potentially important to trauma resuscitation might be analyzed. Two studies3,4 have evaluated the use of clinical simulators as a means of assessing team function during simulated trauma resuscitation. Marshall et al3 reported that use of a clinical simulator in conjunction with standard Advanced Trauma Life Support training significantly improved team effectiveness. Similarly, Holcomb et al4 reported the use of a clinical simulator to assess trauma competency before and after an intensive clinical and didactic rotation at a trauma center; they reported a favorable effect on team function and efficiency. A shortcoming of both studies is that the manner in which team behavior was analyzed and scored was not explicitly defined, limiting the ability to broadly apply these studies' findings.
At present, there are no standardized approaches to assessing team competencies in the setting of trauma. Developing such methodology is essential to quantifying the relationship between coordinated team behavior and clinically important outcomes, such as effects on resource expenditure, reduction in errors or “near misses,” diminished rates of morbidity and mortality, and professional satisfaction. A validated approach to assessing trauma team behavior would have utility for training and assessing teams in postgraduate training programs and continuing medical education courses and for quality assurance initiatives in environments utilizing experienced practitioners (eg, nonacademic health care centers and mass-casualty drills).
The purpose of this study was to develop a prototype instrument to assess team function during trauma resuscitation. Ideally this instrument would capture those team skills required during primary and secondary trauma survey, possess high rates of inter-rater and intra-rater agreement, and enable faculty to rapidly assess teams and provide immediate feedback. As an initial effort, we examined the suitability of the Mayo High Performance Teamwork Scale described by Malec et al5 for this purpose. This 16-item instrument attempts to operationalize principles of crises resource management, specifically situational awareness, communication skills, anticipation of errors, and containment of errors. The instrument was developed and tested using simulation scenarios analogous to trauma resuscitation (critical anesthesia management, emergency response teams). While the focus of our investigation is the initial management of the severely injured patient, this approach is potentially applicable to other settings that require teams to respond to high-acuity, complex situations in a time-sensitive fashion (eg, delivery of care in the intensive care unit or operating room and response of resuscitation teams to in-hospital emergencies).
Methods
Development of Simulation Exercises
We utilized human patient simulator technology with full-scale mannequins to create a consistent and reproducible clinical encounter that allowed us to study trauma resuscitation team behavior and dynamics. While surgical trainees in our program successfully complete the Advanced Trauma Life Support provider course prior to internship, in-depth exposure to trauma resuscitations first occurs during the second postgraduate year as part of the emergency department experience. In consultation with our acute-care surgical and emergency medicine faculty, we identified the core trauma competencies we expected our trainees to master during their emergency department rotation (table 1). We designed 6 trauma resuscitation scenarios to incorporate these competencies and to reflect common injury patterns managed at our center (table 2).
Table 1.
Trauma Competencies Expected of Residents Rotating as Part of the Trauma Resuscitation Team
Table 2.
Scenarios Incorporating Trauma Competencies and Used in Assessing Team Function
The subjects of this investigation were postgraduate year 2 surgical residents who participated in simulation sessions during the early months of the academic year. Each simulation session in this investigation involved 2 to 3 surgical trainees in addition to a scripted confederate participating in the 6 scenarios. Prior to each session, trainees received a detailed orientation to the clinical simulator. For each scenario, the trainees were assigned defined roles (team leader, primary physician, etc) analogous to the roles assigned to members of our emergency department resuscitation team. A facilitator provided a brief clinical summary and scripted prompts as the scenario progressed. A faculty member or fellow experienced in trauma resuscitation provided a debriefing at the conclusion of the scenario. Sessions were videotaped for purposes of scoring team competencies. Individual clinical performance was assessed using established methodology in which competency standards were created by a panel of experienced practitioners, as previously described.6–8 These competency standards were independent of the assessment of team function.
Assessing Team Effectiveness
Three investigators independently of each other reviewed the videotaped record of the 24 resulting trauma resuscitation episodes and selected 10 trauma resuscitation episodes that represented the spectrum of team behavior (ie, from ineffective to effective) based on subjective assessment. These 10 videotaped trauma resuscitation episodes, representing 3 scenarios (blunt cardiac injury, pelvic trauma, and penetrating torso trauma), were presented to 8 physicians experienced in trauma care delivery. These individuals scored these episodes utilizing the Mayo instrument in an attempt to understand the operational characteristics of this instrument when applied in this fashion.5 In brief, the Mayo instrument scores 16 possible elements of team function according to a graded scale (0, never; 1, inconsistent; 2, consistent). Each element is also scored on a temporal basis (0–3 minutes, 3–6 minutes, etc). Based on our experience with the Mayo instrument (noted in the Results section), we introduced modifications to produce a prototype scoring system specifically for trauma resuscitations. We identified 7 attributes of the Mayo High Performance Teamwork Scale that appeared most relevant to team skills in the trauma context. To diminish inter-rater variability and enhance usability, we created a binary scoring system, that is, a given attribute is either present or absent. Further, we eliminated timing parameters. Finally, a free-text field was provided to capture additional information. This scoring instrument is illustrated in table 3.
Table 3.
Prototype Instrument to Assess Team Behavior During Trauma Resuscitation
Simulation Facility Description
This study was conducted using state-of-the-art human patient simulators (Medical Education Technologies, Inc, Sarasota, Florida). These mannequins are equipped to provide a realistic patient encounter (breath and heart sounds, pulses, motors to operate chest excursions during spontaneous ventilation, CO2 and inspired and expired gas analysis, noninvasive blood pressure, electrocardiogram, vocal interaction with operator in remote location, programmed responses to specific interventions, etc). Scenario participants are able to perform all diagnostic and therapeutic actions during each exercise (ie, in “real time”), including drug administration and airway and cardiovascular management.
Statistical Analysis
Standard parametric and nonparametric statistical tests and software (Sigmastat System Software, San Jose, California) were employed. Specifically, concordance among users for key items in the Mayo scale was determined by examining commonality of response. We assigned a normalized ranking score for the videotaped trauma resuscitation episodes by determining the highest score assigned, and used this score to adjust all scores. Conventional contingency tests (such as χ2) were used to examine variability in presence or absence of elements of the prototype scoring instrument among the videotaped episodes. Comparison of binary attribute scores of the prototype instrument observed among low- and high-performing trauma resuscitation teams was accomplished with a rank sum test.
Human Subject Protection
This study was approved by the Human Research Protection Office of Washington University School of Medicine.
Results
Characteristics of Participants
Eleven second-year surgical residents (45.4% women, median age 28 years [interquartile range 28–29 years]) participated in a series of simulation sessions conducted over a 4-week period. Self-reported experience with trauma resuscitations prior to these sessions was limited. Participants reported observing 20 (15–25) resuscitations, participating in 10 (2–23.25) resuscitations, and functioning as team leader in 0 (0–1.5) resuscitations.
Limitations of Mayo Instrument When Applied to Videotaped Trauma Resuscitation Episodes
Several limitations to utilizing the Mayo approach to team effectiveness in the trauma setting emerged. The following elements were not consistently scored and presumably not germane to the simulated trauma resuscitation scenarios:5
The team leader ensures maintenance of an appropriate balance between command authority and team member participation.
Disagreements or conflicts among team members are addressed without a loss of situation awareness.
When appropriate, roles are shifted to address urgent or emergent events.
When directions are unclear, team members acknowledge their lack of understanding and ask for repetition or clarification.
Team members acknowledge—in a positive manner—statements directed at avoiding or containing errors or seeking clarifications.
Team members respond to potential errors or complications with procedures that avoid the error or complication.
When statements directed at avoiding or containing errors or complications do not elicit a response to avoid or contain the error, team members persist in seeking a response.
Team members ask each other for assistance prior to or during periods of task overload.
In addition, the recorded scenarios typically lasted no more than 5 minutes, consistent with the duration and tempo of actual trauma resuscitation. Thus, grading team function in a precise temporal fashion (ie, 0–3 minutes, 3–6 minutes, etc) was not pertinent. Finally, we found assigning a graded score (a grade of 1 corresponding to a behavior being inconsistently present, 2 corresponding to a behavior consistently present, etc) to have poor inter-rater reproducibility. Specifically, mean concordance (± SE) among raters was 0.642 (± 0.012).
Piloting of Prototypical Team-Scoring Instrument for Trauma
We piloted the utility of our prototype instrument in discriminating effective from ineffective team function. We assigned a normalized ranking score to each of the 10 scenarios based on the raw Mayo instrument score provided by each of the 8 raters. Averaging these scores for each of the scenarios allowed us to order them according to effectiveness of team function (figure). We examined how these scenarios were scored by the 8 raters for the 7 attributes contained in our prototype scoring instrument. We found that the presence of these 7 attributes varied significantly among these scenarios (from 52.5% for attribute 6, “All team members are appropriately involved and participate in the resuscitation,” to 93.8% for attribute 3, “Team members prompt one another to attend to significant clinical indicators throughout the resuscitation”; P < .001). Further, when attribute scores provided by the 8 raters were collapsed to either 1 (present) or 0 (absent), median scores differed significantly when the 5 lowest-ranking scenarios were compared with the 5 highest-ranking scenarios (median 6.0 [interquartile range 5.0–7.0] vs 8.0 [7.0–8.0]; P < .001).
Figure 1.
Ability of Prototype Instrument to Discriminate Based on Team Effectiveness.
Discussion
Clinical simulators have been evaluated in a variety of settings, including trauma resuscitation.3,4,9–12 While most trauma applications use this technology as either an adjunct to traditional didactic methods or as a means of assessing competency, clinical simulation is potentially useful for teaching and assessing trauma team resuscitation skills. Specifically, potential barriers to analyzing and critiquing team skills during live or real-time patient resuscitations include the variable nature of the encounter as well as the urgency of the clinical situation. Simulator technology overcomes these obstacles—trauma teams can be confronted with consistent scenarios and challenges in a controlled setting, enabling systematic assessment of group dynamics and how these dynamics and behaviors might be influenced by both clinical (eg, severity, complexity) and personnel (eg, level of experience, competency) factors.
Realizing the potential of simulation technology to develop and assess trauma team behavior, and translating these insights into actual patient care, is predicated on the availability of robust and validated instruments that incorporate the key elements of effective team function and that lend themselves to providing meaningful feedback. While prior investigators3,4 have described use of simulation technology in trauma resuscitations to enhance team behavior, no standard scoring instrument exists. Fletcher et al13 have described 4 behavioral domains essential to assessment, instruction, and critique of crisis resource management as it occurs in urgent or emergent clinical settings. These domains include cooperation and communication (team building and maintenance, consideration and support of team members, and conflict resolution); leadership and management (appropriate use of authority and assertiveness, adherence to standards, team coordination, workload management); situational awareness (cognition of system, environmental, and time factors; anticipation of future events); and decision making. Several team-scoring instruments designed for use in specialized clinical settings utilize elements based on these domains.5,13,14 The purpose of the current investigation was to develop an instrument for assessing team function during trauma resuscitation that incorporates these domains.
We modeled our trauma team behavior assessment instrument on the Mayo High Performance Teamwork Scale described by Malec et al.5 While reported to have acceptable rates of reliability in key domains and to be sensitive to effects of training, we found that this instrument in its original form performed poorly when used to grade team behavior in our simulated trauma resuscitations. For example, the Mayo instrument has a temporal component, where behaviors are graded in 3-minute increments. We found that our scenarios were sufficiently brief (approximately 5 minutes or less) and that partitioning them in such a fashion was not practical. Further, many of the components of the Mayo scale (eg, “Team members acknowledge—in a positive manner—statements directed at avoiding or containing errors or seeking clarifications”) were seldom applicable or graded by our observers.5 Why these elements did not seem relevant in our application is unclear. Conceivably, these elements might not be germane to relatively brief simulated trauma resuscitations such as the ones we utilized. Alternatively, these attributes might be important to effective team function but difficult to objectively assess. Finally, we found that interobserver agreement was poor, with individuals arriving at common agreement less than two-thirds of the time.
Based on these findings, we attempted to develop a streamlined team-scoring instrument more applicable to the trauma setting but that incorporated the essential elements of effective team function. Specifically, in a parsimonious and efficient fashion, we endeavored to capture the 4 domains of team behavior critical to effective crisis resource management articulated by Fletcher et al13 (table 3). In addition to including fewer elements and eliminating a temporal component, we incorporated a binary scoring system (ie, an attribute is scored as either present or absent). By incorporating fewer but nonetheless essential elements relative to its progenitor (the Mayo scale), we envision this instrument as being useful as an educational or instructional device for emphasizing those team elements critical to coordinated team function in trauma resuscitation (ie, members of trauma teams might routinely use this device as a mental checklist during trauma resuscitations). In our pilot analysis, this approach appeared to distinguish teams based on effectiveness of performance (figure).
Limitations
Our study has several limitations. As noted above, our instrument ideally would capture those team skills required during primary and secondary trauma survey, possess high rates of inter-rater and intra-rater agreement, and enable faculty to rapidly assess teams and provide immediate feedback in a variety of settings (in residency training, in a postgraduate course, or as part of ongoing quality assurance initiatives). However, our study is purely descriptive. Thus our trauma team-scoring instrument requires prospective validation to determine whether it possesses these attributes, whether it can meaningfully discriminate between effective and ineffective team function, and whether its operational characteristics are comparable when used by other groups of investigators. Further, the trauma simulation exercises we developed to test our instrument were brief. Examining trauma team performance using longer simulation scenarios may highlight important characteristics of team behavior not captured by our methodology. In addition, because no validated methods of assessing team function in the setting of trauma exist, we selected from the pool of videotaped trauma team resuscitations for purposes of testing our instrument based on the subjective judgment of experienced observers, not quantitative data. Varying the duration, complexity, and nature of our trauma resuscitations will likewise be important for determining whether our instrument is sufficiently robust and generalizable to be useful for the broad spectrum of trauma resuscitation encounters. We are currently conducting studies to assess these aspects of our instrument. Similarly, our study was conducted in a clinical simulation laboratory. We used this technology so as to create a consistent set of clinical circumstances to allow us to study team member dynamics. Nonetheless, such technology may lack sufficient realism so as to create barriers to extrapolating our findings to the clinical realm. Team behaviors learned, modified, and reinforced in a simulated environment might not effectively translate to actual patient care. Lastly, effective team function is presumably partly dependent on individual team member competencies. We did not specifically study the interaction between individual competency and group dynamics. Understanding this relationship may be important to optimizing team performance.
Conclusion
There is interest in integrating team training in the setting of trauma resuscitation into standardized surgical curricula.15 While intellectually appealing, direct evidence linking effectiveness of team function to clinically important outcomes in this setting is lacking. Such a demonstration will require instruments and methods capable of accurately assessing team function. Ideally, these instruments would have high inter-rater and intra-rater reliability, would discriminate effective from ineffective team function, and would possess diagnostic capability such that specific deficiencies in team function could be targeted for remediation. Finally, such instruments should be relatively simple to use such that they require minimal instruction and are applicable to most health care settings. We are currently planning prospective validation to determine whether our prototypical instrument possesses these operational characteristics in anticipation of use in investigative, educational, and quality assurance realms.
Acknowledgments
The authors acknowledge the generous support of the American College of Surgeons and Medical Education Technologies, Inc in funding this effort. Similarly, the authors are appreciative of the time and effort provided by members of the Departments of Emergency Medicine and Surgery at Washington University School of Medicine in assisting with the simulation exercises.
An abstract based on this work has been accepted for presentation at the Surgical Forum, 95th Annual Clinical Congress, American College of Surgeons, Chicago, Illinois, October 2009.
Footnotes
Nicholas Hamilton, MD, is a resident in general surgery at the Department of Surgery, Washington University School of Medicine; Bradley D. Freeman, MD, is Associate Professor in the Department of Surgery, Washington University School of Medicine; Julie Woodhouse, RN, is Nursing Administrator, SIM Research the Department of Anesthesiology, Washington University School of Medicine; Clare Ridley, MD, is a resident in anesthesiology in the Department of Anesthesiology, Washington University School of Medicine; David Murray, MD, is Professor in the Department of Anesthesiology, Washington University School of Medicine; and Mary E. Klingensmith, MD, is Professor at the Department of Surgery, Washington University School of Medicine.
Supported in part by a grant from the American College of Surgeons and Medical Education Technologies, Inc (B.D.F., M.E.K., J.W.)
References
- 1.American College of Surgeons. Advanced Trauma Life Support. 8th ed. Chicago, IL: American College of Surgeons; 2008. [Google Scholar]
- 2.Cherry R. A., Ali J. Current concepts in simulation-based trauma education. J Trauma. 2008;65:1186–1193. doi: 10.1097/TA.0b013e318170a75e. [DOI] [PubMed] [Google Scholar]
- 3.Marshall R. L., Smith J. S., Gorman P. J., Krummel T. M., Haluck R. S., Cooney R. N. Use of a human patient simulator in the development of resident trauma management skills. J Trauma. 2001;51:17–21. doi: 10.1097/00005373-200107000-00003. [DOI] [PubMed] [Google Scholar]
- 4.Holcomb J. B., Dumire R. D., Crommett J. W. Evaluation of trauma team performance using an advanced human patient simulator for resuscitation training. J Trauma. 2002;52:1078–1085. doi: 10.1097/00005373-200206000-00009. [DOI] [PubMed] [Google Scholar]
- 5.Malec J. F., Torsher L. C., Dunn W. F. The Mayo High Performance Teamwork Scale: reliability and validity for evaluating key crew resource management skills. Simul Healthc. 2007;2(1):4–10. doi: 10.1097/SIH.0b013e31802b68ee. [DOI] [PubMed] [Google Scholar]
- 6.Murray D. J., Boulet J., Ziv A., Kras J., McAllister J. D., Woodhouse J. An acute care skills evaluation for graduating medical students: a pilot study using clinical simulation. Med Educ. 2002;36:833–841. doi: 10.1046/j.1365-2923.2002.01290.x. [DOI] [PubMed] [Google Scholar]
- 7.Murray D. J., Boulet J., Kras J., Woodhouse J., Cox T., McAllister J. D. Acute care skills in anesthesia practice: a simulation based resident performance assessment. Anesthesiology. 2004;101:1084–1095. doi: 10.1097/00000542-200411000-00007. [DOI] [PubMed] [Google Scholar]
- 8.Murray D. J., Boulet J., Kras J., Cox T., McAllister J. D. A simulation based acute skills assessment for anesthesia training. Anesth Analg. 2005;101:1127–1134. doi: 10.1213/01.ane.0000169335.88763.9a. [DOI] [PubMed] [Google Scholar]
- 9.Lee S. K., Pardo M., Gaba D. Trauma assessment training with a patient simulator. J Trauma. 2003;55:651–657. doi: 10.1097/01.TA.0000035092.83759.29. [DOI] [PubMed] [Google Scholar]
- 10.Ali J., Gana T. J., Howard M. Trauma mannequin assessment of management skills of surgical residents after advanced trauma life support training. J Surg Res. 2000;93:197–200. doi: 10.1006/jsre.2000.5968. [DOI] [PubMed] [Google Scholar]
- 11.Barsuk D., Ziv A., Lin G. Using advanced simulation for recognition and correction of gaps in airway and breathing management skills in pre-hospital trauma care. Anesth Analg. 2005;100:803–809. doi: 10.1213/01.ANE.0000143390.11746.CF. [DOI] [PubMed] [Google Scholar]
- 12.Gilbart M. K., Hutchison C. R., Cusimano M. D., Regehr G. A computer-based trauma simulator for teaching trauma management skills. Am J Surg. 2000;179:223–228. doi: 10.1016/s0002-9610(00)00302-0. [DOI] [PubMed] [Google Scholar]
- 13.Fletcher G., Flin R., McGeorge P., Glavin R., Maran N., Patey R. Anaesthetists' non-technical skills (ANTS): evaluation of a behavioural marker system. Br J Anaesth. 2003;90(5):580–588. doi: 10.1093/bja/aeg112. [DOI] [PubMed] [Google Scholar]
- 14.Kim J., Neilipovitz D., Cardinal P., Chiu M., Clinch J. A pilot study using high-fidelity simulation to formally evaluate performance in the resuscitation of critically ill patients: the University of Ottawa Critical Care Medicine, High-Fidelity Simulation, and Crisis Resource Management I Study. Crit Care Med. 2006;34:2167–2174. doi: 10.1097/01.CCM.0000229877.45125.CC. [DOI] [PubMed] [Google Scholar]
- 15.Knudson M. M., Khaw L., Gaba D. Teamwork in the trauma bay (ACS/APDS surgical skills curriculum) Available at: http://elearning.facs.org. Accessed October 19, 2009.