Association between measured teamwork and medical errors: an observational study of prehospital care in the USA

Abstract

Objectives

The goal of this study was to examine the relationship between measured teamwork and adverse safety events in the prehospital emergency care of children using high-fidelity simulation. We posit that non-technical skills such as leadership, teamwork, situation awareness and decision-making are associated with the clinical success of teams.

Design

Observational study.

Setting

Emergency medical services (EMS) responders were recruited from public fire and private transport agencies in Oregon State to participate in four simulations of paediatric emergencies using high-fidelity patient simulators, scene design, and professional actors playing parents and bystanders.

Participants

Forty-four fire/transport teams consisting of 259 EMS professionals consented to participate and completed simulations.

Primary and secondary outcome measures

Teams were assessed using the Clinical Teamwork Scale (CTS), a validated instrument that measures overall teamwork and 15 specific elements in five overarching domains: communication, decision-making, role responsibility (leadership and followership), situational awareness/resource management and patient-friendliness. We used generalised estimating equations to estimate the odds of error with increasing overall CTS teamwork score while adjusting for clinical scenario and potential clustering by team.

Results

Across 176 simulations, the mean overall score on the CTS was 6.04 (SD 2.10; range 1=poor to 10=perfect) and was normally distributed. The distribution of scores was similar across the four clinical scenarios. At least one error was observed in 82% of the simulations. In simulations with at least one observed error, the mean CTS score was 5.76 (SD 2.04) compared with 7.16 (SD 1.95) in scenarios with no observed error. Logistic regression analysis accounting for clustering at the team level revealed that the odds of an error decreased 28% with each unit increase in CTS (OR 0.72, 95% CI 0.59 to 0.88).

Conclusions

This study found that overall teamwork among care delivery teams was strongly associated with the risk of serious adverse events in simulated scenarios of caring for critically ill and injured children.

Strengths and limitations of this study.

• This research provides some of the first data that quantifies the relationship between clinical teamwork and the likelihood of medical errors in a paediatric prehospital setting.

• These data suggest that Clinical Teamwork Scale may be a useful tool to measure the immediate impact of interventions on teamwork that matter to the reduction of errors.

• Simulations replicated paediatric emergencies known to be high risk for errors and were conducted using professional actors and high-fidelity paediatric simulators in the field with clinical emergency medical services (EMS) teams assembled and responding as they normally would reflecting everyday clinical care as closely as possible.

• The appraisal of errors and assignment of teamwork scores were performed by clinicians who were not involved in EMS but were humans using their best judgement, which is a method subject to bias.

• It remains uncertain whether poorer teamwork is the cause of errors or if it is simply associated with other deficits in performance that may be contributing; however, this finding provides a tool that can be used to measure impact before and after interventions.

Introduction

Medical errors are estimated to be the third leading cause of death in the USA and the 14th leading cause worldwide.^1–3 The total aggregated cost of adverse safety events and errors is estimated to amount to trillions of dollars each year.³ Medicine is both an important and imperfect field, combining science with art and humans with technology. In 1999, the world became aware that patients die from disease and traumatic events, and some also die or are injured unintentionally through the provision of clinical care.⁴ The landmark Institute of Medicine report estimated that the number of people who die every year due to preventable medical errors in the USA alone is equivalent to three jumbo jet crashes every 2 days, or one patient every 15 min, with over 1 million estimated to be harmed. Studies of medical errors have generally focused on adults and inpatient settings.^5–7 However, medical errors are also an important source of harm for paediatric patients with estimates of 70 000 paediatric inpatients harmed per year by medical errors.^{8 9} A more recent report emphasises that the cost of medical errors is proportionately more devastating to countries of low economic status.¹⁰ Even less is known about the epidemiology, aetiology and risk factors for preventable adverse events in the care of children in the emergency medical services (EMS) system where children first receive care for emergencies outside of the hospital.

Non-technical skills such as good communication and teamwork are critical in avoiding medical errors and assuring the safety of patients.³ WHO considers communication to be the leading cause of unintentional patient harm.¹¹ Past reviews have indicated that aspects of teamwork such as leadership, teamwork, situation awareness and decision-making are largely correlated with the clinical success of teams.^12–14 For example, a study evaluating teamwork in the intensive care unit (ICU) indicated that 37% of teamwork failures were due to verbal communication errors between physicians and nurses.¹⁵ There have been interventions to reduce medical errors such as implementing computerised provider order-entry systems, limiting residents’ work shifts to consecutive hours, and implementing evidence-based care bundles. A large retrospective study from the US Veterans Affair Hospital found that inpatient surgical mortality was reduced 18% following team training interventions.¹⁶ However, lack of prospective and detailed evaluation of teamwork elements and specific errors limit our understanding of the effectiveness of these practices.¹⁷ The goal of this study was to examine the relationship between measured teamwork and adverse safety events in the prehospital emergency care of children using high-fidelity simulation.

Methods

We followed STROBE guidance for reporting of observational studies.¹⁸

Study participants

In total, 176 simulations were performed over a 6-month timeframe with 44 teams of EMS providers recruited from public fire and private transport crews in three large counties surrounding Portland, Oregon in the USA. Portland is the largest city in Oregon and the three-county area that participated in this study serves a population of 1.8 million. The EMS system in this region has dual-Advanced Life Support response to all 911 calls with public fire agencies, responding in teams of three to five individuals with at least one paramedic, and a private transport agency, responding in two-person teams with at least one paramedic. We conducted all simulations in the field, using high-fidelity simulators, scene design, and professional actors playing roles of parents and bystanders, with both fire and transport EMS teams responding to each scenario as they normally would when providing clinical care. Participants ranged from Emergency Medical Technician to paramedic, with each team including at least one paramedic.

Forty-four fire/transport teams consisting of 259 EMS professionals were available in the three-county region during our study timeframe and consented to participate. All who consented completed the study. One individual withdrew consent to have his/her individual data analysed after completing the simulation sessions for personal reasons, but agreed to analysis of team-level data for all simulation sessions he/she had participated. In total, we analysed team-level data from 44 teams and individual-level data from 258 EMS professionals.

Patient involvement

The EMS-C Children’s Safety Initiative is an observational study with a specific focus on EMS providers. Patients were not involved in the design, recruitment or conduct of the study.

Simulations

Each fire/ambulance team participated in four simulation scenarios: (1) cardiac arrest in newborn (newborn resuscitation programme, NRP), (2) cardiac arrest in child (paediatric advanced life support, PALS), (3) non-accidental trauma (NAT) and (4) accidental trauma from pedestrian–motor vehicle collision (MVC). Simulations were developed by a diverse team with experience in patient safety, simulation, medical and EMS education, paediatric emergency medicine, advanced life support training and emergency medical services. Professional actors who each had several years’ experience working as standardised patients in simulations played the roles of patients and family members. They read through a standardised script and the full simulations and debriefings were beta tested exactly as they would run with participants with several EMS teams in two different EMS agencies, not participating in the study. Simulations were conducted in the field using high-fidelity simulators, scene design, and professional actors playing roles of parents and bystanders. EMS crews responded in their agency vehicles after receiving a radio dispatch. Simulators included Victoria birthing patient simulator and the Newborn Hal simulator from Gaumard, and Sim Jr. 6-year-old child and sim NewB newborn simulators from Laerdal. Settings included street (MVC), simulated apartment bedroom for birth (neonatal resuscitation protocol or NRP), dishevelled family room (neonatal abuse trauma or NAT), simulated care facility for 8-year-old with cardiac arrest (PALS) and all simulations were allowed to continue though transport in the agency’s own ambulances.

In the NRP scenario, the patient presented as a newborn who was initially responsive on delivery but subsequently developed a weak cry, flaccid extremities, bradycardia and cyanosis requiring resuscitation. For the PALS case, the patient presented as a 6-year-old boy with symptomatic bradycardia that progressed to pulseless electrical activity with a reported history of developmental disabilities, seizures and cerebral palsy, living in a care facility. In the NAT case, a 6-month-old patient had ‘fallen’ from the couch and was unconscious and unresponsive with bruising consistent with inflicted injury. In the MVC scenario, the patient was a 3-month-old child who, while in a stroller, was struck by a car. The patient had a large scalp wound and was initially crying but shortly after developed decreased responsiveness, hypertension, bradycardia and slow respirations. Makeup was used to simulate bruising, bleeding and lacerations in a standardised manner for each scenario. Each scenario followed a previously determined algorithm for improvement or decompensation of the patient’s status along prespecified time intervals according to performance of specific critical actions also specified in advance. All scenarios were pilot tested among EMS teams who did not subsequently participate in the study. Scenarios were revised as needed based on feedback from pilot testing. Professionally trained actors played all confederate roles in the scenarios to enhance realism.

For the purpose of standardisation, simulation scenarios concluded after approximately 10 min. If the team initiated a procedure close to the 10 min mark, such as calculating and drawing up a medication, the simulation was allowed to continue to complete observation of the task in process and reduce the chances of an awkward end to the scenario. If the care protocols proceeded rapidly, and the condition of the patient was stable and transport action had begun, the simulation could be concluded before the 10 min point. After finishing all four scenarios, teams returned for a 30 min debriefing session.

Simulation scenarios were presented to teams in random order to minimise the potential bias that may occur from conditioning as teams become familiar with the simulators and working with each other. Fire and transport crews responded to each scenario in their own fire engines and ambulances after receiving a scripted radio dispatch. Each crew used their own equipment or alternatively used training kits that were replicas of the kits they usually carry. The crews varied who arrived on scene first (ambulance or fire department) and at times arrived simultaneously to mimic local practice. Crews were able to move the patient simulators to the transport vehicle and mimic patient transport to the hospital. If transport was started, the crews would assign a driver and those who remained inside the transport vehicle would continue to administer patient care. All crews were aware they were participating in simulated emergencies and were oriented to the study and mannequins prior to participation. To allow for anonymous tracking of each participant by role, the crew members wore coloured tape markers on their shoulders. The fire crews also varied the person-in-charge assignment for each scenario, although the crew membership was consistent.

Data collection

Prior to simulations, participants completed a survey asking about demographic characteristics, training and their experience in EMS.

One of two subject matter experts (J-MG and MH) experienced in using the Clinical Teamwork Scale (CTS) directly observed simulations in real time and measured teamwork using the CTS, a validated instrument that measures overall teamwork and 15 specific elements in five overarching domains in addition to overall teamwork: communication, decision-making, role responsibility (leadership and followership), situational awareness/resource management and patient-friendliness.¹⁹ CTS was selected because the authors were skilled in its use having developed it over a decade ago with a diverse team (that included one of the founders of crew resource management), it was designed specifically to allow quick assessments of teamwork during rapidly moving clinical emergencies, it is free, and has been successfully used by us and numerous others to measure teamwork in emergent clinical scenarios similar to our intended purpose: paediatric resuscitations, obstetrics, trauma, advanced cardiac life support and emergency medical services.^20–27 Evaluators rate the performance of the care team using a Likert scale of zero (unacceptable) to 10 (perfect) and are given space to insert narrative comments. In validation studies, the CTS has demonstrated substantial score concordance among raters and excellent inter-rater reliability.^26–28 A systematic review of tools that have been used to measure teamwork recently concluded that CTS was superior to other tools for measuring teamwork citing content and construct validity as well as reliability and ease of use.²⁹ Prior to the study, simulations were pilot tested with EMS teams not involved in the study. During this time, both reviewers scored teamwork independently using CTS arrived at consensus and repeated independent assessments of new simulations to arrive at consistency. Two reviewers were present, independently scored scenarios and came to consensus at the end of pilot tests and during more than half of all study simulations.

Error measurement

During the simulations, the evaluators noted the number of errors and described what the error entailed using standard taxonomy. For purposes of this study, errors were collapsed into whether an error was present or not present. Similar to CTS, when both reviewers were present (>50% of the time), a consensus was reached at the end regarding errors. When there was uncertainty over whether an action may or may not have constituted an error, the team discussed to reach consensus. For the remaining scenarios, a single reviewer rated the errors.

Statistical analysis

A completed case analysis was used; due to missing data on key variables, 6 of 176 (3.4%) of simulation scenarios were dropped from analysis. We summarised error proportion by each simulated clinical scenario, and summarised teamwork (CTS teamwork score) by the presence or absence of an error. We summarised teamwork as the overall CTS score and also by each score in the five subdomains of the CTS. Scores were compared between groups using t-tests for unequal variances. A generalised estimating equation (GEE) logistic regression model was used to estimate the odds of error with increasing CTS score in SAS GENMOD (V.9.4; SAS Institute, Cary, North Carolina, USA). Although the order of the four simulation scenarios was randomised by EMS teams, in analysis we controlled for the potential correlation of errors by team using an exchangeable correlation structure clustered by teams. Our choice to use the exchangeable correlation structure should be robust to errors in adjacent simulation scenarios and to those farther apart in time on the testing day, as well as scenarios that may share similar characteristics (eg, same age of simulated pediatric patient). We tested for confounding by scenario and years of EMS experience. A p value of <0.05 was considered statistically significant.

Results

Table 1 shows demographic characteristics and self-reported professional experience of EMS responders participating in the simulation sessions.

Table 1.

Characteristics of emergency medical services (EMS) personnel participating in simulations

Characteristic	N (%)
Mean age, years (SD)*	36.9	(8.47)
Gender† Female	36	(14.0)
Race/ethnicity‡
White	220	(89.1)
Black or African American	0	(0)
American Indian or Alaska Native	6	(2.43)
Asian	4	(1.62)
Native Hawaiian or Other Pacific Islander	2	(0.81)
Hispanic or Latino	7	(2.83)
Other/two or more races	8	(3.24)
Training level†
Emergency Medical Responder	0	(0)
Emergency Medical Technician (EMT)	100	(38.9)
Advanced EMT	2	(0.78)
EMT-intermediate	25	(9.73)
Paramedic	128	(49.8)
Paramedic intern	2	(0.78)
Years’ experience working in EMS§	11.7	(7.84)
Proficiency in paediatric EMS on a scale of 1 (“Novice”) to 5 (“Expert”)†	2.55	(0.89)

*n=256.

†EMS n=257.

‡n=247.

§n=258.

At least one error was observed in 82% of the 176 simulation scenarios for which the presence or absence of errors was recorded. Due to missing data for overall CTS score, data for six scenarios were not included in the logistic regression analysis. Table 2 describes the types of errors observed.

Table 2.

Taxonomy of observed errors

Errors	Type
Diagnostic and management	Assessment, algorithm, diagnostic, management, decision-making
Medication	Wrong medication, dose, route, sequence, type, intravenous fluid issue, etc
Technical	Wrong device, size, bagging, intubation, CPR, defibrillator, immobilisation
Scene/environment	Scene safety, scene time, positioning, patient/family

The overall CTS score was normally distributed, with a mean of 6.04 (SD 2.10; min=1 (poor) and max=10 (perfect)) in the 170 simulations with recorded overall CTS scores. The distributions of overall CTS scores were similar across the four scenarios. In simulations with an overall CTS score and at least one observed error (n=138), the mean CTS score was 5.76 (SD 2.04) compared with 7.16 (SD 1.95) in scenarios with no observed error (n=32). The t-tests demonstrated that this difference was statistically significant (p=0.0007). For each individual simulation scenario, teams with no errors had higher CTS scores than teams with one or more errors. There were statistically significant differences in CTS scores between teams with no errors and teams with errors in the Non-Accidental Trauma simulation and the Newborn Cardiac Arrest simulation, with the no error teams having significantly greater CTS scores. Table 3 provides comparative data for CTS scores between teams with and without errors by simulation scenario.

Table 3.

Comparing CTS overall teamwork score between teams with and without errors by simulation scenario

Simulation scenario	No error	One or more errors	T*	P value
	(mean, SD)	(mean, SD)
Motor vehicle collision	6.20 (2.25)	5.91 (1.96)	0.37	0.72
Non-accidental trauma	8.20 (1.10)	5.57 (2.03)	4.39	0.0019†
Newborn cardiac arrest	7.55 (1.97)	5.79 (2.18)	2.5	0.022†
Six-year-old cardiac arrest	7.17 (1.60)	5.78 (2.09)	1.88	0.097
All simulations	7.16 (1.95)	5.76 (2.04)	3.61	0.0007†

*Satterhwaite t-test for unequal variances.

†Significant at alpha=0.05.

After comparing an unadjusted GEE logistic regression model to a model controlling for scenario and mean years of EMS experience, there were no significant differences in the effect estimate for CTS score or significance of estimates (table 4). This indicates that the relationship between overall CTS and odds of error was not confounded by scenario or years of EMS experience; thus, the final model reported is unadjusted for these variables. Logistic regression analysis accounting for clustering at the team level revealed that the odds of an error decreased 28% with each unit increase in CTS (OR 0.72, 95% CI 0.59 to 0.88).

Table 4.

Generalised estimating equation logistic regression models testing associations between teamwork and errors

Predictor variable	Unadjusted				Adjusted*
	OR	95% CI	Z	P value	OR	95% CI	Z	P value
CTS score (1-unit increase)	0.72	(0.59 to 0.88)	−3.2	0.0014	0.73	(0.59 to 0.89)	−3.06	0.0022
MVC	–	–	–	–	0.45	(0.18 to 1.17)	−1.63	0.1034
NRP	–	–	–	–	0.46	(0.20 to 1.04)	−1.87	0.0618
PALS	–	–	–	–	0.88	(0.25 to 3.03)	−0.21	0.8352
NAT	–	–	–	–	REF	–	–	–
Mean years EMS experience	–	–	–	–	0.92	(0.79 to 1.07)	−1.07	0.284

*Adjusted for clinical scenario (NAT=reference group) and mean years of EMS experience at the team level.

EMS, emergency medical services; MVC, motor vehicle collision; NAT, non-accidental trauma; NRP, newborn resuscitation programme; PALS, paediatric advanced life support.

Within the simulations, the percentage of teams that completed a simulation without error was highest in the Newborn Cardiac Arrest simulation (25%). The simulation with the highest percentage of teams with one or more error was the Non-Accidental Trauma simulation, with 89% of simulation having at least one error. Bivariable analysis suggested that adverse safety events are more likely in teams with low teamwork scores.

Adverse safety events were also more likely in teams with lower individual scores in communication, situational awareness, decision-making and leadership/followership (table 5).

Table 5.

CTS domain-specific ratings in simulations with and without any observed error

Error	CTS item rating
CTS item	No error (mean, SD)	One or more errors (mean, SD)	P value
Overall communication*	7.13 (2.21)	5.89 (2.00)	0.002
Overall situational awareness†	7.31 (2.07)	6.26 (2.25)	0.01
Overall decision-making‡	7.26 (2.07)	5.49 (2.35)	<0.001
Overall role responsibility (leader/helper)§	7.25 (1.74)	6.06 (1.92)	0.001
Patient friendliness§	7.13 (2.56)	6.44 (2.17)	0.16

*n=160.

†n=170.

‡n=167.

§n=163.

Compared with teams with one or more errors, teams without any errors had higher CTS scores across all itemised categories of the CTS (Overall Communication, Overall Situational Awareness, Overall Decision-Making, Overall Role Responsibility and Patient-Friendliness). Among these categories, teams with no error had significantly higher CTS scores for Overall Communication, Overall Situational Awareness, Overall Decision-Making and Overall Role Responsibility (Leadership/Helper).

Discussion

In this study of simulated paediatric prehospital emergencies, we observed a higher frequency of errors in teams with lower-rated teamwork scores. This finding was observed consistently across a variety of clinical scenarios (eg, both newborn cardiac arrest and non-accidental trauma) and teamwork subdomains (eg, communication and situational awareness). Our study builds on the prior research on the role of teamwork in medical errors, which has mostly been conducted in the hospital setting.^{30 31} Using high-fidelity simulation scenarios and a validated teamwork measurement tool, we extended this research to the paediatric prehospital emergency care setting, and found an association between teamwork and medical errors.

The results were robust to multivariable regression adjustment, suggesting that breakdowns in teamwork are indeed a factor contributing to the high error rates observed in this study. These findings add strength to WHO’s description of communication as a leading cause of unintentional harm and further elaborate on the importance of teamwork in decreasing errors.¹¹ In addition, these findings provide a nice complement to the work on surgical mortality and team training¹⁶ as this provides a mechanism to measure the immediate intervening factors that are likely in the causal pathway.

Given the large cost and morbidity burden associated with medical errors worldwide, these findings suggest directions for future research and also have policy implications.^{3 10} Most immediately, however, more studies are needed to confirm this association both in this specific healthcare setting as well as other clinical areas where team-based care is the norm. For example, future studies should examine the contribution of teamwork to medical errors in hospital emergency medical care to continue building our understanding of the role of teamwork beyond the ICU. Improving teamwork is a relatively low-cost intervention to improve patient outcomes. Thus, this direction of research is important internationally and may be particularly critical for countries of low economic means, who are impacted heavily by the financial burdens.¹⁰

We used a validated measure of teamwork, the CTS. CTS has been used around the world in different settings and clinical care areas.^26–28 Future studies should use this measure and also continue to refine methods for measuring the multimodal concept of teamwork in healthcare systems around the world. Given the complexity of teamwork and the inherent challenges in measuring it (eg, different evaluations based on different parties, the limitations of self-reported/self-scored data, etc), additional research will be needed to provide a full picture of the role of teamwork in patient outcomes. In clinical settings where our findings are replicated, it will be essential to consider interventions, training and policies to prevent teamwork-associated medical errors.

There are several important limitations to consider. We conducted simulations and measured teamwork in one geographical region with a specific EMS system design which may not be representative of other areas. Although we employed best practices in conducting our simulations and measuring teamwork (eg, cutting-edge simulation models, the use of trained actors, a validated teamwork measurement scale), our study examined simulated healthcare encounters rather than actual healthcare. However, due to the rare nature of critical paediatric emergencies in EMS, direct observation of care is not feasible on this scale. Next, we do not know if poorer teamwork is the cause of errors or if it is simply associated with other deficits in performance that may be contributing, such as medical knowledge. Further, the appraisal of errors and assignment of teamwork scores were performed by humans using their best judgement, which is a method subject to bias. Finally, while the amount of missing data were small, we dealt with missing data using complete-case analysis (170 complete of 176 available cases). The analytical approach to missing data continues to be a subject of active investigation with recent writings suggesting that the best approach is to think through the likely causes of bias and select the analytical method that is least likely to introduce bias and best able to reduce bias.³² Although several options are available for addressing missingness (eg, multiple imputation approaches), several features of our specific data raised questions about the applicability of these approaches in our study. Our data structure includes nested, non-independent observations, where the same clinical team participated in clinically distinct simulations and different clinical teams participated in each clinical simulation. Given this, it did not seem appropriate to impute scores from other clinical teams in the same clinical scenario to the team with missing data, as this would assume that the clinical performance of teams was homogeneous. In the same note, it did not seem appropriate to assume that one clinical team would perform similarly on clinical cases that were very distinct. Given that neither of these assumptions seemed appropriate and also that our total number of missing data was small, the most honest interpretation of our data seemed to be complete-case analysis. In conclusion, we found that teamwork is highly correlated with errors in simulated paediatric prehospital emergencies and this finding was robust across four different simulation scenarios as well as across subdomains of teamwork.

Conclusion

Overall teamwork is strongly associated with the risk of adverse events in critically ill and injured children.