Out-of-Hospital Pediatric Patient Safety Events: Results of the CSI Chart Review

Introduction

Sentinel studies have identified medical errors in hospitalized adults as a significant source of morbidity and mortality.^1-3 While the epidemiology and contributors to adverse events in the hospital setting have been characterized, less is known about safety events in the out-of-hospital setting^4,5 where research has largely focused on traffic and vehicular safety,^6–14 or on specific issues such as medication errors^15–23 or airway management as opposed to a more comprehensive analysis.^24–29 Even less is known about the nature and severity of patient safety events in the out-of-hospital care of children, who comprise an estimated 800,000 to 1,600,000 EMS transports annually in the United States.³⁰

EMS personnel including first responders, EMTs, and paramedics care for patients of all ages across a wide range of conditions, often under challenging conditions. Children comprise only 8.9–13% of out-of-hospital calls and EMS personnel have limited exposure to sick and critically ill children.^30-31 Furthermore, children have unique characteristics and medical needs that make them vulnerable to potential medical care errors including: weight-based medication dosing and size-based equipment needs; age-dependent anatomic and physiologic differences; and developmental and communication characteristics that complicate physical assessment and medical decision making.³² These factors, combined with limited pediatric training and provider comfort caring for children,^33–42 create a low-frequency, high-stakes situation of increased risk for medical error.³²

We present results from a blinded medical record review conducted as part of the Children’s Safety Initiative-Emergency Medical Services (CSI-EMS)^43–45 to characterize the frequency, severity, and preventability of patient safety events that occur during the out-of-hospital care of children.

Methods

CSI-EMS is a large multi-phase study funded by the National Institutes of Health (NICHD R01HD062478). We reviewed records for all “Code 3” (lights and sirens) transports from 2009–2011 for patients < 18 years of age in Multnomah County, Oregon, where both fire department and separate private transport agencies respond to calls. We chose Code 3 records in order to capture children with the most serious illness or injury with increased likelihood of requiring EMS care and therefore at risk for safety events. Multnomah County is an urban area (population > 700,000) that includes Portland, the largest city in Oregon. Private ambulance transports are staffed with ALS crews of 2 paramedics and fire units have 4–5 person crews including at least 1 paramedic. Both fire and ambulance agencies respond to all calls. We included all charts in the electronic record systems of the fire and ambulance agencies in 2009–11, effectively creating a population-based period cohort for analysis. We reviewed charts from the transporting agency, and when available the fire response unit (fire department units do not complete a chart when they do not provide care at the scene).

Pairs of paramedics and emergency physicians independently reviewed charts in a blinded manner using a structured chart review tool, mirroring the process established by Brennan et al. in their seminal analysis of hospital safety.¹ We recruited 13 paramedic reviewers among the local EMS community who were not employed by the agencies submitting charts in order to assure anonymity and avoid potential bias in judging co-workers. The 7 physician reviewers were faculty and fellows working in the pediatric Emergency Department of a local tertiary care/trauma center that provides online medical control for much of the Portland metropolitan area. All reviewers completed a 2-hour training providing guidance and hands-on experience with the tool. As an additional quality control, we developed a reference manual with examples of commonly encountered decision points in the chart review process. To assure reliability, all paired chart reviews were arbitrated by a third reviewer who was one of two pediatric emergency medicine co-investigators with experience in out-of-hospital care. The determination of the arbiter was considered final and used in the data analysis.

We modelled the chart review tool after the “Retrospective Case Record Review” from the Harvard Medical Study,^1,2 and adapted it to the out-of-hospital setting based on results from our CSI-EMS focus group study.⁴³ Consistent with recommendations for retrospective chart reviews by Vassar and Holzmann,⁴⁶ we pretested multiple iterations using a sample of 30 high-risk charts from 2011.

We specifically adapted the tool to capture a broad range of potential and manifest safety events similar to the construal of adverse events proposed by Patterson and colleagues.⁴⁷ Our tool assessed: Unintended injury or consequences, Near misses, Suboptimal actions, Errors, and Management complications (UNSEMs). The instrument contained 145 questions including dichotomous answers, lists, Likert-type scales and free text, and was divided into four sections: 1) reviewer and case information, 2) medical care domains, 3) summary of patient condition and UNSEMs, and 4) summary of case. The tool was completed online using SurveyMonkey™ and incorporated automatic skip patterns (Appendix A).

We structured the tool to identify potential UNSEMs within specific domains of medical care: resuscitation; assessment, impression/diagnosis, and clinical decision-making; airway/breathing; fluids and medication; technical procedures (e.g., vascular access); equipment; environment (scene and transport characteristics); and systems (protocols, guidelines, staffing). Within each domain, we asked reviewers to determine whether an UNSEM occurred and the degree to which the UNSEM could have harmed the patient (“no harm likely or a near miss,” “mild or temporary harm, including additional treatment,” or “permanent or severe harm, including death”). Finally, the reviewers assessed the degree to which the UNSEM as a whole was preventable (“0” = impossible to prevent to “10” = entirely preventable).

Analyses:

We excluded cases representing interfacility transport of pediatric patients from final analyses as we felt that such cases represented children already evaluated and stabilized with procedures, medications, and management plans often in place and determined by the transferring or accepting physician. We used frequency distributions and cross-tabulations to examine the distribution of the independent variables (age, gender, reason for dispatch, and interventions performed) with the two outcome variables of at least one severe UNSEM and at least one mild UNSEM. We constructed univariate logistic regression models to examine associations between each independent variable and each outcome separately, assessed statistical significance using Wald F statistics and their associated P-values statistical significance, and expressed associations as unadjusted odds ratios (ORs) and 95% confidence intervals (CIs). We set the initial inclusion criterion for multivariate regression modeling at P<0.25 as the analysis was exploratory. We entered all variables meeting these criteria into a multivariate logistic regression model using the outcome of at least one severe UNSEM. We considered associations with a P < 0.05 or a 95% confidence interval that did not include the value 1 statistically significant. We excluded the variable “reason for dispatch” from the final multivariate model due to high colinearity with types of interventions performed (e.g., resuscitation, airway management). We performed all analyses using SAS (Version 9.4, SAS Institute, Inc., Cary, NC).

The Institutional Review Board from Oregon Health & Science University approved all study components and protocols (IRB Number 00006942).

Results

From a total of 11,328 pediatric transports, we identified 497 critical transports (4.4%) during the study period. We excluded two transports because of missing data and five due to inability to obtain the associated fire department chart, leaving 490 charts for review. Scene calls represented 77.1% of cases (112/490, 22.9% were inter-facility transfers), of which 41.8% (205) involved care delivery by both fire and ambulance teams. The most common reasons for interfacility transfer were trauma (45/112, 40.2%), respiratory distress (35/112, 31.3%), and altered level of consciousness (12/112, 9.1%). Reflecting the urban setting of the study, the median ambulance response, scene, and transport times were 5 minutes, 12 minutes, and 14 minutes, respectively. Table 1 presents the reasons for dispatch by age for the 378 scene calls. The most common reason for the EMS call was trauma (158/378, 41.8%), followed by seizure (95/378, 25.1%). In more than a quarter of calls (101/378, 26.7%), reviewers classified the patient’s condition as “critical/severe” or a death occurred.

Table 1.

Reasons for dispatch by pediatric patient age group among 378 code-3 ambulance transports

Reason for Dispatcha	Age of patient
	0–28 d n (%)b	29 d-11 m n (%)b	12 m-5 y n (%)b	6–11 y n (%)b	12–18 y n (%)b	Total 0–18 y n (%)c
Trauma	1 (0.6)	5 (3.2)	47 (29.8)	27 (17.1)	78 (49.4)	158 (41.8)
Seizure or ALOC	0 (0.0)	8 (8.2)	53 (55.8)	21 (22.1)	13 (13.7)	95 (25.1)
Respiratory distress	5 (11.4)	5 (11.4)	22 (50.0)	9 (20.4)	3 (6.8)	44 (11.6)
Cardio and/or respiratory arrest	5 (15.6)	18 (56.2)	4 (12.5)	1 (3.13)	4 (12.5)	32 (8.4)
Poisoning, ingestion, intoxication	0 (0.0)	0 (0.0)	8 (32.0)	2 (8.0)	15 (60.0)	25 (6.6)
Other (including birth or delivery)	4 (66.7)	0 (0.0)	0 (0.0)	0 (0.0)	2 (33.3)	6 (1.6)
Pain (non-trauma)	0 (0.0)	0 (0.0)	1 (12.5)	0 (0.0)	7 (87.5)	8 (2.1)
Allergic reaction or anaphylaxis	1 (10.0)	3 (30.0)	3 (30.0)	1 (10.0)	2 (20.0)	10 (2.6)
Total	16	39	138	61	124	378

^aReason for Dispatch (Chief Complaint)

^bPercent of total for row

^cPercent of total for column

We identified at least one UNSEM in 262/378 (69.3%) of pediatric transports. Figure 1 shows the proportion of safety events by type of UNSEM. Of all UNSEMs identified, 61 (23.3%) were classified as having the potential to cause severe or permanent harm to the patient, including death, and 201 (76.7%) UNSEMs as potentially causing mild or temporary harm, including the need for additional treatment. The kappa statistic for inter-rater reliability between the two reviewers with regard to the occurrence of a safety event in the care domains was 0.62, a value that is considered substantial and is on par with the original safety study by Brennan.^1,46,47

Figure 1.

Types of UNSEMs observed in 378 code-3 ambulance transports of pediatric patients.

The frequency and severity of UNSEMs varied by patient and call characteristics (Table 2). Although neonates and infants (<1-year-old) comprised only 14.6% (55/378) of all cases, they accounted for 36.1% (22/61) of severe UNSEMs. Similarly, severe UNSEMs were disproportionately higher among dispatches for cardiopulmonary arrest.

Table 2.

Occurrence of severe and mild UNSEMs^a by patient and call characteristics among 378 code-3 ambulance transports of pediatric patients

Characteristics	Total Transports n	One or More Severe UNSEMsa n (%)	One or More Mild UNSEMsa n (%)
Patient Characteristic
Age
0–28 days	16 (4.2)	6 (9.8)	6 (3.0)
29 days – 11 months	39 (10.3)	16 (26.2)	15 (7.5)
12–24 months	138 (36.5)	16 (26.2)	84 (41.8)
25 months – 11 years	61 (16.1)	10 (16.4)	29 (14.4)
12–17 years	124 (32.8)	13 (21.3)	67 (33.3)
Genderb
Male	153 (40.5)	23 (37.7)	114 (56.7)
Female	224 (59.3)	29 (39.7) 37 (60.7)	87 (43.3)
Reason for Dispatch
Trauma	158 (41.8)	15 (24.6)	95 (47.3)
Seizure or ALOCc	95 (25.1)	12 (19.7)	51 (25.4)
Respiratory Distress	44 (11.6)	9 (14.8)	22 (11.0)
Cardiorespiratory Arrest	32 (8.5)	21 (34.4)	7 (3.5)
Poisoning/Ingestion/ Intoxication	25 (6.6)	1 (1.6)	13 (6.5)
Other (including Birth/Delivery)	6 (1.59)	0 (0.0)	4 (2.0)
Pain (non-trauma)	8 (2.1)	0 (0.0)	6 (3.0)
Allergic Reaction/Anaphylaxis	10 (2.6)	3 (4.9)	3 (1.5)
EMS Scene Cared
Fire and Ambulance	198 (52.4)	37 (60.7)	104 (51.7)
Ambulance Only	180 (47.6)	24 (39.3)	97 (48.3)
Total	378	61	201

^aUNSEM = A patient safety event involving: Unintended injury or consequence; Near miss; Suboptimal action; Error; or Management complication

^bIn 2 charts, gender of patient was not recorded.

^cALOC = altered level of consciousness

^dFire Department units respond to each call, however, the ambulance units can elect to waive them off. If no Fire Department chart was available, it was assumed that the Fire Department responders provided no patient care or interventions.

Table 3 presents the frequency and severity of UNSEMs by domain of EMS care provided. UNSEMs were observed across all domains of care, but were more likely to be severe when they occurred within resuscitation. For example, we judged multiple unsuccessful attempts at endotracheal intubation in lieu of CPR with bag-valve-mask ventilation in an infant with asystolic cardiopulmonary arrest to represent a potentially severe error. UNSEMS were relatively less common and less severe when they involved environmental, equipment, or systems issues.

Table 3.

Occurrence of severe and mild UNSEMs^a by EMS care domain among 378 critical ambulance transports of pediatric patients

EMS Care Domain	Severe UNSEMs n (%)b	Mild UNSEMs n (%)b
Assessment, Impression/Diagnosis, and or clinical decision making	46 (9.4)	45 (9.1)
Resuscitation	41 (10.2)	34 (6.9)
Airway Management	32 (6.5)	26 (5.3)
Medications	27 (5.5)	46 (9.4)
Procedures	21 (4.2)	56 (11.4)
Equipment	9 (1.8)	13 (2.6)
System	8 (1.6)	14 (2.9)
Fluids	8 (1.6)	15 (3.1)
Environment	2 (0.04)	13 (2.6)
Total	194	262

^aUNSEM = A patient safety event involving: Unintended injury or consequence; Near miss; Suboptimal action; Error; or Management complication

^bPercent among 378 transports

Table 4 presents the results of univariate regression of patient, call, and EMS care characteristics on the occurrence of a severe UNSEM. We observed statistically significant associations for patient age, with increased risk of a severe UNSEM among younger age groupings. In particular, neonates (0–28 days) had a more than five-fold increase in odds of a severe UNSEM and infants (29 days – 11 months) relative to adolescents (12–17 years). There was no association observed for gender. Among reasons for dispatch, we observed an increased ORs for a severe UNSEM for cardiopulmonary arrest, anaphylaxis, and respiratory distress relative to trauma. Interventions involving resuscitation, airway management, or other procedures, or administration of medications also had increased odds of a severe UNSEM.

Table 4.

Univariate predictors of observing at least one severe UNSEM^a among 378 code-3 ambulance transports of pediatric patients

Characteristic	Odds Ratio	95% CI	P value
Patient Characteristic
Age
0–28 days	5.12	1.53 – 17.1	0.003
29 days – 11 months	5.94	2.38 – 14.8	<0.001
12–24 months	1.12	0.51 – 2.44	0.78
25 months – 11 years	1.67	0.68 – 4.09	0.25
12–17 years (reference category)	1.00	--	--
Gender
Male	1.12	0.64 – 1.97	0.70
Call Characteristic
Reason for Dispatch
Trauma (reference category)	1.00	--	--
Seizure or ALOCC	1.38	0.61 – 3.09	0.43
Respiratory Distress	2.45	0.98 – 6.13	0.05
Cardiorespiratory Arrest	18.2	6.22 – 53.2	<0.001
Poisoning/Ingestion/Intoxication	0.40	0.05 – 3.18	0.37
Other (including Birth/Delivery)	--	--	--
Pain (non-trauma)	-- --	-- --	--
Allergic Reaction/Anaphylaxis	4.08	0.93 – 17.9	0.04
Interventions Performed
Resuscitation	6.12	3.03 – 12.3	<0.001
Airway Management	6.37	2.42 – 16.8	<0.001
Fluid Given	1.51	0.83 – 2.77	0.18
Medication Administered	3.35	0.72 – 3.12	<0.001
Any Other Procedure	1.50	0.72 – 3.11	0.27
No Interventions	0.30	0.07 – 1.30	0.09

^aUNSEM = A patient safety event involving: Unintended injury or consequence; Near miss; Suboptimal action; Error; or Management complication

The results of multivariate analysis are shown in Table 5. Statistically significant predictors of a severe UNSEM included: patient age (infants 29 days-11 months), and resuscitation; neonatal age (0–28 days) and children requiring procedures showed a trend towards increased odds for a severe UNSEM. While we observed an increased odds for calls involving airway management in univariate analysis, the adjusted OR did not meet an alpha of 0.05. We evaluated age as an effect modifier of cardiorespiratory arrest and resuscitation, but the number of observations was too small and estimates were unstable.

Table 5.

Multivariate predictors of observing at least one severe UNSEM^a in code-3 ambulance transports of pediatric patients^b

Patient and Call Characteristics	Odds Ratio	95% CI	P Value
Age
0–28 days	4.09	0.92 – 15.8	0.05
29 days – 11 months	3.33	1.25 – 8.68	0.02
12–24 months	0.86	0.33 −2.11	0.75
25 months – 11 years	1.37	0.48 – 3.63	0.54
Gender
Male	1.21	1.04 – 2.36	0.54
Interventions Performed
Any resuscitation	3.10	1.16 – 8.28	0.02
Any medication	1.58	0.70 – 3.45	0.27
Any fluid given	1.00	0.44 – 2.08	0.99
Any airway management	1.92	0.64 – 6.24	0.27
Other procedure	1.89	0.98 – 3.60	0.06

^aUNSEM = A patient safety event involving: Unintended injury or consequence; Near miss; Suboptimal action; Error; or Management complication

^b378 observations; pseudo R-squared = 0.18

Figure 2 displays the distribution of reviewer ratings of the preventability of identified safety events. Almost half of the UNSEMs (120/165, 45.3%) were judged “entirely preventable,” such as failure to check blood glucose in an infant with altered mental status.

Figure 2.

Preventability of UNSEMs observed in 378 code-3 ambulance transports of pediatric patients.

Discussion

Our review of records for critical pediatric patient transports in a large U.S. metropolitan area suggests that patient safety events are relatively common and have the potential for serious consequences. To our knowledge, this analysis of real-world pediatric transport data is the first to quantify the extent and degree of safety events in the out-of-hospital care of children. While hospital-based patient safety research has focused primarily on direct harm to patients as a result of medical errors or neglect and presupposes knowledge of patient outcomes, we intentionally broadened the definition of safety events to capture all opportunities to improve care for children in the out-of-hospital setting, where long-term patient outcomes are often unknown. We found that safety events in the out-of-hospital care of children occurred in approximately two-thirds of critical pediatric ambulance transports, with nearly one-fourth rated as severe. Calls involving infants and those in which resuscitation was required appear to experience increased odds for the occurrence of severe safety events.

Our study adds important epidemiologic perspective to prior research that has largely focused either on individual aspects of safety or data derived from simulated events rather than the broader landscape of patient safety issues.⁴ A few qualitative studies have attempted to identify key issues or potential contributors to patient safety in the out-of-hospital setting, and our results support some of these findings.^50–52 From the provider perspective, clinical judgment, training, procedural skill performance, medication or calculation problems, equipment size, parental interference, and discomfort with young patients were identified as important contributors to patient safety events. While our findings support these providers’ impression that younger children are potential high risk for patient safety events, we did not find significant contributions related to equipment or parental interference as suggested by these qualitative studies.

Only a few studies have examined the epidemiology using objective data from medical records.^53–61 An Australian study of ambulance transport errors and management deficiencies identified safety events in 77% of cases, of which nearly 2/3 were deemed to have contributed to death.⁶² Only a small proportion of patients in this study were children, but our results suggest that children have a similar risk for out-of-hospital errors associated with procedures required during resuscitation.

In addition to documenting the frequency of pediatric out-of-hospital patient safety events, we explored characteristics of patients, reason for call, and EMS care domain in order to identify factors associated with heightened odds of a severe event. Neonates and infants had the highest odds of experiencing a severe UNSEM in our sample in both univariate and multivariate analyses. There are a number of potential explanations for this observation. Previous studies reported that EMS providers are least comfortable caring for younger children, which may be related to both a lack of exposure and lack of training specific to neonates and infants.^36,43,51,52 While children account for only a small proportion of all EMS transports nationally, neonatal and infant transports are even less common.^30,63

EMS care domains associated with increased odds of a severe UNSEM in univariate analysis in this study included airway procedures, medication administration, and resuscitation. Difficulties with pediatric out-of-hospital airway management are well documented, including a large prospective study which has fueled debate as to whether paramedics should perform endotracheal intubation of children in the field.^24–29,64 Similarly, a number of studies have documented high rates of EMS errors related to medication administration.^{4,17,21,22,23,65} Suggested and observed contributors to medication errors in the out-of-hospital setting include difficulties estimating patient weight for weight-based medication dosing; calculation errors; decimal errors; lack of standardized drug concentrations and packaging; failure or difficulty in using length- or weight-based medication protocols; and limited experience and training in pediatrics.

Two variables significantly associated with severe UNSEMs in our univariate analysis were cardiopulmonary arrest and EMS resuscitation, which are intimately linked. While EMS resuscitation training through PALS has been shown to increase provider knowledge,⁶⁶ self-efficacy and comfort caring for children,³⁸ and even improve critical skills such as airway management and vascular access,^67,68 training in PALS and NRP is typically infrequent,³⁶ and skills decay quickly.^69,70 Provider stress, particularly in managing pediatric cardiopulmonary arrest, is likely an additional factor contributing to patient safety events in this context.⁷¹

Several limitations to this study must be acknowledged. We used EMS patient care records to assess the occurrence of patient safety events and did not have data related to hospital outcomes. We chose to use patient charts over administrative datasets because they are a direct reflection of the clinical care provided. Because providers do not usually write extensive narratives in patient charts, it is likely that record reviews underestimate the incidence of safety events compared with direct observation, which has been demonstrated in studies of medication dosing error.^21,65 Furthermore, we may have identified additional UNSEMs from hospital records (e.g., unrecognized esophageal intubation) had these linkages been available. Both the presence of an UNSEM and the degree of harm caused are based on judgment of the reviewers and is inherently subjective, though this method has been used in other landmark safety studies.²

Our sample was limited to pediatric calls that were transported Code 3. This was a deliberate choice to study a cohort of the sickest children requiring more intervention with the potential for safety events. This sample selection approach, however, does not capture patients who were under-triaged, and it does not evaluate errors in non-critical transports. Finally, we studied a single county EMS system and therefore our findings may not be generalizable to other EMS agencies with different provider compositions, performance standards, or geographic locations, and regional variation in EMS performance has been described.^72,73 The agency in our study includes only paramedics and serves a predominantly urban area; however, compared with the 11 agencies in PECARN, the characteristics of our EMS system are comparable.⁷⁴ The high level of training, short call cycle times, and the availability of pediatric specific protocols and online medical control within our study’s EMS system would tend to render these estimates of patient safety events as conservative, compared to systems that rely on part-time volunteer forces with less training and experience, or those in rural and remote settings with prolonged response and transport times.

Finally, our primary outcome of UNSEM was intentionally broader than simply errors and near misses. This was a deliberate decision to identify as many events as possible with the potential for harm and amenable to intervention in future research. As a result, in addition to 148 errors and 62 near misses, 200 instances of “suboptimal actions” were identified by reviewers; in fact this was the most commonly identified event. It is unclear whether these suboptimal actions represent truly unique categories or whether the expanded taxonomy encouraged reporting by avoiding the stigma associated with judging errors. Nonetheless, the events captured have both the potential for harm and future intervention. Thus, it appears, at least in this early stage of research, that our classification system and analytic approach provides a practical method for research as well as continuous quality improvement in out-of-hospital care.

Conclusions

Among high-risk pediatric ambulance transports, patient safety events are common, potentially severe, and may be largely preventable. Results of our analyses suggest that resuscitation and care of children under one year of age are important areas to focus efforts to reduce out-of-hospital pediatric patient safety events.