Patient Safety Perceptions in Pediatric Out-of-Hospital Emergency Care: Children's Safety Initiative

Abstract

Objective

To characterize emergency medical service (EMS) providers’ perceptions of the factors that contribute to safety events and errors in the out-of-hospital emergency care of children.

Study design

We used a Delphi process to achieve consensus in a national sample of 753 emergency medicine physicians and EMS professionals. Convergence and stability were achieved in 3 rounds and findings were reviewed and interpreted by a national expert panel.

Results

Forty-four (88%) states were represented and 66% of participants were retained through all 3 rounds. From an initial set of 150 potential contributing factors derived from focus groups and literature, participants achieved consensus on the following leading contributors: airway management, heightened anxiety caring for children, lack of pediatric skill proficiency, lack of experience with pediatric equipment, and family members leading to delays or interference with care. Somewhat unexpectedly, medications and communication were low-ranking concerns. After thematic analysis, the overarching domains were ranked by their relative importance: (1) clinical assessment; (2) training; (3) clinical decision-making; (4) equipment; (5) medications; (6) scene characteristics; and (7) EMS cultural norms.

Conclusions

These findings raise considerations for quality improvement and suggest important roles for pediatricians and pediatric emergency physicians in training, medical oversight, and policy development.

Emergency Medical Services (EMS) providers are an integral part of the pediatric care delivery system. Out-of-hospital emergency care relies upon professionals with a range of training to make time-sensitive, critical decisions under conditions of uncertainty while interacting with highly-complex environments and technologies. In these situations, mistakes can occur and their consequences can be severe. Although the contributors to adverse events in hospitals are well described,^1,2 the nature of adverse events and associated contributors in the out-of-hospital environment are largely unknown.^3,4 Furthermore, relatively few, if any, studies take a global look at contributors to safety in out-of-hospital emergency care and even fewer report on the care of children.^4,5 Pediatric care deserves special attention as it poses unique challenges that can increase the occurrence of medical errors, including: the inability of young children to provide a medical history or clearly communicate complaints; age-dependent anatomic and physiologic differences; physical and developmental characteristics; and variations in weight-based medication dosing and size-based equipment needs.^6,7,8,9,10

The Children's Safety Initiative (CSI)-EMS is a large, multi-phase study funded by the National Institutes of Health (NICHD R01HD062478) to describe the epidemiology of patient safety events in the out-of-hospital emergency care of children. The study includes: (1) focus groups; (2) national Delphi study; (3) chart review; and (4) in situ simulation. Each phase of the study builds an increasingly detailed understanding of the nature of safety events and their associated contributors in the out-of-hospital setting. This study presents results from the national Delphi study intended to understand what practicing EMS providers perceive as the major contributors to patient safety events in the emergency care of children.

Methods

We conducted a national Delphi study to gain a consensus among EMS providers around the contributors to patient safety events. The Delphi technique is a multi-stage survey methodology intended to obtain a reliable and objective consensus among independent experts.^11,12 Its design is intended to reduce the biasing effect of dominating individuals and group pressure allowing experts to provide their responses independent from one another.¹³ A response rate of 40-50% is considered the recommended standard for Delphi surveys.¹⁴ Before each new survey round, experts' responses were analyzed and reflected in subsequent rounds. Surveys were administered electronically via SurveyMonkey™ and all rounds were completed between August 2011 and July 2012.

We acknowledge that most states recently reclassified EMS providers, by moving to the national standard for classification of EMS providers as described by the National Registry of Emergency Medical Technicians (NREMT). At the time of this study, the participants indicated their provider level based on their state regulations. The researchers have an understanding of the National Registry system for providers (e.g., know that EMTB is not currently used in EMS provider terminology).

EMS direct care (field) providers (Emergency Medical Technician-Paramedic [EMT-P], EMT-Intermediate [EMT-I], EMT-Basic [EMT-B], and first responders) and administrative leaders, Emergency Department physicians and nurses (general and pediatric specialists), and respiratory therapists participating in out-of-hospital transports were eligible to participate in this study. Participants were recruited through email lists of US EMS and Emergency Medicine specialty societies and professional groups in attempts to obtain an unbiased national sample of providers.

Potential participants were offered the opportunity to be included in a raffle for an electronic tablet. The study was approved by the Oregon Health & Science University Institutional Review Board (IRB00006942) and informed consent was obtained.

Survey Design

Survey questions were developed through focus groups and the literature as previously described.¹⁵ Cognitive interviews were conducted to assess face validity, clarity of content, instructions, and usability of surveys.¹⁶ Briefly a think-aloud interviewing method was used asking EMTs, Emergency Physicians, and EMS program directors to vocalize their comprehension of questions and to think aloud as they retrieved information and analyzed and interpreted which information to use to complete the survey questions. Survey questions included demographic information such as: age, sex, level of training, years of experience, full-or part-time status, paid or volunteer work, geographic location, the number of pediatric patients seen in the previous year, whether they have children, and the ages they consider to be pediatric. We used an established definition of safety events encompassing adverse events, near misses, and errors,¹⁷ followed by structured questions asking participants to rate the likelihood of factors leading to safety events. Response options were formatted as 9-point Likert-type scales with 1 being not at all likely, 5 moderately likely, and 9 highly likely. Space was provided to allow unstructured elaboration, clarification, and case examples.

Analyses

We conducted qualitative and quantitative analyses of group and subgroup responses to identify the factors rated as most and least likely to lead to safety events. The 9-point Likert-type scale was collapsed into three categories (“not at all likely” [1-3], “somewhat likely” [4-6], and “highly likely” [7-9] to lead to patient safety events), and the proportion of respondents selecting a response in each category for each question was calculated. A rank order was determined by the proportion of respondents rating an item as “highly likely” to lead to safety events. Subgroup analyses were conducted to examine the variation among different levels of EMTs, physicians, and nurses. Quantitative analyses were conducted with Microsoft Office 2007 Excel and IBM SPSS Statistics 19 software programs.

Qualitative data were used to understand the degree of consistency among provider types, the clinical context of different risk factors, and to identify other potential contributors to safety events. Qualitative responses were analyzed by trained qualitative analysts using QSR NVivo 9™ to find themes in the narrative data and rank them by frequency. Themes were then reviewed by the research team and considered for inclusion in the next round.

A panel of national experts from major EMS and pediatric organizations was assembled to assist in interpreting results from the organizational perspective, identifying potential solutions, and suggesting needed future research (Table I; available at www.jpeds.com). Two teleconference reviews were conducted with the panel of national experts.

Table 1.

Expert Advisory Panel Participants and Their Organizations

Participant	Participant Organization(s)
Katrina Altenhofen, MPH, EMT-P	Iowa EMSC, National Association of State EMS Officials (NASEMSO)
Melissa Bentley, MS, NREMT-P	National Registry of Emergency Medical Technicians (NREMT)
William Brown Jr., NR, MS, NREMT-P	National Registry of Emergency Medical Technicians (NREMT)
Elizabeth Edgerton, MD, MPH	Human Resources and Services Administration (HRSA), Maternal Child Health Bureau (MCHB)
Dia Gainor, MPA	National Association of State EMS Officials (NASEMSO)
Ritu Sahni, MD, MPH	Providence Health & Services, Oregon Health & Science University (OHSU), National Association of EMS Physician (NAEMSP)
Paul Sirbaugh, DO	Texas Children's Hospital, Baylor College of Medicine, Houston EMS

Results

Figure 1 shows the geographic distribution of survey respondents with 44 (88%) states participating. Of 753 eligible EMS professionals who consented to participate, 722 completed Round 1, 614 (85%) completed Round 2, 492 (68%) completed Round 3; and 477 (66%) participated in all three rounds, and the representation of provider types remained stable (eg, EMT-Ps comprised 50.8% of participants in Round 1, 51.5% in Round 2, and 51.6% in Round 3) across all rounds. As shown in Table II, the majority of respondents (80%) were clinically active EMTs (Paramedic, Intermediate, Basic). Most had been working for a mean of 17 years; 60% were male, approximating the proportion of male EMTs nationally (72%¹⁸); providers were evenly distributed across urban, suburban, and rural settings; and the majority worked for either public or private ambulance agencies. Because focus groups in Phase I of our study suggested that there may be differences of opinion among EMTs in their perceptions at what age a person should still be considered a child, we asked participants ““When you think of EMS care for a child, what age of child do you consider the oldest?” Whereas 69% of physicians reported that the upper ages still considered to be a child were between 16 and 18, only 30-40% of EMTs still considered these ages to be children. Although there was no one age that EMTs agreed was the upper limit, age 12 was the highest single age reported with 20-26% of EMTs reporting this as the upper age of a child.

Figure 1.

Geographic Distribution of Participants

Table 2.

Characteristics of Delphi Participants

	Round 1 N (%)	Round 2 N (%)	Round 3 N (%)
Total	722 (100)	614 (85)	492 (68)
Age, mean (SD)	41.7 (10.3)	41.8 (10.3)	42.3 (10.3)
Sex, male	438 (60.7)	368 (59.9)	301 (61.2)
Has children	558 (77.8)	459 (74.8)	372 (75.6)
Years working in EMS, mean (SD)	16.8 (9.4)	17.0 (9.3)	17.1 (9.2)
Number of years at current level, mean (SD)	14.0 (9.2)	14.0 (9.1)	14.2 (9.1)
Race & Ethnicity*
White	665 (93.1)	566 (92.2)	454 (92.3)
Hispanic	23 (3.8)	17 (2.8)	12 (2.4)
American Indian/Alaska Native	23 (3.8)	22 (3.6)	15 (3.0)
Asian	9 (1.5)	8 (1.3)	7 (1.4)
Native Hawaiian/Pacific Islander	2 (0.3)	2 (0.3)	2 (0.4)
Black	8 (1.1)	6 (1.0)	5 (1.0)
Geographic Location
Urban	244 (33.8)	213 (34.7)	165 (33.5)
Suburban	226 (31.3)	183 (29.8)	145 (29.5)
Rural	250 (34.6)	213 (34.7)	177 (36.0)
Provider Type
EMT-Paramedic	367 (50.8)	316 (51.5)	254 (51.6
EMT-Intermediate	50 (6.9)	39 (6.4)	31 (6.3)
EMT-Basic or First Responder	159 (22.0)	129 (21.0)	99 (20.1)
MD, DO	82 (11.4)	70 (11.4)	64 (13.0)
NP, RN, RT, LPN	64 (8.9)	60 (9.8)	44 (8.9)
Work Setting
Private Ambulance Agency	163 (22.6)	129 (21.0)	98 (19.9)
Public Ambulance Agency	278 (38.6)	237 (38.6)	185 (37.6)
Emergency Room	86 (11.9)	75 (12.2)	65 (13.2)
State Office	19 (2.6)	16 (2.6)	14 (2.8)
Other	174 (24.2)	147 (23.9)	123 (25.0)

^*Participants could select all race/ethnicity categories that apply, and thus, the total percentages exceed 100.

Table III (available at www.jpeds.com) provides the detailed responses by provider type regarding perceived contributors to patient safety events. As shown, six of the top seven issues perceived to lead to patient safety events and errors involve issues of applied clinical knowledge and pediatric skill proficiency. Concerns expressed in qualitative responses included insufficient clinical volume to maintain skills, lack of experience with pediatric patients, not enough hands-on real-time training, need for refresher courses, and inability to recreate the stress and anxiety EMTs face in the field.

Table 3.

Final Delphi Consensus of Factors Most Likely to Cause Safety Events by Provider Type

Delphi Round 3	All (N=492) Ranking	All % Highly Likely	EMT-P (N=254) % Highly Likely	EMT-I (N=31) % Highly Likely	EMT-B, FR (N=99) % Highly Likely	MD (N=64) % Highly Likely	RN (N=44) % Highly Likely
Lack of experience with pediatric airway management	1	73.4	72.1	67.7	64.3	87.5	84.1
Heightened anxiety when working with children	2	72.5	71.7	66.7	66.3	81.3	81.8
Lack of proficiency in pediatric skills among providers on scene	3	66.6	66.5	61.3	58.2	75	77.3
Lack of experience with pediatric equipment	4	57.9	54.8	54.8	54.1	71.9	65.9
Family Member leads to delay or interference	5	49.2	50	41.9	60	37.5	43.2
Lack of ongoing pediatric training (continuing medical education) for EMS providers	6	48.6	43.8	48.4	48	65.6	52.3
Knowing when to alter plans mid-course	7	47.3	44.9	35.5	36.7	59.4	59.1
Lack of debriefing among EMTs after calls	8	45.5	44.8	48.4	33	63.5	48.8
Determining whether patient is sick or not sick	9	45	43.5	25.8	43.9	69.8	50
Knowing when to perform advanced airway procedure (for example, LMA, ETT, King, etc.)	10	44.5	41.7	32.3	35.4	64.1	61.4
Reluctance to report one's own errors	11	43.2	43	35.5	37.2	53.1	47.7
Reluctance to report others' errors	12	40	39.8	32.3	36.8	45.3	45.5
Making the decision to “Scoop and Run”/“Load and Go” or “Stabilize before Transport”	13	38.6	39.4	32.3	33.7	46	38.6
Use of adult equipment on children (for example, immobilizing a child on an adult backboard)	14	34.3	34.4	29	29.6	34.4	47.7
Availability of correct size of equipment (for example, spinal immobilization, cardiac monitoring equipment, O2 monitoring equipment)	15	33.1	27.3	29	38.8	39.1	47.7

Across all three survey rounds, EMTs rated airway issues as the most likely contributor to patient safety events in children ranging from a low of 64% among EMT-Bs to a high of 87% among Emergency Medicine physicians. Common details described in qualitative responses around airway issues included decision-making issues (e.g., determining the severity of illness, best medical approach, and/or need to change management) and proficiency in the performance of advanced airway procedures.

Heightened anxiety when caring for children was the second most common contributor to patient safety events. Respondents reported that their own emotional responses, lack of experience responding to pediatric emergencies, and the emotions of family members negatively impact the provision of care when responding to pediatric emergencies. Even though lack of pediatric skill proficiency and experience with pediatric equipment were rated highly by all provider types, physicians and nurses felt this was a greater problem than EMTs. Physicians were also more likely than EMTs to report that lack of ongoing pediatric training, knowing if the patient is sick or not sick, and lack of debriefing were likely to lead to patient safety events and errors. Looking specifically into diagnostic issues, EMTs reported that children with special needs and chronic conditions were particularly challenging for them to determine if the child was sick or not sick.

After identifying specific factors most likely to contribute to safety events, participants were asked to rank the overarching themes in order of importance at the end of the survey (Table IV). Although there was widespread agreement for the top three and last contributors, providers disagreed on the relative importance of equipment, medications and scene characteristics. EMTPs and nurses ranked medications over equipment and scene characteristics, EMT-Bs ranked equipment over scene characteristics and medications, and physicians ranked equipment issues over medications and scene characteristics.

Table 4.

Rank order of Delphi categories according to their likelihood to lead to a safety event or error in the out-of-hospital care of children, 1=most likely.

Categories	Overall N=483		EMT- P N=250		EMT-I N=31		EMT –B N=94		MD,DO N=64		NP, RN, etc. N=44
	Rank	Mean	Rank	Mean	Rank	Mean	Rank	Mean	Rank	Mean	Rank	Mean
Assessment	1	2.69	1	2.68	1	2.87	1	3	1	2.3	1	2.5
Training	2	3.05	2	3.17	2	2.94	2	3.03	2	2.72	2	2.98
Decision-Making	3	3.29	3	3.25	3	3.35	3	3.53	3	3.17	3	3.11
Equipment	4	4.06	5	4.21	4	4.06	4	3.64	4	4	5	4.23
Medications	5	4.23	4	4.07	5	4.52	6	4.64	5	4.35	4	3.9
Scene Characteristics	6	4.96	6	4.89	6	4.65	5	4.54	6	5.58	6	5.59
EMS Cultural Norms	7	5.71	7	5.74	7	5.61	7	5.62	7	5.84	7	5.68

Mean provided to characterize the spread of scores

After all rounds were completed, we engaged a panel of national experts to assist in interpreting and categorizing results in formats that would be easily understood and likely to affect change. Experts collaboratively collapsed the seven Delphi categories created at the end of the Delphi process into six overarching domains: competency (subdivided into: case exposure, critical thinking and proficiency, and education and knowledge); medication; equipment; scene and environment characteristics; patient safety culture; and communication. Cultural norms was considered to be a cross-cutting concern affecting all six domains. The panel felt that it was particularly important to frame assessment, decision-making, and training as competency-related domains to clearly convey the application of acquired skills. For example, questions about knowing when to perform an advanced airway procedure (e.g., laryngeal mask airway, intubation, supraglottic device, etc.); knowing when to alter plans midcourse, and determining whether a child was sick or not sick were all considered to be in the assessment and decision-making domain. These categories, while reflecting our experts’ professional backgrounds and opinions, are novel.

Discussion

In this large national Delphi study of Emergency Medical Services professionals, we found that the factors EMS professionals perceived to contribute to patient safety events in the out-of-hospital setting are complex and rooted in limited clinical experience with children, anxiety, and limited proficiency in pediatric skills. Though children make up almost a quarter of the US population,²⁰ translating to almost 73 million individuals, they comprise only 13% of EMS transports and fewer than 1% of EMS dispatches requiring advanced life support.^5,21 Limited case exposure is compounded by infrequent training that together create a high potential for patient safety events.^9,22,23 According to hospital-based epidemiological studies, such events have a large effect on national health with errors resulting in significant injury or death in 4% to 17% of hospital admissions.^1,24

Applied knowledge in clinical assessment and decision-making was a particular challenge according to out-of-hospital providers. Accurate diagnosis in the out-of-hospital field has downstream ramifications as it determines hospital destination, critical procedures and medications, and whether lights and sirens are used. Our results emphasize the importance of interventions to reduce diagnostic errors in the out-of-hospital setting.

Airway management is one of the most hotly debated topics in out-of-hospital medicine and was the most likely contributor to out-of-hospital patient safety events in our study. Traditionally, mastery of these skills requires clinical exposure and practice. Although our study does not inform that particular debate, it does emphasize the importance of pediatric airway management as an area of high risk to patient safety and a high priority to EMS providers and experts. Further aims of our study (Phase 3) examine the nature of airway events in everyday clinical care through chart reviews and explore the underlying issues relating to airway events through simulations.

Patient safety studies in hospital-based medicine consistently identify medication errors as a significant contributor to safety problems. Participants ranked medications as 5^th out of 7 overarching domains for patient safety events. This result was unexpected, and discussed with our expert panel who validated that our findings were likely accurate. There are several potential explanations for the relative decrease in perceived importance of medications as potential contributors to pediatric out-of-hospital safety events. First, medication administration is limited by scope-of-practice to EMT-Ps and to a lesser degree, EMT-Is; and, only about 50% of children treated within EMS are cared for by providers with advanced life support training and the ability to administer medications. Second, medications are given in a very small proportion of pediatric EMS transports, fewer than 5% in one study.^25,26 Finally, the number of medications that can be used for pediatric patients is limited and use is often protocol-based which may decrease the perceived potential for medication error. The findings of this study do not negate findings nor diminish the importance of prior studies reporting specifically on medication errors in out-of-hospital medicine, but rather frame them in the context of the lived experiences of EMS professionals.^27,28,29

Our study, like any, has limitations. Although the Delphi method is a robust method for achieving unbiased consensus, the study design measures are perceived rather than observed outcomes. Perceptions of experienced clinicians in the field are critically important to our understanding of patient safety as they are a blend of frequency of events, severity of consequences, and insights into potential mechanisms, causes, and preventability. Subsequent phases of our Children's Safety Initiative are designed to provide objective analyses of clinical practice to complement these perceptions. Another limitation is the representativeness of our sample. Because our recruitment method relied on listservs, social media, and word-of-mouth, we only know the amount of people who initiated the first survey rather than the organizations’ distribution list. This is not considered a limitation for Delphi surveys however as the key is retention of an initial group over sequential rounds. Also, even though our survey sample was geographically diverse, representing 44 of 50 states, we had strong representation from the Northwest, Midwest and Northeast and may under-represent California and the Southeast, which could limit the generalizability of our results to these areas. Finally, some attrition occurred across the three rounds of the survey with 66% of respondents completing all three rounds. Attrition rates among Delphi studies have been reported to range significantly, with larger studies reporting higher attrition rates and some studies losing 40% of their experts.¹⁹ In general as mentioned previously, retention rates of 40-50% are generally considered to be recommended for Delphi studies and our response rate of 66% exceeds that. Additionally, in our study the proportion of participants from each category remained stable, and the number of participants completing all three rounds makes this one of the largest Delphi studies ever conducted in health care.

Our study supports the need for involvement of pediatricians and pediatric emergency medicine physicians to a greater degree in both systems level interventions as well as local training and protocol development.³⁰Further research is needed to define the epidemiology of pediatric outof-hospital safety events and to test the expert consensus opinions identified in this study.

Abbreviations

EMS

Emergency Medical Services

EMT-B

Emergency Medical Technician-Basic

EMT-I

Emergency Medical Technician-Intermediate

EMT-P

Emergency Medical Technician-Paramedic