Use of Electronic Health Record-Based Measures to Assess Quality of Care for Pediatric Agitation

Abstract

Objectives:

Acute agitation during pediatric mental health emergency department (ED) visits presents safety risks to patients and staff. We previously convened multidisciplinary stakeholders who prioritized 20 proposed quality measures for pediatric acute agitation management. Our objective was to assess the feasibility of evaluating performance on these quality measures using electronic health record (EHR) data and to examine performance variation across three EDs.

Methods:

At a children’s hospital and two non-children’s hospitals, we determined the feasibility of evaluating quality measures for pediatric acute agitation management using structured EHR data elements. We retrospectively assessed measure performance during ED visits by children 5–17 years old who presented for a mental health condition, received medication for agitation, or received physical restraints from July 2020 to June 2021. We used bivariate and multivariable models to examine measure performance by patient characteristics and hospital.

Results:

We identified 2785 ED visits for a mental health condition, 275 visits with medication for agitation, and 35 visits with physical restraints. Performance was feasible to measure using EHR data for ten measures. Nine measures varied by patient characteristics, including 4.87 times higher adjusted odds (95% CI 1.28, 18.54) of physical restraint use among children with versus without autism spectrum disorder. Four measures varied by hospital, with physical restraint use varying from 0.5%–3.3% of mental health ED visits across hospitals.

Conclusions:

Quality of care for pediatric acute agitation management was feasible to evaluate using EHR-derived quality measures. Variation in performance across patient characteristics and hospitals highlights opportunities to improve care quality.

Emergency department (ED) visits by children for mental health conditions have increased in the U.S. over the last decade, and acute agitation during these visits presents safety risks to patients and staff.^1,2 Approximately 4% of children seen in the ED for a mental health condition receive intramuscular (IM) medication for acute agitation and 5–10% receive physical restraints.^3,4 However, timely implementation of behavioral strategies and oral medications may reduce the severity of agitation and decrease the need for more restrictive interventions.⁵ Additionally, children who experience acute agitation in the ED are more likely to experience prolonged length of stay, known as boarding, while awaiting inpatient psychiatric care.⁶ Hence, strategies are needed to promote high quality care for children with acute agitation that is safe, timely, and patient-centered.⁷

Quality measurement is the first key step in understanding gaps and variations in care to guide quality improvement initiatives.⁸ While quality measures have been rigorously developed for other pediatric mental health conditions in the ED, including suicidal ideation and psychosis,⁹ quality measures have not been developed for pediatric acute agitation management. To bridge this gap, we previously conducted a modified Delphi consensus process to develop proposed quality measures for pediatric acute agitation management in the ED.¹⁰ An expert panel consisting of multidisciplinary health care team members, parents, and hospital data analysts selected 20 quality measures for pediatric acute agitation care on the basis of their importance and anticipated feasibility.¹¹ A subsequent step in quality measure development is empiric assessment of measure feasibility and performance variation.¹² Feasibility means that data are readily available for measurement and retrievable without undue burden.¹² Demonstration of performance variation highlights opportunities for improvement across population groups or care settings.¹² Since most children in the U.S. receive emergency care at non-children’s hospitals,¹³ it is critical to ensure that new quality measures are feasible to evaluate across hospital types.

Thus, we aimed to assess 20 stakeholder-prioritized quality measures for pediatric ED acute agitation management for: (1) feasibility to obtain from electronic health record (EHR) data and (2) performance variation across patient characteristics and across 3 EDs, which included both children’s and non-children’s hospitals.

METHODS

Study Design and Setting

We retrospectively evaluated performance on quality measures for pediatric ED acute agitation management at one children’s hospital and two non-children’s hospitals in a large metropolitan area. The hospitals belong to different health systems and are each located approximately 30 miles apart.

The children’s hospital has a 12-bed inpatient psychiatric unit serving children ages 3–17 and approximately 56,000 annual ED visits. The non-children’s hospitals are community hospitals with academic affiliations. One has an on-site inpatient psychiatric unit with 8 adolescent beds; the other is affiliated with an off-site inpatient psychiatric facility with 12 adolescent beds. They have 11,000 and 7,000 annual pediatric ED visits, respectively. Each ED has a separate space to care for children that is open 12 and 16 hours per day, respectively. Children are seen by pediatric emergency medicine physicians when the pediatric space is open, and otherwise are seen by general emergency medicine physicians.

The children’s hospital and one non-children’s hospital have an algorithm in place to guide the management of acute agitation in children.¹⁴ The children’s hospital and the same non-children’s hospital require ED nurses to complete formal training in de-escalation techniques. The children’s hospital and the same non-children’s hospital have child life specialists available for 10.5 and 7 hours per day, respectively. All three hospitals use an EHR developed by Epic Systems Corporation (Verona, Wisconsin). Pro re nata (PRN) orders are not placed at any of the participating EDs; all orders are placed as one-time. The study was deemed exempt or as quality improvement by each hospital’s institutional review board.

Specification of Quality Measure Definitions and Determination of Feasibility

Twenty quality measures for pediatric ED acute agitation management were previously ranked by multidisciplinary stakeholders as highly important and potentially feasible.¹⁰ We worked with quality improvement specialists and hospital data analysts to specify each proposed quality measure by defining data sources, numerators, and target population denominators. We considered quality measures as feasible to measure if they could be defined using existing structured data fields in the EHR, rather than unstructured data (e.g., free text) requiring manual review, as this would facilitate use of the measures repeatedly over time for their intended purpose of quality improvement work. Minor changes were made to definitions of the following 5 measures to enhance feasibility: safe room, timely vitals, medication given, restraints, and child life involved. For example, the proposed denominator for the safe room measure, “patients at risk for agitation,” was changed to “mental health ED visits” to enable measurement using existing structured EHR data elements.

The population denominator of “mental health ED visits” was defined as ED visits by children 5–17 years old with a psychiatric chief complaint (“ED PSYCH” selected from an EHR picklist) or a primary ICD-10-CM diagnosis code for a mental health condition, based on the previously validated Child and Adolescent Mental Health Disorders Classification System (CAMHD-CS).^15–17

We developed a definition for the denominator “ED visits with medication given for agitation” by iteratively comparing combinations of structured EHR data elements against documentation of pre-specified keywords in ED notes as the gold standard criterion (Supplemental Figure 1). The keywords included: agitation/agitated, aggression/aggressive, combative, violent, yelling, biting, punching, threatening, or risk of harm to self or others.¹⁸ We selected the combination of structured EHR data elements that maximized sensitivity and specificity (100% and 98.1%, respectively) for medication given for acute agitation (Supplemental Table 1), as follows: (1) administration of diphenhydramine intramuscular (IM), lorazepam IM, olanzapine IM, haloperidol IM, or chlorpromazine IM, or (2) a psychiatric chief complaint and administration of diphenhydramine per os (PO), lorazepam PO, or olanzapine orally dissolving tablet (ODT). The medications included in this definition are recommended in national consensus guidelines by child and adolescent psychiatrists for pediatric acute agitation management and were also listed in the agitation care algorithm used in two of the participating EDs.^5,14,19

We defined the population denominator of “ED visits with physical restraints used” as ED visits with a physician order of physical restraint application for violent or self-destructive behavior.

Assessment of Quality Measure Performance

We assessed overall quality measure performance during ED visits by children 5–17 years old who presented for a mental health condition, received medication for agitation, or received physical restraints from July 2020 to June 2021 at the 3 hospitals.

We assessed performance by the following patient characteristics: sex, age group (5–11, 12–17),^20,21 race and ethnicity (based on EHR documentation and categorized as Hispanic, non-Hispanic Asian, non-Hispanic Black, non-Hispanic White, and other/missing), insurance status (private, public, uninsured/other/missing), and the presence or absence of each of four mental health diagnosis groups (autism spectrum disorder [ASD]; substance-related and addictive disorders; disruptive, impulse control, and conduct disorders; and developmental delay or intellectual disability). The four diagnosis groups were selected based on prior literature demonstrating increased risk for acute agitation among children in medical settings,^3,22–24 and the diagnosis groups were defined using CAMHD-CS, a validated ICD-10-CM classification system for pediatric mental health diagnoses.^15,16 Race and ethnicity were examined as social constructs rather than biologic determinants,²⁵ and were included in the analysis because prior literature has demonstrated disparities in ED mental health care by race and ethnicity.^26,27

We used bivariate and multivariable models to examine measure performance by patient characteristics and by hospital. We assessed differences in measure performance using χ2 test or Fisher’s exact test for categorical measures and analysis of variance for continuous measures. The multivariable analysis included patient characteristics as covariates and used a fixed effect variable for hospitals to account for hospital-level systematic differences in care. Some quality measures had insufficient sample size for fully-adjusted models; in these cases, we performed partially-adjusted models adjusted for age group, sex, and fixed effect for hospital only; or for age group, sex, and variables significant on bivariate analysis only. Analysis was performed using SAS/STAT version 15.1 (SAS Institute, Inc, Cary NC).

RESULTS

Study Population Characteristics

Across the 3 EDs, we identified 2785 ED mental health visits, 275 visits with medication given for agitation, and 35 visits with physical restraint use during the study period. Of mental health ED visits, 81.4% were by adolescents 12–17 years old, 46.4% were by non-Hispanic White children, 31.7% by Hispanic children, and 13.1% by non-Hispanic Black children. Insurance was distributed across private (46.2%) and public (44.5%) payors. One-quarter (25.5%) of children were admitted or transferred to a psychiatric unit and 17.0% were admitted to a medical/surgical unit (Table 1).

Table 1.

Target Population Characteristics for Pediatric Acute Agitation Quality Measures

ED Visit Characteristic	Target Population for Quality Measures
	Mental Health Condition, N (%), N=2785	Medication Given for Agitation, N (%), N=275	Physical Restraints Used, N (%), N=35
Sex
Male	1,073 (38.5)	149 (54.2)	23 (65.7)
Female	1,711 (61.4)	126 (45.8)	12 (34.3)
Missing	1 (0)	0 (0)	0 (0)
Age, years
5–11	518 (18.6)	58 (21.1)	1 (2.9)
12–17	2267 (81.4)	217 (78.9)	34 (97.1)
Race and Ethnicity
Hispanic	882 (31.7)	69 (25.1)	14 (42.9)
Asian, non-Hispanic	114 (4.1)	12 (4.4)	1 (2.9)
Black, non-Hispanic	364 (13.1)	43 (15.6)	3 (8.6)
White, non-Hispanic	1293 (46.4)	137 (49.8)	15 (42.9)
Other/Missing	132 (4.7)	14 (5.1)	1 (2.9)
Insurance Status
Private	1,287 (46.2)	125 (45.5)	11 (31.4)
Public	1,240 (44.5)	117 (42.5)	20 (57.1)
Uninsured/Other/Missing	258 (9.3)	33 (12.0)	4 (11.4)
Disposition
Discharge	1,548 (55.6)	139 (50.5)	15 (42.9)
Admit/transfer to psychiatric unit	711 (25.5)	61 (22.2)	12 (34.3)
Admit to medical/surgical unit	474 (17.0)	71 (25.8)	8 (22.9)
Other/Missing	52 (1.9)	4 (1.5)	0 (0)
Autism Spectrum Disorder
Yes	148 (5.3)	44 (16.0)	4 (11.4)
No	2,637 (94.7)	231 (84.0)	31 (88.6)
Substance-Related and Addictive Disorder
Yes	174 (6.2)	22 (8.0)	9 (25.7)
No	2,611 (93.8)	253 (92.0)	26 (74.3)
Disruptive, Impulse Control, Conduct Disorders
Yes	237 (8.5)	32 (11.6)	5 (14.3)
No	2,548 (91.5)	243 (88.4)	30 (85.7)
Developmental Delay or Intellectual Disability
Yes	88 (3.2)	41 (14.9)	5 (14.3)
No	2,697 (96.8)	234 (85.1)	30 (85.7)
Hospital
Children’s Hospital	1,470 (52.8)	99 (36.0)	8 (22.9)
Non-Children’s Hospital A	670 (24.1)	132 (48.0)	5 (14.3)
Non-Children’s Hospital B	645 (23.2)	99 (36.0)	22 (62.9)

ED: Emergency Department

Quality Measure Feasibility Determination

Of the proposed quality measures, ten were feasible to measure using existing structured data elements in the EHR (Table 2). Supplemental Table 2 describes reasons that the remaining quality measures were not feasible to measure using EHR data.

Table 2.

Definitions of Quality Measures for Pediatric Acute Agitation Management that were Feasible to Measure using Electronic Health Record Data

Target Population Denominator	Brief Quality Measure Description	Numerator	Data Source
Mental health ED visitsa	Safe room	Visits in an ED room designed for safety or documented as having been modified for safety	Room number or nursing flowsheet documentation
	Timely vitals	Visits with temperature, heart rate, respiratory rate, and blood pressure documented within 30 minutes of arrival	Vitals flowsheet
	Medication given	Visits with medication given for agitationb	Medication administration record
	Restraints	Visits with physical restraints used	Physician order indicating restraint application for violent or self-destructive behavior
ED visits with medication given for agitationb	Child life involved	Visits with child life services received	Child life note present
	Patient-specific agitation plan	Visits with a documented plan for prevention or treatment of re-escalation	Personalized care plan flowsheet
	Length of stay	ED length of stay in hours	Arrival and discharge times
	Order to medicationd	Minutes from medication order to administration	Order and medication administration times
	Return visitse	Visits with a return visit to the ED within 7 days	Arrival and discharge times
ED visits with physical restraints usedc	Time restrained	Time in physical restraints per patient in minutes	Nursing flowsheet indicating duration of restraint application for violent or self-destructive behavior

^a.Psychiatric chief complaint selected from an EHR picklist, or primary ICD-10-CM diagnosis code in the Child and Adolescent Mental Health Disorders Classification System

^b.ED visit with medication given for agitation defined by: (1) administration of diphenhydramine IM, lorazepam IM, olanzapine IM, haloperidol IM, or chlorpromazine IM, or (2) a psychiatric chief complaint and administration of diphenhydramine PO, lorazepam PO, or olanzapine ODT

^c.Physical restraints defined by a physician order of physical restraint application for violent or self-destructive behavior

^d.Measure denominator was limited to ED visits with severe agitation defined by: (1) administration of diphenhydramine IM, lorazepam IM, haloperidol IM, chlorpromazine IM, olanzapine IM, or (2) a psychiatric chief complaint and administration of olanzapine ODT

^e.Measure denominator was limited to ED visits with a disposition of discharge

Quality Measure Performance

Overall quality measure performance and performance variation by hospital are presented in Table 3. Of the 2785 ED mental health visits, 6.4% had medication given for acute agitation and 1.2% involved physical restraint use, with a mean time in restraints of 105 minutes. Of 275 ED visits with medication given for acute agitation, involvement of child life specialists and documentation of a patient-specific plan for agitation occurred in 5.1% and 3.3%, respectively. Of 139 ED visits with medication given for acute agitation resulting in discharge, 6.5% had a return ED visit within 7 days. Performance on four quality measures varied significantly by hospital: placement in a safe room, obtaining timely vitals, physical restraint use, and length of stay (P<0.01 for each). In particular, physical restraint use varied from 0.5% to 3.3% of mental health ED visits across hospitals.

Table 3.

Quality Measure Performance Overall and by Hospital

Target Population Denominator	Quality Measure	Measure Numerator, N	Measure Denominator, N	Overall Performance, %	Performance by Hospital, %			Test of Differenceb
					A	B	Ca
Mental health ED visits	Safe room	1,816	2785	65.2	47.0	52.9	78.9	P<0.0001
	Timely vitals	1,638	2785	58.8	83.6	92.6	32.7	P<0.0001
	Medication given	178	2785	6.4	6.4	6.2	6.5	P=0.97
	Restraints	34	2785	1.2	0.7	3.3	0.5	P<0.0001
ED visits with medication for agitation	Child life involved	14	275	5.1	6.8	n/a	5.1	P=0.21
	Patient-specific plan for agitation	9	275	3.3	n/a	n/a	9.1	n/a
	Length of stay, h	n/a	275	6.3 (6.0)c	5.0 (4.0)c	6.6 (10.0)c	8.4 (5.4)c	P=0.007
	Minutes from order to medication administration	n/a	213d	27 (112)e	42 (172)e	13 (12)e	20 (46)e	P=0.15
	Return visits in 7d	9	139f	6.5	6.5	0.0	13.3	P=0.23
ED visits with restraints used	Minutes in restraints	n/a	32g	105 (105)d	171 (201)e	85 (45)e	112 (126)e	P=0.27

ED: Emergency department

^a.Hospital C was the children’s hospital.

^b.χ2 test or Fisher’s exact test for categorical variables and analysis of variance for continuous variables

^c.Median (interquartile range)

^d.Limited to ED visits with medication given for severe agitation

^e.Mean (standard deviation)

^f.Limited to ED visits with a disposition of discharge

^g.Visits with physical restraint use with missing documentation of restraint duration (N=3) were excluded.

Bivariate analyses of quality measure performance by patient characteristics are presented in Supplemental Table 3. In multivariable analyses (Table 4), the adjusted odds of receiving medication for agitation were higher among ED visits by males than females (aOR 1.71, 95% CI 1.20, 2.44); during visits by children who identified as other race and ethnicity (aOR 2.21, 95% CI 1.07, 4.56) compared with non-Hispanic White children; and during visits with versus without the following diagnoses: ASD (aOR 3.88, 95% CI 2.31, 6.54); substance-related and addictive disorders (aOR 2.46, 95% CI 1.35, 4.48); disruptive, impulse control, and conduct disorders (aOR 1.82, 95% CI 1.13, 2.95); and developmental delay or intellectual disability (aOR 9.03, 95% CI 5.02, 16.24). The adjusted odds of restraint use were higher among ED visits by males than females (aOR 2.67, 95% CI 1.26, 5.65), children with than without ASD (aOR 4.87, 95% CI 1.28, 18.54), and children with than without substance-related and addictive disorders (aOR 5.48, 95% CI 2.20, 13.64). The adjusted time from medication order to administration was 13.8 minutes shorter (95% CI −26.9, −0.7) during ED visits by children with ASD than without ASD. The adjusted odds of child life specialist involvement were lower during ED visits by adolescents 12–17 years old than children 5–11 years old (aOR 0.26, 95% 0.08, 0.84). The adjusted odds of documenting a patient-specific plan for agitation were higher for ED visits by children with than without developmental delay or intellectual disability (aOR 10.95, 95% CI 2.30, 52.11). The adjusted ED length of stay was 4.9 hours longer (95% CI 1.2, 8.7) for visits with than without a diagnosis of substance-related and addictive disorders.

Table 4.

Quality Measure Performance by Patient Characteristics, Multivariable Analysis

Patient Characteristica	Quality Measure
	Safe room,c aOR (95% CI)	Timely vitals,c aOR (95% CI)	Medication given,c aOR (95% CI)	Restraints,c aOR (95% CI)	Minutes from order to medication administrationc (95% CI)	Child life involved,d aOR (95% CI)	Patient- specific plan for agitation,e aOR (95% CI)	Length of stay, hc (95% CI)	Return visits in 7d,d aOR (95% CI)	Minutes in restraintsd (95% CI)
Male (Ref: Female)	1.20 (1.00–1.45) h	0.85 (0.69–1.04)	1.71 (1.20–2.44) g	2.67 (1.26–5.65) h	−0.2 (−10.0–9.6)	0.74 (0.23–2.37)	1.32 (0.27–6.53)	−0.2 (−2.2–1.8)	0.40 (0.09–1.78)	48.3 (−25.8–122.4)
Age 12–17 (Ref: 5–11)	1.19 (0.94–1.51)	1.37 (1.07–1.75) h	1.14 (0.74–1.76)	3.30 (0.75–14.60)	5.3 (−5.3–16.0)	0.26 (0.08–0.84) h	1.03 (0.22–4.70)	−0.5 (−2.8–1.8)	0.42 (0.10–1.73)	138.6 (−28.9–306.2)
Hispanica	0.99 (0.79–1.24)	0.89 (0.69–1.14)	0.70 (0.44–1.11)	1.77 (0.77–4.05)	−14.6 (−25.6- −3.6) g	--	--	−0.4 (−2.8–2.1)	--	--
Non-Hispanic Blacka	0.91 (0.67–1.23)	1.22 (0.90–1.66)	1.43 (0.86–2.37)	1.24 (0.36–4.23)	0 (−14.3–14.3)	--	--	0.6 (−2.4–3.5)	--	--
Non-Hispanic Asiana	0.90 (0.58–1.40)	1.62 (0.96–2.73)	0.97 (0.42–2.28)	1.24 (0.23–6.78)	−11.2 (−32.9–10.5)	--	--	0.8 (−3.9–5.5)	--	--
Other Race/Ethnicitya	1.30 (0.77–2.19)	0.85 (0.53–1.38)	2.21 (1.07–4.56) h	1.20 (0.08–17.25)	−12.1 (−34.3–10.1)	--	--	−1.3 (−5.6–2.9)	--	--
Public Insurance (Ref: Private)	0.89 (0.73–1.08)	1.31 (1.05–1.63) h	0.71 (0.48–1.04)	1.06 (0.47–2.37)	4.1 (−5.6–13.9)	--	--	−0.2 (−2.4–1.9)	--	--
Uninsured/Other (Ref: Private)	0.96 (0.33–2.78)	0.98 (0.34–2.80)	0.40 (0.02–7.21)	2.29 (0.11–48.85)	--	--	--	--	--	--
Autism Spectrum Disorderb	2.03 (1.13–3.65) h	1.15 (0.75–1.76)	3.88 (2.31–6.54) f	4.87 (1.28–18.54) h	−13.8 (−26.9- −0.7) h	--	2.87 (0.58–14.21)	2.1 (−0.9–5.1)	--	--
Substance-Related and Addictive Disorderb	0.57 (0.39–0.84) g	1.50 (0.96–2.33)	2.46 (1.35–4.48) g	5.48 (2.20–13.64) f	−6.1 (−24.7–12.4)	--	--	4.9 (1.2–8.7) g	--	--
Disruptive, Impulse Control, and Conduct Disordersb	2.52 (1.69–3.78) f	1.13 (0.80–1.61)	1.82 (1.13–2.95) h	1.43 (0.48–4.27)	−0.9 (−14.7–13.0)	--	--	2.0 (−0.9–4.9)	--	--
Developmental Delay or Intellectual Disabilityb	4.20 (1.47–11.98) g	0.86 (0.50–1.47)	9.03 (5.02–16.24) f	3.42 (0.75–15.49)	4.1 (−9.2–17.4)	--	10.95 (2.30–52.11) g	1.5 (−1.7–4.7)	--	--

The highest level of adjustment possible was performed for each quality measure, based the available sample size. Missing values were excluded from analysis.

^aReference group: Non-Hispanic White

^bReference group: Absence of diagnosis

^cAdjusted for all variables listed, with a fixed effect variable for hospitals to account for hospital-level systematic differences in care.

^dAdjusted for age sex, and fixed effect for hospital.

^eAdjusted for age, sex, and variables significant on bivariate analysis (autism spectrum disorder, and developmental delay or intellectual disability).

^fP<0.001

^gP<0.01

^hP<0.05

DISCUSSION

We determined that assessment of care quality for pediatric acute agitation management in the ED was feasible using EHR-derived quality measures at a children’s hospital and two non-children’s hospitals. Of twenty quality measures previously prioritized as important by multidisciplinary stakeholders, ten visit-level quality measures were feasible to assess using existing structured EHR data elements. We identified variation in care by patient characteristics and by hospital across several measures, suggesting that opportunities may be available to improve care for this population.

Children with ASD, developmental delay, and intellectual disability are at particularly high risk for developing acute agitation during mental health ED visits. In particular, we found that ED visits by children with developmental delay or intellectual disability had 9 times higher adjusted odds of medication use for acute agitation, while ED visits by children with ASD had nearly 5 times higher adjusted odds of physical restraint use. Our findings are consistent with data from inpatient settings indicating that nearly 1 in 5 children with ASD experience acute agitation during hospitalization, and they have nearly twice the odds of receiving physical restraints in this setting compared with children without ASD.^23,28 We found that patient-specific agitation plans were documented more often for children with developmental delay or intellectual disabilities, suggesting that such plans may be particularly beneficial in this population. Further efforts are needed to design streamlined patient-specific agitation treatment plans that contain information considered valuable and relevant to clinicians and families.^29,30 Future work should also evaluate whether increased use of patient-specific agitation plans can prevent or reduce the severity of acute agitation episodes.

Using the quality measures, we evaluated whether care differed for pediatric acute agitation management by race and ethnicity. Prior studies on pediatric physical restraint use in the ED have been limited to single health systems, with one study finding higher rates of restraint use during ED visits for Black than White youth.^4,27,31 In contrast, in the 3 EDs we studied, we did not identify disparities in physical restraint use by race and ethnicity. However, we did find higher rates of medication administration for acute agitation during ED visits by children identifying as other races and ethnicities compared to non-Hispanic White children, a difference that is difficult to interpret given the group’s heterogeneity. Additional large-scale studies are needed to determine whether there are widespread racial and ethnic disparities in pediatric acute agitation care. If so, work is needed to understand the underlying mechanisms and to develop interventions to address these disparities.

To assess quality measure performance, our work relied on existing EHR data, which enhanced feasibility of measurement but lacked comprehensiveness. For instance, verbal de-escalation techniques were only documented in the free text of notes, and agitation severity was not documented objectively using a numeric agitation severity scale.^32,33 Also, there was no EHR marker to indicate which children were at increased risk of becoming agitated. Future steps could include predictive modeling with EHR data or the development of agitation risk screening questions,³⁸ so as to enable identification of children at elevated risk for agitation and facilitate targeted quality improvement efforts. Importantly, no patient-reported outcome measures were documented in the EHR. To achieve care that is patient-centered, the development and routine use of patient-reported outcome measures for this population will be critical.^34,35 Our process relied on hospital data analysts to extract data from the EHR. In the future, automated reports or EHR dashboards could be designed to extract and display quality measures instantaneously.

A strength of our study was the inclusion of two non-children’s hospitals, because most U.S. children receive their emergency care at non-children’s hospitals.¹³ Prior work from the National Pediatric Readiness Project suggests that few EDs have dedicated policies for the care of children with mental health conditions.³⁶ In our study, we identified that one of the non-children’s hospitals had 4-fold lower odds of physical restraint use than the other; this hospital had an algorithm to guide pediatric acute agitation management¹⁴ and required nurses to complete formal training in de-escalation techniques. While this study focused on visit-level quality measures, future work should also examine structural, site-level factors that influence care quality for children with acute agitation.

Next steps will include validation of the quality measure specifications at hospitals with different EHR systems and where clinicians use different medications to manage acute agitation, including medications recommended in guidelines that are not used locally (e.g., clonidine, chlorpromazine, risperidone, quetiapine).^5,19 Quality measurement across additional sites may enable the establishment of achievable benchmarks of care.³⁷ Studies are also needed to determine whether improvements on these quality measures for pediatric acute agitation management result in meaningful changes to health outcomes. In local quality improvement initiatives, these quality measures can be tracked over time along with changes to outcome measures as specific interventions are undertaken, such as the implementation of agitation algorithms, multidisciplinary simulation to practice de-escalation techniques, and implementation of behavioral rapid response teams.^39–42 Using these measures to track progress, quality improvement collaboratives can facilitate simultaneous learning across institutions to enhance the quality of care provided for children who experience acute agitation in the ED.

LIMITATIONS

With quality measures derived from EHR data, there is potential for misclassification. For instance, our definition of visits with medication given for agitation may have included some visits with medications given for another indication or as routine home medications; conversely, the definition may have missed some patients who received medication for agitation. Nevertheless, we chose a definition that demonstrated good sensitivity and specificity when compared against clinician notes. We conducted this study at a limited number of hospitals located in the same metropolitan area. Subsequent testing across more hospitals is needed to determine whether measurement is feasible across different EHR systems, to externally validate measure specifications, and to understand the full breadth of variation in care practices in order to establish achievable benchmarks for care. This study did not assess whether quality measure performance was associated with improved health outcomes, which remains an important future direction. For some quality measures (e.g., minutes spent in physical restraints), our analysis was limited by small numbers of eligible patients, which may limit their use as quality measures unless more patients are determined to be eligible when measured across a larger sample of hospitals.^43,44

CONCLUSIONS

We determined that it was feasible to assess the quality of care provided for pediatric acute agitation management using ten stakeholder-prioritized, EHR-derived quality measures. Performance on these quality measures varied across patient characteristics and hospitals, including children’s and non-children’s hospital settings. This variation suggests that opportunities are available to improve the quality of care provided for a vulnerable and high-priority population.

Abbreviations:

ASD

Autism spectrum disorder

CAMHD-CS

Child and Adolescent Mental Health Disorders Classification System

ED

Emergency department

EHR

Electronic health record

IM

Intramuscular

ODT

Orally disintegrating tablet

PO

Per os

PRN

Pro re nata