Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2023 Jan 21.
Published in final edited form as: J Healthc Qual. 2022 Jan 21;44(4):218–229. doi: 10.1097/JHQ.0000000000000339

Development of Quality Measures for Pediatric Agitation Management in the Emergency Department

Jennifer A Hoffmann 1, Julie K Johnson 2, Alba Pergjika 3, Elizabeth R Alpern 1, Jacqueline B Corboy 1
PMCID: PMC9246821  NIHMSID: NIHMS1767904  PMID: 35067536

Abstract

Objective:

Pediatric mental health emergency department (ED) visits that involve restraints for agitation are increasing. Quality measures are used to assess and improve healthcare quality. Our objective was to develop quality measures for pediatric ED agitation management informed by multidisciplinary perspectives.

Methods:

A multidisciplinary panel developed quality measures for pediatric ED agitation management through the modified Delphi method. Panelists ranked measures in importance and feasibility on a 9-point scale during two survey rounds, with a teleconference discussion between surveys. Consensus was defined by >75% of panelists ranking a quality measure highly (≥7) in importance and a median feasibility score ≥4.

Results:

Panelists included 36 physicians, nurses, social workers, security, child life specialists, hospital data analysts, and parents. The panel reached consensus on 20 quality measures. Measures with the highest percentage of scores with importance ≥7 were related to adverse medication events, patients restrained, staff/patient injuries, re-escalation plans, presence of an algorithm to standardize care, formal staff training on de-escalation techniques, time to medication administration, and room safety.

Conclusions:

Twenty quality measures that incorporate multidisciplinary perspectives were developed for pediatric ED agitation management. Once operationalized and field-tested, these measures may be used to assess and improve healthcare quality for pediatric agitation.

Keywords: Agitation, mental health, quality measure, emergency medicine, pediatrics

INTRODUCTION

Nearly 150,000 U.S. children visit the emergency department (ED) for mental health concerns annually.1 Of children seen in the ED for a mental health condition, 3–4% require pharmacologic restraint and 5–10% require physical restraint to manage acute agitation.24 Acute agitation in the ED can be dangerous and distressing, but timely implementation of behavioral strategies and medications may improve agitation and reduce the need for restraint.5 However, restraint use in children in the ED has been increasing over time. Mental health ED visits by patients 3–21 years old involving pharmacologic restraint increased by 370% from 2009 to 2019, corresponding to rises in overall mental health ED visit numbers.2,6

All children experiencing acute agitation in the ED deserve high-quality, safe care. To meet this goal, quality measures (also known as quality indicators or performance measures) may be used as a tool to improve care quality in terms of structure, process, and outcome.7 Tracking quality measures promotes alignment of care with established quality domains of safety, timeliness, effectiveness, efficiency, equity, and patient-centeredness.8 In 2007, the Institute of Medicine (IOM) recommended developing national standards to measure emergency care performance.9 However, among 405 quality measures for pediatric emergency care identified in 2010, only two quality measures had been developed for mental health conditions.10 Since then, quality measures have been rigorously developed for ED management of suicidal ideation and psychosis among youth, but no such measures have been developed for acute agitation.11 Meaningful development of quality measures must consider the varied perspectives of parents and the multidisciplinary care team members involved in the management of pediatric agitation.7 The modified Delphi technique is a structured consensus process that is widely used to integrate individual perspectives to develop quality measures in healthcare.12 The purpose of this study was to develop quality measures for pediatric acute agitation management in the ED through a modified Delphi process that incorporated the perspectives of multidisciplinary ED care team members and parents.

METHODS

Study Design

We conducted a targeted literature review to inform the drafting of quality measures for managing acute agitation in children in the ED. To prioritize quality measures, we used the modified Delphi method, which is an iterative process consisting of two surveys and a group discussion to systematically integrate the expertise of individual group members.12,13 The method is an established, structured approach to quality measure evaluation with demonstrated reliability and content, construct, and predictive validity.1416 We followed best-practice guidelines for the use and reporting of Delphi techniques for healthcare quality measure selection developed by Boulkedid et al., which emerged from a systematic review of 80 Delphi studies.12

We used the National Quality Forum framework for endorsement of quality measures to guide our study approach. The National Quality Forum, which reviews quality measures for Medicare and Medicaid use, endorses quality measures that are high priority (important to measure), evidence-based, valid, and have a demonstrated performance gap.17 If a measure is not judged to meet requirements for importance, it will not be evaluated against the remaining criteria. On this basis, we focused primarily on the importance of proposed quality measures. We assessed feasibility of proposed quality measures as an additional outcome.

Targeted Literature Review

We conducted a targeted literature review in November 2020 to inform quality measure development. We examined English-language studies related to managing acute agitation or aggression and quality of care for children and adults in the ED or inpatient hospital setting in PubMed, CINAHL, and Ovid databases. We also searched for existing quality measures and clinical care guidelines on websites of professional societies and health care quality organizations (Supplemental Table 1).10

We drafted a set of potential quality measures informed by this literature review, with refinement based on the clinical and quality expertise of the study team members. We reviewed the identified publications and websites for existing quality measures related to the management of acute agitation in children or adults in the ED or inpatient setting. We also considered outcome measures reported in research studies and recommendations offered in clinical practice guidelines and adapted these into quality measures when possible. Additionally, we developed a process map describing the steps undertaken to manage acute agitation in the ED, and we considered whether performance on each step could be considered as a quality measure.

Selection of Panel Participants

We recruited a 36-member panel to review, score, and discuss the draft quality measures. Selection of panelists purposively ensured a diversity of views across relevant ED care team roles. We decided a priori to recruit 6 emergency medicine physicians (3 pediatric trained and 3 general emergency medicine trained), 3 general pediatricians who work in the ED, 3 pediatric emergency medicine fellows, 3 child/adolescent psychiatrists or psychologists, 3 ED advanced practice nurses, 3 ED nurses, 3 psychiatric social workers, 3 security staff or sitters, 3 child life specialists, 3 hospital data analysts with expertise in quality improvement methodology, and 3 parents. We recruited parents of children with behavioral or psychiatric conditions or other conditions that could predispose them to acute agitation (e.g. substance use, autism spectrum disorder, or developmental delay). In determining our panel size, we considered the median panel size of 17 (interquartile range 11–31) reported in a systematic review of Delphi studies for healthcare quality measure selection.12 We opted for a slightly larger panel size, as we included more panelist types than most prior studies, so as to integrate multiple views for each panelist type.12

Panelist recruitment occurred via email using the authors’ hospital listservs, including a parent listserv maintained by the hospital family advisory board. Panelists who agreed to participate were asked to propose additional panelists with relevant expertise. If more than three individuals of a given panelist type were willing to participate, participants were systematically chosen based on years of experience, self-reported relevant expertise in the care of children with agitation, and contribution to the diversity of the panel in terms of gender, race, and ethnicity.18

Panel Scoring: First Round

Panelists were sent an explanation of the Delphi process and the draft quality measures using REDCap, a secure, web-based electronic data capture tool.19,20 Each panelist electronically scored the draft quality measures on importance and feasibility on a scale from 1 (low) to 9 (high).11 Panelists were instructed to apply the following criteria for a measure to be considered important: the extent to which the measure is important to making significant gains in health care quality (safety, timeliness, effectiveness, efficiency, equity, patient-centeredness) and improving health outcomes for children experiencing agitation in the ED.17 Panelists were instructed to apply the following criteria for a measure to be considered feasible: the extent to which the required data are readily available (through medical record data or other existing sources such as patient experience surveys), retrievable without undue burden, and can be implemented for performance measurement.21 We asked panelists to submit comments to explain their rationale, reformulate existing measures, and propose new quality measures.

At the conclusion of the first round, a measure was considered controversial if the median importance score was 4 to 6, if the median feasibility score was ≤3, or if the panel scores indicated a disagreement between panelists as defined by ≤75% of scores falling inside the 3-point domain (1–3, 4–6, or 7–9) that contained the observed median for importance.12 Measures were considered to have reached consensus if >75% of panelists ranked the measure highly (≥7) in importance and the median feasibility score was neutral or high (≥4).12,22 These definitions were adapted from prior work on healthcare quality measure selection.12,22

Group Discussion

Two weeks after the first round, we invited panelists to participate in a 90-minute teleconference discussion on one of two dates to discuss quality measures that were deemed controversial by the above definition. During the teleconference, panelists also discussed measures that reached consensus in the first round but had a median feasibility score in the neutral (4–6) range, to determine whether modifications might enhance feasibility. Panelists were again invited to propose new quality measures, with particular attention to any IOM quality domains that remained underrepresented in the measure set.

Panel Scoring: Second Round

One week after the teleconferences, panelists independently re-ranked the controversial quality measures (modified to incorporate panelist feedback) on importance and feasibility. To aid ranking, panelists received measure score cards consisting of histograms displaying the score distribution from the first round, the measure’s median score, an indication of the panelist’s previous score, and de-identified comments selected by the study team to represent a range of panelist perspectives on the measure’s importance and feasibility (Figure 1). During the second round, panelists also ranked newly proposed quality measures in terms of importance and feasibility. The same score criteria used in the first round were applied to determine which measures reached consensus.

Figure 1. Sample Score Card for Delphi Round Two.

Figure 1.

Delphi panelists were provided scorecards with information from the first survey and teleconference to re-rank controversial quality measures. Score cards consisted of a de-identified summary of representative comments submitted by other panelists, histograms displaying the score distribution for each measure from the first round, the median score for each measure, and an indication of the panelist’s score for each measure in the first round. ED: emergency department; RN: registered nurse.

Measure Classification and Analysis

To assess how the proposed quality measures aligned with various dimensions of healthcare quality and to identify any remaining gaps in measurement, the study team classified each measure by Donabedian classification (structure, process, or outcome measure) and IOM quality domain(s) as effective, timely, efficient, safe, equitable, or patient-centered.10 All categorizations were made by one investigator (JAH) and reviewed for agreement by three investigators (JKJ, ERA, JBC), with disagreements resolved by consensus discussion. Heat maps were used to display differences in quality measure ranking by panelist type. This study was deemed exempt by our hospital’s institutional review board (#2021-4202) with a waiver of informed consent. Summary statistics for quality measure scoring were computed using Stata version 16.1 (StataCorp, College Station, TX).

RESULTS

The modified Delphi panel consisted of 36 physicians, nurses, social workers, security staff and sitters, child life specialists, hospital data analysts, and parents. The panel included 29 (81%) females, 6 (17%) males, and 1 (3%) non-binary individual. Racial/ethnic composition was self-reported as 25 (69%) non-Hispanic White, 3 (8%) Hispanic, 4 (11%) Black, 3 (8%) Asian, and 1 (3%) panelist of other race/ethnicity. The demographics of the 3 parents who participated were similar to those of children seen in our ED for mental health conditions. Of 30 panelists who reported a primary practice location, 26 (87%) worked at an academic children’s hospital, 2 (7%) at a general academic hospital, and 2 (7%) at a community hospital; 28 (93%) reported working in one metropolitan area. Panelists reported a median of 7.5 years of experience working in the ED or caring for children with agitation.

The Delphi process results are summarized in Figure 2. Thirty proposed quality measures were drafted for pediatric ED agitation care, informed by the literature review. In the first survey round, 12 of the 30 proposed quality measures achieved consensus, 18 measures met our definition of being controversial, and two new measures were proposed. Twenty-six (76%) of panelists participated in a teleconference. Due to time constraints, given the large number of controversial measures, the teleconference discussions focused on eight controversial measures with rankings falling just below the threshold for consensus (with 67–75% of panelists ranking the measure as high in importance). Five of these controversial quality measures were modified based on the discussion. Panelists also discussed three measures that achieved consensus in the first round with median feasibility rankings in the neutral (4–6) range, but no changes were suggested to enhance the feasibility of these measures. Through discussion at the teleconferences, four new quality measures were proposed.

Figure 2. Modified Delphi Process to Develop Quality Measures for Pediatric Agitation Management.

Figure 2.

The modified Delphi process for the development of quality measures consisted of a literature review to inform an initial set of quality measures, panelist ranking of quality measures on their importance and feasibility, a teleconference to allow panelists to discuss controversial quality measures and propose additional measures, and a second round in which panelists re-ranked controversial measures and ranked newly proposed measures. The process resulted in 20 quality measures that reached consensus.

Regardless of teleconference participation, all panelists had the opportunity to view summary statistics and representative comments from the first survey and teleconferences via an electronic score card before the second survey round. All 36 (100%) panelists participated in the second survey. Panelists re-ranked the 18 controversial measures (with suggested modifications the first round and teleconference incorporated) and ranked the 6 new measures proposed during the first round and teleconference. Consensus was achieved for 8 additional quality measures. At the conclusion of the modified Delphi process, the panel reached consensus on 20 quality measures for pediatric ED agitation management.

Table 1 lists quality measures that achieved consensus, along with their Donabedian classification, IOM quality domain, and example comments provided by panelists. Quality measures ranked by the greatest percentage of panelists as being highly important (with scores ≥7) were related to: adverse medication events, the number of patients restrained, staff/patient injuries, plans for re-escalation, presence of an algorithm to standardize care, formal staff training on verbal de-escalation techniques, the time between medication order and administration, and room safety. Measures that reached consensus were categorized as 13 (65%) process measures, 5 (25%) outcome measures, and 2 (10%) structure measures. Twenty-nine individual IOM designations were assigned to the 20 measures that reached consensus: 7 (35%) measures addressed effectiveness, 7 (35%) safety, 6 (30%) timeliness, 5 (25%) patient-centeredness, 4 (20%) efficiency, and none directly addressed equity. Proposed measures that did not achieve consensus are listed in Supplemental Table 2.

Table 1.

Quality Measures for Pediatric Agitation Management that Achieved Consensus in the Modified Delphi Process

Brief Measure Descriptiona Full Measure Description Donabedian Classification IOMb Quality Domain(s) Example Panelist Comments
Triage screening Number of patients with a mental health chief complaint screened upon arrival for risk for agitation with discussion of known effective mitigation strategies Process Efficient, Patient-centered “Would need to be scripted so all triage nurses are asking in the same manner. For example: Does your child have a condition that requires additional behavioral support, such as sensory sensitivities, autism spectrum disorder, or a history of agitation or aggression? How can we best support your child while they are here?” –ED Advanced Practice Nurse
Safe room Percent of patients at risk for agitation placed in a specialized ED room designed for safety Process Safe “Very important to have rooms designed for safety, calmness, soothing environment, pleasant distraction, etc.” –Emergency Medicine Attending
Timely vitals Number of agitated patients who have a complete set of vital signs documented within 30 minutes of ED arrival Process Safe Timely “Agitation may preclude blood pressure cuffs, pulse oximetry probes, but this is essential to determining organic causes of agitation, not immediate assumption of psychiatric diagnosis.” –Emergency Medicine Attending
Behavioral rapid response The ED has a structured multidisciplinary “behavioral health rapid response” to coordinate the response of a specialist trained in de-escalation techniques, security, pharmacy, and others as indicated Structure Efficient Timely “Possibly assign a ‘psychiatric nurse’ for the shift, perhaps someone who has some additional training and feels most comfortable.” –ED Nurse
Algorithm in place The ED has a standardized protocol or algorithm available to guide the approach to agitation management Structure Effective Efficient “The algorithmic use of meds, escalating medication administration, when to obtain an ECG, correct consultant to contact -- all of this on an algorithm can be very helpful.” –Emergency Medicine Attending
Verbal de-escalation Number of verbal de-escalation strategies (as measured by nursing documentation) prior to administration of medication for agitation Process Effective “Important to identify interventions prior to medication administration.” –Parent
Formal staff training Proportion of ED staff (including providers, nurses, security, sitters, and child life) who have completed a formal day-long training program in verbal de-escalation techniques and crisis intervention. Process Effective “Mandatory training is needed for all ED staff, not just nurses, and everyone should get the same type of training so that there is a shared approach to management” –ED Nurse
Didactics and simulation Percentage of ED staff (including providers, nurses, security, sitters, and child life) who have participated in a brief didactic training followed by simulation training on verbal de-escalation techniques and crisis intervention. Process Effective “Staff who are trained can serve as a trainer for other staff members.” –ED Advanced Practice Nurse
Medication received Percentage of patients with a mental health chief complaint who receive a medication for agitation Process Effective No comments provided
Time from agitation to medication Time from onset of agitation to time first medication is given for agitation Process Timely “I think timing the onset of the agitation would not be feasible given it would be difficult to know the exact time.” –ED Psychiatric Social Worker
Order to medication Time from physician order of medication to nursing administration of medication for agitation Process Timely “The time of onset of agitation is difficult to capture, so the time of the physician order may be the best proxy measure available for the time that moderate/severe agitation began.” –Pediatric Emergency Medicine Fellow
Patients restrained Number of ED patients who receive physical restraints (manual holds and mechanical devices) per 1000 hours of ED patient care Outcome Safe Patient-centered “Reflects the effectiveness of other components of care prior to needing restraint (e.g. verbal de-escalation, timely medications).” –Child/Adolescent Psychiatrist
Median time restrained Median time in physical restraints per patient in minutes Outcome Safe Patient-centered “Has value because we always strive to get patients out of restraints as promptly as possible, by using other measures to address agitation.” –ED Advanced Practice Nurse
Adverse events Frequency of serious adverse events related to administration of medication for agitation (e.g. low blood pressure, low oxygen level, ECG changes that may predispose to a potentially life-threatening abnormal heartbeat) Outcome Safe “Provides insight to both safety and efficacy of pharmaceutical intervention.” –Emergency Medicine Attending
Injuries Patient/family and staff injuries per month related to episodes of patient agitation Outcome Safe “Will depend on safety reports, may be a measure that is significantly underreported.” –Emergency Medicine Attending
Child life involvement Percent of patients who receive a medication for agitation who receive child life services, during times when the ED is staffed by a child life specialist. Process Effective, Patient-centered “Not all emergency departments have access to child life services, nor will all EDs have funding/resources for child life.” –Emergency Medicine Attending
Plan for re-escalation Percentage of patients who receive medications or restraints who have a documented plan for prevention or treatment of re-escalation Process Effective Timely “I do feel we would make good use of this information if we had it, and it would be beneficial to see if that makes a difference in recurrent agitation episodes.” –ED Advanced Practice Nurse
Courtesy/respect Patient/Family Experience survey response: “Did the care providers (physicians and advanced practice providers) treat you with courtesy and respect?” Outcome Patient-centered “It’s not only what you know in the books it’s learning how to adapt as well in the hospital with everyday experiences.” –Security Staff or Sitter
Length of stay Length of stay in the ED for patients with agitation Process Timely “The duration of stay may be an inciting factor in patient agitation.” –Emergency Medicine Attending
Return visits For patients with agitation who are discharged from the ED, return visits to the ED within 7 days Process Efficient, Safe “Must dive into charts to determine if family’s needs not met with outpatient plan, or failure of family to seek outpatient care/noncompliance, or failure of med/psych team to assess acuity accurately.” –Emergency Medicine Attending
a.

Quality measures are listed in chronological order from patient presentation in the emergency department to disposition

b.

IOM: Institute of Medicine

Heat maps in Figure 3 display differences in quality measure ranking by panelist type. Data analysts tended to provide lower median importance and feasibility rankings, while parents generally offered higher median feasibility rankings than other panelist types. Nurses prioritized removing unsafe belongings, establishing an appropriate level of observation, and workplace safety ratings relative to other panelist types. Parents prioritized family rankings of trust in providers, medication route (intramuscular versus oral), and time from medication administration to sedation more highly than other panelist types.

Figure 3. Heat Maps of Importance and Feasibility Rankings for Quality Measures by Panelist Type.

Figure 3.

Quality measures are displayed in order of the percentage of panelists who ranked the measure as high in importance and are grouped by whether or not the quality measure achieved the study definition of consensus among panelists. Median importance, median feasibility, the percentage of panelists who ranked the measure as high in importance, and the percentage of panelists who ranked the measure as high in feasibility are displayed by panelist type and shaded with the lowest rankings in orange, middle rankings in yellow, and highest rankings in green. The full measure text corresponding to each brief measure description can be found in Table 1 and Supplemental Table 2.

LIMITATIONS

Not all panelists participated in the teleconference, although all panelists had the opportunity to review summary statistics and example comments before re-ranking quality measures. Our process focused on selecting measures with high levels of agreement on importance across panelist groups. However, measures of extremely high importance to one specific group may still be worthy of implementation in some cases. Most panelists worked in metropolitan areas at academic children’s hospitals, reflecting their pediatric expertise, but perspectives of healthcare personnel working in rural areas or at non-children’s hospitals may differ. The proposed quality measures will need to undergo operationalization, field-testing, and assessment of measure properties. Future studies will also need to assess whether improvement on the quality measures leads to improvements in meaningful health outcomes.

DISCUSSION

Through a modified Delphi process, 20 quality measures were developed to assess the quality of care for acute agitation management in children in the ED. Identified measures had high importance and sufficient feasibility based on the perspectives of multidisciplinary ED care team members and parents. The chosen quality measures span a variety of Donabedian classifications and IOM domains.

Previously proposed quality measures for acute agitation in the ED focused on adults, while prior consensus guidelines for agitation management in children concentrated on medication choice and only included the perspectives of child and adolescent psychiatrists.5,23 Our work identifies quality measures that address the full spectrum of ED care for agitation, while incorporating multidisciplinary perspectives. Participants in the measure selection process included physicians with varied training backgrounds, nurses, social workers, security staff and sitters, child life specialists, hospital data analysis, and parents. This process resulted in 25% of measures reflecting the IOM quality domain of patient-centeredness. In contrast, among 405 quality measures for pediatric emergency care identified in 2010, only 7% were patient-centered.10 Inclusion of parents in the measure development process may increase the number of patient-centered measures developed.

As our primary selection criterion, we focused on importance rather than scientific validity, as there is limited existing evidence to guide pediatric ED agitation management. For example, involvement of child life and care plans for re-escalation might reduce episodes of agitation. Still, to our knowledge, these strategies have not been specifically studied in this population. Also, some institutions currently have an algorithm in place to standardize agitation care, including verbal de-escalation strategies and choice of medication.24 However, the impact of such algorithms on outcomes has yet to be established. In contrast, there is limited evidence to support having a behavioral rapid response to ensure all relevant staff members arrive at the bedside of a patient with acute agitation in a coordinated and timely fashion. In one review, behavioral rapid response teams were found to reduce security calls, restraint use, and staff injuries while moderately improving staff knowledge and self-efficacy.25 Additionally, multidisciplinary simulation training at a pediatric hospital has been shown to improve staff confidence in managing aggression, which was sustained 3–6 months after training.26

The next steps for the quality measures should include operationalization and field-testing. In our Delphi process, we deliberately chose a more liberal scoring criterion for feasibility than for importance, because the process of operationalization and field-testing will confirm if measures scored with neutral feasibility are truly feasible to implement. During operationalization, measures will need to be precisely specified by defining cohorts (numerators and denominators) for the population under study. We intentionally did not present this level of detail during the Delphi ranking process to promote accessibility for panelists without prior healthcare quality experience, such as parents. However, we included hospital data analysts on the panel to promote consideration of feasibility for data collection throughout the measure development process. During operationalization, the selection of an appropriate population denominator may differ by measure. An ideal denominator for some measures may be patients at risk for agitation, such as the measure regarding placement in rooms designed for safety. Further work is needed to determine which children are at the highest risk for developing agitation in the ED. Related to this goal, our Delphi panel proposed a quality measure focused on triage screening for agitation risk upon arrival to the ED. An accurate, reliable screening questionnaire has previously been developed to assess agitation risk during an upcoming hospitalization, but a validated tool to predict agitation risk in the ED setting is needed.27

Future directions for this work should also include assessing measures properties in terms of validity, reliability, and performance variation.17 Validity includes the specification of measures to capture the intended target population and ensuring that measure scoring correctly reflects the quality of care provided. Reliability means that the measure can be implemented consistently within and across organizations, and the same results are produced when assessed in the same population in the same time period. Performance variation is necessary to demonstrate that an opportunity exists for improvement across population groups or care settings. Although none of the quality measures that emerged from our study assessed equity as their primary IOM domain, equity issues can be addressed by stratifying measure performance by race, ethnicity, or payer. Assessment of measure properties is needed in both community and children’s hospital settings to ensure translation across all EDs in which children receive care.

CONCLUSIONS

In summary, 20 quality measures were developed to assess the quality of care for acute agitation management in children in the ED. Multidisciplinary ED care team members and parents ranked these measures with high importance and sufficient feasibility for implementation. The proposed quality measures span process, structure, and outcome measures and represent a variety of quality domains. These measures may be used to assess and improve the quality of care for a vulnerable and high-priority population, children with agitation in the ED.

Supplementary Material

Supplemental Table 2
Supplemental Table 1

IMPLICATIONS.

To improve the quality of care for pediatric agitation in the ED in a meaningful way, we must understand what to measure. In this study, the involvement of ED care team members and parents in the measure development process allowed for multiple perspectives to be systematically integrated. Future work should focus on operationalization, field-testing, and assessment of quality measure properties, including the determination of performance variation across populations.

Funding/Support:

Dr. Hoffmann was supported by the Agency for Healthcare Research and Quality (AHRQ) under 5K12HS026385-03. REDCap is supported at Feinberg School of Medicine by the Northwestern University Clinical and Translational Science Institute. Research reported in this publication was supported, in part, by the National Institutes of Health’s National Center for Advancing Translational Sciences, Grant Number UL1TR001422. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The funders did not participate in the work.

Abbreviations:

ED

Emergency department

IOM

Institute of Medicine

Biographical Sketches

Jennifer A. Hoffmann, MD is an Assistant Professor of Pediatrics at Northwestern University Feinberg School of Medicine and an attending physician in pediatric emergency medicine at the Ann & Robert H. Lurie Children’s Hospital of Chicago in Chicago, Illinois. Her research focuses on improving healthcare quality for pediatric mental health emergencies.

Julie K. Johnson, MSPH, PhD is a Professor of Surgery at Northwestern University Feinberg School of Medicine in Chicago, Illinois. She conducts qualitative research on topics related to healthcare quality and safety.

Alba Pergjika, MD, MPH, is an Instructor at Northwestern University Feinberg School of Medicine and an attending physician in child and adolescent psychiatry at the Ann & Robert H. Lurie Children’s Hospital of Chicago in Chicago, Illinois. She practices consultation liaison psychiatry including the management of children with challenging behaviors in the emergency department and hospital setting.

Elizabeth R. Alpern, MD, MSCE is a Professor of Pediatrics at Northwestern University Feinberg School of Medicine and Division Head of in Pediatric Emergency Medicine at the Ann & Robert H. Lurie Children’s Hospital of Chicago in Chicago, Illinois. She uses large databases and research networks to improve the quality of emergent care delivered to children.

Jacqueline B. Corboy, MD is an Assistant Professor of Pediatrics at Northwestern University Feinberg School of Medicine and an attending physician in pediatric emergency medicine at the Ann & Robert H. Lurie Children’s Hospital of Chicago in Chicago, Illinois. She serves as the co-director for quality for the Division of Emergency Medicine and is responsible for overseeing quality improvement initiatives for emergency care.

Footnotes

Conflict of Interests Disclosure: The authors have no conflicts of interest relevant to this article to disclose.

Supplemental Digital Content

Supplemental Table 1.docx

Supplemental Table 2.docx

References

  • 1.Leeb RT, Bitsko RH, Radhakrishnan L, Martinez P, Njai R, Holland KM. Mental Health–Related Emergency Department Visits Among Children Aged <18 Years During the COVID-19 Pandemic — United States, January 1–October 17, 2020. MMWR Morb Mortal Wkly Rep. 2020;69(45):1675–1680. doi: 10.15585/mmwr.mm6945a3 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Foster AA, Porter JJ, Monuteaux MC, Hoffmann JA, Hudgins JD. Pharmacologic Restraint Use During Mental Health Visits in Pediatric Emergency Departments. J Pediatr. 2021;0(0). doi: 10.1016/j.jpeds.2021.03.027 [DOI] [PubMed] [Google Scholar]
  • 3.Dorfman DH, Mehta SD. Restraint use for psychiatric patients in the pediatric emergency department. Pediatr Emerg Care. 2006;22(1):7–12. doi: 10.1097/01.pec.0000195758.12447.69 [DOI] [PubMed] [Google Scholar]
  • 4.Knox DK, Holloman GH. Use and avoidance of seclusion and restraint: Consensus statement of the American Association for emergency psychiatry project BETA seclusion and restraint workgroup. West J Emerg Med. 2012;13(1):35–40. doi: 10.5811/westjem.2011.9.6867 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Gerson R, Malas N, Feuer V, et al. Best Practices for Evaluation and Treatment of Agitated Children and Adolescents (BETA) in the Emergency Department: Consensus Statement of the American Association for Emergency Psychiatry. West J Emerg Med. 2019;20(2):409–418. doi: 10.5811/westjem.2019.1.41344 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Cutler GJ, Rodean J, Zima BT, et al. Trends in Pediatric Emergency Department Visits for Mental Health Conditions and Disposition by Presence of a Psychiatric Unit. Acad Pediatr. 2019;19(8):948–955. doi: 10.1016/j.acap.2019.05.132 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Adirim T, Meade K, Mistry K. A new era in quality measurement: The development and application of quality measures. Pediatrics. 2017;139(1). doi: 10.1542/peds.2016-3442 [DOI] [PubMed] [Google Scholar]
  • 8.Agency for Healthcare Research and Quality. Six Domains of Health Care Quality. Accessed January 22, 2021. https://www.ahrq.gov/talkingquality/measures/six-domains.html
  • 9.Institute of Medicine. Emergency Care for Children: Growing Pains. National Academies Press; 2007. doi: 10.17226/11655 [DOI] [Google Scholar]
  • 10.Alessandrini E, Varadarajan K, Alpern ER, et al. Emergency Department Quality: An Analysis of Existing Pediatric Measures. Acad Emerg Med. 2011;18(5):519–526. doi: 10.1111/j.1553-2712.2011.01057.x [DOI] [PubMed] [Google Scholar]
  • 11.Bardach NS, Burkhart Q, Richardson LP, et al. Hospital-Based Quality Measures for Pediatric Mental Health Care. Pediatrics. 2018;141(6):e20173554. doi: 10.1542/peds.2017-3554 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: A systematic review. PLoS One. 2011;6(6). doi: 10.1371/journal.pone.0020476 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Schmutz J, Eppich WJ, Hoffmann F, Heimberg E, Manser T. Five steps to develop checklists for evaluating clinical performance: An integrative approach. Acad Med. 2014;89(7):996–1005. doi: 10.1097/ACM.0000000000000289 [DOI] [PubMed] [Google Scholar]
  • 14.Shekelle PG, Kahan JP, Bernstein SJ, Leape LL, Kamberg CJ, Park RE. The Reproducibility of a Method to Identify the Overuse and Underuse of Medical Procedures. N Engl J Med. 1998;338(26):1888–1895. doi: 10.1056/nejm199806253382607 [DOI] [PubMed] [Google Scholar]
  • 15.Shekelle PG, Chassin MR, Park RE. Assessing the predictive validity of the RAND/UCLA appropriateness method criteria for performing carotid endarterectomy. Int J Technol Assess Health Care. 1998;14(4):707–727. doi: 10.1017/S0266462300012022 [DOI] [PubMed] [Google Scholar]
  • 16.Brook R The RAND/UCLA Appropriateness Method. In: Clinical Practice Guidelines Development: Methodology Perspectives. (McCormick K, Moore S, Siegel R, eds.). Agency for Health Care Policy and Research; 1994. [Google Scholar]
  • 17.National Quality Forum: Measure Evaluation Criteria. Accessed January 4, 2021. http://www.qualityforum.org/Measuring_Performance/Submitting_Standards/Measure_Evaluation_Criteria.aspx
  • 18.Okoli C, Pawlowski SD. The Delphi method as a research tool: An example, design considerations and applications. Inf Manag. 2004;42(1):15–29. doi: 10.1016/j.im.2003.11.002 [DOI] [Google Scholar]
  • 19.Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap) – A metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inf. 2009;42(2):377–381. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Harris PA, Taylor R, Minor BL, et al. The REDCap consortium: Building an international community of software platform partners. J Biomed Inform. 2019;95. doi: 10.1016/j.jbi.2019.103208 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Zima BT, Murphy JM, Scholle SH, et al. National quality measures for child mental health care: Background, progress, and Next Steps. Pediatrics. 2013;131(SUPPL.1):S38. doi: 10.1542/peds.2012-1427e [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Bardach NS, Burkhart Q, Richardson LP, et al. Hospital-Based Quality Measures for Pediatric Mental Health Care. Pediatrics. 2018;141(6):e20173554. doi: 10.1542/peds.2017-3554 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Schall M, Laderman M, Bamel D, Bolender T. Improving Behavioral Health Care in the Emergency Department and Upstream. Institute for Healthcare Improvement.; 2020. Accessed June 21, 2020. http://www.ihi.org/resources/Pages/IHIWhitePapers/Improving-Behavioral-Health-Care-in-the-Emergency-Department-and-Upstream.aspx
  • 24.Lavelle J, M’Farrej M, Esposito J, et al. Children’s Hospital of Philadelphia Emergency Department Clinical Pathway for Evaluation/Treatment of Children with Behavioral Health Concerns. Published 2013. Accessed April 14, 2021. https://www.chop.edu/clinical-pathway/behavioral-health-issues-clinical-pathway
  • 25.Choi KR, Omery AK, Watkins AM. An Integrative Literature Review of Psychiatric Rapid Response Teams and Their Implementation for De-escalating Behavioral Crises in Nonpsychiatric Hospital Settings. J Nurs Adm. 2019;49(6):297–302. doi: 10.1097/NNA.0000000000000756 [DOI] [PubMed] [Google Scholar]
  • 26.Mitchell M, Newall F, Sokol J, Heywood M, Williams K. Simulation-based education to promote confidence in managing clinical aggression at a paediatric hospital. Adv Simul. 2020;5(1):21. doi: 10.1186/s41077-020-00139-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Barzman DH, Brackenbury L, Sonnier L, et al. Brief rating of aggression by children and adolescents (BRACHA): Development of a tool for assessing risk of inpatients’ aggressive behavior. J Am Acad Psychiatry Law. 2011;39(2):170–179. doi: 10.1016/j.ypsy.2011.09.012 [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplemental Table 2
Supplemental Table 1

RESOURCES