Antibiotic Clinical Decision Support for Pneumonia in the ED: A Randomized Trial

Abstract

Background:

Electronic health record-based clinical decision support (CDS) is a promising antibiotic stewardship strategy. Few studies have evaluated the efficacy of antibiotic CDS in the pediatric emergency department (ED).

Methods:

We conducted a randomized clinical trial comparing CDS vs. usual care for promoting guideline-concordant antibiotic prescribing for pneumonia in the ED at two children’s hospitals. Within each site, CDS or usual care was randomly assigned during 4-week periods. The CDS intervention provided antibiotic recommendations tailored to each encounter and in accordance with national guidelines. The primary outcome was exclusive guideline-concordant antibiotic prescribing within the first 24 hours of care. Safety outcomes included time to first antibiotic order, encounter length of stay, delayed intensive care, and 3- and 7-day revisits.

Results:

1,027 encounters were included, encompassing 478 randomized to usual care and 549 to CDS. Exclusive guideline-concordant prescribing did not differ at 24 hours (CDS, 51.7% vs. usual care, 53.3%; odds ratio [OR] 0.94 [95% CI: 0.73, 1.20]). In pre-specified stratified analyses, CDS was associated with guideline-concordant prescribing among encounters discharged from the ED (74.9% vs. 66.0%; OR 1.53 [95% CI: 1.01, 2.33]), but not among hospitalized encounters. Mean time to first antibiotic was shorter in the CDS group (3.0 vs 3.4 hours; p=0.024). There were no differences in safety outcomes.

Conclusions:

Effectiveness of ED-based antibiotic CDS was greatest among those discharged from the ED. Longitudinal interventions designed to target both ED and inpatient clinicians and to address common implementation challenges may enhance the effectiveness of CDS as a stewardship tool.

INTRODUCTION

Pneumonia is a common childhood infection and accounts for more days of antibiotic use in children’s hospitals than any other condition.^1–3 For bacterial community-acquired pneumonia (CAP), narrow-spectrum antibiotics (i.e., amoxicillin or ampicillin) are recommended for most children.⁴ Less-effective and often broader-spectrum antibiotics, however, are commonly prescribed.^5,6 Appropriate prescribing is critical to slowing antimicrobial resistance and improving outcomes. While hospital-based stewardship programs have demonstrated effectiveness,^7–9 their implementation in the ED environment poses unique challenges.^10–12

Electronic health record (EHR)-based clinical decision support (CDS) is one strategy for addressing the ED antibiotic stewardship gap.^13–15 EHR-based CDS has proven effective, although the magnitude of benefit varies widely.^16–20 Few evaluations of CDS have been conducted in the pediatric ED. Herein, we report the results of a pragmatic, randomized clinical trial comparing EHR-based antibiotic CDS versus usual care for promoting guideline-concordant antibiotic prescribing for CAP in the ED at two children’s hospitals.

METHODS

Study Population

Encounters for children six months to <18 years presenting to the ED at the Monroe Carell Jr. Children’s Hospital at Vanderbilt in Nashville, TN and the Children’s Hospital of Pittsburgh in Pittsburgh, PA with clinician-confirmed radiographic pneumonia were potentially eligible.

Screening was custom-developed within each site’s EHR – Epic (Epic Systems Corporation, Verona, WI) at Vanderbilt and Cerner Millennium (Cerner Corporation, Kansas City, MO) at Pittsburgh. At Vanderbilt, screening was initiated using a real-time, natural language processing classification tool that screened text from chest radiograph reports (based on a random forest model’s predicted likelihood of pneumonia).²¹ Technical challenges prohibited this approach at the Pittsburgh site. Instead, screening was triggered when a chest radiograph was ordered for encounters with a chief complaint reflecting acute respiratory illness (e.g. fever, cough, fast breathing).²² At both sites, screening was restricted to ED encounters. A positive screen prompted an alert to the clinician (faculty and trainee physicians and nurse practictioners) upon chart opening to confirm the pneumonia diagnosis and study eligibility. Children with tracheostomy, cystic fibrosis, or immunosuppression were excluded, as were those hospitalized within the preceding seven days or previously enrolled in the trial within the preceding 28 days. Treating clinicians could choose to acknowledge the alert, to be reminded later, or to opt themselves or the entire encounter out of the trial for any reason. Once eligibity was confirmed, the encounter was considered enrolled. Enrollments occurred continuously from December 17, 2018 to September 9, 2020. The study was approved with a waiver of informed consent at both sites.

Randomization

To minimize contamination and align with trainee rotation schedules, CDS or usual care was assigned to qualifying encounters in four-week periods using randomly determined sequences for each site. Permuted block randomization (block size=4 or 6) was chosen to balance periods of low and high pneumonia prevalence.

CDS Intervention

The EHR-based antibiotic advisor was developed using an iterative, user-centered design process,^23,24 including observations, interviews, and usability testing with ED clinicians (faculty and trainee physicians, as well as nurse practitioners) at both sites in accordance with current CDS design principles.^15,25,26 Key findings emphasized the need for the CDS tool to support efficiency, workflow alignment, and transparency. The resulting recommendations addressed the content and timing of the tool’s user inquiries, triggering alerts, and mechanisms of CDS access embedded in the EHR. Education regarding the trial and the planned CDS intervention was provided in multiple formats, including ED and resident educational conferences, email, and one-on-one discussions. ED clinician champions and study staff shared regular reminders throughout the conduct of the trial.

During CDS periods, tailored antibiotic recommendations were provided for each encounter based on ED clinician input regarding: anticipated ED disposition (home, inpatient, ICU), likelihood of bacterial etiology, beta-lactam antibiotic use within the preceding 48 hours, history of severe penicillin allergy, concern for aspiration, and concern for complicated or severe pneumonia (e.g., moderate to large effusion, necrotizing pneumonia, or suspicion for Staphylococcus aureus). (eFigure 1). Recommendations were in accordance with national and local clinical practice guidelines.⁴ During usual-care periods, no CDS was provided. In both arms, clinicians had usual access to local clinical practice guidelines and retained all decision-making authority regarding antibiotic prescribing.

Primary Outcome

The primary outcome was exclusive guideline-concordant antibiotic prescribing during the first 24 hours of care. Concordance was pre-specified and emphasized use of narrow-spectrum antibiotics. Withholding antibiotics was presumed to be concordant because childhood pneumonia is frequently viral,^4,27 and clinicians were considered highly unlikely to withhold antibiotics in those with suspected bacterial pneumonia. Macrolides were excluded from concordance assessments because clinical equipoise exists regarding use of macrolide therapy for pediatric CAP.^28,29 Otherwise, all systemic antibiotics prescribed during the first 24 hours of care were used to determine concordance.

Secondary Outcomes

Secondary outcomes included any guideline-concordant antibiotic prescribing, defined as prescribing any antibiotic recommended by the guideline, during the first 24 hours of care, as well as exclusive and any guideline-concordant prescribing during the entire encounter. For example, a child hospitalized with uncomplicated CAP and prescribed ampicillin monotherapy during the first 24 hours of care, but later prescribed clindamycin, would be considered concordant for exclusive and any guideline-concordant prescribing during the first 24 hours, discordant for exclusive guideline-concordant prescribing during the entire encounter, and concordant for any guideline-concordant prescribing during the entire encounter. Safety outcomes included time to first antibiotic, length of encounter (calculated in hours from triage to discharge), delayed ICU transfer (transfer of inpatients to ICU after first 24 hours) ED revisits and hospitalizations within three or seven days following discharge, and death within 30 days.

Statistical Analysis

Logistic regression was used to model CDS effects on guideline-concordant prescribing overall, and by ED disposition (discharge, inpatient, ICU), age (<6 years and ≥6 years), and site, as well as restricted to children receiving antibiotics. These same analyses were applied to secondary outcomes. Post hoc analyses were conducted using multivariable logistic regression, adjusting for enrolling site and pneumonia severity score. The severity score estimates risk probabilities for severe pneumonia (e.g., respiratory failure or shock) using age, gender, race and ethnicity, and triage vital signs (systolic blood pressure, respiratory rate, heart rate, and SpO2:FiO2 ratio).³⁰ To examine durability of CDS effects in the transition from the ED to inpatient settings, we also examined exclusive concordance among inpatient encounters separately for the first eight hours of care (a proxy for ED care) and after the first eight hours of care (a proxy for inpatient care). Safety outcomes were compared using Wilcoxon rank sum and Pearson’s chi-squared tests. We also explored alternative models to account for clustering effects by site, although results were unchanged owing to very low intraclass correlation coefficient (0.009 for primary model; not shown). To evaluate for potential contamination effects, we explored the prevalence of guideline-concordant prescribing by arm over time (collapsed into three month blocks) for the primary outcome.

To examine CDS effectiveness under ideal conditions, an exploratory per-protocol analysis was performed at Vanderbilt, which had detailed CDS usage data. For this, multivariable analyses were repeated after restricting the population to encounters that completed the study according to protocol (i.e., confirmation that clinicians viewed treatment recommendations in the CDS group and no protocol deviations in either group).

We anticipated enrolling 750 encounters in each arm of the trial. The study was designed to detect an absolute difference of ≥6.8% in concordant prescribing at an alpha level of 0.05 with 80% power. All analyses were conducted using R version 4.04 (https://www.R-project.org).

RESULTS

Study Population

The study included 1,027 ED encounters (median age 4.2 years) with 478 randomized to usual care and 549 to CDS (Figure 1). Children from usual care encounters were older, more likely to be of Black race or Hispanic ethnicity, and more likely to receive public insurance (Table 1). There were no differences in ED disposition.

Figure 1.

Consort Diagram

Table 1.

Baseline Characteristics of the Study Population

	Combined	Usual Care	CDS
	N=1027	n=478	n=549
Age (years)	4.2 (1.9, 8.4)	4.8 (2.2, 9.5)	4.0 (1.7, 7.4)
Sex
Male	54.9 (564)	54.6 (261)	55.2 (303)
Female	45.1 (463)	45.4 (217)	44.8 (246)
Race
White	67.5 (693)	61.5 (294)	72.7 (399)
Black	19.1 (196)	24.5% (117)	14.4 (79)
Asian	3.0 (31)	2.9 (14)	3.1 (17)
Mixed	2.1 (22)	2.1 (10)	2.2 (12)
Other	4.3 (44)	5.0 (24)	3.6 (20)
Unknown	4.0 (41)	4.0 (19)	4.0 (22)
Ethnicity
Hispanic	9.3 (96)	12.6 (60)	6.6 (36)
Not Hispanic	83.9 (862)	81.6 (390)	86.0 (472)
Unknown	6.7 (69)	5.9 (28)	7.5 (41)
Comorbidity ^
Non-Chronic	62.4 (641)	59.0 (282)	65.4 (359)
Non-complex Chronic	12.8 (131)	12.8 (61)	12.8 (70)
Complex Chronic	24.8 (255)	28.2 (135)	21.9 (120)
Insurance (n=1024)
Public	54.8 (561)	59.2 (281)	51.0 (280)
Private	41.2 (422)	37.1 (176)	44.8 (246)
Self-Pay	4.0 (41)	3.8 (18)	4.2 (23)
Temperature (C) (n=1026)	37.4 (36.9, 38.3)	37.3 (36.8, 38.1)	37.5 (36.9, 38.4)
Heart rate (n=1023)	136 (116, 156)	134 (113, 153)	139 (120, 159)
Respiratory rate (n=1023)	32 (24, 44)	30 (24, 40)	34 (24, 46)
Systolic BP (n=986)	108 (100, 116)	108 (100, 117)	108 (100, 116)
SpO2:FiO2 ratio (n=1019)	457 (328, 474)	462 (360, 479)	457 (306, 474)
ED Disposition
Discharge Home	42.0 (431)	44.4 (212)	39.9 (219)
Inpatient	39.3 (404)	39.5 (189)	39.2 (215)
Intensive Care	18.7 (192)	16.1 (77)	20.9 (115)
Enrolling Site
VUMC	52.1 (535)	67.2 (321)	39 (214)
CHP	47.9 (492)	32.8 (157)	61 (335)

Categorical data presented as % (frequency) and continuous data as median (interquartile range);

^{^}Defined using the Pediatric Medical Complexity Algorithm

Empiric Antibiotic Prescribing and Selection

Antibiotics were prescribed in 825 (80.3%) encounters during the first 24 hours of care (Figure 2). Antibiotic initiation was similar between arms (CDS 82.9% vs usual care 77.4%; p=0.256). Of those receiving antibiotics, 302 (36.6%) received amoxicillin or ampicillin ± macrolide, 162 (19.6%) received a third-generation cephalosporin ± macrolide, and 95 (11.5%) received amoxicillin/clavulanate or ampicillin/sulbactam ± macrolide. Other antibiotic combinations accounted for <10% of prescribing.

Figure 2. Antibiotic Selection, 1^st 24 Hours of Care.

Antibiotic selection was categorized into mutually exclusive groups based on oral or parental antibiotics received during the 31^st 24 hours of care. Macrolide antibiotics were allowed to be grouped with any other combination of antibiotics and are represented with lighter shading. Those not receiving antibiotics, represented as a proportion of all encounters included in the study population, are presented to the left of the vertical double line. Antibiotic groupings, represented as a proportion of all encounters in which antibiotics were received, are presented to the right of the vertical double line

Exclusive Guideline-Concordant Antibiotic Prescribing

Exclusive guideline-concordant prescribing occurred in 539 (52.5%) encounters during the first 24 hours of care, and 501 (48.8%) during the entire episode. Overall, there were no differences in guideline-concordant prescribing by treatment arm at 24 hours (CDS, 51.7% vs. usual care, 53.3%; unadjusted odds ratio [OR] 0.94 [95% CI: 0.73, 1.20]) or the entire episode (47.5% vs 50.2%; OR 0.90 [0.70, 1.15]) (Table 2A). Among discordant encounters, the most common antibiotics prescribed were third-generation cephalosporins (14.3%), amoxicillin-clavulanate (10.7%), and azithromycin (10%).

Table 2A.

Exclusive Guideline-Concordant Antibiotic Prescribing by Study Group

First 24 Hours	Usual Care, % Concordant (no./No.)	CDS, % Concordant (no./No.)	Unadjusted OR (95% CI)
Overall	53.3 (255/478)	51.7 (284/549)	0.94 (0.73, 1.20)
VUMC	54.2 (174/321)	56.1 (120/214)	1.08 (0.76, 1.53)
CHP	51.6 (81/157)	49.0 (164/335)	0.90 (0.62, 1.32)
ED Disposition
Discharge Home	66.0 (140/212)	74.9 (164/219)	1.53 (1.01, 2.33)
Inpatient	49.2 (93/189)	46.0 (99/215)	0.88 (0.60, 1.30)
ICU	28.6 (22/77)	18.3 (21/115)	0.56 (0.28, 1.11)
Age < 6 Years	58.2 (163/280)	53.5 (192/359)	0.83 (0.60, 1.13)
Age ≥ 6 Years	46.5 (92/198)	48.4 (92/190)	1.08 (0.73, 1.61)
Receiving Antibiotics	39.7 (147/370)	42.2 (192/455)	1.11 (0.84, 1.46)
Entire Episode	Usual Care	CDS	Unadjusted OR (95% CI)
Overall	50.2 (240/478)	47.5 (261/549)	0.90 (0.70, 1.15)
VUMC	50.8 (163/321)	51.9 (111/214)	1.05 (0.74, 1.48)
CHP	49.0 (77/157)	44.8 (150/335)	0.84 (0.58, 1.23)
ED Disposition
Discharge Home	-	-	-
Inpatient	43.9 (83/189)	39.5 (85/215)	0.84 (0.56, 1.24)
ICU	22.1 (17/77)	10.4 (12/115)	0.41 (0.18, 0.92)
Age < 6 Years	55.7 (156/280)	48.7 (175/359)	0.76 (0.55, 1.04)
Age ≥ 6 Years	42.4 (84/198)	45.3 (86/190)	1.12 (0.75, 1.68)
Receiving Antibiotics	36.2 (135/373)	37.9 (175/462)	1.08 (0.81, 1.43)

Exclusive guideline-concordant prescribing was highest for encounters discharged from the ED. CDS increased guideline-concordant prescribing in this group (74.9% vs. 66.0%; OR 1.53 [1.01, 2.33]) (Table 2A). There were no differences in inpatient or ICU settings, in analyses stratified by site or age, or among those receiving antibiotics. In post hoc adjusted analyses, the estimates of effect size were similar, although there were no differences in prescribing between groups overall or in stratified analyses (Figure 3A). Among hospitalized encounters stratified by ED (first eight hours) vs. inpatient prescribing (after first eight hours), exclusive concordance was also similar (eTable 1). In our longitudinal exploration of guideline-concordant prescribing by arm, we did not detect evidence of substantial contamination effects in the usual care arm (eFigure 2). Incidentally, we noted a drop in guideline-concordant prescribing in both arms during the last two quarters of the study (Apr-Sep 2020).

Figure 3. Adjusted Odds Ratios for Exclusive and Any Guideline-Concordant Antibiotic Prescribing During First 24 Hours of Care and for Entire Episode.

Forest plots demonstrating adjusted odds ratios with 95% confidence intervals for exclusive (3A) and any (3B) guideline-concordant antibiotic prescribing during first 24 hours of care and for entire episode, overall and for analyses stratified by enrollment site, initial disposition from emergency department, and age, and an analysis restricted to those receiving antibiotics. Odds ratios estimated using logistic regression models adjusted for pneumonia severity score (linear predictor) incorporating age, sex, race/ethnicity, SpO2:FiO2, HR, RR, SBP.

Any Guideline-Concordant Antibiotic Use

During the first 24 hours, antibiotic CDS did not increase any guideline-concordant prescribing (65.2% vs. 62.1%; OR 1.14 [0.89, 1.47]) (Table 2B), except in the subgroup of those receiving antibiotics (58.5% vs. 51.1%; OR 1.35 [95% CI 1.02, 1.78]). For the entire encounter, CDS increased any guideline-concordant prescribing overall (69.8% vs. 63.8%; OR 1.31 [95% CI 1.01, 1.7]) in those six years of age and older (66.8% vs. 55.6%; OR 1.61 [95% CI 1.07, 2.43]), and in those receiving antibiotics (64.3% vs. 53.6%; OR 1.56 [95% CI 1.18, 2.06]). In post hoc adjusted analyses, the estimates of effect size were similar, although most differences were not statistically significant (Figure 3B).

Table 2B.

Any Guideline-Concordant Antibiotic Prescribing by Treatment Group

First 24 Hours	Usual Care, % Concordant (no./No.)	CDS, % Concordant (no./No.)	Unadjusted OR (95% CI)
Overall	62.1 (297/478)	65.2 (358/549)	1.14 (0.89, 1.47)
VUMC	61.7% (198/321)	67.3 (144/214)	1.28 (0.89, 1.84)
CHP	63.1 (99/157)	63.9 (214/335)	1.04 (0.70, 1.54)
ED Disposition
Discharge Home	68.4 (145/212)	76.7 (168/219)	1.52 (0.99, 2.33)
Inpatient	61.9 (117/189)	67.4 (145/215)	1.28 (0.85, 1.92)
ICU	45.5 (35/77)	39.1 (45/115)	0.77 (0.43, 1.38)
Age < 6 Years	67.5 (189/280)	67.4 (242/359)	1.00 (0.71, 1.39)
Age ≥ 6 Years	54.5 (108/198)	61.1 (116/190)	1.31 (0.87, 1.96)
Receiving Antibiotics	51.1 (189/370)	58.5 (266/455)	1.35 (1.02, 1.78)
Entire Episode	Usual Care	CDS	Unadjusted OR (95% CI)
Overall	63.8 (305/478)	69.8 (383/549)	1.31 (1.01, 1.70)
VUMC	62.3 (200/321)	68.2 (146/214)	1.30 (0.90, 1.87)
CHP	66.9 (105/157)	70.7 (237/335)	1.20 (0.80, 1.8)
ED Disposition
Discharge Home	-	-	-
Inpatient	63.0 (119/189)	71.6 (154/215)	1.49 (0.98, 2.26)
ICU	53.2 (41/77)	53.0 (61/115)	0.99 (0.57, 1.77)
Age < 6 Years	69.6 (195/280)	71.3 (256/359)	1.08 (0.77, 1.53)
Age ≥ 6 Years	55.6 (110/198)	66.8 (127/190)	1.61 (1.07, 2.44)
Receiving Antibiotics	53.6 (200/373)	64.3 (297/462)	1.56 (1.18, 2.06)

Safety Outcomes

Antibiotics were initiated faster in the CDS group compared with usual care (median of 3.0 hours vs 3.4 hours, p=0.024). Adverse safety events were uncommon and did not differ between groups (eTable 2). Delayed ICU transfers, ED revisits within 3 days of index discharge, and hospitalizations within 3 or 7 days of index discharge occurred in fewer than 5% of encounters. In both groups, ED revisits within 7 days occurred in <7% and deaths in <1%. Median length of stay was also similar between groups.

Exploratory Per-Protocol Analysis

The single-site per-protocol analysis included 293/321 (91.3%) usual-care encounters and 107/214 (50%) CDS encounters. In the CDS group, nurse practitioners and trainees appeared more likely than faculty physicians to complete the CDS intervention and to use guideline-concordant prescribing (eTable 3A and 3B). Per-protocol analyses were consistent with the main analyses, although odds ratios favoring CDS were generally greater. For exclusive guideline-concordant prescribing, significant differences favoring CDS were noted only among those receiving antibiotics (eFigure 3A). For any guideline-concordant prescribing, significant differences were noted overall, among inpatients, those six years of age or older, and among those receiving antibiotics (eFigure 3B).

DISCUSSION

As a common entry point for children with acute illnesses, the ED is an important target for antibiotic stewardship. High volumes, rapid triage and turnover of patients, a variety of clinician types, and lack of longitudinal care challenge traditional stewardship initiatives.¹² EHR-based CDS is also the modality most preferred by ED clinicians and those engaged in stewardship programs.¹³ In this trial, we demonstrated that EHR-based CDS facilitated judicious antibiotic use in children with pneumonia; effectiveness was greatest among those discharged home from the ED.

Most studies of CDS for promoting evidence-based care demonstrate modest benefits,^{16,17,20,31–36} with a meta-analysis concluding a 5.8% increase in recommended care associated with CDS.¹⁶ In three cluster-randomized trials conducted in pediatric outpatients with acute respiratory illnesses, two studies demonstrated that CDS increased guideline-concordant prescribing (difference 4–8%; baseline concordance 40–70%) as compared to usual care,^31,33 whereas a third study demonstrated no benefit.³² In a randomized trial comparing CDS vs. usual care for antibiotic prescribing in adult outpatients, CDS reduced antibiotic use overall by 25% and broad-spectrum antibiotic use by 50% in adults with possible bacterial pharyngitis or LRTI.³⁷ We are not aware of similar studies focused on antibiotic CDS in the ED, although studies of CDS promoting judicious use of computed tomography in the pediatric ED have demonstrated improvements in care.^17,35

Though CDS improved concordant prescribing among those discharged from the ED, concordant prescribing was lower among inpatients, and we did not observe significant differences overall. There are several potential explanations for these findings. It’s possible that clinicians in both arms relied on other sources (e.g. clinical practice guideline) to inform antibiotic decisions. Second, we hypothesize that patient complexity (e.g., illness severity, presence of comorbidities, and likelihood for more virulent pathogens) contributed to diagnostic uncertainty and cognitive load, resulting in more frequent use of broad-spectrum antibiotics and less CDS adherence, especially among inpatients.³⁸ Supporting this theory, an observational study conducted in the ICU demonstrated no benefit of CDS on antibiotic choice among critically ill children.³⁴ Further, our CDS intervention was presented to ED clinicians only, which may have further limited its effectiveness for inpatient encounters. When examining inpatients only, however, we did not observe substantive decreases in concordance when moving from the ED to the inpatient setting, suggesting that ED treatment strategies were very often continued. Taken together, this suggests that implementation of CDS may be most effective when initiated in the ED and extended throughout a patient’s inpatient stay.

The novel coronavirus pandemic (COVID-19) resulted in rapid and unprecedented declines in pediatric encounters for pneumonia and other respiratory illnesses³⁹ and also likely impacted our study. While study enrollments continued without interruption, accrual was much slower than anticipated and our enrollment target was not attained. This may have limited our ability to detect small differences. The pandemic also likely contributed to the baseline imbalances between the two study arms since subject accrual varied substantially and did not follow anticipated seasonal fluctuations. Post hoc multivariable analyses were performed to mitigate these concerns, although we cannot rule out residual confounding. Further, we hypothesize that increased diagnostic uncertainty (owing to a novel coronavirus associated with severe respiratory disease) and other pandemic-related factors (e.g., access to follow-up) influenced the effectiveness of our CDS intervention. The declines in guideline-concordant prescribing noted in the final two quarters of enrollment across both arms supports this hypothesis.

Our exploratory analyses also indicate that the antibiotic CDS intervention, when used as intended, may have led to greater guideline-concordant prescribing, suggesting higher intervention efficacy and underscoring the paramount importance of CDS design and implementation strategies aimed at enhancing effectiveness.⁴⁰ Only about half of intervention encounters evaluated used the CDS as intended. This aligns with the top end of similar estimates reported in prior studies (range 3–62%),^{19,20,32,33,37} but falls short of ideal and is in spite of careful consideration of ED-specific contextual factors, use of an iterative, user-centered design process,^23,24 adherence to CDS design principles,^15,25,26 and extensive education. Nonetheless, there were appreciable constraints to interface design and deployment within both the Epic and Cerner EHRs, in many cases violating human factors best practices.^41,42 For example, it was not possible to restrict screening alerts to clinicians in the ED, and alerts were sometimes triggered for others (e.g., inpatient or consulting clinicians) who may be less likely to acknowledge and accept recommendations for patients still in the ED. Clinicians were also sometimes required to navigate away from the CDS tool (e.g. to view imaging), a constraint which could have resulted in busy clinicians failing to return to the CDS or only after initial treatment decisions had been made.

In the exploratory per-protocol analysis, faculty physicians were less often represented and appeared less likely to accept CDS recommendations as compared to other clinicians. Although all ED clinician types were represented in user research and educational efforts, faculty physicians may have been more likely to view the CDS after initial treatment decisions had been made, especially in the context of supervising other clinicians. Experienced physicians may also perceive less benefit or reduced autonomy with CDS. It’s also likely that faculty were most engaged in decision-making when diagnostic uncertainty was highest, whereas trainees and nurse practictioners may have relative autonomy with lower acuity, uncomplicated cases. Additional studies are needed to confirm these hypotheses and inform future interventions. Other barriers that may influence perceived utility and uptake of CDS include time and workflow inefficiencies, diagnostic uncertainty, and alert fatigue.^10–12,43 For instance, while viruses are implicated in >70% of children with CAP,²⁷ failure to adequately address uncertainty around viral versus bacterial etiologies may explain why antibiotic initiation was no less common in the CDS arm. In future studies, coupling recommendations with personalized audit and feedback⁴⁴ and other strategies to maximize utility may be more likely to drive meaningful change in clinician behavior.

Despite the implementation challenges, functional benefits of CDS may extend beyond clinical outcomes assessed, including workflow optimization, patient safety, and patient or clinician satisfaction. In this study, CDS decreased time to initial antibiotic ordering and antibiotic CDS has also been associated with reductions in dosing errors and costs.³⁴ Further research is necessary to understand these and other contextual factors to facilitate successful CDS implementation.

The pragmatic approach, with study processes embedded within clinical care, is a notable strength of this study. By design, we were interested in evaluating the real-world effectiveness of CDS in the ED, rather than under ideal experimental conditions of a traditional clinical trial. As such, our results are more readily applicable to other clinical care settings. Our study also has limitations. It’s possible that CDS learning effects carried over into control periods, potentially biasing towards the null. We attempted to minimize contamination using four-week randomization periods that aligned with rotation schedules of residents assigned to the ED, the clinicians most likely to be exposed to the CDS. Our exploratory analyses of quarterly concordant prescribing suggest contamination effects were minimal. It was not feasible to capture all possible scenarios in our guideline-concordance definitions, and concordance may have been misclassified for some encounters. It’s also possible that clinical deterioration following hospitalization necessitated broad-spectrum antibiotics in some cases; in this scenario, concordance may have been misclassified as discordant. As such, concordance in the inpatient settings may have been underestimated. In either case, any misclassification would likely be non-differential. Screening procedures differed across the two sites, and detailed reporting of screening failures and reasons for exclusion were unable to be captured. This may have contributed to heterogeneity in the populations enrolled. Finally, this trial was conducted at two tertiary, university-based children’s hospitals. As compared to children’s hospitals, those cared for in non-children’s hospitals are more likely to receive broad-spectrum antibiotics.⁴⁵ Antibiotic CDS could be more effective in those environments.

In conclusion, the effectiveness of ED-based antibiotic CDS for childhood pneumonia was modest with the greatest effects seen among those discharged home from the ED. Using CDS as a tool for stewardship in this setting may be enhanced by designing longitudinal interventions that target both ED and inpatient clinicians, that address diagnostic uncertainty, and that overcome workflow inefficiencies and other implementation barriers.

Abbreviations

CAP

community-acquired pneumonia

ED

emergency department

EHR

electronic health record

CDS

clinical decision support