Effect of Clinical Decision Support on Diagnostic Imaging for Pediatric Appendicitis: A Cluster Randomized Trial

Anupam B Kharbanda; Gabriela Vazquez-Benitez; Dustin W Ballard; David R Vinson; Uli K Chettipally; Steven P Dehmer; Heidi Ekstrom; Adina S Rauchwerger; Brianna McMichael; Dale M Cotton; Mamata V Kene; Laura E Simon; Jingyi Zhu; E Margaret Warton; Patrick J O’Connor; Elyse O Kharbanda

doi:10.1001/jamanetworkopen.2020.36344

. 2021 Feb 9;4(2):e2036344. doi: 10.1001/jamanetworkopen.2020.36344

Effect of Clinical Decision Support on Diagnostic Imaging for Pediatric Appendicitis

A Cluster Randomized Trial

Anupam B Kharbanda ^1,^✉, Gabriela Vazquez-Benitez ², Dustin W Ballard ^3,⁴, David R Vinson ^3,⁴, Uli K Chettipally ³, Steven P Dehmer ², Heidi Ekstrom ², Adina S Rauchwerger ⁴, Brianna McMichael ¹, Dale M Cotton ³, Mamata V Kene ³, Laura E Simon ⁴, Jingyi Zhu ², E Margaret Warton ⁴, Patrick J O’Connor ², Elyse O Kharbanda ², for the Clinical Research on Emergency Services and Treatments Network (CREST) and the Critical Care Research Center, HealthPartners Institute

¹Department of Pediatric Emergency Medicine, Children’s Minnesota, Minneapolis

²Division of Research, HealthPartners Institute, Minneapolis, Minnesota

³The Permanente Medical Group, Oakland, California

⁴The Kaiser Permanente Northern California Division of Research, Oakland, California

Accepted for Publication: December 16, 2020.

Published: February 9, 2021. doi:10.1001/jamanetworkopen.2020.36344

^✉

Corresponding Author: Anupam B. Kharbanda, MD, MSc, Department of Pediatric Emergency Medicine, Children’s Minnesota, 2525 Chicago Ave S, Minneapolis, MN 55404 (anupam.kharbanda@childrensmn.org).

Author Contributions: Drs Kharbanda and Vazquez-Benitez had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: A. B. Kharbanda, Vazquez-Benitez, Ballard, Vinson, Chettipally, Ekstrom, O’Connor, E. O. Kharbanda.

Acquisition, analysis, or interpretation of data: A. B. Kharbanda, Vazquez-Benitez, Ballard, Vinson, Dehmer, Rauchwerger, McMichael, Cotton, Kene, Simon, Zhu, Warton, O’Connor, E. O. Kharbanda.

Drafting of the manuscript: A. B. Kharbanda, Vazquez-Benitez, Ballard, Ekstrom, Warton.

Critical revision of the manuscript for important intellectual content: Vazquez-Benitez, Ballard, Vinson, Chettipally, Dehmer, Rauchwerger, McMichael, Cotton, Kene, Simon, Zhu, O’Connor, E. O. Kharbanda.

Statistical analysis: A. B. Kharbanda, Vazquez-Benitez, Dehmer, Cotton, Warton.

Obtained funding: A. B. Kharbanda, Ballard, Vinson, Rauchwerger, E. O. Kharbanda.

Administrative, technical, or material support: Ballard, Chettipally, Ekstrom, Rauchwerger, McMichael, Cotton, Kene, Simon, Zhu.

Supervision: A. B. Kharbanda, Ballard, Rauchwerger, E. O. Kharbanda.

Conflict of Interest Disclosures: Ms Ekstrom reported receiving grants from HealthPartners Institute during the conduct of the study. Ms McMichael reported receiving grants from the National Institutes of Health outside the submitted work. Dr Cotton reported grants from Kaiser Permanente during the conduct of the study. No other disclosures were reported.

Funding/Support: This work was supported by grant R01HD079463 from the National Institute of Child Health and Human Development.

Role of the Funder/Sponsor: The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Clinical Research on Emergency Services and Treatments Network (CREST): Site principal investigators: Dustin W. Ballard, MD, MBE, Uli K. Chettipally, MD, MPH, Dale M. Cotton, MD, Mamata V. Kene, MD, MPH, James S. Lin, MD, Dustin G. Mark, MD, D. Ian McLachlan, MD, MPH, Dana R. Sax, MD, MPH, David R. Vinson, MD. Statistical support: Mary E. Reed, DrPH, E. Margaret Warton, MPH. Project management: Adina. S. Rauchwerger, MPH, Laura E. Simon, BA (Kaiser Permanente Northern California Division of Research). Monetary support was provided to several of the study team members.

Critical Care Research Center, HealthPartners Institute: Site leads: Mike Zwank, MD, Bjorn Peterson, MD, Levon Ohaodha, MD, Sandi Wewerka, RN. Advisors: Brad Gordon, MD, Kurt Isenburger, MD, Dave Dries, MD, Martin Richards, MD, Azhar Ali, MD. Monetary support was provided to several of the study team members.

Data Sharing Statement: See Supplement 3.

Additional Contributions: We would like to thank the site investigators, clinical leaders, champions, research managers and coordinators from the following networks for their assistance with developing care recommendations, workflow assessment, training, recruitment, and/or data collection. We would like to thank the Center for Acute Care Outcomes, Children’s Minnesota, David J. Schmeling, MD, and William Mize, MD. We would also like to thank our mentors and advisors for their guidance and support in developing this intervention: Nathan Kuppermann, MD, MPH (University of California, Davis School of Medicine), and Peter S. Dayan, MD, MSc (Columbia University). Each received a minimal stipend to participate in the study.

^✉

Corresponding author.

PMCID: PMC7873779 PMID: 33560426

Key Points

Question

Can an electronic health record–linked clinical decision support tool reduce the use of diagnostic imaging in children and adolescents with emergency department visits for acute abdominal pain?

Findings

In this cluster randomized trial of 17 emergency departments caring for 5940 patients, the clinical decision support tool did not lead to an overall reduction in the use of computed tomography or ultrasonography. In 1 health care system, the clinical support tool was associated with a significant reduction in diagnostic imaging among patients at low or moderate risk of appendicitis.

Meaning

These findings suggest that more research is needed to determine whether clinical decision support tools promote more appropriate imaging in patients with acute abdominal pain.

This cluster randomized trial evaluates the effectiveness of an electronic health record–linked clinical decision support intervention on diagnostic imaging, health care costs, and safety outcomes for patients with suspected appendicitis.

Abstract

Importance

Appendicitis is the most common pediatric surgical emergency. Efforts to improve efficiency and quality of care have increased reliance on computed tomography (CT) and ultrasonography (US) in children with suspected appendicitis.

Objective

To evaluate the effectiveness of an electronic health record–linked clinical decision support intervention, AppyCDS, on diagnostic imaging, health care costs, and safety outcomes for patients with suspected appendicitis.

Design, Setting, and Participants

In this parallel, cluster randomized trial, 17 community-based general emergency departments (EDs) in California, Minnesota, and Wisconsin were randomized to the AppyCDS intervention group or usual care (UC) group. Patients were aged 5 to 20 years, presenting for an ED visit with right-sided or diffuse abdominal pain lasting 5 days or less. We excluded pregnant patients, those with a prior appendectomy, those with selected comorbidities, and those with traumatic injuries. The trial was conducted from October 2016 to July 2019.

Interventions

AppyCDS prompted data entry at the point of care to estimate appendicitis risk using the pediatric appendicitis risk calculator (pARC). Based on pARC estimates, AppyCDS recommended next steps in care.

Main Outcomes and Measures

Primary outcomes were CT, US, or any imaging (CT or US) during the index ED visit. Safety outcomes were perforations, negative appendectomies, and missed appendicitis. Costs were a secondary outcome. Ratio of ratios (RORs) for primary and safety outcomes and differences by group in cost were used to evaluate effectiveness of the clinical decision support tool.

Results

We enrolled 3161 patients at intervention EDs and 2779 patients at UC EDs. The mean age of patients was 11.9 (4.6) years and 2614 (44.0%) were boys or young men. RORs for CT (0.94; 95% CI, 0.75-1.19), US (0.98; 95% CI, 0.84-1.14), and any imaging (0.96; 95% CI, 0.86-1.07) did not differ by study group. In an exploratory analysis conducted in 1 health system, AppyCDS was associated with a reduction in any imaging (ROR, 0.82; 95% CI, 0.73- 0.93) for patients with pARC score of 15% or less and a reduction in CT (ROR, 0.58; 95% CI, 0.45-0.74) for patients with a pARC score of 16% to 50%. Perforations, negative appendectomies, and cases of missed appendicitis by study phase did not differ significantly by study group. Costs did not differ overall by study group.

Conclusions and Relevance

In this study, AppyCDS was not associated with overall reductions in diagnostic imaging; exploratory analysis revealed more appropriate use of imaging in patients with a low pARC score.

Trial Registration

ClinicalTrials.gov Identifier: NCT02633735

Introduction

Appendicitis is the most common pediatric surgical emergency, with more than 75 000 cases in the United States each year.¹ Despite its high frequency, distinguishing appendicitis from other causes of acute abdominal pain remains a challenge.² Pressures to improve emergency department (ED) efficiency, while preventing negative appendectomies and missed appendicitis, have contributed to heavy utilization of computed tomography (CT) and ultrasonography (US) in patients with suspected appendicitis.^3,4

During the past 2 decades, there have been incremental improvements in the evaluation of patients with suspected appendicitis, especially at pediatric EDs,^5,6,7,8,9 where efforts to reduce ionizing radiation exposure have shifted imaging from CT to US.^10,11,12 Clinical pathways for acute abdominal pain have demonstrated the safety and effectiveness^{5,13,14,15,16} of US as first-line imaging.^14,15,16 However, US can be problematic if applied indiscriminately, as it is operator dependent, and equivocal or nondiagnostic results are common.^17,18 Unfortunately, CT use remains high for children with abdominal pain at community-based, general EDs.^3,4,19 Differential management between pediatric and general EDs is concerning, considering that community-based sites are where most children across the United States present for emergent evaluation.²⁰

We previously developed and validated the pediatric appendicitis risk calculator (pARC).²¹ Subsequently, we created a clinical decision support (CDS) system, AppyCDS, integrating pARC with the electronic health record (EHR). Using EHR and web-based algorithms, AppyCDS prompts health care clinicians to screen patients at risk of appendicitis and recommends next steps in care based on calculated risk or pARC score.²² The primary aim of this study was to evaluate the impact of AppyCDS on use of CT or US during an ED visit among pediatric and adolescent patients with acute abdominal pain. Secondary aims were to evaluate the impact of AppyCDS on health care costs and safety outcomes.

Methods

We evaluated AppyCDS in a pragmatic (ie, the intervention was tested in real-world clinical settings), parallel cluster randomized trial in 17 community-based general EDs within 2 large health systems. Of these, 6 were HealthPartners (HP) facilities in Minnesota and Western Wisconsin, with a combined annual ED census of 160 000 visits. Eleven EDs were Kaiser Permanente Northern California (KPNC) facilities, located across Northern California, with a combined annual ED census of 800 000 visits. This study was approved by institutional review boards from participating sites with a waiver of informed consent because it posed minimal risk to patients. This study followed the Consolidated Standards of Reporting Trials (CONSORT) reporting guideline. The trial protocol and statistical analysis plan are available in Supplement 1. For both health systems, patients aged 5 to 20 years old account for approximately 20% of ED visits. All EDs use the Epic EHR. Neither health system is university-based; 5 EDs have academic affiliations. Facilities were staffed by board-certified or board-eligible emergency physicians. Physician assistants also provided care at HP EDs. All EDs had access to CT and US during regular business hours; after-hours US availability varied by facility. Age thresholds to admit for observation or surgery (vs transfer to a dedicated pediatric facility) also varied by ED.

Design, Study Population, and Randomization

We conducted the trial between October 2016 and July 2019, with 2 study phases, ie, preintervention and intervention. Detailed descriptions of the AppyCDS intervention have been published.²² The timing of AppyCDS implementation varied by health system (eFigure in Supplement 2). At HP, following a 5-month pilot, the preintervention phase ran from November 2016 to April 2017. After a 1-month washout period, the intervention phase ran from June 2017 to July 2019. At KPNC, following a 2-month pilot, the preintervention phase ran from October 2016 to June 2017. After a 1-month washout period, the intervention phase ran from August 2017 to July 2019. Data from pilot and washout periods were not included in analyses.

At all sites, patients aged 5 to 20 years presenting to the ED with a chief complaint of abdominal pain were initially assessed using 3 study-specific screening questions, displayed in the EHR, as follows: (1) is the patient’s abdominal pain diffuse (ie, generalized)? (2) does the patient have any right-sided abdominal pain? and (3) what is the duration of abdominal pain? Screening questions were completed by nursing staff (at HP) or treating physicians (at KPNC) prior to evaluation. Patients with 5 days or fewer of right-sided or diffuse abdominal pain were eligible and enrolled. Automated exclusions were applied in real time based on data in the EHR.²² Additional exclusions were applied during analysis (eTable 1 in Supplement 2).

Sites were randomized to the AppyCDS intervention or usual care (UC) in January 2017. We stratified EDs by health system and then grouped them into clusters by patterns of pediatric referral and health care professional coverage. We sorted clusters into pairs by pediatric patient volume. Using a random-number generator, 6 clusters (9 EDs) were randomly assigned to AppyCDS intervention; 6 clusters (8 EDs) were assigned to UC.

Description of Intervention

The overall goals of AppyCDS were as follows: (1) to collect data to calculate risk of appendicitis (pARC score) and (2) to provide point-of-care decision support, recommending next steps in care based on pARC score. As previously described, AppyCDS platforms differed by health system to accommodate existing workflows and EHR-linked CDS infrastructure.²² At HP, clinicians at intervention sites were directed by best practice advisories (BPAs) to use AppyCDS; BPAs displayed for patients who were study eligible when the clinician opened the medical record or attempted to order a CT or US. At KPNC, physicians were trained to access AppyCDS through an existing EHR-linked CDS platform, RISTRA (Risk Stratification).^23,24 In addition, KPNC physicians were notified of potentially eligible patients through automated text message alerts.²⁵ At KPNC, physicians accessing AppyCDS completed initial screening questions, identified exclusions, and, for eligible patients, proceeded with data entry to calculate a pARC score. Across all intervention sites, recommendations for patient management displayed based on pARC score (eTable 2 in Supplement 2). Recommendations were tailored by health system, based on resource availability and consensus from clinical leaders.

Description of UC

At all HP sites preintervention and at HP UC sites during the intervention, no additional data were collected. At all KPNC sites during preintervention and at KPNC UC sites during the intervention, additional clinical information was entered by health care professionals into AppyCDS, but pARC scores and recommended next steps in care were not displayed.

Outcomes

The primary outcome was utilization of diagnostic imaging (ie, CT, US, or any imaging [CT or US]) during the index ED visit. Use of diagnostic imaging was identified through Current Procedural Terminology (CPT) or internal codes.

Appendicitis was confirmed by manual review of pathology and operative notes for appendectomies within 7 days of the index ED visit with definitions based on our prior work.^15,21 When operative or pathology reports were not available, diagnosis was based on data at the index ED visit.

Safety outcomes included negative appendectomies, perforation, and missed appendicitis. Negative appendectomy was defined as a nonincidental appendectomy within 7 days of the index visit with no evidence for appendicitis on histopathology. Perforation was based on manual review of operative notes. Missed appendicitis was defined as appendicitis within 7 days of the index visit among patients discharged home. Appendicitis outcomes were reviewed by 2 study investigators (D.W.B. and E.O.K.) with adjudication by a third investigator (A.B.K.) as needed.

Secondary outcomes included ED length of stay, disposition, ED visits and intensive care admissions within 7 days of the index visit, severe medical conditions (eg, pyelonephritis, nephrolithiasis), and other surgical conditions (eg, intussusception, ovarian torsion). Among patients insured through HP or KPNC, costs of care for the index ED visit and the subsequent 14 days were evaluated using claims data. Costs of care in 2018 US dollars were derived by matching billing codes to Total Care Relative Resource Values,²⁶ a standardized set of measures endorsed by the National Quality Forum for calculating the total cost of care.²⁷ Pharmacy costs were limited to antibiotics, antiemetics, and pain medications.

Statistical Analysis

First, we evaluated patient characteristics and use of imaging by study group and phase using frequency distributions, means, and standard deviations. We evaluated the effectiveness of AppyCDS by estimating the ratio of ratios (RORs) and 95% CIs of the intervention group, from intervention phase to preintervention phase, compared with the UC group for the 2 study phases for primary and safety outcomes. To estimate the ROR, we used a general estimating equation (GEE) Poisson model with a study group indicator, a study phase indicator, a study group–study phase interaction term, and health system as fixed effects. We evaluated whether adding patient characteristics into the model changed the ROR estimates. The GEE model accounted for the data structure, patients within each ED, and randomization by cluster, with a repeat statement and compound symmetry covariance structure. For safety and secondary binary outcomes, similar methods were used. Length of ED stay and hospitalization were estimated using a log normal distribution. Costs were estimated with a GEE model using a γ distribution with a log link for patients enrolled during the intervention phase. For KPNC, where pARC values were collected for all enrolled patients, we evaluated imaging outcomes by pARC strata in exploratory analyses, using a second-order interaction. Furthermore, we evaluated whether the AppyCDS intervention significantly reduced imaging in patients with pARC scores of 50% or less, consistent with the AppyCDS tool recommendations for this low-risk group (eTable 2 in Supplement 2). Analysis for effectiveness of the AppyCDS system was evaluated with a 2-sided test at P < .05 level of significance. Analysis was performed using SAS version 9.4 (SAS Institute). A priori power analysis assumed that with 12 clusters, baseline CT utilization of 30%, 600 patients per cluster during the intervention period, α of .05 with a 2-tailed test, and an intraclass cluster correlation of 0.05, the study would have 80% power to identify an ROR of 0.76, contrasting the post-pre ratios of CT in the intervention vs the UC group.

Results

Between October 2016 and July 2019 there were 40 283 patients aged 5 to 20 years with visits to participating EDs with abdominal pain. Of these, 8605 (21.4%) had an a priori exclusion (eg, prior appendectomy, pregnancy, trauma, other selected comorbidities, or visit during washout period). We excluded 949 patients (2.4%) at intervention sites and 703 (1.7%) at UC sites, based on data entered into AppyCDS indicating pain was not right-sided or generalized or had lasted for more than 5 days (Figure). An additional 12 945 (32.1%) at intervention sites and 11 140 (27.7%) at UC sites did not have data entered in AppyCDS, so their eligibility could not be evaluated. The final enrolled population with 5 days or less of right-sided or diffuse abdominal pain included 3161 patients at intervention EDs (1011 [32.0%] preintervention phase; 2150 [68.0%] intervention phase) and 2779 patients at UC EDs (788 [28.4%] preintervention phase; 1991 [71.6%] intervention phase). Mean age was 11.9 (4.6) years, and 2614 (44.0%) were boys or young men. Patients at UC sites were more likely to be White, non-Hispanic individuals compared with those at intervention sites (preintervention UC, 311 [39.5%]; intervention UC, 901 [45.2%] vs preintervention AppyCDS, 234 [23.2%]; intervention AppyCDS, 721 [34.5%]) (Table 1). Proportions with appendicitis ranged from 11.1% (112 of 1011) to 13.5% (290 of 2150) by study group and study phase (Table 2). Comparison of patients who were not enrolled vs those who were enrolled revealed a higher mean (SD) age (13.4 [5.0] years vs 11.8 [4.5] years) and a higher proportion of female patients (15 166 of 24 203 [62.7%] vs 3293 of 5940 [55.4%]).

Figure. — Cost analyses were limited to enrolled subjects insured through the health system (HealthPartners or Kaiser Permanente, Northern California). CDS indicates clinical decision support; ED, emergency department.

^aNot enrolled indicates that no patient information was entered in the clinical decision support system.

Table 1. Study Population Characteristics by Intervention Group and Study Phase.

Characteristic	No. (%)
	AppyCDS group		Usual care group
	Preintervention phase (n = 1011)	Intervention phase (n = 2150)	Preintervention phase (n = 788)	Intervention phase (n = 1991)
Age, mean (SD), y	11.0 (4.3)	12.4 (4.6)	11.0 (4.3)	11.9 (4.6)
Race/ethnicity
Asian	127 (12.6)	240 (11.0)	45 (5.7)	131 (6.6)
Black	154 (15.2)	319 (15.1)	110 (14.0)	292 (14.7)
Hispanic	403 (30.9)	701 (31.6)	256 (32.5)	532 (26.7)
Other^a	93 (9.2)	169 (7.8)	66 (8.4)	135 (6.8)
White	234 (23.2)	721 (34.5)	311 (39.5)	901 (45.2)
Female patients	559 (55.3)	1221 (56.8)	420 (53.3)	1092 (54.9)
Site
HP	170 (16.8)	721 (33.5)	52 (6.6)	572 (28.7)
KPNC	841 (83.2)	1429 (66.5)	736 (93.4)	1419 (71.3)

Open in a new tab

Abbreviations: HP, HealthPartners; KPNC, Kaiser Permanente, Northern California.

^{^a}

Other category includes American Indian or Alaska Native, Native Hawaiian, more than 1 race, and unknown race.

Table 2. Appendicitis, Imaging Use, and Safety Outcomes During the Index Visit by Intervention Group and Study Phase^a.

Outcome	AppyCDS group			Usual care group			Intervention/usual care, ratio of ratios (95% CI)
	No. (%)		Ratio by phase (95% CI)	No. (%)		Ratio by phase (95% CI)
	Preintervention phase (n = 1011)	Intervention phase (n = 2150)	Ratio by phase (95% CI)	Preintervention phase (n = 788)	Intervention phase (n = 1991)	Ratio by phase (95% CI)
Appendicitis	112 (11.1)	290 (13.5)	1.26 (1.06-1.50)	95 (12.1)	246 (12.4)	1.06 (0.80-1.40)	1.19 (0.86-1.66)
CT use	232 (22.9)	538 (25.0)	1.07 (0.97-1.17)	142 (18.0)	438 (22.0)	1.13 (0.92-1.40)	0.94 (0.75-1.19)
US use	485 (48.0)	986 (45.9)	1.01 (0.91-1.12)	397 (50.4)	958 (48.1)	1.03 (0.92-1.16)	0.98 (0.84-1.14)
Any imaging^b	586 (58.0)	1237 (57.5)	1.02 (0.94-1.11)	455 (57.7)	1163 (58.4)	1.06 (0.98-1.14)	0.96 (0.86-1.08)
Perforation^c	20 (17.9)	43 (14.8)	0.80 (0.48-1.35)	17(17.9)	50 (20.3)	1.09 (0.80-1.48)	0.74 (0.41-1.32)
Negative appendectomy^d	9 (7.4)	11 (3.7)	0.52 (0.28-0.94)	8 (7.8)	19 (7.3)	1.02 (0.36-2.89)	0.51 (.16-1.66)
Missed appendicitis, No./total No. (%)^e	2/845 (0.2)	11/1782 (0.6)	2.62 (1.28-5.34)	5/657 (0.8)	10/1660 (0.6)	0.92 (0.47-1.83)	2.83 (1.0-7.68)

Open in a new tab

Abbreviations: CT, computed tomography; US, ultrasonography.

^{^a}

No statistical significance by study group–pre-post interaction χ² test (CT, P = .62; US, P = .79; any imaging, P = .53; appendicitis, P = .31; perforation, P = .30; negative appendectomy, P = .29; and missed appendicitis, P = .10); general estimating equation Poisson model had pre-post, study group, study group–study phase interaction term, and health system as fixed effects, and compound symmetry covariance structure for emergency department facility.

^{^b}

Any imaging signifies CT or US.

^{^c}

Perforations among patients with appendicitis, based on operative note or clinical adjudication for patients undergoing draining or antibiotic treatment with interval appendectomy.

^{^d}

Negative appendectomy based on pathology report for patients undergoing immediate appendectomy.

^{^e}

Missed appendicitis within 7 days of index emergency department visit, based on clinical adjudication, of patients discharged home at index visit.

Primary Outcomes

There was no statistically significant change in CT use from preintervention to intervention phase between study groups (ROR, 0.94; 95% CI, 0.75-1.19) (Table 2). A nonsignificant decrease in US use was noted between preintervention and intervention phases at intervention and UC sites (485 of 1011 [48.0%] to 986 of 2150 [45.9%] and 397 of 788 [50.4%] to 958 of 1991 [48.1%], respectively; ROR, 0.98; 95% CI, 0.84-1.14). At intervention sites, any imaging decreased nonsignificantly from 586 (58.0%) in the preintervention phase to 1237 (57.5%) in the intervention phase. At UC sites, there was a nonsignificant increase in any imaging, from 455 (57.7%) in the preintervention phase to 1163 (58.4%) in the intervention phase. The ratio of any imaging by study phase did not differ by study group (ROR, 0.96; 95% CI, 0.86-1.08). After adjusting for age, sex, and race, results were similar (CT: ROR, 0.92; 95% CI, 0.76-1.06; US: ROR, 0.98; 95% CI, 0.83-1.14; any imaging: ROR, 0.95; 95% CI, 0.84-1.06).

Exploratory Analyses

At KPNC, where pARC scores were calculated for all enrolled patients, AppyCDS was associated with a reduction in any imaging (ROR, 0.82; 95% CI, 0.73-0.93) for pARC scores of 15% or less and a reduction in CT (ROR, 0.58; 95% CI, 0.45-0.74) for pARC scores between 16% and 50% (Table 3). However, there was no overall association between pARC strata and imaging.

Table 3. Imaging Use by pARC Risk Stratification, Study Group and Phase, at Kaiser Permanente Northern California Sites.

pARC score	AppyCDS group		Usual care group		Ratio of ratios (95% CI)^a
pARC score	Preintervention phase (n = 510)	Intervention phase (n = 1054)	Preintervention phase (n = 490)	Intervention phase (n = 997)	Ratio of ratios (95% CI)^a
≤15%
CT use	73 (26.1)	105 (19.1)	49 (19.3)	84 (15.9)	0.85 (0.54-1.33)
US use	178 (63.6)	291 (52.8)	159 (62.6)	327 (62.0)	0.88 (0.77-0.99)
Any imaging^b	207 (73.9)	331 (60.1)	177 (69.7)	368 (69.8)	0.82 (0.73-0.93)
16%-50%
CT use	63 (46.7)	112 (38.0)	38 (27.9)	109 (38.5)	0.58 (0.45-0.74)
US use	103 (76.3)	199 (67.5)	107 (78.7)	207 (73.1)	1.04 (0.92-1.17)
Any imaging^b	126 (93.3)	240 (81.4)	121 (89.0)	245 (86.6)	0.94 (0.83-1.06)
51%-75%
CT use	25 (45.5)	70 (56.5)	31 (47.7)	50 (47.2)	1.12 (0.64-1.95)
US use	37 (67.3)	89 (71.8)	54 (83.1)	76 (71.7)	1.19 (0.95-1.50)
Any imaging^b	51 (92.7)	117 (94.4)	63 (96.9)	99 (93.4)	1.02 (0.92-1.12)
>75%
CT use	19 (47.5)	48 (57.1)	5 (14.3)	35 (43.2)	0.45 (0.18-1.11)
US use	30 (75.0)	58 (69.0)	29 (82.9)	63 (77.8)	0.98 (0.73-1.30)
Any imaging^b	37 (92.5)	81 (96.4)	33 (94.3)	79 (97.5)	1.02 (0.91-1.15)

Open in a new tab

Abbreviations: CT, computed tomography; pARC, pediatric appendicitis risk calculator; US, ultrasonography.

^{^a}

A test for heterogeneity, with 3 degrees of freedom, was conducted across pARC strata (CT, P = .40; US, P = .12; any imaging, P = .10).

^{^b}

Any imaging signifies CT or US.

Safety, Secondary Outcomes, and Cost Analysis

Perforation rates decreased over time at intervention sites (20 [17.9%] to 43 [14.8%]) and increased at UC sites (17 [17.9%] to 50 [20.3%]); the ratio of perforation by phase did not differ significantly by study arm (ROR, 0.74; 95% CI, 0.41-1.32). Negative appendectomies decreased at both intervention sites (9 [7.4%] to 11 [3.7%]) and at UC sites (8 [7.8%] to 19 [7.3%]); however, the ratio by study arm was not significant (ROR, 0.51; 95% CI, 0.16-1.66). Missed appendicitis was uncommon in intervention and UC arms across study phases; changes in missed appendicitis rates between the preintervention and intervention phases, by study group, were not significant (ROR, 2.83; 95% CI, 1.0-7.7) (Table 2).

Most patients were discharged home following their ED visit with minimal variation by study group and study phase. Mean ED length of stay increased minimally at intervention sites and remained constant at UC sites. Additional secondary outcomes did not differ by study phase by study group (eTable 3 in Supplement 2).

A total of 1372 patients in the intervention arm and 1290 patients in the UC arm had HP or KPNC insurance and were included in cost analyses (Figure). Overall health care costs did not significantly differ by study group. Nevertheless, total outpatient costs on days 4 to 14 after the index visit were significantly lower at intervention sites compared with UC sites (difference, −$31; 95% CI, −$47 to −$15) (Table 4; eTable 4 in Supplement 2).

Table 4. Costs of Care Within 14 Days After Index Emergency Department Visit by Study Group During Intervention Phase^a.

Category	Cost, $		Difference (95% CI), $	P value
Category	AppyCDS group (n = 1372)	Usual care group (n = 1290)	Difference (95% CI), $	P value
Total costs	1995	2020	−25 (−260 to 210)	.83
Total outpatient costs	993	1035	−42 (−172 to 88)	.52
Index ED visit	633	620	13 (−79 to 106)	.78
CT imaging	112	107	5 (−31 to 41)	.80
US imaging	78	81	−3 (−31 to 26)	.85
All other outpatient costs	360	410	−50 (−110 to 9)	.10
Within 3 d of index visit	284	302	−18 (−68 to 33)	.49
4-14 d From index visit	78	108	−31 (−47 to −15)	<.001
Total inpatient costs	947	960	−13 (−312 to 286)	.93
Pharmacy costs	17	20	−3 (−8 to 2)	.28

Open in a new tab

Abbreviations: CT, computed tomography; ED, emergency department; US, ultrasonography.

^{^a}

All costs were measured in 2018 US dollars during 14 days following and including the index ED visit for patients in each study group during the intervention phase. Total costs equal the sum of total outpatient costs, total inpatient costs, and pharmacy costs. All other outpatient costs include all outpatient costs (including emergency department) observed outside of the index visit. Pharmacy costs were limited to prescription fills for antibiotics, antiemetics, and pain medications. Subgroup totals will not necessarily sum to grand totals due to independently modeling costs within each group while accounting for cluster randomization across 2 sites.

Discussion

In this multisite pragmatic cluster randomized trial, AppyCDS did not reduce CT or US use for the full spectrum of pediatric patients with suspected appendicitis. However, in an exploratory analysis, patients with a low (≤15%) or moderate (16%-50%) pARC scores underwent less imaging. Our study demonstrated the feasibility and safety of conducting a large randomized trial to improve care for children presenting for care in general, community-based EDs, while highlighting challenges and areas for improvement in future interventions.

The pARC score was designed to quantify risk of appendicitis and maximize sorting of patients as having high or low risk of appendicitis.²¹ Each participating health system developed care recommendations based on pARC strata, with input from clinical leaders. At intervention sites, clinicians were significantly more likely to defer imaging for patients with a less than 15% risk of appendicitis, consistent with AppyCDS recommendations. For those with a risk of 16% to 50%, we recommended US as first-line imaging, and accordingly, we noted a significant decrease in use of CT at intervention sites. Consistent with data from the National Surgical Quality Improvement Program for Pediatrics, patients with high pARC scores (>75%) continued to undergo diagnostic imaging at high rates (>90%).³ These results suggest that intervention EDs moved toward more appropriate imaging.³⁰

Our findings are also consistent with prior research demonstrating low CDS use in acute health care settings.^28,29 AppyCDS was designed to provide recommendations in real time to influence decision-making; however, alerts to complete the screening or use the CDS were easy to dismiss or ignore. The CDS system was developed in collaboration with end-users, yet fully automated estimation of pARC using EHR data may be preferred to improve efficiency and adherence. Requiring clinicians to provide a reason for overriding the CDS system or providing real-time feedback has also been demonstrated to improve use of CDS systems.^31,32

Lack of agreement with recommendations, along with system-level barriers, may have limited adherence with AppyCDS guidance.³³ For example, calculation of appendicitis risk through pARC required a white blood cell count, yet parents may have preferred imaging to confirm a diagnosis without a blood draw. For patients at low to moderate risk (ie, pARC score of 16%-25%), the CDS recommended observation prior to imaging, but this may be impractical in busy general EDs with limited space for pediatric patients. For many community-based EDs, AppyCDS recommendations represented a deimplementation of care (ie, forgoing US in patients at low risk of appendicitis). Efforts to deimplement care in the ED may require multilevel approaches that cannot be delivered through CDS systems alone. Ultimately, shared decision-making between families and clinicians,³⁴ based on pARC score, along with health system support,³⁵ may be needed to optimize diagnostic imaging in pediatric patients with acute abdominal pain.³⁶

Limitations

Several important limitations should be noted. There were a large number of patients who were not enrolled at intervention and control sites. As previously described, the rate of appendicitis among these patients was low²²; it is likely that physicians ignored alerts to use AppyCDS because they had low suspicion for appendicitis. Improved capture of these patients may have increased the generalizability of our results but would also have resulted in a lower rate of appendicitis. The clinical characteristics of our enrolled population were consistent with prior publications describing children with acute abdominal pain.^15,21,37 Regardless, our findings demonstrated the challenges we faced trying to evaluate and improve care for children in community-based general EDs. Barriers to CDS systems adherence may be amplified in settings where children are only a fraction of the patient population, and thus, alerts to complete CDS screening questions were infrequent. Additional barriers to CDS system use, such as pressure to reduce ED length of stay, lack of consistent availability of US, financial concerns of families, and fear of litigation due to missed appendicitis, were not measured. To conduct this trial in 2 health systems across 17 hospitals, it was necessary to provide system-level recommendations for care and differences in integration of the CDS system.²² However, this approach did not account for varying levels of risk tolerance among physicians and surgeons. Furthermore, our primary a priori study aim was to demonstrate an overall reduction in diagnostic imaging. A more nuanced approach to outcome assessment may have been more appropriate and was consistent with our findings. To achieve sustained reductions in diagnostic imaging, further studies should evaluate targeted approaches to US based on appendicitis risk.

Conclusions

In this cluster randomized trial, we demonstrated the feasibility and safety of an EHR-linked appendicitis CDS tool in community-based EDs. Although we did not show an overall reduction in imaging, exploratory analyses at 1 health system revealed more appropriate use of diagnostic imaging.

Supplement 1.

Trial Protocol and Statistical Analysis Plan

Click here for additional data file.^{(2.7MB, pdf)}

Supplement 2.

eTable 1. Exclusions Applied in Real Time and Post Hoc and Their Data Sources

eTable 2. Recommendations by Pediatric Appendicitis Risk Calculator (pARC) and by Site

eTable 3. Secondary Outcomes by Study Group and Phase

eTable 4. Outpatient Costs of Care, Excluding Index Emergency Department Visit, by Study Group and Time From Index Visit During Intervention Phase

eFigure. Calendar Month and Year for Start and End of Pilot, Preintervention, Washout, and Intervention Phases by Site

Click here for additional data file.^{(858.7KB, pdf)}

Supplement 3.

Data Sharing Statement

Click here for additional data file.^{(21.4KB, pdf)}

References

1.Barrett ML, Hines AL, Andrews RM Trends in rates of perforated appendix, 2001-2010: Statistical Brief #159. In: Healthcare Cost and Utilization Project (HCUP) Statistical Briefs Agency for Healthcare Research and Quality; 2006. Accessed January 5, 2021. https://www.hcup-us.ahrq.gov/reports/statbriefs/sb159.pdf
2.Colvin JM, Bachur R, Kharbanda A. The presentation of appendicitis in preadolescent children. Pediatr Emerg Care. 2007;23(12):849-855. doi: 10.1097/pec.0b013e31815c9d7f [DOI] [PubMed] [Google Scholar]
3.Anderson KT, Bartz-Kurycki MA, Austin MT, et al. Hospital type predicts computed tomography use for pediatric appendicitis. J Pediatr Surg. 2019;54(4):723-727. doi: 10.1016/j.jpedsurg.2018.05.018 [DOI] [PubMed] [Google Scholar]
4.Raval MV, Deans KJ, Rangel SJ, Kelleher KJ, Moss RL. Factors associated with imaging modality choice in children with appendicitis. J Surg Res. 2012;177(1):131-136. doi: 10.1016/j.jss.2012.03.044 [DOI] [PubMed] [Google Scholar]
5.Shah SR, Sinclair KA, Theut SB, Johnson KM, Holcomb GW III, St Peter SD. Computed tomography utilization for the diagnosis of acute appendicitis in children decreases with a diagnostic algorithm. Ann Surg. 2016;264(3):474-481. doi: 10.1097/SLA.0000000000001867 [DOI] [PubMed] [Google Scholar]
6.Warner BW, Kulick RM, Stoops MM, Mehta S, Stephan M, Kotagal UR. An evidenced-based clinical pathway for acute appendicitis decreases hospital duration and cost. J Pediatr Surg. 1998;33(9):1371-1375. doi: 10.1016/S0022-3468(98)90010-0 [DOI] [PubMed] [Google Scholar]
7.Warner BW, Rich KA, Atherton H, Andersen CL, Kotagal UR. The sustained impact of an evidenced-based clinical pathway for acute appendicitis. Semin Pediatr Surg. 2002;11(1):29-35. doi: 10.1053/spsu.2002.29364 [DOI] [PubMed] [Google Scholar]
8.Saucier A, Huang EY, Emeremni CA, Pershad J. Prospective evaluation of a clinical pathway for suspected appendicitis. Pediatrics. 2014;133(1):e88-e95. doi: 10.1542/peds.2013-2208 [DOI] [PubMed] [Google Scholar]
9.Wagenaar AE, Tashiro J, Wang B, et al. Protocol for suspected pediatric appendicitis limits computed tomography utilization. J Surg Res. 2015;199(1):153-158. doi: 10.1016/j.jss.2015.04.028 [DOI] [PubMed] [Google Scholar]
10.Bachur RG, Levy JA, Callahan MJ, Rangel SJ, Monuteaux MC. Effect of reduction in the use of computed tomography on clinical outcomes of appendicitis. JAMA Pediatr. 2015;169(8):755-760. doi: 10.1001/jamapediatrics.2015.0479 [DOI] [PubMed] [Google Scholar]
11.Niles LM, Goyal MK, Badolato GM, Chamberlain JM, Cohen JS. US emergency department trends in imaging for pediatric nontraumatic abdominal pain. Pediatrics. 2017;140(4):e20170615. doi: 10.1542/peds.2017-0615 [DOI] [PubMed] [Google Scholar]
12.Otero HJ, Crowder L. Imaging utilization for the diagnosis of appendicitis in stand-alone children’s hospitals in the United States: trends and costs. J Am Coll Radiol. 2017;14(5):603-608. doi: 10.1016/j.jacr.2016.12.013 [DOI] [PubMed] [Google Scholar]
13.Santillanes G, Simms S, Gausche-Hill M, et al. Prospective evaluation of a clinical practice guideline for diagnosis of appendicitis in children. Acad Emerg Med. 2012;19(8):886-893. doi: 10.1111/j.1553-2712.2012.01402.x [DOI] [PMC free article] [PubMed] [Google Scholar]
14.Depinet H, von Allmen D, Towbin A, Hornung R, Ho M, Alessandrini E. Risk stratification to decrease unnecessary diagnostic imaging for acute appendicitis. Pediatrics. 2016;138(3):e20154031. doi: 10.1542/peds.2015-4031 [DOI] [PubMed] [Google Scholar]
15.Kharbanda AB, Madhok M, Krause E, et al. Implementation of electronic clinical decision support for pediatric appendicitis. Pediatrics. 2016;137(5):e20151745. doi: 10.1542/peds.2015-1745 [DOI] [PubMed] [Google Scholar]
16.Gurien LA, Smith SD, Dassinger MS, Burford JM, Tepas JJ, Crandall M. Suspected appendicitis pathway continues to lower CT rates in children two years after implementation. Am J Surg. 2019;218(4):716-721. doi: 10.1016/j.amjsurg.2019.07.024 [DOI] [PubMed] [Google Scholar]
17.Binkovitz LA, Unsdorfer KM, Thapa P, et al. Pediatric appendiceal ultrasound: accuracy, determinacy and clinical outcomes. Pediatr Radiol. 2015;45(13):1934-1944. doi: 10.1007/s00247-015-3432-7 [DOI] [PubMed] [Google Scholar]
18.Schuh S, Chan K, Langer JC, et al. Properties of serial ultrasound clinical diagnostic pathway in suspected appendicitis and related computed tomography use. Acad Emerg Med. 2015;22(4):406-414. doi: 10.1111/acem.12631 [DOI] [PubMed] [Google Scholar]
19.Kotagal M, Richards MK, Flum DR, Acierno SP, Weinsheimer RL, Goldin AB. Use and accuracy of diagnostic imaging in the evaluation of pediatric appendicitis. J Pediatr Surg. 2015;50(4):642-646. doi: 10.1016/j.jpedsurg.2014.09.080 [DOI] [PMC free article] [PubMed] [Google Scholar]
20.McDermott KW, Stocks C, Freeman WJ Overview of pediatric emergency department visits, 2015: Statistical Brief #242. In: Healthcare Cost and Utilization Project (HCUP) Statistical Briefs. Agency for Healthcare Research and Quality; 2018. [PubMed] [Google Scholar]
21.Kharbanda AB, Vazquez-Benitez G, Ballard DW, et al. Development and validation of a novel pediatric appendicitis risk calculator (pARC). Pediatrics. 2018;141(4):e20172699. doi: 10.1542/peds.2017-2699 [DOI] [PMC free article] [PubMed] [Google Scholar]
22.Ekstrom HL, Kharbanda EO, Ballard DW, et al. Development of a clinical decision support system for pediatric abdominal pain in emergency department settings across two health systems within the HCSRN. EGEMS (Wash DC). 2019;7(1):15. doi: 10.5334/egems.282 [DOI] [PMC free article] [PubMed] [Google Scholar]
23.Vinson DR, Mark DG, Chettipally UK, et al. ; eSPEED Investigators of the KP CREST Network . Increasing safe outpatient management of emergency department patients with pulmonary embolism: a controlled pragmatic trial. Ann Intern Med. 2018;169(12):855-865. doi: 10.7326/M18-1206 [DOI] [PubMed] [Google Scholar]
24.Ballard DW, Vemula R, Chettipally UK, et al. ; KP CREST Network Investigators . Optimizing clinical decision support in the electronic health record: clinical characteristics associated with the use of a decision tool for disposition of ED patients with pulmonary embolism. Appl Clin Inform. 2016;7(3):883-898. doi: 10.4338/ACI-2016-05-RA-0073 [DOI] [PMC free article] [PubMed] [Google Scholar]
25.Simon LE, Rauchwerger AS, Chettipally UK, et al. Text message alerts to emergency physicians identifying potential study candidates increase clinical trial enrollment. J Am Med Inform Assoc. 2019;26(11):1360-1363. doi: 10.1093/jamia/ocz118 [DOI] [PMC free article] [PubMed] [Google Scholar]
26.HealthPartners. Total Care Relative Resource Value (TCRRVTM): a measurement approach to achieve the triple aim. Updated September 21, 2017. Accessed June 12, 2018. https://www.healthpartners.com/ucm/groups/public/@hp/@public/documents/documents/cntrb_039627.pdf
27.National Quality Forum NQF endorses resource use measures. Published January 31, 2012. Accessed June 12, 2018. http://www.qualityforum.org/News_And_Resources/Press_Releases/2012/NQF_Endorses_Resource_Use_Measures.aspx
28.Yan Z, Ip IK, Raja AS, Gupta A, Kosowsky JM, Khorasani R. Yield of CT pulmonary angiography in the emergency department when providers override evidence-based clinical decision support. Radiology. 2017;282(3):717-725. doi: 10.1148/radiol.2016151985 [DOI] [PMC free article] [PubMed] [Google Scholar]
29.Khairat S, Marc D, Crosby W, Al Sanousi A. Reasons for physicians not adopting clinical decision support systems: critical analysis. JMIR Med Inform. 2018;6(2):e24. doi: 10.2196/medinform.8912 [DOI] [PMC free article] [PubMed] [Google Scholar]
30.Young PJ, Nickson CP, Perner A. When should clinicians act on non-statistically significant results from clinical trials? JAMA. 2020;323(22):2256-2257. doi: 10.1001/jama.2020.3508 [DOI] [PubMed] [Google Scholar]
31.O’Connor SD, Sodickson AD, Ip IK, et al. Journal club: requiring clinical justification to override repeat imaging decision support: impact on CT use. AJR Am J Roentgenol. 2014;203(5):W482-90. doi: 10.2214/AJR.14.13017 [DOI] [PMC free article] [PubMed] [Google Scholar]
32.Raja AS, Ip IK, Dunne RM, Schuur JD, Mills AM, Khorasani R. Effects of performance feedback reports on adherence to evidence-based guidelines in use of CT for evaluation of pulmonary embolism in the emergency department: a randomized trial. AJR Am J Roentgenol. 2015;205(5):936-940. doi: 10.2214/AJR.15.14677 [DOI] [PMC free article] [PubMed] [Google Scholar]
33.Cabana MD, Rand CS, Powe NR, et al. Why don’t physicians follow clinical practice guidelines? a framework for improvement. JAMA. 1999;282(15):1458-1465. doi: 10.1001/jama.282.15.1458 [DOI] [PubMed] [Google Scholar]
34.Hess EP, Homme JL, Kharbanda AB, et al. Effect of the head computed tomography choice decision aid in parents of children with minor head trauma: a cluster randomized trial. JAMA Netw Open. 2018;1(5):e182430. doi: 10.1001/jamanetworkopen.2018.2430 [DOI] [PMC free article] [PubMed] [Google Scholar]
35.Davidson KW, Ye S, Mensah GA. Commentary: de-implementation science: a virtuous cycle of ceasing and desisting low-value care before implementing new high value care. Ethn Dis. 2017;27(4):463-468. doi: 10.18865/ed.27.4.463 [DOI] [PMC free article] [PubMed] [Google Scholar]
36.Norton WE, Chambers DA. Unpacking the complexities of de-implementing inappropriate health interventions. Implement Sci. 2020;15(1):2. doi: 10.1186/s13012-019-0960-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
37.Kharbanda AB, Dudley NC, Bajaj L, et al. ; Pediatric Emergency Medicine Collaborative Research Committee of the American Academy of Pediatrics . Validation and refinement of a prediction rule to identify children at low risk for acute appendicitis. Arch Pediatr Adolesc Med. 2012;166(8):738-744. doi: 10.1001/archpediatrics.2012.490 [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplement 1.

Trial Protocol and Statistical Analysis Plan

Click here for additional data file.^{(2.7MB, pdf)}

Supplement 2.

eTable 1. Exclusions Applied in Real Time and Post Hoc and Their Data Sources

eTable 2. Recommendations by Pediatric Appendicitis Risk Calculator (pARC) and by Site

eTable 3. Secondary Outcomes by Study Group and Phase

eTable 4. Outpatient Costs of Care, Excluding Index Emergency Department Visit, by Study Group and Time From Index Visit During Intervention Phase

eFigure. Calendar Month and Year for Start and End of Pilot, Preintervention, Washout, and Intervention Phases by Site

Click here for additional data file.^{(858.7KB, pdf)}

Supplement 3.

Data Sharing Statement

Click here for additional data file.^{(21.4KB, pdf)}

[zoi201089r1] 1.Barrett ML, Hines AL, Andrews RM Trends in rates of perforated appendix, 2001-2010: Statistical Brief #159. In: Healthcare Cost and Utilization Project (HCUP) Statistical Briefs Agency for Healthcare Research and Quality; 2006. Accessed January 5, 2021. https://www.hcup-us.ahrq.gov/reports/statbriefs/sb159.pdf

[zoi201089r2] 2.Colvin JM, Bachur R, Kharbanda A. The presentation of appendicitis in preadolescent children. Pediatr Emerg Care. 2007;23(12):849-855. doi: 10.1097/pec.0b013e31815c9d7f [DOI] [PubMed] [Google Scholar]

[zoi201089r3] 3.Anderson KT, Bartz-Kurycki MA, Austin MT, et al. Hospital type predicts computed tomography use for pediatric appendicitis. J Pediatr Surg. 2019;54(4):723-727. doi: 10.1016/j.jpedsurg.2018.05.018 [DOI] [PubMed] [Google Scholar]

[zoi201089r4] 4.Raval MV, Deans KJ, Rangel SJ, Kelleher KJ, Moss RL. Factors associated with imaging modality choice in children with appendicitis. J Surg Res. 2012;177(1):131-136. doi: 10.1016/j.jss.2012.03.044 [DOI] [PubMed] [Google Scholar]

[zoi201089r5] 5.Shah SR, Sinclair KA, Theut SB, Johnson KM, Holcomb GW III, St Peter SD. Computed tomography utilization for the diagnosis of acute appendicitis in children decreases with a diagnostic algorithm. Ann Surg. 2016;264(3):474-481. doi: 10.1097/SLA.0000000000001867 [DOI] [PubMed] [Google Scholar]

[zoi201089r6] 6.Warner BW, Kulick RM, Stoops MM, Mehta S, Stephan M, Kotagal UR. An evidenced-based clinical pathway for acute appendicitis decreases hospital duration and cost. J Pediatr Surg. 1998;33(9):1371-1375. doi: 10.1016/S0022-3468(98)90010-0 [DOI] [PubMed] [Google Scholar]

[zoi201089r7] 7.Warner BW, Rich KA, Atherton H, Andersen CL, Kotagal UR. The sustained impact of an evidenced-based clinical pathway for acute appendicitis. Semin Pediatr Surg. 2002;11(1):29-35. doi: 10.1053/spsu.2002.29364 [DOI] [PubMed] [Google Scholar]

[zoi201089r8] 8.Saucier A, Huang EY, Emeremni CA, Pershad J. Prospective evaluation of a clinical pathway for suspected appendicitis. Pediatrics. 2014;133(1):e88-e95. doi: 10.1542/peds.2013-2208 [DOI] [PubMed] [Google Scholar]

[zoi201089r9] 9.Wagenaar AE, Tashiro J, Wang B, et al. Protocol for suspected pediatric appendicitis limits computed tomography utilization. J Surg Res. 2015;199(1):153-158. doi: 10.1016/j.jss.2015.04.028 [DOI] [PubMed] [Google Scholar]

[zoi201089r10] 10.Bachur RG, Levy JA, Callahan MJ, Rangel SJ, Monuteaux MC. Effect of reduction in the use of computed tomography on clinical outcomes of appendicitis. JAMA Pediatr. 2015;169(8):755-760. doi: 10.1001/jamapediatrics.2015.0479 [DOI] [PubMed] [Google Scholar]

[zoi201089r11] 11.Niles LM, Goyal MK, Badolato GM, Chamberlain JM, Cohen JS. US emergency department trends in imaging for pediatric nontraumatic abdominal pain. Pediatrics. 2017;140(4):e20170615. doi: 10.1542/peds.2017-0615 [DOI] [PubMed] [Google Scholar]

[zoi201089r12] 12.Otero HJ, Crowder L. Imaging utilization for the diagnosis of appendicitis in stand-alone children’s hospitals in the United States: trends and costs. J Am Coll Radiol. 2017;14(5):603-608. doi: 10.1016/j.jacr.2016.12.013 [DOI] [PubMed] [Google Scholar]

[zoi201089r13] 13.Santillanes G, Simms S, Gausche-Hill M, et al. Prospective evaluation of a clinical practice guideline for diagnosis of appendicitis in children. Acad Emerg Med. 2012;19(8):886-893. doi: 10.1111/j.1553-2712.2012.01402.x [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi201089r14] 14.Depinet H, von Allmen D, Towbin A, Hornung R, Ho M, Alessandrini E. Risk stratification to decrease unnecessary diagnostic imaging for acute appendicitis. Pediatrics. 2016;138(3):e20154031. doi: 10.1542/peds.2015-4031 [DOI] [PubMed] [Google Scholar]

[zoi201089r15] 15.Kharbanda AB, Madhok M, Krause E, et al. Implementation of electronic clinical decision support for pediatric appendicitis. Pediatrics. 2016;137(5):e20151745. doi: 10.1542/peds.2015-1745 [DOI] [PubMed] [Google Scholar]

[zoi201089r16] 16.Gurien LA, Smith SD, Dassinger MS, Burford JM, Tepas JJ, Crandall M. Suspected appendicitis pathway continues to lower CT rates in children two years after implementation. Am J Surg. 2019;218(4):716-721. doi: 10.1016/j.amjsurg.2019.07.024 [DOI] [PubMed] [Google Scholar]

[zoi201089r17] 17.Binkovitz LA, Unsdorfer KM, Thapa P, et al. Pediatric appendiceal ultrasound: accuracy, determinacy and clinical outcomes. Pediatr Radiol. 2015;45(13):1934-1944. doi: 10.1007/s00247-015-3432-7 [DOI] [PubMed] [Google Scholar]

[zoi201089r18] 18.Schuh S, Chan K, Langer JC, et al. Properties of serial ultrasound clinical diagnostic pathway in suspected appendicitis and related computed tomography use. Acad Emerg Med. 2015;22(4):406-414. doi: 10.1111/acem.12631 [DOI] [PubMed] [Google Scholar]

[zoi201089r19] 19.Kotagal M, Richards MK, Flum DR, Acierno SP, Weinsheimer RL, Goldin AB. Use and accuracy of diagnostic imaging in the evaluation of pediatric appendicitis. J Pediatr Surg. 2015;50(4):642-646. doi: 10.1016/j.jpedsurg.2014.09.080 [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi201089r20] 20.McDermott KW, Stocks C, Freeman WJ Overview of pediatric emergency department visits, 2015: Statistical Brief #242. In: Healthcare Cost and Utilization Project (HCUP) Statistical Briefs. Agency for Healthcare Research and Quality; 2018. [PubMed] [Google Scholar]

[zoi201089r21] 21.Kharbanda AB, Vazquez-Benitez G, Ballard DW, et al. Development and validation of a novel pediatric appendicitis risk calculator (pARC). Pediatrics. 2018;141(4):e20172699. doi: 10.1542/peds.2017-2699 [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi201089r22] 22.Ekstrom HL, Kharbanda EO, Ballard DW, et al. Development of a clinical decision support system for pediatric abdominal pain in emergency department settings across two health systems within the HCSRN. EGEMS (Wash DC). 2019;7(1):15. doi: 10.5334/egems.282 [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi201089r23] 23.Vinson DR, Mark DG, Chettipally UK, et al. ; eSPEED Investigators of the KP CREST Network . Increasing safe outpatient management of emergency department patients with pulmonary embolism: a controlled pragmatic trial. Ann Intern Med. 2018;169(12):855-865. doi: 10.7326/M18-1206 [DOI] [PubMed] [Google Scholar]

[zoi201089r24] 24.Ballard DW, Vemula R, Chettipally UK, et al. ; KP CREST Network Investigators . Optimizing clinical decision support in the electronic health record: clinical characteristics associated with the use of a decision tool for disposition of ED patients with pulmonary embolism. Appl Clin Inform. 2016;7(3):883-898. doi: 10.4338/ACI-2016-05-RA-0073 [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi201089r25] 25.Simon LE, Rauchwerger AS, Chettipally UK, et al. Text message alerts to emergency physicians identifying potential study candidates increase clinical trial enrollment. J Am Med Inform Assoc. 2019;26(11):1360-1363. doi: 10.1093/jamia/ocz118 [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi201089r26] 26.HealthPartners. Total Care Relative Resource Value (TCRRVTM): a measurement approach to achieve the triple aim. Updated September 21, 2017. Accessed June 12, 2018. https://www.healthpartners.com/ucm/groups/public/@hp/@public/documents/documents/cntrb_039627.pdf

[zoi201089r27] 27.National Quality Forum NQF endorses resource use measures. Published January 31, 2012. Accessed June 12, 2018. http://www.qualityforum.org/News_And_Resources/Press_Releases/2012/NQF_Endorses_Resource_Use_Measures.aspx

[zoi201089r28] 28.Yan Z, Ip IK, Raja AS, Gupta A, Kosowsky JM, Khorasani R. Yield of CT pulmonary angiography in the emergency department when providers override evidence-based clinical decision support. Radiology. 2017;282(3):717-725. doi: 10.1148/radiol.2016151985 [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi201089r29] 29.Khairat S, Marc D, Crosby W, Al Sanousi A. Reasons for physicians not adopting clinical decision support systems: critical analysis. JMIR Med Inform. 2018;6(2):e24. doi: 10.2196/medinform.8912 [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi201089r30] 30.Young PJ, Nickson CP, Perner A. When should clinicians act on non-statistically significant results from clinical trials? JAMA. 2020;323(22):2256-2257. doi: 10.1001/jama.2020.3508 [DOI] [PubMed] [Google Scholar]

[zoi201089r31] 31.O’Connor SD, Sodickson AD, Ip IK, et al. Journal club: requiring clinical justification to override repeat imaging decision support: impact on CT use. AJR Am J Roentgenol. 2014;203(5):W482-90. doi: 10.2214/AJR.14.13017 [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi201089r32] 32.Raja AS, Ip IK, Dunne RM, Schuur JD, Mills AM, Khorasani R. Effects of performance feedback reports on adherence to evidence-based guidelines in use of CT for evaluation of pulmonary embolism in the emergency department: a randomized trial. AJR Am J Roentgenol. 2015;205(5):936-940. doi: 10.2214/AJR.15.14677 [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi201089r33] 33.Cabana MD, Rand CS, Powe NR, et al. Why don’t physicians follow clinical practice guidelines? a framework for improvement. JAMA. 1999;282(15):1458-1465. doi: 10.1001/jama.282.15.1458 [DOI] [PubMed] [Google Scholar]

[zoi201089r34] 34.Hess EP, Homme JL, Kharbanda AB, et al. Effect of the head computed tomography choice decision aid in parents of children with minor head trauma: a cluster randomized trial. JAMA Netw Open. 2018;1(5):e182430. doi: 10.1001/jamanetworkopen.2018.2430 [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi201089r35] 35.Davidson KW, Ye S, Mensah GA. Commentary: de-implementation science: a virtuous cycle of ceasing and desisting low-value care before implementing new high value care. Ethn Dis. 2017;27(4):463-468. doi: 10.18865/ed.27.4.463 [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi201089r36] 36.Norton WE, Chambers DA. Unpacking the complexities of de-implementing inappropriate health interventions. Implement Sci. 2020;15(1):2. doi: 10.1186/s13012-019-0960-9 [DOI] [PMC free article] [PubMed] [Google Scholar]

[zoi201089r37] 37.Kharbanda AB, Dudley NC, Bajaj L, et al. ; Pediatric Emergency Medicine Collaborative Research Committee of the American Academy of Pediatrics . Validation and refinement of a prediction rule to identify children at low risk for acute appendicitis. Arch Pediatr Adolesc Med. 2012;166(8):738-744. doi: 10.1001/archpediatrics.2012.490 [DOI] [PMC free article] [PubMed] [Google Scholar]

PERMALINK

Effect of Clinical Decision Support on Diagnostic Imaging for Pediatric Appendicitis

Anupam B Kharbanda, MD, MSc

Gabriela Vazquez-Benitez, PhD

Dustin W Ballard, MD, MBE

David R Vinson, MD

Uli K Chettipally, MD, MPH

Steven P Dehmer, PhD

Heidi Ekstrom, MA

Adina S Rauchwerger, MPH

Brianna McMichael, MPH, MSN

Dale M Cotton, MD

Mamata V Kene, MD, MPH

Laura E Simon, BA

Jingyi Zhu, PhD

E Margaret Warton, MPH

Patrick J O’Connor, MD, MPH

Elyse O Kharbanda, MD, MPH

Key Points

Question

Findings

Meaning

Abstract

Importance

Objective

Design, Setting, and Participants

Interventions

Main Outcomes and Measures

Results

Conclusions and Relevance

Trial Registration

Introduction

Methods

Design, Study Population, and Randomization

Description of Intervention

Description of UC

Outcomes

Statistical Analysis

Results

Figure. Flowchart of Patients Who Were Eligible, Excluded, and Enrolled by Intervention and Usual Care Group.

Table 1. Study Population Characteristics by Intervention Group and Study Phase.

Table 2. Appendicitis, Imaging Use, and Safety Outcomes During the Index Visit by Intervention Group and Study Phasea.

Primary Outcomes

Exploratory Analyses

Table 3. Imaging Use by pARC Risk Stratification, Study Group and Phase, at Kaiser Permanente Northern California Sites.

Safety, Secondary Outcomes, and Cost Analysis

Table 4. Costs of Care Within 14 Days After Index Emergency Department Visit by Study Group During Intervention Phasea.

Discussion

Limitations

Conclusions

References

Associated Data

Supplementary Materials

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases

Table 2. Appendicitis, Imaging Use, and Safety Outcomes During the Index Visit by Intervention Group and Study Phase^a.

Table 4. Costs of Care Within 14 Days After Index Emergency Department Visit by Study Group During Intervention Phase^a.