Scope and Impact of EHR integrated Clinical Decision Support in the Emergency Department: A Systematic Review

Abstract

Objective

As electronic health records evolve, integration of computerized clinical decision support (CDS) offers the promise of sorting, collecting, and presenting this information to improve patient care. We conducted a systematic review to examine the scope and impact of EHR-integrated CDS technologies implemented in the ED.

Methods

A literature search was conducted in four databases from their inception through January 18^th, 2018: PubMed, Scopus, CINAHL, Cochrane Central. Studies were included if they examined the effect of a decision support intervention which was implemented in a comprehensive electronic health record in the ED setting. Standardized data collection forms were developed and used to abstract study information and assess risk of bias.

Results:

2,558 potential studies were identified after removing duplicates. Of these, 42 met inclusion criteria. Common targets for CDS intervention included medication and radiology ordering practices, as well as more comprehensive systems supporting diagnosis and treatment for specific disease entities. The majority of studies (83%) reported positive effects on outcomes studied. Most studies (76%) employed a pre-post experimental design, with only three (7%) randomized control trials.

Conclusions:

Numerous studies suggest that CDS interventions are effective in changing physician practice with respect to process outcomes such as guideline adherence, however many studies are small and poorly controlled. Future studies should consider the inclusion of more specific information regarding design choices, attempt to improve on uncontrolled before-after designs, and focus on clinically relevant outcomes wherever possible.

Introduction:

Widespread adoption of Electronic Health Records (EHRs) offers an unprecedented opportunity to apply informatics techniques to clinical and operational data.^{1, 2} While the EHR stores a wealth of clinical data that can potentially improve the quality of clinical ED care,^1–4 this information is often lost in the sea of data collected in the EHR. In their brief patient encounters, emergency physicians are all too often confronted with poorly organized information, which is difficult to synthesize and act on at the bedside.^5–7 As EHRs evolve, integration of computerized clinical decision support (CDS) offers the promise of sorting, collecting, and presenting this information to improve patient care.^{8, 9} CDS technologies have demonstrated the ability to improve patient outcomes across a variety of healthcare settings.^{10, 11} CDS has been promoted by several organizations such as the Centers for Medicare and Medicaid Services (CMS), which will continue to increase implementation.¹²

Considering their promise to improve emergency care, along with regulatory incentives for their adoption, development and implementation of these technologies are proceeding rapidly. Given the complexity of the sociotechnical systems (such as ED) in which they are implemented,^{13, 14} however, CDS technologies have the potential for negative consequences.¹⁵ As we enter an era of increasing CDS development and use in the emergency department, it is crucial that implementation of these technologies be based on the best available clinical evidence indicating improvements in patient care.¹⁶ Furthermore, development of these technologies may benefit from design principles established in both health care and human factors engineering. These range from simple heuristics such as the “5 rights” of decision support (the right information, to the right person, in the right format, through the right channel, at the right time in the workflow)¹⁷ to more comprehensive theories of human factors engineering such as Parasuraman’s model of human interaction with automation.¹⁸

We conducted a systematic review to examine the scope and impact of EHR-integrated CDS technologies implemented in the ED. After reviewing the results of the systematic review, we discuss gaps in the current research and propose recommendations for future studies.

Methods:

This systemic review follows the PRISMA (Preferred Reporting Items for Systemic Reviews and Meta Analyses) guidelines.¹⁹ As a review of existing literature, this study was exempt from IRB review.

Definition of CDS

Consistent with previously published reviews of CDS in the general medical literature, we defined a CDS as any software designed to directly aid in clinical decision making in which characteristics of individual patients are matched to a computerized knowledge base for the purpose of generating patient-specific assessments or recommendations that are then presented to clinicians for consideration.^{10, 11}

Inclusion and Exclusion Criteria

Our literature review considered only peer-reviewed studies published in English. Studies were included if:

• 1They described an intervention which met the above definition of clinical decision support.
• 2They described the implementation of a CDS intervention (as opposed to pre-implementation testing).
• 3The intervention occurred in the ED setting.

During the selection process, we clarified our definition of CDS and added three inclusion criteria to narrow our sample to those studies most relevant to emergency medicine practitioners evaluating the effectiveness of decision support interventions:

• 4The CDS intervention was integrated within an existing EHR. This excluded studies in which tools were piloted without integration into existing workflows (e.g. standalone websites or computers requiring patient information to be entered for the sole purpose of generating a recommendation, or systems which may have fulfilled the role of an EHR but were only used for the purpose of intervention on specific patient populations).
• 5The study reported the impact of an intervention on a care process or patient related outcome (as opposed to outcomes of interest only to further development of the intervention). For instance, we excluded several studies which created an alert but only reported its “firing rate” or sensitivity and specificity, as these evaluated the development of an intervention as opposed to its implementation.
• 6We specifically excluded studies whose primary purpose was to perform automatic triaging of patients into Emergency Severity Index (ESI) or similar categories as these studies are more concerned with automating a care process than supporting a decision.

Search Strategy and Study Selection

The search was conducted in four databases from their inception through January 18^th, 2018: PubMed, Scopus, CINAHL, Cochrane Central. The search combined terms related to two domains: (1) CDS and (2) Emergency Department. Terms from different domains were combined using the Boolean operator ‘AND’; terms within each domain were combined using the Boolean operator ‘OR’. Table 1 contains the terms used in the search, and Appendix 1 contains the full search string. Our initial strategy included only the first three inclusion criteria. The title and abstract of each study were screened by a single reviewer with all possibly relevant studies retained for full text review. Full-text articles were then retrieved for studies retained after the initial screening. These studies were reviewed by two paired authors who excluded studies which clearly did not meet any of the initial three inclusion criteria (above). After the first round of full text review, studies were retained for further screening if either one or both reviewing authors flagged the study article for possible inclusion. At this point in the study, in response to discussions to obtain consensus when reviewing authors disagreed on inclusion, the team refined search criteria by adding selection criteria 4 through 6 (above). These criteria were added to specifically address areas which the team felt papers were not clearly included or excluded based on the original criteria. The remaining papers (any in which one of the two reviewers initially assigned voted to include or keep for further discussion) were then reviewed by two emergency physicians (BWP and MSP) who included only those studies which met all six inclusion criteria. In cases of disagreement between these two reviewers, cases were discussed with all study authors until consensus was reached.

Table 1:

Search terms included. Terms within columns were linked with an “OR” statement, with an “AND” statement between columns. See Appendix 1 for full search strategy

CDS	Emergency Department
diagnosis, computer assisted	emergency service, hospital
decision support systems, clinical	emergency medicine
decision-making, computer assisted	emergency department
decision aid	emergency room
decision models	emergency medicine
decision support
predictive Instrument
diagnostic aid predictive rule
decision rule
decision support techniques

Data Collection

We developed a data collection form which was pilot-tested on three papers and revised subsequently. Each researcher used the data collection form to extract the following data from each project: 1) Study Objective, 2) CDS intervention, 3) Study Design, and 4) Outcomes and Impact, which were further divided into clinical and process measures, and user experience (e.g. perceived usability, acceptance and use). Pairs of researchers reviewed each other’s data collection forms to ensure accurate abstraction.

Risk of Bias

Studies were assessed for risk of bias using questions selected from a bank proposed by Viswanathan et al.²⁰ Risk of bias scoring was completed during the data extraction phase, with the extracting author completing the risk of bias form, and the paired author reviewing the other’s form. Differences were discussed until consensus was achieved. In cases of when consensus could not be reached between the two authors, items were discussed with the full team.

Results:

After removing duplicates, a total of 2,558 potential studies were identified. Of these, we retained 296 for full text review after the initial round of abstract review. We excluded an additional 153 studies in the first round of full text review, leaving 143 studies. After addition of selection criteria 3 through 6, the final round of full text review excluded 99 additional papers, leaving 42 papers which were included in the systematic review reported in this paper. Among these 42 papers, there was a trend towards increasing volume by year (See Figure 2). We did not identify any studies published before 2005 which met our inclusion criteria.

Figure 2:

Included studies by year of publication. Of note, only 18 days of 2018 were included in the data collection

Characteristics of included studies are included in Table 2. While there were a broad range of CDS intervention types and targets described, several themes emerged. Twelve studies (29%), describe interventions in the form of alerts or other modifications to computerized physician order entry (CPOE) designed to support decision making surrounding medication ordering.^{21, 22} These studies were more prevalent earlier in the review period, with 7 of the 12 published in 2012 or earlier. These interventions generally involved order entry alerts and were designed to prevent administration of inappropriate medications (e.g. Beers list for older adults²³),²² dosing errors,²⁴ or ensure guideline compliant antibiotic ordering.²⁵ Another focus was CDS interventions which applied decision support at the time of radiology ordering with the goal of reducing unnecessary or inappropriate imaging for diagnostic workup of conditions such as pulmonary embolus^{26, 27} or intracranial hemorrhage²⁸; these studies accounted for 17 (40%) of the total papers included in the review. More comprehensive technologies supported several decisions along the diagnostic and treatment pathway to improve both detection and care for specific conditions including syncope,²⁹ sepsis,³⁰ asthma management,³¹ pneumonia,³² and appendicitis.^{33, 34}

Table 2:

Study characteristics, organized by year of publication

Author/Year	Study	Study Objective	CDS Intervention Implementation	Study Design (population, setting, comparison)	Reported Outcome and Impact
					Clinical and Process Measures	User Experience
Bernstein et al, 200521	An Electronic Chart Prompt to Decrease Proprietary Antibiotic Prescription to Self-Pay Patients	“To develop a clinical decision support system to display patient insurance status before prescription writing for outpatient conditions”	CDS: Synchronous decision support system with an on-screen prompt “to alert physicians to patients’ insurance status before prescribing antibiotics for patients to be discharged from the ED” Decision: Medications guideline adherence Implementation: Brief ED-based in-service, 30-minute didactic lecture, brief email reminder	Design: Pre-post intervention comparison in 13 weeks leading up to and immediately after implementation Setting: Urban ED with 78,000 visits/year serving a medically underserved population Clinicians: 61 prescribes during control and intervention periods: all attending emergency physicians, PAs, EM residents, medical and PA students, residents rotating from other services in the ED Patients: All ED patients aged 18 years or older discharged from the ED receiving antibiotic prescriptions during the 26-week study period: 543 during control and 514 during intervention	Proprietary antibiotics prescribed decreased: 26.6% to 20.7% (22% reduction; p=0.03) Preplanned subgroup analysis of self-pay patients with respiratory and urinary infections: proprietary antibiotics prescribed decreased 44.4% to 31.3% (30%reduction; p=0.005)	No data reported; but authors discussed the ‘apparent resistance to change’ from prescribers
Kirk et al, 200556	Computer Calculated Dose in Pediatric Prescribing	“To assess the rate of medication errors in predominantly ambulatory pediatric patient and the effect of computer calculated does in medication error rates of two commonly prescribed drugs”	CDS: Computer calculated dosing for acetaminophen or promethazine Decision: Medication appropriateness for pediatric patients Implementation: Department staff training session, reinforced via email.	Design: Prospective cohort between March 2003 (date of implementation) and August 2003 Setting: Outpatient clinic, pediatric ED, at National University Hospital, Singapore Clinicians: All prescribing physicians (84% pediatricians) Patients: 3,347 patients with 4,274 prescriptions analyzed	Medication error rate: 12.6% (299/2381) in the computer calculated dose system, 28.2% in the traditional method (534/1893) Overall error rate was 19.5% with most errors being a result of an underdose (64%) OR for error with CDS vs traditional method 0.436 in LR model controlling for drug, prescriber and patient characteristics	No assessment reported
Buising et al, 200837	Improving antibiotic prescribing for adults with community acquired pneumonia: Does a computerized decision support system achieve more than academic detailing alone? - a time series analysis	“To describe the impact of … both academic detailing (AD) and a computerized decision support system (CDSS) on the management of patients with CAP”	CDS: Web based design, integrated with hospital database to provide PSI to recommend inpatient vs. outpatient care and CURB score to recommend ICU need, as well as provided antibiotics recommendations. Decision: Treatment of CAP Implementation: Intervention implemented after an 8-month period of academic detailing where physicians received 1 on 1 training	Design: Two stage pre-post intervention cohort and time series analysis over 3 time periods: Baseline, April 2003 – March 2004. Academic Detailing, February 2005 - October 2005. CDS, April 2006 - September 2006 Setting: A single urban tertiary care ED with 50,000 visits/year Clinicians: 30 different emergency physicians, both senior and junior medical staff Patients: All ED patients aged 18 years or older with a diagnosis of pneumonia, chest infection, lower respiratory tract infection, pleuritic chest pain, cough, shortness of breath, and/or aspiration who met a pneumonia definition by case review. Baseline N = 392, Academic Detailing N = 215, CDS N = 133	The adjusted odds ratio for concordant therapy in the academic detailing period, compared with the baseline period was 2.79 [1.88,4.14], p < 0.01, and for the CDS period compared to the academic detailing period was 1.99 [1.07, 3.69], p = 0.02 With CDS an improvement in the appropriateness of antibiotic prescribing was demonstrated, which was greater than that expected to have occurred with time and academic detailing alone, based on predictions from a binary logistic model	No assessment reported
Sard et al, 200854	Retrospective evaluation of a computerized physician order entry adaptation to prevent prescribing errors in a pediatric emergency department	“To determine the impact on medication prescribing errors by adding a pediatric medication list (drug dosing support tool) into the CPOE”	CDS: Pediatric medication quick-list (drug dosing support tool) added into CPOE Decision: Medication appropriateness for pediatric patients Implementation: 2-hour training required to access the new system	Design: Pre-post intervention comparison Setting: Academic, urban, pediatric ED, volume ~30,000 visits/year Clinicians: Attending physicians, pediatric EM fellows, pediatric residents Patients: 840 randomly selected visits (420 pre-implementation, 420 post-implementation)	Medication errors/100 visit significantly decreased (24 pre to 13 post) Medication errors/100 orders significantly decreased (31 pre to 14 post)	No assessment reported
Terrell et al, 200922	Computerized decision support to reduce potentially inappropriate prescribing to older emergency department patients: A randomized, controlled trial	“To evaluate the effectiveness of computer-assisted decision support in reducing potentially inappropriate prescribing to older adults”	CDS: Alert issued when one of 9 preselected high-risk medications was provided as a discharge prescription Decision: Medication appropriateness for older adult patients Implementation: Physicians were randomized, with half receiving the alerts when applicable during the study period	Design: Randomized, controlled trial Setting: An academic [urban] emergency department (ED) in Indianapolis, Indiana, with 100,000 visits/year Clinicians: 63 emergency physicians [faculty and PG-2 & 3 residents] were randomized to the intervention (32 physicians) or control (31 physicians) group Patients: 5,162 analyzed visits by patients aged 65 or over, 210 of whom had attempts to provide a potentially inappropriate recommendation	Intervention physicians prescribed one or more inappropriate medications during 2.6% of ED visits by seniors, compared with 3.9% of visits managed by control physicians (P=.02; odds ratio=0.55,95% Cl 0.34–0.89)	No assessment reported
Melnick et al, 201029	Knowledge translation of the American College of Emergency Physicians’ clinical policy on syncope using computerized clinical decision support	“To identify the presence of a change in physician ordering of cranial imaging and admission practices due to implementation of a HPI-based CDSS in patients with a final diagnosis of syncope”	CDS: 3 item module added to the syncope documentation template including recommendations on imaging and admission and forcing documentation Decision: CT brain in syncope Implementation: Module announced via email	Design: Pre-post intervention comparison of consecutive patients presenting with syncope during June 2017 – February 2008 (pre) to February 2008 – June 2008 (post) Setting: Mount Sanai Medical Center (New York, NY), a 1,171-bed tertiary care academic medical center Clinicians: 34 emergency physicians supervising both EM and rotating residents Patients: 410 pre-implementation visits and 301 post-implementation	No significant difference in the head CT imaging rate (39.8% pre-intervention, 43.2% post-intervention, p=0.358) Significant decrease in admission rate (68.1% pre-intervention, 60.5% post-intervention, p=0.036)	No assessment reported
Terrell et al, 201038	Computerized decision support for medication dosing in renal insufficiency: A randomized, controlled trial	“To examine whether decision support in a computerized physician order entry system would reduce the rate of excessive medication dosing for patients with renal impairment”	CDS: CPOE alert providing dosing recommendations for targeted medications when the patient’s estimated creatinine clearance level was below the threshold for dosage adjustment Decision: Medication dosing Implementation: Physicians were randomized, with half receiving the alerts when applicable during the study period	Design: Randomized, controlled trial Setting: An academic [urban] emergency department (ED) in Indianapolis, Indiana, with 100,000 annual visits Clinicians: The sample included 42 physicians who were randomized to the intervention (21 physicians) or control (21 physicians) group; including emergency medicine faculty and year 2 & 3 residents Patients: 2,783 patients with sufficient data to estimate creatinine clearance, of whom 119 were written prescriptions for meds requiring potential adjustment	Decision support was provided 73 times to physicians in the intervention group, who excessively dosed 31 (43%) prescriptions. In comparison, control physicians excessively dosed a significantly larger proportion of medications: 34 of 46,74% (effect size = 31%; 95% confidence interval 14% to 49%; P= .001)	No assessment reported
Venkat et al, 201052	Feasibility of integrating a clinical decision support tool into an existing computerized physician order entry system to increase seasonal influenza vaccination in the emergency department	“We hypothesized that integration of clinical decision support into an existing ED CPOE system would allow large-scale patient screening and provision of seasonal influenza vaccination in the ED setting without added staffing resources”	CDS: Protocolized order for influenza vaccination screen which automatically fired at registration for patients who had no record of prior vaccination Decision: Screening for influenza vaccination Implementation: 15 minutes of training at staff meetings prior to initiation of the order set	Design: Pre-post intervention comparison of October 2009 (immediately post intervention) to October 2008 (pre-intervention) Setting: Urban, tertiary care center with an annual census of 48,000 visits/year, approximately 10% pediatric Clinicians: 59 nurses Patients: 3,900 pre-implementation visits and 3,091 post-implementation	ED influenza vaccination rose by 17.5% (95%CI16–19%, p<0.001)	No assessment reported
Carman et al, 201125	Use of a Clinical Decision Support Tool to Improve Guideline Adherence for the Treatment of Methicillin-Resistant Staphylococcus Aureus Skin and Soft Tissue Infections	“To examine whether development of a CDS tool based on current guidelines for the outpatient management of CAMRSA-related abscesses and interfaced with the documentation and order-entry process would improve prescriptive practice in an ED setting”	CDS: H&P template, abscess order-entry form, and prebuilt discharge pathway to guide prescriptions, follow-up, and discharge instructions specific to the care of MRSA abscesses Decision: Diagnosis and treatment of skin and soft tissue infections Implementation: Email and verbal presentation at department meeting	Design: Pre-post intervention comparison of a pre-implementation baseline to 6 and 12 weeks after intervention Setting: One level-one trauma center with separate adult and pediatric EDs, a smaller community hospital ED, and two freestanding EDs with 200,000 visits/year total Clinicians: Private group practice, with 71 board-certified emergency physicians and 30 mid-level providers Patients: All patients presenting to the ED with a chief complaint of abscess during the time periods. N= 863. Exclusions: animal bites and oral/dental infections	Adherence to recommended antibiotics therapy: 86.8% at baseline, 90.6% at week 6 and 96.7% at week 12 (p <0.001) Wound cultures: 34.6% at baseline, 7.0% at week 6 and 22.0% at week 12 (p=0.002) Chlorhexidine decontamination 0.0% at baseline, 5.0% at week 6 and 7.3% at week 12 (p =0.001)	Usability survey was generated at 6 and 12 weeks post intervention: low response rate (20% at week 6 and 6% at week 12). Median responses were positive for all usability questions
Drescher et al, 201153	Effectiveness and acceptability of a computerized decision support system using modified Wells’ criteria for evaluation of suspected pulmonary embolism	“To determine whether a validated prediction algorithm embedded in a computerized decision support system improves the positive yield rate of CT angiography for pulmonary embolism and is acceptable to emergency physicians”	CDS: Required physicians to enter information into a calculator for the dichotomized version of the Wells’ score, embedded in the CPOE and fired every time a physician ordered a D-dimer or CT-angiography Decision: CT for PE Implementation: Physicians briefed on intervention 1 month prior to implementation	Design: A prospective interventional study with a retrospective pre-interventional comparison group Setting: Academic community hospital ED Clinicians: 19 board-certified attending physicians without residents Patients: 205 pre-implementation visits, 229 post-implementation	Overall increase in the positivity rate of from 8.3% (95% confidence interval [Cl] 4.9% to 12.9%) pre-intervention to 12.7% (95% Cl 8.6% to 17.7%) post-intervention (higher yield)	Clinicians did not complete the algorithm or adhere to its recommendation for 105 cases (26.7%) Reasons given included time requirement and lack of belief that the CDS was helpful for guiding patient evaluations
Nelson et al, 201130	Prospective trial of real-time electronic surveillance to expedite early care of severe sepsis	“An automated, real-time electronic medical record query and caregiver notification system was developed and examined for its utility in improving sepsis care”	CDS: EMR surveillance algorithm ran for patients > 18 having ≥ 2 SIRS criteria and ≥ 2 BP measurements ≤ 90 mm Hg during their ED stay. Algorithm prompted notification by paging and text entry in EHR for fluid resuscitation, blood culture, and antibiotic administration Decision: Diagnosis and treatment of sepsis Implementation: Algorithm was run in background leading up to implementation	Design: Pre-post prospective study, comparing patients identified by the algorithm “dry firing” to a period of full implementation with physician notification Setting: One academic hospital ED with 68,000 visits/year Clinicians: No information Patients: 33,860 total patients screened of whom 398 (1.2%) met criteria	Two interventions had increased frequency after system activation: chest radiograph before admission (OR 3.2; 95% Cl 1.1 to 9.5) and collection of blood cultures (OR 2.9; 95% Cl 1.1 to 7.7) Non-significant increased frequency in lactate ordering or antibiotic administration Only blood cultures exhibited decrease in median time to performance: (pre-intervention 86 min, IQR 31, 296 min; post intervention 81 min, IQR37,245min;P=.032)	No assessment reported
Griffey et al, 201249	Guided medication dosing for elderly emergency patients using real-time, computerized decision support	“To evaluate the impact of a real-time computerized decision support tool in the emergency department that guides medication dosing for the elderly on physician ordering behavior and on adverse drug events (ADEs)”	CDS: Alert providing age-adjusted recommendations, e.g., medication, dosage or frequency. Fires in CPOE for patients older than 65 for pre-specified medication orders Decision: Medication appropriateness for older adult patients Implementation: Not discussed	Design: Prospective controlled trial over 4 consecutive periods consisting of control periods alternating with intervention periods (6 to 7-week blocks) Setting: One urban, academic ED Clinicians: ED residents, off-service Residents, and rotating residents Patients: 2,398 orders among 1,407 patients: 668 control and 739 intervention	Greater % of orders consistent with recommendations during ON periods (31%) as compared to OFF periods (23%) (p<0.001) Physicians declined the recommendation in 93% of suggestions during ON periods No differences in admission rate, reversal drug administration, number of 10-fold orders, or ED LOS 39 ADEs during ON periods and 31 during OFF periods (p=0.02)	No assessment reported
Raja et al, 201227	Effect of Computerized Clinical Decision Support on the Use and Yield of CT Pulmonary Angiography in the Emergency Department	“To determine the effect of evidence-based CDS on the use and yield of CT pulmonary angiography for acute pulmonary embolism (PE) in the emergency department ED”	CDS: Addition to CPOE system for ordering CT angiography of chest. Required physicians to enter D dimer (Not done, normal or elevated) and clinical suspicion of PE (low, intermediate or high), then provided nonbinding advice regarding need for CT imaging Decision: CT for PE Implementation: Rollout accompanied by significant educational efforts	Design: Pre-post intervention evaluation in a single ED from 2003–2009, intervention in 2007 Setting: 793-bed quaternary care center with 60,000 ED visits/year Clinicians: All prescribing clinicians Patients: 338,230 patients seen during study period, of whom 6,838 received pulmonary CTA	20.1% decrease in quarterly use of CTA per 1000 patients 69% increase in yield of CTA studies	No assessment reported
Dexheimer et al, 201339	Modifications and Integration of the Electronic Tracking Board in a Pediatric Emergency Department	“To describe the modifications and decisions made in the implementation of a computerized tracking board in a pediatric ED”	CDS: Tracking board timer that alerts nurses 30 minutes after medication administration to reassess pain levels Decision: Pain reassessment Implementation: Not discussed	Design: Pre-post prospective study Setting: Freestanding children hospital with 577 beds and more than 120,000 ED visits/year Clinicians: Physicians, nurses and patient care attendants Patients: No information	After the inception of a screen icon and alert for pain reassessment, pain reassessments increased from 50–65% of patients to more than 85%	No assessment reported
Prevedello et al, 201357	Does Clinical Decision Support Reduce Unwarranted Variation in Yield of CT Pulmonary Angiogram?	“To determine whether previously documented effects of clinical decision support on computed tomography for pulmonary embolism in the emergency department (i.e., decrease use and increase yield) are due to a decrease in unwarranted variation”	CDS: Addition to CPOE system for ordering CT angiography of chest. Required physicians to enter D dimer (Not done, normal or elevated) and clinical suspicion of PE (low, intermediate or high), then provided nonbinding advice regarding need for CT imaging Decision: CT for PE Implementation: Rollout accompanied by significant educational efforts	Design: Retrospective pre-post evaluation conducted from January 2006 – March 2009, in the emergency department of a 793-bed, quaternary care-academic hospital (adult-only emergency department, Level 1 Trauma and Burn Center, and Stroke and Cerebrovascular Disease Center) Setting: 793-bed quaternary care center with 60,000 ED visits/year Clinicians: All prescribing clinicians Patients: 113,703 ED visits with 2,891 (2.5%) rate of CTA for PE	PE-CT decreased from 26.5 (pre) to 24.3 (post) CT scans/1000 patients Overall yield of CT scans increased from 9.2% to 12.6%, a 3.4% increase post-CDS Inter-physician variability in yield ranged 2.6%–20.5% (pre) and 0%–38.1% (post) Intra-physician variability in yield ranged from a 9.1% decrease to a 21.0% increase in yield of PE—CT; was significant in 3 out of 25 physicians (the intra-physician variability was related to improved yield)	No assessment reported
Demonchy et al, 201458	Impact of a computerized decision support system on compliance with guidelines on antibiotics prescribed for urinary tract infections in emergency departments	“To assess the impact of a CDSS on compliance with guidelines on empirical antibiotic prescriptions for UTIs in EDs”	CDS: Triggered when a UTI diagnosis was validated and launched a tool to collect limited additional data from popup screens. It provided recommendations regarding the investigations, the indications for hospitalization, the antibiotic treatment and the follow-up, tailored to the individual patient data Decision: Diagnosis and treatment of UTI Implementation: Intervention was preceded by passive recommendations at a single site	Design: Pre-post prospective study in 3 French EDs from March 2012 – October 2012. Intervention was rolled out over 3 periods in 3 centers, with one center implementing a passive recommendation screen between baseline and CDS Setting: 3 academic, French EDs Clinicians: Junior and senior physicians as well as rotating residents Patients: All patients aged 15 years or older and diagnosed with community-acquired UTI (N= 912); exclusions from the study if data were missing regarding the diagnosis and/or the antibiotic prescription	The CDSS led to a modification of the initial diagnosis in 23% (42/182) CDSS was not associated with significant increase in appropriate antibiotics or duration in multivariate analysis CDS used in 59% of cases	Usability was rated as good by all clinicians, with the tool considered to be user-friendly and not time-consuming. Most clinicians found the CDSS to be useful in their daily practice
Dexheimer et al, 201431	Implementation and evaluation of an integrated computerized asthma management system in a pediatric ED: a RCT	“To determine whether patient eligibility identification by a probabilistic disease detection system (Bayesian net-work) combined with an asthma management system embedded in the workflow decreases time to disposition decision”	CDS: Asthma management system with 2 components: (1) automatic disease detection system (ADDS) and (2) management system Decision: Asthma management Implementation: 2 months of email and live announcements	Design: Prospective, randomized controlled trial comparing CDS to a paper-based protocol Setting: Urban tertiary care pediatric ED with 55,000 visits/year Clinicians: 68 ED attending physicians, 95 nurses, and 16 respiratory therapists Patients: Intervention group (n=398) was compared to control group (n=390)	Primary outcome measure was the time from triage to disposition decision. Intervention and control patients did not differ significantly in time to disposition Secondary outcome was adherence to guidelines for asthma care. There were no significant differences between intervention and control groups	No assessment reported
Fowler et al, 201459	Electronic health record: integrating evidence-based information at the point of clinical decision making	“To(1) provide a straightforward mechanism for physicians to review alternative diagnoses prior to finalizing a treatment plan and (2) dramatically improve the ease of access to relevant evidence-based information resources without disrupting established workflows”	CDS: Diagnostic decision support tool (DDST) and a web-based knowledge page (KP). Based on age, gender, chief complaint, triage impression, positive findings and history of present illness, the DDST makes suggestions for possible diagnosis. Clicking on a possible diagnosis provides the provider with information from different sources (the KP) Decision: General diagnosis Implementation: Presentation about the tools were given to physicians before, during and after the study	Design: Uncontrolled pilot, intervention available optionally in EHR for 6 months Setting: 250-bed pediatric teaching hospital. Focus groups and survey used to generate results Clinicians: 200 pediatric ED clinicians (attending physicians, fellows, residents) Patients: 34,000 potential visits, DDST used 167 times (0.03%)	Only 2 physicians responded to survey, both stated DDST influenced their DDx.	Data collected during focus groups showed that physicians in the ED would prefer access to KP without the DDST
Gupta et al, 201435	Effect of clinical decision support on documented guideline adherence for head CT in emergency department patients with mild traumatic brain injury	“To determine the impact of an electronic CDS tool, based on validated evidence, designed to guide emergency clinician decision making for use of head CT for patients with MTBI”	CDS: Electronic CDS triggered in CPOE when a head CT for trauma was ordered; clinician prompted to enter justifying information and given recommendation on appropriateness of proceeding with CT Decision: CT for head trauma Implementation: Not discussed	Design: Pre-post prospective study; 27 months before and 27 months after CDS implementation Setting: ED of a 793-bed, urban, academic level 1 trauma center Clinicians: No information Patients: Random sample of 200 head CTs on patients with MTBI from each of baseline and intervention periods	Documented guideline adherence: from 49% to 76.5% (p<.001); absolute increase of 27.5%	No assessment reported
Raja et al, 201436	The use of decision support to measure documented adherence to a national imaging quality measure	“To determine feasibility of measuring adherence to national quality measure regarding CTPA for PE using CDS and whether this would affect use and yield of CTs”	CDS: Wells’ score criteria based CDS within CPOE, required answering yes/no to each Wells’ element and indicating result of D-Dimer (if ordered). CDS messages for non-adherent orders Decision: CT for PE Implementation: Updated an existing CDS	Design: Pre-post prospective study from September 2009 – November 2011, intervention October 2010 Setting: 776 bed quaternary care center, ED volume ~60,000 visits/year Clinicians: Attending physicians and residents; all attendings EM boarded Patients: 1,209 patients with suspected PE in pre-intervention, 1212 in post-intervention	Documented adherence to NQM improved (control 56.9% vs intervention 75.6%, p<.008) No change in CT utilization or yield	No assessment reported
Carnevale et al, 201560	Impact Of An Emergency Medicine Decision Support And Risk Education System On Computed Tomography And Magnetic Resonance Imaging Use	“To assess the impact of a multimodal emergency medicine decision support and risk education system (DS-RES) on the use of CT and MRI imaging”	CDS: Point of care documentation templates selected by chief complaint which collect data and drive recommendations and alerts on imaging, orders, and documentation Decision: General imaging guidelines Implementation: Not discussed	Design: Pre-post prospective study with intervention availably July 2009. Pre period was July 2008 – June 2009, post was October 2009 – September 2010. Data were adjusted for age, complaint, and “risk” status using a DxCG risk score Setting: “A Kaiser Permanente Northwest Medical Center” Clinicians: No information Patients: 18,105 pre-intervention visits, 20,892 post-intervention. Visits were included for 11 specific chief complaints, among only discharged patients	Physicians used the tool in 81.6% of encounters. Overall rate of CT/MRI imaging increased from 26.0% to 28.3% (P<.0001) In low-risk group, CT/MRI use decreased by 6.2% (95% Cl 4.0%–8.5%). In medium risk, decreased 3.3% (95% Cl 1.6%–5.0). In high-risk group, imaging increased by 5.6% (95% Cl 4.3%–7.0%) Proportion of patients with a 3-day re-visit to either the ED or inpatient facility decreased by 1.4% (95% Cl 0.7%–2.2%) from 9.7% to 8.3%. 7-day revisits decreased by 0.7% (95% CI-0.2% to 1.5%), from 13.9% to 13.2%	No assessment reported
Dean et al, 201532	Impact of an Electronic Clinical Decision Support Tool for Emergency Department Patients With Pneumonia	“To assess the effect of [CDS] tool deployment on 30-day all-cause mortality and patient disposition before and 6 months after tool deployment in 4 intervention hospital EDs compared with 3 usual care hospital EDs”	CDS: Alert to identify ED patients with high risk of pneumonia and recommends evidence-based management. Automatically calculated objective severity assessment and management recommendations, including disposition, diagnostic testing, and antibiotic selection Decision: Diagnosis and treatment of PNA Implementation: No discussion	Design: Pre-post prospective study from baseline December 2009 – November 2010 followed by post implementation December 2011 - November 2012 in 4 EDs, with comparison to 3 non-intervention EDs in study Setting: 7 EDs, 19,500–86,400 visits/year Clinicians: No information Patients: 4,758 patients aged 18 or over who met criteria of ICD-9 diagnosis of PNA and had chest imaging performed consistent with PNA	Concordance with admission recommendation increased significantly, from 79% to 84% post-deployment in intervention group Time to antibiotics decreased significantly among CAP patients in intervention group No difference in mortality adjusted or unadjusted among all PNA patients CAP patients treated in intervention EDs had a lower 30-day Mortality post deployment OR 0.53 (0.28–0.99)	CDS was used in 62.6% of patients in the intervention group
Faine et al, 201561	Importance of Decision Support Implementation in Emergency Department Vancomycin Dosing	“To assess the impact of an EMR intervention on vancomycin dosing accuracy in critically ill ED patients. Secondary objectives were to assess the impact of vancomycin dosing on mortality, hospital length of stay, acute kidney injury, and the impact of obesity on vancomycin dosing accuracy”	CDS: Weight-based vancomycin dosing added to (CPOE). Decision: Medication Dosage (Vancomycin) Implementation: Email notification and live presentation prior to rollout	Design: Retrospective pre-post cohort study of all patients (n=278) treated with vancomycin and admitted to an intensive care unit. Pre period March 2008 – May 2009, post November 2009 – April 2011 Setting: 60,000 visits/year Midwestern academic ED (March 2008 and April 2011) and admitted to an intensive care unit Clinicians: ED Physicians Patients: Pre group (n=100) was compared with intervention group (n=178)	The primary outcome was the proportion of vancomycin doses defined as “appropriate” based on recorded actual body weight”. The EMR dose calculation tool was associated with an increase in mean vancomycin dose ([14.1±5.0] vs. [16.5±5.7] mg/kg, p<0.001) and a 10.3% absolute improvement in first-dose appropriateness (34.3% vs. 24.0%, p=0.07) Secondary outcomes included 28-day in-hospital mortality, hospital length of stay and acute kidney injury (safety outcome). None of the differences between pre- and post- measurements was statistically significant	No assessment reported
Ip et al, 201528	Impact of clinical decision support on head computed tomography use in patients with mild traumatic brain injury in the ED	“To examine the impact of real-time computerized clinical decision support (CDS), based on published high-quality evidence, on the use of head CT in adult ED patients diagnosed with MTBI”	CDS: CDS triggered in CPOE when a head CT for trauma was ordered; clinician prompted to enter justifying information and given recommendation on appropriateness of proceeding with CT Decision: CT brain for trauma Implementation: Not discussed	Design: Pre-post prospective study, included all adult EDs for all ED patients with a discharge diagnosis of MTBI between January 2009 and December 2010; direct comparison to a control cohort of ED patients diagnosed with MTBI from the National Hospital Medical Care Survey (NHAMCS) during study period Setting: 793-bed, level-one trauma center Clinicians: ED physicians Patients: 116,009 unique ED visits at study site and 53,477 visits in control cohort: MTBI represented 1.2% of all ED visits	Study ED: MTBI ED visits associated with head CT for 58.1% in pre and 50.3% in post (p=0.005); relative decrease of 13.4% Control cohort: use of head CT did not change: 73.3% in pre and 76.9% in post Decrease in head CT use persisted after accounting for baseline demographic differences No significant changes in rate of delayed imaging or rate of delayed diagnosis of radiologically significant findings	No assessment reported
Sethuraman et al, 201562	Prescription errors before and after introduction of electronic medication alert system in a pediatric emergency department	“To compare prescription errors rates before and after introduction of CPOE with electronic medication alert system in a PED”	CDS: CPOE with electronic medication alert system (EMAS) (Cerner), alerts provider to drug allergies, dose range checked, drug-drug interactions, and drug frequency errors for outpatient prescriptions Decision: Medication appropriateness Implementation: Not discussed	Design: Pre-post prospective study, implementation in October 2010. Pre period January 2010 – May 2010, post January 2011 – May 2011 Setting: Free standing PED/children’s hospital, inner city, level-one trauma center ~92,000 visits/year Clinicians: Children’s ED providers Patients: Randomized selection of 5,000 patients in both pre and post groups, generating a total of 7,268 prescriptions in pre-intervention, and 7292 in post-intervention	Significant reduction in overall error (10.4/100 prescriptions pre vs 7.3/100 prescriptions post) No reduction in rate of prescriptions with serious errors EMAS was 45.1% sensitivity and 57% specific for medication errors 32% of alerts were false positives	No assessment reported
Stevens et al, 201563	Enhancing the quality of prescribing practices of older veterans discharged from the emergency department (EQUiPPED	“To decrease ordering of potentially inappropriate medications to <5% for veterans ≥ 65 years old”	CDS: Geriatric outpatient pharmacy order sets grouped according to common discharge diagnoses with medication options preferred for use in older adults. Dose adjustments for renal impairment, point-of-prescribing education regarding medications to avoid, and links to synthesized geriatric content were embedded within the order sets Decision: Medication Implementation: Concurrent education campaign with lectures, journal club, and reminder cards, academic detailing and individual provider feedback	Design: Pre-post prospective study. 8 months pre-intervention April 2012 – November 2012 compared with the following 17 months. Intervention was rolled out incrementally, with CDS intervention in February 2013. Data analyzed using Poisson regression and piecewise nonlinear regression Setting: One large, urban VAMC ED with 41,000 ED visits/year (40% veterans ≥ 65 years old) Clinicians: 13 physicians, 2 APPs & 1 clinical pharmacist (permanent staff) participated [study excluded > 40 “moonlighters”] Patients: 4,042 total patients aged 65 or older who were treated and discharged	In the 8 months before the intervention, the average monthly proportion of PIMs prescribed was 9.4 +/− 1.5%. This declined to 4.6 +/− 1.0% after the interventions In the top 5 medications or categories, significant reductions were noted in 3 (muscle relaxants, promethazine, chronic NSAIDs) The observed change was sustained for the final year of the evaluation period	Feedback from providers was important in developing order sets that facilitated workflow and were not perceived as burdensome
Kharbanda et al, 201633	Implementation of Electronic Clinical Decision Support for Pediatric Appendicitis	“To standardize care and reduce CT use while maintaining patient safety through implementation of a multicomponent electronic clinical decision support tool for pediatric patients with possible appendicitis”	CDS: 3 component tool: a standardized abdominal pain order-set, a Web-based risk stratification tool, and a “time of ordering alert” Decision: Radiology Implementation: CDS was introduced with “brief (20 minute) group and individual training sessions over a 4-week time period (October 2011)”. “During the implementation period, group quarterly e-mails were sent to ED providers describing the guidelines and risk stratification approach.”	Design: Interrupted time series for analysis Quasi-experimental study of children aged 3 to 18 years who presented with possible appendicitis to 2 urban, tertiary care pediatric EDs between January 2011 and December 2013. Pre-intervention period was 10-months prior to implementation Setting: 2 urban tertiary care pediatric EDs with collective census of 95,000 visits/year Clinicians: 47 PEM physicians, 8 NP’s, 9 PEM fellows Patients: 2,803 patients (767 pre-implementation; 2,085 post-implementation; excluded 49 subjects from week of CDS implementation)	54% relative decrease in CT use from pre-implementation to the end of the study (38.8% to 17.7%; p=,007) No significant change in US or total imaging trends No statistically significant differences in rates of missed appendicitis, ED revisits with 30 days, appendiceal perforation, and ED length of stay	No assessment reported
McGuire et al, 201651	Using a configurable EMR and decision support tools to promote process integration for routine HIV screening in the emergency department	“To initiate routine, opt-out HIV screening in Maricopa Medical Center’s adult ED for patients aged 18–64 years who were having labs drawn via venipuncture as part of their care”	CDS: Integrated elements specific to HIV screening into the triage/intake process, used resulting data to drive new screening protocols Decision: HIV Screening Implementation: Not discussed	Design: Pre-post prospective study over 3-year period, July 2011 – June 2014 with implementation at Q3 2012 Setting: Maricopa Medical Center, not further characterized Clinicians: ED triage nurses Patients: 130,761 total encounters, of whom 33,683 were tested for HIV	Overall increase in tested patients (67% in the first year, 97% in the second year, 98% in the third year) and decrease in missed opportunities (dwindled to fewer than 5% of eligible patients) HIV positivity finding rate of 0.27% (3 times the rate recommended by the CDC for implementing routine testing)	No assessment reported
Silveira et al, 201664	Impact of a clinical decision support tool on adherence to the Ottawa Ankle Rules.	“To determine the impact of CDS tool on documented adherence to the Ottawa Ankle Rules and yield (significant fracture rate) of ankle/foot radiography”	CDS: Ottawa Ankle Rule based CDS embedded into the CPOE Decision: Radiographs for ankle injury Implementation: Not discussed	Design: Pre-post prospective study, 6 months pre and 8 months post intervention in February 2013 Setting: 793 bed quaternary care academic hospital Clinicians: ED clinicians Patients: Adults with ankle injury: 205 pre-intervention, 255 post-intervention	Documented OAR adherence significantly increased from 55.9% to 95.7% Utilization and yield of X-rays did not change	No assessment reported
Austrian et al, 201750	Impact of an emergency department electronic sepsis surveillance system on patient mortality and length of stay.	“To determine whether an electronic health record-based sepsis alert system could improve quality of care and clinical outcomes for patients with sepsis”	CDS: Sepsis alert triggered by abnormal vital signs or laboratory results Decision: Laboratory evaluation for sepsis Implementation: ED physician and nurse sepsis champions provided in-service instruction for staff on the alert	Design: Retrospective cohort study of pre and post intervention of CDS from January 2013 – April 2015. Intervention implemented February 2014 Setting: EDs and urgent care clinics of one urban, academic health system in the USA Clinicians: All physicians and nurses working in the above centers Patients: All adult patients who were subsequently hospitalized with discharge diagnosis of sepsis listed as present on admission	Mean LOS for patients with sepsis decreased from 10.1 to 8.6 days (P<.001) following alert introduction The sepsis alert system had no effect on mortality or other clinical or process measures	No data reported, but discussion of alert fatigue
Baird et al, 201724	Impact of Developing Adult Ketamine Order Panels for the Emergency Department.	“To evaluate the impact of standardizing the ordering process with the intent of increasing appropriate dosing of ketamine in the ED”	CDS: Standardized order set providing ketamine orders with dosage recommendations based on weight and indication Decision: Medication dosage for ketamine Implementation: Provider education at a monthly meeting, and reference materials on EMR login screen	Design: Pre-post intervention comparison; retrospective control collected November 2013 – October 2015 with prospective intervention group collected March 2016 – May 2016 Setting: Multi-institutional analysis of ED use in a large health system Clinicians: ED providers Patients: 240 patients pre-intervention, 33 post-intervention identified as receiving ketamine in the ED	Pre-intervention: 63.8% appropriate dosing, post intervention 81.8%, p = 0.231 by Mantel-Haenszel test 6 emergence reactions documented in pre-phase, 0 in post	No assessment reported
Bookman et al, 201765	Embedded Clinical Decision Support in Electronic Health Record Decreases Use of High-cost Imaging in the Emergency Department: EmbED study	“To evaluate the impact of evidence-based clinical decision support tools integrated directly into provider workflow in the electronic health record on utilization of computed tomography (CT) brain, C-spine, and pulmonary embolism (PE)”	CDS: Alert that fires for specific CT orders that brings a clinical decision tool to the provider, with fields auto populated from the chart to determine appropriateness of study Decision: CT imaging Implementation: Education about the intervention given to all providers by site leadership just prior to insertion of the scoring tools	Design: 18-month longitudinal study comparing pre-implementation from July – December 2014 to post-implementation of July – December 2015 Setting: 5 EDs across a large integrated healthcare system utilizing a common EHR. The healthcare system consists of 1 academic and 4 community sites Clinicians: 163 attending providers across all 5 sites Patients: All adult ED patients with CT Brain, CT Cervical Spine, or CT PE ordered	Decrease of 6,106 CT scan ordering for the three studies compared pre and post implementation. Across all sites, there was greater than 6% decrease in utilization of CT brain and CT C-spine. The use of CT PE also decreased but was not significant For all CT types, high utilizers in the pre-intervention period decreased usage over 14% in the post-intervention period with CT brain, p < 0.001, CT C-spine p = 0.001, and CT PE p < 0.001	No assessment reported
Dayan et al, 201766	Use of Traumatic Brain Injury Prediction Rules With Clinical Decision Support	“To determine whether implementing the Pediatric Emergency Care Applied Research Network (PECARN) traumatic brain injury (TBI) prediction rules and providing risks of clinically important TBIs (ciTBIs) with computerized clinical decision support (CDS) reduces computed tomography (CT) use for children with minor head trauma”	CDS: CT recommendations and risks of ciTBI, both for patients at very low risk of ciTBI (no Pediatric Emergency Care Applied Research Network rule factors) and those not at very low risk Decision: CT head imaging for trauma Implementation: 1-time education consisted of a standard presentation focused on the CDS components and how to navigate to its different components	Design: Non-randomized multicenter clinical trial with concordant controls Setting: 5 pediatric emergency departments (PEDs) and 8 general EDs (GEDs) between November 2011 – June 2014 Clinicians: Attending faculty and physician fellows at each of the clinical sites Patients: patients <18 years of age and experienced minor blunt head trauma, defined by Glasgow Coma Scale (GCS) scores of 14 to 15, within 24 hours of ED presentation	CT rates decreased significantly (P < .05) but modestly (2.3%–3.7%) at 2 of 4 intervention PEDs for children at very low risk The other 2 PEDs had small (0.8%–1.5%) non-significant decreases. CT rates did not decrease consistently at the intervention GEDs, with low baseline CT rates (2.1%–4.0%) in those at very low risk For all children with minor head trauma, intervention sites had small decreases in CT rates (1.7%–6.2%)	No assessment reported
Drescher et al, 201767	Knowledge translation of the PERC rule for suspected pulmonary embolism: A blueprint for reducing the number of CT pulmonary angiograms.	Whether “the adoption of a department-endorsed, evidence-based clinical protocol, a multi-modal educational program, and CDS embedded in our computerized order entry system, would lead to a decrease in the number of CTPA ordered”	CDS: Providers required to enter clinical information into diagnostic pathway prior to ordering CT imaging for PE. PERC and Wells’ scores were then calculated and provided a recommended diagnostic pathway Decision: CT imaging for evaluation of PE Implementation: Education was provided to ordering providers about the clinical rules and how to complete the ordering process at department meetings and grand rounds. Post implementation, providers who ordered CTPA outside the protocol parameters emailed regarding their rationale. A quarterly utilization report was sent to all providers	Design: Pre-post observational study in which data were collected prospectively over 12 months starting October 2012 and compared to the previous 12 months Setting: One urban tertiary referral center with an emergency medicine (EM) residency program and with a census of 100,000 visits/year Clinicians: 17 clinical EM faculty and 36 EM residents Patients: All non-pregnant patients aged 18 years or older in whom the providers suspected a diagnosis of PE and who did not have a contraindication to CTPA, including renal insufficiency and allergy to contrast material.	CTPA declined from 1,033 scans for 98,028 annual visits (10.53 per 1,000 patient visits (95% Cl [9.9–11.2]) to 892 scans for 101,172 annual visits (8.81 per 1,000 patient visits (95% Cl [8.3–9.4]) p<0.001. The absolute reduction in PACT ordered was 1.72 per 1,000 visits (a 16% reduction)	No assessment reported
Hendrickson et al, 201734	Implementation of an electronic clinical decision support tool for pediatric appendicitis within a hospital network	“To assess the utility of an electronic health record (EHR)-based clinical decision support system (E-CDS) to optimize the care of children with acute abdominal pain in a general ED in a community hospital”	CDS: Guideline content built into a CDS order set in the EHR. At time of initial order placement, 3 steps: (1) common orders for initial evaluation, (2) secondary consideration of imaging after lab info (absolute neutrophil count) was available, (3) additional orders based on results of workup Decision: Evaluation and imaging for suspected appendicitis Implementation: Introduction about CDS in grand rounds; informal training sessions to clinicians	Design: Pre- (3 months) and post-intervention (6 months after) retrospective analysis of the use of EHR-based CDS order set for pediatric ED patients with suspected appendicitis Setting: Urban tertiary care children’s hospital with 11,000 pediatric ED visits, also suburban community hospital with about 12,000 children visits to ED Clinicians: No information Patients: 327 eligible visits; age 3–18 with chief complaint of abdominal pain	In community ED: among patients with any imaging, significant increase in use of ultrasound (36–51%, p<.05) and decrease in use of CT (81–66%, p<.05) and a near significant increase in use of ultrasound among all patients (26–38%, p=.055) At children’s hospital: no change; use of ultrasound was already high relative to CT Logistic regression done to adjust for gender difference between pre and post: attenuation of CDS effect, leading to marginally non-significant differences No negative impact on care (e.g., missed appendicitis)	No assessment reported
Martin Rico et al, 201768	Electronic alerting and decision support for early sepsis detection and management: Impact on clinical outcomes	“To assess the validity of an electronic alert for identifying severe sepsis and septic shock”	CDS: EHR alert for the possibility of severe sepsis and septic shock, and direct access to order panel Decision: Treatment for severe sepsis and septic shock Implementation: Training during first 2 months of implementation of alert of all professionals working the ED	Design: Pre-post intervention comparison. Pre-implementation from January 2011 – April 2013 and post-implementation from April 2013 – December 2013 Setting: One international site with 54,000 emergency encounters and 10,000 admissions annually Clinicians: All physicians working in ED at single site Patients: Patients aged 14 or older, with discharge diagnosis of ICD-9 codes related to sepsis	Compared to pre-implementation, mortality rate decreased by 28% and adjusted mortality risk decreased by 36%	No assessment reported
Mills et al, 201769	Clinical decision support increases diagnostic yield of computed tomography for suspected pulmonary embolism	“To determine effects of evidence-based clinical decision support (CDS) on the use and yield of computed tomographic pulmonary angiography for suspected pulmonary embolism (CTPE) in Emergency Department (ED) patients”	CDS: Provider inputs if D-dimer done and level, as well as Wells’ criteria information, and CDS alert displayed with recommendation of obtaining d-dimer vs CT-imaging Decision: CT imaging for PE Implementation: Implemented along with leadership focus on reducing inappropriate imaging	Design: Multi-site, single health care system prospective quality improvement intervention using a pre/post design of clinical decision support at the time of CTPE order Setting: 2 community and 1 academic ED, all associated with a single health system Clinicians: Physicians at the community site and attending physicians and residents at the academic site Patients: ED patients older than 18 with order for CT-PE. 34.7% of patients presented pre- and 65.3% post-CDS implementation	Yield increased a relative 30.8% after CDS implementation (8.1% vs. 10.6%; p=0.0003) There was no statistically significant change in CTPE utilization (1.4% pre- vs. 1.4% post-implementation; p = 0.25)	No data reported, but discussion of alert fatigue and “gaming” of CDS
Min et al, 201770	Clinical Decision Support Decreases Volume of Imaging for Low Back Pain in an Urban Emergency Department	“To determine whether point-of-care clinical decision support can effectively reduce inappropriate medical imaging of patients who present to the emergency department (ED) with low-back pain (LBP)”	CDS: A point-of-care checklist of accepted red flags for LBP was embedded in the existing order entry form for lumbar imaging Decision: Imaging for back pain Implementation: Communication regarding process changes for LBP diagnostic imaging requests and supplementary education material was delivered to physicians	Design: ED electronic health records from January 2013 – May 2016, were examined, with implementation March 2015. Retrospective extraction of data Setting: One major acute care and teaching center in Vancouver, British Columbia, Canada Clinicians: 43 emergency physicians at a major acute care center Patients: 4,562 patients with a diagnosis related to lower back pain	Proportion of LBP patients with an imaging order fell significantly (median: 22% to 17%; mean: 23% to 18%; P ¼ .0002) compared with pre-intervention baseline The percentage of patients without imaging who were later imaged at a hospital outpatient clinic within 30 days was 2.3% before intervention and 2.2% after (P ¼ .974) In addition, the proportion of patients discharged from the ED without imaging who returned to the ED within 30 days was 8.2% before intervention and 6.9% after (P¼.170)	No assessment reported
Nicholson et al, 201771	The Use of a Computerized Provider Order Entry Alert to Decrease Rates of Clostridium difficile Testing in Young Pediatric Patients	“To design a computerized provider order entry (CPOE) alert to decrease testing for C. difficile in young children and infants”	CDS: Computerized provider order entry alert based on age advising against c. difficile in children < 12 months and consider alternative etiologies for ages 12 months – 36 months Decision: C. difficile laboratory testing Implementation: Pediatric healthcare providers given brief educational seminars via PowerPoint	Design: Pre-post intervention comparison with cohorts Setting: One, tertiary care children’s hospital with emergency department Clinicians: All clinicians in ED and inpatient setting at pediatric hospital Patients: All patients who were seen in the emergency room or were admitted to any pediatric inpatient setting between July 2012 – July 2013 (pre-CPOE alert) and between September 2013 – September 2014 (post-CPOE alert)	Average monthly testing rate significantly decreased after the CPOE alert for children 0–11 months old (11.5 pre-alert vs 0 post-alert per 10,000 patient days; P< .001) and 12–35 months old (61.6 pre-alert vs 30.1 post-alert per 10,000 patients days; P< .001), but not for those children ≥36 months old (50.9 pre-alert vs 46.4 post-alert per 10,000 patient days; P= .3) who were not targeted with a CPOE alert	No assessment reported
Stevens et al, 201772	Enhancing Quality of Provider Practices for Older Adults in the Emergency Department (EQUiPPED)	“To evaluate the effectiveness and sustainability of individual provider feedback and electronic EDS to reduce the use of potentially inappropriate medications (PIMs), prescribed to older Veterans at the time of emergency department (ED) discharge”	CDS: Medication order sets for common outpatient geriatric prescriptions with renal dosing adjustments and alternative options Decision: Medications Implementation: Educational seminars by geriatricians and individual provider feedback	Design: Pre-post intervention comparison Setting: 4 VA emergency departments Clinicians: Attending and residents at all 4 VA emergency departments Patients: VA patients aged 65 or older discharged from emergency department	The proportion of PIMs at site one decreased from 11.9% (SD 1.8) pre-intervention to 5.1% (SD 1.4) post-intervention (P < .0001); site 2 from 8.2% (SD 0.8) pre to 4.5% (SD 1.0) post (P < .0001); site 3 from 8.9% (SD 1.9) pre to 6.1% (SD 1.7) post (P = .0007); and site 4 from 7.4% (SD 1.7) pre to 5.7% (SD 0.8) post (P = .04)	No assessment reported
Yan et al, 201773	Yield of CT Pulmonary Angiography in the Emergency Department When Providers Override Evidence-based Clinical Decision Support	“The authors hypothesized that the yield of CT pulmonary angiography would be statistically significantly lower when CDS was overridden compared with when the Wells’ decision rule was followed”	CDS: Wells’ score criteria integrated into the ED radiology CPOE. Recommended D dimer for Wells’ ≤4 Decision: CT Imaging for PE Implementation: Not discussed	Design: Retrospective analysis of visits with CT from January 2011 – August 2014. Compared guideline adherent CT’s to “Override Group” Setting: Urban, level-one trauma center with approx. 60,000 visits/year Clinicians: No information Patients: 2,404 CT’s in adherent group, 589 in override group	Overall diagnostic yield of CT pulmonary angiography: override group = 4.2% vs 11.2% in adherent group. P<001	No assessment reported
Sharp et al, 201874	Implementation of the Canadian CT Head Rule and Its Association With Use of Computed Tomography Among Patients With Head Injury	“To evaluate the association of implementation of the Canadian CT Head Rule on head CT imaging in community emergency departments (EDs) via CDS”	CDS: Clinical decision support triggered when the physician ordered a CT if information in the electronic health record indicated the scan was likely avoidable according to the Canadian CT Head Rule Decision: CT imaging for head injuries Implementation: Electronic learning module distributed via email to all providers along with in-person clinical leadership education	Design: An interrupted time-series analysis of encounters from January 2014 – December 2015 in 13 Southern California EDs Setting: 13 community EDs within a single integrated health care delivery system Clinicians: All physician providers working at the clinical sites Patients: Adult health plan members with a trauma diagnosis and Glasgow Coma Scale score at ED triage were included	An absolute 5.3% (95% confidence interval [Cl] 2.5% to 8.1%) reduction in CT use post-intervention After the intervention, diagnostic yield of CT-identified intracranial injuries increased by 2.3% (95% Cl 1.5% to 3.1%)	No assessment reported

Thirty-five studies (83%) reported positive effects on measured outcomes. The most common outcomes were process measures directly related to workflows altered by CDS interventions such as rate of compliance with guidelines for imaging^{35, 36} and medication orders.³⁷ Nine (21%) studies reported effects on patient-centered clinical outcomes, which we defined as revisits to the ED, hospital or ED length of stay, and admission and mortality rates, with five of these reporting improvements. Seven studies (17%) reported at least some evaluation of the CDS intervention’s usability or acceptance, including either user feedback or adoption rates for optional interventions.

Thirty-two of the included studies (76%) employed a before/after experimental design, with only three (7%) randomized control trials. Most studies were single center. Table 3 shows the results of the bias assessment for each study. We chose not to explicitly score studies, but noted that studies varied greatly in quality, from RCT’s with overall low risk of bias^{22, 38} to uncontrolled observational studies with higher risk of bias³⁹.

Table 3:

Risk of bias assessment results by study

	Prospective, Retrospective, or Mixed?	Are Inclusion/Exclusion Clearly stated?	Are inclusion/Exclusion applied uniformly?	Sufficient Sample Size for Outcome of Interest?	Level of Detail describing Intervention?	Are outcomes pre-specified?	Is the selection of comparison group appropriate?	Was an attempt made to balance allocation between groups?	Did researchers isolate impact from unintended exposure with multivariate analysis or stratification?	Did execution of intervention vary from protocol?	Were outcome assessors blinded?	Were outcome measurements valid and reliable?	Was length of follow-up the same for all groups?	Was follow up sufficient to evaluate outcomes?	Are important primary outcomes missing from results?	Are the statistical Methods used appropriate?	Are any important harms missing from the results?	Are results believable taking study limitations into consideration?	Is the funding source identified?
Bernstein et al 2005	Unknown	Yes	Yes	Yes	High	Yes	Yes	No	No	No	NA	Yes	NA	Yes	No	Yes	No	Yes	Yes
Kirk et al 2005	Prospective	Partially	Yes	No	High	Yes	UK	No	No	No	NA	No	Yes	Partially	No	Partially	No	Partially	Yes
Sard et al 2008	Retrospective	Yes	Yes	Yes	High	Yes	Yes	NA	No	No	No	Yes	NA	NA	No	Yes	Yes	Yes	Yes
Buising et al 2008	Unknown	Yes	NA	Yes	Medium	Yes	UK	NA	Yes	UK	No	Yes	NA	Partially	UK	Yes	UK	Yes	Yes
Terrell et al 2009	Prospective	Yes	Yes	Yes	High	Yes	Yes	Yes	UK	No	NA	Yes	NA	NA	No	Yes	Yes	Yes	Yes
Melnick et al 2010	Mixed	Yes	Yes	Yes	High	Yes	Yes	NA	UK	No	No	UK	Yes	Yes	No	Yes	Yes	Yes	Yes
Terrell et al 2010	Prospective	Yes	Yes	Yes	High	Yes	Yes	Yes	Yes	No	Yes	Yes	NA	NA	No	Yes	No	Yes	Yes
Venkat et al 2010	Prospective	Yes	Yes	Yes	Medium	Yes	Yes	NA	Partially	Yes	NA	Yes	NA	NA	Yes	Partially	Yes	Yes	Yes
Carman et al 2011	Mixed	Yes	Yes	Yes	Medium	Yes	No	No	No	Unknown	No	UK	NA	NA	No	No	Partially
Drescher et al 2011	Mixed	Yes	Part	Yes	High	Yes	Yes	NA	No	UK	No	Yes	NA	NA	No	UK	NA	Yes	No
Nelson et al 2011	Unknown	Yes	Yes	UC	High	Yes	Yes	No	UK	No	No	Yes	NA	NA	No	Yes	Yes	Yes	Yes
Raja et al 2012	Unknown	Yes	Yes	Yes	Yes	Yes	Yes	No	No	No	No	Yes	NA	NA	No	Yes	Yes	Yes	Yes
Griffey et al 2012	prospective	Yes	Yes	Yes	high	Yes	Yes	Yes	Yes	No	Yes	Yes	Yes	Yes	No	Yes	No	Yes	Yes
Dexheimer 2013	NA	No	NA	UK	Low	Yes	Yes	No	No	UK	NA	UK	NA	NA	UK	No	NA	Yes	No
Prevedello et al 2013	Retrospective	Yes	Yes	Yes	High	Yes	Yes	NA	No	No	NA	Yes	Yes	Yes	No	Yes	No	Yes	Yes
Demonchy et al 2014	Prospective	Yes	Yes	No	Medium	Yes	Yes	No	Yes	No	No	Yes	NA	NA	No	Yes	No	Yes	Yes
Raja et al 2014	Prospective	Yes	Yes	Yes	High	Yes	Yes	NA	No	No	No	Yes	NA	NA	No	Yes	Partially	Yes	Yes
Gupta et al 2014	Unknown	Yes	Yes	Yes	high	Yes	Yes	Yes	UK	UK	No	Yes	Yes	Yes	No	Yes	No	Yes	Yes
Dexheimer et al 2014	Prospective	Yes	Yes	Yes	High	Yes	Yes	Yes	Yes	UK	No	Yes	NA	NA	No	Yes	NA	Yes	Yes
Fowler et al 2014	NA	No	NA	No	Medium	No	NA	NA	No	NA	No	UK	NA	NA	UK	No	UK	Yes	No
Faine et al 2015	Retro	Yes	NA	No	High	Yes	Yes	NA	Yes	UK	UK	Yes	NA	NA	No	Yes	NA	Yes	No
Carnevale et al 2015	Mixed	Partially	Yes	Yes	Low	Yes	Yes	No	Partially	UK	No	Yes	NA	NA	No	Yes	Yes
Dean et al 2015	Prospective	Yes	Yes	Yes	Medium	Partially	Yes	No	Yes	No	No	Yes	Yes	Yes	No	Yes	No
Stevens et al 2015	Prospective	Yes	NA	Yes	Medium	Yes	NA	NA	Partially	No	NA	Yes	NA	NA	No	Yes	NA	Yes	Yes
Sethuraman et al 2015	Prospective	Yes	Yes	Yes	High	Yes	Yes	NA	Unknown	No	No	Yes	NA	NA	No	Yes	Yes	Yes	Yes
Ip et al 2015	Mixed	Yes	Yes	Yes	high	Yes	Yes	Yes	Yes	UK	No	Yes	Yes	Yes	No	Yes	No	Yes	No
Kharbanda et al 2016	Prospective	Yes	Yes	Yes	High	Yes	Yes	NA	Yes	No	NA	Yes	Yes	Yes	No	Yes	No	Yes	Yes
McGuire et al 2016	Prospective	Yes	Yes	Yes	High	Yes	UK	NA	Partially	No	NA	UK	Yes	Partially	No	Yes	Yes	Yes	Yes
Silveira et al 2016	Prospective	Yes	Yes	Yes	High	Yes	Yes	NA	No/Do not know	No	No	Yes	NA	NA	No	Yes	Yes	Yes	Yes
Austrian et al, 201	Retrospective	Yes	Yes	Yes	High	Yes	Yes	No	No	No	No	Yes	Yes	Yes	No	Yes	No	Yes	Yes
Baird et al, 2017	Mixed	Yes	Yes	No	Medium	Yes	Yes	No	No	No	No	Yes	NA	No	No	No	No	Yes	Yes
Bookman et al, 2017	Mixed	No	Yes	Yes	High	Yes	UK	Yes	No	No	No	Yes	Yes	Yes	No	Yes	No	No	No
Dayan et al, 2017	Prospective	Yes	Yes	Yes	High	Yes	Yes	No	No	No	No	Yes	Yes	Yes	No	Yes	Yes	Yes	Yes
Drescher et al, 2017	Mixed	Yes	Yes	Yes	Medium	Yes	Yes	No	UK	No	No	Yes	Yes	Yes	Yes	Yes	No	Yes	Yes
Hendrickson et al 2017	Retrospective	Yes	Yes	Yes	high	Yes	Yes	Yes	Yes	UK	UK	Yes	No	Yes	No	Yes	No	Yes	Yes
Martin Rico et al, 2017	Retrospective	Yes	Yes	Yes	Low	Yes	Yes	No	No	No	No	UK	Yes	Yes	No	Yes	No	Partially	No
Mills et al, 2017	Prospective	Yes	Yes	Yes	High	Yes	Yes	Yes	No	No	No	Yes	Yes	Yes	No	Yes	No	Yes	Yes
Min et al, 2017	Retrospective	No	Yes	Yes	High	Yes	Yes	Yes	Yes	No	No	Yes	Yes	Yes	No	Yes	No	Yes	No
Nicholson et al, 2017	Retrospective	Yes	Yes	Yes	Medium	Yes	Yes	Yes	Yes	No	No	Yes	Yes	Yes	No	Yes	No	Yes	Yes
Stevens et al, 2017	Mixed	Yes	Yes	Yes	Medium	Yes	Yes	No	No	No	No	Yes	Yes	Yes	Yes	Yes	No	Yes	Yes
Yan et al, 2017	Retrospective	Yes	NA	Yes	High	Yes	NA	Yes	NA	No	NA	Yes	NA	NA	No	Yes	No	Yes	Yes
Sharp et al, 2018	Prospective	Yes	Yes	Yes	Medium	Yes	Yes	Yes	No	No	No	Yes	Yes	Yes	No	Yes	No	Yes	Yes

Limitations:

As in all systematic reviews, decisions made regarding inclusion and exclusion criteria influenced our sample. Given the volume of literature and the motivation for our study, we chose to focus only on computerized decision support integrated with EHR technology. While this limitation was useful in defining a more cohesive cohort of studies, a few recent, high quality studies involving non-EHR integrated CDS were excluded, including an RCT.⁴⁰ Also excluded were several early studies in which investigators created CDS systems to promote guideline-based documentation and computer aided diagnosis before the era of modern EHR’s.^41–43 Furthermore, our decision to only include studies which reported the impact of an intervention on a care process or patient related outcome excluded literature from our current analysis which evaluated either the design or acceptability of CDS systems in isolation, without evaluation of the effects of implementation.

Discussion:

Our PRISMA-based systematic review identified 42 studies which explicitly measured the clinical impact of EHR-integrated CDS implemented in the ED. The majority of these studies (83%) reported positive impact although few addressed patient-centered clinical outcomes or long-term sustainability. Studies were mixed in quality and many had relatively high risk of bias. Commonly identified biasing factors included uncontrolled or historically controlled designs, lack of blinding to outcome assessment, and lack of inclusion of important potential positive and negative outcomes in evaluation of program impact. Among studies reporting intervention uptake and acceptance, CDS use was often low, even in cases where interventions had positive effects. Our ED specific results echo those found in the overall health informatics literature: while numerous studies report positive impact on process or outcome, well-controlled trials examining clinical outcomes are relatively rare^{8, 44} and uptake and acceptance of health IT can be challenging. Given the widespread clinical rollout of CDS systems driven by health policy,⁴⁵ the relatively small number of studies identified calls attention to the paucity of high quality literature evaluating actual implementation of these systems in the ED.

Our systematic review adds new knowledge regarding the scope and effects of CDS in the ED. Bennet & Hardiker conducted a review of the literature published between 1994 and 2015 to identify the body of CDS research undertaken in EDs, the research methods used, their quality, and the impact of CDS on clinical care in EDs.⁴⁶ While their review had somewhat similar inclusion criteria, it did not specify that CDS must be integrated into the EHR, and did not explicitly define CDS for the purpose of inclusion and exclusion. The authors included 23 studies, with some overlap of our sample, and noted a lower rate of studies with “positive impact” at 50%, however this is due to excluding studies which reported outcomes that were deemed not clinically relevant as opposed to identifying a higher rate of negative studies.

The preponderance of studies included employed a before and after study design. This is undoubtedly a convenient design for studying a single center intervention, and often the only possible design to use for evaluating an intervention that will be implemented solely for quality improvement purposes. Unfortunately, there are several disadvantages of this study design.^{47, 48} Uncontrolled before and after studies are often confounded by changes in practice pattern over time and in the specific setting of CDS interventions. For example, they can be confounded by education and attention surrounding the rollout of a new intervention focused on a condition or process. Fortunately, the included studies also provide proof of concept that more rigorous designs such as RCT’s^{22, 31, 38} and interrupted time series analyses^{37, 49} are feasible for the study of CDS interventions. Furthermore, several studies were able to use a multi-site design to provide better experimental control and generalizeability.^{24, 32, 50}

Many of the outcomes studied were process measures immediately downstream from implemented interventions, including documentation, prescribing or ordering of tests in a manner adherent to guidelines, or timing of interventions supported by CDS systems. While these are useful as they provide evidence that a given CDS modifies behavior, they do not make a strong case for the utility of CDS in improving patient outcomes. Furthermore, few studies explicitly discussed the potential harms of recommendations from CDS systems, including potential negative effects on patients when complying with a given practice guideline. The combination of reporting only process measures while not considering potential harms lays the groundwork for a potentially misleading assessment of a given intervention’s overall effect. Several studies did report significant improvement in patient centered outcomes, including an improved 30-day mortality among pneumonia patients,³² improved rates of screening tests and immunizations,^{51, 52} and decreased imaging utilization coupled with consistent diagnostic performance for various conditions.^{28, 33}

Overall, few papers discussed explicit human factors design considerations or usability evaluation of CDS interventions. This was likely due in part to our exclusion of papers which did not report impact of CDS implementation; many papers were excluded because they described only the design, without testing, of an intervention. Despite this, among the included articles there is a notable lack of emphasis on human factors and design considerations prior to implementation, or usability as an impact measure along with clinical outcomes. Among studies which did report data on acceptance of CDS, intervention uptake was often poor.^{49, 53} Several studies described resistance to use of CDS tools and among those which were optional, rates of CDS use were often low^{25, 54} despite positive outcomes when they were used. This finding is concordant with a recently published review focusing specifically on acceptance of cognitive support technologies at ED point of care, which found that, while clinicians acknowledge the value of these technologies, actual use rates are often low.⁵⁵ This highlights the importance of careful design of interventions and consideration of human factors design principles and usability evaluation in the design and implementation phases.

Based on reviewing the available literature, we propose authors of future studies of CDS interventions consider the following:

1. While uncontrolled before and after trials remain common, the literature demonstrates the feasibility of more robust experimental designs including interrupted time series, controlled before/after studies using multiple sites, or RCT designs.

2. While process outcomes are often the most convenient from a study design perspective, potentially impactful studies increasingly are focusing on more patient centered outcomes. Furthermore, while uncommonly included, potential negative (e.g. missed diagnoses, return ED visits) as well as positive consequences of recommendations should be included in studies of CDS implementation.

3. Given low acceptance rates even among potentially impactful interventions, usability and acceptance outcomes remain an important component of CDS evaluation. Additionally, further attention and effort should be dedicated towards human centered design and implementation of CDS to improve acceptance and uptake.

In conclusion, literature evaluating the implementation of CDS technologies within the emergency department is expanding as hospitals and physicians take advantage of supportive systems within electronic health records to drive patient-specific recommendations. Numerous studies suggest that CDS interventions are effective in changing physician practice with respect to process outcomes such as guideline adherence, however many studies are small and poorly controlled. Continuing to move from single center to multi-center study designs will offer the ability to utilize improved experimental designs. While the results of the current literature review highlight the promise of decision support for improving care, future studies should consider the inclusion of more specific information regarding design choices, attempt to improve on uncontrolled before-and-after designs, and focus on clinically relevant outcomes wherever possible.

Figure 1:

Study screening and inclusion