The use of computerized clinical decision support systems in emergency care: a substantive review of the literature

Abstract

Objectives: This paper provides a substantive review of international literature evaluating the impact of computerized clinical decision support systems (CCDSSs) on the care of emergency department (ED) patients.

Material and Methods: A literature search was conducted using Medline, Cumulative Index of Nursing and Allied Health Literature (CINAHL), Embase electronic resources, and gray literature. Studies were selected if they compared the use of a CCDSS with usual care in a face-to-face clinical interaction in an ED.

Results: Of the 23 studies included, approximately half demonstrated a statistically significant positive impact on aspects of clinical care with the use of CCDSSs. The remaining studies showed small improvements, mainly around documentation. However, the methodological quality of the studies was poor, with few or no controls to mitigate against confounding variables. The risk of bias was high in all but 6 studies.

Discussion: The ED environment is complex and does not lend itself to robust quantitative designs such as randomized controlled trials. The quality of the research in ∼75% of the studies was poor, and therefore conclusions cannot be drawn from these results. However, the studies with a more robust design show evidence of the positive impact of CCDSSs on ED patient care.

Conclusion: This is the first review to consider the role of CCDSSs in emergency care and expose the research in this area. The role of CCDSSs in emergency care may provide some solutions to the current challenges in EDs, but further high-quality research is needed to better understand what technological solutions can offer clinicians and patients.

OBJECTIVES

This paper provides a description of a substantive review of published international literature evaluating the impact of computerized clinical decision support systems (CCDSSs) on the care of emergency department (ED) patients. The principal aims of this review are to identify the body of CCDSS research undertaken in EDs, the research methods used, their quality, and the impact of CCDSSs on clinical care in EDs. The discussion synthesizes what is known and not known about the effectiveness of CCDSSs in emergency care and the quality of the current evidence base.

BACKGROUND AND SIGNFICANCE

There are huge challenges facing providers of emergency and urgent care within the UK National Health Service (NHS) and internationally. Ever-increasing use and ongoing economic uncertainties have created a situation where the demand for emergency health care cannot consistently be matched by the resources available to deliver it. There is clear international evidence of the deleterious effects of long waits on the mortality and morbidity of ED patients.^1–4 The effect of waiting in the ED has been directly correlated with mortality even when comorbidities are controlled for.⁵ In counties where there are performance targets, the pressure to deliver timely and efficient emergency care is compounded.^6,7

Within the NHS in England, there are additional challenges that complicate matters further. Changes to medical education have reduced the time doctors in training spend working in EDs.⁸ There is evidence that as a result of this, decision-making takes longer, more tests are ordered, and more referrals are made to inpatient specialties.⁹ In emergency care, the use of locum (temporary replacement) doctors is proportionately high, with reports that one-fifth of medical staffing on weekends is by locums.¹⁰ Locum doctors will not know the ED’s policies or clinical guidelines and often require a level of support and supervision, adding to the pressure within the system.¹¹

A parallel and equally important issue is that of the aging population. The number of people over the age of 85 years in the UK has increased by 20% since 2006¹² and is expected to continue to increase by two-thirds over the next 20 years.¹³ In 2009–2010, 15% of ED patients in the UK were over 70, and this will increase.¹⁴ An aging population adds to the increase in demand for EDs and subsequent hospital admissions. Compared to patients <30 years, those >70 years are 5 times more likely to be admitted.¹⁵ The elderly have unique care needs that can be difficult to meet in a busy emergency care environment.¹⁶ The elderly spend more time in the ED, their problems are more complex, they undergo more tests, and they often require critical care.¹⁵

The current climate within emergency care settings presents significant safety challenges. The workforce is less experienced and there are numerous vacancies at the consultant level.^9,17 The pressure within the emergency care system is further compounded by rising numbers of patients. The global economic crisis is preventing any increase in health care spending.^18,19 The clinical work in EDs is often complex and the environment more challenging than in other more traditional settings.²⁰ One distinctive feature of decision-making in the ED is the frequency and complexity of decisions with limited time and information.²¹ This creates a clinical environment with additional risk of adverse events and clinical incidents.²²

Novel approaches are required to ensure that the workforce can deliver high-quality care. Over the last 20 years, there has been an explosion in the number of studies evaluating the role of CCDSSs. There is a growing consensus that CCDSSs have the potential to significantly improve health care.²³ Several systematic reviews have identified the benefits of CCDSSs with regard to patient safety, improved clinical performance, and improved patient outcomes.^24,25

A recent series of 6 comprehensive systematic reviews has drawn overall conclusions that CCDSSs across a range of health care settings can improve the process of care for some patients.^26–31 However, the number of studies that have demonstrated a positive impact on patient outcomes is relatively low, and these are only of moderate quality.²⁷ Within this series of systematic reviews only randomized controlled studies (RCTs) were reviewed, as this method is regarded as the gold standard for CCDSS evaluation.³² CCDSS research using an RCT method in the ED setting is fraught with methodological challenges. It is unsurprising that the acute care review only identified 3 studies undertaken in EDs.²⁶ To date, there are no comprehensive reviews of all types of studies on the effectiveness of CCDSSs in emergency care settings. This paper sets out to redress that balance and expose the body of CCDSS research in EDs and the quality of the research, and to identify the contribution that CCDSS can make to enhancing care in this challenged setting.

MATERIALS AND METHODS

Search strategy

The strategy involved searching 3 bibliographic databases and multiple electronic resources and gray literature websites for resources published between 1994 and 2015; see Table 1.

Table 1.

Search of electronic resources

Journal databases
Medline hosted by EBSCO
CINAHL hosted by EBSCO
Embase via NHS evidence
Electronic resources
The Cochrane Collaboration: www.cochrane.org
Centre for Reviews and Dissemination, including database of abstract of reviews of effectiveness (DARE), NHS economic evaluation database, and the Health Technology Assessment Programme: via www.crd.york.ac.uk
National Institute for Health Service Research: http://www.nets.nihr.ac.uk
Implementation Science: http://www.implementationscience.com
Gray literature websites
Zetoc (database of conference proceedings): http://zetoc.mimas.ac.uk
EThOS (electronic theses online service): http://ethos.bl.uk
OpenGrey (European gray literature): http://www.opengrey.eu/
NHS Evidence Health Information Resources: https://www.evidence.nhs.uk/nhs-evidence-content/journals-and-databases

For the search terms used in the database searches, see the Medline example in Table 2.

Table 2.

Medline search terms

Computer assisted decision making OR computer assisted instruction OR decision support systems OR reminder systems OR clinical decision support system OR computerised clinical decision support system OR computerized clinical decision support system OR computer decision support OR decision aid OR computerised decision aid, computerized decision aid, computerised decision support OR computerized decision support OR CDSS OR decision support systems, clinical AND accident and emergency OR emergency department OR emergency care OR A&E OR emergency service OR urgent care OR unscheduled care OR Emergency Service, Hospital OR Ambulatory Care

Figure 1 depicts the searching and screening process that took place against the inclusion and exclusion criteria. The preliminary results identified 1773 papers from the 3 databases and 49 papers/documents from other sources. These were screened by title and abstract for CCDSS research in ED. This reduced the number of papers to 399. The second review screened for primary research and reduced the number of papers to 180. The final review screened each abstract (if needed, the entire paper was reviewed) against the inclusion and exclusion criteria to identify 23 studies.

Figure. 1.

Database search results and screening process

Inclusion and exclusion criteria

The studies included in this review had to meet 3 principal inclusion criteria. Studies that met any of the exclusion criteria were rejected.

Inclusion

1. The study had to take place in an ED.

2. The study had to report primary research on the use of a CCDSS for an acute problem in a face-to-face situation.

3. The study had to compare care supported by CCDSSs with usual care.

Exclusion

• Studies identifying only the beliefs, attitudes, and opinions of system users

• ED tracking systems

• Technical development of CCDSS (bench testing/simulated settings)

• Bed management systems

• Paper-based decision support tools

• Radiology imaging systems

• Pathology ordering systems

• Pharmacy systems for drug prescribing (dosing/error reduction)

• Systems used by patients

• Health screening/surveillance

Justification of inclusion and exclusion criteria

For practical reasons, only English-language research papers were included. As CCDSSs are a relatively novel health care intervention with little research before the 1990s, studies were selected from the last 21 years (1994–2015). As the literature search progressed, it become evident that there was only a small number of CCDSS studies that had taken place in EDs. All study designs were included to increase the understanding of how CCDSSs in EDs had been evaluated. Computerized physician/provider order entry (CPOE) systems have been the subject of extensive research in the United States. There is a small but growing number of studies that have evaluated the benefits of CPOE on ED patients, workflow, and safety, and some have incorporated decision support.³³ Only the studies involving the use of CPOE that had a specific clinical decision support tool embedded in the system were included. Once the set of included studies was finalized, a review of their reference lists was undertaken to identify any additional relevant publications.

Methodological review of the studies

Traditional critical appraisal tools to assess the quality of all the study designs in this review are not available. A pragmatic approach was taken to assess study quality by analyzing the risk of bias that each design presents. Assessing the risk of bias is critical in understanding the validity of results and the effects of CCDSS intervention.³⁴ The approach was based on the Cochrane Collaboration Effective Practice and Organization of Care Group (EPOC)³⁵ criteria for assessing the risk of bias in RCTs and interrupted time series (ITS) designs (see Table 3).

Table 3.

EPOC criteria for assessing risk of bias³⁵

RCT risk of bias assessment criteria
1. Was the allocation concealment adequately generated?
2. Was the allocation adequately concealed?
3. Were baseline outcome measurements similar?
4. Were baseline characteristics similar?
5. Were incomplete outcome data adequately addressed?
6. Was knowledge of the allocated interventions adequately prevented during the study?
7. Was the study adequately protected against contamination?
8. Was the study free from selective reporting?
9. Was the study free from other risks of bias?
ITS risk of bias assessment criteria
1. Was the intervention independent of other changes?
2. Was the shape of the intervention effect prespecified?
3. Was the intervention unlikely to affect data collection?
4. Was knowledge of the allocated intervention adequately prevented during the study?
5. Were incomplete outcome data adequately addressed?
6. Was the study free from selective outcome reporting?
7. Was the study free from other risks of bias?

As before and after studies are not recognized by the Cochrane Collaboration as a reliable method to evaluate complex interventions, no EPOC criteria exist for such studies.³⁵Table 4 describes the criteria used in this review to detect bias, developed from various sources.^36–38

Table 4.

Threats to internal validity in before and after studies

1. Selection bias

2. Performance bias

a. Historical changes

b. Testing bias

c. Hawthorne effect

d. Maturation effect

3. Detection bias

4. Attrition bias

5. Reporting bias

6. Regression to the mean

7. Confounding variables

PB and NH constructed the search. PB screened all the papers for inclusion/exclusion. NH audited a sample of the selected papers to ensure rigor in the selection process. As the body of evidence was small (23 papers in total) and little is known about the evaluation of CCDSSs in EDs, all papers were included in the review irrespective of the quality of research.

RESULTS

Overview of included studies

Twenty-three studies met the inclusion criteria and were included in the review (see Table 5). Sixty percent of the studies were undertaken in the United States (n = 14); the remainder were undertaken in Canada (n = 3), Australia (n = 3), UK (n = 1), France (n = 1), and the Netherlands (n = 1). A majority evaluated CCDSSs in one ED (n = 18). There were 4 multicenter studies: 2 in the United States that evaluated the same CCDSS in 4 EDs, a Canadian study involving 4 EDs, and a study in France involving 20 EDs. The vast majority of studies were undertaken in academic EDs (n = 21).

Table 5.

Overview of the key characteristics of the studies included in the literature review

Author and country	Aim of study and clinical condition	Study design and sample	Key outcome measures	Results
Bond et al. (2013)39 Canada	To determine if electronic clinical practice guidelines (eCPGs) improve management of patients with severe sepsis and septic shock.	Before and after study	Time to lactate measurement Time to blood culture collection Time to antibiotic administration Delivery of fluid bolus Administration of vasopressors for unresponsive hypotension Measurement of central venous pressure (CVP) and cardiovascular system O2	There was a statistically significant difference in antibiotics within 3 h (P = .034), use of vasopressors (P = .019), measurement of CVP (P < .0001).
		Pre- and post-eCPG outcomes were compared between 51 cases and 51 matched retrospective controls.
				Overall finding suggests that patients in the eCPG group were treated more aggressively.
Dexheimer et al. (2013)40	Does the use of a CCDSS detect and then improve the care of children with asthma by prompting clinicians to use the asthma guideline?	Prospective RCT	Time to disposition Admission rate ED length of stay (LOS)	There was no statistically significant difference between control and intervention groups in time to disposition, admission rate, ED LOS.
		Intervention n = 358; control n = 346
United States
Kwok et al. (2009)41	To assess the effects of a CDSS on improving adherence to asthma guidelines in adults.	Before and after study	Quality of care judged by documentation of: Precipitating factors Previous ICU history Smoking history Peak flow results Asthma severity Discharge plan Smoking cessation advice Steroids on discharge	Increased documentation of asthma severity (P < .01) and discharge plan documentation (P < .01) despite regression model adjustment for triage category and seniority of doctor.
Australia		Observational pre- and post-intervention design comparing CDSS with historical controls (n = 50; study group n = 50 control)
Nelson et al. (2011)42	Does a CCDSS alert the physician of sepsis (2 or more SIRS criteria and 2 or more BP <90 mmHg) and suggest specific recommendations to improve care?	Before and after prospective studyPhase 1 data collected but staff not alerted; Phase 2 staff alerted, sample size not evidence in paper; no reply from e-mail correspondence with author	Timeliness of: Blood cultures Lactate Chest X-ray Antibiotics	The doctor often diagnosed sepsis before the CDSS. With the CDSS, chest X-ray before admission (OR = 3.2, 95% CI, 1.1–9.5) and collection of blood cultures were more frequent (OR = 2.9, 95% CI, 1.1–7.7). Blood cultures were the only thing performed significantly faster (P = .032). Frequency and timeliness of lactate sampling and provision of antibiotics remained unchanged.
United States
Gibbs et al. (2012)43	To evaluate the impact of a CDSS on core measures for the treatment of patients with CAP.	Interrupted time series design	Blood cultures prior to antibiotics Antibiotics within 6 h of arrival Appropriate antibiotic selection Mean time to antibiotic administration	There was a statistically significant difference in blood cultures prior to antibiotics (P < .001), antibiotics within 6 h (P = .004), appropriate antibiotic selection (P = .0112).
United States		Before (n = 613) and after (n = 572) CCDSS implementation
Roukema et al. (2008)44	To assess compliance with a CCDSS for the management of children with fever without an apparent source. To assess the effects on time in the ED and number of laboratory investigations ordered.	Prospective randomized controlled trial; total of 164 children	Compliance with CCDSS LOS in ED Number of investigations ordered	Length of time in ED was no different between the 2 groups; this was an unexpected result.
Netherlands		N = 74 randomized into CCDSS group that gave immediate instructions on what laboratory investigations to order; n = 90 randomized to “usual care” assessed by a physician who then decided on the investigations.		Adherence to advice from the CCDSS was deemed successful. In intervention group, 82% had investigations ordered; in control, 44%. This was in contrast to expectations.
				The prediction rule in the CCDSS needed adjustment, as it was not specific enough to discriminate between children at high risk of serious infection. The CCDSS was discontinued.
Schriger et al. (1997)45	To assess the effects of guidelines incorporated into electronic patient records for health care workers exposed to body fluids on guideline adherence.	Prospective 3-phase off-on-off interrupted time series with intention-to-treat analysis	Quality of care as determined by 7 essential items documented in the medical record Provision of aftercare instructions; compliance with 4 testing decisions and 5 treatment decisions Overall cost of care	Documentation improved by 42% (95% CI, 34%–49%) between phases 1 and 2 but decreased to baseline in phase 3. Compliance with issuing aftercare instructions increased by 62% (95% CI, 51%–74%) but decreased to baseline in phase 3. Compliance with laboratory testing increased by 20% (95% CI, 9%–31%) and decreased to below baseline in phase 3. Compliance with treatment increased by 13% (95% CI, 9%–17%) and decreased to baseline in phase 3. Charges incurred for test and treatments increased by 37% (95% CI, 22%–52%) and decreased below baseline in phase 3.
United States		Phase 1: off n = 50; phase 2: on n = 156; phase 3: off n = 74
Schriger, Bareff, Buller et al. (2000)	To assess the effects of guidelines incorporated into an electronic patient record for children under 3 years with fever in guideline adherence.	Prospective off-on-off interrupted time series with intent-to-treat analysis	Quality of clinical documentation Provision of aftercare instructions Appropriateness of testing Treatment decisions Diagnosis Overall cost of care	Quality of clinical documentation improved from 80% to 92% between phases 1 and 2 (13% improvement, 95% CI, 10%–15%). Documentation of aftercare instructions improved from 48% to 81% between phases 1 and 2 (33% improvement, 95% CI, 28%–38%). All documentation returned to baseline during phase 3. There was no difference in appropriateness of care and no change in overall cost of care.
United States
		Phase 1: off n = 352; phase 2: on n = 374; phase 3 off n = 104
Drescher et al. (2011)47	Does use of a CCDSS embedded in a CPOE improve the use and yield of computerized tomography pulmonary angiography (CTPA) in the diagnosis of pulmonary embolism (PE)?	Before and after intervention study	Did the rate of positive CTPA increase with CCDSS?	Pre-intervention positive CTPA 8.3% (95% CI, 4.9%–12.9%). Post-intervention positive CTPA 12.7% (95% CI, 8.6%–17.7%). This was a 4.4% (95% CI, −1.4%, 10.1%) increase in the proportion of positive results. The proportion of d-dimer tests decreased from 70% to 63%
		Prospective intervention group with retrospective pre-intervention comparison group; pre (n = 205) and post (n = 229) implementation
United States
Raja et al. (2012)48	To assess the effect of CPOE on the use and yield of CPTA for PE.	Before and after intervention study over 6-year period	Use and yield of CTPA	Decrease in the use of CTPA with CCDSS from 26.4% to 21.2% (P = .0379).
United States
		Pre CCDSS n = 3855; post n = 2983 had CPTA ordered
				The percentage of positive CPTA increased with the CDSS from 5.8% to 9.8%, P = .0323.
Roy et al. (2009)49	To assess the effectiveness of a handheld CCDSS to improve the diagnostic workup of patients with suspected PE.	Cluster RCT	Primary outcome was appropriateness of diagnostic work-up in each center	Appropriate workups increased in all patients compared to before the trial began. The greatest increase was in the CCDSS. Primary outcome was appropriateness of diagnostic workup; this increased by 19.3% after adjusting for variables (P = .023). Pre-test probability scoring was greater in the CCDSS, P < .001.
		After all 20 EDs were accustomed to inputting clinical data and using the devices, they were randomly allocated to either activation of the CDSS on the device or posters and pocket cards that showed the validated diagnostic strategies.
France
		Pre-intervention (20 centers): 1103 patients. Post intervention: 10 centers randomized to CDDSS, n = 753; 10 centers to posters and pocket cards, n = 1052.
Jones et al. (2013)50	To assess the impact of CCDSS on the management of patients presenting to ED with CAP.	Before and after study	Overdiagnosis Hospital admission Guideline-concordant triage Length of stay Inpatient mortality	Use of the CCDSS increased the number of appropriate hospitalizations (P = .02) and reduced inpatient mortality (P < .02). Outcomes were adjusted for disease severity.
		A real-time CCDSS was developed to assess pneumonia severity and make management recommendations. Pre (n = 2349) and post (n = 2583) implementation outcomes were compared in 4 EDs.
United States
Niemi et al. (2009)51	To assess CDSS’s ability to detect CAP and heart failure (HF) and then improve compliance with national quality indicators.	Before and after study Seven months pre- and post-implementation of CCDSS. Before and after comparison for quality indicators. No sample sizes were available despite e-mail correspondence with author.	CAP Antibiotics within 4 h Pneumococcal vaccination status documented Appropriate antibiotics HF Left ventricular (LV) function assessment Initiation of medication Provision of discharge instructions	Measures of antibiotic administration within 4 h and vaccination status recording slightly increased.
United States
				Correct antibiotic selection decreased slightly; none were statistically significant. LV function assessment and medication initiation increased slightly but were not statistically significant.
				Provision of discharge instructions for HF increased, P < .01.
Buising et al. (2008)52	To evaluate the impact of different methods of guideline implementation (academic detailing, CCDSS) on prescribing behavior for patients with community-acquired pneumonia (CAP).	A 2-stage before and after intervention cohort study and time series analysis	Prescription of antibiotic that adequately covered likely pathogens and followed recommendations Recognition of severe illness and adjustment of antibiotics Adjustment of antibiotics for allergies	There was an improvement in antibiotic prescribing with both academic detailing and CCDSS. The improvement with the CCDSS was greater than expected, based on binary logistic predictions.
Australia
		All patients who presented during the study period (baseline n = 392, academic detailing n = 215, and CCDSS n = 133) were compared for concordance with recommendations on prescribing.
Dong et al. (2005)64	To determine agreement between a computer-based triage tool and memory-based triage (the usual method).	Prospective observational study	Determine agreement between CCDSS for triage and usual care	Agreement between the computer-based triage system and the usual memory-based system was poor.
Canada		A total of 693 patients were triaged in the usual way (memory-based), then by a blinded research nurse using the computer-based tool. The results of both triage decisions were compared. An expert panel was used to judge the accuracy of both decisions.
				There was more agreement between the expert panel and the computer-based system than the expert panel and the memory-based system.
Lim et al. (2012)65 Canada	To assess the effect of an electronic clinical practice guideline (eCPG) on adults presenting to ED with febrile neutropenia.	Retrospective comparative cohort study	Informatics and clinical outcomes were measured: Use of eCPG Time interval from triage to antibiotic LOS in ED Investigations Treatment	eCPG use was 37.8% overall. Intervention, 57%; control ED 1, 19%; 2, 0%; 3, 0%. In intervention ED, ECGs (P = .03) and blood cultures (P = .04) were performed more frequently. Reduction in triage to doctor assessment (P = .001), triage to first antibiotic (P = .02).
		Four EDs; 1 the intervention ED (the designated ED destination center for patients on cancer treatment; developed the eCPG and used it widely). The other 3 EDs were controls. eCPG not mandatory.
				There was no statistically significant difference in the eCPG group for time to antibiotics, although it was slightly lower. Knowledge of eCPG improved care for all patients.
		Total of 201 patients in study, 128 in intervention ED, 73 in the 3 control EDs.
Asaro, Sheldahl, and Char (2006)54	What is the effect of introducing a CCDSS on guideline adherence for patients presenting with acute coronary syndrome (ACS)?	Descriptive retrospective before and after study (4 phases)	Risk stratification: acute ST elevation MI, high-risk ACS, intermediate-risk ACS, and low-risk ACS Drugs appropriately prescribed: β-blocker, heparin, aspirin	No improvement in overall compliance with any of the ACS recommendations.
United States		Pre CPOE, preprinted order form was available, n = 45; pre-CPOE several weeks after introduction of paper guideline, n = 66; several weeks after CPOE introduction, n = 25; 3 months after CPOE with additional education about the order sets, n = 16
Britton et al. (2013)55	What is the effect of a computerized order set within a CPOE system for sexual assault victims with regard to adherence to treatment guidelines?	Before and after study	Compliance with the order set	Pre-test compliance was 4.4%; post-test was 82.4% (P < .001).
		Pre-test (n = 322), post-test (n = 131) design over a 10-year period
United States
Carman et al. (2011)57	To evaluate CCDSS embedded into Allscripts to improve adherence to national guidelines for management of skin and soft-tissue infections in ED.	Before and after study	Correct antibiotics Wound culture Chlorhexidine scrubs	Adherence to antibiotics (P < .001), chlorhexidine scrubs (P < .001). Use of the CCDSS declined over the course of the study. Use of the CCDSS only had impact on chlorhexidine. Although antibiotic adherence improved, this did not correlate with use of the CCDSS.
		Pre-test, n = 205; post-test at 6 weeks, n = 383, at 12 weeks, n = 285
United States
Goergen et al. (2006)58	To assess whether introducing an imaging guideline based on NEXUS criteria can reduce the numbers of radiological investigations in patients with neck trauma in the ED; imaging guideline was converted into CCDSS.	Before and after study; prospective nonrandomized clinical trial (October 2001 to September 2002) using historical controls (June 2000 to July 2001)	How many used the CCDSS Of those managed with CCDSS who were advised no X-ray, how many were not X-rayed	Statistically significant reduction in the number of neck images (P = 0.03) in the total sample.
Australia
		Study group, n = 353; control n = 403. Of the 353 study patients, 141 (40%) were managed with the CCDSS.		Largest reduction in those patients managed with the CCDSS (P = .01).
				There was no delayed diagnosis in those not imaged.
Jadav et al. (2009)59	To evaluate the effects of making pain scoring mandatory for children attending ED: did it increase provision of analgesia?	Before and after study; retrospective case note review comparing audit results before and after introduction of mandatory pain scoring	Number of children who had their pain scored	There was an increase in the number of children who had their pain scored (P < .001). No significant change in those given analgesia.
United Kingdom
		Pre-intervention, n = 187; post-intervention, n = 163
Melnick et al. (2010)60	Does a CCDSS for syncope change doctor behavior and improve guideline adherence?	Before and after study; prospective study with retrospective controls	Admission rate Number of CT scans ordered	Statistically significant difference in admission rates (P = .036). No difference in CT imaging (P = .358).
United States
		Pre-intervention, n = 410; post-intervention, n = 301
Day et al. (1995)53	Does use of a CCDSS for care of ED patients with acute low back pain improve guideline adherence?	Prospective time series on-off design; paper vs CDSS for acute low-back pain	Accurate documentation Discharge instructions given Investigations ordered Treatment Patient disposition	Statically significant improvement in documentation (P < .001), discharge instructions (P < .001). There was no statistically significant difference in X-rays ordered, medication use, and cost of care.
United States
		Control, n = 206, random sample of 103 charts analyzed. Test period, n = 259; 202 were treated with the CCDSS. All 259 were analyzed.

Rationale for undertaking CCDSS research

All the studies, by way of rationale, referred to the challenges of guideline adherence in an era when there is rapid expansion of clinical guidelines, quality indicators, risk assessment tools, and core measures for specific clinical conditions. Several papers cited the unique challenges of the ED environment: multiple interruptions, complex patients, and overcrowding, indicating that these add considerably to the difficulties of guideline implementation and adherence.^39–43

The consensus within many of the selected studies was that CCDSSs can provide an effective means of improving the quality of ED patients’ care.^39–52 Some departments were so encouraged by the results that they developed additional CCDSSs, for example to cover a range of presenting conditions, and generated several additional studies.^45,46,53

Clinical conditions investigated

There was a wide variety of clinical conditions studied. Conditions covered by more than 1 study included:

• Community-acquired pneumonia (n = 3)

• Pulmonary embolism (n = 3)

• Sepsis (n = 2)

• Fever in children (n = 2)

• Asthma in children (n = 2)

One study covered both heart failure and community-acquired pneumonia. The remaining 11 studies covered a range of clinical conditions:

• Acute coronary syndrome

• Sexual assault

• Soft-tissue infection

• Acute low-back pain

• Triage

• Neck trauma

• Pain in children

• Febrile neutropenia

• Syncope

• Blood-borne virus exposure

• Heart failure

Type of CCDSSs and their functionality

Various types of functionality were used within the CCDSSs. Four of them were incorporated into CPOE using an embedded order set.^47,48,54,55 The most common method of decision support was via popup alerts (n = 12), which provided suggestions to clinicians regarding assessment and/or treatment options. See Table 6 for a more comprehensive overview of functionality.

Table 6.

Types of CCDSS in use in EDs and their functionality

Author and clinical condition	Commercial system		Locally developed	Incorporated in CPOE	Mandatory	Decision support via:
						E-mail	Popup alert	Pager	Embedded order set	Other
Bond et al. (2013)39	✗		✓	✗	✗	✗	✗	✗	✗	Prompts within CCDSS
Sepsis
Dexheimer et al. (2013)40	✗		✓	✗	✓	✗	✗	✗	✗	Printed paper guideline
Asthma in children
Kwoket al. (2009)41	✗		✓	✗	✗	✗	✓	✗	✗
Asthma in children
Nelson et al. (2011)42	✗	✓		✗	✗	✗	✓	✓	✗	Automatic text suggesting treatments
Sepsis
Gibbs et al. (2012)43	✗		✓	✗	?	?	?	?	?	No other details available
Community-acquired pneumonia
Roukema et al. (2008)44	✗		✓	✗	✗	✗	✓	✗	✗
Fever in children
Schriger et al. (1997)45	✗		✓	✗	✓	✗	✓	✗	✗
Blood-borne virus exposure
Schriger et al. (2000)46	✗		✓	✗	✗	✗	✓	✗	✗
Fever in children <3 years
Drescher et al. (2011)47	✓		✓	✓	✗	✗	✓	✗	✓
Pulmonary embolism
Raja et al. (2012)48	✗		✓	✓	✓	✗	✓	✗	✓
Pulmonary embolism
Roy et al. (2009)49	✗		✓	✗	✗	✗	✓	✗	✗
Pulmonary embolism
Jones et al. (2013)50	✗		✓	✗	?	?	?	?	?	No other details available
Community-acquired pneumonia
Niemi et al. (2009)51	✗		✓	✗	✗	✓	✓	✗	✗	E-mails and printed alerts sent to various places
Community-acquired pneumonia and heart failure
Buising et al. (2008)52	✗		✓	✗	✗	✗	✓	✗	✗	Prompts within CCDSS
Community-acquired pneumonia
Dong et al. (2005)64	✗		✓	✗	✓	✗	✗	✗	✗	CCDSS presents triage guideline
Triage
Lim et al. (2012)65	✓		✓	✗	✗	✗	✓	✗	✗
Febrile neutropenia
Asaro et al. (2006)54	✓		✗	✓	✗	✗	✗	✗	✓
Acute coronary syndrome
Britton et al. (2013)55	✓		✗	✓	✗	✗	✗	✗	✓	Prompts within CPOE
Sexual assault
Carman et al. (2011)57	✓		✗	✗	✗	✗	✓	✗	✗	Linked to CPOE for ordering swabs
Skin/soft-tissue infections
Goergen et al. (2006)58	✗		✓	✗	✗	✗	✗	✗	✗	CCDSS advised if X-ray required
Neck trauma
Jadav et al. (2009)59	✓		✗	✗	✓	?	?	?	?	No other details available
Pain in children
Melnick et al. (2010)60	✓		✓	✗	✗	✗	✗	✗	✗
Syncope
Day et al. (1995)53	✗		✓	✗	✗	✗	✗	✗	✗	Prompts within CCDSS suggesting treatment
Acute low-back pain

Impact of CCDSSs on clinical care in EDs

The following section analyzes how each study design evaluated the impact of CCDSSs in EDs. Analyzing the robustness methods used for evaluation is an intrinsic part of understanding the evidence base and how reliable it is. Ultimately, this will enable firm conclusions to be drawn about the effectiveness of CCDSSs in EDs.

Five research designs were employed:

1. Randomized controlled trial (n = 3)

2. Before and after study (n = 13)

3. Interrupted time series (n = 5)

4. Prospective observational design (n = 1)

5. Comparative cohort design (n = 1)

Randomized controlled trials that have assessed the use of CCDSSs in EDs

Three studies used an RCT design.^40,44,49 The studies by Roukema et al.⁴⁴ and Dexheimer et al.⁴⁰ were conducted in academic pediatric EDs. The study by Roy et al.⁴⁹ was a cluster randomized controlled trial across 20 EDs in France, approximately half of which were academic departments.

In the study by Dexheimer et al.,⁴⁰ there was no statistically significant difference between the control and intervention groups for time to referral, admission rate, or length of stay in the ED. The results of the study by Roukema et al.⁴⁴ demonstrated that there was no difference in length of stay between the 2 groups. Adherence to the guidance within the CCDSS was deemed successful. Eighty-four percent had tests ordered in the intervention group, compared to 44% in the control group. However, the prediction rule in the CCDSS was not specific enough to discriminate between children at high risk of serious bacterial infection and was discontinued.

The cluster RCT by Roy et al.⁴⁹ randomized 20 EDs across France to either control or intervention departments. Appropriate diagnostic workups increased in all patients compared to the pretrial data. The greatest increase was in the CCDSS group. After adjusting for confounding variables, the appropriateness of the diagnostic workup increased by 19.3% (P = .023). Pretest probability scoring was greater in the CCDSS group (P < .001).

Finally, the asthma detection and management CCDSS evaluated by Dexheimer et al.⁴⁰ did not appear to demonstrate any benefits. Roukema et al.⁴⁴ did demonstrate an increase in initial tests for children with fever. However, the CCDSS was discontinued due to its inability to discriminate between children with low or high risk of serious infection. The cluster RCT by Roy et al.⁴⁹ is the only study using a randomized experimental design that demonstrated tangible improvements in the process of care. However, it is unlikely that cluster RCTs are a feasible option for future studies due to the high cost and logistical challenges of implementation.

Before and after studies that have assessed the use of CCDSSs in EDs

Before and after studies were the most commonly used design identified by this review (n = 13). They are also the leading research method in general CCDSS research.⁵⁶ All the before and after studies in this review compared the process of care and/or patient outcomes before and after introduction of the CCDSS.^{39,41,42,47,48,50,51,54,55,57–60} None included a control. Four of the studies assessed the effect of order sets and prompts embedded within the CPOE system on the management of acute coronary syndrome (ACS),⁵⁴ sexual assault,⁵⁵ and pulmonary embolism.^47,48 Both of the studies assessing the impact of ordering appropriate investigations for pulmonary embolism revealed statistically significant results, with an increase in the number of positive results when the CCDSS guided the ordering of computerized tomography pulmonary angiography.^47,48 In the study by Britton et al.,⁵⁵ post-test compliance with adherence to treatment guidelines in sexual assault was highly statistically significant (P < .001). However, in the study by Asaro et al.,⁵⁴ there was no improvement in overall compliance with the recommendations for ACS management.

Three studies assessed the impact of CCDSSs on the management of infection in the ED; 2 of these concerned sepsis^39,42 and the third focused on soft-tissue infections.⁵⁷ For the management of sepsis, the study by Britton et al.⁵⁵ suggested that patients in the CCDSS group were treated more aggressively. However, this contrasts with the study by Nelson et al.,⁴² who identified that the only difference in the process of care between the before and after groups was the speed with which blood cultures were taken. The study by Carman et al.⁵⁷ evaluated the use of embedded guidelines within the electronic patient record for management of soft-tissue infection. The CCDSS had a statistically significant impact on wound cleansing only (P < .001). Although antibiotic adherence improved, this did not appear to correlate with use of the CCDSS.

Two before and after studies in pediatric EDs considered the effects of CCDSSs on pain assessment and management⁵⁹ and adherence to asthma guidelines.⁴¹ Both studies demonstrated an increase in appropriate documentation (pain score P < .001, asthma severity P < .01, and asthma discharge plan P < .01). However, although pain scores were more frequently recorded, there was no corresponding increase in the number of children who were given analgesia.⁵⁹

The management of community-acquired pneumonia (CAP) was evaluated by Jones et al.⁵⁰ and Niemi et al.⁵¹ Introduction of a real-time CCDSS to assess CAP severity and make management recommendations demonstrated positive results when outcomes were compared across 4 EDs.⁵⁰ The appropriateness of hospitalization was increased (P = .02) and a reduction in inpatient mortality was demonstrated (P < .02). Conversely, in the study by Niemi et al.,⁵¹ which first assessed the ability to detect CAP and/or heart failure and then measured compliance against national indicators for CAP and heart failure combined, the results were less convincing. The only statistically significant result was the provision of discharge advice for patients with heart failure (P < .01).

The final 2 before and after studies considered the impact of CCDSS adherence on national guidelines for syncope⁶⁰ and imaging following neck trauma.⁵⁸ Both studies demonstrated positive results in adherence to admission guidelines in syncope⁶⁰ and reduction in the number of inappropriate neck images ordered following neck trauma.⁵⁸

Although the results of the majority of these studies appear encouraging and justify the use and introduction of CCDSSs in EDs, they should be viewed with caution. There is a threat to internal validity and an intrinsic risk of bias in before and after studies. Indeed, several authors discount the method as a means of contributing to the CCDSS evidence base, as the risk of bias is unacceptably high.^56,61

Interrupted time series studies that have assessed the use of CCDSSs in EDs

Five studies used an interrupted time series design.^{43,45,46,52,53} ITS studies collect data at multiple time points before and after implementation of an intervention.⁶² Collection of data at multiple points before the intervention reveals the underlying secular trend, which will have naturally occurring variation. The analysis, which compares the data points after implementation of the intervention, takes account of this underlying trend to enable the truer effect of the CCDSS to be revealed.⁶¹

Buising et al.⁵² and Gibbs et al.⁴³ evaluated the impact of a CCDSS on CAP at multiple time points. Both studies were concerned with improving the correct antibiotic selection for CAP. Gibbs et al.⁴³ also evaluated the timeliness of antibiotics. Both studies demonstrated statistically significant improvements in appropriate antibiotic selection; Gibbs et al.⁴³ also demonstrated an improvement in the number of antibiotics given within 6 hours (P < .001) when a CCDSS was introduced to aid clinicians in assessment of patients with pneumonia.

The ITS studies by Schriger⁴⁵ and Day et al.⁵³ were conducted in the same academic ED. Both studies measured various elements of guideline adherence, documentation, and provision of aftercare instructions. All measures increased during the implementation phase and returned to or were below baseline when it was removed. There were also improvements in the documentation (P < .001) and discharge advice (P < .001) for patients with back pain.⁶³ However, there was no statistically significant difference in X-rays ordered, medication use, and cost of care.

Overall, the results from the studies using the ITS design appear encouraging. However, unless such studies are rigorously conducted to take into account the effect of confounding variables and the underlying secular trend, it is difficult to draw firm conclusions about the true impact of CCDSS in the ED. The study by Buising et al.⁵² was the only ITS study to consider the underlying trend and does demonstrate that a properly conducted ITS design is an appropriate method for investigating the impact of a CCDSS.

Prospective observational studies that have assessed the use of CCDSS in EDs

One study used a prospective observational design to analyze the effectiveness of a CCDSS in use of the Canadian Acuity and Triage Scale (CTAS).⁶⁴ A convenience sample of 693 patients were triaged by the usual “memory-based” method by the triage nurse who was on duty. Patients were then “re-triaged” by a blinded research nurse using the CTAS CCDSS. The results of both triage decisions were then compared using kappa statistics. Agreement was poor (κ = 0.202). An expert panel assessed 100 triage records, and there was more agreement between the experts and the CCDSS than the triage nurses. The results suggest that a CTAS CCDSS might support better triage decisions, according to expert panel views, than the usual triage method.

However, there are some weaknesses in how the study was conducted. This study did not consider any confounding factors, eg, triage nurse experience. Moreover, through its design it actually introduced confounding factors itself. For example, the research nurse using the CTAS CCDSS triaged the same patients some time after their initial triage and when they had already been directed to a clinical area in the ED for their subsequent care, eg, majors, resuscitation, minors. This process eliminated the time pressure that triage nurses face when having to make rapid decisions, often in an environment fraught with interruptions. The presence of this detection bias does not allow any firm conclusions to be drawn regarding the results of this study.

Comparative cohort studies that have assessed the use of CCDSSs in EDs

Finally, the last study in this review used a retrospective comparative cohort design to evaluate the impact of use of an electronic clinical practice guideline (eCPG) on the management of patients with neutropenic sepsis.⁶⁵ Outcome data were retrospectively extracted from clinical records across 4 EDs in Canada over a 3-year period. Overall, use of the eCPG was low, at 37.8% overall, although in the intervention ED it was 57%. When the eCPG was used, there was a statistically significant improvement in ECG recording and in collection of blood cultures. There were statistically significant reductions in triage to doctor assessment time and triage to first antibiotic. Again, this study, like many others in this review, failed to address confounding factors. An important factor overlooked in this study was the experience of clinical staff in managing the care of patients with neutropenic sepsis.

DISCUSSION

This substantive review identified and critically appraised 23 studies that evaluated the impact of CCDSSs on care in EDs. The results of 13 of the studies identified a statistically significant positive impact on aspects of clinical care with use of a CCDSS.^{39,41,43,45,47–50,52,55,58,59,65} Two studies showed no benefit after introduction of the CCDSS.^40,54 The remaining 8 studies showed small tangential improvements, mainly concerned with documentation.^{42,44,46,51,53,57,60,64} While more than half of the included studies showed favorable results, a brief analysis of the methodological quality revealed a high risk of bias in all but 6 studies.^{41,48–50,52,60} Only one RCT adequately addressed performance and detection bias.⁴⁹ Only 1 of the 5 ITS studies considered the underlying secular trend within the analysis.⁵² And of the before and after studies, fewer than one-third considered and statistically adjusted for confounding variables.^41,48,50,60

Systematic reviewers have been highly critical of the lack of robust evidence on the effectiveness of eHealth interventions.⁶⁶ A substantial review by the Agency for Healthcare Research and Quality drew similar conclusions.⁶⁷ They identified strong evidence for improvements in the process of care when CCDSSs were used, but little evidence of a positive effect on patient outcome or cost.⁶⁷ When considering the quality and safety benefits of IT in ED, Handel et al.⁶⁸ also determined that the evidence is mixed. The critical appraisal of studies in this review supports this position. The evidence is weak regarding the impact of CCDSSs in EDs due to the inherent risk of bias in the design of most studies and/or the inability to control for or address confounding variables

This review of the current primary research on the effectiveness of CCDSSs in EDs has identified the body of evidence and the research methods used. Firm conclusions cannot be drawn due to weak study designs. Of the studies with higher-quality designs,^{41,48–50,52,60} results are more promising in terms of improving the process of care. In the management of pulmonary embolism, both Roy et al.⁴⁹ and Raja et al.⁴⁸ demonstrated improved guideline adherence and more appropriate diagnostic test ordering. Melnick et al.⁶⁰ also demonstrated improved diagnostic test ordering in the management of syncope. Enhanced quality of care as judged by improved documentation was demonstrated by Kwok et al.⁴¹ in the management of asthma. Finally, in 2 studies of CAP, improved antibiotic prescribing was demonstrated,⁵² and, most significantly, patient outcome was improved in the study by Jones et al.,⁵⁰ who demonstrated a reduction in inpatient mortality.

Limitations

The rigor of this review was limited, as a single researcher (PB) constructed the search critieria, reviewed the studies for inclusion/exclusion, and extracted the data. Any papers where decisions were not easily reached were discussed with NH, who also advised on data extraction processes.

CONCLUSION

Clinicians face ongoing and ever-increasing challenges in the delivery of consistently high-quality care in emergency care settings. Research in other settings would suggest that CCDSSs might help to address some of these challenges. This is the first review to consider the effectiveness of CCDSSs specifically in EDs. The review has revealed an increasing body of literature but continuing shortcomings in research design and a predominance of before and after studies. Although such studies are relatively easy to implement in a clinical environment, they are fraught with challenges; the threats to internal validity do not permit confident conclusions to be drawn about causal relationships. The results of the higher-quality studies included in this review are encouarging, but more high-quality evidence is required for the role of CCDSSs in enhancing quality and safety in EDs to be more fully understood.

FUNDING

There was no funding associated with this review.

COMPETING INTERESTS

There are no competing interests.

CONTRIBUTORS

PB and NH designed the review. PB undertook the literature searching, analyzed the paper, revised the review, and approved the final version. NH critically reviewed the drafts and reviewed papers for inclusion. PB is accountable for all aspects of the work.