Real-Time Emergency Department Electronic Notifications Regarding High-Risk Patients: A Systematic Review

Abstract

Background: To systematically review evidence on the feasibility and efficacy of real-time electronic notifications about patients at high risk of emergency department (ED) recidivism.

Methods: Eight electronic databases were searched for empirical studies of real-time ED-based electronic tools, identifying adult patients at high risk of frequent utilization. Study selection and data extraction were performed independently by two reviewers. Qualitative data synthesis and assessment of strength of evidence were conducted through consensus discussion.

Results: Of 2,256 records found through the search, 210 were duplicates, 2,004 were excluded based on abstract review, and 31 were excluded after full text review. The final sample consisted of 10 studies described in 11 articles describing the effect of real-time ED-based electronic notifications for high-risk patients. Three were randomized controlled trials (RCTs). All notifications were based on prespecified markers of risk. Seven studies integrated complex care plans into the electronic health record. Effect on ED use and length of stay (LOS) was mixed: nine studies reported decreased ED use, although results were statistically significant in only three studies; for LOS, one study reported a statistically significant reduction. Impact on cost and financial metrics was promising, with three (of three studies reporting this metric) showing improved organizational financial metrics. Three RCTs reported a reduction in opioid prescriptions.

Conclusions: Real-time electronic notifications of ED providers regarding patients at high risk of ED recidivism are feasible. They may help reduce resource utilization and costs. Large knowledge gaps remain regarding patient- and provider-centered outcomes.

Introduction

Asmall number of patients represent a disproportionate fraction of emergency department (ED) visits in the United States.¹ Certain patient populations—notably those with history of frequent ED use, chronic pain diagnoses, and multiple chronic conditions—are more likely to have frequent ED use.^2,3

Although definitions of frequent ED use and high utilization vary,^1,4–8 patients meeting any definition of high ED utilization are at risk of multiple negative outcomes. For instance, compared with nonfrequent users, frequent ED users have higher healthcare costs (both within and outside the ED),^1,9,10 greater physical and mental disease burdens,¹ and higher ratings of acute illness severity.⁵ They may increase system overload and provider burnout.¹¹ Individualized case management, care plans, and integration of nonmedical services may improve these patients' health and reduce resource use.^12–14 Systematic reviews of interventions targeting frequent ED utilization have shown some evidence of benefit.^15,16

One of the challenges of these interventions, however, is incomplete application of the intervention, often because of difficulty in identifying both patients' high-risk status and appropriate resources. Real-time electronic clinical decision support applications are increasingly used to reduce providers' cognitive load, increase adherence to evidence-based care, and improve patient safety and outcomes for a variety of conditions.^17–24 Real-time clinical decision support for ED patients at risk of recurrent visits may similarly improve health outcomes.²⁵

To advance policy and practice, it is important to know the degree to which these automated alert systems may improve care and outcomes, and to identify which types of alerts (if any) are effective. This systematic review, therefore, aimed to summarize evidence on feasibility and efficacy of real-time electronic notifications, including predictive analytics and decision support applications, for adult patients at high risk of ED recidivism.

Materials and Methods

Search Strategy

Per guidelines on conduct and reporting of systematic reviews,^26,27 a systematic search of eight electronic databases—PubMed, CINAHL, PsycINFO, SocINDEX, Academic Search Premier, Embase, the Cochrane Central Register of Controlled Trials (CENTRAL), and OVID Nursing—was developed and conducted in consultation with a medical librarian to identify potential studies through April 2017. Additional studies were identified through hand search of eligible articles' bibliographies. The search strategy for one of the databases is shown in the supplementary materials (Supplementary Table S1). The review protocol is registered, PROSPERO 2017: CRD42017065388.

Study Selection

Eligibility criteria included being conducted in the ED; comparing patient outcomes from using (vs. not using) real-time electronic notifications of ED recidivism risk; including adult (at least 18 years old) patients at risk of “high ED utilization” according to each study's definition; and presenting real-time electronic notifications at the time of the ED visit. Any study design was eligible. All outcomes, primary and secondary, were of interest.

Studies of tools that were not available in real time or those used for triage, classification of specific disease states, or identification of medication adverse events were excluded. We imposed no time or geographic restrictions.

Three reviewers (H.K., Y.B., and M.R.) screened a random sample of 200 citations to ensure uniform application of criteria. The remaining citations were screened in duplicate and independently by two reviewers (H.K. and Y.B.), using the open-source online software Abstrackr.²⁸ Full-text articles were assessed independently for eligibility by two reviewers (H.K. and Y.B.). Disagreements were resolved through arbitration by the senior author.

Data Extraction, Analytic Approach, and Assessment of Risk of Bias

A data extraction sheet was used to extract relevant information including (1) study characteristics; (2) ED setting characteristics; (3) exposure characteristics, such as whether the tool was integrated into an electronic health record (EHR) and how data were presented; (4) outcome measures and effect sizes; and (5) study definition of high utilization and high recidivism. Data abstraction was carried out independently by two reviewers (H.K. and Y.B.).

The outcomes were then discussed by the entire team. Given the heterogeneity of study elements and outcome measures, a qualitative synthesis of data was conducted.

We assessed risk of bias of randomized controlled trials (RCTs) by using the Cochrane Risk Of Bias In Non-randomized Studies—of Interventions (ROBINS-I) tool, which involves examining characteristics of study design, conduct, analysis, and reporting.^29,30

Results

Study Selection

A total of 2,256 records were identified: 1,422 from PubMed and 834 from the additional electronic databases and hand-searched citations (Fig. 1). After excluding 210 duplicates and 2,004 records not meeting selection criteria, 42 full-text articles were reviewed for eligibility, of which 31 were excluded. The final analytic sample consisted of 11 publications of 10 unique studies published between 2009 and 2017. Findings were summarized at the study level rather than by publication. These 10 studies comprised three RCTs and seven observational assessments. Nine of the 10 studies were conducted in the United States and 1 in the United Kingdom.

Fig. 1.

PRISMA flow diagram for study sample construction. Source: Adapted from Moher et al. (2009). PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses; ED, emergency department; RCTs, randomized controlled trials.

Study Characteristics

Select key characteristics for the study sample and for study outcomes are summarized in Tables 1 and 2.

Table 1.

Summary of Study Characteristics and Evidence Map

STUDY DESIGN	AUTHOR(S) (YEAR)	STUDY GOAL	NO. OF ANALYZED FOLLOW-UP	UTILIZATION OUTCOMES					COST AND FINANCIAL OUTCOMES
				ED USE	IP/OP VISIT READMIT	LENGTH OF STAY	OPIOID USE (O) IMAGING USE (I)	LAB BED DAY	COSTS	FINANCIAL METRICS
RCT	Neven et al. (2016)	To reduce ED visits and decrease controlled substance prescriptions to high utilizers; to assess the ROI of the program	Treatment: 79 Control: 76 12 months	+*			O: +*ED + *D		?	+
	Rathlev et al. (2016)	To assess the impact of electronic alerts on provider opioid prescribing behaviors, patient outcomes, and utilization	Treatment: 20 Control: 20 12 months	−	−IP		O: −ED − IP +*D I: −CT/MRI (ED)		−
	Ringwalt et al. (2015)	To reduce number of ED visits and opioid prescriptions for patients with chronic noncancer pain	Treatment: 13 Control: 16 12 months	+*			O: +* ? ED
Pre/post	Adams and Nielson (2012)	To identify recently discharged psychiatric patients revisiting the ED, and to reduce hospital readmissions and ED visits	NR 12 months	?
	Chan et al.# (2009)	To examine the effect of referral appointment system in the ED on patient access to and adherence with follow-up post-ED visit	Pre: 399 Post: 326 6 months	−
	Hardin et al. (2017)	To reduce healthcare costs and improve outcomes for “high-risk, high-need” patients	Pre: 339 Post: 339 12 months	+*	+*IP + *OP	+*	I: +*CT		+*	+*
	Mercer et al. (2015)	To reduce unnecessary healthcare utilization through individualized care plans for high utilizers of ED and inpatient services	Pre: 24 Post: 23 (6 months), 12 (12 months)	− −	+* IP + *R +* IP + *R	− −			−ED +*IP −ED +IP
	Murphy and Neven (2014)	To examine the cost-effectiveness of computerized reminders about individualized care plans in the ED	Pre: 141 Post: 141 12 months	+*	+*IP +*OP				+*	+*
	Stokes-Buzzelli et al. (2010)	To examine the use of health IT to manage patients frequently presenting to the ED	Pre: 20 Post: 20 24 months	+		−		+*Lab	+
	Stowe (2011)	To evaluate the effect of an automated alert system for cancer patients presenting to the ED	NR 16 weeks			?		?Bed day	?	?

+^* = statistically significant change; + = marginally significant; − = not significant; ? = unclear, p-value not reported. #: study reported significant change in adherence with follow-up.

D, prescribed upon discharge; CT: computed tomography; ED, emergency department, IP, hospital inpatient; MRI, magnetic resonance imaging; NR, not reported; OP, outpatient; R, readmission; RCT, randomized controlled trial; ROI, return on investment.

Table 2.

Synthesis of Study Populations, Interventions, and Results

AUTHOR (S) (YEAR)	SETTING	PATIENT INCLUSION CRITERIA	INTERVENTION DESCRIPTION	OUTCOMES INVESTIGATED	REPORTED RESULTS
Adams and Nielson (2012)	Single ED Psychiatric ED census 6,000	Psychiatric patients returning to the ED within 30 days of inpatient discharge from single included hospital	An alert notified ED psychiatric providers if patients had been discharged from inpatient psychiatric setting within 30 days and provided decisional support through accessing care plans and outpatient services	30-Day psychiatric ED revisit rates	Preintervention 30-day revisit rates: 6.51% (monthly range 1.83–9.53%) Postintervention 30-day revisit rates: 4.3% (monthly range 3.53–5.56%) NB: Statistical testing not reported
Chan et al. (2009)	Single ED ED census 39,000	1. Reside in 15 zip-code regions served by three clinics 2. State they do not have a primary care physician	Electronic referral system with embedded scheduling capabilities in the EHR for follow-up at clinics (IMPACT-ED)	1. Frequency of 2-week follow-up at clinics 2. Frequency of 2-week ED return visits 3. Frequency of referrals during 6-month intervention period to one of three local community clinics from the ED	2-Week absolute change in frequency of follow-up visits with an outpatient provider among patients receiving referrals: +23.8% (95% CI = 19.1–28.6), p < 0.05 Preintervention percentage of patients with 2-week ED return visits: 14.8% (95% CI = 11.3–18.3) Postintervention percentage of patients with 2-week ED return visits: post 11.7% (95% CI = 8.2–15.2), p > 0.05 6-Month change in frequency of referrals: no change
Hardin et al. (2017)	Single ED ED census 80,000	1. Referred to have a Complex Care Map by any hospital, emergency, or community health professional who believed patients could benefit from the program based on their current healthcare utilization 2. 3+ ED visits within 12 months before intervention 3. 18 Years or older	A pop-up alert fired the first time any hospital provider (including ED providers) opened the medical record of a patient with a Complex Care Map; this alert is linked to the patient's care plan	1. Frequency of 12-month healthcare utilization (ED visits; urgent care visits; observation/inpatient admissions LOS; computed tomography scans ordered) 2. Change in 12-month status of social and healthcare access (safe housing; insurance; primary care provider)	12-Month change in total number healthcare visits: −5.581; percentage change: −37.4, p < 0.001 12-Month change in mean number ED visits: −4.419; percentage change: −43.1, p < 0.001 12-Month change in mean number IP visits: −0.575; percentage change: −44.4, p < 0.001 12-Month change in mean number OP visits −0.582; percentage change: −17.3, p < 0.001 12-Month change in mean number of CT scans: −0.918; Percentage change: −62, p < 0.001 12-Month change in mean LOS −0.918; Percentage change: −40.5, p < 0.001 12-Month gross healthcare cost charges ($) Total difference in means = −17.764, p < 0.001 ED difference in means = −6,290, p < 0.001 12-Month stable housing difference in means 11.5; percentage change: 14, p < 0.001 12-Month identifiable PCP difference in means: 12.1; percentage change: 14.9%, p < 0.001 12-Month insured difference in means: 13.0, percentage change: 15.6%, p < 0.001
Mercer et al. (2015)	Multiple EDs 924-bed academic tertiary care center, with 36,000 inpatient discharges annually	1. 3+ ED visits or admissions within 6 months 2. Medical, social, or behavioral complexity (defined as multiple medical comorbidities, concomitant psychiatric illness, substance abuse, homelessness, etc.) determined by a multidisciplinary team of hospital medicine, emergency medicine, psychiatry, ambulatory care, social work, nursing, risk management, and performance services (system analysts)	Multidisciplinary teams created individual care plans for patients in the Complex Care Plan program (those with 3+ ED visits or admissions within 6 months and medical, social, or behavioral complexity). These care plans were embedded in the EHR and became visible when a patient registered in the ED. At the same time they registered, a secure page was sent to the inpatient team; the Complex Care Plan chairperson received an automated e-mail notification of the patient's ED visit at the point of patient ED registration	1. 6- and 12-month ED visits 2. 6- and 12-month inpatient LOS 3. 6- and 12-month inpatient admissions 4. 6- and 12-month 30-day inpatient readmissions 5. 6- and 12-month direct costs (ED and inpatient)	ED visits 6-Month change in total ED visits per enrolled patient: +42.9% (SD = 148.4%), p = 0.836; change in mean ED visits per enrolled patient: 0.7 (SD = 11.92), p = 0.836 12-Month change in total ED visits per enrolled patient: +48.4% (SD = 145.1%), p = 0.941; change in mean number of ED visits per enrolled patient: 10.2 (SD = 43.19), p = 0.941 Inpatient LOS 6-Month change in total inpatient LOS per enrolled patient: −50.8% (SD = 51.4%), p = 0.506; change in mean inpatient LOS per enrolled patient: −0.3 (SD = 4.3) 12-Month change in total inpatient LOS per enrolled patient: −37.8% (SD = 78.8%), p = 0.910; change in mean LOS per enrolled patient: −0.3 (SD = 2.27), p = 0.910 Inpatient admissions 6-Month change in total number of inpatient admissions per enrolled patient: −56.0% (SD = 41.6%), p < 0.001; change in mean number of inpatient admissions per enrolled patient: −3.9 (3.76), p < 0.001 12-Month change in total number of inpatient admissions per enrolled patient: −50.5% (SD = 43.9%), p = 0.003; change in mean number of inpatient admissions per enrolled patient: −6.1 (SD = 6.02) 30-Day inpatient readmissions 6-Month change in total number 30-day readmissions per enrolled patient: −66.0% (SD = 32.4%), p < 0.001; change in mean number of 30-day readmissions per enrolled patient: −3.7 (3.79) 12-Month change in total number of 30-day readmissions per enrolled patient: −51.5% (SD = 32.0%), p = 0.002; change in mean number 30-day readmissions per enrolled patient: −5.1 (5.71), p = 0.002 ED direct costs ($) 6-Month change in total ED costs per enrolled patient: +12.5% (SD = 147.5%), p = 0.143; change in mean ED costs per enrolled patient: −852.4 (2780.01), p = 0.143 12-Month change in total ED costs per enrolled patient: +48.0% (SD = 161.8%), p = 0.850; change in mean ED costs per enrolled patient: 1319.7 (SD = 10348.89), p = 0.850 Inpatient direct costs ($) 6-Month change in total inpatient costs per enrolled patient: −47.7% (SD = 52.3%), p < 0.001; mean inpatient costs per enrolled patient: −14264.90 (SD = 19301.75), p < 0.001 12-Month change in total inpatient costs per enrolled patient: −35.8% (SD = 76.1%), p = 0.052; change in mean inpatient costs per enrolled patient: −19923.2 (SD = 31891.69), p = 0.052
Murphy and Neven (2014)	Single ED ED census 80,000	1. Enrollment in ED care coordination program (Consistent Care Program) based on referrals by ED physicians and/or Medicaid managed care plans 2. Enrollment in program between January 1, 2008 and December 31, 2010 3. Complete data for year before enrollment in program 4. At least three ED visits in year before index date 5. 18 years or older	EHR push notification to the ED provider, stating that patient was a member of the care coordination program and that the patient's care plan existed in the hospital system	1. 12-Month number of ED visits 2. 12-Month ED visit direct treatment cost 3. 12-Month ED visit direct treatment cost per visit 4. 12-Month hospital net income	Extreme users (12 or more ED visits in prior year) 12-Month change in median no. of ED visits: −15 (95% CI −17 to −13), p < 0.001 12-Month change in direct treatment costs: −$6,091; (95% CI −$8,998 to −$4,298), p < 0.001 12-Month change in direct treatment cost per visit: −133 (95% CI −$211 to −$50), p < 0.001 12-Month change in net income: $1,925 (95% CI $1,093 to $3,159), p < 0.001 Frequent users (3–11 ED visits in prior year) 12-Month change in median no. of ED visits = −5 (95% CI −5 to −2), p < 0.001 12-Month change in direct treatment costs: −$1,285 (95% CI −$2,364 to −$492), p < 0.001 12-Month change in direct treatment cost per visit: −$88 (95% CI −$150 to −$33), p < 0.001 12-Month change in net income $431 (95% CI $112 to $878), p < 0.001
Neven et al. (2016)	Multiple EDs Three EDs in separate health systems with combined ED census of 112,000	Selection from aggregated patients from all three hospitals: 1. 5+ ED visits at study hospitals over previous 12 months 2. ≥50% of ED visits for pain complaints or drug-seeking behavior 3. 18+ years old	Multidisciplinary teams created care coordination plans for patients randomized to the intervention. Within 3 min of patient presenting to study ED, automatic fax of the patient-specific 8-point care plan (including recommendations against opioid prescriptions) was sent to the ED provider. In addition, a case manager met with patient when they presented in ED or, if unable to meet in ED, called to follow-up the next day	1. 12-month ED visits (mean number and incidence) 2. Prevented fraction of opioid use (morphine milligram equivalents and opioid prescriptions with a refill, measured at month 10 of intervention) 3. 12-month total treatment costs (third-party) 4. 12-month ROI of intervention	12-Month ED visits: Mean no.: 5.59 (SD 4.65) in intervention vs. 8.49 (SD 7.02) in control, p = 0.0003 OR 0.673, 95% CI = 0.538–0.841 p < 0.001 12-Month opioid prescriptions from the ED: Mean no.: 0.28 (SD = 0.74) in intervention vs. 1.44 (SD 2.05) in control, p < 0.001 OR = 0.21, 95% CI = 0.122–0.353, p = 0.001 Prevented fraction of opioid use at month 10: Morphine milligram equivalents: 43.7% (95% CI = 41.4–45.9), p < 0.001 Opioid prescriptions with a refill: 58.4% (95% CI = −27.8 to 86.5), p = 0.136 Estimated 12-month costs: Estimated 12-month third-party cost difference: $5,785 in intervention vs. $8,985 for control, (−$3,200 cumulative cost difference, SE = 1,345), p = 0.002 Estimated 12-month ROI of intervention: $3.39 per $1 spent (SE = 1.85), p = 0.07
Rathlev et al. (2016)	Multiple EDs Two EDs within the same health system with censuses of 16,000 and 27,000	Monthly review by the High Frequency User Task Force of all potentially eligible patients, defined as:1. 1. 4+ ED visits to Baystate Health System in previous 12 months 2. A history of opioid use disorder (defined by E-codes for opioid overdose, a referral to the High Frequency User Task Force by a treating provider, or supportive evidence from the local EHR or state Prescription Drug Monitoring Program) per review of the prescription Drug Monitoring Program review of the State Prescription Drug Monitoring Program 3. 18+ years old	A push electronic alert appeared the first time any hospital provider (including ED, inpatient, and nursing staff) accessed the chart of a patient who had a care plan developed by the High Frequency User Task Force and who was randomized to intervention	1. Opioids prescribed to patients upon discharge and administered to ED and inpatients 2. 12-Month total medical charges within hospital system 3. 12-Month number of ED visits 4. 12-Month number of ED visits with advanced radiologic imaging 5. 12-Month number of inpatient admissions	Morphine milligram equivalents administered to ED/inpatients: ratio treatment vs. control: 0.29 (95% CI = 0.07–1.12), p = 0.07 Morphine milligram equivalents prescribed to discharged patients: ratio treatment vs. control = 0.11 (95% CI = 0.01–0.92), p = 0.04 12-Month total medical charges: ratio treatment vs. control: 0.92 (95% CI = 0.31–2.7), p = 0.88 12-Month no. of ED visits: treatment mean change = −10.7 (95% CI = −17.5 to −4.0), control mean change −12.8 (95% CI = −19.8 to −5.8), p = 0.68 12-Month no. of ED visits with CT/MRI: treatment mean change = −5.7 (95% CI = −10.0 to −1.4), control mean change −5.8 (95% CI = −9.1 to −2.5), p = 0.98 12-Month no. of inpatient admissions: treatment mean change = −2.6 (95% CI = −5.0 to −0.2), control mean change −1.3 (95% CI = −2.8 to 0.2), p = 0.46
Ringwalt et al. (2015)	Multiple EDs Thirteen of the EDs in a single health system (unknown total number of EDs) with 12 million patient interactions each year	1. 10+ ED visits between October 2010 and September 2011 2. At least two of those visits with a discharge code related to nonspecific subjective pain 3. 18–89 years old	EHR alert to ED providers of patient making multiple visits for treatment of chronic noncancer pain, notifying them that a multidisciplinary team suggests the patient receive care from a community-based provider. The alert also prompted providers to provide a list of community resources for the patient and not prescribe opioids. Letters were also sent to patients and community-based providers stating the patient would no longer receive opioids from the ED.	1. 12-Month number of ED visits 2. Opioid administration (in the ED, and prescribed at discharge)	12-Month no. of ED visits: 11.9 (SD 13.8) in intervention vs. 16.6 (SD 14.8) in control, unadjusted, p < 0.0001 12-Month mean no. of ED visits: 0.690 (95% CI = 0.57–0.84), p < 0.0002 Opioids administered in ED: intervention 16% of visits (95% CI = 0.15–0.18), control 26% of visits (95% CI = 0.25 to −0.28), p-value not reported No. of prescribed opioids, adjusted for baseline opioids: intervention vs. control rate ratio 0.57 (95% CI = 0.46–0.70), p < 0.0001
Stokes-Buzzelli et al. (2010)	Single ED census 95,000	1. Query of 100 highest number of ED visits within ED 2. 18+ Years old 3. Without sickle cell anemia (managed in a separate program)	Multidisciplinary “Community Resources for Emergency Department Overuse [CREDO]” volunteer team summarized (twice per month) all eligible ED patients' medical histories, and created/reviewed individualized care plans for them that were embedded in the EHR. These patients were flagged in the EHR so that all other providers had access to the previously determined care plan	1. 6-Month number of laboratory studies ordered 2. 6-Month total number of visits 3. 6-Month total ED contact time 4. 6-Month average LOS 5. 6-Month total ED charges	6-Month no. of laboratory studies ordered: preintervention mean: 1,847 (SD = 1,826); postintervention mean = 1,328 (SD = 1,191), (95% CI = −1,252 to −26) p = 0.04 6-Month average no. of ED visits: preintervention mean = 67.4 (SD = 47.4); postintervention mean = 50.5 (SD = 49.0), (95% CI = −33 to −0.3), p = 0.046 6-Month ED LOS (minutes): preintervention mean = 388 (SD = 186); postintervention mean = 342 (SD = 180), (95% CI = −98 to 6) p = 0.08 6-Month total emergency department contact time (hours): preintervention mean = 443.7 (SD = 381.7); postintervention mean = 270.6 (SD = 245.8), (95% CI = −17,072 to −3,701); decrease of 39%, p = 0.003 (Represents a mean of 7.21 days less in the ED) 6-Month total ED charges: preintervention mean = $64,721 (SD = $52,448); postintervention mean = $49,208 (SD = $49,239), (95% = CI −$30,943 to $83), p = 0.049
Stowe (2011)	Single ED Census not specified	Any cancer patient presenting to a single ED in a healthcare trust	RAPA system to provide real-time notifications to nominated health workers when known cancer patients presented to the ED	1. No. of alerts received in 16 weeks 2. LOS for nonelective admissions for 16 weeks 3. Hospital system savings (British pound and bed-days) 4. User experience	16-Week total no. of alerts received: 155 16-Week estimated no. of bed-days saved: 650 (Results reported for lung cancer patients only) Projected annual savings: 143,000 pounds Qualitative data: positive provider experience, system promoted better communication between outpatient providers and ED providers

CI, confidence interval; CT, computed tomography; ED, emergency department; EHR, electronic health record; IP, hospital inpatient; LOS, length of stay; MRI, magnetic resonance imaging; OP, hospital outpatient; RAPA, recurring admissions patient alert; ROI, return on investment; SD, standard deviation; SE, standard error; TAU, treatment as usual.

Patient Population and Definition of “High Risk” Status

Study sample size ranged between 12 and 399 participants. Participants' age ranged between 18 and 80 years. Diagnoses included cancer, noncancer pain, opioid use disorder, substance use, psychiatric conditions, and other chronic illnesses. Definitions and means of identification of “high risk” ED patients varied widely. Three studies used predefined static risk markers—for example, cancer diagnosis, recent discharge from a psychiatric hospital, patient residential zip-code, and lack of a primary care physician (PCP).^31–33 Seven studies preidentified eligible patients based on multidisciplinary committees' review of ED records of potential “high utilizers.” Inclusion criteria for review by the multidisciplinary committees included referrals from providers for possible inappropriate ED use or high social/behavioral care complexity,^34–36 absolute number of ED visits,^34,36–40 and history of opioid use disorder, or at least half of the patient's annual ED visits attributed to pain or drug-seeking behaviors.^39–41

Intervention Features

All interventions consisted of electronic notifications integrated directly into patients' EHR. Four studies reported the types of EHR used: CarePlus, Allscripts, Epic, or Cerner FirstNet.^32,36,38,41 In seven studies,^34–39,41 the notification consisted of an alert to the ED provider that a patient had an individualized complex care plan that had been created by a multidisciplinary team. In two studies, the EHR alert was exclusively presented to ED-affiliated specialists.^31,32 Additional components, such as secure pages sent to the inpatient team,³⁶ e-mails sent to members of the multidisciplinary care team,³⁶ letters to patients and PCPs,^37,39 notification to an ED case manager,³⁹ and community resources,^32,33 were in four interventions. One intervention suggested to the ED provider that a patient be given a PCP appointment, and allowed ED providers to access local clinic scheduling software.³³ No studies described how providers were trained in the intervention.

Outcomes Investigated

All 10 studies reported healthcare utilization outcomes extracted directly from the EHR: ED use (n = 10)^{31–33,35–41}; inpatient and outpatient visits (n = 4)^33,34,36,41; 30-day hospital readmissions (n = 1)³⁶; ED or hospital length of stay (LOS) (n = 4)^31,34,36,38; use of and prescription of opioids (n = 3)^37,39–41; and use of diagnostic imaging and laboratory tests (n = 3).^34,38,41 Seven studies evaluated healthcare cost^{31,34–36,38,40,41} and four evaluated other organizational financial performance metrics.^31,34,35,40 One study reported participants' access to medical insurance, primary care, and safe housing access.³⁴ One study reported opioid prescriptions.^39,40 No studies analyzed patient-reported outcomes. One study qualitatively examined provider experience with the system³¹; none included quantitative measurements of provider utilization or provider workflow. The total length of follow-up ranged from 16 weeks to 24 months, with most studies (n = 7) assessing outcomes at 12 months.^{32,34–37,39,41}

Synthesis of Evidence

The study team identified three key dimensions of study outcomes (see also Tables 1 and 2).

Feasibility of Real-Time Ed-Based Electronic Notifications about High-Risk Adult Patients

All 10 studies supported the feasibility of real-time ED-based electronic notifications, reporting that they successfully used an at-risk patient's ED visit to trigger real-time notifications. No studies reported intervention-level feasibility data, such as the proportion of at-risk patients for whom notifications were actually delivered, or the proportion of alerts that were viewed or utilized.

Efficacy of Real-Time ED-Based Electronic Notifications about High-Risk Adult Patients

Utilization outcomes

Nine of the10 studies reported a decrease in ED use with intervention implementation (Table 2).^32–39,41 This decrease in ED use was only statistically significant in three studies.^34,35,37 One study reported a marginally significant effect (p = 0.046)³⁸; one did not provide inferential statistics³²; in three studies, the effect failed to reach statistical significance.^33,36,41 Only one study reported ED LOS, suggesting a reduction after intervention implementation.³⁸

Regarding non-ED-specific outcomes, two of three studies suggested a reduction in hospital admission with intervention implementation,^34,36 and one showed no effect on inpatient admission frequency.⁴¹ The interventions did not consistently reduce hospital LOS: two studies had no significant findings,^36,38 one claimed a reduction but failed to report significant threshold,³¹ and only one study showed a statistically significant reduction.³⁴ One study reported no change in rate of referrals to primary care, but increased rates of primary care follow-up,³³ and one reported increased use of a primary care provider after intervention implementation.³⁴

The three RCTs suggested that electronic notifications may reduce the number of opioid prescriptions provided, both in the ED and upon discharge.^37,39,41

Healthcare Cost and Organizational Financial Metrics

Three of the six studies that assessed cost reported a reduction in direct treatment cost,³⁵ gross charges,³⁴ and average daily bed cost.³¹ One reported a marginally significant reduction in total ED charges,³⁸ whereas another suggested that the intervention significantly reduced direct inpatient but not ED cost.³⁶ One study concluded that there was no statistically significant difference in total medical charges.⁴¹ One study found a marginally significant return on investment of the intervention such that every dollar spent on the intervention, by 1 year, would have a $3.39 net benefit for third-party payers.⁴⁰ Among the three studies that examined organizational financial metrics, all reported improvement, such as (1) net income,³⁵ (2) contribution and operating margin as well as total direct expenses,³⁴ and (3) projected annual cost-savings.³¹ Only two studies reported cost of implementation and maintenance of the intervention.^35,40

Patient- And Individual Provider-Centered Measures and Other Outcomes

One study reported improvement in proportion of patients with medical insurance and safe housing.³⁴ None of the included studies reported the effect of real-time electronic notifications on patient- and individual provider-centered outcomes, such as patient morbidity and mortality. None of the included studies reported on harms or unintended consequences.

Assessment of Risk of Bias

The three RCTs presented low-to-moderate risk of bias, due to nonblinding of participants and personnel, which was inevitable due to the nature of the interventions (Appendix Table A1). The seven observational (pre/postdesign) studies were lower quality, exhibiting moderate-to-high risk of bias (Appendix Table A2). None of the observational studies had a concurrent comparison group, subjecting them to a high risk of regression to the mean (RTM). Participant selection process was at high risk of bias in five of the seven observational studies, due to lack of systematic processes for participant inclusion. All observational studies had a high risk of incomplete outcome data (including not fully capturing visits to other EDs) and selective outcome reporting (notably lack of clarity of study aims).

Discussion

This systematic review summarizes evidence on the feasibility and efficacy of real-time ED-based electronic notifications—including real-time electronic decision support—about adult patients at high risk of ED recidivism. Ten studies were identified from 11 articles, assessing >40 outcomes. All 10 studies support the feasibility of such interventions. Regarding efficacy, interventions' effect on ED use, LOS, number of opioids prescribed, and imaging use was promising but mixed. The effect on inpatient and outpatient visits, cost, and organizational financial metrics was more consistently favorable. Overall, there is low-to-moderate evidence that real-time ED-based electronic notifications about high-risk patients improve healthcare utilization and organizational financial performance. This review, therefore, highlights both the potential benefits of health informatics for the management of high-risk patients in the ED and the large knowledge gaps regarding patient- and provider-centered outcomes and feasibility of real-time calculations of patient risk.

The benefits of EHR-integrated complex care plans, as described in seven studies,^34–39,41 are not surprising. Individualized care plans implemented in other care settings reduce excessive utilization and improve patient health outcomes.^12,13,42 In the interventions in this review, care plans were directly integrated into the EHR, and push notifications or alerts were delivered to the ED provider at the time of treatment, often in conjunction with availability of intensive case management services. Although only measured in three studies,^37,39,41 the decrease in the number of opioid prescriptions provided after an ED visit is noteworthy—given the current epidemic of opioid overdoses in the United States.^43,44 As shown in the 2014 study by Murphy et al.,³⁵ the impact of these alerts was greatest among patients with the highest baseline values for annual ED visits. This suggests that, among the subset of highest risk among the already high-risk population, utilization patterns may be influenced by incorporation of provider notification of specific care needs for those patients. The effect of these alerts on patient-level health outcomes (such as patient quality of life, mortality, and morbidity) is yet to be determined. Future research should include assessment of more patient-centered outcomes.

This review also underscores the need for more comprehensive measures to assess usage, usability, and acceptability of these technologies. Compared with non-EHR-integrated interventions, real-time alerts may operate through various mechanisms to influence outcomes. Real-time alerts may increase provider attention to individualized care plans, may improve provider awareness of care guidelines, may decrease provider cognitive overload, or may synthesize sources of data unavailable to provider in real time in the ED setting.^45,46 Unfortunately, none of the studies rigorously evaluated workflow, provider usage, provider behavior change, or provider acceptability—despite ample evidence from other researchers that EHR alerts are often overridden, ignored, or designed in a way that is unusable by clinicians.^47,48

Although four of the studies provided examples of the alert that was shown to providers,^32,34,38,39 only one reported that providers were educated on the system, and if guidelines were not followed, the providers were contacted.³⁹ The remaining three studies did not report their method of training or introduction of the tools to providers.^32,34,38 Dissemination of these types of interventions would be improved by detailed examination of best practices for provider education and by qualitative assessments of usefulness of various forms of presentation of information to providers. Future research should incorporate experiential data gathered, for example, from analysis of EHR timestamps and number of “clicks,” ethnographic observations, and focus groups.

Although not explicitly reported in these reviewed studies, implementation and upkeep of patient information in the interventions may have been time intensive. For instance, seven studies reported using multidisciplinary team meetings—often in combination with provider referrals and EHR searches for high utilizers—to identify patients who should get a care plan and create these plans.^34–41 These systems may be difficult to disseminate, implement, and maintain. Exploration of more efficient and innovative ways to identify at-risk patients is needed. For example, retrospective and prospective risk models predicting future ED use have been developed using administrative and treatment data sets.^2,3,49 Application of such models in real time may influence the ability to disseminate these types of interventions. It may also be possible to create predictive models using big data that are more nuanced than those applied by many of the studies in this review to identify patients at their moment of transition to poor health rather than after they have already demonstrated opioid misuse or frequent ED utilization. Future research should explore the development and integration of such real-time modeling of patient risk and compare automated predictive analytics' accuracy to that of in-person teams.

Finally, future research should monitor potential harms and unintended consequences of these notifications. Real-time notifications about at-risk patients may increase stigma, restraining patients from seeking care.⁵⁰ As a result, reported reduction in ED use may reflect reduction in necessary as well as “unnecessary” care. Therefore, future research should include assessments of potential negative outcomes from such notifications and explicitly discuss how potential patient stigmatization is minimized.

Limitations

Heterogeneity in patient populations, timeline for outcome assessments, and definition of outcome measures create difficulty in comparing findings among studies. A lack of longitudinal data raises concerns about RTM among patient populations. Moreover, the use of pre/postdesigns suggests potential contamination of effects from the intervention to the control group as provider behavior may have been altered across the conditions. The lack of data from other hospitals not participating in the study may bias results toward stronger treatment effects as patients may have more frequently visited EDs where the interventions were not implemented. Finally, it is possible that it is the presence of care plans, rather than their integration into the EHR, which influenced measured outcomes.

In the same manner, it is possible that the present review was limited by the specific terms used in the search and databases used to conduct the search itself. In addition, unpublished data and gray literature were not searched, which may increase publication bias. Finally, because of the small number of retrieved studies, the review team was unable to conduct a meta-analytic assessment of the body of evidence, and as such, all results are qualitative.

Conclusion

Preliminary studies, from an emerging body of literature, support feasibility and potential utility of integrated real-time electronic notifications of ED providers about patients at high risk of ED recidivism. However—although there is evidence of efficacy in improving organizational financial performance, in promoting social outcomes, and in reducing cost and healthcare utilization, including ED use, inpatient and outpatient visit—the true utility of these interventions is yet to be determined. Determining utility would require, for instance, assessment of available community resources, provider use of the notifications, and benefits measured from a societal perspective. Future studies should explore automated risk prediction and comprehensive patient- and provider-level outcome measures.

Appendix

Appendix Table A1.

Risk of Bias Summary from Included Randomized Controlled Trials

AUTHOR(S) (YEAR)	RANDOM SEQUENCE GENERATION	ALLOCATION CONCEALMENT	RESIDUAL CONFOUNDING	BLINDING OF PARTICIPANTS AND PERSONNEL	BLINDING OF OUTCOME ASSESSORS	INCOMPLETE OUTCOME DATA	SELECTIVE REPORTING	OTHER BIASES	OVERALL RISK OF BIAS RATING
Neven et al. (2016)	+	−	−	+A	?	−	−	+	Low-to-moderate
Rathlev et al. (2016)	−	−	−	+A	−	?	−	+	Low-to-moderate
Ringwalt et al. (2015)	−	+	−	+A	−	−	−	+	Low-to-moderate

Domain-based evaluation adapted from the Cochrane Collaboration risk of bias table.

− = low risk of bias; + = high risk of bias; ? = unclear; A, not possible to blind participants due to nature of intervention.

Appendix Table A2.

Risk of Bias Summary from Included Observational Studies

AUTHOR(S) (YEAR)	SELECTION OF PARTICIPANTS	GROUP ALLOCATION	RESIDUAL CONFOUNDING	DEVIATION FROM INTENDED INTERVENTION	MEASUREMENT OF OUTCOMES	INCOMPLETE OUTCOME DATA	SELECTIVE REPORTING	OTHER BIASES	OVERALL RISK OF BIAS RATING
Adams and Nielson (2012)	+	+	+	−	+	+	+	+	High
Chan et al. (2009)	+	+	+	−	−	+	?	+	Moderate-to-high
Hardin et al (2017)	+	+	+	−	−	+	?	+	Moderate
Mercer et al. (2015)	+	+	+	−	−	+	?	+	Moderate-to-high
Murphy and Neven (2014)	+	+	+	−	−	+	?	+	Moderate
Stokes-Buzzelli et al. (2010)	−	−	+	−	−	+	?	+	Moderate
Stowe (2011)	−	+	+	−	+	+	+	+	High

Domain-based evaluation adapted from the Cochrane Collaboration risk of bias table.

− = low risk of bias; + = high risk of bias; ? = unclear.

Authors' Contribution

M.L.R. conceived of the study and takes responsibility for the study as a whole. M.L.R., H.K., and Y.B. made substantial contributions to the design of the work, the acquisition, analysis, and interpretation of data for the work, drafting and revision of the work, and final approval of the version to be published. T.A.T. and I.N.S. made substantial contributions to the design of the work, interpretation of data for the work, critical revisions of the article for important intellectual content, and final approval of the version to be published.

Disclosure Statement

No competing financial interests exist.

Supplementary Material

Supplementary Table S1