Diagnostic Accuracy and Effectiveness of Automated Electronic Sepsis Alert Systems: A Systematic Review

Abstract

Background

Although timely treatment of sepsis improves outcomes, delays in administering evidence-based therapies are common.

Purpose

To determine whether automated real-time electronic sepsis alerts can: 1) accurately identify sepsis, and 2) improve process measures and outcomes.

Data Sources

We systematically searched MEDLINE, Embase, The Cochrane Library, and CINAHL from database inception through June 27, 2014.

Study Selection

Included studies that empirically evaluated one or both of the prespecified objectives.

Data Extraction

Two independent reviewers extracted data and assessed the risk of bias. Diagnostic accuracy of sepsis identification was measured by sensitivity, specificity, positive (PPV) and negative predictive values (NPV) and likelihood ratios (LR). Effectiveness was assessed by changes in sepsis care process measures and outcomes.

Data Synthesis

Of 1,293 citations, 8 studies met inclusion criteria, 5 for the identification of sepsis (n=35,423) and 5 for the effectiveness of sepsis alerts (n=6,894). Though definition of sepsis alert thresholds varied, most included systemic inflammatory response syndrome criteria ± evidence of shock. Diagnostic accuracy varied greatly, with PPV ranging from 20.5-53.8%, NPV 76.5-99.7%; LR+ 1.2-145.8; and LR- 0.06-0.86. There was modest evidence for improvement in process measures (i.e., antibiotic escalation), but only among patients in non-critical care settings; there were no corresponding improvements in mortality or length of stay. Minimal data were reported on potential harms due to false positive alerts.

Conclusions

Automated sepsis alerts derived from electronic health data may improve care processes but tend to have poor positive predictive value and do not improve mortality or length of stay.

Introduction

Sepsis is the most expensive condition treated in the hospital, resulting in an aggregate cost of $20.3 billion or 5.2% of total aggregate cost for all hospitalizations in the United States.¹ Rates of sepsis and sepsis-related mortality are rising in the United States.^{2, 3} Timely treatment of sepsis, including adequate fluid resuscitation and appropriate antibiotic administration, decreases morbidity, mortality, and costs.^4-6 Consequently, the Surviving Sepsis Campaign recommends timely care with the implementation of sepsis bundles and protocols.⁴ Though effective, sepsis protocols require dedicated personnel with specialized training, who must be highly vigilant and constantly monitor a patient's condition for the course of an entire hospitalization.^{7, 8} As such, delays in administering evidence-based therapies are common.^{8, 9}

Automated electronic sepsis alerts are being developed and implemented to facilitate the delivery of timely sepsis care. Electronic alert systems synthesize electronic health data routinely collected for clinical purposes in real- or near real-time to automatically identify sepsis based on prespecified diagnostic criteria, and immediately alert providers that their patient may meet sepsis criteria via electronic notifications (e.g., through electronic health record (EHR), e-mail, or pager alerts).

However, little data exist to describe whether automated, electronic systems achieve their intended goal of earlier, more effective sepsis care. To examine this question, we performed a systematic review on automated electronic sepsis alerts to assess their suitability for clinical use. Our two objectives were: 1) to describe the diagnostic accuracy of alert systems in identifying sepsis using electronic data available in real-time or near real-time; and 2) to evaluate the effectiveness of sepsis alert systems on sepsis care process measures and clinical outcomes.

Materials and Methods

Data Sources and Search Strategies

We searched PubMed MEDLINE, Embase, The Cochrane Library, and CINAHL from database inception through June 27, 2014, for all studies that contained the following three concepts: sepsis, electronic systems, and alerts (or identification). All citations were imported into an electronic database (EndNote X5, Thomson Reuters). Our complete search strategy is provided in detail in the eAppendix.

Study Selection

Two authors (ANM and OKN) reviewed the citation titles, abstracts, and full-text articles of potentially relevant references identified from the literature search for eligibility. References of selected articles were hand searched to identify additional eligible studies. Inclusion criteria for eligible studies were: 1) adult patients (≥18 years) receiving care either in the emergency department or hospital; 2) outcomes of interest including a) diagnostic accuracy in identification of sepsis, and/or b) effectiveness of sepsis alerts on process measures and clinical outcomes were evaluated using empiric data; and 3) sepsis alert systems used real-time or near real-time electronically available data to enable proactive, timely management. We excluded studies that: 1) tested the effect of other electronic interventions that were not sepsis alerts (i.e., computerized order sets) for sepsis management; 2) studies solely focused on detecting and treating central line-associated bloodstream infections, shock (not otherwise specified), bacteremia, or other device-related infections; and 3) studies evaluating the effectiveness of sepsis alerts without a control group.

Data Extraction and Quality Assessment

Two reviewers (ANM and OKN) extracted data on the clinical setting, study design, dates of enrollment, definition of sepsis, details of the identification and alert systems, diagnostic accuracy of the alert system, and the incidence of process measures and clinical outcomes using a standardized form. Discrepancies between reviewers were resolved by discussion and consensus. Data discrepancies identified in one study were resolved by contacting the corresponding author.¹⁰

For studies assessing the diagnostic accuracy of sepsis identification, study quality was assessed using the Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) revised tool.¹¹ For studies evaluating the effectiveness of sepsis alert systems, studies were considered ‘high quality’ if a contemporaneous control group was present to account for temporal trends (e.g., randomized controlled trial or observational analysis with a concurrent control). ‘Fair quality’ studies were before-and-after studies that adjusted for potential confounders between time periods. ‘Low quality’ studies included those that did not account for temporal trends, such as before-and-after studies using only historical controls without adjustment. Studies that did not use an intention-to-treat analysis were also considered ‘low quality’. The strength of the overall body of evidence, including risk of bias, was guided by the Grading of Recommendations Assessment, Development, and Evaluation Working Group Criteria adapted by the Agency of Healthcare Research and Quality.¹²

Data Synthesis

To analyze the diagnostic accuracy of automated sepsis alert systems to identify sepsis and to evaluate the effect on outcomes, we performed a qualitative assessment of all studies. We were unable to perform a meta-analysis due to significant heterogeneity in study quality, clinical setting, and definition of the sepsis alert. Diagnostic accuracy of sepsis identification was measured by sensitivity, specificity, positive and negative predictive values and likelihood ratios. Effectiveness was assessed by changes in sepsis care process measures (i.e. time to antibiotics) and outcomes (length of stay, mortality).

Results

Description of Studies

Of 1,293 titles, 183 qualified for abstract review, 84 for full-text review, and 8 articles met our inclusion criteria (eFigure in Supplement). Five articles evaluated the diagnostic accuracy of sepsis identification^{10, 13-16} and five articles^{10, 14, 17-19} evaluated the effectiveness of automated electronic sepsis alerts on sepsis process measures and patient outcomes. All articles were published between 2009 and 2014 and were single-site studies conducted at academic medical centers (Tables 1 and 2). The clinical settings in the included studies varied and included the emergency department (ED), hospital wards, and the intensive care unit (ICU).

Table 1. Characteristics of Studies Evaluating the Diagnostic Accuracy of Automated Electronic Sepsis Alerts.

Source	Site No./Type	Setting	Alert threshold	Gold standard definition	Gold standard measurement	N	Study Qualitya
Hooper et al.,10 2012	1/academic	MICU	≥2 SIRS criteriab	Reviewer judgment, not otherwise specified	Chart review	560	High
Meurer et al.,13 2009	1/academic	ED	≥2 SIRS criteria	Reviewer judgment whether diagnosis of infection present in ED plus SIRS criteria	Chart review	248	Low
Nelson J. et al.,14 2011	1/academic	ED	≥2 SIRS criteria and 2 SBP measurements < 90 mm Hg	Reviewer judgment whether infection present, requiring hospitalization with at least 1 organ system involved	Chart review	1386	High
Nguyen et al.,15 2014	1/academic	ED	≥2 SIRS criteria and 1 sign of shock (SBP≤90 mm Hg or lactic acid ≥ 2.0 mmol/L)	Reviewer judgment to confirm SIRS, shock, and presence of a serious infection	Chart review	1095	Low
Thiel et al,16 2010	1/academic	Wards	Recursive partitioning tree analysis including vitals and laboratory resultsc	Admitted to the hospital wards and subsequently transferred to the ICU for septic shock and treated with vasopressor therapy	ICD-9 discharge codes for acute infection, acute organ dysfunction, and need for vasopressors within 24 hours of ICU transfer	27674	Low

Abbreviations: MICU, medical intensive care unit; SIRS, systemic inflammatory response syndrome; ED, emergency department; SBP, systolic blood pressure; ICD-9, International Classification of Diseases, Ninth Revision

^aAssessed using the Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) revised tool.¹⁰

^bRecorded within a 24 hour period, mandating either abnormal temperature or WBC.

^cIncluded shock index, mean arterial pressure, international normalized ratio, white blood cell count, absolute neutrophil count, bilirubin, albumin, hemoglobin, and sodium.

Table 2. Characteristics of Studies Evaluating the Effectiveness of Automated Electronic Sepsis Alerts.

Source	Design	Site No./Type	Setting	N	Alert system type	Alert threshold	Alert notificationa	Treatment Recommendation	Study Qualityb
Berger et al.,17 2010	Before-after(6 months pre and 6 months post)	1/academic	ED	5796c	CPOE system	≥ 2 SIRS criteria	CPOE passive alert	Yes: lactate collection	Low
Hooper et al.,10 2012	RCT	1/academic	MICU	443	EHR	≥ 2 SIRS criteriad	Text page and EHR passive alert	No	High
McRee et al.,18 2014	Before-after (6 months pre and 6 months post)	1/academic	Wards	171e	EHR	≥ 2 SIRS criteria	Notified nurse, specifics unclear	No, but the nurse completed a sepsis risk evaluation flow sheet	Low
Nelson et al.,14 2011	Before-after (3 months pre and 3 months post)	1/academic	ED	184f	EHR	≥ 2 SIRS criteria and 2 or more SBP readings < 90 mmHg	Text page and EHR passive alert	Yes: fluid resuscitation, blood culture collection, antibiotic administration, “among others”	Low
Sawyer et al.,19 2011	Prospective, non-randomized (2 intervention and 4 control wards)	1/academic	Wards	300	EHR	Recursive partitioning regression tree algorithm including vitals and lab valuesg	Text page to charge nurse who then assessed patient and informed treating physicianh	No	High

Abbreviations: ED, emergency department; CPOE, computerized provider order entry; SIRS, systemic inflammatory response syndrome; RCT, randomized control trial; MICU, medical intensive care unit; EHR, electronic health record; SBP, systolic blood pressure;

^aPassive alerts do not require the provider to acknowledge the alert or take action. Text pages were automatically generated and sent.

^bAssessed by prespecified criteria of study design and intention-to-treat protocol.

^cNot an intention-to-treat analysis. Only patients meeting SIRS criteria with a provider's affirmative response to a computerized query regarding suspected infection were analyzed.

^dRecorded within 24 hour period, mandating either abnormal temperature or WBC.

^eNot an intention-to-treat analysis. Only medical records reviewed of individuals with an ICD-9 discharge code of sepsis

^fNot an intention-to-treat analysis. 398 patients triggered the alert but only the 184 (46%) confirmed to have an admission diagnosis of infection by chart review were included in the analysis.

^gIncluded shock index, mean arterial pressure, international normalized ratio, white blood cell count, absolute neutrophil count, bilirubin, albumin, hemoglobin, and sodium

^hNurses and physicians on intervention wards received sepsis alert education prior to implementation; no education provided to control wards.

Among the eight included studies, there was significant heterogeneity in threshold criteria for sepsis identification and subsequent alert activation. The most commonly defined threshold was the presence of 2 or more systemic inflammatory response syndrome (SIRS) criteria.^{10, 13, 17, 18}

Diagnostic Accuracy of Automated Electronic Sepsis Alert Systems

The prevalence of sepsis varied substantially between the studies depending on the ‘gold standard’ definition of sepsis used and the clinical setting (ED, wards, or ICU) of the study (Table 3). The two studies^{14, 16} that defined sepsis as requiring evidence of shock had a substantially lower prevalence (0.8-4.7%) compared to the two studies^{10, 13} that defined sepsis as having only two or more SIRS criteria with a presumed diagnosis of an infection (27.8%-32.5%).

Table 3. Diagnostic Accuracy of Automated Electronic Sepsis Alerts.

Source	Setting	Alert threshold	Prevalence, %	Sensitivity, % (95% CI)	Specificity, % (95% CI)	PPV, % (95% CI)	NPV, % (95% CI)	LR+, (95% CI)	LR-, (95% CI)
Hooper et al.,10 2012	MICU	≥ 2 SIRS criteriaa	36.3	98.9 (95.7-99.8)	18.1 (14.2-22.9)	40.7 (36.1-45.5)	96.7 (87.5-99.4)	1.21 (1.14-1.27)	0.06 (0.01-0.25)
Meurer et al.,13 2009	ED	≥ 2 SIRS criteria	27.8	36.2 (25.3-48.8)	79.9 (73.1-85.3)	41.0 (28.8-54.3)	76.5 (69.6-82.2)	1.80 (1.17-2.76)	0.80 (0.67-0.96)
Nelson et al.,14 2011	ED	≥ 2 SIRS criteria and 2 SBP measurements < 90 mm Hg	0.8	63.6 (31.6-87.8)	99.6 (99.0-99.8)	53.8 (26.1-79.6)	99.7 (99.2-99.9)	145.8 (58.4-364.1)	0.37 (0.17-0.80)
Nguyen et al.,15 2014	ED	≥ 2 SIRS criteria and 1 sign of shock (SBP≤90 mm Hg or lactic acid ≥ 2.0 mmol/L)	Unable to estimate b	Unable to estimate b	Unable to estimate b	44.7 (41.2-48.2)	100.0 c (98.8-100.0)	Unable to estimate b	Unable to estimate b
Thiel et al.,16 2010	Wards	Recursive partitioning tree analysis including vitals and laboratory resultsd	4.7	17.1 (15.1-19.3)	96.7 (96.5-96.9)	20.5 (18.2-23.0)	95.9 (95.7-96.2)	5.22 (4.56-5.98)	0.86 (0.84-0.88)

Abbreviations: MICU, medical intensive care unit; SIRS, systemic inflammatory response syndrome; ED, emergency department; SBP, systolic blood pressure

^aRecorded within 24 hour period, mandating either abnormal temperature or WBC

^bFalse negative and true negatives unknown due to random sampling of non-alert cases

^cEstimated value based on random sample of 300 non-alert cases

^dIncluded shock index, mean arterial pressure, international normalized ratio, white blood cell count, absolute neutrophil count, bilirubin, albumin, hemoglobin, and sodium

All alert systems had suboptimal positive predictive value (20.5%-53.8%). The two studies that designed the sepsis alert to activate by SIRS criteria alone^{10, 13} had a positive predictive value of 41% and a positive likelihood ratio of 1.21-1.80. The ability to exclude the presence of sepsis varied considerably depending on the clinical setting. The study by Hooper et al.¹⁰ that examined the alert among patients in the medical ICU appeared more effective at ruling out sepsis (NPV=96.7%; negative LR = 0.06) compared to a similar alert system used by Meurer et al.¹³ which studied patients in the ED (NPV=76.5%, negative LR = 0.80).

There were also differences in the diagnostic accuracy of the sepsis alert systems depending on how the threshold for activating the sepsis alert was defined and applied in the study. Two studies evaluated a sepsis alert system among patients presenting to the ED at the same academic medical center.^{13, 14} The alert system (Nelson et al.) that was triggered by a combination of SIRS criteria and hypotension (PPV = 53.8%, LR+ = 145.8; NPV = 99.7%, LR- = 0.37) outperformed the alert system (Meurer et al.) that was triggered by SIRS criteria alone (PPV = 41.0%, LR+ = 1.80; NPV = 76.5%, LR- = 0.80). Further, the study by Meurer and colleagues evaluated the accuracy of the alert system only among patients who were hospitalized after presenting to the ED, rather than all consecutive patients presenting to the ED. This selection bias likely falsely inflated the diagnostic accuracy of the alert system used by Meurer et al., suggesting the alert system that was triggered by a combination of SIRS criteria and hypotension was comparatively even more accurate.

Two studies evaluating the diagnostic accuracy of the alert system were deemed to be ‘high quality’ (Table 4). Three studies were considered ‘low quality’ -- one study did not include all patients in their assessment of diagnostic accuracy;¹³ one study consecutively selected alert cases but randomly selected non-alert cases, greatly limiting the assessment of diagnostic accuracy;¹⁵ and the other study applied a gold standard that was unlikely to correctly classify sepsis (septic shock requiring ICU transfer with vasopressor support in the first 24 hours was defined by discharge ICD-9 diagnoses without chart review) with a considerable delay from the alert system trigger (alert identification was compared to the discharge diagnosis rather than physician review of real-time data).¹⁶

Table 4. Assessment of Bias in Studies Evaluating Diagnostic Accuracy of Automated Electronic Sepsis Alerts^a.

Study	Patient Selection	Index Test	Reference Standard	Flow and Timing
Hooper et al.,10 2012	+++	+++	++b	+++
Meurer et al.,13 2009	+++	+++	++b	+c
Nelson et al.,14 2011	+++	+++	++b	+++
Nguyen et al.,15 2014	+d	+++	+e	+++
Thiel et al.,16 2010	+++	+++	+f	+g

^aDetermined by 2 independent abstractors using the revised QUADAS-2 tool ¹⁰. Three plus signs indicate the lowest risk for bias and one plus sign indicates highest risk for bias

^bUnclear if the gold standard was interpreted without knowledge of the results of the sepsis alert

^cNot all patients accounted for in the study. Only patients in the emergency department who were subsequently hospitalized were subjected to the gold standard and were included in the analysis.

^dConsecutive selection for cases, but random selection of non-cases greatly limited evaluation of diagnostic accuracy

^eGold standard was interpreted with knowledge of the results of the sepsis alert

^fDischarge ICD-9 diagnosis codes unlikely to correctly classify patients admitted to the hospital wards and subsequently transferred to the intensive care unit for septic shock and needing vasopressor support in the first 24 hours.

^gThere was a delay in time between the sepsis alert triggering and ascertainment of the gold standard (discharge ICD-9 diagnoses) which may result in misclassification.

Effectiveness of Automated Electronic Sepsis Alert Systems

Characteristics of the studies evaluating the effectiveness of automated electronic sepsis alert systems are summarized in Table 2. Regarding activation of the sepsis alert, two studies notified the provider directly by an automated text page and a passive EHR alert (not requiring the provider to acknowledge the alert or take action),^{10, 14} one study notified the provider by a passive electronic alert alone,¹⁷ and one study only employed an automated text page.¹⁹ Furthermore, if the sepsis alert was activated, two studies suggested specific clinical management decisions^{14, 17} two studies left clinical management decisions solely to the discretion of the treating provider, ^{10, 19} and one study assisted the diagnosis of sepsis by prompting nurses to complete a second manual sepsis risk evaluation.¹⁸

Table 5 summarizes the effectiveness of automated electronic sepsis alert systems. Two studies evaluating the effectiveness of the sepsis alert system were considered to be ‘high quality’ studies based on the use of a contemporaneous control group to account for temporal trends and an intention-to-treat analysis.^{10, 19} The two studies evaluating the effectiveness of a sepsis alert system in the ED were considered ‘low quality’ due to before-and-after designs without an intention-to-treat analysis.^{14, 17}

Table 5. Effectiveness of Automated Electronic Sepsis Alerts.

Source	Outcomes evaluated	Key Findings	Quality
Hooper et al.,10 2012	Primary: time to receipt of antibiotic (new or changed) Secondary: sepsis-related process measures and outcomes	No difference (6.1 hours for control vs. 6.0 hours for intervention, p=0.95) No difference in amount of 6 hour IV fluid administration (964 mL vs. 1019 mL, p=.6), collection of blood cultures (adjusted HR 1.01, 95% CI, 0.76-1.35), collection of lactate (adjusted HR 0.84, 95% CI, 0.54-1.30), ICU length of stay (3.0 vs. 3.0 days, p=.2), hospital length of stay (4.7 vs. 5.7 days, p=.08), and hospital mortality (10% for control vs. 14% for intervention, p=.3)	High
Sawyer et al.,19 2011	Primary: sepsis-related process measures (antibiotic escalation, IV fluids, oxygen therapy, vasopressor initiation, diagnostic testing (blood culture, CXR) within 12 hours of alert Secondary: ICU transfer, hospital length of stay, hospital length of stay after alert, in-hospital mortality	Increases in receiving ≥ 1 measure (56% for control vs. 71% for intervention, p=.02), antibiotic escalation (24% vs. 36%, p=.04), IV fluid administration (24% vs. 38%, p=.01), and oxygen therapy (8% vs. 20%, p=.005). There was a nonsignificant increase in obtaining diagnostic tests (40% vs. 52%, p=.06) and vasopressor initiation (3% vs. 6%, p=.4) Similar rate of ICU transfer (23% for control vs. 26% for intervention, p=.6), hospital length of stay (7 vs. 9 median days, p=.8), hospital length of stay after alert (5 vs. 6 median days, p=.7), and in-hospital mortality (12% vs. 10%, p=.7).	High
Berger et al.,17 2010	Primary: lactate collection in ED Secondary: lactate collection among hospitalized patients, proportion of patients with abnormal lactate (≥4 mmol/L), and in-hospital mortality among hospitalized patients	Increase in lactate collection in the ED (5.2% before vs. 12.7% after alert implemented, absolute increase of 7.5%, 95% CI, 6.0%-9.0%) Increase in lactate collection among hospitalized patients (15.3% vs. 34.2%, absolute increase of 18.9%, 95% CI, 15.0%-22.8%); decrease in the proportion of abnormal lactate values (21.9% vs. 14.8%, absolute decrease of 7.6%, 95% CI, -15.8% to -0.6%), and no significant difference in mortality (5.7% vs. 5.2%, absolute decrease of 0.5%, 95% CI, -1.6%-2.6%, p=.6)	Low
McRee et al.,18 2014	Stage of sepsis, length of stay, mortality, discharge location	Nonsignificant decrease in stage of sepsis (34.7% with septic shock before vs. 21.9% after, p>.05); No difference in length-of-stay (8.5 days before vs. 8.7 days after, p>.05). Decrease in mortality (9.3% before vs. 1.0% after, p<.05) and proportion of patients discharged home (25.3% before vs. 49.0% after, p<.05)	Low
Nelson et al.,14 2011	Frequency and time to completion of process measures: lactate, blood culture, CXR, and antibiotic initiation	Increases in blood culture collection (OR 2.9, 95% CI 1.1-7.7) and CXR (OR 3.2, 95% CI 1.1-9.5); nonsignificant increases in lactate collection (OR 1.7, 95% CI, 0.9-3.2) and antibiotic administration (OR 2.8, 95% CI, 0.9-8.3). Only blood cultures were collected ina more timely manner (median of 86 minutes before vs. 81 minutes after alert implementation, p=.03).	Low

Abbreviations: IV, intravenous; CXR, chest radiograph; ICU, intensive care unit; ED, emergency department

Neither of the two high quality studies that included a contemporaneous control found evidence for improving inpatient mortality or hospital and ICU length of stay.^{10, 19} The impact of sepsis alert systems on improving process measures for sepsis management depended on the clinical setting. In an RCT of patients admitted to a medical ICU, Hooper et al. did not find any benefit of implementing a sepsis alert system on improving intermediate outcome measures such as antibiotic escalation, fluid resuscitation, and collection of blood cultures and lactate.¹⁰ However, in a well-designed observational study, Sawyer et al. found significant increases in antibiotic escalation, fluid resuscitation, and diagnostic testing in patients admitted to the medical wards.¹⁹ Both studies that evaluated the effectiveness of sepsis alert systems in the ED showed improvements in various process measures,^{14, 17} but without improvement in mortality.¹⁷ The single study that showed improvement in clinical outcomes (in-hospital mortality and disposition location) was of ‘low quality’ due to the pre-post study design without adjustment for potential confounders and lack of an intention-treat analysis (only individuals with a discharge diagnosis of sepsis were included, rather than all individuals who triggered the alert).¹⁸ Additionally, the pre-intervention group had a higher proportion of individuals with septic shock compared to the post-intervention group, raising the possibility that the observed improvement was due to difference in severity of illness between the two groups rather than due to the intervention.

None of the studies included in this review explicitly reported on the potential harms (e.g. excess antimicrobial use or alert fatigue) after implementation of sepsis alerts, but Hooper et al. found a nonsignificant increase in mortality, and Sawyer et al. showed a nonsignificant increase in the length of stay in the intervention group compared to the control group.^{10, 19} Berger et al. showed an overall increase in the number of lactate tests performed, but with a decrease in the proportion of abnormal lactate values (21.9% vs. 14.8%, absolute decrease of 7.6%, 95% CI, -15.8% to -0.6%), suggesting potential overtesting in patients at low risk for septic shock. In the study by Hooper et al., 88% (442/502) of the patients in the medical intensive care unit triggered an alert, raising the concern for alert fatigue.¹⁰ Furthermore, three studies did not perform intention-to-treat analyses, rather, included only patients who triggered the alert and also had provider suspected or confirmed sepsis,^{14, 17} or had a discharge diagnosis for sepsis.¹⁸

Discussion

The use of sepsis alert systems derived from electronic health data and targeting hospitalized patients improve a subset of sepsis process of care measures, but at the cost of poor positive predictive value and no clear improvement in mortality or length of stay. There is insufficient evidence for the effectiveness of automated electronic sepsis alert systems in the emergency department.

We found considerable variability in the diagnostic accuracy of automated electronic sepsis alert systems. There was moderate evidence that alert systems designed to identify severe sepsis (e.g. SIRS criteria plus measures of shock) had greater diagnostic accuracy than alert systems that detected sepsis based on SIRS criteria alone. Given that SIRS criteria are highly prevalent among hospitalized patients with non-infectious diseases,²⁰ sepsis alert systems triggered by standard SIRS criteria may have poorer predictive value with an increased risk of “alert fatigue” – excessive electronic warnings resulting in physicians disregarding clinically useful alerts.²¹ The potential for alert fatigue is even greater in critical care settings. A retrospective analysis of physiological alarms in the ICU estimated on average six alarms per hour with only 15% of alarms considered to be clinically relevant.²²

The fact that sepsis alert systems improve intermediate process measures among ward and ED patients but not ICU patients likely reflects differences in both the patients and the clinical settings.²³ First, patients in the ICU may already be prescribed broad spectrum antibiotics, aggressively fluid resuscitated, and have other diagnostic testing performed before the activation of a sepsis alert, so it would be less likely to see an improvement in the rates of process measures assessing initiation or escalation of therapy compared to patients treated on the wards or in the ED. The apparent lack of benefit of these systems in the ICU may merely represent a “ceiling” effect. Second, nurses and physicians are already vigilantly monitoring patients in the ICU for signs of clinical deterioration, so additional alert systems may be redundant. Third, patients in the ICU are connected to standard bedside monitors that continuously monitor for the presence of abnormal vital signs. An additional sepsis alert system triggered by SIRS criteria alone may be superfluous to the existing infrastructure. Fourth, the majority of patients in the ICU will trigger the sepsis alert system,¹⁰ so there likely is a high noise-to-signal ratio with resultant alert fatigue.²¹

In addition to greater emphasis on alert systems of greater diagnostic accuracy and effectiveness, our review notes several important gaps that limit evidence supporting the usefulness of automated sepsis alert systems. First, there is little data to describe the optimal design of sepsis alerts,^{24, 25} or the frequency with which they are appropriately acted upon or dismissed. In addition, we found little data to support whether effectiveness of alert systems differed based on whether clinical decision support was included with the alert itself (e.g., direct prompting with specific clinical management recommendations) or the configuration of the alert (e.g. interruptive alert or informational).^{24, 25} Most of the studies we reviewed employed alerts primarily targeting physicians; we found little evidence for systems which also alerted other providers (e.g. nurses or rapid response teams). Few studies provided data on harms of these systems (e.g. excess antimicrobial use, fluid overload due to aggressive fluid resuscitation) or how often these treatments were administered to patients who did not eventually have sepsis. Few studies employed study designs that limited biases (e.g. randomized or quasi-experimental designs) or used an intention-to-treat approach. Studies that exclude false positive alerts in analyses could bias estimates towards making sepsis alert systems appear more effective than they actually were. Finally, while presumably deploying automated sepsis alerts in the ED would facilitate more timely recognition and treatment, more rigorously conducted studies are needed to identify whether using these alerts in the ED are of greater value compared to the wards and ICU. Given the limited number of studies included in this review, we were unable to make strong conclusions regarding the clinical and cost effectiveness of implementing automated sepsis alerts.

Our review has certain limitations. First, despite our extensive literature search strategy, we may have missed studies published in the grey literature or in non-English languages. Second, there is potential publication bias given the number of abstracts that we identified addressing one of our prespecified research questions compared to the number of peer-reviewed publications identified by our search strategy.

Conclusion

Automated electronic sepsis alert systems have promise in delivering early-goal directed therapies to patients. However, at present, automated sepsis alerts derived from electronic health data may improve care processes but tend to have poor positive predictive value and have not been shown to improve mortality or length of stay. Future efforts should develop and study methods for sepsis alert systems which avoid the potential for alert fatigue while improving outcomes.