Abstract
Background
Although timely treatment of sepsis improves outcomes, delays in administering evidence-based therapies are common.
Purpose
To determine whether automated real-time electronic sepsis alerts can: 1) accurately identify sepsis, and 2) improve process measures and outcomes.
Data Sources
We systematically searched MEDLINE, Embase, The Cochrane Library, and CINAHL from database inception through June 27, 2014.
Study Selection
Included studies that empirically evaluated one or both of the prespecified objectives.
Data Extraction
Two independent reviewers extracted data and assessed the risk of bias. Diagnostic accuracy of sepsis identification was measured by sensitivity, specificity, positive (PPV) and negative predictive values (NPV) and likelihood ratios (LR). Effectiveness was assessed by changes in sepsis care process measures and outcomes.
Data Synthesis
Of 1,293 citations, 8 studies met inclusion criteria, 5 for the identification of sepsis (n=35,423) and 5 for the effectiveness of sepsis alerts (n=6,894). Though definition of sepsis alert thresholds varied, most included systemic inflammatory response syndrome criteria ± evidence of shock. Diagnostic accuracy varied greatly, with PPV ranging from 20.5-53.8%, NPV 76.5-99.7%; LR+ 1.2-145.8; and LR- 0.06-0.86. There was modest evidence for improvement in process measures (i.e., antibiotic escalation), but only among patients in non-critical care settings; there were no corresponding improvements in mortality or length of stay. Minimal data were reported on potential harms due to false positive alerts.
Conclusions
Automated sepsis alerts derived from electronic health data may improve care processes but tend to have poor positive predictive value and do not improve mortality or length of stay.
Keywords: Sepsis, early diagnosis, hospital information systems, diagnosis, computer-assisted, clinical alarms, review
Introduction
Sepsis is the most expensive condition treated in the hospital, resulting in an aggregate cost of $20.3 billion or 5.2% of total aggregate cost for all hospitalizations in the United States.1 Rates of sepsis and sepsis-related mortality are rising in the United States.2, 3 Timely treatment of sepsis, including adequate fluid resuscitation and appropriate antibiotic administration, decreases morbidity, mortality, and costs.4-6 Consequently, the Surviving Sepsis Campaign recommends timely care with the implementation of sepsis bundles and protocols.4 Though effective, sepsis protocols require dedicated personnel with specialized training, who must be highly vigilant and constantly monitor a patient's condition for the course of an entire hospitalization.7, 8 As such, delays in administering evidence-based therapies are common.8, 9
Automated electronic sepsis alerts are being developed and implemented to facilitate the delivery of timely sepsis care. Electronic alert systems synthesize electronic health data routinely collected for clinical purposes in real- or near real-time to automatically identify sepsis based on prespecified diagnostic criteria, and immediately alert providers that their patient may meet sepsis criteria via electronic notifications (e.g., through electronic health record (EHR), e-mail, or pager alerts).
However, little data exist to describe whether automated, electronic systems achieve their intended goal of earlier, more effective sepsis care. To examine this question, we performed a systematic review on automated electronic sepsis alerts to assess their suitability for clinical use. Our two objectives were: 1) to describe the diagnostic accuracy of alert systems in identifying sepsis using electronic data available in real-time or near real-time; and 2) to evaluate the effectiveness of sepsis alert systems on sepsis care process measures and clinical outcomes.
Materials and Methods
Data Sources and Search Strategies
We searched PubMed MEDLINE, Embase, The Cochrane Library, and CINAHL from database inception through June 27, 2014, for all studies that contained the following three concepts: sepsis, electronic systems, and alerts (or identification). All citations were imported into an electronic database (EndNote X5, Thomson Reuters). Our complete search strategy is provided in detail in the eAppendix.
Study Selection
Two authors (ANM and OKN) reviewed the citation titles, abstracts, and full-text articles of potentially relevant references identified from the literature search for eligibility. References of selected articles were hand searched to identify additional eligible studies. Inclusion criteria for eligible studies were: 1) adult patients (≥18 years) receiving care either in the emergency department or hospital; 2) outcomes of interest including a) diagnostic accuracy in identification of sepsis, and/or b) effectiveness of sepsis alerts on process measures and clinical outcomes were evaluated using empiric data; and 3) sepsis alert systems used real-time or near real-time electronically available data to enable proactive, timely management. We excluded studies that: 1) tested the effect of other electronic interventions that were not sepsis alerts (i.e., computerized order sets) for sepsis management; 2) studies solely focused on detecting and treating central line-associated bloodstream infections, shock (not otherwise specified), bacteremia, or other device-related infections; and 3) studies evaluating the effectiveness of sepsis alerts without a control group.
Data Extraction and Quality Assessment
Two reviewers (ANM and OKN) extracted data on the clinical setting, study design, dates of enrollment, definition of sepsis, details of the identification and alert systems, diagnostic accuracy of the alert system, and the incidence of process measures and clinical outcomes using a standardized form. Discrepancies between reviewers were resolved by discussion and consensus. Data discrepancies identified in one study were resolved by contacting the corresponding author.10
For studies assessing the diagnostic accuracy of sepsis identification, study quality was assessed using the Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) revised tool.11 For studies evaluating the effectiveness of sepsis alert systems, studies were considered ‘high quality’ if a contemporaneous control group was present to account for temporal trends (e.g., randomized controlled trial or observational analysis with a concurrent control). ‘Fair quality’ studies were before-and-after studies that adjusted for potential confounders between time periods. ‘Low quality’ studies included those that did not account for temporal trends, such as before-and-after studies using only historical controls without adjustment. Studies that did not use an intention-to-treat analysis were also considered ‘low quality’. The strength of the overall body of evidence, including risk of bias, was guided by the Grading of Recommendations Assessment, Development, and Evaluation Working Group Criteria adapted by the Agency of Healthcare Research and Quality.12
Data Synthesis
To analyze the diagnostic accuracy of automated sepsis alert systems to identify sepsis and to evaluate the effect on outcomes, we performed a qualitative assessment of all studies. We were unable to perform a meta-analysis due to significant heterogeneity in study quality, clinical setting, and definition of the sepsis alert. Diagnostic accuracy of sepsis identification was measured by sensitivity, specificity, positive and negative predictive values and likelihood ratios. Effectiveness was assessed by changes in sepsis care process measures (i.e. time to antibiotics) and outcomes (length of stay, mortality).
Results
Description of Studies
Of 1,293 titles, 183 qualified for abstract review, 84 for full-text review, and 8 articles met our inclusion criteria (eFigure in Supplement). Five articles evaluated the diagnostic accuracy of sepsis identification10, 13-16 and five articles10, 14, 17-19 evaluated the effectiveness of automated electronic sepsis alerts on sepsis process measures and patient outcomes. All articles were published between 2009 and 2014 and were single-site studies conducted at academic medical centers (Tables 1 and 2). The clinical settings in the included studies varied and included the emergency department (ED), hospital wards, and the intensive care unit (ICU).
Table 1. Characteristics of Studies Evaluating the Diagnostic Accuracy of Automated Electronic Sepsis Alerts.
Source | Site No./Type | Setting | Alert threshold | Gold standard definition | Gold standard measurement | N | Study Qualitya |
---|---|---|---|---|---|---|---|
Hooper et al.,10 2012 | 1/academic | MICU | ≥2 SIRS criteriab | Reviewer judgment, not otherwise specified | Chart review | 560 | High |
Meurer et al.,13 2009 | 1/academic | ED | ≥2 SIRS criteria | Reviewer judgment whether diagnosis of infection present in ED plus SIRS criteria | Chart review | 248 | Low |
Nelson J. et al.,14 2011 | 1/academic | ED | ≥2 SIRS criteria and 2 SBP measurements < 90 mm Hg | Reviewer judgment whether infection present, requiring hospitalization with at least 1 organ system involved | Chart review | 1386 | High |
Nguyen et al.,15 2014 | 1/academic | ED | ≥2 SIRS criteria and 1 sign of shock (SBP≤90 mm Hg or lactic acid ≥ 2.0 mmol/L) | Reviewer judgment to confirm SIRS, shock, and presence of a serious infection | Chart review | 1095 | Low |
Thiel et al,16 2010 | 1/academic | Wards | Recursive partitioning tree analysis including vitals and laboratory resultsc | Admitted to the hospital wards and subsequently transferred to the ICU for septic shock and treated with vasopressor therapy | ICD-9 discharge codes for acute infection, acute organ dysfunction, and need for vasopressors within 24 hours of ICU transfer | 27674 | Low |
Abbreviations: MICU, medical intensive care unit; SIRS, systemic inflammatory response syndrome; ED, emergency department; SBP, systolic blood pressure; ICD-9, International Classification of Diseases, Ninth Revision
Assessed using the Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) revised tool.10
Recorded within a 24 hour period, mandating either abnormal temperature or WBC.
Included shock index, mean arterial pressure, international normalized ratio, white blood cell count, absolute neutrophil count, bilirubin, albumin, hemoglobin, and sodium.
Table 2. Characteristics of Studies Evaluating the Effectiveness of Automated Electronic Sepsis Alerts.
Source | Design | Site No./Type | Setting | N | Alert system type | Alert threshold | Alert notificationa | Treatment Recommendation | Study Qualityb |
---|---|---|---|---|---|---|---|---|---|
Berger et al.,17 2010 | Before-after(6 months pre and 6 months post) | 1/academic | ED | 5796c | CPOE system | ≥ 2 SIRS criteria | CPOE passive alert | Yes: lactate collection | Low |
Hooper et al.,10 2012 | RCT | 1/academic | MICU | 443 | EHR | ≥ 2 SIRS criteriad | Text page and EHR passive alert | No | High |
McRee et al.,18 2014 | Before-after (6 months pre and 6 months post) | 1/academic | Wards | 171e | EHR | ≥ 2 SIRS criteria | Notified nurse, specifics unclear | No, but the nurse completed a sepsis risk evaluation flow sheet | Low |
Nelson et al.,14 2011 | Before-after (3 months pre and 3 months post) | 1/academic | ED | 184f | EHR | ≥ 2 SIRS criteria and 2 or more SBP readings < 90 mmHg | Text page and EHR passive alert | Yes: fluid resuscitation, blood culture collection, antibiotic administration, “among others” | Low |
Sawyer et al.,19 2011 | Prospective, non-randomized (2 intervention and 4 control wards) | 1/academic | Wards | 300 | EHR | Recursive partitioning regression tree algorithm including vitals and lab valuesg | Text page to charge nurse who then assessed patient and informed treating physicianh | No | High |
Abbreviations: ED, emergency department; CPOE, computerized provider order entry; SIRS, systemic inflammatory response syndrome; RCT, randomized control trial; MICU, medical intensive care unit; EHR, electronic health record; SBP, systolic blood pressure;
Passive alerts do not require the provider to acknowledge the alert or take action. Text pages were automatically generated and sent.
Assessed by prespecified criteria of study design and intention-to-treat protocol.
Not an intention-to-treat analysis. Only patients meeting SIRS criteria with a provider's affirmative response to a computerized query regarding suspected infection were analyzed.
Recorded within 24 hour period, mandating either abnormal temperature or WBC.
Not an intention-to-treat analysis. Only medical records reviewed of individuals with an ICD-9 discharge code of sepsis
Not an intention-to-treat analysis. 398 patients triggered the alert but only the 184 (46%) confirmed to have an admission diagnosis of infection by chart review were included in the analysis.
Included shock index, mean arterial pressure, international normalized ratio, white blood cell count, absolute neutrophil count, bilirubin, albumin, hemoglobin, and sodium
Nurses and physicians on intervention wards received sepsis alert education prior to implementation; no education provided to control wards.
Among the eight included studies, there was significant heterogeneity in threshold criteria for sepsis identification and subsequent alert activation. The most commonly defined threshold was the presence of 2 or more systemic inflammatory response syndrome (SIRS) criteria.10, 13, 17, 18
Diagnostic Accuracy of Automated Electronic Sepsis Alert Systems
The prevalence of sepsis varied substantially between the studies depending on the ‘gold standard’ definition of sepsis used and the clinical setting (ED, wards, or ICU) of the study (Table 3). The two studies14, 16 that defined sepsis as requiring evidence of shock had a substantially lower prevalence (0.8-4.7%) compared to the two studies10, 13 that defined sepsis as having only two or more SIRS criteria with a presumed diagnosis of an infection (27.8%-32.5%).
Table 3. Diagnostic Accuracy of Automated Electronic Sepsis Alerts.
Source | Setting | Alert threshold | Prevalence, % | Sensitivity, % (95% CI) | Specificity, % (95% CI) | PPV, % (95% CI) | NPV, % (95% CI) | LR+, (95% CI) | LR-, (95% CI) |
---|---|---|---|---|---|---|---|---|---|
Hooper et al.,10 2012 | MICU | ≥ 2 SIRS criteriaa | 36.3 | 98.9 (95.7-99.8) | 18.1 (14.2-22.9) | 40.7 (36.1-45.5) | 96.7 (87.5-99.4) | 1.21 (1.14-1.27) | 0.06 (0.01-0.25) |
Meurer et al.,13 2009 | ED | ≥ 2 SIRS criteria | 27.8 | 36.2 (25.3-48.8) | 79.9 (73.1-85.3) | 41.0 (28.8-54.3) | 76.5 (69.6-82.2) | 1.80 (1.17-2.76) | 0.80 (0.67-0.96) |
Nelson et al.,14 2011 | ED | ≥ 2 SIRS criteria and 2 SBP measurements < 90 mm Hg | 0.8 | 63.6 (31.6-87.8) | 99.6 (99.0-99.8) | 53.8 (26.1-79.6) | 99.7 (99.2-99.9) | 145.8 (58.4-364.1) | 0.37 (0.17-0.80) |
Nguyen et al.,15 2014 | ED | ≥ 2 SIRS criteria and 1 sign of shock (SBP≤90 mm Hg or lactic acid ≥ 2.0 mmol/L) | Unable to estimate b | Unable to estimate b | Unable to estimate b | 44.7 (41.2-48.2) | 100.0 c (98.8-100.0) | Unable to estimate b | Unable to estimate b |
Thiel et al.,16 2010 | Wards | Recursive partitioning tree analysis including vitals and laboratory resultsd | 4.7 | 17.1 (15.1-19.3) | 96.7 (96.5-96.9) | 20.5 (18.2-23.0) | 95.9 (95.7-96.2) | 5.22 (4.56-5.98) | 0.86 (0.84-0.88) |
Abbreviations: MICU, medical intensive care unit; SIRS, systemic inflammatory response syndrome; ED, emergency department; SBP, systolic blood pressure
Recorded within 24 hour period, mandating either abnormal temperature or WBC
False negative and true negatives unknown due to random sampling of non-alert cases
Estimated value based on random sample of 300 non-alert cases
Included shock index, mean arterial pressure, international normalized ratio, white blood cell count, absolute neutrophil count, bilirubin, albumin, hemoglobin, and sodium
All alert systems had suboptimal positive predictive value (20.5%-53.8%). The two studies that designed the sepsis alert to activate by SIRS criteria alone10, 13 had a positive predictive value of 41% and a positive likelihood ratio of 1.21-1.80. The ability to exclude the presence of sepsis varied considerably depending on the clinical setting. The study by Hooper et al.10 that examined the alert among patients in the medical ICU appeared more effective at ruling out sepsis (NPV=96.7%; negative LR = 0.06) compared to a similar alert system used by Meurer et al.13 which studied patients in the ED (NPV=76.5%, negative LR = 0.80).
There were also differences in the diagnostic accuracy of the sepsis alert systems depending on how the threshold for activating the sepsis alert was defined and applied in the study. Two studies evaluated a sepsis alert system among patients presenting to the ED at the same academic medical center.13, 14 The alert system (Nelson et al.) that was triggered by a combination of SIRS criteria and hypotension (PPV = 53.8%, LR+ = 145.8; NPV = 99.7%, LR- = 0.37) outperformed the alert system (Meurer et al.) that was triggered by SIRS criteria alone (PPV = 41.0%, LR+ = 1.80; NPV = 76.5%, LR- = 0.80). Further, the study by Meurer and colleagues evaluated the accuracy of the alert system only among patients who were hospitalized after presenting to the ED, rather than all consecutive patients presenting to the ED. This selection bias likely falsely inflated the diagnostic accuracy of the alert system used by Meurer et al., suggesting the alert system that was triggered by a combination of SIRS criteria and hypotension was comparatively even more accurate.
Two studies evaluating the diagnostic accuracy of the alert system were deemed to be ‘high quality’ (Table 4). Three studies were considered ‘low quality’ -- one study did not include all patients in their assessment of diagnostic accuracy;13 one study consecutively selected alert cases but randomly selected non-alert cases, greatly limiting the assessment of diagnostic accuracy;15 and the other study applied a gold standard that was unlikely to correctly classify sepsis (septic shock requiring ICU transfer with vasopressor support in the first 24 hours was defined by discharge ICD-9 diagnoses without chart review) with a considerable delay from the alert system trigger (alert identification was compared to the discharge diagnosis rather than physician review of real-time data).16
Table 4. Assessment of Bias in Studies Evaluating Diagnostic Accuracy of Automated Electronic Sepsis Alertsa.
Study | Patient Selection | Index Test | Reference Standard | Flow and Timing |
---|---|---|---|---|
Hooper et al.,10 2012 | +++ | +++ | ++b | +++ |
Meurer et al.,13 2009 | +++ | +++ | ++b | +c |
Nelson et al.,14 2011 | +++ | +++ | ++b | +++ |
Nguyen et al.,15 2014 | +d | +++ | +e | +++ |
Thiel et al.,16 2010 | +++ | +++ | +f | +g |
Determined by 2 independent abstractors using the revised QUADAS-2 tool 10. Three plus signs indicate the lowest risk for bias and one plus sign indicates highest risk for bias
Unclear if the gold standard was interpreted without knowledge of the results of the sepsis alert
Not all patients accounted for in the study. Only patients in the emergency department who were subsequently hospitalized were subjected to the gold standard and were included in the analysis.
Consecutive selection for cases, but random selection of non-cases greatly limited evaluation of diagnostic accuracy
Gold standard was interpreted with knowledge of the results of the sepsis alert
Discharge ICD-9 diagnosis codes unlikely to correctly classify patients admitted to the hospital wards and subsequently transferred to the intensive care unit for septic shock and needing vasopressor support in the first 24 hours.
There was a delay in time between the sepsis alert triggering and ascertainment of the gold standard (discharge ICD-9 diagnoses) which may result in misclassification.
Effectiveness of Automated Electronic Sepsis Alert Systems
Characteristics of the studies evaluating the effectiveness of automated electronic sepsis alert systems are summarized in Table 2. Regarding activation of the sepsis alert, two studies notified the provider directly by an automated text page and a passive EHR alert (not requiring the provider to acknowledge the alert or take action),10, 14 one study notified the provider by a passive electronic alert alone,17 and one study only employed an automated text page.19 Furthermore, if the sepsis alert was activated, two studies suggested specific clinical management decisions14, 17 two studies left clinical management decisions solely to the discretion of the treating provider, 10, 19 and one study assisted the diagnosis of sepsis by prompting nurses to complete a second manual sepsis risk evaluation.18
Table 5 summarizes the effectiveness of automated electronic sepsis alert systems. Two studies evaluating the effectiveness of the sepsis alert system were considered to be ‘high quality’ studies based on the use of a contemporaneous control group to account for temporal trends and an intention-to-treat analysis.10, 19 The two studies evaluating the effectiveness of a sepsis alert system in the ED were considered ‘low quality’ due to before-and-after designs without an intention-to-treat analysis.14, 17
Table 5. Effectiveness of Automated Electronic Sepsis Alerts.
Source | Outcomes evaluated | Key Findings | Quality |
---|---|---|---|
Hooper et al.,10 2012 | Primary: time to receipt of antibiotic (new or changed) Secondary: sepsis-related process measures and outcomes |
No difference (6.1 hours for control vs. 6.0 hours for intervention, p=0.95) No difference in amount of 6 hour IV fluid administration (964 mL vs. 1019 mL, p=.6), collection of blood cultures (adjusted HR 1.01, 95% CI, 0.76-1.35), collection of lactate (adjusted HR 0.84, 95% CI, 0.54-1.30), ICU length of stay (3.0 vs. 3.0 days, p=.2), hospital length of stay (4.7 vs. 5.7 days, p=.08), and hospital mortality (10% for control vs. 14% for intervention, p=.3) |
High |
Sawyer et al.,19 2011 | Primary: sepsis-related process measures (antibiotic escalation, IV fluids, oxygen therapy, vasopressor initiation, diagnostic testing (blood culture, CXR) within 12 hours of alert Secondary: ICU transfer, hospital length of stay, hospital length of stay after alert, in-hospital mortality |
Increases in receiving ≥ 1 measure (56% for control vs. 71% for intervention, p=.02), antibiotic escalation (24% vs. 36%, p=.04), IV fluid administration (24% vs. 38%, p=.01), and oxygen therapy (8% vs. 20%, p=.005). There was a nonsignificant increase in obtaining diagnostic tests (40% vs. 52%, p=.06) and vasopressor initiation (3% vs. 6%, p=.4) Similar rate of ICU transfer (23% for control vs. 26% for intervention, p=.6), hospital length of stay (7 vs. 9 median days, p=.8), hospital length of stay after alert (5 vs. 6 median days, p=.7), and in-hospital mortality (12% vs. 10%, p=.7). |
High |
Berger et al.,17 2010 | Primary: lactate collection in ED Secondary: lactate collection among hospitalized patients, proportion of patients with abnormal lactate (≥4 mmol/L), and in-hospital mortality among hospitalized patients |
Increase in lactate collection in the ED (5.2% before vs. 12.7% after alert implemented, absolute increase of 7.5%, 95% CI, 6.0%-9.0%) Increase in lactate collection among hospitalized patients (15.3% vs. 34.2%, absolute increase of 18.9%, 95% CI, 15.0%-22.8%); decrease in the proportion of abnormal lactate values (21.9% vs. 14.8%, absolute decrease of 7.6%, 95% CI, -15.8% to -0.6%), and no significant difference in mortality (5.7% vs. 5.2%, absolute decrease of 0.5%, 95% CI, -1.6%-2.6%, p=.6) |
Low |
McRee et al.,18 2014 | Stage of sepsis, length of stay, mortality, discharge location | Nonsignificant decrease in stage of sepsis (34.7% with septic shock before vs. 21.9% after, p>.05); No difference in length-of-stay (8.5 days before vs. 8.7 days after, p>.05). Decrease in mortality (9.3% before vs. 1.0% after, p<.05) and proportion of patients discharged home (25.3% before vs. 49.0% after, p<.05) | Low |
Nelson et al.,14 2011 | Frequency and time to completion of process measures: lactate, blood culture, CXR, and antibiotic initiation | Increases in blood culture collection (OR 2.9, 95% CI 1.1-7.7) and CXR (OR 3.2, 95% CI 1.1-9.5); nonsignificant increases in lactate collection (OR 1.7, 95% CI, 0.9-3.2) and antibiotic administration (OR 2.8, 95% CI, 0.9-8.3). Only blood cultures were collected ina more timely manner (median of 86 minutes before vs. 81 minutes after alert implementation, p=.03). | Low |
Abbreviations: IV, intravenous; CXR, chest radiograph; ICU, intensive care unit; ED, emergency department
Neither of the two high quality studies that included a contemporaneous control found evidence for improving inpatient mortality or hospital and ICU length of stay.10, 19 The impact of sepsis alert systems on improving process measures for sepsis management depended on the clinical setting. In an RCT of patients admitted to a medical ICU, Hooper et al. did not find any benefit of implementing a sepsis alert system on improving intermediate outcome measures such as antibiotic escalation, fluid resuscitation, and collection of blood cultures and lactate.10 However, in a well-designed observational study, Sawyer et al. found significant increases in antibiotic escalation, fluid resuscitation, and diagnostic testing in patients admitted to the medical wards.19 Both studies that evaluated the effectiveness of sepsis alert systems in the ED showed improvements in various process measures,14, 17 but without improvement in mortality.17 The single study that showed improvement in clinical outcomes (in-hospital mortality and disposition location) was of ‘low quality’ due to the pre-post study design without adjustment for potential confounders and lack of an intention-treat analysis (only individuals with a discharge diagnosis of sepsis were included, rather than all individuals who triggered the alert).18 Additionally, the pre-intervention group had a higher proportion of individuals with septic shock compared to the post-intervention group, raising the possibility that the observed improvement was due to difference in severity of illness between the two groups rather than due to the intervention.
None of the studies included in this review explicitly reported on the potential harms (e.g. excess antimicrobial use or alert fatigue) after implementation of sepsis alerts, but Hooper et al. found a nonsignificant increase in mortality, and Sawyer et al. showed a nonsignificant increase in the length of stay in the intervention group compared to the control group.10, 19 Berger et al. showed an overall increase in the number of lactate tests performed, but with a decrease in the proportion of abnormal lactate values (21.9% vs. 14.8%, absolute decrease of 7.6%, 95% CI, -15.8% to -0.6%), suggesting potential overtesting in patients at low risk for septic shock. In the study by Hooper et al., 88% (442/502) of the patients in the medical intensive care unit triggered an alert, raising the concern for alert fatigue.10 Furthermore, three studies did not perform intention-to-treat analyses, rather, included only patients who triggered the alert and also had provider suspected or confirmed sepsis,14, 17 or had a discharge diagnosis for sepsis.18
Discussion
The use of sepsis alert systems derived from electronic health data and targeting hospitalized patients improve a subset of sepsis process of care measures, but at the cost of poor positive predictive value and no clear improvement in mortality or length of stay. There is insufficient evidence for the effectiveness of automated electronic sepsis alert systems in the emergency department.
We found considerable variability in the diagnostic accuracy of automated electronic sepsis alert systems. There was moderate evidence that alert systems designed to identify severe sepsis (e.g. SIRS criteria plus measures of shock) had greater diagnostic accuracy than alert systems that detected sepsis based on SIRS criteria alone. Given that SIRS criteria are highly prevalent among hospitalized patients with non-infectious diseases,20 sepsis alert systems triggered by standard SIRS criteria may have poorer predictive value with an increased risk of “alert fatigue” – excessive electronic warnings resulting in physicians disregarding clinically useful alerts.21 The potential for alert fatigue is even greater in critical care settings. A retrospective analysis of physiological alarms in the ICU estimated on average six alarms per hour with only 15% of alarms considered to be clinically relevant.22
The fact that sepsis alert systems improve intermediate process measures among ward and ED patients but not ICU patients likely reflects differences in both the patients and the clinical settings.23 First, patients in the ICU may already be prescribed broad spectrum antibiotics, aggressively fluid resuscitated, and have other diagnostic testing performed before the activation of a sepsis alert, so it would be less likely to see an improvement in the rates of process measures assessing initiation or escalation of therapy compared to patients treated on the wards or in the ED. The apparent lack of benefit of these systems in the ICU may merely represent a “ceiling” effect. Second, nurses and physicians are already vigilantly monitoring patients in the ICU for signs of clinical deterioration, so additional alert systems may be redundant. Third, patients in the ICU are connected to standard bedside monitors that continuously monitor for the presence of abnormal vital signs. An additional sepsis alert system triggered by SIRS criteria alone may be superfluous to the existing infrastructure. Fourth, the majority of patients in the ICU will trigger the sepsis alert system,10 so there likely is a high noise-to-signal ratio with resultant alert fatigue.21
In addition to greater emphasis on alert systems of greater diagnostic accuracy and effectiveness, our review notes several important gaps that limit evidence supporting the usefulness of automated sepsis alert systems. First, there is little data to describe the optimal design of sepsis alerts,24, 25 or the frequency with which they are appropriately acted upon or dismissed. In addition, we found little data to support whether effectiveness of alert systems differed based on whether clinical decision support was included with the alert itself (e.g., direct prompting with specific clinical management recommendations) or the configuration of the alert (e.g. interruptive alert or informational).24, 25 Most of the studies we reviewed employed alerts primarily targeting physicians; we found little evidence for systems which also alerted other providers (e.g. nurses or rapid response teams). Few studies provided data on harms of these systems (e.g. excess antimicrobial use, fluid overload due to aggressive fluid resuscitation) or how often these treatments were administered to patients who did not eventually have sepsis. Few studies employed study designs that limited biases (e.g. randomized or quasi-experimental designs) or used an intention-to-treat approach. Studies that exclude false positive alerts in analyses could bias estimates towards making sepsis alert systems appear more effective than they actually were. Finally, while presumably deploying automated sepsis alerts in the ED would facilitate more timely recognition and treatment, more rigorously conducted studies are needed to identify whether using these alerts in the ED are of greater value compared to the wards and ICU. Given the limited number of studies included in this review, we were unable to make strong conclusions regarding the clinical and cost effectiveness of implementing automated sepsis alerts.
Our review has certain limitations. First, despite our extensive literature search strategy, we may have missed studies published in the grey literature or in non-English languages. Second, there is potential publication bias given the number of abstracts that we identified addressing one of our prespecified research questions compared to the number of peer-reviewed publications identified by our search strategy.
Conclusion
Automated electronic sepsis alert systems have promise in delivering early-goal directed therapies to patients. However, at present, automated sepsis alerts derived from electronic health data may improve care processes but tend to have poor positive predictive value and have not been shown to improve mortality or length of stay. Future efforts should develop and study methods for sepsis alert systems which avoid the potential for alert fatigue while improving outcomes.
Supplementary Material
Acknowledgments
The authors thank Gloria Won, M.L.I.S., for her assistance with developing and performing the literature search strategy and wish her a long and joyous retirement. Dr. Makam had full access to the data in the study and takes responsibility for the integrity of the date and accuracy of the data analysis. Study concept and design: All authors. Acquisition of Data: Makam and Nguyen. Analysis and Interpretation of Data: All authors. Drafting of the manuscript: Makam. Critical revision of the manuscript: All authors. Statistical Analysis: Makam and Nguyen. We have no conflicts of interest to disclose. Part of Dr. Makam's work on this project was completed while he was a primary care research fellow at the University of California, San Francisco, funded by a National Research Service Award (training grant T32HP19025-07-00). Dr. Makam is currently supported by the National Center for Advancing Translational Sciences of the National Institutes of Health (KL2TR001103). Dr. Nguyen was supported by the Agency for Healthcare Research and Quality (R24HS022428-01). Dr. Auerbach was supported by an NHLBI K24 grant (K24HL098372).
Footnotes
Conflicts of Interest: We have no conflicts of interest to disclose.
References
- 1.Torio CM, Andrews RM. Statistical Brief #160. Healthcare Cost and Utilization Project (HCUP) Statistical Briefs; Rockville (MD): 2013. National Inpatient Hospital Costs: The Most Expensive Conditions by Payer, 2011. [Google Scholar]
- 2.Hall MJ, Williams SN, DeFrances CJ, Golosinskiy A. Inpatient care for septicemia or sepsis: a challenge for patients and hospitals. NCHS Data Brief. 2011;(62):1–8. [PubMed] [Google Scholar]
- 3.Martin GS, Mannino DM, Eaton S, Moss M. The epidemiology of sepsis in the United States from 1979 through 2000. N Engl J Med. 2003;348(16):1546–1554. doi: 10.1056/NEJMoa022139. [DOI] [PubMed] [Google Scholar]
- 4.Dellinger RP, Levy MM, Rhodes A, et al. Surviving sepsis campaign: international guidelines for management of severe sepsis and septic shock: 2012. Crit Care Med. 2013;41(2):580–637. doi: 10.1097/CCM.0b013e31827e83af. [DOI] [PubMed] [Google Scholar]
- 5.Rivers E, Nguyen B, Havstad S, et al. Early goal-directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med. 2001;345(19):1368–1377. doi: 10.1056/NEJMoa010307. [DOI] [PubMed] [Google Scholar]
- 6.Pro CI, Yealy DM, Kellum JA, et al. A randomized trial of protocol-based care for early septic shock. N Engl J Med. 2014;370(18):1683–1693. doi: 10.1056/NEJMoa1401602. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Turi SK, Von Ah D. Implementation of early goal-directed therapy for septic patients in the emergency department: a review of the literature. J Emerg Nurs. 2013;39(1):13–19. doi: 10.1016/j.jen.2011.06.006. [DOI] [PubMed] [Google Scholar]
- 8.Fong JJ, Cecere K, Unterborn J, Garpestad E, Klee M, Devlin JW. Factors influencing variability in compliance rates and clinical outcomes among three different severe sepsis bundles. Ann Pharmacother. 2007;41(6):929–936. doi: 10.1345/aph.1K041. [DOI] [PubMed] [Google Scholar]
- 9.Ferrer R, Artigas A, Levy MM, et al. Improvement in process of care and outcome after a multicenter severe sepsis educational program in Spain. JAMA. 2008;299(19):2294–2303. doi: 10.1001/jama.299.19.2294. [DOI] [PubMed] [Google Scholar]
- 10.Hooper MH, Weavind L, Wheeler AP, et al. Randomized trial of automated, electronic monitoring to facilitate early detection of sepsis in the intensive care unit*. Crit Care Med. 2012;40(7):2096–2101. doi: 10.1097/CCM.0b013e318250a887. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Whiting PF, Rutjes AW, Westwood ME, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155(8):529–536. doi: 10.7326/0003-4819-155-8-201110180-00009. [DOI] [PubMed] [Google Scholar]
- 12.Owens DK, Lohr KN, Atkins D, et al. AHRQ series paper 5: grading the strength of a body of evidence when comparing medical interventions--agency for healthcare research and quality and the effective health-care program. J Clin Epidemiol. 2010;63(5):513–523. doi: 10.1016/j.jclinepi.2009.03.009. [DOI] [PubMed] [Google Scholar]
- 13.Meurer WJ, Smith BL, Losman ED, et al. Real-time identification of serious infection in geriatric patients using clinical information system surveillance. J Am Geriatr Soc. 2009;57(1):40–45. doi: 10.1111/j.1532-5415.2008.02094.x. [DOI] [PubMed] [Google Scholar]
- 14.Nelson JL, Smith BL, Jared JD, Younger JG. Prospective trial of real-time electronic surveillance to expedite early care of severe sepsis. Ann Emerg Med. 2011;57(5):500–504. doi: 10.1016/j.annemergmed.2010.12.008. [DOI] [PubMed] [Google Scholar]
- 15.Nguyen SQ, Mwakalindile E, Booth JS, et al. Automated electronic medical record sepsis detection in the emergency department. PeerJ. 2014;2014(1) doi: 10.7717/peerj.343. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Thiel SW, Rosini JM, Shannon W, Doherty JA, Micek ST, Kollef MH. Early prediction of septic shock in hospitalized patients. J Hosp Med. 2010;5(1):19–25. doi: 10.1002/jhm.530. [DOI] [PubMed] [Google Scholar]
- 17.Berger T, Birnbaum A, Bijur P, Kuperman G, Gennis P. A Computerized Alert Screening for Severe Sepsis in Emergency Department Patients Increases Lactate Testing but does not Improve Inpatient Mortality. Appl Clin Inform. 2010;1(4):394–407. doi: 10.4338/ACI-2010-09-RA-0054. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.McRee L, Thanavaro JL, Moore K, Goldsmith M, Pasvogel A. The impact of an electronic medical record surveillance program on outcomes for patients with sepsis. Heart Lung. 2014 doi: 10.1016/j.hrtlng.2014.05.009. [DOI] [PubMed] [Google Scholar]
- 19.Sawyer AM, Deal EN, Labelle AJ, et al. Implementation of a real-time computerized sepsis alert in nonintensive care unit patients. Crit Care Med. 2011;39(3):469–473. doi: 10.1097/CCM.0b013e318205df85. [DOI] [PubMed] [Google Scholar]
- 20.Brun-Buisson C. The epidemiology of the systemic inflammatory response. Intensive Care Med. 2000;26(Suppl 1):S64–74. doi: 10.1007/s001340051121. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Nanji KC, Slight SP, Seger DL, et al. Overrides of medication-related clinical decision support alerts in outpatients. J Am Med Inform Assoc. 2013 doi: 10.1136/amiajnl-2013-001813. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Siebig S, Kuhls S, Imhoff M, Gather U, Scholmerich J, Wrede CE. Intensive care unit alarms--how many do we need? Crit Care Med. 2010;38(2):451–456. doi: 10.1097/CCM.0b013e3181cb0888. [DOI] [PubMed] [Google Scholar]
- 23.Singal G, Currier P. How can we best use electronic data to find and treat the critically ill?*. Crit Care Med. 2012;40(7):2242–2243. doi: 10.1097/CCM.0b013e318256b99b. [DOI] [PubMed] [Google Scholar]
- 24.Ash JS, Sittig DF, Dykstra R, et al. Identifying best practices for clinical decision support and knowledge management in the field. Stud Health Technol Inform. 2010;160(Pt 2):806–810. [PMC free article] [PubMed] [Google Scholar]
- 25.Wright A, Phansalkar S, Bloomrosen M, et al. Best Practices in Clinical Decision Support: the Case of Preventive Care Reminders. Appl Clin Inform. 2010;1(3):331–345. doi: 10.4338/ACI-2010-05-RA-0031. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.