Impact of an emergency department electronic sepsis surveillance system on patient mortality and length of stay

Abstract

Objective

The purpose of this study was to determine whether an electronic health record–based sepsis alert system could improve quality of care and clinical outcomes for patients with sepsis.

Materials and Methods

We performed a patient-level interrupted time series study of emergency department patients with severe sepsis or septic shock between January 2013 and April 2015. The intervention, introduced in February 2014, was a system of interruptive sepsis alerts triggered by abnormal vital signs or laboratory results. Primary outcomes were length of stay (LOS) and in-hospital mortality; other outcomes included time to first lactate and blood cultures prior to antibiotics. We also assessed sensitivity, positive predictive value (PPV), and clinician response to the alerts.

Results

Mean LOS for patients with sepsis decreased from 10.1 to 8.6 days (P < .001) following alert introduction. In adjusted time series analysis, the intervention was associated with a decreased LOS of 16% (95% CI, 5%-25%; P = .007, with significance of α = 0.006) and no change thereafter (0%; 95% CI, −2%, 2%). The sepsis alert system had no effect on mortality or other clinical or process measures. The intervention had a sensitivity of 80.4% and a PPV of 14.6%.

Discussion

Alerting based on simple laboratory and vital sign criteria was insufficient to improve sepsis outcomes. Alert fatigue due to the low PPV is likely the primary contributor to these results.

Conclusion

A more sophisticated algorithm for sepsis identification is needed to improve outcomes.

BACKGROUND AND SIGNIFICANCE

Sepsis is the cause of 3 million hospitalizations annually and is associated with in-hospital mortality of 15%–30%.¹ Early identification and treatment are mainstays of management, as goal-directed therapy has been associated with, on average, a 20% reduction in mortality.² Yet early detection of sepsis remains a challenge in the emergency department (ED)^3,4 due to the lack of a gold standard for rapid diagnosis.

The Surviving Sepsis Campaign international guidelines recommend screening potentially infected patients for severe sepsis.⁵ Manual screening interventions have led to a reduction in sepsis mortality, but they tend to be very resource intensive.^6,7 Prior studies have shown mixed findings on the effect of automated electronic early recognition tools on process measures for sepsis,^8–11 while most ED-based sepsis alerts have not shown benefit for clinical outcomes, including mortality.^8,13,14 However, few of these studies have employed quasi-experimental designs to account for temporal trends,^8,11,13,15 an important limitation given concurrent clinical and policy initiatives to improve outcomes for patients with sepsis.⁷ Furthermore, prior evaluations of sepsis alerts have not assessed adoption of the alerts.^8,13,15 Indeed, a recent systematic review of sepsis alerts concluded that there was a gap in the evidence on alert uptake and utilization.¹⁵

With the goal of improving health outcomes for patients diagnosed with severe sepsis, we developed a system of 3 interruptive sepsis alerts to communicate to clinicians that their patients might already have, or might be at risk for, severe sepsis. These alerts were primarily targeted toward ED nurses, a novel aspect of our electronic alert.¹⁵

OBJECTIVE

The purpose of this study was to evaluate the effect of the introduction of these alerts on quality of care and clinical outcomes for patients with sepsis. Additionally, we measured the accuracy of the alerts for sepsis identification as well as provider utilization of the alerts.

MATERIALS AND METHODS

We performed a retrospective cohort study of individuals at New York University Langone Medical Center, an urban academic institution, from January 1, 2013, to April 16, 2015. We included patients ≥18 years of age who were seen in the urgent care center or ED. For our primary analysis, evaluation of the impact of the alerts on quality and outcomes, we included patients who were subsequently hospitalized with a final discharge diagnosis of sepsis, which we defined as either severe sepsis (ICD-9 code 995.92) or septic shock (ICD-9 code 785.52), listed as present on admission. We included only diagnoses present on admission to make sure that patients in our cohort did indeed have severe sepsis/septic shock prior to alert firing.

For our analysis of alert performance and utilization, we included all ED patients for whom alerts were triggered. The primary data source for the study was the Epic (Epic Systems, Verona WI, USA) electronic health record (EHR).

The intervention was an ED-based sepsis alert system that was deployed on February 5, 2014. The alerts were developed by a team comprising ED, infectious disease, and critical care physicians, clinical pharmacists, ED nurses, and representatives from the departments of clinical quality and effectiveness and medical center information technology. ED physician and nurse sepsis champions provided in-service instruction for staff on the alert and the workflow surrounding escalation of care and the institutional sepsis protocol. This protocol involved calling a sepsis alert that necessitated an immediate bedside huddle between the provider and the nurse, ordering of blood cultures and lactate, and administration of appropriate antibiotics and intravenous fluids. The clinical sepsis champions were also responsible for monitoring compliance with the alert.

The ED sepsis alert system comprised 3 alerts that fired only while the patient was in the ED: the Systemic Inflammatory Response Syndrome (SIRS) advisory alert, which targeted ED nurses, and 2 versions of the sepsis advisory alert, 1 targeting nurses and the other targeting providers, including physicians, physician assistants, and nurse practitioners (Table 1 and Figure 1). All alerts were developed as screening tools for early detection of sepsis. While the alerts had varying triggering criteria, discussed below, they were all suppressed for patients <18 years old and for patients who were designated as “Comfort Care Only” in the EHR.

Table 1.

Description of components of sepsis alert program

Alert name	Trigger	Target provider	Response options
SIRS advisory alert	2/4 SIRS criteria: Temperature >38°C OR <36°C; heart rate >90 beats/min (sinus rhythm); respiratory rate >20 breaths/min or PaCO2 <32 mm Hg; white blood cell count <4 × 109/L, >12 × 109/L OR >10% bands	Nurse	Place nurse sepsis order seta Call sepsis or other critical alerta Already on sepsis protocola No suspected or new signs of infection Need additional information/reassessb
Sepsis advisory alert (nurse)	Systolic blood pressure <90 mm Hg OR lactate ≥4 mg/dL	Nurse	Call sepsis or other critical alerta Already on sepsis protocola No suspected or new signs of infection Need additional information/reassessb
Sepsis advisory alert (provider)	Systolic blood pressure <90 mm Hg OR lactate ≥4 mg/dL	Provider	Place sepsis adult order seta Call sepsis or other critical alerta Already being treated for sepsisa No suspected or new signs of infection Need additional information/reassessb

^aResponses that suggested the need for further evaluation for sepsis.

^bOption discontinued on February 11, 2015.

Figure 1.

Screenshot of SIRS advisory alert.

The SIRS advisory alert was triggered by 2 out of 4 SIRS criteria (Table 1).⁴ These criteria were selected because, at the time, the primary definition of sepsis (Sepsis 2) included 2 of 4 SIRS criteria.⁶ In developing the alert, the goal was to maximize sensitivity, as the team did not want to miss cases; although the team did not do a precision recall assessment, they chose 2 SIRS criteria to trigger the alert, with the assumption that this rule would have a higher sensitivity than ones requiring additional SIRS criteria. Furthermore, these values were captured as part of routine clinical care and did not require additional documentation burden for clinicians, which could reduce compliance.

Upon triggering, the alert presented nurses with an explanation of the triggering criteria, recent relevant vital signs and laboratory results, and response options (Table 1 and Figure 1). Available responses for nurses included calling the sepsis alert or another disease-specific alert (eg, ST-elevation myocardial infarction), each of which was expected to result in an immediate bedside huddle between physician and nurse, or documenting that the patient was either already on a sepsis protocol or had no suspected signs of infection. Additionally, from within the alert, the nurse could open an order set, which included a comprehensive metabolic panel, lactate, urine and blood cultures, chest X-ray, and EKG.

The sepsis advisory alerts were triggered for patients who had a systolic blood pressure <90 mm Hg or a lactate ≥4 mg/dL. These alerts were intended to serve as safeguards for patients who were severely ill but might not have satisfied SIRS criteria. The clinical team chose to name this alert “Sepsis” to highlight sepsis as an often overlooked etiology in patients with unexplained severe illness. The response options were the same as for the SIRS advisory alert, with the exception that the severe illness alert targeting nurses did not have a link to the sepsis order set; the version targeting providers did have the order set response.

The primary clinical outcomes were length of stay (LOS) and in-hospital mortality. LOS was measured as the time from ED arrival to hospital discharge. Secondary clinical outcomes included transfer to the intensive care unit (ICU), ICU LOS, and use of a vasopressor. Process measures included obtaining a lactate (excluding lactate results ≥24 h after ED arrival), time to first lactate, and obtaining blood cultures prior to antibiotic administration.

We assessed a number of covariates, including demographics, vital signs, clinical conditions or comorbidities, and whether the patient had an ICU stay. Demographic covariates included age, sex, race, and insurance. Systolic blood pressure, diastolic blood pressure, and creatinine were the first recorded measures following presentation to the ED. Comorbidities or conditions were based on discharge diagnoses using the Agency for Healthcare Research and Quality Clinical Classification Software; conditions assessed included cardiac arrest, respiratory failure, diabetes, acute myocardial infarction, dysrhythmias, chronic obstructive pulmonary disease, dementia, heart failure, and pneumonia.

We also evaluated utilization metrics of the alert. We measured the number of total alerts that fired. We determined the number of sepsis hospitalizations that had at least 1 alert with any response by a clinician, ie, was not canceled. We calculated the proportion of sepsis hospitalizations with any alert that had a response suggesting active sepsis evaluation or treatment, including open the sepsis order set, call sepsis or other critical alert, and already on sepsis protocol. We also determined the proportion of sepsis hospitalizations with any alert that had a response of no new signs of infection and determined the responses to all of the alerts that were triggered.

Statistical analysis

Baseline characteristics were compared before and after introduction of the alert using chi-squared tests for categorical variables and Wilcoxon rank sum test for continuous variables, as appropriate. We also used these tests to compare differences in outcomes before and after introduction of the alert.

We performed a patient-level interrupted time series study with calendar month as the unit of time. This quasi-experimental design accounts for temporal trends while examining the association between introduction of an intervention and outcomes¹⁶; these associations were analyzed using segmented regression analysis.^17–19 For the co-primary outcome of LOS, as well as secondary outcomes of ICU LOS and time to first lactate, we fit negative binomial models; a logistic model was fit for outcomes of in-hospital mortality, use of vasopressor, lactate results (limited to those within 24 h of ED arrival), and obtaining blood cultures prior to antibiotic administration. In each model, we included 3 variables to measure the relationship of time and the outcome of interest: (1) a continuous time variable to represent the underlying temporal trends; (2) a dummy variable for the date after February 25, 2014, to determine the change in outcome related to the sepsis alert; and (3) a continuous time variable beginning on that date, to represent the change in slope.^17–19 The coefficients of the second and third variables indicated whether the sepsis alert had an immediate or ongoing effect on each outcome, respectively. Models for LOS and mortality were adjusted for all covariates, including demographics, vital signs, clinical conditions, and whether the patient had an ICU stay. Models for other outcomes were adjusted for these same variables except for ICU stay. For outcomes of ICU LOS and time to first lactate, we only modeled hospitalizations with an ICU stay and those with a documented lactate, respectively. We accounted for multiple testing in our analysis using the Bonferroni method,²⁰ with α = 0.006 for each of the 8 outcomes examined.

We determined the sensitivity and positive predictive value (PPV) for the sepsis alert system. PPV was calculated as the total number of alerts that triggered for hospitalized patients with sepsis divided by the total number of alerts. We calculated sensitivity as the proportion of patients hospitalized with sepsis for whom an alert was triggered. Additionally, we determined PPV and sensitivity for each of the 3 individual alerts in the sepsis alert system. We also calculated prevalence as number of hospitalized patients with sepsis divided by all hospitalized adults admitted from the ED.

RESULTS

Of 2144 hospitalizations with a final diagnosis of severe sepsis or septic shock, 838 (39.1%) occurred prior to and 1306 (60.9%) occurred after introduction of the sepsis alert system. Patient characteristics were overall similar between the 2 groups (Table 2). Patients hospitalized after introduction of the sepsis alerts were slightly older (mean age 67.8 vs 66.0 years) and were more likely to have dementia (19.5% vs 14.6%) than patients hospitalized pre-alert.

Table 2.

Baseline characteristics of 2144 hospitalized patients with severe sepsis or septic shock present on admission

Characteristic	All hospitalizations	Prior to sepsis alert	After sepsis alert	P-value
	n = 2144	n = 838	n = 1306
Age, mean (SD)	67.1 (18.9)	66.0 (18.4)	67.8 (19.3)	<0.01
Female	47.2	46.8	47.5	0.75
Black	8.8	9.6	8.4	0.34
Medicaid insurance	2.1	2.2	2.1	0.90
Cardiac arrest	1.9	1.7	2.1	0.51
Respiratory failure	21.5	21.4	21.6	0.89
Diabetes	30.5	30.2	30.7	0.82
Acute myocardial infarction	2.0	1.9	2.0	0.89
Cardiac dysrhythmias	31.8	30.4	32.6	0.28
COPD	15.8	14.3	16.8	0.13
Dementia	17.6	14.6	19.5	<0.01
Heart failure	24.3	24.6	24.1	0.78
Pneumonia	29.4	28.0	30.3	0.27
Systolic blood pressure, mm Hg (SD)	124.3 (25.6)	123.2 (25.6)	125.0 (25.5)	0.12
Diastolic blood pressure, mm Hg (SD)	69.7 (17.1)	70.0 (17.)	69.6 (17.1)	0.28
Creatinine, mg/dL (SD)	1.6 (1.5)	1.7 (1.7)	1.5 (1.4)	0.03

SD: standard deviation; COPD: chronic obstructive pulmonary disease.

During the study period, mean (SD) LOS was 9.2 days (8.9). LOS was shorter among hospitalizations following introduction of the sepsis alert system than prior to the alerts in unadjusted analysis (8.6 vs 10.1; P < .001; Table 3). When considering both temporal trends and patient covariates, we observed a 16% decrease in LOS with introduction of the sepsis alert system, but no change in either the underlying time trend or trends following introduction of the alerts (Table 4). However, this decrease did not quite reach statistical significance when accounting for multiple testing (P = .007 with α = 0.006).

Table 3.

Process measures and clinical outcomes for sepsis hospitalizations that occurred prior to and after the introduction of an EHR sepsis alert system

Outcome	Prior to sepsis alert	After sepsis alert	P-value
	N = 838	N = 1306
Length of stay, days (SD)	10.1 (10.1)	8.6 (7.9)	<0.001
Mortality	8.5	7.0	0.22
Transfer to ICU	36.9	25.8	<0.001
ICU length of stay, days (SD)	1.8 (3.7)	1.2 (3.1)	<0.001
Vasopressor use	28.8	22.7	<0.01
Lactatea	90.7	91.3	0.65
Time to first lactate, days (SD)	0.19 (0.94)	0.16 (0.58)	<0.001
Blood cultures prior to antibiotics	79.0	79.2	0.92

^aExcluding lactate results ≥24 h after ED arrival.

SD: standard deviation; ICU: intensive care unit.

Table 4.

Results of the interrupted time series analysis evaluating the effect of the introduction of the EHR sepsis alert system on process measures and clinical outcomes

Outcome	Rate of change prior to sepsis alert	Change in level, introduction of sepsis alert	Change in rate post-alert
Length of stay	1.00 (0.99-1.01)	0.84 (0.75-0.95)	1.00 (0.98-1.02)
Mortality	1.04 (0.97-1.12)	0.58 (0.29-1.19)	0.96 (0.87-1.06)
Transfer to ICU	0.95 (0.91-1.00)	0.69 (0.45-1.07)	1.07 (1.01-1.13)
ICU length of stay	1.01 (0.99-1.04)	0.98 (0.77-1.26)	0.98 (0.95-1.01)
Vasopressor use	0.99 (0.94-1.04)	0.69 (0.43-1.11)	1.01 (0.95-1.08)
Lactatea	0.99 (0.93-1.07)	1.67 (0.86-3.25)	0.96 (0.88-1.04)
Time to first lactate	0.98 (0.93-1.03)	0.63 (0.39-1.04)	1.07 (1.00-1.14)
Blood cultures prior to antibiotics	1.04 (0.99-1.09)	0.87 (0.56-1.37)	0.95 (0.89-1.00)

^aExcluding lactate results ≥24 h after ED arrival.

The in-hospital mortality rate was 7.6%, with rates of 8.5% prior to and 7.0% following the introduction of the sepsis alert system (Table 3). We found no evidence for differences in mortality in the pre- and post-alert period after adjustment (Table 4). In unadjusted analysis, other clinical measures were improved in hospitalizations following introduction of the sepsis alert; these markers included reduced ICU admissions, reduced ICU LOSs among patients admitted to the ICU, and reduced use of vasopressors (Table 3). In the adjusted interrupted time series analysis for the outcome of ICU transfer, we found a nonsignificant decrease in odds of ICU transfer of 31% (95% CI, −7%, 55%) with introduction of the alert system, followed by a nonsignificant increase in odds of ICU transfer of 7% (95% CI, 1%-13%) per month as compared to the underlying time trend (Table 4); the latter value was not significant (P = .03) in consideration of multiple testing. Introduction of the alert was not associated with other outcome measures in adjusted analyses (Table 4).

In hospitalizations both prior to and following introduction of the sepsis alert system, we observed a high rate of lactate results (Table 3). However, in the post-alert period, lactate results were obtained earlier than in the pre-alert period (mean 3.8 h vs 4.6 h). In the adjusted interrupted time series analysis, there was no difference between groups for this process measure (Table 4). Similarly, we observed no differences in percent of patients who had blood cultures obtained prior to antibiotics administration in hospitalizations that occurred prior to versus following introduction of the sepsis alert system (Tables 3 and 4).

The 3 alerts in the ED sepsis alert system were triggered a total of 97 216 times. Of these triggers, 14 207 targeted patients in the sepsis cohort, for a PPV of 14.6% for any alert. The sensitivity of the alert system was 80.4% for patients hospitalized for sepsis. At least 1 alert was responded to by a clinician, ie, was not canceled, in 96.3% of the 1050 sepsis hospitalizations with an alert. Of these hospitalizations, 722 (68.8%) had a response that suggested an active treatment. Among sepsis hospitalizations with an alert, 258 (24.6%) had at least 1 ED alert that had the response of “no suspected or new signs of infection” and 180 (17.1%) had this response and never a response of active treatment, despite all patients in this cohort having a diagnosis code of severe sepsis or septic shock that was present on admission. The majority of alerts were the SIRS advisory alerts (83 385 of 97 216), which had a sensitivity of 73.0% for sepsis and PPV of 13.0%. The provider and nurse sepsis advisory alerts had sensitivities of 23.0% and 23.8% and PPVs of 26.6% and 22.4%, respectively. Of the 97 216 total alerts, 66.6% were dismissed. The prevalence of sepsis in hospitalized patients admitted from the ED was 9.7%.

DISCUSSION

Following implementation of a novel EHR alert system for sepsis in the ED, we observed an absolute reduction in hospital LOS, although no difference in hospital mortality, for patients hospitalized with sepsis. After adjusting for confounders and temporal trends, the introduction of the alert system was associated with a modest decrease in LOS that did not reach statistical significance. Notably, the alerts had no effect on any either intermediate outcome measures, including ICU admissions and LOS, or process of care measures for sepsis, including time to first lactate measurement or antibiotics prior to blood cultures.

The explanation for our negative results is likely related to findings on characteristics and utilization of the alert system. The PPV of the alert system was quite low. Because of the poor PPV of the alert system, repeated firings likely contributed to the well-documented phenomenon of alert fatigue. The implication of alert fatigue – clinicians frequently ignoring alerts – was suggested by our measures of adoption. Only about two-thirds of patients with documented sepsis had a clinician respond to the alert that sepsis was to be evaluated or treated, and >20% of patients with sepsis had a clinician respond that sepsis was not present. Although nearly all alerts eventually had some response, clinicians may simply have been trying to silence the alert and return to their usual routine, thus defeating the purpose of the alert.

On the positive side, the alerts had a high sensitivity, an important characteristic, as missed cases of sepsis can lead to delayed treatment and associated high mortality.^2,21 Indeed, the sensitivity of the alerts compared favorably to that of other ED-based sepsis alerts.^22,28

Other factors may have contributed to the lack of positive effect of the alert on care processes and outcomes. The isolated alert system trigger may have been insufficient to effect robust changes in ED workflow and clinical outcomes. Conversely, Narayanan et al. showed decreased time to fluids and antibiotics using an alert that was one part of a robust application of resources, including direct communication with attending physicians on all alert firings and a sepsis protocol that mobilized a team consisting of nurse, ED technician, pharmacist, and radiology technicians.¹³ Similarly, Manaktala and Claypool¹¹ emphasized the 5 months of training and change management supplementing their sepsis alert implementation as key drivers in reducing inpatient sepsis-related mortality.¹¹ Notably, this study took place in the inpatient setting, where the prevalence of sepsis is higher than in the ED. Partly as a result of this difference in prevalence, the PPV of the alert used in that study was 50%, as compared to 15% in our ED-based study. The differences in results between our intervention and that deployed in the study by Manaktala suggest that high PPV is critical for successful deployment of clinical decision support interventions. Our clinical team is currently revising the alert criteria to create more targeted versions of the alert to increase PPV in hopes of improving its success.

While we observed a significant decrease in LOS before and after introduction of the alert in unadjusted analysis, the final adjusted analysis suggested that much of the improvement was related to temporal trends in decreased LOS. Given both the risks²³ and financial disincentives²⁴ of extended hospital stays, our hospital has had a number of initiatives to decrease LOS around this period.²⁵ Concurrently, readmission reduction efforts may have contributed to these temporal trends, although prior studies have not found a consistent relationship between LOS and readmissions.^26,27

Some studies have demonstrated a modest improvement in process measures such as increased frequency of lactate collection⁸ and slightly earlier blood cultures²⁸ and antibiotics¹³ following implementation of electronic alerting in the ED, although results have not been consistent. Amland et al.¹² observed a decrease in adverse outcomes over time following introduction of a clinical decision support system targeting hospitalized patients.

However, these studies were limited in that their pre-post design did not account for secular trends that could also explain this difference in process outcomes.^8,13–15,28 Notably, in our unadjusted pre-post analysis, the sepsis alert system was associated with improvements in process measures, such as time to first lactate, as well as outcome measures, including LOS and ICU transfer. However, when we considered temporal trends and covariates, a more robust analytic approach, the alerts no longer demonstrated benefits in any of these measures. This difference in analytic approach may account for the difference between our findings and those of prior positive studies. Similar to our findings, prior studies of EHR-based sepsis alerts found no improvement in mortality with the introduction of alerts.^{8–10,13,15,29}

The high sensitivity and low PPV of our alerting system is in part attributable to the limitations of the definition of sepsis during our study time period. Based on the definitions of SIRS and sepsis in effect through early 2016, the sepsis screening alert fired when 2/4 SIRS criteria were fulfilled. Positive SIRS criteria is not specific to sepsis alone, as many patients who present to the ED will meet 2/4 SIRS criteria and not have sepsis.³⁰ Subsequent to our study, the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis 3) were released, in which the task force unanimously declared that the use of ≥2 SIRS criteria to identify sepsis was insufficiently specific.³¹ This statement is consistent with both our work and a prior study suggesting that the Sepsis 2 definition has a high sensitivity and low specificity for patients in the ED.³⁰

This study has several limitations that should be considered. First, this was a single-site study, so findings may not be applicable to other institutions. Second, there were other quality initiatives at our hospital during the study period that may have affected both ED throughput and appropriate use of critical care resources. We attempted to control for these potential differences in care delivery during the period with our experimental design. Third, the ED was limited to an urgent care center between January 2013 and April 2014 in the aftermath of damage caused by Hurricane Sandy.¹⁷ Sepsis protocols were the same in ED and urgent care, although there may have been differences in patient acuity; nonetheless, we performed rigorous adjustment for patient risk. Fourth, we used ICD-9 coding for our definition of sepsis rather than the gold standard of clinician chart review. Although this coding definition is subject to misclassification, it has been well studied.³² Fifth, prior studies have suggested an increased prevalence of coding of sepsis among hospitalized patients over time.¹ Furthermore, introduction of the alert may have led to increased awareness and documentation of sepsis. Nonetheless, patient characteristics were generally similar before and after introduction of the alert, and our analytic approach considered temporal changes, which would include temporal changes in coding. Sixth, we excluded patients designated as “Comfort Care Only” from our alert triggering to minimize the risk of patients receiving care inconsistent with their stated goals. While this exclusion reduced the number of eligible patients, it is consistent with the approach in other sepsis alert studies.^9–11,13

CONCLUSION

With high-level investment in health information systems by hospitals, syndromic surveillance has been viewed as a promising tool to improve outcomes. Yet our study suggests that implementation of relatively straightforward and basic clinical decision support alerting will not result in satisfactory improvement in process or outcome measures. More concerning, an intervention with many false positives might contribute to alert fatigue, excess time overdiagnosing, and treatment of nonsepsis patients. Our results suggest that more sophisticated approaches to early identification of sepsis patients are needed to consistently improve patient outcomes. Furthermore, this study strongly supports the principle that high PPV is critical for clinical decision support interventions in general.

Funding

This work was supported by Agency for Healthcare Research and Quality grant number K08HS23683.

Competing Interests

The authors have no competing interests to declare.

Contributors

All persons who meet authorship criteria are listed as authors.

JSA, SB, and CTJ conceived of the study. SB and JSA acquired the data. All 4 authors participated in interpretation of the data, drafting the manuscript, and approving the final version.