Epic Sepsis Model Inpatient Predictive Analytic Tool: A Validation Study

OBJECTIVES:

Earlier treatment of sepsis leads to decreased mortality. Epic is an electronic medical record providing a predictive alert system for sepsis, the Epic Sepsis Model (ESM) Inpatient Predictive Analytic Tool. External validation of this system is lacking. This study aims to evaluate the ESM as a sepsis screening tool and determine whether an association exists between ESM alert system implementation and subsequent sepsis-related mortality.

DESIGN:

Before-and-after study comparing baseline and intervention period.

SETTING:

Urban 746-bed academic level 1 trauma center.

PATIENTS:

Adult acute care inpatients discharged between January 12, 2018, and July 31, 2019.

INTERVENTIONS:

During the before period, ESM was turned on in the background, but nurses and providers were not alerted of results. The system was then activated to alert providers of scores greater than or equal to 5, a set point determined using receiver operating characteristic curve analysis (area under the curve, 0.834; p < 0.001).

MEASUREMENTS AND MAIN RESULTS:

Primary outcome was mortality during hospitalization; secondary outcomes were sepsis order set utilization, length of stay, and timing of administration of sepsis-appropriate antibiotics. Of the 11,512 inpatient encounters assessed by ESM, 10.2% (1,171) had sepsis based on diagnosis codes. As a screening test, the ESM had sensitivity, specificity, positive predictive value, and negative predictive value rates of 86.0%, 80.8%, 33.8%, and 98.11%, respectively. After ESM implementation, unadjusted mortality rates in patients with ESM score greater than or equal to 5 and who had not yet received sepsis-appropriate antibiotics declined from 24.3% to 15.9%; multivariable analysis yielded an odds ratio of sepsis-related mortality (95% CI) of 0.56 (0.39–0.80).

CONCLUSIONS:

In this single-center before-and-after study, utilization of the ESM score as a screening test was associated with a 44% reduction in the odds of sepsis-related mortality. Due to wide utilization of Epic, this is a potentially promising tool to improve sepsis mortality in the United States. This study is hypothesis generating, and further work with more rigorous study design is needed.

KEY POINTS

Question: Is the Epic Sepsis Model (ESM) a useful screening tool for identifying patients at risk of a sepsis diagnosis during hospitalization and can implementation of the alert system reduce sepsis mortality?

Findings: At ESM set point greater than or equal to 5, overall discrimination (area under the receiver operating characteristic curve) was 83.4% (p < 0.001) with 86.0% sensitivity, 80.8% specificity, and positive and negative predictive values of 33.8% and 98.1%, respectively. Implementation of the ESM as a screening tool alert system was associated with a reduction in mortality rate among patients with a score greater than or equal to 5 and who had not yet received sepsis-related antibiotics (odds ratio, 0.56; 95% CI, 0.39–0.80).

Meaning: The Epic sepsis alert system is a potentially promising tool for improving sepsis mortality throughout the United States; however, more rigorous controlled studies are needed.

Sepsis is life-threatening organ dysfunction caused by a dysregulated host response to infection (1). The impact of sepsis on healthcare is substantial. The frequency of sepsis in hospitalized patients is 2–6% with a mortality rate of 15–40% (2–4) and costs 23.6 billion dollars (about $73 per person in the United States) per year, equivalent to 6.2% of the U.S. healthcare budget (5, 6).

Several studies found that early identification and treatment of sepsis leads to decreased mortality (7–10). The Updated 2018 Surviving Sepsis Campaign has made recommendations to bundle care into a “hour-1 bundle” with the intention to begin resuscitation and management of sepsis at time of presentation (11). To improve outcomes and to comply with this new “hour-1 bundle” recommendation, early identification systems and notification systems are needed to identify sepsis sooner.

Multiple early detection methods of degree of illness exist such as the modified Early Warning Score and the National Early Warning Score (12–15); however, early detection alert systems specifically for sepsis detection have not been widely adopted. Barriers to implementing these detection methods include system processes issues, and difficulty integrating these various early alert systems into electronic health records. Some hospitals have invested in developing or purchasing alert systems, which have been employed with varying success (16–20). However, these disparate alert systems inhibit widespread dissemination of standardized practice.

Epic Systems is the largest provider of information health technology in the United States. According to Epic, the hospitals and healthcare groups using their electronic medical record (EMR) serve 54% of patients in the United States (21). In 2018, Epic developed a predictive model for sepsis using 500,000 patient encounters, and 80 demographic and clinical variables called the Epic Sepsis Model (ESM) Inpatient Predictive Analytic Tool. The benefit of this alert system is that it is easily integrated into the electronic health record and is generalizable to many healthcare systems.

Our health system implemented Epic as our EMR provider in 2016 and was one of the early adopters of this sepsis alert model. Epic reports its hospital level performance as an area under the receiver operating characteristic curve (AUC) at 0.76–0.83 (22). There has been one other study that externally validated the ESM, and reported an AUC of 0.63, which is substantially worse than the performance reported by Epic (23). Widespread external validation of this tool is important, owing to the high mortality rate and high cost of sepsis. In this study, we set out to externally validate ESM at our institution. The purposes of this study were to determine the best score cut-point for utilization as a screening tool for potential sepsis in our population, and to then assess the ESM as an alert system to improve mortality in patients with sepsis.

MATERIALS AND METHODS

Sample and Setting

The study was conducted at a 746-bed academic level 1 trauma center, Prisma Health, in Greenville, SC. The study began as a quality improvement project to validate the new Epic sepsis analytic tool in our acute-care population. The ESM is a proprietary tool developed to assess the risk of sepsis in patients. It uses a set of approximately 80 data elements, encompassing various clinical parameters and patient characteristics. These elements include vital signs, laboratory results, comorbidities, and demographic factors. Each data element is weighted according to its relative contribution in determining the risk of sepsis and a numerical score is generated that reflects this risk. One feature of the ESM tool is its ability to identify the specific data elements that are contributing most significantly to an elevated score.

This study was conducted as a before-and-after study. The Prisma Health Institutional Review Board reviewed and approved this study (protocol number Pro00081382) with a waiver for informed consent.

Determination of Cut-Point Score

All adult inpatients were assessed using the ESM from January 12, 2018, to April 30, 2018. During this timeframe, the ESM was turned on in the background; however, the data were not visible to staff. There was a total of 11,512 inpatient encounters with ESM data available for analysis. For the “gold standard” definition of sepsis, we used the International Classification of Diseases, 10th Revision sepsis diagnosis codes listed in Appendix A (http://links.lww.com/CCX/B217). Thus, any patient with an identified sepsis diagnosis code, in any sequence, was counted as sepsis positive. The ESM screening test “cut point” was determined using receiver operating characteristic (ROC) curve analysis. The ESM is continually updated and the maximum ESM score during the hospitalization was used in the ROC analysis.

Healthcare Worker Training

Next, we educated physicians and nurses on the ESM. Education for the physicians was performed using a required completion of a Computer Based Training module. Physicians were educated on how the tool was developed with the goal of earlier recognition and treatment of sepsis. The nurses were provided sepsis alert system education during staff meetings and explained the rationale, importance, and objectives of the tool. An ESM score above the threshold was to be treated as a “critical value,” whereby the nurses were to call physicians, provide physicians with the pertinent clinical information that led to the elevated score, and receive physician instruction regarding ongoing patient care.

Evaluating Model As Alert System

After clinical team instruction was completed, the ESM score alert system was turned on and implemented in the acute care settings. We then compared patient characteristics and outcomes between the baseline period (e.g., pre-implementation when the score was not visible to staff from January 12, 2018, to October 21, 2018) and the intervention period (e.g., post-implementation, when the ESM education had been completed, the score was made visible to staff, and there was an expectation of staff to act on the ESM score from October 22, 2018, to July 31, 2019). The study population for analysis in both time frames consisted of those patients with an ESM score above the optimal cutoff; these patients were deemed “at risk” for a sepsis diagnosis during the hospitalization. Patients who received antibiotics for suspected sepsis before arrival to the hospital were excluded from the study analysis (n = 371). The primary outcome was mortality during the hospitalization. Secondary outcomes were sepsis order set utilization, hospital and ICU length of stay (LOS), and timing of administration of sepsis-appropriate antibiotics.

Statistical Analysis

Bivariate analyses of the differences in factors between the two time periods were conducted using the chi-square test for categorical data and the Wilcoxon rank-sum test for continuous data. Multivariable logistic regression analysis was used to assess independent predictive factors for the outcome of mortality. Factors with p values of less than 0.20 in bivariate analyses were selected for inclusion in the logistic regression model. Factors with a clinical association with mortality (i.e., patient age and need for intensive care), or an observed association with the study time periods were included in the logistic models. p values of less than 0.05 and nonoverlapping Fisher exact 95% CIs were used to assess statistical significance. SAS Enterprise Guide 8.3 software was used for all statistical analyses (SAS Institute, Cary, NC).

RESULTS

Determination of Cut-Point Score

From January 12, 2018, to April 30, 2018, a total of 11,512 inpatient encounters were assessed by the ESM. Of these 11,512 encounters, 10.2% (1,171) were identified as having sepsis based on diagnosis codes. For identifying the ESM “cut-point” screening value, we used the maximum score during the hospitalization. The average maximum score of patient encounters with a diagnosis of sepsis was 14.9 (range, 1–64) and the average maximum score of patient encounters with no diagnosis of sepsis was 3.2 (range, 0–67). Of the 1,171 patients with a sepsis diagnosis, 1,007 (86.0%) had a maximum ESM score greater than or equal to 5 (i.e., sensitivity); of the 10,341 patients without a sepsis diagnosis, 8,359 (80.8%) had a maximum ESM score less than 5 (i.e., specificity). Positive and negative predictive values were 33.8% and 98.1%, respectively. An ESM score of 5 or greater yielded the best result to serve as a screening tool for sepsis diagnosis during hospitalization (Supplemental Fig. 1, http://links.lww.com/CCX/B218; AUC, 0.834; p < 0.001).

Evaluating Model As Alert System

Results of bivariate analyses between the two time periods are provided for three subgroups of patients in order of clinical specificity: 1) patients with ESM score greater than or equal to 5, 2) patients with score greater than or equal to 5 and a diagnosis of sepsis, and 3) patients with score greater than or equal to 5, a diagnosis of sepsis, and no receipt of antibiotics before ESM score greater than or equal to 5 (Tables 1–3). Tables 4 and 5 provide multivariable logistic regression analysis results in the latter two populations.

TABLE 1.

Characteristics and Outcomes of Patients With Epic Sepsis Model Score Greater Than or Equal to 5

Characteristic	Hospitalizations, n (%)		p
	Baseline Period	Intervention Period
n	4,397	5,159	—
Age, median (IQR), yr	65 (54–76)	65 (54–76)	0.689
Gender
Female	2,065 (47.0)	2,491 (48.3)	0.198
Male	2,332 (53.0)	2,668 (51.7)
Race/ethnicity
Caucasian	3,232 (73.5)	3,762 (72.9)	0.600
African American	976 (22.2)	1,172 (22.7)
Hispanic	104 (2.4)	137 (2.7)
Other	85 (1.9)	88 (1.7)
Admitted through emergency department	3,234 (73.6)	3,995 (77.4)	< 0.001
Sepsis score (first score ≥ 5), median (IQR)	6 (5–7)	6 (5–7)	0.001
Sepsis order set used	907 (20.6)	1,049 (20.3)	0.722
Diagnosis of sepsis	1,477 (33.6)	1,511 (29.3)	< 0.001
Hospital length of stay, median (IQR), d	6.3 (3.7–11.8)	6.0 (3.3–10.8)	< 0.001
Admitted to ICU during hospitalization	1,722 (39.2)	2,194 (42.5)	< 0.001
ICU length of stay, median (IQR), d	2.8 (1.5–6.0)	2.5 (1.3–4.8)	< 0.001
Mortality	524 (11.9)	519 (10.1)	0.004

IQR = interquartile range.

TABLE 3.

Characteristics and Outcomes of Patients With Epic Sepsis Model Score Greater Than or Equal to 5, Diagnosis of Sepsis and No Sepsis-Related IV Antibiotic Administered Prior to Epic Sepsis Model Greater Than or Equal to 5

Characteristic	Hospitalizations, n (%)		p
	Baseline Period	Intervention Period
n	304	628	—
Age, median (IQR), yr	69 (57–79)	66 (53–76)	0.023
Gender
Female	136 (44.7)	316 (50.3)	0.110
Male	168 (55.3)	312 (49.7)
Race/ethnicity
Caucasian	219 (72.0)	450 (71.7)	0.250
African American	66 (21.7)	147 (23.4)
Hispanic	15 (4.9)	17 (2.7)
Other	4 (1.3)	14 (2.2)
Admitted through emergency department	229 (75.3)	497 (79.1)	0.189
Sepsis score (first score ≥ 5), median (IQR)	6 (5–7)	6 (5–7)	0.338
Sepsis order set used	105 (34.5)	271 (43.2)	0.012
Hospital length of stay, median (IQR), d	6.1 (3.8–13.0)	7.2 (3.9–12.9)	0.380
Admitted to ICU during hospitalization	115 (37.8)	313 (49.8)	< 0.001
ICU length of stay, median (IQR), d	3.2 (1.7–8.3)	2.8 (1.3–5.2)	0.020
Time from ESM ≥ 5 to antibiotic administration, median (IQR), min	150 (45–900)	90 (33–255)	< 0.001
Time from ESM ≥ 5 to antibiotic administration ≤ 180 min (3 hr)	169 (55.6)	438 (69.8)	< 0.001
Mortality	74 (24.3)	100 (15.9)	0.002

ESM = Epic Sepsis Model, IQR = interquartile range.

TABLE 4.

Multivariable Logistic Regression Analysis for Predictors of Hospital Mortality in Patients With a Sepsis Diagnosis and Epic Sepsis Model Score Greater Than or Equal to 5 (n = 2,988)

Predictive Factor	OR (95% CI)	p
Patient Age (10-yr intervals)	1.21 (1.13–1.29)	< 0.001
Admitted through the emergency department
No	Referent
Yes	0.78 (0.61–1.00)	0.051
Admitted to ICU during hospitalization
No	Referent
Yes	2.62 (2.13–3.22)	< 0.001
Order set used
No	Referent
Yes	0.70 (0.57–0.87)	< 0.001
Study time period
Baseline	Referent
Intervention	0.71 (0.58–0.87)	< 0.001

OR = odds ratio.

TABLE 5.

Multivariable Logistic Regression Analysis for Predictors of Hospital Mortality in Patients With a Sepsis Diagnosis and No Sepsis-Related IV Antibiotic Administered Prior to Epic Sepsis Model Score Greater Than or Equal to 5 (n = 932)

Predictive Factor	OR (95% CI)	p
Patient Age (10-yr intervals)	1.18 (1.06–1.32)	0.003
Admitted through the emergency department
No	Referent
Yes	0.60 (0.41–0.89)	0.012
Admitted to ICU during hospitalization
No	Referent
Yes	2.46 (1.73–3.49)	< 0.001
Order set used
No	Referent
Yes	0.85 (0.58–1.25)	0.412
Time from Epic Sepsis Model ≥ 5 to antibiotic administration
> 3 hr	Referent
≤ 3 hr	0.83 (0.57–1.21)	0.328
Study time period
Baseline	Referent
Intervention	0.56 (0.39–0.80)	0.001

OR = odds ratio.

Observed mortality rates and 95% CIs for baseline and intervention time periods are presented for all inpatients, subgroups of patients with and without a sepsis diagnoses, and ESM scores greater than or equal to 5 and less than 5 (Fig. 1). Statistically significant differences were noted between the baseline and intervention periods in the ESM score greater than or equal to 5 population (all patients and patients with a sepsis diagnosis); however, no differences were discerned in the overall population or in those with ESM scores less than 5.

Figure 1.

Observed mortality rates by time period, Epic Sepsis Model (ESM) group, and presence or absence of sepsis diagnosis. Asterisk indicates a statistically significant difference in mortality rates between baseline and intervention time periods.

In the sepsis “alert” population (i.e., ESM score ≥ 5), statistically significant differences between the two time periods were noted in the proportions of patients with a diagnosis of sepsis (Table 1), the proportion being admitted through the emergency department (ED) (Tables 1 and 2), and the proportion having an ICU admission during the hospital stay (Tables 1–3). Median differences occurred in sepsis score (Tables 1 and 2), hospital LOS (Table 1), and ICU LOS (Tables 1–3). Mortality rates were statistically significantly lower during the intervention period compared with the baseline period in all three populations—10.1% versus 11.9% (Table 1; p = 0.004), 14.5% versus 18.5% (Table 2; p = 0.003), and 15.9% versus 24.3% (Table 3; p = 0.002).

TABLE 2.

Characteristics and Outcomes of Patients With Epic Sepsis Model Score Greater Than or Equal to 5 and Diagnosis of Sepsis

Characteristic	Hospitalizations, n (%)		p
	Baseline Period	Intervention Period
n	1,477	1,511	—
Age, median (IQR), yr	65 (53–76)	64 (53–75)	0.282
Gender
Female	715 (48.4)	710 (47.0)	0.437
Male		762 (51.6)	801 (53.0)
Race/ethnicity
Caucasian	1,114 (75.4)	1,111 (73.5)	0.477
African American	289 (19.6)	330 (21.8)
Hispanic	42 (2.8)	41 (2.7)
Other	32 (2.2)	29 (1.9)
Admitted through emergency department	1,172 (79.4)	1,258 (83.3)	0.006
Sepsis score (first score ≥ 5), median (IQR)	6 (5–8)	6 (5–7)	0.018
Sepsis order set used	680 (46.0)	731 (48.4)	0.200
Hospital length of stay, median (IQR), d	6.8 (4.0–12.8)	7.0 (3.8–12.5)	0.841
Admitted to ICU during hospitalization	644 (43.6)	768 (50.8)	< 0.001
ICU length of stay, median (IQR), d	2.9 (1.5–6.1)	2.7 (1.3–5.2)	0.025
Mortality	273 (18.5)	219 (14.5)	0.003

IQR = interquartile range.

In the population of patients who had not received antibiotics prior to ESM score greater than or equal to 5 (Table 3), there was increased utilization of the sepsis order set during the intervention period (43.2% vs 34.5%; p = 0.012) and decreased time to administration of antibiotics after the ESM score greater than or equal to 5 alert (median 90 vs 150 min; p < 0.001). Receipt of antibiotics within three hours of the ESM alert increased from 55.6% to 69.8% during the baseline and intervention periods, respectively (p < 0.001).

Multivariable logistic regression analyses results for predictors of hospital mortality are provided in Tables 4 and 5. In patients with ESM Scores greater than or equal to 5 and a sepsis diagnosis (Table 4), increasing age (10-yr intervals) and admission to the ICU were independently associated with increased risk of mortality (odds ratios [ORs], 1.21 and 2.62, respectively). Conversely, order set utilization and interventional time period were associated with decreased risk of mortality (OR, 0.70 and 0.71, respectively). In patients with ESM scores greater than or equal to 5, a diagnosis of sepsis, and no IV antibiotics prior to ESM greater than or equal to 5 (Table 5); patient age and admission to the ICU were independently associated with increased risk of mortality (ORs, 1.18 and 2.46, respectively). Admission through the ED and the interventional time period were associated with decreased risk of mortality (ORs, 0.60 and 0.56, respectively); however, shorter time to antibiotic administration and order set utilization did not reach statistical significance in this smaller subset of patients.

DISCUSSION

This is one of the first studies attempting to externally validate the ESM Inpatient Predictive Analytic Tool and demonstrate its effect on mortality. Using this screening tool, we observed an absolute unadjusted mortality rate reduction of 4% in all patients with ESM greater than or equal to 5 and a sepsis diagnosis, and of 8% in those who had not received antibiotics before ESM score was greater than or equal to 5, respectively. In addition, the corresponding risk-adjusted ORs were 0.71 and 0.56 in the two populations comparing the intervention time period to the baseline time period. While a causal relationship cannot be definitively identified in this uncontrolled before-and-after study, the decrease in mortality rate observed was accompanied by differences in process measures that lend support to a causal effect. These include a decreased time to antibiotic administration, which was seen in bivariate analyses (Table 3) but did not reach statistical significance in multivariate analyses (Table 5). Similarly, sepsis order set usage increased in the intervention time period and was an independent predictor of decreased mortality in the ESM greater than or equal to 5 with sepsis diagnosis population (Table 4). However, this did not reach statistical significance in the smaller subset of patients who had not yet received antibiotics (Table 5).

Several other studies have also demonstrated the effectiveness of a sepsis early warning system (18–20, 23–27). These studies have also shown an improvement in early intervention for antibiotic administration and escalation (18, 26), diagnostic test (18–20), improvement of LOS (19, 28), and mortality (19, 22, 23, 25, 26). Another potential cause for this decrease mortality is the early admission of septic patients to the ICU. Although we can speculate on the reasons behind the increased and early admissions to the ICU observed in the intervention group, we cannot definitively establish a causal relationship based on the data obtained from our study.

However, not all studies have found electronic sepsis alert systems to improve patient outcomes (17, 28). A recent study from the University of Michigan by Wong et al (23) looked at 27,697 patients and concluded that the ESM had “poor discrimination and calibration in predicting the onset of sepsis.” There are several limitations of the study by Wong et al (23) that could explain the differing results. First, in the study by Wong et al (23), the score threshold for alerting the physicians for the possibility of sepsis was arbitrarily chosen at a score threshold of 6 or higher without validating the scoring system at their institution. Alternatively, our study used a cut point of 5 after an extensive validation period at our institution. Second, Wong et al (23) found that only 7% of patients with sepsis who were missed by a clinician based on timely administration of antibiotics prior to an ESM score of greater than 6, while 31.2% of patients in our study with sepsis were not on sepsis-appropriate antibiotics prior to the ESM score of greater than or equal to 5. This large discrepancy of already being on the appropriate IV antibiotics between the study by Wong et al (23) and the study reported here might be explained by the lower cut point of 5 we used, or different prescribing patterns of antibiotics between hospitals. While the study by Wong et al (23) was a retrospective review after the alert system was already implemented, the strength of our study is that we were able to compare sepsis-related outcomes using ESM scores prior to and after making them available for clinical use.

Another recent trial by Downing et al (17) failed to find a difference in sepsis-related outcomes including mortality, antibiotics given, lactic acid order, or rate of ICU admission/LOS with implementation of a sepsis alert system. The study by Downing et al (17) was a randomized controlled trial of 1,123 patients using a sepsis alert based on the definition of systemic inflammatory response syndrome (SIRS) plus a source of infection. Clinicians were alerted whether the patients were on antibiotics or not. A substantial portion of these patients (66%) were already on antibiotics when the alert system paged; therefore, less than 200 patients in each group did not have antibiotics when the sepsis alert fired. With a false positive rate of 40% in their study, approximately 100 patients in each group had sepsis without antibiotics prior to alert; therefore, the study was not powered to demonstrate a difference in mortality (17).

Sepsis alert systems that use clinical information not exclusively based on SIRS criteria tend to demonstrate improvement of more meaningful patient centered outcomes such as mortality (19, 24, 25, 27) compared with studies that had the presence of SIRS as an indication for alert (17, 20, 28). As of this article, Guirgis et al (29) is the only study, which demonstrated a mortality benefit for septic patients that used SIRS criteria for sepsis for an alert system.

Alert fatigue is also a well-recognized weakness of sepsis warning systems. An alert system becomes less efficacious in changing clinical practice as the frequency of the alerts increases or as the time the alert system has been in place increases. A study by Austrian et al (30) examined a sepsis alert system in the ED with 2/4 SIRS criteria being met. In that study, the positive predictive value (PPV) of the alert was only 13.0%, and there were 97,216 sepsis alerts over a 2-year period (2). The Sepsis alert system that we implemented using the ESM had a better PPV of sepsis at 33.8%. While a low PPV may raise concerns about the accuracy of positive test results, this trade-off is justified in certain clinical scenarios such as sepsis where there is a high disease prevalence in the hospital and a low PPV is still accompanied by a reasonable absolute number of true positives. By setting a cut point for “screening” that maximized sensitivity we ensured that a greater number of potential sepsis patients would receive the alert and the possibility of earlier intervention. With a PPV of 33.8%, roughly one in every three patients evaluated for sepsis would have a diagnosis of sepsis. Approximately 6–7% of the inpatient population in our hospital had an ESM score alert, which is roughly 45–50 patients per day. Physicians would be alerted only for patients who were not already being treated for sepsis. This less frequent firing of the sepsis alert system may mitigate alert fatigue as a possible reason for our improved results.

The present study has several limitations that should be acknowledged. First, this study used billing diagnosis codes to identify patients with sepsis. To study the outcomes of sepsis, an accurate and consistent way to retrospectively identify large groups of septic patients is necessary; however, no gold standard exists (31). This project began as a quality improvement effort, which relies on billing codes to identify sepsis. Although these billing codes have poor construct validity, they have wide availability and application to other institutions (32–34). A detailed human review for a more specific definition of sepsis was not feasible due to the measurement burden on this large of a scale.

Second, it is important to note that this investigation employed an uncontrolled before-and-after study design, which is inherently susceptible to bias. The primary vulnerability associated with before-and-after studies is the potential bias introduced by temporal trends or other longitudinal changes that occur between the before and after periods, thereby influencing the study outcomes. While efforts were made to minimize confounding factors, such as the absence of institution-wide changes in sepsis treatment during the study period, it should be recognized that the physicians and nurses involved in the study were educated on the ESM. This education may have contributed to an increased overall awareness of sepsis among the healthcare providers, potentially leading to a reduced time to administer antibiotics. Consequently, this could have influenced the observed outcomes.

Furthermore, it is important to acknowledge that this study focused on a specific single institutional setting, which may limit the generalizability of the findings to other contexts. Factors such as variations in healthcare practices, patient demographics, or resource availability could affect results in different healthcare settings.

Critics of sepsis early warning systems may postulate that the over-reliance of early warning systems for sepsis may increase the use of inappropriate antibiotics, testing, and costs in patients who had a high ESM score but did not eventually have sepsis. Clinical judgment remains essential, and further efforts are needed to improve the specificity of the ESM.

CONCLUSIONS

In this uncontrolled before-and-after study, we observed a 44% reduction in the odds of sepsis-related mortality after implementation of the ESM alert (OR, 0.56; 95% CI, 0.39–0.80). Due to the wide utilization of Epic for EMRs and the ease of integration into the EMR, this is a promising tool for an immediate impact on the improvement of sepsis mortality throughout the United States. Our study is hypothesis generating, and further work should use a controlled cohort or interrupted time series design to minimize bias and strengthen conclusions regarding the potential impact of the ESM alert system on sepsis mortality.