Abstract
Background
Early recognition and timely intervention significantly reduce sepsis-related mortality.
Objective
Describe the development, implementation and impact of an Early Warning and Response System (EWRS) for Sepsis.
Design
After tool derivation and validation, a pre/post study with multivariable adjustment measured impact.
Setting
Urban academic healthcare system
Patients
Adult non-ICU patients admitted to acute inpatient units from: 10/01–10/31/2011 for tool derivation, 06/06–07/05/2012 for tool validation, and 06/06–09/04/2012 and 06/06–09/04/2013 for the pre/post analysis.
Intervention
An EWRS in our electronic health record monitored laboratory values and vital signs in real time. If a patient had >= 4 predefined abnormalities at any one time, the provider, nurse, and rapid response coordinator were notified and performed an immediate bedside patient evaluation.
Measurements
Screen positive rates, test characteristics, predictive values and likelihood ratios; system utilization; and resulting changes in processes and outcomes.
Results
The tool’s screen positive, sensitivity, specificity, and positive and negative predictive values and likelihood ratios for our composite of intensive care unit (ICU) transfer, rapid response team call or death in the derivation cohort was 6%, 16%, 97%, 26%, 94%, 5.3 and 0.9, respectively. Validation values were similar. The EWRS resulted in a statistically significant increase in early sepsis care, ICU transfer, and sepsis documentation, and decreased sepsis mortality and increased discharge to home, although neither of these latter two findings reached statistical significance.
Conclusions
An automated prediction tool identified at risk patients and prompted a bedside evaluation resulting in more timely sepsis care, improved documentation, and a suggestion of reduced mortality.
Keywords: Electronic health record, electronic medical record, information technology, early warning system, sepsis and shock
BACKGROUND
There are as many as 3 million cases of severe sepsis and 750,000 resulting deaths in the US annually.1 Interventions such as goal directed resuscitation and antibiotics can reduce sepsis mortality, but their effectiveness depends on early administration. Thus, timely recognition is critical.2–5
Despite this, early recognition in hospitalized patients can be challenging. Using chart documentation as a surrogate for provider recognition, we recently found only 20% of patients with severe sepsis admitted to our hospital from the emergency department were recognized.6 Given these challenges, there has been increasing interest in developing automated systems to improve the timeliness of sepsis detection.7–10 Systems described in the literature have varied considerably in triggering criteria, effector responses, and study settings. Of those examining the impact of automated surveillance and response in the non-intensive care unit (ICU) acute inpatient setting, results suggest an increase in the timeliness of diagnostic and therapeutic interventions 10, but less impact on patient outcomes 7. Whether these results reflect inadequacies in the criteria used to identify patients (parameters or their thresholds) or an ineffective response to the alert (magnitude or timeliness) is unclear.
Given the consequences of severe sepsis in hospitalized patients, as well as the introduction of vital sign (VS) and provider data in our electronic health record (EHR), we sought to develop and implement an electronic sepsis detection and response system to improve patient outcomes. This study describes the development, validation and impact of that system.
METHODS
Setting and Data Sources
The University of Pennsylvania Health System (UPHS) includes three hospitals with a capacity of over 1,500 beds and 70,000 annual admissions. All hospitals use the EHR Sunrise Clinical Manager version 5.5 (Allscripts, Chicago, Illinois). The study period began in October 2011 when VS and provider contact information became available electronically. Data were retrieved from the Penn Data Store, which includes professionally coded data as well as clinical data from our EHRs. The study received expedited approval and a HIPAA waiver from our Institutional Review Board.
Development of the Intervention
The Early Warning and Response System for Sepsis (EWRS) was designed to monitor laboratory values and VSs in real time in our inpatient EHR to detect patients at risk for clinical deterioration and development of severe sepsis. The development team was multi-disciplinary, including informaticians, physicians, nurses and data analysts from all three hospitals.
To identify at risk patients, we used established criteria for severe sepsis, including the systemic inflammatory response syndrome criteria (temperature <36°C or >38°C; heart rate > 90 beats/minute; respiratory rate >20 breaths/minute or PaCO2 <32mm Hg; and total WBC count <4,000 or >12,000 or >10% bands) coupled with criteria suggesting organ dysfunction (cardiovascular dysfunction based on a systolic blood pressure < 100 mm Hg, and hypoperfusion based on a serum lactate measure >2.2 mmol/L [the threshold for an “abnormal” result in our lab]).11, 12
To establish a threshold for triggering the system, a derivation cohort was used and defined as patients admitted between 10/1–31/2011 to any inpatient acute care service. Those less than 18 years old or admitted to hospice, research, and obstetrics services were excluded. We calculated a risk score for each patient, defined as the sum of criteria met at any one time during their visit. At any given point in time, we used the most recent value for each criteria, with a look back period of 24 hours for VSs and 48 hours for labs. The minimum and maximum number of criteria that a patient could achieve at any one time was zero and six, respectively. We then categorized patients by the maximum number of criteria achieved, and estimated the proportion of patients in each category who: 1) were transferred to an ICU during their hospital visit; 2) had a rapid response team (RRT) called during their visit; 3) died during their visit; 4) had a composite of 1, 2, or 3; or 5) were coded as sepsis at discharge (see the Supplement for further information). Once a threshold was chosen, we examined the time from first trigger to: 1) any ICU transfer, 2) any RRT, 3) death, or 4) a composite of 1, 2 or 3. We then estimated the screen positive rate, test characteristics, predictive values and likelihood ratios of the specified threshold.
The efferent response arm of the EWRS included: the covering provider (usually an intern), the bedside nurse, and rapid response coordinators, who were engaged from the outset in developing the operational response to the alert. This team was required to perform a bedside evaluation within 30 minutes of the alert, and enact changes in management if warranted. The rapid response coordinator was required to complete a three question follow-up assessment in the EHR asking whether all three team members gathered at the bedside, the most likely condition triggering the EWRS, and whether management changed (Supplement Figure 1). To minimize the number of triggers, once a patient triggered an alert, any additional alert triggers during the same hospital stay were censored.
Implementation of the EWRS
All inpatients on non-critical care services were screened continuously. Hospice, research, and obstetrics services were excluded. If a patient met the EWRS criteria threshold, an alert was sent to the covering provider and rapid response coordinator by text page. The bedside nurses, who do not carry text enabled devices, were alerted by pop-up notification in the EHR (Supplement Figure 2). The notification was linked to a “task” that required nurses to verify in the EHR the VSs triggering the EWRS, and adverse trends in VSs or labs (Supplement Figure 3).
The Pre-Implementation (“Silent”) Period and EWRS Validation
The EWRS was initially activated for a pre-implementation “silent period” (6/6–9/4/2012) to both validate the tool and provide baseline data to which the post-implementation period was compared. During this time, new admissions could trigger the alert, but notifications were not sent. We used admissions from the first 30 days of the pre period to estimate the tools screen positive rate, test characteristics, predictive values, and likelihood ratios.
The Post-Implementation (“Live”) Period and Impact Analysis
The EWRS went “live” 9/12/2012, upon which new admissions triggering the alert would result in a notification and response. Unadjusted analyses using the chi-square test for dichotomous variables and the Wilcoxon rank sum test for continuous variables compared demographics and the proportion of clinical process and outcome measures for those admitted during the silent period (6/6–9/4/2012) and a similar timeframe one year later when the intervention was live (6/6–9/4/2013). To be included in either of the time periods, patients had to trigger the alert during the period and be discharged within 45 days of the end of the period. The pre and post sepsis mortality index (SMI) was also examined. See the Supplement for a detailed description of study measures. Multivariable regression models estimated the impact of the EWRS on process and outcome measures, adjusted for differences between the patients in the pre and post periods with respect to age, gender, Charlson index on admission, admitting service, hospital, and admission month. Logistic regression models examined dichotomous variables. Continuous variables were log transformed and examined using linear regression models. Cox regression models explored time to ICU transfer from trigger. Among patients with sepsis, a logistic regression model was used to compare the odds of mortality between the silent and live periods, adjusted for expected mortality, both within each hospital and across all hospitals.
Because there is a risk of providers becoming overly reliant on automated systems and overlooking those not triggering the system, we also examined the discharge disposition and mortality outcomes of those in both study periods not identified by the EWRS.
The primary analysis examined the impact of the EWRS across UPHS; we also examined the EWRS impact at each of our hospitals. Lastly, we performed subgroup analyses examining the EWRS impact in those assigned an ICD-9 code for sepsis at discharge or death. All analyses were performed using SAS v9.3 (SAS Institute Inc., Cary, NC).
RESULTS
In the derivation cohort, 4,575 patients met inclusion criteria. The proportion of those in each category (0–6) achieving our outcomes of interest are described in Supplement Table 1. We defined a “positive” trigger as a score >=4, as this threshold identified a limited number of patients (3.9% [180/4575]) with a high proportion experiencing our composite outcome (25.6% [46/180])). The proportion of patients with an EWRS score >= 4 and their time to event by hospital and health system is described in Supplement Table 2. Those with a score >= 4 were almost four times as likely to be transferred to the ICU, almost seven times as likely to experience an RRT, and almost ten times as likely to die. The screen positive, sensitivity, specificity, and positive and negative predictive values and likelihood ratios using this threshold and our composite outcome in the derivation cohort was 6%, 16%, 97%, 26%, 94%, 5.3, and 0.9, respectively, and were 6%, 17%, 97%, 28%, 95%, 5.7, and 0.9, respectively, in our validation cohort.
In the pre period, 3.8% of admissions (595/15,567) triggered the alert, as compared to 3.5% (545/15,526) in the post period. Demographics were similar across periods, except in the post period patients were slightly younger, and had a lower Charlson co-morbidity index at admission.(Table 1) The distribution of alerts across medicine and surgery services were similar. (Table 1)
Table 1.
Descriptive statistics of the study population before and after implementation of the early warning and response system
| Hospitals A-C | ||||
|---|---|---|---|---|
| Pre | Post | p-value | ||
| N encounters | 15,567 | 15,526 | ||
| N alerts | 595 (4%) | 545 (4%) | 0.14 | |
| Age (years) | Median (IQR) | 62.0 (48.5 – 70.5) | 59.7 (46.1 – 69.6) | 0.04 |
| Female | 298 (50%) | 274 (50%) | 0.95 | |
| Race | White | 343 (58%) | 312 (57%) | 0.14 |
| Black | 207 (35%) | 171 (31%) | . | |
| Other | 23 (4%) | 31 (6%) | . | |
| Unknown | 22 (4%) | 31 (6%) | . | |
| Admission type | Elective | 201 (34%) | 167 (31%) | 0.40 |
| ED | 300 (50%) | 278 (51%) | . | |
| Transfer | 94 (16%) | 99 (18%) | . | |
| BMI (kg/m2) | Median (IQR) | 27.0 (23.0 – 32.0) | 26.0 (22.0 – 31.0) | 0.24 |
| Previous ICU admission | 137 (23%) | 127 (23%) | 0.91 | |
| RRT before alert | 27 (5%) | 20 (4%) | 0.46 | |
| Admission Charlson Index | Median (IQR) | 2.0 (1.0 – 4.0) | 2.0 (1.0 – 4.0) | 0.04 |
| Admit service | Medicine | 398 (67%) | 364 (67%) | 0.18 |
| Surgery | 173 (29%) | 169 (31%) | . | |
| Other | 24 (4%) | 12 (2%) | . | |
| Service where alert fired | Medicine | 391 (66%) | 365 (67%) | 0.18 |
| Surgery | 175 (29%) | 164 (30%) | . | |
| Other | 29 (5%) | 15 (3%) | . | |
BMI: body mass index, DRG: diagnosis related group, ED: emergency department, ICU: intensive care unit, IQR: interquartile range, RRT: rapid response team.
In our post period, 99% of coordinator pages and over three quarters of provider notifications were sent successfully. Almost three quarters of nurses reviewed the initial alert notification, and over 99% completed the electronic data verification and adverse trend review, with over half documenting adverse trends. Ninety five percent of the time the coordinators completed the follow-up assessment. Over 90% of the time, the entire team evaluated the patient at bedside within 30 minutes. Almost half of the time, the team thought the patient had no critical illness. Over a third of the time, they thought the patient had sepsis, but reported over 90% of the time that they were aware of the diagnosis prior to the alert. Supplement Table 3 includes more details about the responses to the electronic notifications and follow-up assessments.
In unadjusted and adjusted analyses, ordering of antibiotics, intravenous fluid boluses, lactate and blood cultures within 3 hours of the trigger increased significantly, as did ordering of blood products, chest radiographs, and cardiac monitoring within 6 hours of the trigger.(Tables 2 and 3)
Table 2.
Clinical process measures before and after implementation of the early warning and response system
| Hospitals A-C | |||
|---|---|---|---|
| Pre | Post | p-value | |
| N alerts | 595 | 545 | |
| >= 500cc IV bolus order <3hrs | 92 (15%) | 142 (26%) | <.01 |
| IV/PO antibiotic order <3hrs | 75 (13%) | 123 (23%) | <.01 |
| IV/PO sepsis antibiotic order <3hrs | 61 (10%) | 85 (16%) | <.01 |
| Lactic acid order <3hrs | 57 (10%) | 128 (23%) | <.01 |
| Blood culture order <3hrs | 68 (11%) | 99 (18%) | <.01 |
| Blood gas order <6hrs | 53 (9%) | 59 (11%) | 0.28 |
| CBC or BMP <6 hrs | 247 (42%) | 219 (40%) | 0.65 |
| Vasopressor <6hrs | 17 (3%) | 21 (4%) | 0.35 |
| Bronchodilator administration <6hrs | 71 (12%) | 64 (12%) | 0.92 |
| RBC, plasma or platelet transfusion order <6hrs | 31 (5%) | 52 (10%) | <.01 |
| Naloxone order <6hrs | 0 (0%) | 1 (0%) | 0.30 |
| AV node blocker order <6hrs | 35 (6%) | 35 (6%) | 0.70 |
| Loop diuretic order <6hrs | 35 (6%) | 28 (5%) | 0.58 |
| CXR <6hrs | 92 (15%) | 113 (21%) | 0.02 |
| CT head, chest or abd < 6hrs | 29 (5%) | 34 (6%) | 0.31 |
| Cardiac monitoring (EKG or telemetry) <6hrs | 70 (12%) | 90 (17%) | 0.02 |
ABD: abdomen, AV: atrioventricular, BMP: basic metabolic panel, CBC: complete blood count, CT: computed tomography, CXR: chest radiograph, EKG: electrocardiogram, HRS: hours, IV: intravenous, PO: oral; RBC: red blood cell.
Table 3.
Adjusted analysis for clinical process measures for all patients and those discharged with a sepsis diagnosis
| All alerted patients | Discharged with sepsis code* | |||
|---|---|---|---|---|
| Unadjusted odds ratio | Adjusted odds ratio^ | Unadjusted odds ratio | Adjusted odds ratio^ | |
| >= 500cc IV bolus order <3hrs | 1.93 (1.44 – 2.58) | 1.93 (1.43 – 2.61) | 1.64 (1.11 – 2.43) | 1.65 (1.10 – 2.47) |
| IV/PO antibiotic order < 3hrs | 2.02 (1.48 – 2.77) | 2.02 (1.46 – 2.78) | 1.99 (1.32 – 3.00) | 2.02 (1.32 – 3.09) |
| IV/PO sepsis antibiotic order < 3hrs | 1.62 (1.14 – 2.30) | 1.57 (1.10 – 2.25) | 1.63 (1.05 – 2.53) | 1.65 (1.05 – 2.58) |
| Lactic acid order < 3hrs | 2.90 (2.07 – 4.06) | 3.11 (2.19 – 4.41) | 2.41 (1.58 – 3.67) | 2.79 (1.79 – 4.34) |
| Blood culture < 3hrs | 1.72 (1.23 – 2.40) | 1.76 (1.25 – 2.47) | 1.36 (0.87 – 2.10) | 1.40 (0.90 – 2.20) |
| Blood gas order < 6hrs | 1.24 (0.84 – 1.83) | 1.32 (0.89 – 1.97) | 1.06 (0.63 – 1.77) | 1.13 (0.67 – 1.92) |
| BMP or CBC order< 6hrs | 0.95 (0.75 – 1.20) | 0.96 (0.75 – 1.21) | 1.00 (0.70 – 1.44) | 1.04 (0.72 – 1.50) |
| Vasopressor order < 6hrs | 1.36 (0.71 – 2.61) | 1.47 (0.76 – 2.83) | 1.32 (0.58 – 3.04) | 1.38 (0.59 – 3.25) |
| Bronchodilator administration < 6hrs | 0.98 (0.69 – 1.41) | 1.02 (0.70 – 1.47) | 1.13 (0.64 – 1.99) | 1.17 (0.65 – 2.10) |
| Transfusion order < 6hrs | 1.92 (1.21 – 3.04) | 1.95 (1.23 – 3.11) | 1.65 (0.91 – 3.01) | 1.68 (0.91 – 3.10) |
| AV node blocker order < 6hrs | 1.10 (0.68 – 1.78) | 1.20 (0.72 – 2.00) | 0.38 (0.13 – 1.08) | 0.39 (0.12 – 1.20) |
| Loop diuretic order < 6hrs | 0.87 (0.52 – 1.44) | 0.93 (0.56 – 1.57) | 1.63 (0.63 – 4.21) | 1.87 (0.70 – 5.00) |
| CXR < 6hrs | 1.43 (1.06 – 1.94) | 1.47 (1.08 – 1.99) | 1.45 (0.94 – 2.24) | 1.56 (1.00 – 2.43) |
| CT < 6hrs | 1.30 (0.78 – 2.16) | 1.30 (0.78 – 2.19) | 0.97 (0.52 – 1.82) | 0.94 (0.49 – 1.79) |
| Cardiac monitoring < 6hrs | 1.48 (1.06 – 2.08) | 1.54 (1.09 – 2.16) | 1.32 (0.79 – 2.18) | 1.44 (0.86 – 2.41) |
AV: atrioventricular, BMP: basic metabolic panel, CBC: complete blood count, CT: computed tomography, CXR: chest radiograph, EKG: electrocardiogram, IV: intravenous, HRS: hours, PO: oral.
Sepsis definition based on ICD-9 diagnosis at discharge (‘790.7’,’995.94’,’995.92’,’995.90’,’995.91’,’995.93’,’785.52’).
Adjusted for log transformed age, gender, log transformed Charlson Index at admission, admitting service, hospital and admission month.
Odds ratios compare the odds of the outcome after versus before implementation of the early warning system.
Hospital and ICU length of stay (LOS) were similar in the pre and post periods. There was no difference in the proportion of patients transferred to the ICU following the alert; however, the proportion transferred within 6 hours of the alert increased, and the time to ICU transfer was halved (Supplement Figure 4), but neither change was statistically significant in unadjusted analyses. Transfer to ICU within 6 hours became statistically significant after adjustment. All mortality measures were lower in the post period, but none reached statistical significance. Discharge to home and sepsis documentation were both statistically higher in the post period, but discharge to home lost statistical significance after adjustment. (Tables 4 and 5 and Supplement Table 4)
Table 4.
Clinical outcome measures before and after implementation of the early warning and response system.
| Hospitals A-C | ||||
|---|---|---|---|---|
| Pre | Post | p-value | ||
| N alerts | 595 | 545 | ||
| Hospital LOS (days) | Median (IQR) | 10.1 (5.1 – 19.1) | 9.4 (5.2 – 18.9) | 0.92 |
| ICU LOS (days) after alert | Median (IQR) | 3.4 (1.7 – 7.4) | 3.6 (1.9 – 6.8) | 0.72 |
| ICU transfer <6 hrs after alert | 40 (7%) | 53 (10%) | 0.06 | |
| ICU transfer <24 hrs after alert | 71 (12%) | 79 (14%) | 0.20 | |
| ICU transfer any time after alert | 134 (23%) | 124 (23%) | 0.93 | |
| Time (hrs) to first ICU after alert | Median (IQR) | 21.3 (4.4 – 63.9) | 11.0 (2.3 – 58.7) | 0.22 |
| RRT <=6 hrs after alert | 13 (2%) | 9 (2%) | 0.51 | |
| Mortality (of all patients) | 52 (9%) | 41 (8%) | 0.45 | |
| Mortality <= 30 days after alert | 48 (8%) | 33 (6%) | 0.19 | |
| Mortality (of those transferred to ICU) | 40 (30%) | 32 (26%) | 0.47 | |
| Deceased or IP hospice | 94 (16%) | 72 (13%) | 0.22 | |
| Discharge to home | 347 (58%) | 351 (64%) | 0.04 | |
| Disposition location | Home | 347 (58%) | 351 (64%) | 0.25 |
| SNF | 89 (15%) | 65 (12%) | . | |
| Rehab | 24 (4%) | 20 (4%) | . | |
| LTC | 8 (1%) | 9 (2%) | . | |
| Other hospital | 16 (3%) | 6 (1%) | . | |
| Expired | 52 (9%) | 41 (8%) | . | |
| Hospice IP | 42 (7%) | 31 (6%) | . | |
| Hospice other | 11 (2%) | 14 (3%) | . | |
| Other location | 6 (1%) | 8 (1%) | . | |
| Sepsis discharge diagnosis | 230 (39%) | 247 (45%) | 0.02 | |
| Sepsis O/E | 1.37 | 1.06 | 0.18 | |
HRS: hours, ICU: intensive care unit, IP: inpatient, IQR: interquartile range, LOS: length of stay, LTC: long term care, O/E: observed to expected, REHAB: rehabilitation, RRT: rapid response team, SNF: skilled nursing facility.
Table 5.
Adjusted analysis for clinical outcome measures for all patients and those discharged with a sepsis diagnosis.
| All alerted patients | Discharged with sepsis code* | |||
|---|---|---|---|---|
| Unadjusted estimate | Adjusted estimate^ | Unadjusted estimate | Adjusted estimate^ | |
| Hospital LOS (days)a | 1.01 (0.92 – 1.11) | 1.02 (0.93 – 1.12) | 0.99 (0.85 – 1.15) | 1.00 (0.87 – 1.16) |
| ICU transferb | 1.49 (0.97 – 2.29) | 1.65 (1.07 – 2.55) | 1.61 (0.92 – 2.84) | 1.82 (1.02 – 3.25) |
| Time (hrs) to first ICU transfer afterc | 1.17 (0.87 – 1.57) | 1.23 (0.92 – 1.66) | 1.21 (0.83 – 1.75) | 1.31 (0.90 – 1.90) |
| ICU LOS (days)a | 1.01 (0.77 – 1.31) | 0.99 (0.76 – 1.28) | 0.87 (0.62 – 1.21) | 0.88 (0.64 – 1.21) |
| RRT b | 0.75 (0.32 – 1.77) | 0.84 (0.35 – 2.02) | 0.81 (0.29 – 2.27) | 0.82 (0.27 – 2.43) |
| Mortality b | 0.85 (0.55 – 1.30) | 0.98 (0.63 – 1.53) | 0.85 (0.55 – 1.30) | 0.98 (0.63 – 1.53) |
| Mortality within 30 days of alertb | 0.73 (0.46 – 1.16) | 0.87 (0.54 – 1.40) | 0.59 (0.34 – 1.04) | 0.69 (0.38 – 1.26) |
| Mortality or inpatient hospice transferb | 0.82 (0.47 – 1.41) | 0.78 (0.44 – 1.41) | 0.67 (0.36 – 1.25) | 0.65 (0.33 – 1.29) |
| Discharge to homeb | 1.29 (1.02 – 1.64) | 1.18 (0.91 – 1.52) | 1.36 (0.95 – 1.95) | 1.22 (0.81 – 1.84) |
| Sepsis discharge diagnosisb | 1.32 (1.04 – 1.67) | 1.43 (1.10 – 1.85) | NA | NA |
ICU: intensive care unit, LOS: length of stay, NA: not applicable, RRT: rapid response team.
Sepsis definition based on ICD-9 diagnosis at discharge (‘790.7’,’995.94’,’995.92’,’995.90’,’995.91’,’995.93’,’785.52’).
Adjusted for gender, age, Present on Admission Charlson Comorbidity Score, admit service, hospital, and admission month (June, July or August + Sep).
For each outcome, the estimate is identified as:
Coefficient;
Odds Ratio; or
Hazard Ratio.
Estimates compare the mean, odds, or hazard of the outcome after versus before implementation of the early warning system.
In a sub-analysis of EWRS impact on patients documented with sepsis at discharge, unadjusted and adjusted changes in clinical process and outcome measures across the time periods were similar to that of the total population. (Supplement Tables 5 and 6 and Figure 5) The unadjusted composite outcome of mortality or inpatient hospice was statistically lower in the post period, but lost statistical significance after adjustment.
The disposition and mortality outcomes of those not triggering the alert were unchanged across the two periods. (Supplement Tables 7, 8 and 9)
DISCUSSION
This study demonstrates that a predictive tool can accurately identify non-ICU inpatients at increased risk for deterioration and death. In addition, we demonstrated the feasibility of deploying our EHR to screen patients in real time for deterioration and to trigger electronically a timely, robust, multidisciplinary bedside clinical evaluation. Compared to a control (silent) period, the EWRS resulted in a marked increase in early sepsis care, transfer to the ICU, and sepsis documentation, and an indication of a decreased sepsis mortality index and mortality and increased discharge to home, although none of these latter three findings reached statistical significance.
Our study is unique in that it was implemented across a multi-hospital health system, which has identical EHRs, but diverse cultures, populations, staffing, and practice models. In addition, our study includes a pre-implementation population similar to the post-implementation population (in terms of setting, month of admission, and adjustment for potential confounders).
Interestingly, patients identified by the EWRS who were subsequently transferred to an ICU had higher mortality rates (30% and 26% in the pre and post periods respectively across UPHS) than those transferred to an ICU who were not identified by the EWRS (7% and 6% in the pre and post periods respectively across UPHS). (Table 4 and Supplement Table 7) This finding was robust to study period, so is likely not related to the bedside evaluation prompted by the EWRS. It suggests the EWRS could help triage patients for appropriateness of ICU transfer, a particularly valuable role that should be explored further given the typical strains on ICU capacity13, and the mortality resulting from delays in patient transfers into ICUs14, 15.
Although we did not find a statistically significant mortality reduction, our study may have been underpowered to detect this outcome. Our study has other limitations. First, our pre-post design may not fully account for secular changes in sepsis mortality. However, our comparison of similar time periods and our adjustment for observed demographic differences allow us to estimate with more certainty the change in sepsis care and mortality attributable to the intervention. Second, our study did not examine the effect of the EWRS on mortality after hospital discharge, where many such events occur. However, our capture of at least 45 hospital days on all study patients, as well as our inclusion of only those who died or were discharged during our study period, and our assessment of discharge disposition such as hospice increase the chance that mortality reductions directly attributable to the EWRS were captured. Third, although the EWRS changed patient management, we did not assess the appropriateness of management changes. However, the impact of care changes was captured crudely by examining mortality rates and discharge disposition. Fourth, our study was limited to a single academic healthcare system and our experience may not be generalizable to other healthcare systems with different EHRs and staff. However, the integration of our automated alert into a commercial EHR serving a diverse array of patient populations, clinical services, and service models throughout our healthcare system may improve the generalizability of our experience to other settings.
CONCLUSION
By leveraging readily available electronic data, an automated prediction tool identified at risk patients and mobilized care teams resulting in more timely sepsis care, improved sepsis documentation, and a suggestion of reduced mortality. This alert may be scalable to other healthcare systems.
Supplementary Material
Acknowledgments
We would like to thank Jennifer Barger MS, BSN, RN, Patty Baroni MSN, RN, Patrick J Donnelly MS, RN, CCRN, Mika Epps MSN, RN, Allen L Fasnacht MSN, RN, Neil O Fishman MD, Kevin M Fosnocht MD, David F Gaieski MD, Tonya Johnson MSN, RN, CCRN, Craig R Kean MS, Arash Kia MD, MS, Matthew D Mitchell PhD, Stacie Neefe BSN, RN, Nina J Renzi BSN, RN, CCRN, Alexander Roederer, Jean C Romano MSN, RN, NE-BC, Heather Ross BSN, RN, CCRN, William D Schweickert MD, Esme Singer MD, and Kendal Williams MD, MPH for their help in developing, testing and operationalizing the EWRS examined in this study; their assistance in data acquisition; and for advice regarding data analysis. This study was previously presented as an oral abstract at the 2013 American Medical Informatics Association Meeting in Washington, DC.
Footnotes
DISCLOSURES
Dr. Umscheid’s contribution to this project was supported in part by the National Center for Research Resources, Grant UL1RR024134, which is now at the National Center for Advancing Translational Sciences, Grant UL1TR000003. The content of this paper is solely the responsibility of the authors and does not necessarily represent the official views of the NIH. The authors report no potential financial conflicts of interest relevant to this article.
References
- 1.Gaieski DF, Edwards JM, Kallan MJ, Carr BG. Benchmarking the incidence and mortality of severe sepsis in the United States. Crit Care Med. 2013;41(5):1167–1174. doi: 10.1097/CCM.0b013e31827c09f8. [DOI] [PubMed] [Google Scholar]
- 2.Dellinger RP, Levy MM, Rhodes A, et al. Surviving sepsis campaign: international guidelines for management of severe sepsis and septic shock: 2012. Crit Care Med. 2013;41(2):580–637. doi: 10.1097/CCM.0b013e31827e83af. [DOI] [PubMed] [Google Scholar]
- 3.Levy MM, Dellinger RP, Townsend SR, et al. The Surviving Sepsis Campaign: results of an international guideline-based performance improvement program targeting severe sepsis. Crit Care Med. 2010;38(2):367–374. doi: 10.1097/CCM.0b013e3181cb0cdc. [DOI] [PubMed] [Google Scholar]
- 4.Otero RM, Nguyen HB, Huang DT, et al. Early goal-directed therapy in severe sepsis and septic shock revisited: concepts, controversies, and contemporary findings. Chest. 2006;130(5):1579–1595. doi: 10.1378/chest.130.5.1579. [DOI] [PubMed] [Google Scholar]
- 5.Rivers E, Nguyen B, Havstad S, et al. Early goal-directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med. 2001;345(19):1368–1377. doi: 10.1056/NEJMoa010307. [DOI] [PubMed] [Google Scholar]
- 6.Whittaker SA, Mikkelsen ME, Gaieski DF, Koshy S, Kean C, Fuchs BD. Severe sepsis cohorts derived from claims-based strategies appear to be biased toward a more severely ill patient population. Crit Care Med. 2013;41(4):945–953. doi: 10.1097/CCM.0b013e31827466f1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Bailey TC, Chen Y, Mao Y, et al. A trial of a real-time alert for clinical deterioration in patients hospitalized on general medical wards. J Hosp Med. 2013;8(5):236–242. doi: 10.1002/jhm.2009. [DOI] [PubMed] [Google Scholar]
- 8.Jones S, Mullally M, Ingleby S, Buist M, Bailey M, Eddleston JM. Bedside electronic capture of clinical observations and automated clinical alerts to improve compliance with an Early Warning Score protocol. Crit Care Resusc. 2011;13(2):83–88. [PubMed] [Google Scholar]
- 9.Nelson JL, Smith BL, Jared JD, Younger JG. Prospective trial of real-time electronic surveillance to expedite early care of severe sepsis. Ann Emerg Med. 2011;57(5):500–504. doi: 10.1016/j.annemergmed.2010.12.008. [DOI] [PubMed] [Google Scholar]
- 10.Sawyer AM, Deal EN, Labelle AJ, et al. Implementation of a real-time computerized sepsis alert in nonintensive care unit patients. Crit Care Med. 2011;39(3):469–473. doi: 10.1097/CCM.0b013e318205df85. [DOI] [PubMed] [Google Scholar]
- 11.Bone RC, Balk RA, Cerra FB, et al. Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. The ACCP/SCCM Consensus Conference Committee. American College of Chest Physicians/Society of Critical Care Medicine. Chest. 1992;101(6):1644–1655. doi: 10.1378/chest.101.6.1644. [DOI] [PubMed] [Google Scholar]
- 12.Levy MM, Fink MP, Marshall JC, et al. 2001 SCCM/ESICM/ACCP/ATS/SIS International Sepsis Definitions Conference. Crit Care Med. 2003;31(4):1250–1256. doi: 10.1097/01.CCM.0000050454.01978.3B. [DOI] [PubMed] [Google Scholar]
- 13.Sinuff T, Kahnamoui K, Cook DJ, Luce JM, Levy MM. Rationing critical care beds: a systematic review. Crit Care Med. 2004;32(7):1588–1597. doi: 10.1097/01.ccm.0000130175.38521.9f. [DOI] [PubMed] [Google Scholar]
- 14.Bing-Hua YU. Delayed admission to intensive care unit for critically surgical patients is associated with increased mortality. Am J Surg. 2014 doi: 10.1016/j.amjsurg.2013.08.044. [DOI] [PubMed] [Google Scholar]
- 15.Cardoso LT, Grion CM, Matsuo T, et al. Impact of delayed admission to intensive care units on mortality of critically ill patients: a cohort study. Crit Care. 2011;15(1):R28. doi: 10.1186/cc9975. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.