Randomized Trial of Automated, Electronic Monitoring to Facilitate Early Detection of Sepsis in the Intensive Care Unit

Abstract

Objective

To determine whether automated identification with physician notification of the systemic inflammatory response syndrome in medical intensive care unit (MICU) patients expedites early administration of new antibiotics or improvement of other patient outcomes in patients with sepsis.

Design

A prospective, randomized, controlled, single-center study.

Setting

MICU of an academic, tertiary-care medical center.

Patients

442 consecutive patients admitted over a 4 month period who met modified SIRS criteria in a MICU.

Intervention

Patients were randomized to monitoring by an electronic “Listening Application” to detect modified (SIRS) criteria vs. usual care. The Listening Application notified physicians in real-time when modified SIRS criteria were detected, but did not provide management recommendations.

Measurements and Main Results

The median time to new antibiotics was similar between the intervention and usual care groups whether comparing among all patients (6.0h vs 6.1h, p=0.95), patients with sepsis (5.3h vs. 5.1h; p=0.90), patients on antibiotics at enrollment (5.2h vs. 7.0h, p= 0.27), or patients not on antibiotics at enrollment (5.2h vs. 5.1h, p= 0.85). The amount of fluid administered following detection of modified SIRS criteria was similar between groups whether comparing all patients or only patients hypotensive at enrollment. Other clinical outcomes including ICU length of stay, hospital length of stay, and mortality were not shown to be different between patients in the intervention and control groups.

Conclusions

Real-time alerts of modified SIRS criteria to physicians in one tertiary care MICU were feasible and safe but did not influence measured therapeutic interventions for sepsis or significantly alter clinical outcomes.

INTRODUCTION

Each year sepsis affects more than 650,000 individuals in the United States with a cost of approximately $17 billion^1,2. Severely septic patients generally require intensive care unit management, and indeed approximately 20% of all ICU admissions are complicated by infection³. Sepsis is defined as the systemic inflammatory response (SIRS) to infection⁴, but in a recent worldwide survey of physicians, the ACCP definition was not recognized or utilized by the majority of respondents⁵. The lack of consensus among practicing physicians is a barrier to early recognition of sepsis in a complex ICU environment. The time-sensitive nature of diagnosis is emphasized by clinical trials that have demonstrated benefits of early treatment^6–8. However, early diagnosis of sepsis may be challenging in health care settings that require repeated assembly of historical, laboratory, and physiologic data from unconnected information systems. Even after intensive education of health care providers to improve early detection and management of sepsis, rates of broad-spectrum antibiotic administration within 6 hours remained less than 70% in severely septic patients⁹, and of the other 10 recommended treatments, only 2 occurred in more than 50% of patients within 6 hours. Although compliance with guidelines improved with education, the residual deficit suggests the need for additional interventions. Using electronic tools to assist in the diagnosis of complicated illnesses is of interest to many physicians and hospitals and may improve the timeliness, consistency, and reliability of diagnosis. In the critical care setting, the early recognition of ARDS using electronic tools has been pioneered and shown to be accurate¹⁰. We hypothesized that automated physician notification of patients meeting SIRS criteria using an electronic “Listening Application” would facilitate the diagnosis of sepsis, thereby shortening the time to initiation of antibiotics and other sepsis-related therapies.

MATERIALS AND METHODS

Definitions and Terms

Sepsis

Defined as a systemic inflammatory response syndrome secondary to a known or suspected infection. The systemic inflammatory response syndrome (SIRS) is present when two of the following four criteria are satisfied: 1. Temperature > 38 degrees Celsius or <36 degrees Celsius. 2. Heart rate > 90 beats/min. 3. Respiratory Rate > 20 breaths/min or PaCO2 < 32mm Hg. 4. WBC > 12,000 cells/mm3 or < 4,000 cells/mm3, or > 10% immature (band) forms

Modified SIRS Criteria

Within a rolling 24 hour window, two or more of the four SIRS criteria with at least one being either an abnormal temperature or white blood cell count.

Listening Application (L.A.)

The L.A. is an electronic tool designed to monitor patient data in real-time to identify patients who meet modified SIRS criteria. The moment modified SIRS criteria are detected, the L.A. electronically notifies the physician, soliciting an evaluation to determine if the patient has a known or suspected infection. The information flow of the L.A. is shown in Figure 1.

Figure 1. Schematic Representation of the Listening Application.

Upon admission to the MICU, patients were electronically monitored by the L.A. for development of modified SIRS criteria. Upon detection, patients were randomized to either the control or intervention arm. For patients randomized to the intervention arm, the L.A. generated pages and EMR notifications to physicians alerting them to the presence of modified SIRS criteria and requesting an assessment of whether or not the patient was septic.

^a If a patient was determined to be septic, then monitoring was suppressed for 7 days. After 7 days, the patient was again eligible for monitoring and notifications. Patients were only enrolled and analyzed after their first notification.

^b In the case of a patient determined NOT to be septic, monitoring was suppressed for 2 days. However, if either WBC or temperature was normal at the time of the alert and then became abnormal during the 2 days of suppression, then the rules engine allowed an additional alert to be sent to the provider. After 2 days, monitoring and notifications resumed as if the patient was new to the system. Patients were only enrolled and analyzed after their first notification.

Subjects

Inclusion Criteria

All patients under the care of MICU teams at Vanderbilt University Medical Center upon meeting modified SIRS criteria (as defined above) were eligible. Enrollment time was considered to be the first time a patient met modified SIRS criteria while under the care of the MICU team, regardless of whether or not the patient had previously met modified SIRS criteria while outside of the ICU. After discharge from the hospital patients were eligible for re-enrollment during subsequent re-admissions.

Exclusion Criteria

Patients were excluded from our study if they had been previously enrolled during the same hospital admission or if they were being cared for in the MICU on a team other than one of the MICU teams.

Study Design

From May to August of 2009, we conducted a randomized, controlled trial in the medical intensive care unit of Vanderbilt University Hospital. This study was approved by the Institutional Review Board at Vanderbilt University. The need for informed consent was waived. Patients were randomized to computerized monitoring or usual care when modified SIRS criteria were first detected. In the L.A. group, notifications that modified SIRS criteria were met were sent via text messages to the pagers of the primary team physician contact and a flag appeared next to the patient’s name on the physician’s electronic patient list. Pages were sent to whoever was listed as the current primary contact for the admitting team, but notifications in the medical record were available to all physicians taking care of the patient. Physicians were asked to acknowledge receipt of the notification and indicate if current data suggested that the patient had sepsis. If a physician failed to respond to a notification, a reminder was resent after 1 hour. No management recommendations were given by our system and providers were not instructed to treat alerted patients in a different manner than any other patients. If patients were determined to be septic by the physician, further notifications were suppressed for 7 days. If patients were determined not to be septic, then further alerts were withheld for 2 days unless a previously normal WBC count or temperature became abnormal. In the control group, time-date stamps of modified SIRS were generated by the L.A., but notifications were not relayed to any physicians.

Electronic Medical Record and Information Systems

The Biomedical Informatics environment at Vanderbilt University Hospital relies on several locally developed tools. Starpanel is an integrated application that allows clinicians to access a variety of electronic information from one screen, including information from the electronic medical record and physician order entry system. Notifications were available in Starpanel at the time that alerts were sent to pagers. Due to concerns about artifact and other inaccuracies, the L.A. did not monitor “live data from bedside monitors, but instead monitored information from the Starpanel electronic medical record, including confirmed laboratory or nurse-entered data. This clinical information was located in an operational database to allow fast, categorical access to the data by various applications, such as the L.A. The information in the operational database was available for the L.A. within seconds of entry into the medical record or laboratory reporting system. Pager alerts and Starpanel notifications were generated almost immediately.

Data Collection

Patients were followed for 28 days or until hospital discharge, whichever occurred first. All data were collected from the electronic medical record. Medical records were examined to determine general demographics and baseline characteristics on each patient. Route of admission to the intensive care unit was recorded. The primary endpoint was time to administration of a new (i.e first or changed) antibiotic. Secondary sepsis-related therapies, including 6-hour and daily fluid intake/output, lactate measurement, and daily vasopressor administration were also collected. Baseline and daily data were collected on all patients, including demographics, basic metabolic and hematologic labs (if available), documentation of infection by the treating physician, positive cultures, antibiotic use, use of mechanical ventilation, presence of hypotension (defined as MAP of 60 or less OR requiring vasopressors), medical co-morbidities, and APACHE II score. To assess whether physician practices were associated with sepsis-related outcomes, we also collected hypotension-free days, ventilator-free days, lengths of stay (hospital and ICU), route of discharge, and in-hospital mortality. Study physicians used data from the medical record to make a retrospective determination of whether or not a patient was septic at the time of study enrollment.

Statistical Analysis

Ferrer and colleagues⁹ found the mean time to administration of antibiotics in septic patients to be 156 minutes (standard deviation 167 minutes) in an emergency room setting. Using these data, PS Sample size software calculated a need for 120 alert events in each arm (or 240 total) to detect a reduction of 60 minutes for the prompting physicians to administer antibiotics with probability (power) 0.8. The Type I error probability associated with testing of this null hypothesis is 0.05. Preliminary data suggested that 60% of the alerts represent actual septic events and estimated that 400 alerts were needed to identify 240 septic alert events where antibiotics might be initiated or changed.

Data was analyzed with an intention-to-treat approach. Mann-Whitney U tests were used to compare the intervention and control groups for the primary endpoint and other continuous outcomes. χ² tests were used to compare categorical variables between the groups. A Cox proportional hazards model was used to estimate the effect of the intervention on the probability of giving new antibiotics at any time during the study. In this model, 28 days was assigned as the time-to-new-antibiotic in patients who died, were discharged, or did not receive antibiotics within 28 days of follow-up. Cox proportional hazard models were also constructed to estimate the effect of the intervention on the probability of obtaining new blood cultures and measuring lactate at any time during the study. Baseline sepsis, hypotension, APACHE II score (continuous), and whether or not the patient was already receiving antibiotics at enrollment were included in the Cox model as covariates. The control group was the reference group for the Cox model.

RESULTS

A total of 443 patients met modified SIRS criteria while under the care of the MICU teams with 221 patients randomized to the L.A. intervention and 222 patients to the control group (Figure 1). One patient in the intervention group was removed from the analysis because the patient died before their data triggered an alert. The demographic and baseline data of patients included in the analysis is similar between groups and displayed in table 1. Patients averaged 55 years old with a slight male predominance. Slightly less than one third were mechanically ventilated with just under half hypotensive at the time they met modified SIRS criteria. The emergency department represented the most common route of ICU admission.

Table 1.

Baseline Characteristics of Patients Monitored by Listening Application vs. Controls

Characteristics	Intervention (n=220)	Controls (n=222)
Age, mean (SD), y	55 (±18)	54 (±18)
Male, No. (%)	125 (57%)	118 (53%)
Race, No. (%)
African-American	43 (20%)	51 (23%)
Asian	0	2 (1%)
Hispanic	0	9 (4%)
White	162 (74%)	150 (68%)
Other	15 (6%)	9 (4%)
APACHE II score, median (IQR)	18.0 (11.8 – 24.0)	16.0 (11.2 – 22.0)
Mechanically Ventilated at Enrollment, No. (%)	61 (28%)	66 (30%)
Hypotension at Enrollment	95 (48%)	91 (44%)
Route of Admission to MICU, No. (%)
Emergency Dept.	90 (41%)	103 (46%)
Transfer from Hospital	69 (31%)	67 (30%)
In-hospital transfer	59 (26%)	48 (21%)
Direct admission	1 (0%)	1 (0%)
Post-procedure	1 (0%)	3 (1%)

APACHE II = Acute Physiology and Chronic Health Evaluation II Score; MICU = Medical Intensive Care Unit

Hypotension defined as MAP < 60 or vasopressor use at enrollment

Physicians entered a response to confirm receipt of the alert and record a sepsis assessment in 185/220 (84%) of initial text alerts. The median time from detection of modified SIRS by the L.A. to an assessment by a physician was 0.9 [IQR 0.18 – 3.47] hours.

New (i.e. first or changed) antibiotics were administered after enrollment for 62% of patients in the intervention group and 60% of patients in the control group. Among patients who received a new antibiotic, the median time from enrollment to administration in the intervention group was 6.0 [IQR 2.4–18.8] hours compared to 6.1 [IQR 2.5 –21.0] hours in the control group (p=0.95) (Table 2). Patients in the intervention group received new antibiotics 0.1 hours earlier than controls (95% CI for the difference: −3.8 to 2.2h). Of the 231 (52%) patients not on antibiotics at the time of enrollment, 131 (57%) were subsequently administered antibiotics at a median time of 5.2 hours [IQR 2.1 – 13.0] in the intervention group (n=66) and 5.1 hours [IQR 1.5 – 17.0] in the control group (n=65) (p=0.93).

Table 2.

Outcomes of Patients by Group

Outcome Variable	Intervention	Controls	p-value
Time to 1st New Antibiotic, median (IQR), hours	6.0 (2.4–18.8)	6.1 (2.5–21.0)	p=0.95
6-hour Fluid Administration, mean, mL	1019±1241	964±1196	p=0.57
Hypotensive at Enrollment	1589±1874	1479±1600	p=0.92
ICU LOS, median (IQR), days	3.0 (2.0–5.0)	3.0 (2.0–4.0)	p=0.22
Hospital LOS, days	5.7 (2.8–10.5)	4.7 (2.7–8.1)	p=0.08
Mortality, %	14%	10%	p=0.29

At the time of enrollment, 43% of patients in the intervention group and 39% of patients in the control group were septic (retrospectively determined by chart review). Among the 61 septic patients not on antibiotics at enrollment, the median time to antibiotics did not differ between groups with 3.4 hours [IQR 1.7 – 12.3] in the intervention group (n=28) vs. 3.5 hours [IQR 1.2 – 13.8] in the control group (n=33), (p=0.92).

After adjusting for baseline sepsis, hypotension, and APACHE II score, as well as whether the patient was already on antibiotics at enrollment, the intervention did not change the probability of receiving a new antibiotic at any given time during the study period (HR 1.00; 95% CI 0.78 to 1.27). After similar adjustments, the probability of new blood culture orders (HR 1.01, 95% CI 0.76 to 1.35) or lactate measurements (HR 0.84, 95% CI 0.54 to 1.30) were similar between groups.

The administration of fluids in the six hours following enrollment was similar among groups whether considering all patients (1019 mL in the intervention vs. 964 mL in the control group, p=0.57) or only patients with hypotension at enrollment (1589 mL in the intervention vs. 1479 mL in the control group, p=0.92) (Table 2). Only 2% of patients enrolled in our study received activated protein c. The rate of activated protein c use and the time to administration was similar between groups.

The intervention group and control group did not differ in ICU length of stay (3.0 vs. 3.0 days; p= 0.22), hospital length of stay (5.7 vs. 4.7 days; p= 0.08), or in-hospital mortality (14% vs. 10%, p=0.29) (Table 2).

Of the 442 patients enrolled in the trial (and thus met modified SIRS criteria at some point during their MICU stay), 180 were retrospectively determined to have been septic when modified SIRS criteria were first detected. 262 were retrospectively determined to have not been septic at the time modified SIRS criteria were first detected. Sixty of the 560 patients admitted to the MICU during the study period did not meet modified SIRS criteria at any point during their stay. Two of these patients were determined to have been septic during their MICU stay. If applied to all MICU patients in our study, the sensitivity of our system for detecting sepsis was 99% and the specificity was 82%. The positive predictive value was 41% and the negative predictive value was 97%.

DISCUSSION

This randomized, controlled trial of a Listening Application to alert physicians when patients develop modified SIRS criteria did not demonstrate a significant difference in the primary outcome of time to administration of new antibiotics. The lack of effect in our trial may be due to a combination of factors. The majority of patients enrolled in our trial had received some resuscitative and/or sepsis-related care prior to MICU admission. New antibiotic orders or other sepsis-related therapies may not have been indicated at the time an alert was generated. It is also possible that monitoring by the Listening Application may not be sufficient to alter physician practices.

Use of alerting systems and rapid response teams have been proposed as interventions to improve the time to treatment and outcomes in patients with severe sepsis¹¹. The implementation of such systems is often complicated and requires significant resources and training of medical staff¹². In addition, these alerting mechanisms are usually implemented for patients being cared for outside of the highly monitored ICU environment where data are sparse and monitoring is less intense. Protocols that lead to activation of rapid response teams for deteriorating patients use criteria that can be found in most medical records. As medical records transition to electronic entry and storage, the information within is becoming available for automated pattern recognition and analysis. The rapid adoption of electronic medical records has not been mirrored by a proliferation of tools that assimilate complicated information and assist physicians in the recognition of developing illness. Our study represents one of the first uses of electronic monitoring of the medical record for the detection of SIRS. Although the alerting system was feasible and physicians responded to 84% of the alerts, the system did not significantly alter clinical practice as measured by time to antibiotics, fluid administration, and other sepsis-related therapies.

Our MICU is a closed environment, staffed 24 hours a day by physicians. Although this structure provided a logistically simple environment to deploy our Listening Application, the index of suspicion for sepsis in the ICU is high and intensive monitoring of patients is often in place. Both ICU nurses and physicians are experienced in the early recognition and management of septic patients. The high rate of antibiotic administration prior to enrollment in our study suggests that infection had already been suspected with treatment initiated in many patients. Thus, similar to an electronic monitoring study in the emergency department, the biggest shortcoming of the L.A. may have been the failure to identify patients with modified SIRS before the treating physician¹³. Modified SIRS criteria is almost entirely based on physiologic data available at the bedside. The L.A. could not access physiologic data until it was recorded in the medical record by the bedside nurse. It is possible that delays in data entry, particularly in sicker patients, led to delays in alerting by the L.A. The relatively low volume of intravenous fluid administration in the 6 hours after the alert further supports the notion that many patients were undergoing treatment for sepsis at the time of the alert. In fact, the alert occurred within 2 hours of admission in 50% of patients suggesting the patient met the criteria for sepsis prior to even being admitted to the ICU. This may help to explain the difference in our results from a recent pilot study by Sawyer et al¹⁴ which showed that implementation of real-time sepsis alerts on non-ICU patients can influence therapies.

It is possible that a lower severity of illness may have influenced the observed results in our trial. The mean APACHEII score of 17 in patients enrolled in our trial is lower than many published trials that implemented sepsis bundles ^6,7,15,16. Mortality was also lower, likely due to a lower severity of illness as well as our inclusion of septic patients without organ dysfunction (i.e. non-severe sepsis) and non-septic patients with SIRS. Early recognition of sepsis may be less likely to affect physician actions and patient outcomes in patients with a lower severity of illness, however, the Cox Proportional Hazards model did not detect a difference in physician actions when adjusted for baseline sepsis, hypotension, or APACHEII score.

The high exposure to antibiotics prior to enrollment (48%) and lower than expected rates of sepsis (41%) suggest that new antibiotics may not have been indicated in many patients at the time of enrollment in this study, diminishing the power to detect a difference in the primary endpoint. However, analysis of the 61 septic patients not on antibiotics at enrollment also did not show any difference in time to administration of new antibiotics.

While our study does not demonstrate significant changes in physician behavior or patient outcomes, it was not paired with any specific educational materials or recommendations for treating septic patients. It is possible that the L.A. helped physicians recognize septic patients earlier, but did not influence their care. Pairing the L.A. with protocols or decision support systems may help physicians or other providers comply with recommended treatments more effectively than notification alone.

Previous studies demonstrate that decision support in the form of protocols and bundles significantly impacts the care of septic patients^7,14,16,17. The lack of effect from the L.A. does not prove that an electronic alerting system is not valuable, but suggests that the criteria for alerts and the patients under surveillance must be carefully considered when such a system is implemented. A similar system utilized in an emergency department increased the number of patients who had blood cultures¹³. It is reasonable to hypothesize that the use of the L.A. in an environment with low rates of SIRS at admission, less clinical suspicion for sepsis, or less intense monitoring, such as non-medical ICUs or non-ICU patient care areas may have resulted in substantially different findings, like those demonstrated by Nelson et al¹³ and Sawyer and colleagues¹⁴. Similarly, we propose that refining systems similar to our L.A. and pairing them with therapeutic decision support could still significantly improve adherence to and expedite initiation of recommended therapies with improved outcomes in patients with sepsis. While not in septic or medical patients, a study of an open-loop computer decision support tool decreased the amount of crystalloid resuscitation in patients with severe burns while increasing the time patients had adequate urine output¹⁸.

We also propose that as more complicated diagnostic algorithms for sepsis emerge that tools such as the L.A. may have increased benefit. The criteria in our algorithm were based on a modified version of the 1992 ACCP/SCCM Consensus Conference⁴. While definitions of sepsis have not radically changed, the 2001 SCCM/ESICM/ACCP/ATS/SIS International Sepsis Definitions Conference¹⁹ identifies at least 25 commonly available clinical criteria suggestive of sepsis as well as numerous investigational biochemical assays which could assist in diagnosing septic patients.

Our study does not show improved compliance to sepsis-related treatments or outcomes, but also does not show any adverse effects. Changes to the alerting criteria, methods for notification, or the patient population under surveillance could produce different results. The similar administration of treatments and similar outcomes in the intervention group suggests that quality improvement or research using similar methods can safely be pursued.

The high specificity, sensitivity, and negative predictive values of the L.A. is promising, but is at least partially a reflection of the high prevalence of sepsis in the MICU. Furthermore, the time between onset of sepsis and detection of modified SIRS criteria by the L.A. is not known.

Conclusion

An electronic monitoring system which alerts ICU physicians when their patients meet modified SIRS criteria was not sufficient by itself to reduce the time to new antibiotic administration, increase the rate of blood cultures, or increase the rate of serum lactic acid measurements. The effect of pairing such an application with computerized physician support suggesting diagnostic and treatment plans should be studied. Further research exploring changes to the criteria for alerting, methods for notification, or population under surveillance should be pursued.