Electronic health record-based clinical decision support alert for severe sepsis: a randomised evaluation

Abstract

Background

Sepsis remains the top cause of morbidity and mortality of hospitalised patients despite concerted efforts. Clinical decision support for sepsis has shown mixed results reflecting heterogeneous populations, methodologies and interventions.

Objectives

To determine whether the addition of a real-time electronic health record (EHR)-based clinical decision support alert improves adherence to treatment guidelines and clinical outcomes in hospitalised patients with suspected severe sepsis.

Design

Patient-level randomisation, single blinded.

Setting

Medical and surgical inpatient units of an academic, tertiary care medical centre.

Patients

1123 adults over the age of 18 admitted to inpatient wards (intensive care units (ICU) excluded) at an academic teaching hospital between November 2014 and March 2015.

Interventions

Patients were randomised to either usual care or the addition of an EHR-generated alert in response to a set of modified severe sepsis criteria that included vital signs, laboratory values and physician orders.

Measurements and main results

There was no significant difference between the intervention and control groups in primary outcome of the percentage of patients with new antibiotic orders at 3 hours after the alert (35% vs 37%, p=0.53). There was no difference in secondary outcomes of in-hospital mortality at 30 days, length of stay greater than 72 hours, rate of transfer to ICU within 48 hours of alert, or proportion of patients receiving at least 30 mL/kg of intravenous fluids.

Conclusions

An EHR-based severe sepsis alert did not result in a statistically significant improvement in several sepsis treatment performance measures.

Background

Sepsis affects an estimated 650 000–750 000 Americans annually, costing the American healthcare system an estimated $20 billion.^1–3 Early recognition, antibiotics and intravenous fluids are the cornerstones of sepsis management. The Surviving Sepsis Campaign recommends administration of antibiotics within 1 hour, citing evidence that even short delays in antibiotic administration lead to increases in mortality. However, despite this recommendation, delays in treatment still often occur.⁴

Given the difficulties in the timely recognition of sepsis, healthcare institutions are increasingly leveraging clinical data captured in electronic health records (EHR) and clinical decision support (CDS) systems that incorporate patient-specific context such as vital signs and laboratory values to alert clinicians to the possible presence of sepsis and other clinical deteriorations.⁵ Despite a growing body of research, the impact of electronic sepsis alerts remains unclear. Alerts have been developed for emergency departments, intensive care units (ICU), and medical wards, and in a range of software platforms: home-grown EHRs, stand-alone software programs and commercial EHRs. However, published studies have reported a wide range of findings: ranging from no impact on bundle compliance or outcome measures, to impact on process measures like antibiotic administration, and one pre-post study that showed a very large effect on mortality.^6–14 Inconsistent findings may reflect methodological weaknesses in the existing literature with only a few studies employing rigorous methodologies such as randomisation or reflect implementation differences such as alert algorithms and associated downstream care.

Most sepsis alert studies have incorporated either home-grown EHRs or other custom software. While this approach can increase flexibility, it creates significant barriers to dissemination and generalisability–most medical centres cannot invest in building, buying or supporting stand-alone alerting systems. Since few studies have limited alert design to the constraints of a standard EHR functionality, implications for broader dissemination, replication and long-term sustainability have been limited.

In addition to information technology infrastructure, the clinical action downstream of the alert is critical to positive clinical impact. This entails recognition of the alert, acceptance of the recommendation and correct action by the clinician. Few studies have described the protocolised care, change management and follow-up assessment methodology associated with and necessary for an alert to positively effect clinical care.

To address these gaps, we studied the implementation of a sepsis alert for hospitalised patients with the following approach: (1) patient-level randomisation to control for confounding; (2) using only the EHR functionality to deliver real-time CDS alerts; and (3) a change management strategy that incorporates user-centred design via clinician feedback and standardised clinical workflow.

Methods

Definitions and terms

Severe Sepsis Alert Criteria: The co-occurrence of one or more criteria of suspected infection, one or more criteria of organ dysfunction and three or more criteria of systemic inflammatory response syndrome (online supplementary file 1). Definition was based on the most current Surviving Sepsis Campaign guidelines published at the time of development.

EHR-based alert: This EHR allows customised alerts via pop-up screen or text-page based on specified criteria. A previously developed alert was modified to increase positive predictive value (PPV) to above 60% to decrease alert fatigue which is comparable to most previously published alerts. This standard functionality was used to implement the Severe Sepsis Alert Criteria into an alert that used criteria collected as a part of usual patient care (vital signs, laboratory values, physician orders). The antibiotics (anti-infectives) included in this definition were determined by the hospital quality committee (online supplementary file 2). The alert was delivered via text-page by the paging system directly to prespecified users rather than a pop-up window within the EHR to ensure users received the alert even when not logged into the EHR.

Alert recipients and workflow: Severe sepsis alerts were sent via text-page to a pool of intensive care-trained nurses (crisis nurse) whose dedicated role is to circulate throughout the hospital and assist providers with patients with deteriorating clinical status. The alert was also delivered to the medicine primary treatment teams if the patient was under their care. These roles were decided by stakeholders in nursing, quality improvement and the medical staff.

Design

Screening and enrolment

From December 2014 to July 2016, we conducted a pragmatic, randomised, single-blind evaluation in the hospital medical and surgical wards and step-down units. This protocol was reviewed by the university institutional review board and determined to be a quality improvement project so consent requirement was waived. This project was registered to ClinicalTrials.gov (NCT02376842). Inclusion criteria: age 18 years or older, admitted to the hospital. Exclusion criteria included patients admitted to the ICU, patient code status is comfort care only. Because we believe that even patients on antibiotics with ongoing signs of sepsis may benefit from a consideration of change in therapy, patients on antibiotics prior to the alert were included.

Randomisation and blinding

Each inpatient encounter was randomised 1:1 to either control group (silent alert visible retrospectively to study staff) or intervention group (live alert transmitted to clinical staff). Assignment to control or intervention group was not known by patients but, in the case of the intervention group, known to the alert recipients (figure 1).

Figure 1.

Enrolment and randomisation.

Intervention and change management

For patients assigned to the intervention group, when the alert criteria were met, the alert automatically generated a text-page sent via the paging system to the crisis nurse and the primary treatment team physician for medical teams. Surgical and other services opted out of receiving the alert directly so alerts generated for their patients were sent to the crisis nurse only.

The text-page displays the patient’s bed location and the most recent set of vital signs. Before and during implementation of this alert, nursing and physician leadership developed and codified standardised clinical workflow that should occur downstream of the alert. Recipients are instructed to go to the patient bedside to assess and discuss next steps including a clinical assessment of whether or not severe sepsis is present. The physician places appropriate orders and the crisis registered nurse (RN) mobilises other necessary clinical resources and documents the clinical diagnosis and actions taken. The crisis RN also has protocol orders which enable them to order blood cultures and repeat lactate if the primary team is not available. After firing, the alert is silenced for 48 hours. The implementing team educated care teams and the crisis nurses about the alert, the expected response and the rationale for implementation. Educational efforts included how to recognise severe sepsis, treatment guidelines and the associated order set, including early antibiotic administration and fluid resuscitation recommendations.

Ad hoc feedback was obtained from end users after implementation to iterate on educational efforts and to improve buy-in. We also formally surveyed end users to understand attitudes towards order sets and the impact of educational efforts.

Physician surveys

A member of the research team reviewed 98 alerts during a random 4-week period in March 2015, of which 63 alerts (64%) were deemed to be a true positive. The associated clinicians were then contacted by phone or email to understand why treatment decisions often differed from institution recommendations. Since antibiotic compliance was high, the survey focused on fluid resuscitation and the use of an order set designed to improve fluid resuscitation. Forty-two clinicians responded to the survey, answering whether they used the EHR sepsis order set, and what, if any, concerned them about 30 mL/kg intravenous fluid resuscitation.

Data collection

Patients were followed until hospital discharge. Demographic and baseline characteristics, vital signs, laboratory results, orders, medication administration, ICU transfer and clinical outcomes were collected within the EHR as part of routine clinical care and extracted for analysis with assistance from an institutional clinical data warehouse (Stanford Translational Research Integrated Database Environment).¹⁵ These data were abstracted into a database by blinded personnel. The primary endpoint was the percentage of patients who had a new order for antibiotics within 3 hours of the alert firing (see online supplementary file 2 for definition of antibiotics). Post hoc secondary endpoints included: blood cultures ordered, serum lactate ordered, intravenous fluids, at least 30 mL/kg of intravenous fluids, length of stay (LOS) greater than 72 hours, rate of transfer to ICU and inpatient mortality.

Statistical analysis

Based on prior studies, we predicted that the alert might lead to a 10% increase in antibiotic administration.⁹ To be conservative, we assumed that administration would increase only among true positives. Given an initially estimated true positive rate of 40%, we therefore expected a 4% overall change. With a significant threshold of 0.05, planned enrolment was 3000 encounters for both groups. However, randomisation was terminated early by the hospital quality committee to expand the alert to all eligible patients—only 1123 encounters were randomised before early termination.

Continuous outcomes were evaluated using a two-sided t-test. Categorical outcomes were assessed with a Pearson’s χ². A significance threshold of 0.05 was used for all comparisons. Bonferroni correction was employed for secondary analyses. All analyses were performed using R statistical software V.3.1.1, with epitools V.0.5–7 and forestplot V.1.1 packages (RStudio Team 2015). All analyses were performed at the hospitalisation level.

Results

Enrolment and baseline characteristics

Before termination of randomisation, 1123 patients met the alert criteria and were randomised. The 528 patients to the silent alert and the 595 patients randomised to the active alert had similar baseline characteristics and prealert vital signs and relevant laboratory values. There were no significant differences between groups in baseline characteristics (table 1). Patients averaged 63 years old, were equally male or female, largely non-Hispanic and White and largely cared for by the internal medicine teaching service. Approximately 20% had a diagnosis of heart failure (International Classification of Diseases, Ninth Revision, ICD-9 428.0) and 26% sepsis, respectively (ICD-9 038.0, 038.12, 038.2, 038.3, 038.40, 038.42, 038.43, 038.49, 038.8, 038.9, 003.1).

Table 1.

Baseline characteristics by treatment group

Demographics	Control	Intervention
Gender
Female, n (%)	268 (51)	304 (51)
Age (mean, SD)	63 (0%)±19.1	63 (0%)±18.6
Ethnicity, n (%)
Hispanic/Latino	104 (20)	98 (16)
Non-Hispanic/non-Latino	420 (80)	493 (83)
Unknown	4 (1)	3 (1)
Race, n (%)
White	273 (52)	322 (54)
Asian	78 (15)	86 (14)
Black or African-American	36 (7)	36 (6)
American-Indian or Alaska Native	3 (1)	1 (0)
Native Hawaiian or Other Pacific Islander	6 (1)	1 (0)
Other	123 (23)	127 (21)
Clinical characteristics
Heart rate (mean, SD)	94.05±17.6	93.76±17.7
Mean arterial blood pressure (mean, SD)	109.5±15.9	110.6±16.6
Most recent temp > 38.3° C (n, %)	115 (21.7)	135 (22.6)
Most recent white cell count >12 x10∧9/μL (n, %)	138 (26.1)	157 (25.5)
Most recent lactate, mmol/L (mean, SD)	2.31±1.67	1.96±1.18
Diagnoses at discharge, n (%)
Heart failure (ICD-9 code XXX)	109 (20.5)	115 (19.2)
Sepsis (ICD-9 code YYY)	146 (27.5)	150 (25.1)
Clinical service, n (%)
Bone marrow transplant	29 (5)	32 (5)
Cardiac surgery	15 (3)	12 (2)
Cardiology	19 (4)	17 (3)
Cardiology transplant	5 (1)	6 (1)
Critical care	14 (3)	13 (2)
Cystic fibrosis adult	6 (1)	9 (2)
General medicine (private)	63 (12)	71 (12)
General medicine (teaching)	185 (35)	198 (33)
General surgery	8 (2)	23 (4)
Gynaecologic oncology	7 (1)	7 (1)
Haematology	32 (6)	35 (6)
Haematology/oncology	21 (4)	29 (5)
Hepatology	4 (1)	7 (1)
Liver transplant	5 (1)	4 (1)
Medicine	22 (4)	21 (4)
Neurology	4 (1)	4 (1)
Oncology	51 (10)	62 (10)
Pulmonary hypertension	7 (1)	7 (1)
Pulmonary transplant	4 (1)	4 (1)
Trauma	3 (1)	3 (1)

ICD-9, International Classification of Diseases, Ninth Revision.

Primary and secondary outcomes

There was no significant difference between intervention and control groups in the percentage of patients with new antibiotic orders within 3 hours of the alert (35.0% vs 36.7%, p=0.53), though a large proportion were already on broad spectrum antibiotics at the time of the alert (66.0% vs 66.3%, p=0.93). There was no difference in secondary outcomes of the percentage of patients who received any intravenous fluids (23.70%% vs 19.89%, p=0.86), lactate ordered (14.12% vs 10.8%, p=0.75), at least 30 mL/kg intravenous fluids (1.5% vs 1.5%, p=1.0), in-hospital mortality at 30 days, (6.2% vs 6.8%, p=1.0), LOS greater than 72 hours (3.2 vs 3.2, p=1.0) or rate of transfer to ICU within 48 hours of alert (6.7% vs 5.7%, p=1.0) (figure 2).

Figure 2.

Primary and secondary outcomes. ICU, intensive care unit.

Survey results

Of the 63 clinician contacts, 42 completed the survey (67%), which was administered either in person or over the phone. Most were first year, male resident physicians (table 2). Twenty-six of the 42 surveys that were completed related to episodes that were chart validated as true positive for severe sepsis (62%). Only two out of the 26 physicians (8%) used the EHR sepsis order set. Physicians reported a variety of reasons, most commonly reporting that they preferred to enter sepsis orders individually (69%) and that the alert did not capture a true change in clinical status (58%). Only two out of the 26 physicians administered the recommended 30 mL/kg of intravenous fluids in cases of chart review confirmed sepsis hypoperfusion (hypotension or hyperlactaemia). The physicians reported that the primary reasons they did not do so were that they were worried about fluid overload (63%) and that the patient was not sufficiently hypotensive to warrant that degree of intravenous fluids (54%) though several also reported disagreement of the diagnosis of sepsis (17%) or were not aware of fluid resuscitation guidelines (13%).

Table 2.

Demographic characteristics and physician attitudes of sepsis order set and intravenous fluid recommendations

Characteristics	n (%)
Gender
Female	15 (33)
Clinical service
General medicine (teaching)	19 (45)
Haematology	8 (19)
Cardiology	5 (12)
Oncology	3 (7)
General medicine (private)	2 (5)
Other	6 (14)
Which of the following are the reasons why you did not use the EPIC sepsis order set? (n=26 providers)
Preferred to place sepsis orders separately	18 (69)
Sepsis alert did not capture a meaningful change in clinical condition	15 (58)
Unaware of sepsis alert order set	7 (27)
Inappropriate to use sepsis alert order set (eg, patient does not have sepsis)	3 (12)
Which of the following are the reasons why you did not give 30 mL/kg intravenous fluids in response to the alert? (n=26 providers)
Provider worried about fluid overload	15 (63)
Provider did not feel patient was hypotensive (eg, the blood pressure measured was not sufficiently different from their baseline)	13 (54)
Provider did not feel that the patient had sepsis	4 (17)
Provider unaware of the 30 mL/kg intravenous fluid guidelines	3 (13)

Discussion

This pragmatic, randomised evaluation of an EHR-based sepsis CDS alert in an inpatient population demonstrated no significant improvement in either the primary endpoint or any secondary endpoint. While some prior studies have shown an impact of sepsis alerts on antibiotics and other therapeutic endpoints, many of these observational studies have weak designs.⁵ More robust evaluations have yielded inconsistent results. One randomised evaluation of a modified systemic inflammatory response alert in an ICU found no impact on therapeutic endpoints, while a ward-level comparison of an alert to usual care showed an increase in antibiotic administration.^{8 9} Given the multifactorial nature of sepsis care, pre-post comparisons are subject to significant confounding and these results should be interpreted with caution.^{12 16 17} Indeed, in the two randomised trials done in the same academic centre, no difference was seen in outcomes.^{7 8}

Our study offers a new contribution to the small body of randomised evaluations of sepsis alerts. It is the first patient-level randomised evaluation of severe sepsis alert, one targeting patients with organ dysfunction from sepsis. Our alert is therefore more consistent than prior studies with the trend to conceptualise sepsis as organ dysfunction caused by host dysregulation.¹⁸ In addition, most prior studies leveraged expensive, customised software that would be difficult to replicate—akin to using a specialised medication or procedure unavailable to most of the world. Our alert uses only off-the-shelf functionality from a common EHR platform, making it potentially generalisable to many health systems. Finally, we invested considerable time and resources to develop a protocolised response to the alert and train staff appropriately, believing that this investment would increase the likelihood that the alert would impact care. However, despite the protocolised care, we did not observe an effect of the alert.

There are multiple possible reasons that we did not observe a significant effect of the alert. First, baseline levels of compliance were high with approximately 66% of patients already receiving antibiotics at the time of alert (online supplementary file 3). Our institution had previously taken steps to educate staff and promote awareness of sepsis management. These steps led to a high baseline compliance, leading to a lower marginal return on improved detection through an alert. Second, survey data revealed that clinicians had already diagnosed severe sepsis and initiated treatment in a significant portion of true positive cases—the alert lagged behind their clinical judgement. Third, some of the non-responses may be attributable to false positives. We determined by chart review that the alert’s PPV was 61%. This PPV is equal or superior to that of most prior studies: one review found alerts to have PPVs ranging from 20.5% to 53.8%.⁵ Nevertheless, nearly 40% of alerts were false positives, and clinicians may have reacted appropriately to these by not initiating treatment. Although we calibrated the alert to a high PPV, we were limited by the detection parameters available from the EHR and the baseline prevalence of severe sepsis among non-ICU patients. Lastly, randomisation was terminated early by the institution’s quality improvement leadership, outside of the authors’ control, so there was insufficient power to detect differences between groups. Early termination occurred because of institutional priorities and reflects the realities of conducting a randomised evaluation in the context of a quality improvement programme. Another notable finding was that the in-hospital mortality rate was 6%, lower than many previously published reports of sepsis mortality.^{2 3} There are several possible reasons the mortality in this study was lower. First, our study excludes ICU patients, whereas many prior studies included them. Second, changes in identification, treatment and coding of severe sepsis make it difficult to establish a clear reference point for severe sepsis. Past estimates including ICU patients using severe sepsis diagnostic codes may obtain a higher rate. In addition, the alert’s PPV was approximately 60%, and not all subjects had sepsis.

Results of a clinician survey suggest some reasons that the alert did not impact care. In a survey of clinicians’ attitudes towards the alert, respondents’ concern for fluid overload appears to have greatly limited adherence to the recommended amount of intravenous fluids though only 20% of patients carried a diagnosis of heart failure. Though the recommended volume of intravenous fluid is not always indicated, the low achievement in this area represents a potential opportunity to change attitudes towards fluid resuscitation. It highlights the importance of change management and local culture in achieving change in clinician behaviour. A significant barrier in improved care remained clinician adherence to treatment recommendations—a common theme identified in CDS.^19–22 Though there are advantages to providing CDS within the EHR at the point of order entry, we opted for text-paging to ensure prompt delivery. This, and the inability to provide an explicit recommendation via the text message due to length constraints, may have decreased the alert’s impact as well.

Future research should explore how to improve the effectiveness of sepsis alerts. Improved accuracy, and in particular PPV, is greatly needed as low specificity alert contributes to alert fatigue which further reduces the potential efficacy.²³ Even if accuracy is improved, achieving clinical outcomes depends on the subsequent clinical actions. Implementation science research is needed to understand why alerts alone do not change provider behaviour. Why do providers still fail to always meet established process outcomes, such as administration of antibiotics and intravenous fluids, even when the alert is accompanied by an educational campaign? What could be done to redesign alerts to be more responsive to provider needs? Human–computer interaction research could contribute to the ergonomics of alerts, finding the best channels and mechanisms to deliver them. Machine learning techniques offer potential to produce more accurate and predictive alerts which in turn may reduce alert fatigue and engender trust by clinicians. However, routine use of more advanced models may not permit explanations for predictions and will be limited by the capabilities of commercial EHR systems until these platforms support more complex algorithms in real time. Collaboration between vendors and researchers will be needed to ensure that advances in sepsis detection have a practical path to implementation and additional randomised evaluations will be critical to evaluating the ultimate clinical impact.

Conclusions

Although it is feasible for a health system to develop and randomise the implementation of a severe sepsis alert using only functionality from a commercial EHR, the effect on sepsis care outcomes is unclear. Results of observational studies of sepsis alerts likely overestimate the impact of alerts, due in part to increased identification and documentation of sepsis. More research is needed to understand how to design sepsis alerts to change provider behaviour.