Development and implementation of sepsis alert systems

Synopsis/Summary

Development and implementation of sepsis alert systems is challenging, particularly outside the monitored intensive care unit (ICU) setting. Important barriers to wider use of sepsis alerts include evolving clinical definitions of sepsis, information overload & alert fatigue, due to suboptimal alert performance. Outside the ICU, additional barriers include differences in health care delivery models, charting behaviors, and availability of electronic data. Currently available evidence does not support routine use of sepsis alert systems in clinical practice. However, continuous improvement in both the afferent (data availability and accuracy of detection algorithms) and efferent (evidence-based decision support and smoother integration into clinical workflow) limbs of sepsis alert systems will help translate theoretical advantages into measurable patient benefit.

Introduction

Development and implementation of sepsis alert systems has occurred primarily in acute care settings, such as the intensive care unit (ICU) and emergency department (ED).¹ The development and implementation of these systems outside the acute care setting (ICU and ED) is limited for a variety of reasons. As a critical care syndrome,^{2, 3} the pathogenesis of sepsis has been studied. Thus, the basic pathophysiology of sepsis is best understood primarily in this context.^{4, 5} Sophisticated technologies and large quantities of data present in the acute care setting, combined with relatively short lengths of stay and clear outcomes (such as mortality), provide a natural environment for clinical informatics research in general.⁶ However, sepsis is not limited to the ICU setting. As a result of advances in the technology and data granularity underlying clinical informatics systems, it is now possible to consider the development and implementation of sepsis alert systems within and outside the ICU.

The reason for considering electronic sepsis surveillance is ultimately to facilitate timely and error-free treatment through early recognition and decision support. However, multiple barriers prevent the development and implementation of hospital-wide sepsis alert systems. These barriers and potential solutions to these barriers are explored. A vision of alert systems of the future is also presented.

Development and Implementation of Sepsis Alert Systems

Early sepsis alert systems were developed primarily for clinical trial enrollment purposes. In 2003, Thompson and colleagues published a sepsis alert and diagnostic system for integrating clinical systems to enhance study coordinator efficiency.⁷ In 2005, Embi and colleagues published the effects of a clinical trial alert system on physician participation in clinical trial recruitment.⁸ In 2008, Herasevich and colleagues published a computer based screening engine for severe sepsis and septic shock,⁹ which was subsequently used to enroll patients in the critical care setting into a time sensitive clinical study.¹⁰ The development of these early alert systems generated considerable interest in how to best use electronic data to find and treat critically ill patients,¹¹ as well as lay the foundation for the implementation of sepsis alert systems in the ICU setting (Table 1).^12-15

Table 1. Sepsis alert system studies in the acute care setting.

Reference	Stage	Subjects	N	Design	Setting	Illness	Primary outcome	Result
2003 Thompson [7]	Early development	Patients	203	Retrospective & Prospective	Medical & Surgical ICU	Sepsis	Time (study coordinator screening)	Study coordinator screening time was reduced
2005 Embi [8]	Early development	Physicians	114	Prospective	Ambulatory & Outpatient	Diabetes (not sepsis)	Rate (trial participation) & Time (recruitment)	Physician participation and trial recruitment rates increased
2008 Herasevich [9]	Early development	Patients	320	Prospective	Medical ICU	Severe sepsis & Septic shock	Feasibility (computerized screening)	Feasibility of automatic screening was demonstrated
2011 Herasevich [10]	Recent development	Patients	8609	Before-after	Medical, Surgical, & Mixed ICU	Severe sepsis & Septic shock	Rate (time sensitive study enrollment)	Automated screening improved enrollment efficiency
2011 Sawyer [11]	Recent development	Patients	300	Observational & Interventional	Nonintensive medical wards	Sepsis	Rate (early therapeutic & diagnostic intervention)	Early therapeutic and diagnostic interventions increased
2011 Nelson [12]	Recent development	Patients	33460	Before-after & Prospective	ED	Severe sepsis	Rate & Time (interventions)	Frequency and timeliness of some ED interventions increased
2012 LaRosa [13]	Recent development	Patients	58	Interventional	Mixed ICU	Severe sepsis & Septic shock	Rate (treatment compliance) & Outcomes (mortality)	Potential to improve treatment compliance and mortality was demonstrated
2012 Hooper & Selmer [14]	Recent development	Patients	442	Interventional (Randomized & Controlled)	Medical ICU	Early sepsis (SIRS criteria)	Rate (treatment compliance) & Outcomes (mortality)	Measurements of interest were not influenced
2015 Selmer & Hooper [16]	Current implementation	Patients	407	Interventional (Randomized & Controlled)	Medical & Surgical ICU	Sepsis (admission diagnosis)	Feasibility (electronic evaluation & management tool)	Measurements of interest were not influenced
2015 Harrison & Herasevich [21]	Current implementation	Patients	587	Diagnostic performance (Observational)	Medical ICU	Severe sepsis & Septic shock	Time & Accuracy (failed to recognize & treat)	Feasibility of failure to recognize and treat was demonstrated

The first methodologically rigorous clinical trials have failed to demonstrate improvements in clinically significant endpoints. In the first study,¹⁵ Hooper, Selmer, and colleagues deployed a modified Systemic Inflammatory Response Syndrome (SIRS) detection algorithm within an ICU setting. They randomized patients to groups monitored with the algorithm and those who were not. When modified SIRS criteria were met, clinicians were notified via text message. The hypothesis being tested was that automated notification would facilitate a diagnosis of sepsis and shorten the time to initiation of antibiotics, fluid administration, and other sepsis related cares. The study demonstrated the feasibility and safety of the approach but failed to demonstrate a difference in the time to administration of appropriate cares. More recently (2015), the same Vanderbilt group (Semler, Hooper, and colleagues) subsequently performed another randomized trial of an electronic tool for the evaluation and treatment of sepsis in the ICU.¹⁶ This system combined their existing automated, electronic monitoring system with a clinical decision support (CDS) system. As with their previous study, this system did not improve clinically significant outcomes in the ICU setting, including hospital/ICU length of stay (LOS) and timely completion of appropriate interventions.

At Mayo Clinic, an ICU-specific patient viewer has been clinically validated and implemented in the medical ICU setting.^17-19 In this context, Harrison, Herasevich, and colleagues developed a surveillance system for the detection of failure to recognize and treat severe sepsis.²⁰ The rationale for this system was to not only detect sepsis, but prevent clinically important deterioration and complications due to failure to treat this underlying illness in a timely manner (“failure to rescue”).^{21, 22} However, the validity of this or any other implementation approach has yet to be tested in a clinical trial.

Barriers to Development and Implementation of Clinically Useful Sepsis Alert Systems

In addition to real-time availability of accurate electronic data, the ability of a sepsis detection algorithm to reliably identify sepsis is influenced by many external factors. Critically, algorithms are developed using current knowledge of the condition of sepsis and on data derived from a particular health care setting or patient population. The performance is optimized for those conditions and will be unpredictably altered if used in any other context. In the face of evolving definitions of sepsis and treatment guidelines, changing patient populations or clinical settings, the performance of sepsis algorithms must be continuously monitored and tweaked. Even small changes in the sensitivity or specificity of these algorithms can lead to high rates of false positive or negative alerts. These changes can undermine confidence in the alert and render it ineffective in clinical practice (Figure 1).

Figure 1.

Sepsis Alert Schematic: The expected outcome is that a sepsis detection algorithm will reduce time to recognition of sepsis and in doing so will prompt clinicians to intervene earlier than they might otherwise. Some potential points of failure are illustrated in this figure.

Clinical diagnostic cues not available in the EMR

Often the critical rate limiting step for efficacy of sepsis alert systems is the availability of real time data in the EMR. The data has to be in the record before the algorithm can “see” it and make a prediction about whether the patient is at risk of sepsis or not. Delayed data entry or validation, lack of interconnectivity of EMR department systems and infrequent sampling times all contribute to patchy, absent or much delayed data availability in the EMR. Furthermore, the clinical diagnosis of sepsis often relies on judgments and measurements not easily captured in the EMR. These measurements can range from physical findings (patient “not looking good”, rigors, increased capillary refill time, bounding pulse or increased work of breathing) to physiological markers (such as low blood pressure and shock index) or molecular biomarkers (such as lactate, C-reactive protein, and procalcitonin).²³ At the present time clinical algorithms to detect sepsis are blinded to many of the cues a bedside clinician takes for granted. The result is that, from the perspective of the clinician, sepsis detection alerts will often fire “late”. Late alerts are nuisance alerts and, understandably, are very poorly tolerated by clinicians.

Algorithm alert performance

Changing definitions of sepsis and its treatment, in addition to patient and health setting characteristics have a significant impact on the performance of algorithms developed to detect sepsis. Evolving clinical definitions of sepsis are particularly difficult to deal with using the current model of algorithm development which is largely based on these definitions.^{24, 25} Alerts developed carefully on the best available evidence, often over an extended period of time, become obsolete when the definitions upon which they are based change. The guidelines for the management of severe sepsis and septic shock by the international Surviving Sepsis Campaign have evolved over time^26-28 and have been challenged by independent, international, multicenter randomized controlled trials (Table 2).^29-31

Table 2. Overview of studies of the evolving clinical definitions of sepsis and recommended treatment.

Reference	Institutions	N	Design	Setting	Illness	Primary outcome	Result
1992 Bone [25]	United States (US)	N/A	Consensus conference	N/A	Sepsis	Formal definition of sepsis	Consensus definition established
2001 Rivers [26]	1 (US)	263	Interventional (randomized)	ED	Severe sepsis & Septic shock	Early goal-directed therapy (EGDT)	EGDT demonstrated to significantly reduce mortality
2004 Dellinger [27]	International	N/A	Consensus guidelines	N/A	Severe sepsis & Septic shock	Surviving Sepsis Campaign - 2004	Consensus definition established
2008 Dellinger [28]	International	N/A	Consensus guidelines	N/A	Severe sepsis & Septic shock	Surviving Sepsis Campaign - 2008	Consensus definitions updated
2013 Dellinger [29]	International	N/A	Consensus guidelines	N/A	Severe sepsis & Septic shock	Surviving Sepsis Campaign - 2012	Consensus definitions updated
2014 Yealy [29]	31 (US)	1341	Interventional (randomized & controlled)	ED	Early septic shock	Reevaluation of EGDT	“Early” (E) demonstrated to reduce mortality, but not “goal-directed” (GD)
2014 Peake [30]	51 (Australia & New Zealand)	1600	Interventional (randomized & controlled)	ED	Early septic shock	Reevaluation of EGDT	“Early” (E) demonstrated to reduce mortality, but not “goal-directed” (GD)
2015 Mouncey [31]	56 (England)	1260	Interventional (randomized & controlled)	Any (post-ED admission)	Early septic shock	Reevaluation of EGDT	“Early” (E) demonstrated to reduce mortality, but not “goal-directed” (GD)
2015 Kaukonen [32]	172 (Australia & New Zealand)	109663	Retrospective	ICU	Severe sepsis	Diagnostic value of SIRS criteria	A significant portion of severe sepsis patients are “SIRS negative”

Typically alerting algorithms are developed on a fixed dataset. The performance of the clinical algorithm is tweaked to perform best on that data set. Depending on the characteristics of the patient population and health care setting, the derived clinical algorithm may not perform as expected in different environments. The healthcare delivery systems differ significantly between different clinical settings.³² As an example, nursing ratios are lower, vital sign capture is less frequent, and clinicians are less tuned to rapid intervention with limited information on the floor compared to in the ICU. In addition there are important differences in the clinical characteristics and demographics of the patients in each of these settings.^33-35 These factors combine to make the pretest probability a patient in the ICU has sepsis much higher than that of a floor patient. This alone may be sufficient to skew the performance of an alert developed on an ICU cohort of patients but deployed on the hospital floor. Finally, even if the alert is optimized to perform well in a specific population and environment, if the patient mix and health care delivery system change over time, performance may drift from the original optimum.

Sepsis alert systems must therefore, possess the flexibility to adapt to changes in definitions that affect the clinical management and treatment of sepsis, to changing patient population characteristics and evolving health settings. Evolution in medicine is not new, but the incorporation of new knowledge into practice is potentially greatly accelerated by electronic alerts and clinical decision support systems. Such systems require constant updating if they are not to have an unexpected negative impact on clinicians and patients.

Information overload & alert fatigue

Sophisticated technology and increasingly large quantities of data are currently flowing into existing, imperfect electronic medical record (EMR) systems in the acute care setting. This inevitable application of “big data” to health care cannot and should not be avoided.³⁶ However, the development and implementation of sepsis alert systems without consideration of the rise of big data in clinical practice has the potential to result in alert fatigue,³⁷ interruption,³⁸ human error,³⁹ and information overload.^{40, 41} The recognition of the importance of alert fatigue in the hospital setting has increased significantly in recent years.⁴² Thus, implementation of any automated alert system must be performed in the context of information overload and complex task interruption.^{43, 44} It is known that information overload can alter alert perception in the medical setting.⁴⁵ This can cause clinicians to perceive alert systems negatively and deter future use.^{46, 47}

The task of generating clinically meaningful alerts while concurrently minimizing information overload and task interruption is challenging.⁴⁸ Successful sepsis alert systems must be designed to minimize alert fatigue, interruption, human error, and information overload.

Variability in the systems of healthcare delivery

The primary purpose of any alert system should be to elicit a response from the clinical team that would otherwise not occur or would be delayed. As might be expected, what a team does with an alert has a tremendous impact on the efficacy of an automated alerting system. Variability in resource availability, leadership engagement, and clinical stakeholder buy-in have a fundamental impact on whether and alert will deliver its intended impact on outcomes. A poorly planned implementation can have a devastating effect on performance.⁴⁹ Perhaps the best illustration of this comes from an examination of a Computer Provider Order Entry (CPOE) implementation in a pediatric hospital which resulted in an unanticipated doubling in ICU standardized mortality ratio.⁵⁰ Subsequent root cause analysis implicated a highly flawed implementation process as the main contributor to these unanticipated outcomes.

Potential Solutions to Development and Implementation of Clinicaly Useful Sepsis Alert Systems

The difference between success and failure is often due to very small tweaks in approach. Slightly better algorithm performance, more frequent data, ergonomic user interfaces, well defined responses, all contribute to overall efficacy of an alert (Figure 2).

Figure 2.

Potential solutions: Some examples of the types of interventions which have the potential to improve the efficacy of sepsis alert systems.

Availability of real time data & alternatives

Often the critical rate limiting step for efficacy of sepsis alert systems is the availability of real time data in the EMR. All of the sepsis detection alerts described here rely on values found within the EMR. The data has to be in the record before the algorithm can “see” it and make a prediction about whether the patient is at risk of sepsis or not. Delayed data entry or validation, lack of interconnectivity of EMR department systems and infrequent sampling times all contribute to patchy, absent or much delayed data availability in the EMR. In contrast the clinical diagnosis of sepsis often relies on judgments and measurements not easily captured in the EMR. These measurements can range from the patient “not looking good”, increased capillary refill time, cold toes, bounding pulse or increased work of breathing, indicating suspicion of infection to physiological markers (such as low blood pressure) or molecular biomarkers (such as lactate, C-reactive protein, and procalcitonin).²³ At the present time, clinical algorithms to detect sepsis are blinded to many of the cues a bedside clinician takes for granted. The result is that, from the perspective of the clinician, sepsis detection alerts will often fire “late”. Late alerts are nuisance alerts and greatly reduce the efficacy of the alert system.

Improved alert performance - mathematical modeling & machine learning

In addition to more ergonomic design, sepsis alert systems of the future must be more accurate. Advanced mathematical modeling and complex machine learning techniques have already been applied to sepsis detection. Machine learning algorithms are already in use for the classification of human physical activity from on-body accelerometers.⁵¹ Thus, the use of these methodologies in sepsis alert systems to improve the clinical management and treatment of sepsis is forthcoming.

The syndromic nature of critical illness,⁵² such as sepsis, can result in disruption of homeostasis with inflammation and shock (inadequate oxygen supply/demand). In the context of evolving definitions and gold standards of sepsis, this means it is frequently important to recognize and treat with or without confirmation of infection. Given the current inability to diagnose invasive infection, as exemplified in the gut microbiome hypothesis of sepsis syndrome,⁵³ this means measures for infection control must be introduced empirically and deescalated if no infection is identified. Although these factors may appear to initially reduce to potential accuracy of any sepsis alert system, global awareness of homeostasis in sepsis alert systems of the future will increase the accuracy of these systems. This approach is the basis for the ICU-specific patient viewer that has already been clinically validated and implemented in the medical ICU setting at Mayo Clinic.¹⁹

Alert delivery & integration into workflow

In parallel with advances in proxy measurements for the diagnosis, prognosis, management, and treatment of sepsis, there is a need to optimize delivery of alerts for these proxy measurements and suspicion of sepsis to clinicians. Studies of methods of alert delivery for clinical information and clinical trial enrollment have been performed outside the ICU setting.^{54, 55} Increasingly sophisticated electronic decision support systems are developed with specific consideration of human factors.⁵⁶

The best method(s) of alert delivery (text paging, EMR systems, email, phone calls, and/or text messaging) for urgent and non-urgent alerts in the hospital setting is poorly understood.⁵⁷ Likewise, investigation into the most appropriate clinician recipient (attending physicians, fellows, residents, and/or nurse practitioners/physician assistants) for alert delivery is limited.^{58, 59} As multidisciplinary response teams have been demonstrated to improve the process of care and mortality in septic shock,⁶⁰ an appropriate trigger for multidisciplinary team activation needs to be determined for each individual setting. In the medical ICU of the Mayo Clinic, the electronic sepsis alert is currently send to a single team member who, if appropriate, activates a multidisciplinary response team (currently “shock response team” consisting of physicians, nurses, respiratory therapists, pharmacists and laboratory technicians) in charge of rapid execution of time-sensitive interventions.

Reengineering the hospital environment

Another potential strategy to the barriers of sepsis alert system implementation is the use of new models of care delivery. Models which recognize the limitations of the current generation of alert systems but which also exploit their potential to provide a safety net for the busy bedside clinician. As we gain more experience in the use of telemedicine, this may prove to be a useful way in which to incorporate of a wide range of alerts into health systems. In this model, a dedicated clinician with the appropriate information tools would e-monitor the entire hospital, validate or silence alerts and trigger appropriate responses for true positive alerts. A similar approach has already been used by Khan et al where remotely located nurse screening and prompting for appropriate evidence-based interventions improved processes and outcomes of ICU patients.⁶¹

Key Points.

• Barriers to implementation of sepsis alert systems in include evolving clinical definitions of sepsis, delayed availability of data through the EMR, information overload & alert fatigue

• To be clinically useful, alert systems of the future will need to be more reliable with lower rates of false positive alerts and be much better integrated into clinical workflow.

• Emerging concepts and strategies which may increase the clinical utility of alerts include: wearable physiologic monitoring devices, cognitive ergonomics, human-centered interface design, use of more sophisticated mathematical modeling & machine learning techniques, and integrated prevention, patient education, & public awareness.