Evaluating a digital sepsis alert in a London multisite hospital network: a natural experiment using electronic health record data

Abstract

Objective

The study sought to determine the impact of a digital sepsis alert on patient outcomes in a UK multisite hospital network.

Materials and Methods

A natural experiment utilizing the phased introduction (without randomization) of a digital sepsis alert into a multisite hospital network. Sepsis alerts were either visible to clinicians (patients in the intervention group) or running silently and not visible (the control group). Inverse probability of treatment-weighted multivariable logistic regression was used to estimate the effect of the intervention on individual patient outcomes.

Outcomes

In-hospital 30-day mortality (all inpatients), prolonged hospital stay (≥7 days) and timely antibiotics (≤60 minutes of the alert) for patients who alerted in the emergency department.

Results

The introduction of the alert was associated with lower odds of death (odds ratio, 0.76; 95% confidence interval [CI], 0.70-0.84; n = 21 183), lower odds of prolonged hospital stay ≥7 days (OR, 0.93; 95% CI, 0.88-0.99; n = 9988), and in patients who required antibiotics, an increased odds of receiving timely antibiotics (OR, 1.71; 95% CI, 1.57-1.87; n = 4622).

Discussion

Current evidence that digital sepsis alerts are effective is mixed. In this large UK study, a digital sepsis alert has been shown to be associated with improved outcomes, including timely antibiotics. It is not known whether the presence of alerting is responsible for improved outcomes or whether the alert acted as a useful driver for quality improvement initiatives.

Conclusions

These findings strongly suggest that the introduction of a network-wide digital sepsis alert is associated with improvements in patient outcomes, demonstrating that digital based interventions can be successfully introduced and readily evaluated.

INTRODUCTION

Sepsis is recognized as a common cause of serious illness and death. It is estimated that there are 123 000 cases and 46 000 deaths in England each year.¹ Similar high levels of sepsis have been reported internationally^2,3 and sepsis is recognized by World Health Organization as a global health priority.⁴ Many countries have nationwide sepsis action plans, and in England there are targets for hospitals to rapidly diagnose and treat patients with sepsis.

Timely, appropriately targeted, intravenous (IV) antibiotics have been shown to be effective in improving outcomes for patients, with a 4% increase in odds of mortality for every hour’s delay in administration of IV antibiotics.^5–7 This evidence has resulted in UK hospitals having a target (with financial incentives) of sepsis patients receiving IV antibiotics in 1 hour.^8,9

To ensure rapid diagnosis and early treatment, sepsis screening tools have been introduced and refined, and include qSOFA (quick sequential organ failure assessment score),¹⁰ NEWS (National Early Warning Score),¹¹ and NEWS2.¹² Early warning scores have been shown to be effective in predicting mortality¹³ and intensive care unit (ICU) admission.¹⁴ There is limited evidence that the introduction of track and trigger style warning systems have been associated with improved outcomes for patients.

The introduction of electronic health records (EHRs) has provided the opportunity to embed digital alerts based on current and past clinical measurements. A range of screening algorithms have been used, including the St John Sepsis Algorithm,^15,16 the Severe-Sepsis Best Practice Alert,¹⁷ and hospital-designed alerts.¹⁸ The evidence for the effectiveness of these alerts on patient outcomes is mixed.¹⁹ Some studies have shown that introduction of digital sepsis alerts have led to increases in the proportion of patients with suspected sepsis receiving IV antibiotics in 1 hour,¹⁷ reduced ICU and hospital length of stay (LOS),²⁰ and reduced in-hospital mortality,^18,20 while others have shown no significant effect on LOS^20,21 or in-hospital mortality.¹⁷ A recent randomized controlled trial (RCT) analyzing the impact of the introduction of an alert for inpatients in a U.S. hospital found no association between the introduction of an alert and an improvement in patient outcomes, although the study was terminated when the hospital decided to roll the alert out, and hence underpowered to detect associations.²² The majority of evidence comes from studies in the U.S. healthcare system and sample sizes have been relatively small. It is not known if similar impacts on patient outcomes will be seen in larger scale studies, particularly in an English hospital, where care is free at the point of delivery and accessible to all.

The aims of this study were to determine the effect of the introduction of a digital sepsis alert on 1) key process measure (timely antibiotics); and 2) patient outcomes (extended LOS and in-patient 30-day mortality).

MATERIALS AND METHODS

Study design

In this natural experiment, a weighted multiple logistic regression was used to examine the effect of the digital sepsis alert. Data from October 2016 to May 2018 was included in the study which utilized a “silent” running phase, during which time digital alerts were active but not visible to clinicians. The silent phase provides a control group. Robust statistical methods were used to balance characteristics between the live and control phases. The primary outcome was 30-day inpatient mortality and a secondary outcome of prolonged LOS (≥7 days). Additionally, the impact of the introduction of the alert on the key process measure of timely IV antibiotics (≤60 minutes after the alert) was studied.

Digital sepsis alert

The digital sepsis alert is based on the St John Sepsis Algorithm developed by Cerner Corporation,²⁶ shown in Figure 1. The alert is an integrated part of the EHR and has a silent running mode. Silent alerts are not visible to clinical staff at the “front end” of the system. Once the alert is turned on (live) in a clinical area nurses and doctors are notified of patients who have triggered the alert. Nurses are notified of patients who have triggered the alert either through a pop-up warning on the EHR (in inpatient wards) or as a dashboard which highlights any patient with an active alert (in the emergency department [ED] and inpatient wards). Doctors are presented with a sepsis warning only when they open the patient’s record. In addition to the alert, a novel multidisciplinary care pathway, designed by the Trust, launches from the digital record when the clinician confirms suspicion of sepsis. These “Treatment Plans” support the clinician to start treatment in-line with hospital guidance, including fluids, oxygen, diagnostic tests (blood and other cultures), and early antibiotics. Content from local infection guidelines is built in, so that for any given potential sepsis diagnosis, the appropriate antibiotics with appropriate dosing and directions are prompted.

Figure 1.

Criteria associated with the St John Sepsis Alert.²³ MAP: Mean Arterial Pressure; SPB: Systolic Blood Pressure.

The introduction of the alerts was part of a framework for system redesign and improvement, see Box 1 for more details. The alert was introduced in a phased approach over an 18-month period across the Trust, summarized in Figure 2. The alert was switched from silent to live as recommended by improvement approaches for scale and spread.^25,27 Initially in the acute medical unit at one site, expanding out to both EDs and hematology wards and then Trust-wide. Initial areas were selected based on their interest in assistance in identifying patients with suspicion of sepsis.

BOX 1. The local picture.

The hospital network (Trust) comprises of 5 main sites. In recent years, the Trust had over 1 million outpatient contacts, quarter of a million ED attendances, 200 000 inpatient contacts, and 100 000 inpatient operations. The Trust employs more than 2500 doctors, 4000 nurses, 720 allied healthcare professionals, and 130 pharmacists.

Work to improve care for sepsis patients at the Trust centers on 3 key priorities: The identification and treatment of sepsis across the whole patient pathway Consistency of standards and reporting The prudent use of antimicrobials within the wider antimicrobial stewardship and resistance agenda.

A key focus has been to ensure that patients identified with sepsis receive the appropriate antibiotics within 1 hour, in line with national targets. The work is integrated with the digital transformation and the use of an embedded digital sepsis alert in the EHR.

The digital sepsis alert embedded in the EHR and available to the Trust is the St John Sepsis Algorithm.

Implementation of the alert was part of a collaborative improvement approach through the Sepsis Big Room. A “big room” is a weekly coached meeting which provides time and space for a range of staff to come together to discuss improvements to the quality of patient care. Staff from all disciplines are welcome and the meetings operate a flattened hierarchy. Patient stories are reviewed and real-time data displayed to support the identification of specific improvements to healthcare processes within the pathway of care. In an approach similar to one others have used, a series of tests of change were undertaken to improve decision making and communication for sepsis patients. For each test, a small-scale Plan-Do-Study-Act cycle, based on Toyota Big Room methodology,23,24 was performed and, if this proved successful, the test was tried more widely. Data from the evaluation were used to provide feedback to the team and shape discussion on implementation. A key aspect of work undertaken by the Sepsis Big Room was the development of “Treatment Plans” for specific diagnosis which are embedded in the EHR system. A sepsis treatment plan was developed, which includes relevant investigations, fluids, oxygen prescriptions, and specific antibiotics according to guidelines. This is in line with the Institute for Healthcare Investigations sepsis bundle.25

Figure 2.

Phased introduction of live alerts across multisite hospital from autumn 2016. The digital alert was switched from silent to live in acute wards, followed by emergency departments in 2 hospital sites in autumn and hematology departments soon after. The alert was switched to live across all inpatient wards in August 2018, after data were extracted for this study.

Patient population

Patients who triggered the alert were included in the analysis. These are patients who were identified as potentially having sepsis by the clinical thresholds included in the St John’s Sepsis alert (Figure 1). The unit of analysis was an adult inpatient “encounter.” An encounter was defined as a continuous spell in the Trust. In this analysis all encounters of adult patients (18 years of age and older) in which a sepsis alert was triggered were included. Although the alert may be triggered repeatedly for a patient during a hospital encounter, only first alerts were considered. All patients admitted to the 3 hospitals in the network that have general acute admissions were included. Only encounters ending in admission were included in final models to ensure detailed comorbidity and outcome information were available. For LOS and timely antibiotics, only patients who alerted in the ED were included, to reduce potential confounders including the high proportion of inpatients who are on antibiotics when the alert fires for reasons not directly related to the alert and the impact of prior long inpatient stays on LOS postinfection. Patients who were already on antibiotics were excluded from the analysis of timely antibiotics. Patients who did not receive antbiotics within 24 hours of the alert were excluded-it was assumed the alert had triggered for patients who did not require antibiotics (further details in Supplementary Appendix S1). See Figure 3 for a summary of patient cohorts.

Figure 3.

Cohort definition. Three cohorts were developed to investigate the outcomes of interest. Cohort A comprised all patients who alerted and the outcome of interest in this cohort was mortality. Cohort B comprised patients who alerted in the emergency departments (EDs) only, as the main outcome of interest was length of stay. Cohort C comprised patients who alerted in the ED who received antibiotics within 24 hours postalert. The main outcome of interest was timely antibiotics, defined a receiving antibiotic within 1 hour of the alert (as per NICE guidelines).²⁸

Outcomes

The outcomes were (1) in-hospital all-cause mortality within 30 days of alert, (2) long hospital stay (≥7 days), and (3) timely antibiotics (IV antibiotics ≤60 minutes). Both long hospital stay and timely antibiotics were investigated only for patients who alerted in the ED. LOS was measured as time from alert to discharge.

For the purposes of this study, timely antibiotics was defined as patients who received IV antibiotics within 1 hour of the alert. This definition was informed by the current target for hospitals in England.⁸

Statistical analysis

Three separate analyses were undertaken in 3 cohorts to explore the 3 main outcomes. The switch from silent to live was considered as a natural experiment. Inverse probability of treatment weighting (IPTW) was used²⁹ to account for confounding the nonrandom allocation introduced and balance characteristics between the live and control phases. Multivariable logistic regression was used to determine the propensity score weights. As the sample was different for each outcome, separate regression models were used to determine propensity score weight observations in each sample. Further details are available in Supplementary Appendix S2. Potential confounders for the three outcomes and alert status allocation were included in models. These included patient characteristics, admission details and clinical measures. Data were obtained from patient digital medical records. Details of variables are included in Box 2.

BOX 2. Covariates included in both the propensity score models and the multivariable logistic models.

	Patient characteristics
Age	Grouped into 18-44; 45-64; 65-69; 70-74; 75-79; 80-84; and 85+ years of age
Sex	Male or female
Ethnicity	Based on the following groupings:
	White; Asian; Black; other; and not known.
Comorbidities	Based on any relevant International Classification of Diseases–Tenth Revision code appearing in the discharge diagnosis codes (25 possible codes)
	Myocardial infarction; congestive heart failure; peripheral vascular disease; stroke; dementia; pulmonary; rheumatic; peptic ulcer disease; liver (mild); liver (severe); diabetes; diabetes (complex); paralysis; renal; metastatic cancer; human immunodeficiency virus.
Deprivation quintile	Measured as the deprivation score of the patient’s primary care practice, obtained by matching patients to their registered practice. If patients did not have a registered primary care practice or the practice was not included in the Public Health England practice profiles a “missing” categorization was allocated. There are therefore 6 deprivation categories, with Quintile 1 being the least deprived.
	Admission characteristics
Admitting hospital	A has an ED department
	B no ED department
	C has an ED department
Season of admission	Spring: March, April, May
	Summer: June, July, August
	Autumn: September, October, November
	Winter: December, January, February
	Patient severity
NEWS score	Categorized into: zero, low, medium, high, and none recorded
	NEWS = 0: zero
	1 ≤ NEWS < 5: low
	5 ≤ NEWS < 7: medium
	7 ≥ NEWS: high
	A NEWS score is available for 19 599 (90%) of the patients.
Alert	Suspected sepsis
	Suspected severe sepsis

Balance between treatment populations was evaluated using standardized mean differences (SMDs) of all baseline covariates. A threshold of 10% indicates possible imbalance and 25% was an indication of unacceptable imbalance.

The odds ratio (OR) and 95% confidence interval (CI) were estimated for each outcome using logistic regression applying the propensity score weights. A doubly robust approach was employed,³⁰ including covariates in both the propensity score models and the multivariable logistic models of the outcomes. When modeling death, a random-effects model was used to account for clustering within the acute, hematology, and ED areas, with all other alerts included in an additional cluster. Additionally the outcomes were modeled using logistic regression without applying the propensity score weights but adjusting for confounders. All analyses were done with R version 3.2.3 (R Foundation for Statistical Computing, Vienna, Austria).

RESULTS

Study population

In total, there were 21 732 patient encounters with at least 1 alert between October 2016 and May 2018. A total of 9988 of these were in the ED, 942 alerted in acute wards, and 1218 alerted in hematology wards. A total of 4622 ED patients were not on IV antibiotics at the time of the alert and did receive IV antibiotics within 24 hours of the alert. See Figure 3 for cohort details.

Table 1 summarizes the characteristics of patients for whom the alert was during a silent phase and those for whom the alert was in a live phase (clearly visible to clinicians). The phased introduction of the live alert means that the patient and encounter characteristics of the 2 groups are not balanced and there are more live alerts for patients admitted to site C, admitted through the ED, and admitted in the autumn and winter. Within live alerts, there was a higher proportion of younger patients (18-44 years of age) and older patients (85 years of age and older). In comparison with silent running, patients who alerted in the live phase were more likely to have pulmonary conditions but less likely to have renal conditions. In addition, these patients were less likely have an unknown ethnicity. A higher proportion of patients who alerted in the live phase had medium, high, or missing NEWS score. A higher proportion had suspected severe sepsis in comparison to suspected sepsis.

Table 1.

Distribution of patient and encounter characteristics for all alerts and standardized mean difference before and after weighting

Factor		Level	Control phase alerts (n = 15 056)		Live phase alerts (n = 6127)		All alerts (n = 21 183)		Standardized mean difference (%)
			n	%	n	%	n	%	Before IPTW	After IPTW
Age group		18-44 y (ref)	2588	17.2	1239	20.2	3827	18.1	13.0	11.5
		45-64 y	4431	29.4	1644	26.8	6075	28.7
		65-69 y	1594	10.6	515	8.4	2109	10.0
		70-74 y	1583	10.5	603	9.8	2186	10.3
		75-79 y	1605	10.7	636	10.4	2241	10.6
		80-84 y	1393	9.3	601	9.8	1994	9.4
		85 y and older	1862	12.4	889	14.5	2751	13.0
Sex		Female	6936	46.1	2735	44.6	9671	45.7	2.0	0.4
		Male	8120	54.0	3392	55.4	11512	54.4
Ethnicity		White	6986	46.4	3063	50.0	10049	47.4	12.8	2.7
		Black	1667	11.1	708	11.6	2375	11.2
		Not known	2473	16.4	784	12.8	3257	15.4
		Other	2923	19.4	1257	20.5	4180	19.7
		Asian	1007	6.7	315	5.1	1322	6.2
Deprivation quintile		Least deprived	2787	18.5	1591	26.0	4378	20.7	22.4	9.1
			3953	26.3	1662	27.1	5615	26.5
			4024	26.7	1468	24.0	5492	25.9
			2243	14.9	713	11.6	2956	14.0
		Most deprived	1277	8.5	319	5.2	1596	7.5
		Not known	772	5.1	374	6.1	1146	5.4
Myocardial infarction			1755	11.7	579	9.5	2334	11.0	7.2	8.4
Heart failure			2420	16.1	858	14.0	3278	15.5	5.9	9.4
Peripheral vascular disease			1186	7.9	384	6.3	1570	7.4	6.9	1.2
Stroke			2257	15.0	909	14.8	3166	15.0	0.8	1.7
Dementia			1082	7.2	558	9.1	1640	7.7	8.0	1.8
Pulmonary			3626	24.1	1928	31.5	5554	26.2	16.0	1.5
Rheumatic			567	3.8	225	3.7	792	3.7	0.3	4.0
Peptic ulcer disease			215	1.4	109	1.8	324	1.5	1.8	7.3
Liver disease–mild			957	6.4	426	7.0	1383	6.5	2.0	0.6
Diabetes–uncomplicated			3700	24.6	1478	24.1	5178	24.4	1.1	4.6
Diabetes–complicated			1145	7.6	340	5.6	1485	7.0	8.8	11.3
Paralysis			649	4.3	243	4.0	892	4.2	1.9	4.8
Renal			3141	20.9	801	13.1	3942	18.6	21.6	14.2
Liver disease–severe			328	2.2	160	2.6	488	2.3	1.9	0.6
Metastatic cancer			1329	8.8	348	5.7	1677	7.9	10.6	1.2
Human immunodeficiency virus			150	1.0	64	1.0	214	1.0	0.5	3.1
Trust site	A		6342	42.1	2427	39.6	8769	41.4	48.3	11.5
	B		4684	31.1	872	14.2	5556	26.2
	C		4030	26.8	2828	46.2	6858	32.4
Season of admission	Spring		2359	15.7	2173	35.5	4532	21.4	58.8	6.0
	Summer		2608	17.3	394	6.4	3002	14.2
	Autumn		5553	36.9	1380	22.5	6933	32.7
	Winter		4536	30.1	2180	35.6	6716	31.7
Severity	Suspected sepsis (ref)		8025	53.3	2775	45.3	10800	51.0	10.3	3.1
	Suspected severe sepsis		7031	46.7	3352	54.7	10383	49.0
NEWS score	Zero (ref)		617	4.1	183	3.0	800	3.8	23.3	17.7
	Low		8513	56.5	2887	47.1	11400	53.8
	Medium		2331	15.5	1165	19.0	3496	16.5
	High		2277	15.1	1208	19.7	3485	16.5
	Missing		1318	8.8	684	11.2	2002	9.5

IPTW: inverse probability of treatment weighting;NEWS: National Early Warning Score.

IPTW was used to account for these baseline differences. Models were adjusted for differences in the 2 groups using IPTW. After weighting, all models were balanced. SMDs for cohort B used to model LOS and cohort C used to model timely antibiotics were 3.0% or less (compared with a highest value of over 70% before imposing estimated weights). For cohort A, to model death, the SMDs for hospital site, alert introduction area (cluster), age, NEWS, diabetes, and renal disease were all above 10% but under 25%. As a doubly robust method all potential confounders were included in the final model accounting for any confounding from known confounders. Further details are available in Supplementary Appendix S3.

Association of alert status with death

A total of 21 732 inpatients alerted during the period of study, across all wards, across the 3 sites. A total of 1293 (6.0%) patients died within 30 days of the alert being triggered, which is similar to mortality rates reported elsewhere.²¹ Live alert status was associated with lower in-hospital mortality (5.1% compared with 6.4%; P < .001) (see Table 2). After accounting for patient characteristics, using IPTW propensity scores and patient characteristics in the multivariable logistic model, patients who alerted during the live phase had 24% lower odds (95% CI, 16%-30% lower odds) of in-hospital death.

Table 2.

Summary data and results of models, including adjustment for confounders

	Death		Extended LOS		Timely Antibiotics
	Control	Live	Control	Live	Control	Live
Total encounters	15 061	6671	4494	5494	1927	2695
Number of events	959	339	1846	2209	712	1204
% events	6.4	5.1	41.1	40.2	36.9	44.7
	OR (95% CI)		OR (95% CI)		OR (95% CI)
Unadjusted	0.67 (0.67-0.90)		0.97 (0.89-1.05)		1.38 (1.22-1.55)
Adjusted (reg)a	0.79 (0.67-0.93)		0.97 (0.87-1.05)		1.70 (1.43-1.95)
Adjusted (IPTW)b	0.76 (0.70-0.84)		0.93 (0.88-0.99)		1.71 (1.57-1.87)
	RR (95% CI)c		RR (95% CI)c		RR (95% CI)c
Adjusted (IPTW)b	0.76 (0.70-0.84)d		0.96 (0.93-0.99)		1.35 (1.28-1.41)

After adjustment for potential confounding and IPTW, measures of association did not change markedly, but were more precise.

CI: confidence interval; IPTW: inverse probability of treatment weighting; LOS: length of stay; OR: odds ratio; RR: relative risk.

^aAdjusted for all confounders summarized in Table 1.

^bPropensity score–weighted log-linear fully adjusted model used to estimate Odds Ratios.

^cRelative risks determined using a fully adjusted, propensity score–weighted log-linear model.

^dWith the exception of death, which is estimated directly from the OR, as the event is rare.

Association of alert status with long LOS

A total of 9988 patients alerted in the ED and were subsequently admitted and 4055 (40.6%) of these patients subsequently had a long LOS (≥ 7 days), measured as time from the alert to discharge. After accounting for patient characteristics, live alert status was significantly associated with decreased odds of long LOS. Patients who alerted during the live phase had a 7% lower odds of a long LOS.

Association of alert status with timely antibiotics

A total of 6563 of the 9988 (65.7%) patients who alerted in the ED received antibiotics within ±24 hours of the alert firing.

Of the 4622 patients who were not on antibiotics when the alert fired, 36.9% of encounters that activated during the control period resulted in IV antibiotics administered within 1 hour of the alert and 44.7% of encounters activated the alert when it was visible to clinicians. After accounting for patient characteristics, live alert status was associated with 71% higher odds of receiving timely antibiotics (OR, 1.71; 95% CI, 1.57-1.87). This approximates to a relative risk of 1.35 (95% CI, 1.28-1.41), a 35% increase in chance of receiving timely IV antibiotics.

Patients who did not receive antibiotics are summarized in the Supplementary AppendixTable S1A. It is assumed that the majority of these patients did not need IV antibiotics. This assumption is supported by the improved prognosis of these patients (6.4% died and 46.6% had a prolonged LOS compared with 3.8% and 29.1% of those who did not receive antibiotics). Patient characteristics were compared in those that received antibiotics and those that did not. A higher proportion of elderly patients, patients with a high NEWS score, and patients who alerted in the spring and winter received antibiotics. Ethnicity, deprivation, and sex were not significantly associated with receipt of antibiotics, suggesting that clinical aspects of the patient and not the underlying health inequalities are associated with receipt of antibiotics. Sensitivity analysis was carried out to determine the impact of excluding patients who had not received antibiotics within 24 hours, and when these were included the introduction of the live alert was associated with a 81% higher odds of receiving timely antibiotics (OR, 1.81; 95% CI, 1.68-1.96).

DISCUSSION

This is the first evaluation of a digital sepsis alert in an English hospital and the largest undertaken anywhere to date. Robust methods were used to show that the introduction of a digital sepsis alert was associated with improvements in patient outcomes. Overall, 6.0% of patients who alerted as possible sepsis patients died within 30 days of the alert (all-cause, in-hospital); this is in a hospital network with a lower than expected overall in-hospital mortality.³¹ Patients for whom their first alert was during the live phase had lower odds of death (OR, 0.76; 95% CI, 0.70-0.84), lower odds of a long hospital stay (≥7 days) (OR, 0.93; 95% CI, 0.88-0.99), and increased odds of receiving timely antibiotics (OR, 1.71; 95% CI, 1.57-1.87). The magnitude and interpretation of these results is similar when using either a weighted or unweighted multiple logistic regression model. These results suggest an important clinical benefit from the introduction of alerting, although it is not possible to say the extent to which the presence of alerting, per se, is responsible for the benefits seen, or whether the alert acted as a useful driver for other quality improvement initiatives.

The phased nature of the introduction of the alert allows a detailed analysis, with live alerts and controls coming from a range of wards, settings, and time periods. Previous studies have not given a consistent picture of the effectiveness of electronic sepsis alerts on improving patient outcomes. Reasons for this may include the variation in patient cohorts studied, that is, ICU patients,¹⁸ ED patients,³² patients with a confirmed sepsis diagnosis, or all alerting patients as in this study. Studies also vary in their choice of outcomes, delivery of the complete 3-hour sepsis bundle, which includes therapeutic and diagnostic steps, is commonly reported in U.S. studies; see Shah et al³³ as an example. LOS is also reported, but modeled in a variety of ways, from 72 hours²² to analyzing mean differences.²¹ Many studies are small with <500 patients,^17,22,33 which means that statistical power to detect less common outcomes, such as mortality, is low. Furthermore, the majority of studies are observational studies and robust approaches to accounting for selection bias and time trends are not commonly used, although these were addressed by Austrian et al.³²

A recent RCT found that the introduction of a sepsis alert had no impact on receipt of antibiotics within 3 hours or additional outcomes, including in-hospital mortality.²² Possible reasons for the lack of effects suggested by the authors include a baseline high compliance and the alert firing after clinicians had diagnosed patients. In addition, this RCT focuses on a different patient cohort to our study including patients in wards, not in the ED, and further work is necessary to determine if alerts have different impacts in different settings. The RCT was terminated early when the hospital quality committee decided to roll out the alert to all eligible patients, meaning that the study did not have the planned statistical power to detect change. A study by Austrian et al³² did not find any impact on patient outcomes when an alert was introduced in an ED, although there was a significant impact on LOS. Smaller studies in the United States have found interventions which included an alert triggered by an EHR based diagnosis resulted in improved outcomes for patients. McRee et al²¹ found a reduction in mortality from 9.3% to 1.0% but no significant effect on LOS, although this was a small pilot study (n = 171). In another small study (n = 214) in the United States, the introduction of an alert was associated with an improvement in patients receiving timely antibiotics, from 48.6% to 76.7%, and a significant reduction in LOS.¹⁷ Shah et al³³ also found an increase in the receipt of antibiotics within one hour, although they did not take into account any trends in antibiotic administration in their pre- and postphase analysis. There was no significant impact on in-hospital mortality. Westra et al³⁴ and Guirgis et al³⁵ found the introduction of an alert, bundled with education, training, and structured care sets, to be associated with reductions in mortality and LOS.

This analysis of a large sample of patients who have been admitted across 3 sites to a busy hospital network in England is one of the largest to date. For mortality, both patients who presented with suspected sepsis in the ED and those who developed symptoms congruent with sepsis during their inpatient stay were included. For LOS and timely antibiotics, patients who alerted in the ED and were subsequently admitted were included in the analysis. Outcomes were selected based on their importance to both patients and hospitals, including those based on UK government targets, and were applied to all patient encounters where there was an alert, not limited to patients who were confirmed as having sepsis. A key methodological strength of this study was the inclusion of a “silent” running phase, during which time digital alerts were active but not visible to clinicians. The silent phase provides a control group. In addition, robust statistical methods were used to balance characteristics between the live and control phases.

There are a number of limitations to our study. First, the quasi-experimental design limits ability to imply causation. Although RCTs are considered the gold standard for analyzing health interventions, this approach was not deemed possible in the complex environment of this busy, multisite hospital. Difficulties of conducting an RCT on digital alerts for sepsis have been documented elsewhere.²² Statistical approaches recommended for the analysis of data derived from natural experiment were used.³⁶ Propensity scores are a recognized and recommended method to adjust for confounding factors, in this case introduced by the phased introduction of the alert. The majority of live alerts were ED patients who attended in autumn and winter, which resulted in a higher proportion of severe sepsis and higher NEWS scores in the live group. The impact of the wider sepsis quality improvement initiatives on improved outcomes for patients could not be robustly modeled, as it was not possible to associate the introduction of these initiatives with specific periods of time. Aspects of our analysis were limited by data availability; only admitted patients were included because clinical information was limited for patients discharged from the ED without admission. The analysis of the association of the alert on timely antibiotics and LOS was limited to patients in the ED to reduce potential confounding caused by patients being on antibiotics for reasons unconnected to the sepsis alert and the impact of prior long inpatient stays on LOS postinfection. We only found a significant impact on LOS after applying weights to correct in-balance between the 2 samples. Our definition of extended LOS was informed by national policy in England. Different definitions of LOS leading to different categorizations may have resulted in different interpretation of the impact of the alert.

In this study we have not considered unintended consequences of the introduction of the alert, including the possibility of increases in the use of inappropriate or unnecessary IV antibiotics, increases in patients being diagnosed as having sepsis without confirmation, and the possibility of alert fatigue as a result of relatively low specificity of the alert. This is the focus of future work.

The introduction of digital altering for sepsis is an opportunity to improve care for patients who may have sepsis. Despite the current emphasis on the use of sepsis screening tools, including the recommendation of the uptake of NEWS2 as the best current option for standardizing the management of deterioration and sepsis,³⁷ there is uncertainty around the use of digital screening to improve patient outcomes.³⁸ This study, the largest to date, is an important addition to the body of knowledge of appropriate digital-based screening, and shows that when associated with quality improvement approaches is associated with improved patient outcomes.

CONCLUSION

In 2 busy acute hospitals, the introduction of a digital sepsis alert has been shown to be associated with improved patient outcomes, including lower risk of mortality and extended LOS. A 70% increase in odds of receiving timely antibiotics was found, which is likely to be important in explaining the causal pathway for the alert improving outcomes for patients. This study has clearly shown that the introduction of a network-wide digital screening tool embedded in EHRs is associated with improvement in patient outcomes, demonstrating that digital-based interventions can be successfully introduced and readily evaluated.

FUNDING

This report is independent research funded by the National Institute for Health Research (NIHR) Biomedical Research Centre NIHR-BRC-P68711. CC is supported by a personal NIHR Career Development Fellowship (NIHR-2016-090-015). GSC is supported by NIHR Research Professorship. EW is supported by Medical Research Council Project Grant MR/M013278/1. AH is supported by NIHR Health Protection Research Unit (grant number HPRU-2012-10, 047) in Healthcare Associated Infections and Antimicrobial Resistance at Imperial College London in partnership with Public Health England. Cerner have not been involved at any stage of this service evaluation, which has been carried out completely independently of them.

AUTHOR CONTRIBUTIONS

CC and GSC conceived the study. KH, CC, GSC, and AK developed the protocol. AM, BG, MG, AK, CC, and KH defined the variables of interest from the clinical data and extracted the data. KH conducted the statistical analysis, which was reviewed and improved by EW and CC. MG and AK, working with the Big Room, refined and developed the data definitions and their clinical relevance and provided ongoing feedback on results and their interpretation. KH wrote the first draft of the manuscript. All authors reviewed and contributed to the final draft of the manuscript. CC is the guarantor. KH attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted. The views expressed are those of the author(s) and not necessarily those of the National Health Service, National Institute for Health Research, or the Department of Health and Social Care.