A two-stage clinical decision support system for early recognition and stratification of patients with sepsis: an observational cohort study

Abstract

Objective

To examine the diagnostic accuracy of a two-stage clinical decision support system for early recognition and stratification of patients with sepsis.

Design

Observational cohort study employing a two-stage sepsis clinical decision support to recognise and stratify patients with sepsis. The stage one component was comprised of a cloud-based clinical decision support with 24/7 surveillance to detect patients at risk of sepsis. The cloud-based clinical decision support delivered notifications to the patients’ designated nurse, who then electronically contacted a provider. The second stage component comprised a sepsis screening and stratification form integrated into the patient electronic health record, essentially an evidence-based decision aid, used by providers to assess patients at bedside.

Setting

Urban, 284 acute bed community hospital in the USA; 16,000 hospitalisations annually.

Participants

Data on 2620 adult patients were collected retrospectively in 2014 after the clinical decision support was implemented.

Main outcome measure

‘Suspected infection’ was the established gold standard to assess clinical decision support clinimetric performance.

Results

A sepsis alert activated on 417 (16%) of 2620 adult patients hospitalised. Applying ‘suspected infection’ as standard, the patient population characteristics showed 72% sensitivity and 73% positive predictive value. A postalert screening conducted by providers at bedside of 417 patients achieved 81% sensitivity and 94% positive predictive value. Providers documented against 89% patients with an alert activated by clinical decision support and completed 75% of bedside screening and stratification of patients with sepsis within one hour from notification.

Conclusion

A clinical decision support binary alarm system with cross-checking functionality improves early recognition and facilitates stratification of patients with sepsis.

Introduction

Sepsis is an uncontrolled inflammatory response to an infection.¹ A spectrum of sepsis exists and mortality increases as the severity of sepsis increases. Early recognition is paramount in improving sepsis outcomes as early intervention and resuscitation have been proven to improve patient outcomes, including survival.² Without early recognition, early intervention is impossible. When treatment is delayed, sepsis can rapidly advance to septic shock, multiple organ dysfunction syndrome, and death.³ Sepsis may be associated with one in two in-hospital deaths.⁴ The interval from diagnosis to treatment affects longer term patient outcomes too.^5,6 A sepsis programme enabled by clinical decision support (CDS) functionality offers a systematic application of health-related knowledge, which may be integrated with an electronic health record (EHR) system to achieve earlier intervention, as well as provide real-time surveillance and analysis.^7–9

Given the adverse consequences of sepsis among hospitalised patients, a hospital-based sepsis programme enabled by a two-stage CDS was established to accurately identify and stratify patients at risk of sepsis, and increase and expedite diagnostics and treatments. This study examined the programme’s accuracy in identifying patients at risk of sepsis, performance of clinical processes, and clinical outcomes.

Methods

Patients and data collection

This was an observational cohort study of prospectively screened patients admitted to a 284-bed urban, non-profit community hospital with more than 16,000 annual admissions. The hospital had an enterprise electronic health record (EHR) system (Millennium: Cerner Corporation, Kansas City, MO) and wanted to demonstrate the effectiveness of a sepsis programme enabled by cloud-based surveillance and computerised CDS to detect and stratify patients with sepsis. The U.S. Department of Health and Human Services’ Office for Human Research Protections clarified that quality improvement activities, described herein, often qualify for Institutional Review Board exemption and do not require individual informed consent.¹⁰

The sepsis programme was designed to improve early recognition and intervention of sepsis. The sepsis protocol was guided by the Surviving Sepsis Campaign resuscitation and management bundles.¹¹ Education regarding the ‘magic four’ as a simple concept highlighted clinical events and processes key to saving lives, i.e. STAT lactic acid, obtain cultures prior to antibiotics, early administration of antibiotics, and early administration of fluids in patients with severe sepsis or septic shock. An alert system was developed by combining resources of a sepsis CDS with the resources of the onsite IT department. The CDS is consistent with a human factors design for a binary (off/on) alarm system with user access to additional clinical information that can be used to cross-check the validity of the alarm before responding with a decision. Binary alarm systems are notable for high sensitivity but lower specificity. A postalarm cross-check activity has shown to improve specificity, but cross-checking can be time consuming,¹² which has implications for provider adoption.

The hospital population surveillance screening tool component was a cloud-based early recognition system (St. John Sepsis Rescue Agent: Cerner Corporation; Kansas City, MO). The base system definition^13,14 and its performance characteristics are described in a previous publication.¹⁵ The alert system’s crawler values and suppression parameters are described in the ‘Definitions’ section. Essentially, the alert system applied a binary alarm system paradigm with two alerts: (1) indications of SIRS (proxy for sepsis) and (2) indications of Severe SIRS (proxy for severe sepsis). The system, running continuously to monitor patient diagnostics from arrival until discharge, was integrated into the EHR and clinical workflow. The project team defined positions and relationships to present alerts to specific nursing and provider groups; designed rules to drive alerts, diagnosis, orders, and documentation; and built forms and plans of care for clinical documentation. The core enabling content for clinical decision-making includes an alert notification generated in the cloud and delivered to a nurse (Figure 1), a rules-driven alert notification delivered to a provider (Figure 2), and a sepsis screening and stratification form presented to a provider (Figure 3).

Figure 1.

Alert notification to nurse.

Figure 2.

Alert notification to provider.

Figure 3.

Sepsis screening and stratification form.

The clinical workflow included the following process: alert notifications with clinical indications were delivered to a designated nurse^16,17 who became responsible for contacting a provider within 5 minutes of receiving the alert. The provider was responsible for conducting the sepsis assessment and stratification within 15 minutes of contact, and, if indicated, submitting orders for the suggested sepsis plan of care, including initiating the initial resuscitation bundle as delineated in the surviving sepsis guidelines. A provider–nurse relationship was established to ensure completion of the resuscitation bundle.

After receiving notification from nursing that an alert had ‘fired’, providers could either complete the EHR sepsis screening and stratification form or bypass it. This cross-check option allows providers to establish whether the alert does or does not meet criteria, and if indicated, document the severity of sepsis. The form, developed to meet Surviving Sepsis bundle compliance, pulled in details of the alerting criteria. Based on severity of sepsis, the provider was prompted to place appropriate orders and/or consider critical care consultation. Also, based on level of sepsis risk stratification, a patient’s sepsis-related International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnosis code was added to the EHR Problems and Diagnosis tab, and the suggested plan of care was added to the Orders Tab as well. The provider could also submit orders once the sepsis plan of care was initiated.

Provider education with regards to the importance of early recognition was paramount when compared to a standalone technical solution.¹⁸ Provider education was conducted prior to go-live; which included a live lecture series, an online teaching tool, navigating the EHR with the locally developed sepsis screening and stratification tool, and highlighting clinimetric performance of the CDS related to sensitivity and specificity.

Data collection

The study included adult (≥ 18 years old) hospitalised patients following implementation of the sepsis programme in 2014. The clinical process applied a day in the life of a patient paradigm from arrival to hospital discharge. Source data included EHR registration, vital signs, laboratory, pharmacy, and clinical orders. Discharge codes for all encounters were reviewed to identify false negative system performance; i.e. documented sepsis-related International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnosis code but no CDS alert notification delivered. The study then focused on a cohort of patients screened-in by the sepsis CDS and then screened and stratified by a provider at the patient bedside.

Definitions

Sepsis and severe sepsis were defined per the American College of Chest Physicians/Society of Critical Care Medicine Consensus Conference.^13,14 Sepsis was defined as suspected or confirmed infection with clinical evidence of SIRS; while Severe Sepsis additionally required evidence of organ system dysfunction. ‘Suspected infection’ gold standard required at least one microbiology culture be obtained (e.g. blood, urine, sputum, ORSA/MRSA, or soft tissue) and administration of at least one IV/PO anti-infective antibiotics (e.g. antibacterial/fungal medication). Thresholds for SIRS were established when ≥ 3 of the following five criteria were satisfied: (1) temperature > 38.3℃ or < 35℃; (2) heart rate > 95 beats/min; (3) respiratory rate > 22 breaths/min; (4) white blood cell count > 12,000 cells/mm³, or < 4000 cells/mm³, or > 10% immature (band) forms; or (5) glucose 141 < 200 mg/dL. (Note: the threshold for temperature was lowered to < 35℃ because the surveillance alerting system was unable to distinguish core from non-core temperature.) Threshold for severe SIRS was established when ≥ 2 SIRS criteria present, and ≥ 1 of the following four organ system dysfunction criteria were satisfied: (1) cardiovascular system: SPB < 90 mmHg and/or MAP < 65 mmHg; (2) tissue perfusion: serum lactate > 2.0 mmol/L; (3) hepatic system: total bilirubin: ≥ 2.0 mg/dL and < 10.0 mg/dL; and (4) renal system: serum creatinine: Δ↑0.5 mg/dL from baseline. A look-back period consisted of 12 hours for serum lactate, 30 hours for the other criteria, and 72 hours for Δ↑ serum creatinine. Alert notifications for patients in an ICU location were not delivered to providers. Sepsis-related ICD-9-CM diagnosis codes at discharge included 038.xx, 995.91, 995.92, and 785.52.

Statistical analysis

Data were retrospectively analysed. A confusion matrix was applied to report sepsis prevalence, sensitivity and specificity, and positive (PPV) and negative (NPV) predictive values for the sepsis programme. Unadjusted analyses applied Fisher’s exact and chi-square (two-tail, p-value) for dichotomous variables in 2 × 2 and 2 × n contingency tables, respectively. All analyses were conducted using SPSS v21 (IBM, Inc., Armonk, NY).

Results

A total of 2620 patients with 14,907 hospitalisation days were included in the study; 417 (16%) patients were screened-in by the CDS, corresponding to an alert activation rate of 28 patients per 1000 patient days ([417/14,907 days] × 1000). Of the 417 patients screened-in, 210 (50%) were women. Patients’ age was median 68 (IQR = 54–83) years. Nearly all (94%) patients arrived at the emergency department. The CDS first alert activated median 3.9 (IQR = 1.3–31.9) hours after arrival; 89% (n = 370 of 417) patients were screened and stratified by providers. Stratification was completed median 19 (IQR = 6–50) min after alert activation. Patients’ hospitalisation duration was median 6.0 (IQR = 3.0–10.1) days.

Clinimetric performance of the sepsis CDS was established by assigning the 2620 patients into a confusion matrix (Table 1(a)), from which several accuracy metrics were then derived.

Table 1.

Response to two-stage CDS screen and stratification.

(a) Response to CDS Screen
Reality (N = 2620 patients)	Decision: suspected infection
	Microbiology culture and IV/PO antibiotics		No
CDS alert activated	304	(hit)	113	(miss)
CDS no alert	116	(false alarm)	2087	(correct rejection)
(b) Response to sepsis screening by provider
Reality (n = 417 patients)	Decision: suspected infection
	Microbiology culture and IV/PO antibiotics		No
Screened-in sepsis	246	(hit)	15	(miss)
Screen no sepsis	58	(false alarm)	98	(correct rejection)

Table 1(a) presents a 16% prevalence of sepsis when applying the blended ‘suspected infection’ gold standard and diagnosis code methodological approach. Approximately one in four (n = 116 of 420, 28%) patients had a sepsis diagnosis code documented, but not an activated alert, were considered simple sepsis because they remained below the threshold to activate an alert. Under this construct, the cloud-based sepsis CDS correctly classified 2391 of 2620 patients (91% correct classification) and accurately detected 304 of 420 patients (72% sensitivity) and appropriately rejected 2087 of 2200 patients (95% specificity). Activation was appropriate in 304 of 417 patients (73% positive predictive value) and absence of activation was appropriate in 2087 of 2203 patients (95% negative predictive value). Of special note, these metrics included about one in 11 (n = 39 of 417, 9%) patients with an activated alert (i.e. CDS screened-in), but did not have an order (or combination of orders) for lactic acid, or microbiology cultures, or antibiotics; these 39 (9%) patients screened-in by the CDS, but missing diagnostics and antibiotics, were designated as false positives.

Shifting the focus onto the cohort of 417 patients screened-in by the CDS, nearly nine in 10 (n = 370 of 417, 89%) patients were assessed and risk stratified by a provider, reflecting an 11% bypass rate. Clinical process and outcomes data for all 417 patients are reported in Table 2.

Table 2.

Sepsis screening and stratification by provider.

	Screening and stratification					Patients with an CDS alert	P
	SS/SS	Sepsis	SIRS	Not Sepsis	Bypass
	(n = 29) (%)	(n = 98) (%)	(n = 104) (%)	(n = 139) (%)	(n = 47) (%)	(N = 417)
Clinical process
Severe SIRS was first activated alert	83	43	38	38	26	41	.001
Lactic acid	100	97	87	50	55	74	.001
Microbiology cultures	100	100	94	62	83	84	.001
IV/PO antibiotics	100	99	90	50	68	77	.001
IV fluids	66	78	68	65	77	70	.21
Cultures and antibiotics	100	99	87	42	64	73	.001
ICU admission	83	24	31	32	57	36	.001
Documented sepsis diagnosis code	90	87	63	24	40	55	.001
Clinical outcomes
Expired or hospice	34	18	15	17	06	17	.004
Survivors were discharged to home	58	74	77	80	73	76	.30

SS/SS: severe sepsis/septic shock. Cultures and antibiotics: ‘Suspected Infection’ gold standard; severe SIRS alert includes ≥ 2 SIRS criteria and organ dysfunction.

The ensuing clinical process analysis showed that patients with a Severe SIRS alert (i.e. SIRS with Organ Dysfunction criteria) as the first activated alert were more likely to be stratified into the Severe Sepsis/Septic Shock risk group (p = .001). Stratification clearly illustrated patients with sepsis were resource intensive compared to sepsis rule-out; they received a significantly higher volume of diagnostic orders for lactic acid and microbiology cultures, and interventions of IV/PO antibiotics and ICU admission (p = .001). On a continuum, higher acuity sepsis patients were more likely to have supporting clinical documentation for Health Information Management to apply sepsis diagnosis codes (p = .001). Outcomes analysis showed patients with Severe Sepsis/Septic Shock compared to other patients with sepsis were nearly three times at greater risk of an adverse outcome (i.e. death or referred to hospice) (odds ratio = 2.6, 95% CI [1.1–6.1], p = .028), and among sepsis survivors they were 60% less likely to be discharged home (odds ratio = 0.4, 95% CI [0.2–1.2], p = .10).

Clinimetric performance of screening and stratification of patients with sepsis is presented in Table 1(b). First, 417 (16%) of 2620 patients were screened-in by the CDS. Second, applying ‘suspected infection’ gold standard reported in Table 2, all 417 patients screened-in by the CDS were mapped into a second confusion matrix (i.e. from Table 1(a) to (b)). Under this construct, screening and stratification by providers accurately detected 246 of 304 patients (81% sensitivity) and activation was appropriate in 246 of 261 patients (94% positive predictive value).

Regarding outcomes, 17% (n = 71 of 417) of patients screened-in by the CDS either expired or were discharged to hospice. Subgroup analysis identified 17% (n = 45 of 261) of patients screened-in and ruled-in either expired or were discharged to hospice, compared to 17% (n = 26 of 156) of patients screened-in by the CDS, but ruled-out by providers. This latter outcome was driven primarily by patients with sepsis rule-out but suspected of infection (n = 14 of 58, 24%) versus patients with sepsis rule-out and not suspected of infection (n = 12 of 98, 12%). Among patients without an activated CDS alert but with documentation of sepsis at discharge, 12% (n = 14 of 116) expired or were discharged to hospice.

In summary, the sepsis cohort comprised one in five (n = 533 of 2620, 20%) patients when including all hospitalisations, with one in six (n = 88 of 533, 17%) patients either expired or discharged to hospice. However, these patients accounted for one in two (n = 88 of 152, 58%) deaths or discharges to hospice among all 2620 hospitalised patients. The two-stage CDS for early recognition and stratification of patients with sepsis resulted in sepsis prevalence of 14% (n = 377 of 2620), comprised of 261 patients screened-in by the CDS and ruled-in by providers, and another 116 patients who did not activate an CDS alert but did have documentation of sepsis in their EHR. These (n = 377 of 2620, 14%) patients’ discharge disposition showed one in seven (n = 62 of 377, 16%) patients had either died in-hospital or discharged to hospice.

Discussion

Statement of principle findings

The sepsis programme increased awareness among providers and nursing, enabled by active surveillance and CDS to accurately screen-in patients at risk of sepsis; providers then conducted a near-immediate sepsis screening and stratification. Providers screened and risk stratified nine in 10 patients, and completed the assessment within one hour from alert activation for 75% of patients. By using the sepsis CDS in this manner, patients screened-in were successfully stratified into risk groups, with differentiated intensity of diagnostics and interventions. Clearly, ‘suspected infection’ gold standard is a reliable measure of provider decision based on a patient’s clinical indications and stratification, and also supports estimation of sepsis prevalence and outcomes.

Strengths and limitations

Study findings speak to the validity of the sepsis CDS alert definition for early recognition, and the fidelity of the sepsis screening and stratification protocol. Providers’ compliance with the CDS was likely predicated upon a positive association with the sepsis screening and stratification definitions because an alert demonstrating a high positive predictive value (PPV) increases user compliance.¹⁹ The initial alert activated on 16% of patients, with screening achieved 72% sensitivity and 73% PPV, and stratification by providers increased accuracy to 81% sensitivity and 94% PPV. Of the nine in 10 patients who were risk stratified, 91% had at least one diagnostic ordered or antibiotic administered. A substantial finding is that of the remaining 39 (one in 11, 9%) patients without orders, 37 of them were ruled out by a provider, while the other two patients were sepsis rule-in. Moreover, nearly all patients (n = 123 of 127, 97%) screened-in by the CDS and stratified into higher acuity sepsis had orders for lactic acid, microbiology cultures, and IV/PO antibiotics. To place this discussion in proper context, in a previously published study unaffiliated with this current study, with a sepsis CDS running in silent surveillance mode (i.e. alerts were not delivered to a provider), one in four patients recognised by the system did not have diagnostics resulted or antibiotics administered.¹⁵

There are some limitations to this study to consider. First, the setting was a single centre initiative at a 284-bed urban non-profit community hospital, which may not be generalisable to other clinical settings. Second, while the CDS was built on current evidence and good best practices for a binary alarm system with cross-check and validation functionality, the application was developed to promote broad adoption across the hospital’s provider and nursing groups; other health systems may have different circumstances. Third, the programme’s adoption by providers may not be fully known because the study began a few weeks after the CDS go-live date; some variance in usability and fidelity may exist because the sepsis programme enabled by the two-stage CDS was relatively new to providers. Fourth, the study design incorporated a retrospective analysis of cohort data after launch of the sepsis programme, which may have introduced some selection bias associated with real-world clinical practice and processes, temporal relationship of the sepsis programme’s maturation, and missing data in the patient EHR system (e.g. sepsis screening and stratification tool). Fifth, although the sepsis programme is grounded in current guidelines for recognition, assessment, and treatment of patients with sepsis, these guidelines may evolve over time and the sepsis programme should evolve too.

Interpretation of the findings in light of previous research

This study is unique because education regarding the ‘magic four’ instilled a simple concept that illuminated clinical events and processes key to saving lives (i.e. STAT lactic acid, obtain cultures prior to antibiotics, early administration of antibiotics, and early administration of fluids in patients with severe sepsis or septic shock). Realising ‘time is tissue’, the sepsis CDS was localised modestly to appeal to clinical workflow and achieve broad adoption. Cloud-based surveillance began when a patient’s initial vital signs or other diagnostics were resulted in their EHR, with continuous 24/7 monitoring of clinical results until discharge, thereby mitigating problems reported elsewhere.²⁰ Nearly all patients with an activated alert arrived via the emergency department, which supports findings from prior studies.²¹ The demographic profile of screened-in patients showed they were remarkably similar, which, coupled with expediency of prospective stratification by providers, minimised confounders to study findings. A number of patients with sepsis were admitted to the ICU. A majority of them, however, were managed in general medical/surgical units. Findings from the stratification accentuate this situation. A picture has emerged where surveillance can detect patients at risk of sepsis; diagnostic specificity is improved when the CDS includes postalert screening and stratification. A reasonable line of demarcation exists between patients with rule-out versus rule-in sepsis, severe sepsis, and septic shock. On a progressive continuum, higher acuity sepsis patients were clinically intense, more likely to require ICU services, and more likely to have a poor outcome. These findings on service intensity are consistent with other recent studies.^22,23

Implications for policy and practice

A sepsis programme enabled by cloud-based surveillance and computerised CDS can expedite accurate recognition and stratification of patients with sepsis, with goals of improving hospital survival and increasing the likelihood of discharge to home. Indeed, CDS systems provide a platform for discussion, debate, and innovation among sepsis programmes with an eye on improving patient outcomes. Moreover, sepsis is under-documented in the EHR and code assignment suffers. By adopting CDS, opportunities to improve revenue cycle and value-based programmes exist.

The CDS design incorporating a socio-technical systems approach for rule-in/out is consistent with a human factors design for a two-stage binary alarm system that activates an alert when threshold criteria are met, delivers notifications containing criterion variance, and includes an option to cross-check and validate the alert, as well as consider other pertinent criteria before responding with a medical decision. Leveraging existing human capital to improve outcomes, rather than adopting a rapid response team model, may also be possible by emphasising provider education; incorporating evidence-based CDS and surveillance alert notification functionality into the EHR; standardising nursing and provider workflows; mandating risk stratification; and if indicated, ordering of diagnostics, interventions, and consultation.

Direction for future research

An area of future research includes a focus on improving outcomes of patients within sepsis risk groups, as well as survivors at risk of early readmission because many unfortunate patients stricken with sepsis syndrome experience loss of productivity following hospitalisation.²⁴ A two-stage sepsis CDS as described in this study can be applied for demand planning and facilitate future study. Further research would add depth to our understanding of transition care, the care continuum, and patient health and wellness.

Declarations

Competing Interests

None declared

Funding

None declared

Ethical Approval

As this research was part of quality improvement activities, employed an observational cohort study design, and applied retrospective analysis of results of normal care and routine surveillance to detect sepsis among hospitalised patients, and individual patients cannot be recognised from this manuscript, the authors did not seek formal ethical approval.

Guarantor

RCA

Contributorship

RCA had the original idea for the study. RCA and JJL conducted the review and synthesis of the literature. RCA designed the study. JJL and JMH described the sepsis programme, while TLG and RCA described the two-stage sepsis clinical decision support (CDS) system. TLG managed the source data infrastructure. RCA built the analytic database and performed the statistical analyses. JJL and JMH provided clinical expertise during the interpretation of results written by RCA. JJL, TLG, and JMH in conjunction with RCA co-authored and approved the final manuscript.

Acknowledgements

We appreciate the help of Jeremy Wolford who was responsible for extracting and compiling transaction-level clinical event data into encounter-level analytic datasets.

Provenance

Not commissioned; peer-reviewed by Peter Wilson