Abstract
The 2016 Sepsis-3 guidelines included the Quick Sequential [Sepsis-related] Organ Failure Assessment (qSOFA) tool to identify patients at risk of sepsis. The objective was to compare the utility of qSOFA to the St. John Sepsis Surveillance Agent among patients with suspected infection. The primary outcomes were in-hospital mortality or admission to the intensive care unit. A multiple center observational cohort study design was used. The study population comprised 17 044 hospitalized patients between January and March 2016. For the primary analysis, receiver operator characteristic curves were constructed for patient outcomes using qSOFA and the St. John Sepsis Surveillance Agent, and the areas under the curve were compared against a baseline risk model. Time-to-event clinical process modeling also was applied. The St. John Sepsis Surveillance Agent, when compared to qSOFA, activated earlier and was more accurate in predicting patient outcomes; in this regard, qSOFA fell far behind on both objectives.
Keywords: sepsis surveillance, Quick SOFA (qSOFA), St John Sepsis Surveillance Agent, sepsis clinical decision support, early recognition of sepsis
A spectrum of sepsis exists and mortality risk increases with severity,1,2 with approximately 1 in 2 all-cause in-hospital deaths being associated with sepsis.3 Although patients at risk of sepsis are more likely to be recognized and admitted to the hospital by emergency medicine providers, a disproportionate mortality risk exists for inpatients.4 Historic consensus definitions of sepsis include systemic inflammatory response syndrome (SIRS) as a causal factor of serious infection.5-7 Clinical results derived from routine diagnostics and vital signs illustrate complexities toward diagnosing sepsis because nearly half of hospitalized patients experience SIRS at some point during their hospitalization.8 In contrast, approximately 1 in 8 patients diagnosed with severe sepsis in the intensive care unit (ICU) may not have SIRS indicated (ie, 88% sensitivity).9 To alleviate this inherent tension, deploying reliable sepsis surveillance with integrated clinical decision support (CDS) capabilities may help improve accuracy and timeliness of detecting patients at risk of sepsis.10-13
The 2016 consensus guideline on sepsis (Sepsis-3)14 includes a new sepsis surveillance alert definition termed Quick Sequential [Sepsis-related] Organ Failure Assessment (qSOFA). The purpose of qSOFA is to identify non-ICU patients currently under suspicion of infection who could be deteriorating into a sepsis complication, and to notify providers of this clinical event. The qSOFA algorithm differentiates from earlier consensus guidelines, such that it incorporates latency and places limitations on the use case by narrowing clinical indications to tachypnea, hypotension, and altered mentation within the immediate 24 hours after onset of infection.15
Critiques of the qSOFA model for sepsis surveillance, when compared to SIRS, emphasize problems with it potentially activating later in the disease process, and its perceived clinimetric performance trade-off being overly tilted toward increasing specificity at the expense of degrading sensitivity.16 Findings from a recent study showed qSOFA, when compared to an alert defined by only SIRS ≥ 2 criteria, was more predictive of in-hospital mortality or ICU admission, but less accurate in predicting these similar patient outcomes when compared to other more clinically comprehensive early warning tools such as the Modified Early Warning Score and the National Early Warning Score (NEWS).17 Because that particular study did not include the St. John Sepsis Surveillance Agent, which provides 24/7 monitoring and CDS to more than 550 hospitals in the United States, the objectives of the present study were 3-fold: (1) to establish an incidence rate of qSOFA clinical events and compare it to St. John Sepsis Surveillance Agent activation; (2) to better understand the temporal relationship between onset of infection and qSOFA trigger versus the onset of infection and St. John Sepsis Surveillance Agent activation; and (3) to estimate the clinimetric performance of qSOFA and St. John Sepsis Surveillance Agent predictive models on in-hospital mortality and ICU admission outcomes, and the potential reclassification benefit these models may offer.
Methods
Patients and Data Collection
This multiple center retrospective cohort study was performed at 8 different medical centers in 2 distinct geographic regions in the United States (see Supplemental Tables S1a and S1b, available online with this article). All facilities had an enterprise electronic health record system (Millennium: Cerner Corporation, Kansas City, Missouri) and cloud-based sepsis surveillance and CDS (St. John Sepsis Surveillance Agent: Cerner Corporation, Kansas City, Missouri).18,19 Data were included for adults (≥18 years of age) who were hospitalized between January 2016 and March 2016 and met the definition of Sepsis-3 suspected infection.15 The US Department of Health and Human Services’ Office for Human Research Protections clarified that quality improvement activities, described herein, often qualify for institutional review board exemption and do not require individual informed consent.20
The hospitals’ sepsis programs were designed to improve early recognition and early intervention of patients at risk of sepsis. The sepsis protocol was guided by the Surviving Sepsis Campaign resuscitation and management bundles.21 The St. John Sepsis Surveillance Agent applied a binary alarm paradigm with 2 alert definitions: (1) indications of SIRS (proxy for sepsis) and (2) indications of sepsis (proxy for severe sepsis).18 Unless suppressed by a localized rule, each alert delivered a real-time notification to a provider that included specific clinical criteria responsible for activation.19 Subsequently, the study team developed the analytic data set with a qSOFA flag and its trigger time stamp, and a suspected infection clinical event with its onset time stamp. These 2 events were joined by examining their respective time stamps to create a qSOFA clinical event. Consistent with Sepsis-3, the qSOFA clinical event was suppressed if the qSOFA triggered before onset of suspected infection, after 24 hours from the onset of suspected infection, or after ICU admission.15 The first alert for each patient was studied for each of the 2 surveillance models (ie, qSOFA or St. John Sepsis Surveillance Agent).
Definitions
The primary outcome of the study was in-hospital mortality, and the secondary outcome was the composite of death or ICU admission. The composite outcome metric was established because critical care interventions may have a moderating influence on patient survival. Time to clinical event process metrics included onset of suspected infection to the primary or secondary outcome, alert activation to onset of suspected infection, onset of suspected infection to alert activation, and alert activation to the primary or secondary outcome.
Suspected infection was defined from the original Sepsis-3 qSOFA study,15 in which suspicion of infection was defined as either a microbiology culture drawn time stamp followed by an intravenous anti-infective antibiotic administration time stamp within 72 hours or an administered intravenous anti-infective antibiotic followed by a drawn microbiology culture within 24 hours. The onset of suspected infection was denoted as the time stamp of the first clinical event (ie, culture drawn or antibiotic given). The onset of suspected infection was linked to patient location at that time, and categorized as onset in the emergency department (ED), inpatient care units, or ICU. Microbiology cultures drawn included blood, body fluid, bronchial, catheter tip, cerebrospinal fluid, fungal, ova and parasites, sputum, stool, tissue, urine, and wound. Anti-infective antibiotics administered included ampicillin-sulbactam, azithromycin, cefepime, ceftriaxone, ciprofloxacin, clindamycin, fluconazole, fluticasone-salmetrerol, levofloxacin, meropenem, piperacillin-tazobactam, and vancomycin.
The qSOFA flag was defined from the original Sepsis-3 qSOFA study15 and was indicated when ≥2 criteria were present: systolic blood pressure ≤100 mm Hg, respiratory rate ≥22 breaths per minute, and Glasgow Coma Scale score <15 (see Supplemental Table S2, available online with this article). The qSOFA clinical event occurred when the second qSOFA criteria time stamp was documented within 24 hours from onset of suspected infection. The qSOFA flag was otherwise suppressed if the second criterion time stamp was documented either before onset of suspected infection or after ICU admission.
The threshold for activating the St. John Sepsis Surveillance Agent “SIRS alert” was established when ≥3 of the following 5 criteria were satisfied: (1) temperature >38.3°C or <36°C; (2) heart rate >90 beats/min; (3) respiratory rate >20 breaths/min; (4) white blood cell count >12 000 cells/mm3, or <4000 cells/mm3, or >10% immature (band) forms; or (5) glucose 140 to 200 mg/dL (see Supplemental Table S2, available online with this article). The threshold for activating the St. John Sepsis Surveillance Agent “Sepsis alert” was established when ≥2 SIRS criteria were present, and ≥1 of the following 4 organ system dysfunction criteria were satisfied: (1) cardiovascular system: systolic blood pressure <90 mm Hg and/or mean arterial pressure <65 mm Hg; (2) tissue perfusion: serum lactate >2.0 mmol/L; (3) hepatic system: total bilirubin: ≥2.0 mg/dL and <10.0 mg/dL; and (4) renal system: serum creatinine: Δ↑0.5 mg/dL from baseline (see Supplemental Table S2, available online with this article). A look-back period consisted of 12 hours for serum lactate, 30 hours for the other criteria, and 72 hours for Δ↑ serum creatinine.
Recent discharge was established when a patient was previously discharged from the same hospital within 30 days of current admission date.
Statistical Analysis
Data were analyzed retrospectively. Patient characteristics were compared between those first meeting the suspected infection definition in the ED, inpatient care units, or ICU. Unadjusted bivariate analyses applied Fisher exact test and χ2 test for dichotomous variables, respectively. Mann-Whitney U test for independent and related samples was applied to estimate differences in medians and distributions. Hierarchical multivariable logistic regression was used to identify risk factors of the primary and secondary outcomes. Three predictive models on the primary and secondary outcomes were established: first, the baseline risk model included 3 patient variables (block 1: age, male sex, and recent discharge from hospital), and 2 hospital variables (block 2: onset of infection in the ED and admitting facility); second, the qSOFA risk model incorporated the qSOFA clinical event into block 1 along with the other 5 variables included in the base model; and third, the St. John Sepsis Surveillance Agent risk model incorporated the first SIRS or first Sepsis alerts into block 1 along with the other 5 variables that comprised the baseline model. Accuracy comparisons were performed using the area under the receiver operating characteristic (AUROC) curve to discriminate baseline risk, qSOFA, and St. John Sepsis Surveillance Agent models on the primary and secondary outcomes. AUROC output tables containing positive and negative predictive values were analyzed to potentially reclassify patients when holding specificity constant. The Hanley-McNeil method for comparing the area under the curve (AUC) derived from the 3 models’ same cases was applied.22 The Kaplan-Meier survival model framework (ie, time to event estimator) was applied to measure the proportion of patients experiencing the composite outcome in one-hour increments after detection by each alert type (ie, qSOFA clinical event, St. John Sepsis Surveillance Agent “SIRS alert,” or St. John Sepsis Surveillance Agent “Sepsis alert”). Cases were censored at 96.0 hours after detection. The log-rank (Mantel-Cox) statistic reported the differences in survival distributions between the 3 alert types mentioned above. A 2-tailed P value <.05 was considered statistically significant. All analyses were conducted using SPSS v23 (IBM Corp., Armonk, New York).
Results
Of the 17 044 hospitalizations encompassing 68 964 patient days examined, 5992 (35%) patients met the definition of suspected infection, corresponding to 87 patients per 1000 patient days [(5992/68 964) × 1000]. The patient flow diagram (see Supplemental Figure S1, available online with this article) illustrates that approximately two thirds (n = 3 808, 64%) of patients’ onset of suspected infection occurred while in the ED, 33% (n = 1 967) of patients’ onset of suspected infection occurred after being admitted to inpatient care, and 4% (n = 217) of patients’ onset of suspected infection occurred after ICU admission (see Supplemental Table S3, available online with this article). Characteristics of the 5992 patients with suspected infection and screened-in by either the St. John Sepsis Surveillance Agent or qSOFA are shown in Table 1.
Table 1.
Alert Type by Patient Characteristics.a
| Characteristic | No SJSA Alert | SJSA Alert | No qSOFA | qSOFA |
|---|---|---|---|---|
| Patients with suspected infection (N = 5992), n (%) | 3764 (63) | 2228 (37) | 5343 (89) | 649 (11) |
| Demographics | ||||
| Age, median (IQR), years | 65 (51-77) | 64 (51-75) | 65 (51-76) | 66 (53-77) |
| Female sex, n (%) | 2046 (54) | 1068 (48) | 2797 (52) | 317 (49) |
| Emergency admit, n (%) | 2657 (71) | 1940 (87) | 4059 (76) | 538 (83) |
| Readmission (<30 days), n (%) | 438 (12) | 197 (9) | 555 (10) | 80 (12) |
| Patient location at onset of infection | ||||
| Emergency department, n (%) | 2131 (57) | 1677 (75) | 3319 (62) | 489 (75) |
| Inpatient unit, n (%) | 1471 (39) | 496 (22) | 1807 (34) | 160 (25) |
| ICU, n (%) | 162 (4) | 55 (3) | 217 (4) | 0 (0) |
| Clinical outcomes | ||||
| Transferred to ICU, n (%) | 469 (13) | 536 (24) | 874 (16) | 131 (20) |
| In-hospital mortality, n (%) | 81 (2) | 144 (7) | 170 (3) | 55 (9) |
| Composite outcome, n (%) | 502 (13) | 598 (27) | 937 (18) | 163 (25) |
| LOS, median (IQR), days | 4 (2-7) | 4 (3-7) | 4 (2-7) | 5(3-8) |
Abbreviations: ICU, intensive care unit; IQR, interquartile range; LOS, length of stay; qSOFA, Quick Sequential [Sepsis-related] Organ Failure Assessment; SJSA, St. John Sepsis Surveillance Agent.
Emergency admit indicates patient admission source was the emergency department. Readmission (<30 days) indicates patient was discharged within previous 30 days and now admitted to the same hospital.
Figure 1 illustrates the number of hours before or after sepsis surveillance detection to the onset of suspected infection by alert type. Exactly 1 in 10 (n = 591 of 5992, 10%) patients activated the St. John Sepsis Surveillance Agent before onset of suspected infection, median 1.13 (interquartile range [IQR] = 0.43 to 4.00) hours. In contrast, about 1 in 4 (n = 1637, 27%) patients with suspected infection activated the St. John Sepsis Surveillance Agent after onset, median 0.97 (IQR = 0.55 to 3.63) hours, compared to 1 in 9 (n = 649, 11%) patients who experienced a qSOFA clinical event after onset, median 4.82 (IQR = 1.98 to 11.15) hours (P < .001).
Figure 1.
Incidence and timing of alert activation.
Abbreviation: qSOFA, Quick Sequential [Sepsis-related] Organ Failure Assessment.
As a proportion of patients with suspected infection (n = 5992), a factor of 3 to 4 times more patients (P < .001) activated a St. John Sepsis Surveillance Agent and experienced in-hospital mortality (n = 144, 2.4%) or the composite outcome (n = 598, 9.8%), when compared to qSOFA clinical event in-hospital mortality (n = 55, 0.9%) or composite outcome (n = 163, 2.7%). Figure 2 illustrates the predictive characteristics of the 3 different models (ie, baseline risk, qSOFA + baseline, and St. John Sepsis Surveillance Agent + baseline) on in-hospital mortality and the composite outcome. Presented on the left side of Figure 2 is the baseline risk model on mortality (AUC = .67, 95% confidence interval [CI] = .63 to .70), the qSOFA model (AUC = .69, 95% CI = .66 to .73), and the St. John Sepsis Surveillance Agent model (AUC = .74, 95% CI = .71 to .77) (P < .001). In analysis of each model’s positive and negative predictive values derived from the AUROC result output tables, holding specificity constant, the St. John Sepsis Surveillance Agent and qSOFA correctly reclassified, in absolute terms, 13% and 5% of patients, respectively, from the baseline risk model. This reclassification translates to a relative improvement in sensitivity of Δ↑ 23% for St. John Sepsis Surveillance Agent and Δ↑ 9% for qSOFA clinical event from the estimated baseline risk.
Figure 2.
Overall test performance on patient outcomes.
Abbreviation: qSOFA, Quick Sequential [Sepsis-related] Organ Failure Assessment.
In contrast to the predictive mortality model, the right side of Figure 2 illustrates the 3 predictive models on the composite outcome (P < .001). The baseline risk model (AUC = .65, 95% CI = .64 to .67), the qSOFA model (AUC = .67, 95% CI = .65 to .68), and the St. John Sepsis Surveillance Agent model (AUC = .72, 95% CI = .71 to .74). In analysis of positive and negative predictive values for each model derived from the AUROC result tables, holding specificity constant, the St. John Sepsis Surveillance Agent and qSOFA correctly reclassified, in absolute terms, 15% and 2% patients, respectively, from the baseline risk model. This reclassification translates into a relative improvement in sensitivity of Δ↑ 31% for St. John Sepsis Surveillance Agent and Δ↑ 4% for qSOFA from the estimated baseline risk.
The increased sensitivity corresponds to the St. John Sepsis Surveillance Agent having screened-in an additional 22.9 patients per 1000 patient days, where the screen-in rate for St. John Sepsis Surveillance Agent compared to qSOFA was 32.3 patients per 1000 patient days = [(2228/68 964 days) × 1000] for the former, and 9.4 patients per 1000 patient days = [(649/68 964 days) × 1000] for the latter, respectively.
Four separate hierarchical multivariable logistic regression models were applied to predict the effects of qSOFA and St. John Sepsis Surveillance Agent on mortality and composite outcomes, after controlling for other risk factors (Table 2). The first 2 models incorporated qSOFA and St. John Sepsis Surveillance Agent on mortality, while the latter 2 models targeted the composite outcome. Each model demonstrates increased odds of experiencing an adverse outcome associated with qSOFA and St. John Sepsis Surveillance Agent. Essentially, the 4 models illustrate that patients with suspected infection who activated either the qSOFA or St. John Sepsis Surveillance Agent “SIRS alert” were similar in their likelihood of experiencing an adverse outcome of in-hospital mortality and composite outcome (Table 2). In contrast, patients who activated the St. John Sepsis Surveillance Agent “Sepsis alert” compared to qSOFA were 87% more likely to experience in-hospital mortality and 66% more likely to experience the composite outcome (Table 2).
Table 2.
| Patients With Suspected Infection (N = 5992) | Unadjusted Odds Ratio | Adjusted Odds Ratio (95% Confidence Interval) |
|---|---|---|
| Model 1: Mortality outcome | ||
| Age | 1.02 | 1.03 (1.02-1.04) |
| Male sex | 1.69 | 1.73 (1.31-2.28) |
| Readmission (<30 days) | 1.88 | 1.81 (1.26-2.59) |
| qSOFA | 2.82 | 2.49 (1.79-3.45) |
| Model 2: Mortality outcome | ||
| Age | 1.02 | 1.03 (1.02-1.04) |
| Male sex | 1.69 | 1.65 (1.25-2.18) |
| Readmission (<30 days) | 1.88 | 1.99 (1.38-2.86) |
| SJSA “Sepsis alert” | 3.40 | 4.65 (3.35-6.46) |
| SJSA “SIRS alert” | 1.36 | 2.36 (1.67-3.35) |
| Model 3: Composite outcome | ||
| Age | 1.01 | 1.01 (1.00-1.01) |
| Male sex | 1.26 | 1.27 (1.11-1.46) |
| Readmission (<30 days) | 1.31 | 1.24 (1.01-1.53) |
| qSOFA | 1.58 | 2.10 (1.70-2.58) |
| Model 4: Composite outcome | ||
| Age | 1.01 | 1.01 (1.00-1.01) |
| Male sex | 1.26 | 1.21 (1.05-1.39) |
| Readmission (<30 days) | 1.31 | 1.32 (1.07-1.63) |
| SJSA “Sepsis alert” | 2.64 | 3.49 (2.93-4.16) |
| SJSA “SIRS alert” | 1.40 | 2.19 (1.85-2.59) |
Abbreviations: CI, confidence interval; qSOFA, Quick Sequential [Sepsis-related] Organ Failure Assessment; SIRS, systemic inflammatory response syndrome; SJSA, St. John Sepsis Surveillance Agent.
Model performance:
Model 1: C-statistic = 0.69, 95% CI (0.66 to 0.73).
Model 2: C-statistic = 0.74, 95% CI (0.71 to 0.77).
Model 3: C-statistic = 0.65, 95% CI (0.64 to 0.67).
Model 4: C-statistic = 0.72, 95% CI (0.71 to 0.74).
Age in one-year increments. Readmission (<30 days) indicates patient was discharged within previous 30 days and now admitted to the same hospital.
Applying the Kaplan-Meier survival framework, Figure 3 illustrates the number of hours elapsed after triggering the St. John Sepsis Surveillance Agent or a qSOFA clinical event to the cumulative composite outcome, trimmed at 96 hours after the alert. A difference in estimated survival curves existed between the 3 patient cohorts (P = .026). Time-to-event measures with cases censored at 96 hours after detection showed the cohort with the St. John Sepsis Surveillance Agent “Sepsis alert” (n = 306 of 930, 33%) median 5.73 (IQR = 4.88 to 6.58) hours after detection, and approximately 1 in 17 (n = 17 of 306, 6%) patients were censored; compared to the cohort with a qSOFA clinical event (n = 163 of 649, 25%) median 5.30 (IQR = 3.46 to 7.14) hours after detection, and there were twice the rate of censored cases (n = 19 of 163, 12%); and last, the cohort with a St. John Sepsis Surveillance Agent “SIRS alert” (n = 292 of 1 298, 23%) median 6.35 (IQR = 4.68 to 8.02) hours after detection, and there was a similar rate of censored cases (n = 37 of 292, 13%) as qSOFA.
Figure 3.
Time after alert to composite outcome.
Abbreviation: qSOFA, Quick Sequential [Sepsis-related] Organ Failure Assessment; SIRS, systemic inflammatory response syndrome.
Discussion
This multiple center study of patients with suspected infection found the incidence of St. John Sepsis Surveillance Agent, when compared to qSOFA, to be 3-fold higher for the same patient cohort. Not only was the qSOFA rate substantially lower, but the elapsed time after onset of suspected infection took the majority of patients beyond the immediate 3-hour opportunity-risk window. In contrast, patients who activated the St. John Sepsis Surveillance Agent after onset of suspected infection, when compared to qSOFA, did so much earlier, and a sizable proportion of patients actually triggered the St. John Sepsis Surveillance Agent prior to the onset of suspected infection. Furthermore, 1 in 2 patients who experienced the composite outcome (ie, death or ICU admission) had previously activated the St. John Sepsis Surveillance Agent, whereas only 1 in 7 patients had a qSOFA clinical event. In addition, the St. John Sepsis Surveillance Agent model, when compared to the qSOFA model, was more accurate in predicting mortality and ICU admission outcomes, resulting in greater reclassification benefit from baseline risk. Last, the time-to-event analysis showed patients with a St. John Sepsis Surveillance Agent “Sepsis alert” were more likely to experience an accelerated composite outcome, when compared to qSOFA or the St. John Sepsis Surveillance Agent “SIRS alert.”
The study team sought balance between clinical precision of detection and timing of alert activation. The St. John Sepsis Surveillance Agent is perhaps unique in striking this balance. Despite the initial focus on early recognition of sepsis, from the very onset of this work one of the most important factors has been to identify patients at risk of death, irrespective of the final diagnosis of sepsis. The team aimed to bring appropriate clinical attention to these vulnerable patients, and believe the St. John Sepsis Surveillance Agent met these objective criteria. To place the present study in proper context, the study team reviewed the original qSOFA study conducted at the University of Pittsburgh Medical Center (UPMC)15 and a similar, but independent, study conducted at the University of Chicago Medical Center (UCMC).17 All 3 studies applied a retrospective observational cohort study design to examine the validity of qSOFA on detecting patients at risk of sepsis, and analyzed the qSOFA model’s clinimetric performance on patient outcomes. The UPMC observation period encompassed 2010 through 2012, nested within the UCMC observation period spanning late 2008 through January 2016. On the other hand, the present study’s observation period was more recent and applied a tighter time horizon (January 2016 through March 2016). Thus, the study team anticipated differences in clinical process and outcomes metrics between these 3 studies, notably because awareness of sepsis is increasing leading to earlier treatment,23 surveillance strategies as a component of sepsis performance improvement initiatives are being adopted,24 and evidence-based sepsis management programs are being disseminated.25 Indeed, the team found a 3-fold difference in prevalence of suspected infection between UPMC (10%), UCMC (20%), and the present study (34%), with a bimodal incidence of qSOFA among them: UPMC (17%) versus UCMC (9%) and the present study (11%); a noteworthy difference is the concentration of high-risk patients among these 3 studies. Interestingly, discrimination characteristics of qSOFA on in-hospital mortality showed the following: UPMC (AUC = .81, 95% CI = .80 to .82) versus UCMC (AUC = .69, 95% CI = .67 to .70), mirroring the present study (AUC = .69, 95% CI = .66 to .73). Last, although UPMC did not include a composite outcome measure similar to UCMC and the present study, the UCMC reported clinimetric characteristic of the NEWS on the composite outcome was similar to the present study’s reported St. John Sepsis Surveillance Agent, with NEWS (AUC = .73, 95% CI = .70 to .75) and the St. John Sepsis Surveillance Agent (AUC =.72, 95% CI = .71 to .74), respectively.
There are some limitations to this study to consider. First, although this was a multiple center observational cohort study involving 8 hospitals located across 2 different geographic regions in the United States, the results may not be generalizable to other clinical settings. Second, despite the lack of a gold standard for the definition of suspected infection, the study team followed the original qSOFA study definition15 to reduce potential selection bias while establishing the target patient cohort. Third, the study design incorporated a retrospective analysis of cohort data beginning almost 5 months after launch of the hospitals’ sepsis management programs, which may have introduced some selection bias associated with real-world clinical practice and processes.
An area of future research includes examining the utility of a multitiered surveillance system, which couples additional detection systems to the St. John Sepsis Surveillance Agent. The study team is particularly interested in prospectively testing a NEWS alert with the St. John Sepsis Surveillance Agent, particularly among identified patients but not screened-in at that specific time by a provider at bedside. Moreover, the team would like to gain insights into the incidence of a NEWS in the absence of St. John Sepsis Surveillance Agent activation, and the associated clinical process and impact outcomes.
In comparing earlier consensus definitions of sepsis with the 2016 Sepsis-3 qSOFA, this study found the earlier definitions more robust for surveillance systems. The St. John Sepsis Surveillance Agent, when compared to qSOFA, detected more high-risk patients with suspected infection; activated much earlier in their infectious process; and was more accurate in predicting mortality and ICU admission outcomes. Considering the consequences of missing patients or inadvertent delay, qSOFA fell far behind on these objectives while the St. John Sepsis Surveillance Agent was promising.
Supplementary Material
Footnotes
Declaration of Conflicting Interests: The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Drs Amland and Sutariya are employed by Cerner Corporation, developer of the Millennium electronic health record system and the St. John Sepsis Surveillance Agent system.
Funding: The authors received no financial support for the research, authorship, and/or publication of this article.
Supplemental Material: Supplemental Tables S1-S3 and Supplemental Figure S1 are available online with this article.
References
- 1. ProCESS Investigators;Yealy DM, Kellum JA, Huang DT, et al. A randomized trial of protocol-based care for early septic shock. N Engl J Med. 2014;370:1683-1693. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. ARISE Investigators; ANZICS Clinical Trials Group,Peake SL, Delaney A, Bailey M, et al. Goal-directed resuscitation for patients with early septic shock. N Engl J Med. 2014;371:1496-1506. [DOI] [PubMed] [Google Scholar]
- 3. Liu V, Escobar GJ, Greene JD, et al. Hospital deaths in patients with sepsis from 2 independent cohorts. JAMA. 2014;212:90-92. [DOI] [PubMed] [Google Scholar]
- 4. Levy MM, Rhodes A, Phillips GS, et al. Surviving Sepsis Campaign: Association between performance metrics and outcomes in 7.5 year study. Intensive Care Med. 2014;40:1623-1633. [DOI] [PubMed] [Google Scholar]
- 5. Bone RC, Balk RA, Cerra FB, et al. American College of Chest Physicians/Society of Critical Care Medicine Consensus Conference: definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. Crit Care Med. 1992;20:864-874. [PubMed] [Google Scholar]
- 6. Levy MM, Fink MP, Marshall JC, et al. 2001 SCCM/ESICM/ACCP/ATS/SIS International Sepsis Definitions Conference. Intensive Care Med. 2003;29:530-538. [DOI] [PubMed] [Google Scholar]
- 7. Dellinger RP, Levy MM, Rhodes A, et al. Surviving Sepsis Campaign: international guidelines for management of severe sepsis and septic shock: 2012. Crit Care Med. 2013;41:580-637. [DOI] [PubMed] [Google Scholar]
- 8. Churpek MM, Zadravecz FJ, Winslow C, Howell MD, Edelson DP. Incidence and prognostic value of the systemic inflammatory response syndrome and organ dysfunctions in ward patients. Am J Resp Crit Care Med. 2015;192:958-964. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Kaukonen KM, Bailey M, Pilcher D, Cooper J, Bellomo R. Systemic inflammatory response syndrome criteria in defining severe sepsis. N Engl J Med. 2015;372:1629-1638. [DOI] [PubMed] [Google Scholar]
- 10. Umsheid CA, Betesh J, VanZandbergen C, et al. Development, implementation, and impact of an automated early warning and response system for sepsis. J Hosp Med. 2015;10:26-31. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Amland RC, Haley JM, Lyons JJ. A multidisciplinary sepsis program enabled by a two-stage clinical decision support system: factors that influence patient outcomes. Am J Med Qual. 2016;31:501-508. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Khurana HS, Groves RH, Simons MP. Real-time automated sampling of electronic medical records predicts hospital mortality. Am J Med. 2016;129:688-698. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Manaktala S, Claypool SR. Evaluating the impact of a computerized surveillance algorithm and decision support system on sepsis mortality. J Am Med Inform Assoc. 2017;24:88-95. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Singer M, Deutschman CS, Seymour CW, et al. The third international consensus definitions for sepsis and septic shock (Sepsis-3). JAMA. 2016;315:801-810. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Seymour CW, Liu VX, Iwashyna TJ, et al. Assessment of clinical criteria for sepsis for the third international consensus definitions for sepsis and septic shock (Sepsis-3). JAMA. 2016;315:762-774. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Simpson SQ. New sepsis criteria: a change we should not make. Chest. 2016;149:1117-1118. [DOI] [PubMed] [Google Scholar]
- 17. Churpek MM, Snyder A, Han X, et al. qSOFA, SIRS, and early warning scores for detecting clinical deterioration in infected patients outside the ICU [published online September 20, 2016]. Am J Resp Crit Care Med. doi: 10.1164/rccm.201604-0854OC. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Amland RC, Hahn-Cover KE. Clinical decision support for early recognition of sepsis. Am J Med Qual. 2016;31:103-110. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Amland RC, Lyons JJ, Greene TL, Haley JM. A two-stage clinical decision support system for early recognition and stratification of patients with sepsis: an observational cohort study. J R Soc Med Open. 2015;6(10):1-10. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. US Department of Health and Human Services, Office for Human Research Protections. Frequently asked questions. http://www.hhs.gov/ohrp/regulations-and-policy/guidance/faq/index.html. Accessed April 4, 2016.
- 21. Dellinger RP, Levy MM, Carlet JM, et al. Surviving Sepsis Campaign: international guidelines for management of severe sepsis and septic shock: 2008. Crit Care Med. 2008;34:17-60. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. Hanley JA, McNeil BJ. Method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology. 1983;148:839-843. [DOI] [PubMed] [Google Scholar]
- 23. Rhee C, Kadri S, Huang SS, et al. Objective sepsis surveillance using electronic clinical data. Infect Control Hosp Epidemiol. 2016;37:163-171. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. Dellinger RP. The future of sepsis performance improvement. Crit Care Med. 2015;43:1787-1789. [DOI] [PubMed] [Google Scholar]
- 25. Rhee C, Murphy MV, Li L, Platt R, Klompas M. Comparison of trends in sepsis incidence and coding using administrative claims versus objective clinical data. Clin Infect Dis. 2015;60:88-95. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.