Abstract
Background
Quick Sepsis‐related Organ Failure Assessment (qSOFA) is recommended for use by the most recent international sepsis definition taskforce to identify suspected sepsis in patients outside the intensive care unit (ICU) at risk of adverse outcomes. Evidence of its comparative effectiveness with existing sepsis recognition tools is important to guide decisions about its widespread implementation.
Aim
To compare the performance of qSOFA with the adult sepsis pathway (ASP), a current sepsis recognition tool widely used in NSW hospitals and systemic inflammatory response syndrome criteria in predicting adverse outcomes in adult patients on general wards.
Methods
A retrospective observational cohort study was conducted which included all adults with suspected infections admitted to a Sydney teaching hospital between December 2014 and June 2016. The primary outcome was in‐hospital mortality with two secondary composite outcomes.
Results
Among 2940 patients with suspected infection, 217 (7.38%) died in‐hospital and 702 (23.88%) were subsequently admitted to ICU. The ASP showed the greatest ability to correctly discriminate in‐hospital mortality and secondary outcomes. The area under the receiver‐operating characteristic curve for mortality was 0.76 (95% confidence interval (CI): 0.74–0.78), compared to 0.64 for the qSOFA tool (95% CI: 0.61–0.67, P < 0.0001). Median time from the first ASP sepsis warning to death was 8.21 days (interquartile range (IQR): 2.29–16.75) while it was 0 days for qSOFA (IQR: 0–2.58).
Conclusions
The ASP demonstrated both greater prognostic accuracy and earlier warning for in‐hospital mortality for adults on hospital wards compared to qSOFA. Hospitals already using ASP may not benefit from switching to the qSOFA tool.
Keywords: sepsis, quick Sepsis‐related Organ Failure Assessment, hospital mortality, early detection of disease, adult sepsis pathway, systemic inflammatory response syndrome
Introduction
Sepsis, defined as ‘life‐threatening organ dysfunction caused by a dysregulated host response to infection’, 1 is estimated to affect 20–30 million patients each year worldwide. 2 Despite advances in vaccines, antibiotics and acute care, sepsis remains the primary cause of death from infection. 1 Between 30 and 50% of patients with sepsis die. 3 , 4 Early recognition allows for prompt treatment, which has been shown to be associated with reduced mortality. 5 Sepsis recognition tools can play an important role in facilitating early sepsis diagnosis and initiation of treatment.
The Third International Consensus Definition for Sepsis and Septic Shock (SEPSIS‐3) was published in 2016. 1 , 6 The task force for SEPSIS‐3 advocated the quick Sequential Organ Failure Assessment (qSOFA) score to be used as a tool to recognise adult patients with suspected sepsis at risk of adverse outcomes outside the intensive care unit (ICU). 1 , 6 These patients can be rapidly identified if they have an infection and at least two of the following three clinical criteria: respiratory rate of 22/min or greater, altered mentation, or systolic blood pressure of 100 mmHg or less, which places them at particular risk of adverse outcomes, including prolonged ICU stay and mortality. 6 The use of the qSOFA is a significant departure from the older systemic inflammatory response syndrome (SIRS) criteria, introduced in the first two international sepsis consensus definitions. 7 , 8 Several sepsis recognition tools have been developed based on the SIRS criteria and are currently in use in many countries.
The adult sepsis pathway (ASP) 9 , 10 is one of the most widely implemented sepsis recognition tools in Australia. The ASP tool was developed based on criteria used within the NSW deteriorating patient programme (Between the Flags) for recognising and responding to a deteriorating patient. 11 , 12 It is currently used in more than 180 hospitals in New South Wales to assist recognition of suspected sepsis cases among adult patients, and to facilitate treatment and appropriate referral to specialist teams. 10 Robust assessment of the performance of the newer qSOFA tool in predicting adverse outcomes of sepsis patients in comparison with existing sepsis recognition tools, such as the ASP, is important in determining its suitability for widespread adoption.
The performance of the sepsis recognition tools, including the newer qSOFA tool, may vary by clinical setting. While the research attention has focussed on the early identification and treatment of patients with sepsis in emergency departments (ED) and ICU, many patients admitted to hospital general care wards acquire sepsis or deteriorate after initial ED diagnosis and treatment. 13 Previous studies indicate that only 32% to 50% of patients with sepsis require ICU care, and the majority of patients with sepsis are not transferred to an ICU but cared for on general wards. 14 , 15 , 16 Limited evidence exists about the patient outcomes associated with using sepsis recognition tools on general wards. We conducted a retrospective analysis of a large cohort of general ward patients. Our aim was to compare the performance of qSOFA, with the ASP and SIRS recognition criteria, in predicting mortality and extended ICU length of stay. We further aimed to compare differences in the timing of sepsis warnings triggered by these three tools.
Methods
Study design, setting and population
We conducted a retrospective observational cohort study, which included all adult patients (≥18 years) admitted between December 2014 and June 2016 to a 570‐bed metropolitan tertiary referral hospital in Sydney. Patients were excluded if they: (i) had a principal diagnosis of pregnancy and/or childbirth; (ii) were admitted directly to ICU; or (iii) had no vital signs or laboratory data documented during their admission (Fig. 1). Ethics approval was provided by the Macquarie University Human Research Ethics Committee (reference no: 5201600265).
Figure 1.
Flow diagram of study patients.
Defining suspected infection
A patient was defined as having a suspected infection if any blood culture was ordered during the patient's hospital stay. Three other definitions were applied in sensitivity analyses using different surrogate markers: (i) ordering of two or more blood cultures; (ii) any blood culture and C‐reactive protein/procalcitonin (CRP/PCT); and (iii) any blood culture or CRP/PCT.
Suspected sepsis identified using sepsis recognition tools
A suspected sepsis case was identified when a sepsis warning was triggered based on a recognition tool among patients with suspected infection. The criteria for triggering a sepsis warning for each of the three sepsis recognition tools, that is the ASP, qSOFA and SIRS, are listed in Table 1.
Table 1.
Three sepsis recognition tools
| Measurements | qSOFA criteria 6 (SOS warning: at least two criteria are met) | SIRS criteria 7 (SOS warning: at least two criteria are met) | Adult sepsis pathway 9 | |
|---|---|---|---|---|
| Red Zone (SOS warning: at least one criterion is met) | Yellow Zone (SOS warning: at least two criteria are met) | |||
| Systolic blood pressure (mmHg) | ≤100 | <90 | <100 | |
| Respiratory rate (breaths/min) | ≥22 | >20 | ≤10 or ≥25 | |
| Glasgow Coma Scale score | <15 | <15 | ||
| Temperature (°C) | — | >38 or <36 | <35.5 or >38.5 | |
| Heart rate (b.p.m.) | — | >90 | ≤50 or ≥120 | |
| White blood cell count (/mm3) | — | >12 000 or <4000 | — | — |
| Lactate (mmol/L) | ≥4.0 | ≥2.0 | ||
| Base excess (mEq/L) | <−5.0 | |||
| SpO2 (%) | <95 | |||
—, not included as part of criteria; qSOFA, quick Sepsis‐related Organ Failure Assessment; SIRS, systemic inflammatory response syndrome; SOS, suspicion of sepsis; SpO2, peripheral oxygen saturation.
The ASP tool provides two levels of sepsis warnings: (i) Red Zone for patients with suspected severe sepsis/septic shock; and (ii) Yellow Zone for patients who have suspected sepsis (Table 1). 9
Data sources and linkage
This study utilised routinely collected hospital clinical and administrative datasets. Patient demographic, admission and clinical data, including age, gender, vital signs, laboratory results, blood cultures and ICU admissions, were extracted from various clinical information systems, including Patient Admission Systems and Laboratory Information Systems. More than 3.7 million time‐stamped vital signs and laboratory test results were included. Data sets from these sources were linked using de‐identified medical record numbers and time stamp data where relevant.
Outcomes
The primary outcome of the study was in‐hospital mortality, and two secondary outcomes were composite outcomes: (i) in‐hospital mortality or ICU admission; and (ii) in‐hospital mortality or prolonged ICU stay (≥3 days), which was previously defined in the study on assessing clinical criteria for SEPSIS‐3. 6
Statistical analysis
Algorithms which encapsulated the criteria for each sepsis recognition tool (Table 1) were developed. A sepsis score was calculated for each relevant vital sign or laboratory test result during hospital stays (excluding ED and/or ICU stays). A relevant sepsis warning was produced if the score met the minimum sepsis score for a recognition tool, for example a qSOFA score ≥2 would result in a qSOFA sepsis warning, and an ASP warning from either Red or Yellow Zone would lead to an overall ASP warning. Patients might experience multiple sepsis warnings during their admissions. If ASP warnings from both Red and Yellow Zones were triggered during a hospital stay, the ASP warning level was recorded as a Red Zone warning. No patients were excluded in any analysis due to lack of other laboratory parameters required for the SIRS or ASP tools.
The performance of each of the three tools in predicting study outcomes for ward patients were compared. Comparisons were made using sensitivity, specificity, positive predictive value, and negative predictive value. The area under the receiver‐operating characteristic curve (AUROC) was used to assess the discrimination of each tool, that is the ability of each tool to correctly predict those with and without the study outcomes. The non‐parametric approach of DeLong 17 for comparing AUROC was applied to take into account the correlated nature of the data. Sensitivity analyses were conducted for the primary outcome based on: (i) three additional definitions for suspected infection; and (ii) regardless of the infection status, that is including all eligible patients.
A secondary analysis of two levels of the ASP warning was also conducted. Patient characteristics were compared between cohorts with two levels of the ASP warnings using t‐tests, Wilcoxon‐Mann–Whitney tests and χ2 tests as appropriate based on the distribution of the data.
The median time from first sepsis warning to the primary outcome, that is in‐hospital death, was presented for each recognition tool. We plotted the cumulative proportions of patients who died in hospital in one‐hour increments after the first warning of suspected sepsis by each tool. Analyses were performed using R (version 3.5.0; R Foundation for Statistical Computing, Vienna, Austria) and SAS (version 9.4; SAS Institute Inc., Cary, NC, USA).
Results
A total of 42 870 patient admissions was recorded during the study period, of which 28 978 ward admissions were eligible for inclusion (Fig. 1). The final study cohort consisted of 2940 patients who had a suspected infection during their stay (Table 2). A total of 702 (23.88%) of these patients had an ICU admission and 217 (7.38%) died in hospital (Table 2).
Table 2.
Characteristics of study patients and suspected sepsis patients recognised by three sepsis recognition tools
| Characteristic | All study patients (n = 2940) | qSOFA score ≥ 2 (n = 332, 11.29%†) | SIRS score ≥ 2 (n = 2054, 69.86%†) | Adult sepsis pathway | |||
|---|---|---|---|---|---|---|---|
| Overall (n = 1305, 44.39%†) | Yellow zone (n = 669, 22.75%†) | Red Zone (n = 636, 21.63%†) | P‐value‡ | ||||
| Age, median (IQR) (years) | 69 (53–80) | 75 (60–83) | 70 (55–81) | 71 (59–81) | 72 (60–82) | 71 (57–81) | 0.65 |
| Male sex, n (%) | 1442 (49.05) | 174 (52.41) | 985 (47.96) | 660 (50.57) | 339 (50.67) | 321 (50.47) | 0.94 |
| ATSI, n (%) | 96 (3.27) | 8 (2.41) | 61 (2.97) | 41 (3.14) | 24 (3.59) | 17 (2.67) | 0.34 |
| ICU admissions, n (%) | 702 (23.88) | 99 (29.82) | 475 (23.13) | 535 (41.00) | 162 (24.22) | 373 (58.65) | <0.0001 |
| ICU LOS, median (IQR) (h) | 83 (34–169) | 92 (39–198) | 68 (26–142) | 78 (35–165) | 70 (27–137) | 85 (38–188) | <0.0001 |
| Hospital LOS, median (IQR) (days) | 8.25 (4.66–15.79) | 14.12 (7.73–24.28) | 10.07 (5.55–18.28) | 12.12 (6.68–21.03) | 11.33 (6.22–18.81) | 12.86 (6.95–24.09) | <0.0001 |
| In‐hospital mortality, n (%) (died on wards + died in ICU) | 217 (7.38) (103 + 114) | 80 (24.10) (51 + 29) | 163 (7.94) (98 + 65) | 202 (15.48) (93 + 109) | 47 (7.03) (36 + 11) | 155 (24.37) (57 + 98) | <0.0001 |
| Composite outcome 1§, n (%) | 805 (27.38) | 150 (45.18) | 573 (27.90) | 628 (48.12) | 198 (29.60) | 430 (67.61) | <0.0001 |
| Composite outcome 2§, n (%) | 519 (17.65) | 125 (37.65) | 383 (18.65) | 463 (35.48) | 129 (19.28) | 334 (52.52) | <0.0001 |
Percentages were calculated out of 2940 patients with suspected infection.
P‐values were obtained to compare between patients recognised with Red Zone sepsis and those with Yellow Zone sepsis using the adult sepsis pathway.
Composite outcome 1 = In‐hospital mortality or ICU admission; composite outcome 2 = In‐hospital mortality or prolonged ICU stay (≥3 days).
ATSI, Aboriginal and/or Torres Strait Islander origin; ICU, intensive care unit; IQR, interquartile range; LOS, length of stay; qSOFA, quick Sepsis‐related Organ Failure Assessment; SIRS, systemic inflammatory response syndrome.
Patients identified with suspected sepsis by the sepsis recognition tools
Among these 2940 patients, 69.86% (n = 2054) would be recognised as suspected sepsis cases based on SIRS criteria, 44.39% (n = 1305) on ASP and 11.29% (n = 332) on qSOFA criteria (Fig. 1).
Patients with suspected sepsis based on the qSOFA tool were more likely to be older, male, have a longer ICU and overall hospital stay, and higher in‐hospital mortality rate compared to patients meeting suspected sepsis criteria for the SIRS or ASP (Table 2).
Of all 217 deaths that occurred in the study period, a greater proportion occurred in ICU (52.53%, n = 114) than on wards (47.47%, n = 103). Among suspected sepsis patients identified using ASP, nearly half died on wards (46% = 98/202) (Table 2).
Prediction of adverse outcomes for patients on general wards
The ASP tool performed best in terms of predicting in‐hospital mortality among ward patients with suspected infection based on whether an overall ASP warning was triggered, with AUROC 0.76 (95% confidence interval (CI): 0.74–0.78), followed by the qSOFA tool (0.64, 95% CI: 0.61–0.67), and SIRS (0.53, 95% CI: 0.50–0.56) (P < 0.0001 for all pairwise comparisons; Fig. 2). ASP also performed best for the two secondary composite outcomes (P < 0.0001 for all pairwise comparisons of three tools within each outcome measure).
Figure 2.
Comparison of the discriminatory capacities of three sepsis recognition tools for in‐hospital mortality and two composite outcomes: Composite outcome 1 = in‐hospital mortality or intensive care unit (ICU) admission; composite outcome 2 = in‐hospital mortality or prolonged ICU stay (≥3 days) (n = 2940). Dots represent point estimates, and error bars represent 95% confidence intervals. AUROC, area under receiver operating characteristic curve.
The ASP had the highest sensitivity across the three outcome measures, followed by SIRS and qSOFA (Table 3). The qSOFA tool had the highest specificity across the three outcome measures, followed by the ASP and SIRS.
Table 3.
Performance of the three sepsis recognition tools by patient outcomes (n = 2940)
| Patient outcome | Sepsis recognition tool | Sensitivity, % (95% CI) | Specificity, % (95% CI) | PPV, % (95% CI) | NPV, % (95% CI) |
|---|---|---|---|---|---|
| In‐hospital mortality | qSOFA | 36.87 (30.45–43.29) | 90.75 (89.66–91.83) | 24.10 (19.5–28.70) | 94.75 (93.89–95.60) |
| SIRS | 75.12 (69.36–80.87) | 30.55 (28.82–32.28) | 7.94 (6.77–9.10) | 93.91 (92.33–95.48) | |
| ASP – overall | 93.09 (89.71–96.46) | 59.49 (57.65–61.34) | 15.48 (13.52–17.44) | 99.08 (98.62–99.54) | |
| ASP – yellow | 21.66 (16.18–27.14) | 76.94 (75.36–78.52) | 6.96 (5.04–8.88) | 92.49 (91.41–93.58) | |
| ASP – red | 71.43 (65.42–77.44) | 82.34 (80.9–83.77) | 24.37 (21.03–27.71) | 97.31 (96.65–97.97) | |
| Composite outcome 1† | qSOFA | 18.63 (15.94–21.32) | 91.48 (90.29–92.66) | 45.18 (39.83–50.53) | 74.89 (73.22–76.55) |
| SIRS | 71.18 (68.05–74.31) | 30.63 (28.68–32.59) | 27.90 (25.96–29.84) | 73.81 (70.92–76.71) | |
| ASP – overall | 78.01 (75.15–80.87) | 68.29 (66.32–70.26) | 48.12 (45.41–50.83) | 89.17 (87.67–90.68) | |
| ASP – yellow | 25.22 (22.22–28.22) | 77.89 (76.13–79.65) | 30.07 (26.61–33.53) | 73.42 (71.6–75.24) | |
| ASP – red | 53.42 (49.97–56.86) | 90.35 (89.1–91.6) | 67.61 (63.97–71.25) | 83.72 (82.22–85.23) | |
| Composite outcome 2† | qSOFA | 24.08 (20.41–27.76) | 91.45 (90.34–92.56) | 37.65 (32.44–42.86) | 84.89 (83.52–86.27) |
| SIRS | 73.8 (70.01–77.58) | 30.98 (29.14–32.82) | 18.65 (16.96–20.33) | 84.65 (82.28–87.02) | |
| ASP – overall | 89.21 (86.54–91.88) | 65.22 (63.32–67.12) | 35.48 (32.88–38.07) | 96.57 (95.69–97.46) | |
| ASP – yellow | 25.63 (21.87–29.38) | 77.61 (75.95–79.27) | 19.70 (16.70–22.70) | 82.96 (81.41–84.51) | |
| ASP – red | 64.35 (60.23–68.48) | 87.53 (86.21–88.84) | 52.52 (48.63–56.40) | 91.97 (90.86–93.08) |
Composite outcome 1 = In‐hospital mortality or intensive care unit (ICU) admission; composite outcome 2 = In‐hospital mortality or prolonged ICU stay (≥3 days).
ASP, adult sepsis pathway; CI, confidence interval; NPV, negative predictive value; PPV, positive predictive value; qSOFA, quick Sepsis‐related Organ Failure Assessment; SIRS, systemic inflammatory response syndrome.
Time from first sepsis warning to in‐hospital death
Of the 217 patients who died in hospital, most of them (93.09%, n = 202) would have received an overall ASP sepsis warning during their hospital stays; 75.12% (n = 163) with a SIRS warning; and only 36.87% (n = 80) with a qSOFA warning. Overall, ASP warnings for these patients would have occurred earlier than the SIRS or qSOFA warnings. The median time from a patient first meeting the ASP criteria to death was 8.21 days (interquartile range (IQR): 2.29–16.75), followed by SIRS (6.42 days, IQR: 0.29–17.08) and qSOFA (0 days, IQR: 0–2.58). Over time, a greater proportion of patients who died would have been recognised by the ASP compared to the use of the other two tools (Fig. 3). For example, more than 25% of patients would have received an ASP sepsis warning 4 days (96 h) before death, compared to less than 16% from the other two tools.
Figure 3.
Cumulative proportions of 217 patients who died in hospital would have received a sepsis warning using the three sepsis tools (time intervals longer than 120 h (5 days) were trimmed). (
), Quick Sepsis‐related Organ Failure Assessment; (
), systemic inflammatory response syndrome; (
), adult sepsis pathway.
Comparison of two levels of ASP warning
Among 1305 suspected septic patients identified using ASP, 636 (48.74%) had a Red Zone warning and 669 (51.26%) had a Yellow Zone warning (Table 2). Patients who experienced these two warnings had similar age and gender composition, but those with a Red Zone warning were more likely to be admitted to ICU and/or die after a much longer stay in ICU and hospital. Overall, the Red Zone warning had much better predictive validity, that is higher AUROCs, across all three outcomes than the Yellow Zone sepsis warning (Fig. 2).
Discussion
We compared the performance of three sepsis recognition tools in predicting adverse outcomes. We applied an innovative data‐driven process utilising a large volume of clinical data routinely collected as part of various clinical information systems in a large Australian hospital. Our findings provide support for the use of the ASP among hospital general ward patients, over qSOFA. We found that the ASP outperformed the qSOFA and SIRS tools in predicting patients at risk of experiencing typical adverse outcomes from sepsis, but also presented a significantly earlier sepsis warning for patients who died than either SIRS or qSOFA. The qSOFA demonstrated low sensitivity, identifying only 36.87% of patients with suspected infection who later died in hospital. In contrast, the ASP was successful at recognising 93.09% of these patients. A sepsis recognition tool with low sensitivity could result in missed cases and potentially lead to increased mortality rates.
A sepsis recognition tool generating an earlier warning may facilitate early treatment, which has been shown to be associated with reduced mortality. Seymour et al. found, in a sample of 49 331 patients with suspected infection from 149 US ED, that every hour delay in the administration of antibiotics was associated with a 4% significant increase in risk‐adjusted mortality. 5 We found that about half of patients who died would have had an ASP warning 8.21 days before death while qSOFA warning would not have been triggered for two‐thirds of patients before death. These findings are consistent with those from another study of ward patients by Churpek et al. published in 2017. 18 That study found almost half of patients who had an adverse outcome (i.e. death or prolonged ICU stay) did not have a qSOFA warning prior to the outcome.
While attention has been placed on the identification of sepsis among ED and ICU patients, 19 , 20 , 21 previous studies have showed that the majority of patients with sepsis are treated on hospital general care wards. 14 , 15 These studies also reveal mortality rates of 26% to 30% among patients with sepsis who are not admitted to an ICU compared to 11–33% for those in the ICU. 14 , 15 We found that nearly half of all patients with suspected sepsis identified using ASP died on general wards, which suggests that there could be a significant underappreciation of the potential risks of mortality from sepsis on general wards. This may reflect a problem with access to ICU beds as has been highlighted in previous studies. 19 Two Australian studies found that ward patients with suspected sepsis identified using qSOFA or SIRS at the time of Rapid Response Reviews were associated with poor outcomes, that is in‐hospital mortality and longer length of stay. 22 , 23 Further investigation is warranted to improve patient outcomes for patients with suspected sepsis on hospital general wards.
A systematic review, published in 2018, 24 compared the qSOFA and SIRS in predicting mortality and identified only three studies 18 , 25 , 26 involving patients on general wards, all three published in 2017. Two of these studies reported the AUROC results for in‐hospital mortality comparing the qSOFA and SIRS tools. Although direct comparison is difficult due to the heterogeneity of study populations and methods used, our AUROC results for in‐hospital mortality (ASP 0.76, qSOFA 0.64 and SIRS 0.53) were slightly lower, but at a similar level to these two studies. Churpek et al. 18 reported similar AUROC for qSOFA (0.69) and SIRS (0.67) while Finkelsztein et al. 26 had higher AUROC for qSOFA (0.74) and lower result for SIRS (0.59).
The main strength of this study is the application of an innovative data‐driven approach to link and exploit routinely collected data from hospital clinical information systems. We developed algorithms to retrospectively evaluate the performance of three sepsis recognition tools using these rich data sets, including vital signs and laboratory test results. This data driven approach allowed us to test different tools without consuming the substantial resources which would be required to implement and trial these tools in the field. This is an efficient first step in assessing whether an intervention may provide clinical benefits prior to introduction. Further investigation, including prospective studies, after the implementation of a sepsis recognition tool would be beneficial to determine whether the tool significantly improves patient outcomes.
Clinical information systems provide a valuable mechanism for incorporating sepsis recognition algorithms which automatically generate alerts and provide decision support to guide appropriate, prompt treatment. However, adoption of such algorithms requires close attention to both the specificity and sensitivity of alerting to avoid the increasingly recognised problem of alert fatigue among clinical system users. 27 Few evaluations of the effectiveness of such electronic decision support have been conducted. 13 Many hospitals rely on paper‐based sepsis recognition tools, susceptible to transcription and interpretation errors, and highly reliant on vigilant and timely review by clinicians. In contrast, appropriately designed automated systems have the potential to decrease delays and increase the accuracy of sepsis detection. 13
Limitations
Our study had several limitations. First, this was a single‐centre study and therefore the results may not be generalisable to other settings. Second, there is no gold standard to determine when a patient is suspected to have infection. Blood culture ordering and antibiotic usage have been used as a flag for suspected infection in the study for developing the qSOFA criteria by Seymour et al. 6 In our study, suspected infection was inferred if a blood culture was ordered because data on antibiotic usage were not available. In addition, blood culture ordering was endorsed by ASP, which was recommended to be used at the study hospital during the study period. To address this limitation, we conducted four sensitivity analyses using: (i) three cohorts based on additional definitions for suspected infection; and (ii) all 28 978 patients regardless of infection status (see the Methods section). Results from all four analyses (Tables A1, A2) were consistent with those in Table 3 comparing the three recognition tools. The ASP outperformed qSOFA in prognostic accuracy although in‐hospital mortality rates were different. Third, we did not have information on Not for Resuscitation (NFR) or limitation of care orders, which may have inflated our primary outcome. To be conservative, we conducted a subgroup analysis excluding all 103 patients who died on wards. The ASP tool still had the highest AUROC among the three tools (0.78, 95% CI: 0.75–0.80, Table A3), which was similar to that from the main analysis (0.76). Fourth, our study examined typical adverse outcomes from sepsis, which were used in the study for developing the qSOFA criteria. 6 However, it is acknowledged that these adverse outcomes could also be caused by other critical clinical conditions.
Implications for clinicians, researchers and policy makers
The present study provides evidence that hospitals currently using the ASP may not gain any additional benefit in switching to the qSOFA tool. The ASP demonstrated greater prognostic accuracy and earlier warning for in‐hospital mortality for adult ward patients with suspected infection compared to the qSOFA and SIRS tools. Early detection of sepsis would increase the chances to facilitate the prompt involvement of senior clinicians to confirm the diagnosis and support rapid treatment with appropriate intravenous antibiotics and fluids. Improving the availability of tools that support clinicians to more efficiently incorporate sepsis risk information into their clinical workflows, such as electronic decision support, should be investigated.
Sensitivity analyses using different definitions for suspected infection (see the Methods section)
Table A1.
Characteristics of patient cohorts defined by four definitions of suspected infection
| Characteristics | Patients with suspected infection were identified if during the hospital stay there was (were) | |||
|---|---|---|---|---|
| Any BC order | >1 BC orders | Any BC and any CRP/PCT orders | Any BC or CRP/PCT order | |
| Number of patient admissions | 2940 | 1435 | 2767 | 10 827 |
| Age in years – median (IQR) | 69 (53–80) | 69 (54–80) | 69 (54–80) | 68 (50–80) |
| Male sex – n (%) | 1442 (49.05) | 694 (48.36) | 1358 (49.08) | 5101 (47.11) |
| ATSI – n (%) | 96 (3.27) | 46 (3.21) | 93 (3.36) | 393 (3.63) |
| ICU admissions – n (%) | 702 (23.88) | 419 (29.20) | 694 (25.08) | 1164 (10.75) |
| ICU LOS in hours – median (IQR) | 83 (34–169) | 99 (34–219 | 84 (35–172) | 62 (24–129) |
| Hospital LOS in days – median (IQR) | 8.25 (4.66–15.79) | 11.12 (6.05–20.39) | 8.68 (4.90–16.09) | 5.64 (3.15–10.25) |
| In‐hospital mortality – n (%, 95% CI) | 217 (7.38, 6.44–8.33) | 132 (9.20, 7.70–10.69) | 210 (7.59, 6.6–8.58) | 361 (3.33, 3.00–3.67) |
| Composite outcome 1– n (%, 95% CI) | 805 (27.38, 25.77–28.99) | 475 (33.10, 30.67–35.54) | 793 (28.66, 26.97–30.34) | 1382 (12.76, 12.14–13.39) |
| Composite outcome 2 – n (%, 95% CI) | 519 (17.65, 16.27–19.03) | 326 (22.72, 20.55–24.89) | 512 (18.50, 17.06–19.95) | 784 (7.24, 6.75–7.73) |
Composite outcome 1 = in‐hospital mortality or ICU admission; composite outcome 2 = In‐hospital mortality or prolonged ICU stay (≥3 days).
ATSI, Aboriginal and/or Torres Strait Islander origin; CI, confidence interval; ICU, intensive care unit; IQR, interquartile range; LOS, length of stay.
Table A2.
Performance of the three recognition tools on predicting in‐hospital mortality by different suspected infection definitions
| Suspected infection identified | Recognition tools | Sensitivity, % (95% CI) | Specificity, % (95% CI) | PPV, % (95% CI) | NPV, % (95% CI) | AUROC (95% CI) |
|---|---|---|---|---|---|---|
| Any BC order | qSOFA | 36.87 (30.45–43.29) | 90.75 (89.66–91.83) | 24.10 (19.50–28.70) | 94.75 (93.89–95.6) | 0.64 (0.61–0.67) |
| SIRS | 75.12 (69.36–80.87) | 30.55 (28.82–32.28) | 7.94 (6.77–9.10) | 93.91 (92.33–95.48) | 0.53 (0.5–0.56) | |
| ASP – overall | 93.09 (89.71–96.46) | 59.49 (57.65–61.34) | 15.48 (13.52–17.44) | 99.08 (98.62–99.54) | 0.76 (0.74–0.78) | |
| ASP – yellow | 21.66 (16.18–27.14) | 76.94 (75.36–78.52) | 6.96 (5.04–8.88) | 92.49 (91.41–93.58) | 0.51 (0.48–0.54) | |
| ASP – red | 71.43 (65.42–77.44) | 82.34 (80.90–83.77) | 24.37 (21.03–27.71) | 97.31 (96.65–97.97) | 0.77 (0.74–0.8) | |
| >1 BC orders | qSOFA | 40.15 (31.79–48.51) | 87.41 (85.61–89.21) | 24.42 (18.71–30.14) | 93.51 (92.13–94.9) | 0.64 (0.59–0.68) |
| SIRS | 81.06 (74.38–87.74) | 21.64 (19.41–23.88) | 9.49 (7.78–11.20) | 91.86 (88.80–94.92) | 0.51 (0.48–0.55) | |
| ASP – overall | 98.48 (96.4–100.57) | 52.11 (49.4–54.82) | 17.24 (14.55–19.94) | 99.71 (99.3–100.11) | 0.75 (0.74–0.77) | |
| ASP – yellow | 23.48 (16.25–30.72) | 73.29 (70.89–75.69) | 8.18 (5.42–10.94) | 90.44 (88.66–92.21) | 0.52 (0.48–0.55) | |
| ASP – red | 75 (67.61–82.39) | 78.36 (76.12–80.59) | 25.98 (21.58–30.39) | 96.87 (95.82–97.92) | 0.77 (0.73–0.81) | |
| Any BC and any CRP/PCT orders | qSOFA | 37.14 (30.61–43.68) | 90.34 (89.2–91.49) | 24.00 (19.36–28.64) | 94.59 (93.7–95.49) | 0.64 (0.60–0.67) |
| SIRS | 76.19 (70.43–81.95) | 29.14 (27.37–30.9) | 8.11 (6.91–9.32) | 93.71 (92.02–95.40) | 0.53 (0.50–0.56) | |
| ASP – overall | 93.33 (89.96–96.71) | 57.96 (56.05–59.87) | 15.42 (13.44–17.41) | 99.06 (98.58–99.55) | 0.76 (0.74–0.78) | |
| ASP – yellow | 21.90 (16.31–27.50) | 76.22 (74.57–77.87) | 7.03 (5.07–8.99) | 92.24 (91.1–93.38) | 0.51 (0.48–0.54) | |
| ASP – red | 71.43 (65.32–77.54) | 81.50 (80–83.01) | 24.08 (20.72–27.43) | 97.20 (96.5–97.9) | 0.76 (0.73–0.8) | |
| Any BC or CRP/PCT order | qSOFA | 33.24 (28.38–38.10) | 95.01 (94.6–95.43) | 18.69 (15.68–21.71) | 97.63 (97.34–97.93) | 0.64 (0.62–0.67) |
| SIRS | 78.12 (73.85–82.38) | 52.56 (51.6–53.52) | 5.37 (4.76–5.98) | 98.58 (98.27–98.89) | 0.65 (0.63–0.68) | |
| ASP – overall | 87.81 (84.44–91.19) | 76.50 (75.69–77.32) | 11.42 (10.24–12.6) | 99.45 (99.29–99.61) | 0.82 (0.80–0.84) | |
| ASP – yellow | 27.15 (22.56–31.73) | 85.30 (84.63–85.98) | 5.99 (4.84–7.14) | 97.14 (96.8–97.48) | 0.56 (0.54–0.59) | |
| ASP – red | 61.22 (56.19–66.25) | 91.10 (90.55–91.64) | 19.17 (16.90–21.44) | 98.55 (98.31–98.79) | 0.76 (0.74–0.79) | |
| Regardless of infection status (all 28 978 eligible patients) | qSOFA | 31.00 (26.77–35.24) | 97.49 (97.31–97.67) | 16.57 (14.08–19.06) | 98.88 (98.75–99.00) | 0.64 (0.62–0.66) |
| SIRS | 73.58 (69.54–77.62) | 75.48 (74.98–75.98) | 4.60 (4.12–5.08) | 99.44 (99.34–99.54) | 0.75 (0.73–0.77) | |
| ASP – overall | 81.00 (77.41–84.60) | 88.70 (88.34–89.07) | 10.33 (9.33–11.32) | 99.66 (99.59–99.73) | 0.85 (0.83–0.87) | |
| ASP – yellow | 31.00 (26.77–35.24) | 97.49 (97.31–97.67) | 16.57 (14.08–19.06) | 98.88 (98.75–99.00) | 0.60 (0.58–0.62) | |
| ASP – red | 73.58 (69.54–77.62) | 75.48 (74.98–75.98) | 4.60 (4.12–5.08) | 99.44 (99.34–99.54) | 0.75 (0.73–0.77) |
ASP, adult sepsis pathway; CI, confidence interval; NPV, negative predictive value; PPV, positive predictive value; qSOFA, quick Sepsis‐related Organ Failure Assessment; SIRS, systemic inflammatory response syndrome.
Subgroup analysis excluding 103 patients who died on wards (n = 2837)
Table A3.
Performance of the three recognition tools on predicting in‐hospital mortality
| Recognition tool | Sensitivity, % (95% CI) | Specificity, % (95% CI) | PPV, % (95% CI) | NPV, % (95% CI) | AUROC (95% CI) |
|---|---|---|---|---|---|
| qSOFA | 25.44 (17.44–33.43) | 90.75 (89.66–91.83) | 10.32 (6.76–13.88) | 96.67 (95.98–97.37) | 0.58 (0.54–0.62) |
| SIRS | 57.02 (47.93–66.11) | 30.55 (28.82–32.28) | 3.32 (2.53–4.12) | 94.44 (92.92–95.95) | 0.56 (0.52–0.61) |
| ASP – overall | 95.61 (91.85–99.37) | 59.49 (57.65–61.34) | 8.99 (7.38–10.60) | 99.69 (99.42–99.96) | 0.78 (0.75–0.80) |
| ASP – yellow | 9.65 (4.23–15.07) | 76.94 (75.36–78.52) | 1.72 (0.71–2.73) | 95.31 (94.43–96.20) | 0.57 (0.54–0.60) |
| ASP – red | 85.96 (79.59–92.34) | 82.34 (80.90–83.77) | 16.93 (13.87–19.98) | 99.29 (98.95–99.64) | 0.84 (0.81–0.87) |
ASP, adult sepsis pathway; AUROC, area under the receiver‐operating characteristic curve; CI, confidence interval; NPV, negative predictive value; PPV, positive predictive value; qSOFA, quick Sepsis‐related Organ Failure Assessment; SIRS, systemic inflammatory response syndrome.
Funding: This project was funded by the Clinical Excellence Commission (CEC) and eHealth NSW.
Conflict of interest: None.
References
- 1. Singer M, Deutschman CS, Seymour CW, Shankar‐Hari M, Annane D, Bauer M et al. The third international consensus definitions for sepsis and septic shock (Sepsis‐3). JAMA 2016; 315: 801–10. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Fleischmann C, Scherag A, Adhikari NK, Hartog CS, Tsaganos T, Schlattmann P et al. Assessment of global incidence and mortality of hospital‐treated sepsis. Current estimates and limitations. Am J Respir Crit Care Med 2016; 193: 259–72. [DOI] [PubMed] [Google Scholar]
- 3. Angus DC, Linde‐Zwirble WT, Lidicker J, Clermont G, Carcillo J, Pinsky MR. Epidemiology of severe sepsis in the United States: analysis of incidence, outcome, and associated costs of care. Crit Care Med 2001; 29: 1303–10. [DOI] [PubMed] [Google Scholar]
- 4. Engel C, Brunkhorst FM, Bone HG, Brunkhorst R, Gerlach H, Grond S et al. Epidemiology of sepsis in Germany: results from a national prospective multicenter study. Intensive Care Med 2007; 33: 606–18. [DOI] [PubMed] [Google Scholar]
- 5. Seymour CW, Gesten F, Prescott HC, Friedrich ME, Iwashyna TJ, Phillips GS et al. Time to treatment and mortality during mandated emergency care for sepsis. N Engl J Med 2017; 376: 2235–44. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Seymour CW, Liu VX, Iwashyna TJ, Brunkhorst FM, Rea TD, Scherag A et al. Assessment of clinical criteria for sepsis: for the third international consensus definitions for sepsis and septic shock (Sepsis‐3). JAMA 2016; 315: 762–74. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Levy MM, Fink MP, Marshall JC, Abraham E, Angus D, Cook D et al. 2001 SCCM/ESICM/ACCP/ATS/SIS International Sepsis Definitions Conference. Intensive Care Med 2003; 29: 530–8. [DOI] [PubMed] [Google Scholar]
- 8. Bone RC, Balk RA, Cerra FB, Dellinger RP, Fein AM, Knaus WA et al. Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. The ACCP/SCCM Consensus Conference Committee. American College of Chest Physicians/Society of Critical Care Medicine. Chest 1992; 101: 1644–55. [DOI] [PubMed] [Google Scholar]
- 9. Clinical Excellence Commission . SEPSIS KILLS – Adult Sepsis Pathway. [cited 2019 Oct]. Available from URL: http://www.cec.health.nsw.gov.au/__data/assets/pdf_file/0005/291803/Adult-Sepsis-Pathway-Sept-2016-with-watermark.pdf
- 10. Clinical Excellence Commission . Sepsis Toolkit – Inpatient Program Implementation Guide. 2014 [cited 2019 Jul]. Available from URL: http://www.cec.health.nsw.gov.au/__data/assets/pdf_file/0010/276067/Sepsis_Toolkit_inpatient-full.pdf
- 11. Green M, Lander H, Snyder A, Hudson P, Churpek M, Edelson D. Comparison of the between the flags calling criteria to the MEWS, NEWS and the electronic Cardiac Arrest Risk Triage (eCART) score for the identification of deteriorating ward patients. Resuscitation 2018; 123: 86–91. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Clinical Excellence Commission . Between the Flags [cited 2019 Jul]. Available from URL: http://www.cec.health.nsw.gov.au/patient-safety-programs/adult-patient-safety/between-the-flags
- 13. Bhattacharjee P, Edelson DP, Churpek MM. Identifying patients with sepsis on the hospital wards. Chest 2017; 151: 898–907. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Esteban A, Frutos‐Vivar F, Ferguson ND, Peñuelas O, Lorente JÁ, Gordo F et al. Sepsis incidence and outcome: contrasting the intensive care unit with the hospital ward. Crit Care Med 2007; 35: 1284–9. [DOI] [PubMed] [Google Scholar]
- 15. Rohde JM, Odden AJ, Bonham C, Kuhn L, Malani PN, Chen LM et al. The epidemiology of acute organ system dysfunction from severe sepsis outside of the intensive care unit. J Hosp Med 2013; 8: 243–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Sundararajan V, Macisaac CM, Presneill JJ, Cade JF, Visvanathan K. Epidemiology of sepsis in Victoria, Australia. Crit Care Med 2005; 33: 71–80. [DOI] [PubMed] [Google Scholar]
- 17. DeLong ER, DeLong DM, Clarke‐Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 1988; 44: 837–45. [PubMed] [Google Scholar]
- 18. Churpek MM, Snyder A, Han X, Sokol S, Pettit N, Howell MD et al. Quick Sepsis‐related Organ Failure Assessment, systemic inflammatory response syndrome, and early warning scores for detecting clinical deterioration in infected patients outside the intensive care unit. Am J Respir Crit Care Med 2017; 195: 906–11. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Burrell AR, McLaws ML, Fullick M, Sullivan RB, Sindhusake D. SEPSIS KILLS: early intervention saves lives. Med J Aust 2016; 204: 73.e71–7. [DOI] [PubMed] [Google Scholar]
- 20. Raith EP, Udy AA, Bailey M, McGloughlin S, MacIsaac C, Bellomo R et al. Prognostic accuracy of the SOFA score, SIRS criteria, and qSOFA score for in‐hospital mortality among adults with suspected infection admitted to the intensive care unit. JAMA 2017; 317: 290–300. [DOI] [PubMed] [Google Scholar]
- 21. Shetty AL, Brown T, Booth T, van KL, Dor‐Shiffer DE, Vaghasiya MR et al. Systemic inflammatory response syndrome‐based severe sepsis screening algorithms in emergency department patients with suspected sepsis. Emerg Med Australas 2016; 28: 287–94. [DOI] [PubMed] [Google Scholar]
- 22. Cross G, Bilgrami I, Eastwood G, Johnson P, Howden BP, Bellomo R et al. The epidemiology of sepsis during rapid response team reviews in a teaching hospital. Anaesth Intensive Care 2015; 43: 193–8. [DOI] [PubMed] [Google Scholar]
- 23. LeGuen M, Ballueer Y, McKay R, Eastwood G, Bellomo R, Jones D et al. Frequency and significance of qSOFA criteria during adult rapid response team reviews: a prospective cohort study. Resuscitation 2018; 122: 13–18. [DOI] [PubMed] [Google Scholar]
- 24. Serafim R, Gomes JA, Salluh J, Póvoa P. A comparison of the quick‐SOFA and systemic inflammatory response syndrome criteria for the diagnosis of sepsis and prediction of mortality: a systematic review and meta‐analysis. Chest 2018; 153: 646–55. [DOI] [PubMed] [Google Scholar]
- 25. Dorsett M, Kroll M, Smith CS, Asaro P, Liang SY, Moy HP. qSOFA has poor sensitivity for prehospital identification of severe sepsis and septic shock. Prehosp Emerg Care 2017; 21: 489–97. [DOI] [PubMed] [Google Scholar]
- 26. Finkelsztein EJ, Jones DS, Ma KC, Pabón MA, Delgado T, Nakahira K et al. Comparison of qSOFA and SIRS for predicting adverse outcomes of patients with suspicion of sepsis outside the intensive care unit. Crit Care 2017; 21: 73. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27. van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006; 13: 138–47. [DOI] [PMC free article] [PubMed] [Google Scholar]