qSOFA and SIRS as Predictors of Critical Care Intervention among Patients with Suspected Infection

Abstract

Objective

The Sepsis III clinical criteria for the diagnosis of sepsis rely on scores derived to predict in-hospital mortality. In this study, we introduce the novel outcome of ‘received critical care intervention (CCI)’ and investigate the related predictive performance of both the quick-Sequential Organ Failure Assessment (qSOFA) and the Systemic Inflammatory Response Syndrome (SIRS) criteria.

Design

This was a single center, retrospective analysis of electronic health records.

Setting

Tertiary care hospital in the United States.

Patients

Patients with suspected infection who presented to the Emergency Department (ED) and were admitted to the hospital between January 2010 and December 2014.

Interventions

SIRS and qSOFA scores were calculated and their relationships to the receipt of CCI and in-hospital mortality were determined.

Measurement and Main Results

24,164 patients were included of whom 6,693 (27.7%) were admitted to an ICU within 48-hours. 4,453 (66.5%) patients admitted to the ICU received a CCI. Among those with qSOFA <2, 13.4% received a CCI and 3.5% died compared to 48.2% and 13.4% respectively for qSOFA ≥2. The area under the receiver operating characteristic (AUROC) was similar whether qSOFA was used to predict receipt of CCI or in-hospital mortality (0.74 [95%CI 0.73, 0.74] vs. 0.71 [0.69, 0.72]). The AUROC of SIRS for CCI (0.69) and mortality (0.66) were lower than that for qSOFA (p<0.001 for both outcomes). The sensitivity of qSOFA for predicting CCI was 38%.

Conclusions

ED patients with suspected infection and low qSOFA scores frequently require CCIs. The misclassification of these patients as ‘low risk,’ in combination with the low sensitivity of qSOFA ≥2, may diminish the clinical utility of the qSOFA score for patients with suspected infection in the ED.

Introduction

The Society of Critical Care Medicine and European Society of Intensive Care Medicine Sepsis III Task Force recently defined sepsis as ‘life-threatening organ dysfunction caused by a dysregulated host response to infection.’(1) To allow for the clinical diagnosis of ‘life-threatening organ dysfunction,’ the Task Force identified a set of measurable criteria that utilized markers of organ dysfunction to risk-stratify patients on the basis of in-hospital mortality. Patients with an increase in the Sequential Organ Failure Assessment (SOFA) score of ≥2 or a quick-SOFA (qSOFA) score of ≥2 in the setting of suspected infection had a high predicted in-hospital mortality rate and could be considered ‘septic.’(2)

While a change in SOFA of ≥2 or a qSOFA of ≥2 were shown to discriminate well between patients who survive and those who expire during their hospital stay (2), the use of in-hospital mortality as a way of defining ‘life-threatening organ dysfunction’ accounts only for risk that may be modified by the care later received. A young patient with sepsis from pneumonia, for example, may ultimately receive intubation and mechanical ventilation but survive. Classifying this genre of patient as ‘low-risk’ based on a qSOFA <2 would seemingly imply that ICU-level interventions (e.g. mechanical ventilation) are unlikely to be needed. Thus, the calibration of these severity scores does not account for the interventions subsequently provided, which are associated with survival.

If qSOFA were utilized solely for prognostication of mortality and risk-adjusted outcome comparisons, the focus on the outcome of mortality would be acceptable, however these scores have been proposed for use in clinical decision making.(1,2) As such, the score should be optimally calibrated to an actionable endpoint of the clinical decision. In addition, as previously noted by our group, clinical decision tools and severity-of-illness scores are separate entities though they are often erroneously used interchangeably.(3) Whereas severity-of-illness scores are generally derived from the regression model that optimizes the combination of sensitivity and specificity (i.e. the area under the receiver operating characteristic [AUROC]), clinical decision tools must balance one desired outcome at the expense of another. This is especially true in the Emergency Department (ED) setting where early recognition and timely intervention has proven critical in a variety of disease states, including sepsis and septic shock. (4, 5)

In the present study, we introduce the novel clinical end-point of ‘received critical care intervention (CCI)’ to account for risk unmodified by inpatient care. In addition, we sought to determine the test characteristics of the Sepsis III clinical criteria (specifically qSOFA) and the Systemic Inflammatory Response Syndrome (SIRS) criteria as predictors of receipt of a CCI. We hypothesized that, despite a high overall AUROC, qSOFA would have a low sensitivity for CCI and that patients who are classified by qSOFA as ‘low-risk’ on the basis of a low predicted mortality will have a relatively high rate of receipt of CCI.

Material and Methods

Study Design and Cohort Selection

This was a single center, retrospective study at an urban tertiary care center. Patients admitted to the ED between January 2010 and December 2014 with suspected infection (defined by the collection of any microbial cultures and initiation of antibiotics within 24-hours of ED triage time) were included.

Data Collection

Demographic data, vital signs, and laboratory results were abstracted from the electronic medical record (EMR). Vital signs in the ED at the study center are not recorded as structured data in the EMR for some patients requiring rapid resuscitation. For cases where ED vital signs were missing, the investigators reviewed the scanned records and abstracted the vital signs from paper flowsheets. Vital signs considered outside of the physiologic range were interpreted as chart documentation errors and were considered missing. Medical comorbidities were determined using ICD-9 codes.(6)

Score Calculation

For calculation of qSOFA and SIRS, the worst values for each criterion measured in the ED (for vital signs) or in the first 24-hours after ED triage (for laboratory values) were used. In addition, to better reflect clinical practice, the first vital signs and laboratory values measured after ED arrival were substituted in the calculation of qSOFA and the below analyses were repeated.

As the Glasgow Coma Scale (GCS) is not routinely documented in the ED at the study site, an ICD-9 code suggesting altered mental status (AMS) (780.0, 780.09, 780.02, 780.97, 349.82, 348.31) documented by an ED clinician or a documented ED Chief Complaint suggesting AMS (e.g. altered mental status, confusion, change in mental status etc.) was substituted. Given that this approach may have lower sensitivity to detect AMS than using GCS, a modified qSOFA (mqSOFA) was calculated substituting a lactate ≥2mmol/L for AMS.

Outcomes

The primary outcome was ‘received CCI’ within 48-hours of ED triage. Interventions classified as CCIs were determined by review of the literature (7, 8, 9) and modified for patients with suspected infection by a consensus of the authors. CCIs included receipt of vasopressor agents, receipt of assisted ventilation (either invasive or non-invasive), receipt of a continuous insulin infusion, receipt of ≥4,000mL intravenous fluid within 12-hours of ICU admission, placement of invasive catheters, or renal replacement therapy (see Table e1). To ensure accuracy, 25 cases were reviewed manually by a physician (AM). Patients initially admitted to a ward level of care but subsequently transferred to an ICU and provided a CCI were captured as having received a CCI. Therefore, any CCI was included regardless of initial admission location.

Statistical Analysis

Descriptive data are presented as means with standard deviations or medians with interquartile ranges depending on the distribution of the data. Categorical data are presented as counts with frequencies. Between group comparisons were made with chi-square tests for categorical data and two-sample t-tests or wilcoxon rank sum tests for continuous data as appropriate.

Model discrimination was determined by the area under the receiver operating characteristic (AUROC). Sensitivities and specificities were calculated with cut-points of qSOFA ≥2, mqSOFA ≥2, and SIRS ≥2 as has been previously suggested. (2) Model calibration was assessed for qSOFA using observed vs. predicted plots for the outcomes of CCI and mortality separately (Figure e2). Performance of the qSOFA score was determined both before and after adjustment for baseline risk (including age, gender, and selected medical comorbidities) using multivariable logistic regression. Variables in the baseline risk model were similar to those used in the original Sepsis III derivation.(2) AUROCs were compared using a test of equality of ROC areas obtained from applying two (or more) test modalities to the same sample. In addition to AUROC, the continuous net reclassification improvement (cNRI), a category-free version of the NRI, was determined for qSOFA vs. SIRS when used to predict the outcomes of CCI and mortality. (10)

Information regarding missing data can be found in Table e3. For the above analyses, standard single value imputation was performed with substitution of normal values for missing values. A two-tailed p-value <0.05 was considered significant. This study was approved by the Institutional Review Board at Beth Israel Deaconess Medical Center. All statistics were performed using STATA, version 14 (College Station, TX, StataCorp LP, USA).

Results

Cohort Characteristics

A total of 24,164 patients were included. The mean age of patients was 63.8 (±18.1) years old and 49.1% were female. Complete data regarding demographic information can be found in Table 1.

Table 1.

Baseline Cohort Characteristics

Characteristic	All Patients (n=24,164)	Received Critical Care Intervention (n=4,453)	Did not Receive Critical Care Intervention (n=19,711)	p-Value
Demographics
Age (mean years, SD*)	63.8 (18.1)	64.7 (17.2)	63.6 (18.3)	<0.001
Gender (%female)	49.1	46.2	49.8	<0.001
Vital Signs – mean, SD
First HR (bpm*)	91.8 (20.4)	96.5 (23.7)	90.7 (19.4)	<0.001
Highest HR (bpm)	95.7 (20.8)	103.9 (24.4)	93.8 (19.4)	<0.001
First Systolic Blood Pressure (mmHg)	129.2 (25.2)	120.1 (29.3)	131.2 (23.7)	<0.001
Lowest Systolic Blood Pressure (mmHg)	118.2 (23.9)	103.7 (26.1)	121.5 (22.2)	<0.001
First Temperature (°F*)	98.7 (1.7)	98.5 (2.2)	98.7 (1.5)	<0.001
Highest Temperature (°F)	99.2 (1.7)	99.2 (2.3)	99.2 (1.6)	0.29
First Respiratory Rate (bpm)	18.9 (4.2)	20.6 (6.1)	18.5 (3.5)	<0.001
Highest Respiratory Rate (bpm)	20.8 (5.3)	24.1 (7.0)	20.1 (4.5)	<0.001
Altered Mental Status (%)	8.9	13.8	7.7	<0.001
Maximum Laboratory Values – Mean, SD
WBC* (x103)	12.0 (10.3)	15.6 (14.8)	11.2 (8.8)	<0.001
Bilirubin (mg/dL)	1.0 (2.0)	1.4 (3.2)	0.9 (1.7)	<0.001
Creatinine (mg/dL)	1.5 (1.6)	2.0 (2.0)	1.4 (1.5)	<0.001
Lactate (mmol/L)	1.9 (1.5)	3.1 (2.6)	1.7 (0.9)	<0.001
Medical Comorbidities
Congestive Heart Failure	20.7	31.4	18.2	<0.001
Liver Disease	10.1	13.1	9.4	<0.001
Renal Disease	22.5	25.4	21.8	<0.001
COPD*	25.4	29.6	24.4	<0.001
Diabetes	30.7	34.4	30.0	<0.001
Dementia	1.0	1.4	0.9	0.02
Malignancy	14.8	13.2	15.2	<0.001

^*SD=standard deviation, bpm=beats per minute, F=fahrenheit, WBC=white blood cell count, COPD=chronic obstructive pulmonary disease

Overall, 5,268 (21.8%) patients were admitted from the ED to the intensive care unit (ICU). An additional 1,425 patients (5.9%) were initially admitted to a ward-level of care but were subsequently transferred to the ICU within 48-hours. Therefore, a total of 6,693 (27.7%) patients with suspected infection were admitted to an ICU within 48-hours of ED triage. Of those admitted to the ICU within 48-hours, 4,453 (66.5%) received a CCI. See Figure 1 for the related flow diagram. The most commonly received CCIs were central venous line placement, intubation, and receipt of vasopressors. The receipt of a CCI was tightly correlated with mortality both in unadjusted analysis (p<0.0001) and after controlling for age, serum lactate, and qSOFA score (aOR 5.6, 95%CI 4.9,6.5, p<0.0001). There was also a tight correlation between receipt of CCI and mortality in the low risk (qSOFA <2) group (p<0.0001). Additional information regarding CCIs can be found in Supplemental Figures e4 and e5.

Figure 1.

Cohort Selection Flow Diagram

Primary Analysis

The distributions of qSOFA and SIRS scores can be found in Supplemental Figure e6. In the full cohort, 3,495 (14.5%) patients had a qSOFA of ≥2 and 13,952 (57.5%) patients had a SIRS of ≥2. Of those patients with a qSOFA of ≥2, 1,684 (48.2%) received a CCI and 467 (13.4%) died. Among those with qSOFA <2, 2,769 (13.5%) patients received a CCI and 726 (3.5%) died—for patients with a qSOFA of 1, 1,864 (23.5%) received a CCI and 487 (6.1%) died. Of those with SIRS of ≥2, 3,578 (25.7%) of patients received a CCI and 969 (6.9%) died while of those with SIRS <2 875 (8.6%) received a CCI and 224 (2.2%) died. The receipt of CCI and mortality increased in a stepwise fashion with increasing qSOFA, mqSOFA, and SIRS scores as shown in Figure 2.

Figure 2.

Rate of CCI and In-Hospital Mortality by qSOFA and SIRS Scores

The AUROC when using qSOFA to predict CCI was 0.74 (95%CI: 0.73, 0.74) and 0.71 (0.69, 0.72) when used to predict in-hospital mortality. When using mqSOFA, the AUROCs were 0.77 (0.76, 0.78) and 0.74 (0.73,0.76) for CCI and mortality respectively. The AUROC of SIRS to predict CCI was 0.68 (0.68, 0.69) and, when used to predict mortality, 0.66 (0.65, 0.68). The AUROC of qSOFA and mqSOFA to predict receipt of CCI was significantly higher than that for SIRS (p<0.001) (Figure 3). For predicting CCI, the cNRI when qSOFA is used in place of SIRS was 36.8% (SE 0.02, p<0.0001). Similarly, the cNRI when qSOFA is used in place of SIRS to predict mortality was 31.8% (SE 0.03, p<0.0001).

Figure 3.

AUROC for qSOFA, mqSOFA, and SIRS to Predict CCI

When the first measured vital signs and laboratory values were used to calculate the qSOFA score, the AUROC for qSOFA to predict receipt of CCI and mortality were both significantly lower than when the worst values (as described in the Methods section) were used (0.67 [0.66,0.68] vs. 0.74 [0.73, 0.74] and 0.66 [0.64, 0.67] vs. 0.71 [0.69, 0.72], p<0.001 for both comparisons).

The sensitivity of qSOFA ≥2 was low when used to predict both receipt of CCI and mortality (38% and 39% respectively). The sensitivity of qSOFA ≥2 to predict receipt of CCI when the earliest patient information was used fell to 13%. The specificity of qSOFA for CCI was 91% and was 87% for mortality. Please see Table 2 for complete test characteristics for each score and outcome.

Table 2.

Model Diagnostic Characteristics

Model	AUROC		Sensitivity*		Specificity*
	Critical Care Intervention	Mortality	Critical Care Intervention	Mortality	Critical Care Intervention	Mortality
qSOFA	0.74	0.71	38%	39%	91%	87%
mqSOFA	0.77	0.74	57%	60%	84%	79%
qSOFA – First Values	0.67	0.66	13%	14%	97%	96%
SIRS	0.69	0.66	80%	82%	47%	44%
SIRS – First Values	0.65	0.64	66%	68%	57%	54%

^*For a score ≥2

Adjustment for Baseline Risk

When baseline risk was included in the model in addition to qSOFA, the AUROC improved slightly to 0.76 (0.76,0.77) for predicting receipt of CCI and 0.77 (0.76, 0.78) for predicting mortality. In the multivariate model including baseline risk, increasing age and the presence of malignancy were associated with a lower rate of CCI but a higher rate of mortality. For each increase per decade in age, the odds for receiving a CCI fell by 8% (aOR 0.92; 0.90, 0.94; p<0.001) whereas the odds of in-hospital mortality increased by 23% (aOR 1.23; 1.18–1.28; p<0.001). See Supplemental Table e7 for complete results of multivariable regression.

Discussion

In the present study of ED patients with suspected infection admitted to the hospital, we found that although mortality was low (<5%) for patients with qSOFA or SIRS <2, many patients with qSOFA or SIRS <2 received CCIs. Among patients in this cohort with a qSOFA of 1, for instance, 23.5% received a CCI. We further demonstrate that while the discriminatory power of qSOFA is similar when used to predict either receipt of CCI or in-hospital mortality, the sensitivity of qSOFA for CCI and mortality are quite low. Notably, when the earliest measured clinical values were used to calculate qSOFA (as opposed to the worst values over a certain time period), the sensitivity of a qSOFA score ≥2 was just 13% for predicting need for CCI.

The Sepsis III definition of sepsis is ‘life-threatening organ dysfunction caused by a dysregulated host response to infection,’ and the qSOFA has been demonstrated to be a reasonable predictor of hospital mortality in patients with sepsis by this definition. (1) This prediction is, however, only for how likely the patient is to die, not for how likely they are to need aggressive CCIs that could prevent their death. As we have shown in our analysis, 13.4% of patients with a qSOFA of 0 or 1 received a CCI within 48-hours but just 3.5% died. Classifying a patient as ‘low risk’ on the basis of a low predicted mortality clearly does not account for risk modified by inpatient care.

Along these same lines, while qSOFA more accurately classified patients with respect to outcomes as compared to SIRS (whether comparing model performance by AUROC or reclassification techniques), the low sensitivity of qSOFA to predict receipt of CCI is problematic for a tool being considered for use as an early identifier of impending clinical deterioration.(11, 12) In a recent qSOFA external validation study, the sensitivity of qSOFA ≥2 for in-hospital mortality was only 70% as compared to 93% for SIRS ≥2.(13) Similarly, Chuprek et. al. found that SIRS ≥2 had a sensitivity of 91% as compared to 54% for qSOFA ≥2. (14) It is notable that in our cohort, the sensitivity of qSOFA ≥2 when measured at triage and used to predict receipt of CCI was just 13%. In the ED, where timely recognition and intervention insepsis is paramount (4, 5), the low sensitivity of the qSOFA score (despite a relatively high AUROC when compared to SIRS) highlights the limitations of using AUROC alone when selecting a model to assist with clinical decision making.

The Sepsis III task force suggested that qSOFA be used as a “simple bedside criteria to identify adult patients with suspected infection who are likely to have poor outcomes” and “suggests that qSOFA criteria be used to prompt clinicians to further investigate for organ dysfunction, to initiate or escalate therapy as appropriate, and to consider referral to critical care or increase the frequency of monitoring.”(1, 2) The Surviving Sepsis Guidelines, in response to the release of the Sepsis III definitions, added that “in patients who have screened positive for infection, [qSOFA] may be used as a secondary screen to identify patients at risk for clinical deterioration.” (11) As demonstrated by our results, however, the implementation of qSOFA for the purposes of clinical decision making may result in delayed care escalation, incorrect triage decisions, and ultimately harm for some patients who are likely to survive but will receive a CCI.

To our knowledge, this is the first study to explore factors associated with receipt of CCI amongst patients with suspected infection. Interestingly, increasing age and presence of malignancy, both generally strong predictors of increased mortality amongst patients with infection, were negative predictors of receipt of CCI. This may be due to the fact that young patients are more likely to receive a CCI and survive as compared to more elderly patients who might die despite maximal, aggressive care. The relationship between presence of malignancy and CCI may reflect a cohort of patients with advanced directives (e.g. DNR/DNI) that preclude certain interventions, however our study was not designed to address this.

The strengths of our study include the large sample size, relatively low rate of missing data, and availability of a high-temporal resolution electronic ICU database. The identification of ‘CCI’ as a more proximal and actionable end-point than all-cause mortality is novel and may be useful for the future derivation of clinical decision tools in sepsis. While other validation studies have included need for ICU stay as either a secondary outcome or part of a composite outcome (13, 14, 15), the decision to triage a patient to the ICU might be influenced by a high severity-of-illness score, which may not always predict the care a patient actually requires. (16) Highlighting this, more than a third of patients admitted to the ICU in our cohort did not require a CCI.

Our study has a number of limitations. Foremost, use of ICD-9 codes documented in the ED may have a lower sensitivity for AMS than using GCS score (i.e. not all patients with GCS<15 will get an ICD-9 diagnosis of AMS). It is notable, however, that lack of GCS data was a problem faced by the Sepsis III Task Force where over 75% of patients outside of the ICU in the derivation cohort did not have GCS scores measured. These investigators imputed normal values for GCS scores without recorded measurements. To counter the potential low sensitivity of our AMS measurement, we substituted lactate ≥2mmol/L for GCS score to calculate an mqSOFA. It is notable that our mqSOFA model (including lactate ≥2) yielded an AUROC quite similar to that achieved by qSOFA outside the ICU in the initial Sepsis III derivation studies. As GCS appears to be a commonly missing variable outside of the ICU, substitution of lactate ≥2 may be a way for many hospitals that routinely measure lactate to query retrospective data for research and quality improvement initiatives in the Sepsis III era. Finally, there are likely additional ICU-specific interventions/practices that were not included in our specified ‘CCIs’ (e.g. nurse-to-patient ratio).

Conclusions

Utilization of a risk-stratification scheme based on in-hospital mortality may fail to adequately identify ED patients with infection who ultimately survive but are in need of early, CCIs. This finding, in combination with the relatively low sensitivity of qSOFA≥2, should give pause to clinicians seeking to use qSOFA as a tool for clinical decision making in the ED.