Epidemiology of Quick Sequential Organ Failure Assessment Criteria in Undifferentiated Patients and Association With Suspected Infection and Sepsis

Abstract

Background

The role of Quick Sequential Organ Failure Assessment (qSOFA) criteria in sepsis screening and management is controversial, particularly as they were derived only in patients with suspected infection. We examined the epidemiology and prognostic value of qSOFA in undifferentiated patients.

Methods

We identified patients with ≥ 2 qSOFA criteria within 1 day of admission among all adults admitted to 85 US hospitals from 2012 to 2015 and assessed for suspected infection (using clinical cultures and administration of antibiotics) and sepsis (as defined on the basis of Sepsis-3 criteria). We also examined the discrimination of qSOFA for in-hospital mortality among patients with and without suspected infection, using regression models.

Results

Of 1,004,347 hospitalized patients, 271,500 (27.0%) were qSOFA-positive on admission. Compared with qSOFA-negative patients, qSOFA-positive patients were older (median age, 65 vs 58 years), required ICU admission more often (28.5% vs 6.5%), and had higher mortality (6.7% vs 0.8%) (P < .001 for all comparisons). Sensitivities of qSOFA for suspected infection and sepsis were 41.3% (95% CI, 41.1%-41.5%) and 62.8% (95% CI, 62.4%-63.1%), respectively; positive predictive values were 31.0% (95% CI, 30.8%-31.1%) and 17.4% (95% CI, 17.2%-17.5%). The area under the receiver operating characteristic curve for mortality was lower for qSOFA in patients with suspected infection vs those without (0.814 vs 0.875; P < .001).

Conclusions

Only one in three patients who are qSOFA-positive on admission has suspected infection, and one in six has sepsis. qSOFA also has low sensitivity for identifying suspected infection and sepsis, and its prognostic significance is not specific to infection. More sensitive and specific tools for sepsis screening and risk stratification are needed.

FOR EDITORIAL COMMENT, SEE PAGE 197

Sepsis is a leading cause of death and disability.¹ Timely treatment can reduce the risk of mortality, but early recognition of sepsis is often challenging.² The systemic inflammatory response syndrome (SIRS) criteria have historically been used to screen for possible sepsis, but they have long been criticized for their lack of specificity.^{3, 4} In 2016, the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3) proposed the Quick Sequential Organ Failure Assessment (qSOFA) score as an alternative strategy to efficiently identify patients with suspected infection at risk for poor outcomes.⁵ The qSOFA criteria were developed and validated in large datasets based on superior predictive validity and discrimination for mortality compared with SIRS.^{6, 7, 8}

Despite its prognostic value, the appropriate role of qSOFA in sepsis screening, diagnosis, and triggering of empiric antibiotics remains confusing and controversial.^{9, 10, 11, 12} One challenge is that qSOFA has been evaluated primarily in patients already suspected to have infection. It therefore remains unclear whether qSOFA criteria should be used to alert clinicians to possible sepsis in undifferentiated patients. The Sepsis-3 task force “considered that positive qSOFA criteria should also prompt consideration of possible infection in patients not previously recognized as infected” but did not provide data to support this recommendation.⁵ Furthermore, the few studies that have explicitly examined the sensitivity or specificity of qSOFA for sepsis have been small, single-center studies or used sepsis diagnoses as a reference standard, which themselves have low sensitivity and are variably applied by clinicians.^{13, 14, 15, 16} Finally, it is unclear whether the prognostic significance of qSOFA extends to patients without suspected infection.¹⁷

In this study, we sought to inform the role of qSOFA in sepsis identification and risk stratification by examining its epidemiology and prognostic value in patients with and without suspected infection, using clinical data from a diverse set of hospitals.

Methods

Study Design, Population, and Data Source

We retrospectively analyzed electronic health record (EHR) data for adults (age, ≥ 20 years) admitted between January 2012 and September 2015 to hospitals participating in the Cerner Health Facts data set. This data set draws from academic and community hospitals throughout the United States and contains both administrative data and detailed clinical data.¹⁸ Our starting cohort included adult inpatient encounters from 119 hospitals previously used in a national epidemiologic study of sepsis that reported laboratory, microbiology, and medication data.¹⁸ A subset of these hospitals also routinely reported vital signs and Glasgow Coma Scale (GCS) measurements. To maximize the data quality for this analysis of qSOFA criteria, we first excluded hospitals where > 75% of encounters were missing GCS scores on admission or > 25% were missing systolic blood pressure or respiratory rate values, since it is likely that those hospitals do not systematically utilize GCS measurements or consistently record vital signs in their EHRs. Next, we excluded patients with unknown vital status at discharge. Last, we excluded any remaining patients with missing blood pressure or respiratory rate values within 1 day of admission. Within this analytic cohort, we then identified patients with and without ≥ 2 qSOFA criteria (systolic blood pressure ≤ 100 mm Hg, respiratory rate ≥ 22 breaths/min, or GCS score < 15) within one calendar day of admission. Missing GCS scores were assumed to be normal (score of 15).

Definitions for Suspected Infection, Sepsis, and Conditions Associated With qSOFA

We examined the prevalence and clinical characteristics associated with ≥ 2 qSOFA criteria on admission. We further assessed the test characteristics of qSOFA on admission for suspected infection and sepsis. Suspected infection was deemed present if clinicians obtained specimens for culture from any anatomic site and administered any duration of antibiotics starting within one calendar day of admission.⁶ Sepsis was defined, using Sepsis-3 criteria, as presumed infection with a Sequential Organ Failure Assessment (SOFA) score of ≥ 2 points within 1 day of admission.^{5, 6, 19} Presumed infection was deemed present if there was suspected infection on admission and antibiotics were continued for ≥ 4 days. The antibiotic regimen had to include at least one intravenous dose, and fewer days of antibiotics were permitted if the patient died, was discharged to hospice, or transferred to another acute care hospital within 4 days.¹⁸ The intravenous and 4-day antibiotic requirements were used to identify patients more likely to have serious infections and to exclude patients started on empiric antibiotics that were then stopped within 2 to 3 days once infection was no longer suspected. Because the Cerner Health Facts data set does not include vasopressor doses or urine output, we utilized the number of concurrent vasopressors for the cardiovascular SOFA score and creatinine alone for the renal SOFA score, as previously described.²⁰ We conducted sensitivity analyses using International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) discharge diagnosis codes (principal or secondary) for infection based on the method described by Angus et al²¹ and for sepsis, using the codes for septicemia (038), sepsis (995.91), severe sepsis (995.92), and septic shock (785.52).¹ We used multilevel diagnosis categories from the Agency for Healthcare Research and Quality (AHRQ) Clinical Classifications Software to classify discharge diagnoses.²²

Statistical Analyses

We compared discrimination for in-hospital mortality of ≥ 2 qSOFA criteria in patients with and without suspected infection on admission, in accordance with Seymour et al.⁶ A baseline model was created for the outcome of in-hospital death based on age, sex, race, and comorbidities among all patients with suspected infection on admission. To define comorbidities, we used the Elixhauser method as implemented by the AHRQ,^{23, 24} as prior studies suggest it has better predictive validity for mortality than other administrative comorbidity scores.^{25, 26} Within each decile of baseline risk of death, we compared mortality rates among patients with and without ≥ 2 qSOFA criteria. We assessed model discrimination with area under the receiver operating characteristic (AUROC) curves for death when qSOFA was added to the baseline risk model, with hospitals included as random effects.

All analyses were conducted with SAS version 9.3 (SAS Institute). All tests of significance used two-sided P values at ≤ .05. This study was approved by the Institutional Review Board at Partners HealthCare with a waiver of informed consent (Protocol #2016P001291).

Results

Study Cohort and Characteristics of qSOFA-Positive vs qSOFA-Negative Patients

The cohort derivation process is summarized in Figure 1. The final study cohort included 1,004,347 adult patients admitted to 85 hospitals from January 2012 to September 2015 with vital signs recorded within 1 calendar day of admission. The characteristics of the hospitals and patients removed from the analysis due to incomplete vital sign/GCS reporting or missing vital status on discharge are shown in e-Tables 1 and 2 in the online article. Excluded hospitals tended to be smaller, nonteaching hospitals. Excluded patients were similar with respect to age, comorbidity burden, hospital length of stay, frequency of suspected infection, and in-hospital mortality.

Figure 1.

Flowchart for study cohort derivation. GCS = Glasgow Coma Scale; qSOFA = Quick Sequential Organ Failure Assessment; RR = respiratory rate; SBP = systolic blood pressure.

Table 1.

Characteristics and Outcomes of Patients With and Without ≥ 2 qSOFA Criteria on Admission

Characteristics and Outcomes	≥ 2 qSOFA	< 2 qSOFA
	(n = 271,500)	(n = 732,847)
Median age, y (IQR)	65 (51-78)	58 (40-72)
Male sex,a No. (%)	121,192 (44.6)	299,704 (40.9)
Race,b No. (%)
White	211,928 (78.1)	538,385 (73.5)
Black	38,745 (14.3)	123,164 (16.8)
Other	20,827 (7.7)	71,298 (9.7)
Median AHRQ Elixhauser scorec (IQR)	5 (0-14)	0 (–1 to 8)
Elixhauser comorbidities, No. (%)
Chronic lung disease	64,238 (23.7)	117,696 (16.1)
Congestive heart failure	48,874 (18.0)	70,672 (9.6)
Diabetes,d	68,372 (25.2)	162,769 (22.2)
Neurologic disease	45,179 (16.6)	58,472 (8.0)
Cancere	19,799 (7.3)	38,695 (5.3)
Renal failuree	39,629 (14.6)	76,362 (10.4)
Median SOFA score on admission (IQR)	5 (0-14)	1 (0-2)
Median hospital LOS, d (IQR)	5 (3-8)	4 (2-5)
Required ICU admission,e No. (%)	77,252 (28.5)	47,299 (6.5)
Median ICU LOS, d (IQR)	3 (2-5)	2 (1-4)
In-hospital mortality, No. (%)	18,141 (6.7)	5,605 (0.8)

All comparisons between the qSOFA⁺ vs qSOFA^– groups were statistically significant, with P < .001. AHRQ = Agency for Healthcare Research and Quality; IQR = interquartile range; LOS = length of stay; qSOFA = Quick Sequential Organ Failure Assessment; SOFA = Sequential Organ Failure Assessment.

^aSex was missing for 128 patients (0.01%).

^bRace was missing for 16,860 patients (1.7%).

^cThe AHRQ Elixhauser Comorbidity Index score is weighted and allows for negative points for comorbidities with an inverse association with mortality.

^dDiabetes comorbidity includes diabetes with and without complications.

^eCancer comorbidity includes solid tumor, metastatic cancer, and lymphoma.

Table 2.

Sensitivity and Positive Predictive Value of qSOFA for Suspected Infection and Sepsis

qSOFA Sensitivity or Positive Predictive Value	Infection		Sepsis
	Suspected Infection on Admission	Infection Diagnoses on Discharge	Sepsis on Admission	Sepsis Diagnoses on Discharge
	(n = 203,378)	(n = 302,063)	(n = 75,140)	(n = 57,492)
Prevalence of ≥ 2 qSOFA points on admission in patients with infection or sepsis (qSOFA sensitivity)	84,028 (41.3%) [41.1%-41.5%]	109,624 (36.3%) [36.1%-36.5%]	47,175 (62.8%) [62.4%-63.1%]	36,046 (62.7%) [62.3%-63.1%]
Prevalence of infection or sepsis in patients with ≥ 2 qSOFA points on admissiona (qSOFA positive predictive value)	31.0% [30.8%-31.1%]	40.4% [40.2%-40.6%]	17.4% [17.2%-17.5%]	13.3% [13.2%-13.4%]

Brackets indicate 95% CIs calculated using binomial distributions. The raw counts for the bottom row cells are the same as for the top row. “Suspected infection on admission” was defined as any antibiotic administration and clinical culture sampling from any anatomic site within 1 d of admission. “Sepsis” was defined as presumed infection (suspected infection on admission + antibiotics continued for ≥ 4 d, or fewer if death, discharge to hospice, or transfer to another hospital occurred prior to 4 d, with at least one intravenous antibiotic dose) and a Sequential Organ Failure Assessment score of ≥ 2 within 1 d of admission. “Infection diagnoses on discharge” included one of 1,286 International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes for infection based on the method of Angus et al.²¹ “Sepsis diagnoses on discharge” included ICD-9-CM codes for septicemia (038), sepsis (995.91), severe sepsis (995.92), or septic shock (785.52). See Table 1 legend for expansion of abbreviation.

^aThe cohort included 271,500 patients with ≥ 2 qSOFA points on admission.

Of the 1 million hospitalizations, 271,500 (27.0%) were qSOFA-positive (with ≥ 2 qSOFA points) on admission (92.2% tachypnea, 85.0% hypotension, and 46.7% altered mental status). GCS was not measured in 238,330 (23.7%) of cases. The characteristics of hospitalized patients with and without qSOFA on admission are shown in Table 1. qSOFA-positive patients were older (median age, 65 vs 58 years; P < .001), had more comorbidities (median AHRQ Elixhauser score of 5 vs 0; P < .001), and had higher median SOFA scores on admission (median, 5 vs 1; P < .001) compared with qSOFA-negative patients. qSOFA-positive patients also required ICU admission more often (28.5% vs 6.5%; P < .001) and had higher in-hospital mortality rates vs qSOFA-negative patients (6.7% vs 0.8%; P < .001).

Relationship of qSOFA to Infection, Sepsis, and Other Diagnoses

Amongst the 271,500 qSOFA-positive patients, 84,028 (31.0%; 95% CI, 30.8%-31.1%) had suspected infection and 47,175 (17.4%; 95% CI, 17.2%-17.5%) met Sepsis-3 criteria on admission (Table 2). The sensitivities of qSOFA for suspected infection and sepsis were 84,028/203,378 (41.3%; 95% CI, 41.1%-41.5%) and 47,175/75,140 (62.8%; 95% CI, 62.4%-63.1%), respectively. qSOFA had a specificity of 76.6% (95% CI, 76.5%-76.7%) for suspected infection on admission and 75.9% (95% CI, 75.8%-76.0%) for sepsis. The negative predictive value of qSOFA for suspected infection was 83.7% (95% CI, 83.6%-83.8%) and 96.2% (95% CI, 96.1%-96.2%) for sepsis. Results were similar using discharge diagnosis codes for infection and sepsis (Table 2).

The 30 most common discharge diagnosis categories in patients who were qSOFA-positive on admission are shown in Table 3. The most common diagnosis categories overall were essential hypertension (38.7% of qSOFA-positive patients), disorders of lipid metabolism (29.6%), coronary atherosclerosis (23.4%), and diabetes mellitus without complication (20.9%). The most commonly potentially acute conditions were congestive heart failure (18.3%), atrial fibrillation (17.9%), and acute renal failure (17.1%). The only diagnoses indicative of infection among the top 30 diagnosis categories were septicemia (13.8%), urinary tract infection (12.4%), and pneumonia (12.2%).

Table 3.

Most Common Diagnoses in Patients With ≥ 2 qSOFA Criteria on Admission

Diagnosis Category	No. (%) of qSOFA-Positive Hospitalizations
Essential hypertension	105,137 (38.7)
Disorders of lipid metabolism	80,360 (29.6)
Coronary atherosclerosis	63,434 (23.4)
Diabetes mellitus without complication	56,674 (20.9)
Congestive heart failure	49,797 (18.3)
Atrial fibrillation	48,671 (17.9)
Acute renal failure	46,495 (17.1)
Other esophageal disorders	44,039 (16.2)
Other/unspecified lower respiratory disease	43,844 (16.1)
Chronic kidney disease	41,419 (15.3)
Codes related to substance-related disorders	40,496 (14.9)
Respiratory failure	39,032 (14.4)
Unspecified septicemia	37,558 (13.8)
Obesity	36,431 (13.4)
Codes related to mental health disorders	36,195 (13.3)
Hypertensive heart and/or renal disease	35,694 (13.1)
Other connective tissue disease	35,625 (13.1)
Congestive heart failure; nonhypertensives	35,162 (13.0)
Depressive disorders	34,843 (12.8)
Other and unspecified circulatory disease	34,839 (12.8)
Other fluid and electrolyte disorders	34,546 (12.7)
Hypopotassemia	34,160 (12.6)
Other thyroid disorders	34,136 (12.6)
Anemia, unspecified	34,020 (12.5)
Urinary tract infection; site not specified	33,741 (12.4)
Chronic airway obstruction; not otherwise specified	33,553 (12.4)
Pneumonia; organism unspecified	33,072 (12.2)
Other forms of chronic heart disease	31,101 (11.5)
Delirium, dementia, and amnestic and other cognitive disorders	30,627 (11.3)
Other nervous system symptoms and disorders	329,628 (10.9)

Discharge diagnoses were classified according to the Agency for Healthcare Research and Quality Clinical Classifications Software. See Table 1 legend for expansion of abbreviation.

Prognostic Accuracy of qSOFA in Patients With and Without Suspected Infection

qSOFA-positive patients with suspected infection on admission had higher crude in-hospital mortality rates (9,223/84,028; 11.0%) vs qSOFA-positive patients without suspected infection (8,918/187,472; 4.8%) (P < .001). Patients with suspected infection who were qSOFA-positive on admission had a 4- to 52-fold increase in the adjusted odds of in-hospital death across baseline risk deciles; this increase in risk of death with qSOFA was generally slightly higher than in patients without suspected infection (Fig 2). However, the overall discrimination for in-hospital death on top of the baseline risk model was lower for qSOFA amongst patients with suspected infection (AUROC, 0.814; 95% CI, 0.811-0.818) vs patients without suspected infection (AUROC, 0.875; 95% CI, 0.873-0.878) (P < .001 for comparison). Discrimination for death was also lower for qSOFA in patients with vs without infection codes on discharge (AUROC, 0.741 vs 0.816; P < .001 for comparison). When restricting the analysis to patients not in the ICU on admission, results were similar, with lower discrimination for mortality with qSOFA among patients with suspected infection vs patients without suspected infection (AUROC, 0.823 vs 0.872; P < .001).

Figure 2.

Fold change in rate of in-hospital mortality by deciles of baseline risk of death for ≥ 2 qSOFA criteria vs < 2 qSOFA criteria in patients with and without suspected infection on admission. The x axis divides the cohort into deciles of baseline risk, which were created on the basis of age, sex, race, and Elixhauser Comorbidity Index. The y axis shows the fold increase in the odds of death (log scale) for a patient with suspected infection or without suspected infection who meets ≥ 2 qSOFA criteria for each decile of risk. For example, a patient who falls into the 5th decile of baseline risk (based on moderate burden of comorbidities) with suspected infection (eg, pneumonia) has an approximately 10-fold increased odds of death if he has ≥ 2 qSOFA criteria vs < 2 qSOFA criteria. He has similarly increased odds of death with ≥ 2 qSOFA criteria vs < 2 qSOFA criteria even if he did not have suspected infection. See Figure 1 legend for expansion of abbreviation.

Discussion

In this large US cohort, we found that ≥ 2 qSOFA criteria were present within 1 day of admission in one in four hospitalized patients. qSOFA-positive patients tended to be older and sicker at baseline compared with qSOFA-negative patients, with substantially higher rates of ICU admission and death. Only one in three patients who were qSOFA-positive had suspected infection and only one in six had clinical indicators of sepsis by Sepsis-3 criteria, while qSOFA was absent in more than one-third of patients with sepsis. qSOFA was associated with an increased risk of mortality both in patients with and without suspected infection.

The Sepsis-3 derivation analyses and most subsequent external validations of qSOFA have been conducted in patients already suspected to have infection.^{6, 7, 8, 27} There are few prior data detailing the prevalence of qSOFA criteria in undifferentiated patients. A retrospective analysis of 19,670 ED patients in a large academic hospital also found that qSOFA had a low positive predictive value (12%) for sepsis requiring ICU admission.²⁸ A prospective cohort study of 258 patients who triggered rapid response teams found that 43% met qSOFA criteria and only one-half were presumed to be infected.²⁹ Our findings expand on these studies and underscore the fact that hypotension, tachypnea, and altered mental status are common in conditions other than infection or sepsis. In our cohort, the most common potentially acute conditions in patients who were qSOFA-positive on admission, such as congestive heart failure, atrial fibrillation, and acute renal failure, were not clearly related to infection. The low positive predictive value of qSOFA suggests that it has limited value on its own, without other clinical signs of infection, in informing the need for immediate empiric antibiotics for possible sepsis.^{9, 10, 30, 31, 32}

The low sensitivity of qSOFA for infection and sepsis observed in our study also calls into question its role in screening undifferentiated patients for sepsis. Most prior studies have compared qSOFA criteria with SIRS or other early warning scores in terms of their predictive accuracy for short-term mortality,³³ but a recent meta-analysis also suggested that the sensitivity of qSOFA for sepsis was low, ranging from 10% to 54%.¹⁵ Other studies have also demonstrated that qSOFA criteria are met later in the course of sepsis than SIRS.^{14, 34} One retrospective analysis of undifferentiated ED patients, for example, found that qSOFA criteria were only present at a median of 0.7 h after triage in patients who were ultimately admitted to the ICU with sepsis.²⁸ Our study builds on and extends this literature by applying clinical criteria that closely matched the infection and SOFA criteria used by the Sepsis-3 task force to a very large and diverse cohort of patients from a large number of hospitals.

Three prior single-center studies, in different settings (prehospital, ED, and non-ICU wards) demonstrated moderate to strong performance of qSOFA for discriminating in-hospital mortality in both infected and noninfected patents.^{17, 35, 36} Our analysis confirms that qSOFA is strongly associated with poor outcomes regardless of infection status and shows consistency of this association across most deciles of baseline risk. Taken together, our findings indicate that qSOFA should not be used as a sepsis-specific risk assessment tool, but rather as a general marker of illness in high-risk patients who might require close clinical attention. This does not obviate its value as a high-risk marker in patients with suspected infection but does provide broader context for the use and interpretation of qSOFA in clinical practice. Whether or not qSOFA merits prioritization over existing early warning scores, such as the National Warning Score or Modified Early Warning Score, remains a topic of active debate that this study did not address.^{35, 37}

The findings of this study must be interpreted in the context of its limitations. First, we used a convenience sample of hospitals that use the Cerner electronic health record system, and our findings may not be generalizable to other hospitals. However, our cohort was large and drew from academic and community hospitals distributed around the country. Second, a substantial number of patient encounters were removed from the analysis because of insufficient data. However, the excluded patients were comparable to those included in the analysis, making a major systematic bias unlikely. Third, we looked for qSOFA criteria within 1 day of admission, but clinicians and hospitals may screen patients using physiologic values within a shorter timeframe. However, the time windows we used are narrower than the 72-h window used in the primary Sepsis-3 analyses.⁶ Fourth, we did not compare the performance of qSOFA with SIRS or other early warning scores as we felt this has been carefully evaluated already in prior studies.^{13, 34, 35, 38, 39} Fifth, the AHRQ diagnosis categories we used to identify the most common conditions associated with qSOFA do not always clearly distinguish acute vs chronic conditions (eg, congestive heart failure or atrial fibrillation). Last, any attempt to estimate the accuracy of qSOFA or other criteria for sepsis is limited by the lack of a reliable gold standard.^{16, 40} However, we utilized both clinical Sepsis-3 criteria and diagnosis code strategies to identify patients with sepsis and found similar results.

In conclusion, we found that only one in three patients with qSOFA on admission had evidence of suspected infection, only one in six met Sepsis-3 criteria, and qSOFA was negative in one-third of patients with sepsis. In addition, we found that qSOFA is associated with higher mortality in all hospitalized patients, not just those with suspected infection. These findings suggest that qSOFA has limited utility in sepsis screening and diagnosis and may be better considered a general marker of severe illness and impending clinical deterioration in all patients. There remains a pressing need to develop new screening tools that are both sensitive and specific for sepsis.