Evaluation of repeated qSOFA measurements among patients with suspected infection

Abstract

Objective:

Among patients with suspected infection, a single measurement of qSOFA has good predictive validity for sepsis, yet the increase in validity from repeated measurements is unknown. We sought to determine the incremental predictive validity for sepsis of repeated qSOFA measurements over 48 hours compared to the initial measurement.

Design:

Retrospective cohort study.

Setting:

Twelve hospitals in southwestern Pennsylvania in 2012.

Patients:

All adult medical and surgical encounters in the emergency department, hospital ward, post-anesthesia care unit, and ICU.

Interventions:

None.

Measurements and Main Results:

Among 1.3 million adult encounters, we identified those with a first episode of suspected infection. Using the maximum qSOFA score in each 6-hour epoch from onset of suspected infection until 48 hours later, we characterized repeated qSOFA with i.) summary measures (e.g., mean over 48 hours), ii.) crude trajectory groups, and iii) group-based trajectory modeling (GBTM). We measured the predictive validity of repeated qSOFA using incremental changes in the area under the receiver operating characteristic (AUROC) curve for in-hospital mortality beyond that of baseline risk (age, sex, race/ethnicity, and co-morbidity). Of 37,591 encounters with suspected infection, 1,769 (4.7%) died before discharge. Both the mean qSOFA at 48 hours (AUROC=0.86 [95% CI: 0.85–0.86]) and crude trajectory groups (AUROC=0.83 [95% CI: 0.83–0.83]) improved predictive validity compared to initial qSOFA (AUROC=0.79 [95% CI: 0.78–0.80]) (p<0.001 for both). GBTM found five trajectories (qSOFA always low, increasing, decreasing, moderate, and always high) with greater predictive validity than the initial measurement (AUROC=0.85 [95% CI: 0.84–0.85], p<0.001).

Conclusions:

Repeated measurements of qSOFA improve predictive validity for sepsis using in-hospital mortality compared to a single measurement of qSOFA at the time a clinician suspects infection.

The Third International Consensus Task Force defined sepsis as an acute dysregulated host response to infection leading to life-threatening organ dysfunction (Sepsis-3) (1). The Task Force explored the predictive validity of a number of clinical criteria for sepsis in multiple clinical datasets. Balancing several aspects of predictive validity, reliability, and practicality, they recommended criteria for sepsis as an increase of 2 points or more in the Sequential Organ Failure Assessment (SOFA) score in response to infection (2–4). However, for patients outside the ICU, a set of 3 simple measures available without laboratory tests (respiratory rate ≥22 breaths per minute, systolic blood pressure ≤100mm Hg, and Glasgow Coma Scale [GCS] score <15) had similar predictive validity for sepsis compared to the SOFA score. Termed the quick Sepsis-related Organ Failure Assessment (qSOFA), these signs may be a useful clinical prompt to help clinicians identify those patients with suspected infection most likely to be septic. Subsequent studies have confirmed the predictive validity of a single qSOFA measured with outcomes like hospital mortality (5–9).

Yet, important knowledge gaps about qSOFA remain. First, the predictive validity of repeated measurements is unknown and has not been tested in prior cohorts (6–9). Similar to other bedside tools (10–12), repeated measurements of qSOFA may help clinicians find patients with infection who are at greatest risk of sepsis, while using only objective, routinely measured vital signs. These measurements may better capture the evolving host response and organ dysfunction in sepsis. Therefore, re-examining the cohort in whom the qSOFA score was developed, we studied if repeated measurements of qSOFA in the 48 hours after suspected infection improve predictive validity for sepsis compared to a single measurement alone, and whether repeated measurements can identify distinct clinical trajectories of qSOFA.

Methods

The institutional review board of the University of Pittsburgh approved this study with waiver of informed consent. These data have been reported previously in abstract form (13).

Study design, setting, and population

As reported previously (4), we performed a retrospective cohort study among adult hospital encounters (age ≥18 years) with suspected infection in 2012 at 12 community and academic hospitals in southwestern Pennsylvania. We included all medical and surgical encounters in the emergency department (ED), hospital ward, post-anesthesia care unit (PACU), and intensive care unit (ICU). For each encounter, we abstracted demographic, disposition, and administrative data, as well as time- and location-stamped vital signs, laboratory results, and orders (e.g., body fluid cultures, medications, etc.) from the electronic health record (EHR). We defined a cohort with suspected infection using the first combination of a body fluid culture order (e.g., blood, urine, cerebrospinal fluid, etc.) and at least one dose of antibiotics (oral or parenteral) within a specified time frame (4). Please see the online supplement for more details.

Measurement of qSOFA

We evaluated the first 48 hours after suspected infection in 6-hour epochs from the occurrence of the first culture or antibiotic event (time zero). We chose 6-hours epochs a priori, as this provides a pragmatic window in the EHR for re-assessment and measurement of qSOFA variables. In each epoch, we calculated the maximum qSOFA score as previously reported (4), using the most parsimonious model accounting for GCS score of less than 15, systolic blood pressure of 100 mmHg or less, and respiratory rate of 22/min or more (1 point each; score range, 0 to 3). A qSOFA score ≥2 was considered high, a qSOFA of 1 was moderate, and qSOFA of 0 was low. The qSOFA score during the first 6-hour epoch after time zero was considered the initial qSOFA. Encounters missing an initial qSOFA score were excluded.

Trajectory of qSOFA in the 48 hours after the digital signature of the suspicion of infection

We evaluated the trajectory of qSOFA in three ways: i.) simple summary measures, ii.) crude trajectory groups of qSOFA, and iii) group-based trajectory modeling (GBTM). First, we calculated the mean qSOFA over 48 hours and the maximum qSOFA at both 24 and 48 hours. Second, we determined six groups corresponding to the crude trajectory of qSOFA using a threshold of 2 or more qSOFA points at the initial and maximum measurements. Crude groups included: I.) initial qSOFA=0, maximum qSOFA<2; II.) initial qSOFA=0, maximum qSOFA≥2; III.) initial qSOFA=1, maximum qSOFA<2; and IV.) initial qSOFA=1, maximum qSOFA≥2; V.) initial qSOFA=2 or 3, maximum qSOFA<2; and VI.) initial qSOFA=2 or 3, maximum qSOFA≥2. Third, we used GBTM to further explore the trajectory of qSOFA, and further describe these methods in the online supplement (14).

Outcome

To compare the predictive validity of repeated qSOFA, we assessed the gain in discrimination of in-hospital deaths not predicted by baseline risk (i.e., the gain in ability to discriminate those deaths not predicted by age, sex race/ethnicity, and pre-existing co-morbidity), as such unexpected deaths are presumed to be more common among infected patients who are septic than among infected patients who are not.

Statistical analysis

For descriptive statistics, we calculated mean (SD) and median [IQR] as appropriate to normality for continuous variables, and used frequency (proportions) for categorical variables. To illustrate the general behavior of qSOFA over the 48 hours after suspected infection, we created heat maps of qSOFA among a randomly selected subset stratified by initial qSOFA (N=3,538). Please see the online supplement for additional details on the generation of the heat map cohort.

To compare the predictive validity of repeated measurements of qSOFA, we followed Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) recommendations (15). We evaluated results among all eligible encounters, as well as a priori restricted to encounters with suspected infection outside the ICU in a pre-planned subgroup analysis. First, we assessed the frequency and pattern of missing qSOFA data by each 6-hour epoch (eTable 1, Supplemental Digital Content 1). Then to account for information already available to clinicians at the bedside when they recognize infection, we created a baseline model for in-hospital mortality using multivariable logistic regression that included age (as a fractional polynomial), sex, race/ethnicity (black, white, or other), and the weighted Charlson comorbidity score (as fractional polynomial) as a measure of chronic co-morbidities (16, 17). Race/ethnicity was derived from clinical documentation in the EHR.

Next, we determined predictive validity of the baseline model and after adding: i.) the initial qSOFA, ii.) simple summary measures of qSOFA over time, and iii.) crude trajectory groups of qSOFA. We report the adjusted odds ratios (OR) with 95% confidence intervals (95% CI) and changes in the area under the receiver operating characteristic curve (AUROC) from baseline risk for each qSOFA measure. For the purposes of understanding whether a measure improved predictive validity, we considered a >10% relative gain in discrimination to be a significant improvement between models. We determined relative gains based on the residual uncaptured discrimination (defined as 1 – AUROC) between a new model and the comparator. For example, if a baseline risk model’s AUROC is 0.7, then the uncaptured residual discrimination is 0.3. A model incorporating a new measure with an AUROC of 0.74 would thus capture 13% of the residual uncaptured discrimination compared to the baseline model.

Sensitivity analysis

We performed multiple sensitivity analyses to assess the robustness of our findings. These included assessment of the predictive validity of qSOFA measurements across deciles of baseline risk, changes to the cohort, when adding limitations in life-sustaining therapies at the time of suspected infection to the baseline model, and use of multiple imputation for missing data. Please see the online supplement for more details.

All analyses were performed with STATA 14.2 (StataCorp, College Station, TX) and used a two-sided p≤0.05 for all tests of significance. Heat maps were generated in R 3.2.3 using the “heatmap3” package. GBTM was performed using the STATA traj plugin.

Results

Cohort and patient characteristics

Among 1,309,025 encounters (eFigure 1, Supplemental Digital Content 1), 48,319 had suspected infection. After excluding those missing an initial qSOFA (n=10,728), we studied 37,591 encounters, of whom 1,769 (4.7%) died in hospital. Compared to survivors, those that died were older, more frequently male, and more likely to have suspected infection occur in the ICU (p<0.001 for all, Table 1). Encounters that died were less likely to have onset of infection within 48 hours of admission, had greater serum lactate measurements on the day of suspected infection, and were more likely to require intensive care compared to survivors (p<0.001 for all). Compared to the analysis cohort, encounters excluded due to missing an initial qSOFA were younger, less frequently male, more likely to have suspected infection occur in the ED, had lower SIRS and SOFA scores, and had lower overall mortality (n=82; 0.8%) [eTable 2, Supplemental Digital Content 1].

Table 1.

Patient characteristics

Variable		All patients	Survivors	Deaths	p-valueb
Total patients, no.		37,591	35,822	1,769
Age, years, mean (SD)		62.5 (18.3)	62.2 (18.3)	69.0 (15.2)	<0.001
Male gender, no. (%)		16,635 (44)	15,711 (44)	924 (52)	<0.001
Race or ethnicity, no. (%)					0.073
	White	28,724 (76)	27,350 (76)	1,374 (78)
	Black	4,596 (12)	4,400 (12)	196 (11)
	Other	4,271 (11)	4,072 (11)	199 (11)
Weighted Charlson sum, mean (SD)		1.3 (1.7)	1.3 (1.7)	1.8 (2.0)	<0.001
Surgery prior to infection suspected, no. (%)		6,801 (18)	6,550 (18)	251 (14)	0.094
Onset of infection within 48 hours of admission, no. (%)		32,428 (86)	30,964 (86)	1,474 (83)	<0.001
Unit location at the time infection suspected, no. (%)					<0.001
	ED	16,265 (43)	15,677 (44)	588 (33)
	Ward	11,903 (32)	11,588 (32)	315 (18)
	ICU	4,726 (13)	4,003 (11)	723 (41)
	PACU or Procedure unit	548 (1)	540 (2)	8 (0)
	Stepdown unit	4,095 (11)	3,962 (11)	133 (8)
	Other or missing	54 (0)	52 (0)	2 (0)
SIRS day of infection suspected, mean (SD)		1.4 (1.1)	1.4 (1.1)	2.3 (1.1)	<0.001
	median [IQR]	1 [1, 2]	1 [1, 2]	2 [1, 3]	<0.001
SOFA score day of infection suspected, mean (SD)		2.1 (2.8)	1.9 (2.5)	6.1 (4.8)	<0.001
	median [IQR]	1 [0, 3]	1 [0, 3]	5 [2, 9]	<0.001
Serum lactate measured on day of infection suspected, no. (%)		4,883 (13)	4,041 (11)	842 (48)	<0.001
Serum lactate ≥2.0 mmol/L, no. (%) if measured		2,122 (6)	1,540 (4)	582 (33)	<0.001
Limitation of life-sustaining therapies at time infection suspected, no. (%)a		917 (2)	833 (2)	84 (5)	<0.001
Outcomes
	ICU admission, no. (%)	11,152 (30)	9,578 (27)	1,574 (89)	<0.001
	Hospital length of stay, median [IQR] days	7 [4, 11]	6 [4, 11]	9 [5, 16]	<0.001

^aLimitation of life-sustaining therapy defined as do-not-resuscitate, do-not-intubate, or comfort care order in place at time of suspected infection

^bp-value for survivors compared to deaths

Abbreviations: ED – emergency department, ICU – intensive care unit, IQR – interquartile range, PACU – post-anesthesia care unit, SD – standard deviation, SIRS – systemic inflammatory response syndrome, SOFA – Sequential Organ Failure Assessment.

Repeated measurement of qSOFA

Most initial qSOFA values were low (qSOFA=0, n=25,337; 67%), one quarter were moderate (qSOFA=1, n=9,801; 26%), and only a small proportion were high (qSOFA=2, n=2,292; 6%; qSOFA=3, n=161; <1%). The mean initial qSOFA was greater among encounters that died than those who survived (eTable 3, Supplemental Digital Content 1), and remained higher during the 48-hour period after suspected infection (Figure 1). When the initial qSOFA was low, it usually remained low or moderate (n=24,136; 95%) and overall mortality was <2%. When the initial qSOFA was moderate, one quarter (n=2,494; 25%) increased to high and incurred a mortality of 16%, whereas mortality remained low (4%) for the remainder who remained low (n=7,307; 75%). When the initial qSOFA was high, it usually remained high (n=1,925; 78%) and overall mortality was 25%. A minority of those with initially high qSOFA decreased to low (n=528; 25%), but still had a 11% mortality (Table 2).

Figure 1.

Heat map of qSOFA in 6-hour time windows after the onset of infection, from 0 to 48 hours, stratified by initial qSOFA (columns) and outcome (deaths [N=1,769], Panel A vs survivors [N=1,769], Panel B)

Interpretive example: Patients with low initial qSOFA scores (qSOFA=0, bottom panel, first column) who survive are much more likely to remain at low qSOFA values (i.e., have green or missing values) during the 48 hours after infection. In comparison, patients with high risk initial qSOFA scores (qSOFA≥2, top panel, third and fourth columns) who go on to die are much more likely to have qSOFA values that are higher during the 48 hours after infection (i.e., more pink and red scores).

Abbreviations: qSOFA – quick Sepsis-related Organ Function Assessment

Table 2.

Rule of thumb for qSOFA trajectory and outcome based on all encounters (N=37,591)

Initial qSOFAa	How common is this when infection is suspected?	What is the expected course?b	What is the outcome?	Simple management steps
HIGH (≥2)	Infrequent (1 in 14)	Most remain HIGH (4 of 5)	If remain HIGH, 1 in 4 die; if decrease to LOW, 1 in10 die	Manage aggressively
MODERATE (=1)	Common (1 in 4)	One in four increase to HIGH (1 in 4)	If become HIGH, nearly 1 in 5 die; otherwise, mortality is 4%	Watch carefully
LOW (=0)	Very common (2 in 3)	Very few will become HIGH (1 in 20)	If become HIGH, 1 in 10 die; otherwise, mortality is <2%	Monitor for improvement

^aCorresponds to the initial qSOFA measured during the six hours after suspected infection

^bCorresponds to the maximum qSOFA measured from seven to 48 hours after suspected infection

Predictive validity of single and repeated qSOFA

When added to a baseline model of characteristics available to clinicians at the time of suspected infection, initial qSOFA was associated with an increase in the odds of death among all encounters (OR=3.58; 95% CI: 3.37–3.80) and those outside the ICU (OR=3.07; 95% CI: 2.81–3.34) [eTable 4, Supplemental Digital Content 1]. When summary measures of repeated qSOFA were added, the adjusted odds of death were greatest for the mean qSOFA score over 48 hours (OR=8.55; 95% CI: 7.91–9.24). When comparing crude qSOFA trajectory groups to a referent group with consistently low qSOFA, the adjusted odds of death were greatest among groups II.) initial qSOFA=0, maximum qSOFA≥2 (OR=8.76; 95% CI, 7.14–10.74), IV.) initial qSOFA=1, maximum qSOFA≥2 (OR=12.10; 95% CI, 10.40–14.09), and VI.) initial qSOFA=2 or 3, maximum qSOFA≥2 (OR=23.16; 95% CI 19.96–26.88). These associations were similar among encounters outside the ICU (eTable 4, Supplemental Digital Content 1).

The initial qSOFA improved the predictive validity for sepsis compared to the baseline model alone (AUROC for in-hospital mortality among all encounters=0.79; 95% CI: 0.78–0.80 vs 0.63; 95% CI: 0.62–0.65, p<0.001), a 43% reduction in uncaptured discrimination (Figure 2 and eTable 5, Supplemental Digital Content 1). Mean qSOFA over 48 hours had the greatest reduction in uncaptured discrimination compared to the baseline model alone (AUROC=0.86; 95% CI: 0.85–0.86, p<0.001, 62% reduction). Crude qSOFA trajectory groups also improved predictive validity compared to both the baseline model and initial qSOFA (p<0.001). Results were similar among encounters outside the ICU (Figure 2 and eTable 5, Supplemental Digital Content 1).

Figure 2.

Area under the receiver operating characteristic curve of repeated qSOFA measurements versus baseline risk model among all encounters (N=37,591; Panel A) and those that began outside the ICU (N=32,865; Panel B)

Abbreviations: AUROC – area under the receiver operating characteristic curve, ICU – intensive care unit, qSOFA – quick Sepsis-related Organ Function Assessment

Using group-based trajectory modeling, the best-fitting model had five groups: i.) low (n=13,317; 35%), ii.) increasing (n=3,208; 9%), iii.) decreasing (n=10,921; 29%), iv.) moderate (n=7,301; 19%), and v.) high (n=2,853; 8%) [Figure 3 and eTable 2, Supplemental Digital Content 1]. Crude mortality was greater in moderate (33%) and high trajectory groups (45%) and these groups had a greater adjusted odds of in-hospital mortality (eTable 6, Supplemental Digital Content 1). The predictive validity of GBTM qSOFA trajectories (AUROC=0.85; range, 0.84–0.85) was similar to that of the mean qSOFA and greater than that of both initial qSOFA and the crude trajectory groups (p<0.001; eFigure 2 and eTable 7, Supplemental Digital Content 1). Results were similar among encounters outside the ICU.

Figure 3.

Mean qSOFA score for each trajectory group among 37,591 encounters in the 48 hours after suspected infection. qSOFA trajectory groups are low (black), decreasing (grey), moderate (pink), increasing (blue), and high (red). Inset are odds ratios for in-hospital mortality for each group category compared to referent (low qSOFA trajectory group.

^aAdjusted for age, sex, race, and comorbidities

Abbreviations: OR – odds ratio, qSOFA – quick Sepsis-related Organ Function Assessment

Sensitivity analyses

In sensitivity analyses, repeated measurements of qSOFA improved predictive validity for sepsis across deciles of baseline risk compared to both the baseline model and initial qSOFA (eFigure 2). Analysis restricted to infections suspected in the ED (eTable 8, Supplemental Digital Content 1) also found similar odds ratios (eTable 4, Supplemental Digital Content 1) and predictive validity (eTable 5, Supplemental Digital Content 1). Model performance was unchanged when incorporating limitations in life-sustaining therapies in the baseline model (eTables 9 and 10, Supplemental Digital Content 1). Complete case analysis found similar results for both odds ratios (eTable 11, Supplemental Digital Content 1) and predictive validity (eFigure 3, Supplemental Digital Content 1). Similarly, in multiple imputations, both odds ratios (eTable 6, Supplemental Digital Content 1) and predictive validity for in-hospital mortality were unchanged (eTable 7, Supplemental Digital Content 1).

Discussion

In an integrated health system, repeated measures of qSOFA during the 48 hours after suspected infection had a greater predictive validity for sepsis using in-hospital mortality than a baseline measurement alone. These findings were consistent among encounters outside the ICU and in multiple sensitivity analyses. Encounters with a high or increasing qSOFA were at greater risk than those with low or decreasing qSOFA trajectory. Taken together, these findings suggest that repeated measurements of qSOFA could help prompt clinicians about patients more likely to be septic.

There are several reasons why repeated measurement of qSOFA can improve predictive validity for sepsis among patients with suspected infection. On one hand, individuals may present for care at different stages in their episode of illness. And among those who present at the same time, there may be evolving host response and organ injury on different timelines. In both cases, additional measurement of data over time may account for the dynamics of illness course and the effect of treatment on organ dysfunction. Our findings support recent literature confirming that qSOFA improves predictive validity for in-hospital mortality compared to SIRS (5–9), but add to them by showing an incremental benefit using repeated measurements.

Our findings have important clinical implications. The Third International Sepsis Definitions Task Force recommended qSOFA of more than 2 points as a prompt to suspect sepsis outside the ICU. A significant advantage of qSOFA is that it requires no labs or tests. Yet, a threshold of 2 or more points has a low sensitivity (47–70%) (5, 6, 8, 9), and may miss patients at high risk for sepsis if measured at a single time. Our data suggest greater predictive validity with repeated measurements (Figure 2 and eTable 5, Supplemental Digital Content 1). As shown in Table 2, patients with initial high qSOFA who decrease to low have reduced mortality (11% mortality) compared to those who remain high (25% mortality). Such data could be used in family discussions, de-escalation of interventions, or assist with down-triage decisions in resource-constrained settings. Those with initial moderate qSOFA who increase to high (16% mortality) could prompt up-triage or initiation of organ support therapies (18). Yet if a moderate qSOFA decreases (4% mortality), therapies might be focused with greater precision (e.g., avoiding fluid over-resuscitation) or de-escalated more rapidly. Repeated measures may also lead to greater intensity of monitoring, particularly among those outside the ICU, and may be scalable inside the electronic health record. Lastly, our data may have particular importance for sepsis care in low- and middle-income countries, where repeated measurements of vital signs may be the only way to monitor patients over time (19).

From a research perspective, these data suggest that trajectory modeling may identify clusters of patients at greater risk for sepsis. The GBTM technique identified five distinct groups of septic encounters. Similar to the efforts of other groups (20), these clusters may provide a method to identify different pathophysiology of sepsis, leading to more clues about the underlying mechanisms of host response and organ dysfunction. They may be useful in future studies evaluating the heterogeneity of treatment effects of new therapies aimed at preventing organ dysfunction and shock (21). However, group-based modeling trajectory groups require additional validation in separate cohorts.

This study has several limitations. First, because there is no gold standard for sepsis in the electronic health record, we used an outcome more common in septic patients than not--in-hospital mortality. This approach is now used in multiple qSOFA validation studies (5–9). Second, missing data was common, particularly among encounters discharged prior to 48 hours after infection. However, in both complete cases analysis and after multiple imputations our results were similar. Third, we only used data in an integrated health system in southwestern Pennsylvania, and our results may not be generalizable to regions with different sepsis case mix or in low-middle income countries. Fourth, the effects of interventions may impact the trajectory of qSOFA. Future studies which model ongoing treatment and location status in the hospital (e.g., transfer from ward to ICU after infection suspected) on qSOFA trajectories are warranted. Fifth, our study did not compare the predictive validity of qSOFA trajectories to other repeated measures, such as the full SOFA score, which may not have complete laboratory values at frequent intervals.

Conclusions

Among encounters with suspected infection, the predictive validity of repeated qSOFA measurements for sepsis using in-hospital mortality was greater than a single measurement alone.