Comparison of different versions of the quick sequential organ failure assessment for predicting in-hospital mortality of sepsis patients: A retrospective observational study

Abstract

BACKGROUND

The quick sequential organ failure assessment (qSOFA) is recommended to identify sepsis and predict sepsis mortality. However, some studies have recently shown its poor performance in sepsis mortality prediction. To enhance its effectiveness, researchers have developed various revised versions of the qSOFA by adding other parameters, such as the lactate-enhanced qSOFA (LqSOFA), the procalcitonin-enhanced qSOFA (PqSOFA), and the modified qSOFA (MqSOFA). This study aimed to compare the performance of these versions of the qSOFA in predicting sepsis mortality in the emergency department (ED).

METHODS

This retrospective study analyzed data obtained from an electronic register system of adult patients with sepsis between January 1 and December 31, 2019. Receiver operating characteristic (ROC) curve analyses were performed to determine the area under the curve (AUC), with sensitivity, specificity, and positive and negative predictive values calculated for the various scores.

RESULTS

Among the 936 enrolled cases, there were 835 survivors and 101 deaths. The AUCs of the LqSOFA, MqSOFA, PqSOFA, and qSOFA were 0.740, 0.731, 0.712, and 0.705, respectively. The sensitivity of the LqSOFA, MqSOFA, PqSOFA, and qSOFA were 64.36%, 51.40%, 71.29%, and 39.60%, respectively. The specificity of the four scores were 70.78%, 80.96%, 61.68%, and 91.62%, respectively. The LqSOFA and MqSOFA were superior to the qSOFA in predicting in-hospital mortality.

CONCLUSIONS

Among patients with sepsis in the ED, the performance of the PqSOFA was similar to that of the qSOFA and the values of the LqSOFA and MqSOFA in predicting in-hospital mortality were greater compared to qSOFA. As the added parameter of the MqSOFA was more convenient compared to the LqSOFA, the MqSOFA could be used as a candidate for the revised qSOFA to increase the performance of the early prediction of sepsis mortality.

INTRODUCTION

Sepsis is one of the leading causes of mortality worldwide, characterized by infection-induced physiological, pathological, and biochemical abnormalities resulting in significant healthcare and social implications.[1–4] In the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3)[1], experts recommend the use of quick sequential organ failure assessment (qSOFA) for the diagnosis of sepsis and screening of patients with severe sepsis who are at higher risk of mortality.[1,5]

The criteria of the qSOFA to predict poor outcomes in adult patients with sepsis are a respiratory rate of ≥22 breaths per minute, systolic blood pressure of ≤100 mmHg (1 mmHg=0.133 kPa), and altered mentation. The qSOFA is fast and straightforward, making it appropriate for emergency department (ED) use. However, recent studies have shown the poor performance of qSOFA in predicting sepsis mortality.[6–9] To enhance its effectiveness, researchers have developed various revised versions of the qSOFA by including additional parameters, including the lactate-enhanced qSOFA (LqSOFA),[10] the procalcitonin (PCT)-enhanced qSOFA (PqSOFA),[11] and the modified qSOFA (MqSOFA).[12]

LqSOFA assigned a lactate level of ≥2 mmol/L one point based on the qSOFA score assignment method. Similarly, the PqSOFA assigned an additional point if PCT was at a threshold of >0.5 ng/mL. Likewise, the MqSOFA assigned an additional point calculated from the ratio of peripheral oxygen saturation and a fraction of inspired oxygen (SpO₂/FiO₂ ratio); a score of 0 was assigned if the SpO₂/FiO₂ ratio was ≥315, 1 point if 314–236, and 2 points if ≤235. For the details of the revised versions of the qSOFA, see the Supplementary Table 1.

Table 1.

Comparison of the baseline characteristics of survivors and non-survivors

However, there are few studies verifying and comparing the effectiveness of these revised versions. Therefore, we aimed to evaluate these revised versions of the qSOFA as an in-hospital mortality indicator and for the prognosis of patients with sepsis.

METHODS

Study design

This retrospective study reviewed data obtained from an electronic register system of patients with sepsis who visited West China Hospital. The electronic register system was a part of Hospital Information System (HIS) in which emergency physicians recorded the information of patients.

The local Institutional Review Committee approved the study and waived the requirement for informed consent due to the retrospective nature of the study. The study complied with the international ethical guidelines for human research, such as the Declaration of Helsinki. The accessed data were anonymized.

Settings and subjects

This study included 974 adult patients diagnosed with sepsis in the ED of West China Hospital between January 1 and December 31, 2019. Patients with the following characteristics were included: (1) age ≥18 years and; (2) diagnosis of sepsis (based on the Sepsis-3 criteria,[1] sepsis was defined as life-threatening organ dysfunction caused by the infection. Organ dysfunction was identified as a sequential organ failure assessment (SOFA) score ≥2. The exclusion criteria were: (1) lack of information which hindered the calculation of scores; (2) lack of outcome information; and (3) patients who were intubated when they arrived at the ED.

For all included cases, the following data were retrieved from the electronic register system: demographic characteristics; inital vital signs (body temperature, heart rate, systolic and diastolic blood pressure, and respiratory rate); inital mental status (alert, verbal, pain, unresponsive [AVPU] system); inital oxygen saturation level, initial bedside lactate; initial PCT; other parameters used to calculate the scoring system, and final diagnosis. The outcome was death or survival at discharge. The observation point for survival calculations was set as the discharge time of the last admitted patient.

Statistical analysis

Data analysis was conducted using the SPSS 20.0 (IBM, USA) and MedCalc 18.2.1 (MedCalc Software Ltd., Belgium). Continuous variables are presented as means with standard deviations or median with interquartile range and were compared using the Student’s t-test or Mann-Whitney U-test. Categorical variables are described as number and composition ratios and were compared using the Chi-square test or Fisher’s exact test (when the expected value was <5 in one cell).

Receiver operating characteristic (ROC) curve analyses were conducted to evaluate the effectiveness of different score systems as predictors of mortality. The performance of these scoring systems to discriminate between survivors and non-survivors was evaluated by calculating the area under the curve (AUC) of ROC with a 95% confidence interval. The AUCs were compared using Delong’s approach.[13,14] The cut-off value for each scoring scale was determined as the maximum value of Youden’s index.

Finally, the corresponding accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) related to the optimal cut-off point for each score were calculated. A P-value <0.05 was considered statistically significant.

RESULTS

Among the 974 patients, 38 were excluded due to missing data. In total, 936 patients were included in the final analysis of the study (Figure 1).

Figure 1.

Flow chart of case exclusions.

Among the enrolled cases, there were 835 survivors and 101 deaths. The age of survivors (50.13±18.32 years) was significantly younger than that of the non-survivors (58.91±20.92 years) (P<0.05). Moreover, the mental status, oxygen saturation, serum lactate, qSOFA, MqSOFA, LqSOFA, and PqSOFA were significantly different between the two groups (P<0.05). However, no significant difference was observed on sex, temperature, respiratory rate, systolic blood pressure, and PCT between the two groups (Table 1).

Among the four score systems, ROC curve analysis found that LqSOFA was the best predictor of in-hospital mortality of sepsis patients, with an AUC of 0.740, followed by MqSOFA, PqSOFA, and qSOFA with AUC of 0.731, 0.712, and 0.705, respectively (Table 2, Figure 2).

Table 2.

The performance of scores with different cut-offs in predicting in-hospital mortality of sepsis patients

Figure 2.

ROC curves of different versions of the qSOFA in predicting in-hospital mortality. ROC: receiver operating characteristic; qSOFA: quick sequential organ failure assessment; LqSOFA: lactate-enhanced qSOFA; MqSOFA: modified qSOFA; PqSOFA: procalcitonin- enhanced qSOFA.

Based on the maximum Youden’s index, an optimum cut-off value of 2 was used to predict in-hospital mortality using the LqSOFA, with a sensitivity, specificity, PPV, and NPV of 64.36%, 70.78%, 21.04%, and 94.26%, respectively. The optimum cut-off value of MqSOFA was 2 with sensitivity, specificity, PPV, and NPV of 51.49%, 80.96%, 24.65%, and 93.24%, respectively. Likewise, an optimum cut-off value of 2 was used in the PqSOFA, with sensitivity, specificity, PPV, and NPV of 71.29%, 61.68%, 18.37%, and 94.67%, respectively. The optimum cut-off value of qSOFA was 2 points with sensitivity, specificity, PPV, and NPV of 39.60%, 91.62%, 36.36%, and 92.62%, respectively (Table 2).

There were significant differences of AUCs between the LqSOFA and qSOFA and between the MqSOFA and qSOFA (both P<0.05). However, no other significant difference was observed (Table 3).

Table 3.

Pairwise comparison of AUC of scores for predicting in-hospital mortality of sepsis patients

DISCUSSION

The recent increase in sepsis mortality has raised the awareness of the importance of early risk stratification among emergency physicians. Since 2016, early identification and in-hospital mortality prediction of patients with sepsis in the ED with qSOFA has been recommended.[1] The simplicity and availability of the qSOFA has warranted its wide use in the ED.[15]

However, our study showed that the sensitivity and specificity of the qSOFA in predicting in-hospital mortality in patients with sepsis were 39.60% and 91.62%, respectively, indicating a weak sensitivity. Our result is consistent with recent studies that emphasized the poor performance of qSOFA in the prediction of sepsis mortality.[16, 17] A retrospective study by Moskowitz et al[18] found that the qSOFA had a sensitivity and specificity of 39% and 87%, respectively, when used in predicting in-hospital mortality at the time of ED presentation among 24,164 patients with sepsis. Similarly, a prospective study by Askim et al[19] on 1,535 adult patients found that the qSOFA had a sensitivity and specificity of 16% and 96%, respectively, in predicting mortality at the time of ED presentation. Other studies found that the sensitivity of the qSOFA in predicting mortality ranged from 13% to 90%, while the specificity ranged from 86% to 96%.[8, 20–22]

These studies indicated acceptable qSOFA specificity but unsatisfactory sensitivity, which was consistent with our results. However, high sensitivity is more important than high specificity in predicting sepsis mortality.[23] This is due to the higher risk and cost of false negativity compared to false positivity in patients with high mortality risk.[23, 24]

Based on previous studies, researchers have recently added simplely available indices to the qSOFA to improve sensitivity. In a retrospective study of 836 patients, Liu et al[10] introduced bedside serum lactate and put forward the LqSOFA, which was shown to be superior to the qSOFA. Our findings were consistent with their study, indicating the superiority of the LqSOFA compared to the qSOFA (P<0.05). Additionally, previous studies have shown the effectiveness of lactate as a predictor of sepsis mortality;[25–27] therefore, lactate will expectedly increase the effectiveness of the qSOFA in predicting sepsis mortality.[28, 29] This addition to the qSOFA for comprehensive scoring could help emergency physicians to make clinical decisions by combining two indicators instead of using one in isolation.

The SpO₂/FiO₂ ratio is associated with respiratory status; some studies have included it as a key parameter for sepsis mortality.[30,31] In a retrospective study of 1,137 cases, Guarino et al[12] introduced the SpO₂/FiO₂ ratio into the qSOFA and developed a modified qSOFA (MqSOFA), which showed better accuracy than the qSOFA. Our study also suggested that the performance of MqSOFA was better than qSOFA. Therefore, the MqSOFA may be a viable tool in predicting sepsis mortality.

Xia et al[11] added PCT to the qSOFA and developed the PqSOFA, which was indicated to have a higher sensitivity but lower specificity in predicting sepsis mortality compared to qSOFA in a retrospective study of 821 patients. However, no significant difference was observed between the AUCs of the PqSOFA and the qSOFA in our research. Similarly, previous studies have inconsistent results regarding the effectiveness of PCT as a predictor of sepsis mortality.[32–35] Therefore, further studies are required to determine the utility of the PqSOFA as a revised scoring system for the early prediction of sepsis.

According to our results, the best among the four score systems was the LqSOFA, followed by the MqSOFA, PqSOFA, and qSOFA. Generally, the revised qSOFAs obtained better sensitivity and accuracy in predicting the mortality of sepsis patients than the qSOFA; however, the results remain unsatisfactory. Among the revised versions of the qSOFA, LqSOFA and MqSOFA had similar performance in predicting sepsis mortality (P>0.05). The additional parameters from these measures can be obtained rapidly and conveniently; therefore, the LqSOFA and MqSOFA can be used in the early assessment of mortality in the ED. SpO₂/FiO₂ ratio is more convenient than lactate; therefore, we suggest that the MqSOFA can be used as a candidate for the revised qSOFA to increase the performance of early prediction of sepsis mortality. Furthermore, both the LqSOFA and PqSOFA require a bedside test to be calculated. The MqSOFA was the only altered version of the qSOFA, which contains completely clinical information that makes this score helpful in pre-hospital settings.

This study had several limitations. Firstly, the study population was small. Further studies employing a large sample will provide a higher level of evidence for assessing these scores. Additionally, pre-hospital care was neglected because of unavailable information. Secondly, because of the nature of the retrospective study, we identified altered mentation using the AVPU system. Indeed, altered mentation in the qSOFA should be explained as a different mental status of the patient. Even so, since the altered mentation of each score uses the AVPU system, we believe that this has little effect on the comparison of each score. However, this may affect the test characteristics of each score. Therefore prospective research should be carried out in the future. Finally, our single-center study may cause selection bias. Further multi-center studies are required to validate the performance of these revised qSOFAs prospectively.

CONCLUSIONS

Among the three revised qSOFAs, the PqSOFA had a higher sensitivity and lower specificity than the qSOFA; however, their performances were similar. Moreover, the values of the LqSOFA and MqSOFA in predicting in-hospital mortality were greater compared to the qSOFA. As the added parameter of the MqSOFA was more convenient compared than that of the LqSOFA, the MqSOFA is a potential candidate for the revised qSOFA to increase the performance of the early prediction of sepsis mortality.