Dear Editor,
We read the study by Liu et al. with great interest [1]. We believe that the authors' evaluation of the predictive performance of the Sequential Organ Failure Assessment (SOFA) and Quick Sequential Organ Failure Assessment (qSOFA) scores for hospital mortality in patients with severe and critical coronavirus 2019 (COVID-19) infection can be more easily applied at the bedside using likelihood ratios. Likelihood ratios are a measure of diagnostic accuracy that are derived from sensitivity and specificity. When patients test positive, the post-test probability that patients have the disease increases as the value of the positive likelihood ratio (LR+) becomes greater. For example, LRs+ of 2, 5, and 10 increase post-test probability by roughly 15, 30%, and 45% respectively (though the increment in post-test probability decreases as it approaches 100%). Conversely, when patients test negative, the post-test probability that patients have the disease decreases as the value of the negative likelihood ratio (LR-) becomes smaller. LRs- of 0.5, 0.2, and 0.1 decrease post-test probability by roughly 15, 30, and 45% respectively (though the increment in post-test probability decreases as it approaches 0) [2]. We calculated the LR+ and LR- of SOFA and qSOFA scores for hospital mortality in patients with severe and critical COVID-19 infection using the results from Liu et al.'s study (Table 1 ).
Table 1.
Likelihood ratios of SOFA and qSOFA as predictors for in-hospital mortality of severe/critical COVID-19.
| Models | Cutoff value | Sn (%) | Sp (%) | LR+ (95% CI) | LR- (95% CI) |
|---|---|---|---|---|---|
| All | |||||
| SOFA | 1 | 100 | 50.47 | 2.02 (1.61, 2.41) | Not calculated |
| 2 | 95 | 71.96 | 3.39 (2.46, 4.66) | 0.07 (0.01, 0.47) | |
| 3 | 90 | 83.18 | 5.35 | 0.12 (0.03, 0.45) | |
| 4 | 70 | 87.85 | 5.76 (3.21, 10) | 0.34 (0.17, 0.67) | |
| 5 | 55 | 94.39 | 9.8 (4.10, 23) | 0.48 (0.29, 0.78) | |
| 6 | 25 | 98.13 | 13 (2.79, 64) | 0.76 (0.59, 0.99) | |
| 7 | 20 | 98.13 | 11 (2.08, 53) | NS | |
| 8 | 20 | 100 | Not calculated | 0.8 (0.63, 0.99) | |
| qSOFA | 1 | 70 | 80.37 | 3.57 (2.21, 5.76) | 0.37 (0.19, 0.73) |
| 2 | 0 | 97.2 | Not calculated | NS | |
| 3 | 0 | 100 | Not calculated | Not calculated | |
| Age < 65 years | |||||
| SOFA | 1 | 100 | 54.24 | 2.19 (1.72, 2.64) | Not calculated |
| 2 | 100 | 71.19 | 3.47 (2.49, 4.55) | Not calculated | |
| 3 | 100 | 81.36 | 5.36 (3.47, 7.66) | Not calculated | |
| 4 | 66.67 | 86.44 | 4.92 (2.78, 8.69) | 0.39 (0.21, 0.72) | |
| 5 | 33.33 | 93.22 | 4.92 (1.93, 13) | 0.72 (0.52, 0.98) | |
| 6 | 33.33 | 98.31 | 20 (4.09, 95) | 0.68 (0.50, 0.93) | |
| 7 | 33.33 | 98.31 | 20 (4.09, 95) | 0.68 (0.50, 0.93) | |
| 8 | 33.33 | 100 | Not calculated | 0.67 (0.49, 0.90) | |
| qSOFA | 1 | 66.67 | 76.27 | 2.81 (1.77, 4.45) | 0.44 (0.23, 0.82) |
| 2 | 0 | 96.61 | Not calculated | NS | |
| 3 | 0 | 100 | Not calculated | Not calculated | |
| Age ≥ 65 years | |||||
| SOFA | 1 | 100 | 45.83 | 1.85 (1.50, 2.17) | Not calculated |
| 2 | 94.12 | 72.92 | 3.48 (2.50, 4.83) | 0.08 (0.01, 0.47) | |
| 3 | 88.24 | 85.42 | 6.05 (3.72, 9.84) | 0.14 (0.04, 0.46) | |
| 4 | 70.59 | 89.58 | 6.77 (3.63, 13) | 0.33 (0.17, 0.65) | |
| 5 | 58.82 | 95.83 | 14 (5.30, 38) | 0.43 (0.25, 0.73) | |
| 6 | 23.53 | 97.92 | 11 (2.47, 52) | NS | |
| 7 | 17.65 | 97.92 | 8.4 (1.69, 42) | NS | |
| 8 | 17.65 | 100 | Not calculated | NS | |
| qSOFA | 1 | 70.59 | 85.42 | 4.84 (2.82, 8.30) | 0.34 (0.17, 0.68) |
| 2 | 0 | 97.92 | Not calculated | NS | |
| 3 | 0 | 100 | Not calculated | Not calculated | |
Sn: Sensitivity, Sp: Specificity, LR+: Positive Likelihood Ratio, LR-: Negative Likelihood Ratio, CI: Confidence Interval, NS: Not Significant.
Not Calculated: LRs were not reported if there were no patients in one or more groups in the 2 × 2 table constructed to calculate the value because this would create an LR of 0 or infinity, which is unlikely.
A SOFA score ≥ 3 is a moderately good positive predictor (LR+ 5.35, 95% confidence interval [CI] 3.43 to 8.36) and a strong negative predictor (LR- 0.12, 95% CI 0.03 to 0.45) for hospital mortality. Conversely, a qSOFA score ≥ 1 is a less positive predictor (LR+ 3.57, 95% CI 2.21 to 5.76) and a moderately good negative predictor (LR- 0.37, 95% CI 0.19 to 0.73). Using the pre-test probability for hospital mortality of (20/127) or 15.7% from the paper by Liu et al., the post-test probability of mortality in patients with SOFA ≥3 would be 49.9% and 2.2% for patients with a SOFA score 0 to 2, while the post-test probability for hospital mortality would be 39.3% in patients with qSOFA score ≥ 1 and 6.4% for patients with a qSOFA score of 0.
There are two key limitations that should be considered before applying these results in clinical practice. First, the likelihood ratios that we calculated may be overestimates as the data from Liu and colleagues were derived from an exploratory analysis of a single-center retrospective cohort using a data-driven approach [3]. A larger prospective cohort study to validate the optimal cutoff values reported in their study is warranted. Second, the strength of both the SOFA and qSOFA as predictors of mortality may be different today as new treatments have been shown to reduce mortality in patients with COVID-19 infection since this study was performed [4].
Declaration of Competing Interest
The authors declare no competing sources of interest.
References
- 1.Liu S., Yao N., Qiu Y., He C. Predictive performance of SOFA and qSOFA for in-hospital mortality in severe novel coronavirus disease. Am J Emerg Med. 2020;38(10):2074–2080. doi: 10.1016/j.ajem.2020.07.019. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.McGee S. Simplifying likelihood ratios. J Gen Intern Med. 2002;17(8):646–649. doi: 10.1046/j.1525-1497.2002.10750.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Leeflang M.M., Moons K.G., Reitsma J.B., Zwinderman A.H. Bias in sensitivity and specificity caused by data-driven selection of optimal cutoff values: mechanisms, magnitude, and solutions. Clin Chem. 2008;54(4):729–737. doi: 10.1373/clinchem.2007.096032. [DOI] [PubMed] [Google Scholar]
- 4.Dennis J.M., McGovern A.P., Vollmer S.J., Mateen B.A. Improving survival of critical care patients with coronavirus disease 2019 in England: a National Cohort Study, March to June 2020. Crit Care Med. 2021;49(2):209–214. doi: 10.1097/CCM.0000000000004747. [DOI] [PMC free article] [PubMed] [Google Scholar]