Abstract
Background
The COVID-19 pandemic continues with new waves that could persist with the arrival of new SARS-CoV-2 variants. Therefore, the availability of validated and effective triage tools is the cornerstone for proper clinical management. Thus, this study aimed to assess the validity of the ISARIC-4C score as a triage tool for hospitalized COVID-19 patients in Saudi Arabia and to compare its performance with the CURB-65 score.
Material and methods
This retrospective observational cohort study was conducted between March 2020 and May 2021 at KFHU, Saudi Arabia, using 542 confirmed COVID-19 patient data on the variables relevant to the application of the ISARIC-4C mortality score and the CURB-65 score. Chi-square and t-tests were employed to study the significance of the CURB-65 score and the ISARIC-4C score variables considering the ICU requirements and the mortality of COVID-19 hospitalized patients. In addition, logistic regression was used to predict the variables related to COVID-19 mortality. In addition, the diagnostic accuracy of both scores was validated by calculating sensitivities, specificities, positive predictive value, negative predictive value, and Youden's J indices (YJI).
Results
ROC analysis showed an AUC value of 0.834 [95% CI; 0.800–0.865]) for the CURB-65 score and 0.809 [95% CI; 0.773–0.841]) for the ISARIC-4C score. The sensitivity for CURB-65 and ISARIC-4C is 75% and 85.71%, respectively, while the specificity was 82.31% and 62.66%, respectively. The difference between AUCs was 0.025 (95% [CI; −0.0203-0.0704], p = 0.2795).
Conclusion
Study results support external validation of the ISARIC-4C score in predicting the mortality risk of hospitalized COVID-19 patients in Saudi Arabia. In addition, the CURB-65 and ISARIC-4C scores showed comparable performance with good consistent discrimination and are suitable for clinical utility as triage tools for hospitalized COVID-19 patients.
Keywords: COVID-19, SARS-CoV-2, ISARIC 4C Score, CURB-65 score, Triage
1. Introduction
COVID-19 is a disease caused by SARS-CoV-2 that emerged in Wuhan, China, in December 2019 and spread around the world, causing a pandemic with over 6.8 million deaths worldwide, according to the World Health Organization (WHO) update [1]. The pandemic continues with new waves that could persist as new variants of SARS-CoV-2 spread [2].
Effective triage of more patients hospitalized with suspected or confirmed COVID-19 is an important starting point for proper clinical management. Additionally, early identification of subgroups at increased risk of deterioration and needing ventilation or critical care is essential for more effective medical care. Appropriate and rational patient triage strategies improve the decision-making process related to resource allocation, including hospital bed occupancy, critical care resource consumption, and preservation of the most needful and expensive therapy options for critically ill patients [[3], [4], [5]].
To develop effective triage strategies, several predictors have been proposed as early determinants of COVID-19 severity and mortality, such as clinical presentations, radiological findings, and laboratory test results [[6], [7], [8], [9]]. In addition, various existing risk scores were tested, such as the CURB-65 score and the Pneumonia Severity Index (PSI) [10,11], as well as newly developed COVID-19 morbidity and mortality scores, such as Acute Respiratory Infection Consortium Clinical Characterization Protocol-Coronavirus Clinical Characterization Consortium (ISARIC-4C) Score, the Predictive Risk Score (COVID-GRAM), which is a prediction tool for COVID-19 severity using ten variables, and the Veterans Health Administration COVID -19 (VACO) index [[12], [13], [14]]. However, the main concern before accepting these risk assessments as part of local COVID-19 triage policies is demonstrating their validity and good performance.
The CURB-65 is a well-known score prior to the COVID-19 pandemic to assess the severity of community-acquired pneumonia (CAP). It is a simple predictive tool to stratify patients based on five clinical variables (confusion, urea, respiratory rate, systolic or diastolic blood pressure, and age >65) into low, intermediate, or high mortality risk groups [11]. This score has been used for the assessment of COVID-19 patients at triage points of many institutions and has been shown to be a useful triage tool with good performance in identifying hospitalized COVID-19 patients at high risk of mortality [15,16]. Conversely, the ISARIC-4C score is a newly developed score to predict clinical deterioration and mortality in hospitalized COVID-19 patients. This score is easy to use and requires commonly available parameters after the primary assessment in the Emergency Department (ED) or triage area. The following variables are considered in the ISARIC-4C score: The first positive SARS-CoV-2 test or the onset of symptoms, gender, age, number of comorbidities, respiratory rate, oxygen saturation on admission, need for oxygen therapy, Glasgow Coma scale, blood urea nitrogen (BUN) and C-reactive protein [13]. Comorbidities considered for the ISARIC-4C score include chronic respiratory diseases other than asthma, chronic heart disease, chronic kidney disease, dementia, connective tissue disease, chronic liver disease, chronic neurological disorders, diabetes mellitus, HIV/AIDS, malignancy and obesity [13]. Notably, the ISARIC mortality scores are usually available at the time of primary evaluation and do not depend on imaging findings or other parameters that become available later [17,18].
To reach a generalization, several studies were conducted to validate the ISARIC-4C tool in different populations [13,19]. Likewise, few studies have been conducted using the CURB-65 tool, showing good performance in predicting in-hospital mortality [20,21]. In addition, Aletreby et al. (2021) recently validated the performance of the ISARIC score in a Saudi Arabian intensive care unit [22]. However, studies still need to be conducted to compare ISARIC-4C to the CURB-65 tool in the Saudi patient population, which is considered a research gap. Therefore, this study was conducted with two objectives: (i) to determine the external validity of applying the ISARIC-4C score to hospitalized patients with confirmed SARS-CoV-2 infection in Saudi Arabia and (ii) to compare the performance of the ISARIC mortality score with the CURB-65 score as an effective tool for correct and rational triage of COVID-19 patients.
2. Materials & methods
2.1. Study design, settings, and participants
This single-center retrospective observational cohort study was conducted at King Fahad Hospital of the University (KFHU), a 500-bed academic tertiary care institution in Al-Khobar, Eastern Province, Saudi Arabia. All patients confirmed positive for COVID-19 with moderate or severe disease by real-time PCR between March 2020 and May 2021 according to the Saudi Ministry of Health (MOH) protocol were included. Specifically, the data of the patients in the medical records with complete information on the variables of the CURB65 and ISARIC-4C scores were included in this study.
The 4C mortality score includes eight variables: first positive SARS-CoV-2 test or onset of symptoms, gender, age, number of comorbidities, respiratory rate, oxygen saturation on admission, need for oxygen therapy, calculated points of Glasgow coma scale, blood urea nitrogen (BUN) and C-reactive protein [13], while CURB65 is a user-friendly clinical score that only requires the assessment of five clinical variables (confusion, blood urea level, respiratory rate, blood pressure, and age) [11]. The scores were divided into two steps for each patient by reviewing clinical and laboratory data at admission; each score was applied individually. Precisely, the result of the ISARIC-4C score was divided into four risk groups: ‘low’ (0–3), ‘medium’ (4–8), high (9-14), and ‘very high’ (≥15), whereas the result of the CURB-65 score, was divided into three risk groups: ‘low’ (0–1), ‘medium’ (2), and ‘high’ (3–5). Finally, the performance of both scores was evaluated and compared.
2.2. Data collection
Data were secured from hospital patient files and electronic medical records using a Microsoft Excel spreadsheet to extract the required data, including selected variables to apply the ISARIC 4C mortality score and the CURB-65 score. Data collected included patient demographics, clinical findings, results of routine laboratory tests including urea and C-reactive protein, comorbidities as defined by the Charlson Comorbidity Index [17] including chronic respiratory diseases other than asthma, chronic heart disease, chronic kidney disease, dementia, connective tissue disease, Liver disease, chronic neurological disorders, diabetes mellitus, HIV/AIDS, and malignancy with the addition of clinically defined obesity by the treating physician. In addition, information was collected to assess the severity of COVID-19, including ICU admission, need for mechanical ventilation, length of hospital stays, and outcome (discharge or death).
2.2.1. Ethical considerations
Formal ethical approval (IRB-2022-01-153) was obtained from the Institutional Review Board (IRB) of Imam Abdulrahman bin Faisal University. All personal patient information has been kept confidential, secured and used for research purposes only. In addition, this research was conducted in accordance with the World Medical Association's ethical principles for medical research.
2.3. Statistical analysis
Data were imported from a Microsoft Excel spreadsheet after cleansing, and Statistical Package for Social Sciences (SPSS) version 26.0 was used for statistical analysis. Categorical variables were presented as frequencies and percentages, and quantitative variables were presented as mean ± standard deviation.
The chi-square test was used to study the association between variables such as gender, age and ICU admission, and mortality of hospitalized COVID-19 patients. To examine the significance of both the CURB-65 score and the ISARIC-4C score variables considering the ICU requirements and the mortality of COVID-19 hospitalized patients, both chi-square and t-tests were deployed. Comparative results are presented with a corresponding 95% confidence interval (CI), which is a range of values at which researchers are 95% confident that the observed value represents the true mean of the population [23], and is significant when the calculated p-values are <0.05. Logistic regression was used to predict the variables of the ISARIC-4C mortality score and CURB-65 score associated with COVID-19 mortality. In addition, the Hosmer-Lemeshow test, a logistic regression goodness-of-fit test specific to the risk prediction model, was applied. This test assesses whether or not the observed event rates match with the expected event rates in subgroups of the model population and is used for a binary response only.
To compare the ISARIC 4C mortality score and the CURB-65 score, the ISARIC-4C score was standardized into three risk groups (low [0–3], medium [[4], [5], [6], [7], [8]], and high [9,21]) to avoid a different number of risk groups between both scores according to the guidelines [18,19]. The diagnostic accuracy of both scores was evaluated by calculating sensitivities, specificities, positive predictive value, negative predictive value, and Youden's J indices (YJI) [24]. In addition, a receiver operating characteristic (ROC) analysis and an area under the curve (AUC) calculation were performed to ascertain the precision of discrimination of the estimated scores.
2.4. Results
Of these hospitalized patients with COVID-19 infection, 542 patients were included in this study, 58% (n = 314) were Saudis, and 67% (n = 362) of the participants were males. The mean age of the study participants was 52.06 (16.265 SD). 28% (n = 152) of all patients were admitted to the ICU, and 15.49% (n = 84) of patients did not survive. For age, survivors had significantly different values than non-survivors (p < 0.05), while gender was not significant (p = 0.082). In addition, a significant difference was observed between the ward group and the ICU group for gender (p < 0.000), while no significant difference was observed for age (p = 0.063) (Table 1 ).
Table 1.
Gender and age association with ICU admission and mortality of COVID-19 hospitalized patients.
| Variables | Category | Survivors (n = 458) |
Non-Survivors (n = 84) |
p-value | Ward (n = 390) |
ICU (n = 152) |
p-value | ||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| n | % | n | % | n | % | n | % | ||||
| Gender | Female | 159 | 88.3 | 21 | 11.7 | 0.082 | 148 | 82.2 | 32 | 17.8 | 0.000 |
| Male | 229 | 82.6 | 63 | 17.4 | 242 | 66.9 | 120 | 33.1 | |||
| Age | <50 | 212 | 93.8 | 14 | 6.2 | 0.000 | 176 | 77.9 | 50 | 22.1 | 0.063 |
| 50 to 59 | 114 | 86.4 | 18 | 13.6 | 92 | 69.7 | 40 | 30.3 | |||
| 60 to 69 | 69 | 70.4 | 29 | 29.6 | 63 | 64.3 | 35 | 35.7 | |||
| 70 to 79 | 51 | 79.7 | 13 | 20.3 | 46 | 71.9 | 18 | 28.1 | |||
| ≥80 | 12 | 54.5 | 10 | 45.5 | 13 | 59.1 | 9 | 40.9 | |||
Data on the CURB-65 score variables were summarized in Table 2 by type of patient admission (ward or intensive care unit) and patient mortality (survivor or non-survivor). In particular, a significant association was found between patient mortality and confusion (p < 0.05), respiratory rate (RR) [≥ 30] (p < 0.05), age, and blood urea nitrogen (BUN) levels (p < 0.05). A similar observation in these variables of the CURB-65 score was found in the patients admitted to wards and intensive care units (Table 2).
Table 2.
Significance of CURB-65 score variables according to ICU requirement and mortality of COVID-19 hospitalized patients.
| Variables | All patients |
Mortality |
p- value |
ICU requirement |
p- value |
||
|---|---|---|---|---|---|---|---|
| Survivors (n = 458) | Non-Survivors (n = 84) | Ward (n = 390) | ICU (n = 152) | ||||
| Confusion (GCS < 15)a | 67 | 33 (49.3%) | 34 (50.7%) | 0.001 | 21 (31.3%) | 46 (68.7%) | 0.001 |
| Blood urea nitrogen (BUN)b | 18.24 ± 17.31 | 16.78 ± 16.91 | 26.20 ± 17.38 | 0.001 | 16.99 ± 16.88 | 21.44 ± 18.12 | 0.007 |
| Respiratory Rate (RR) ( ≥ 30)a | 136 | 85 (62.5%) | 51 (37.5%) | 0.001 | 49 (36%) | 87 (64%) | 0.001 |
| Systolic Blood Pressure (SBP)b | 130.73 ± 23.32 | 129.89 ± 22.92 | 135.31 ± 25.03 | 0.067 | 129.56 ± 21.40 | 133.72 ± 27.49 | 0.062 |
| Diastolic Blood Pressure (DBP)b | 77.23 ± 14.22 | 77.37 ± 14.03 | 76.46 ± 15.31 | 0.593 | 76.94 ± 12.42 | 77.95 ± 18.07 | 0.458 |
| Ageb | 52.06 ± 16.26 | 50.06 ± 16.05 | 62.94 ± 12.84 | 0.001 | 50.67 ± 16.49 | 55.61 ± 15.14 | 0.001 |
a p < 0.05 by using Chi-square test. b p < 0.05 by using t-test test.
Among the comorbidities that counted for the ISARIC 4C mortality score, there were significant differences between ward/ICU groups and survivor/non-survivor groups for diabetes mellitus, chronic renal disease, and clinically defined obesity (p < 0.05). Table 3, Table 4 presented data related to the ISARIC 4C mortality variables by patient groups (ward vs. intensive care unit or survivor vs. non-survivor) at 0.05 levels of significance. A significant association was found between all variables except the number of comorbidities between the ward/ICU groups (p > 0.05).
Table 3.
The accounted comorbidities of ISARIC 4C mortality score and their statistical significance.
| Variables | Category | Survivors (n = 458) |
Non-Survivors (n = 84) |
p-value | Ward (n = 390) |
ICU (n = 152) |
p-value | ||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| n | % | n | % | n | % | n | % | ||||
| Chronic cardiac conditions | No | 397 | 85.0 | 70 | 15.0 | 0.414 | 333 | 71.3 | 134 | 28.7 | 0.401 |
| Yes | 61 | 81.3 | 14 | 18.7 | 57 | 76 | 18 | 24 | |||
| Chronic neurological conditions | No | 435 | 85.1 | 76 | 14.9 | 0.102 | 367 | 71.8 | 144 | 28.4 | 0.775 |
| Yes | 23 | 74.2 | 8 | 25.8 | 23 | 74.2 | 8 | 25.8 | |||
| Dementia | No | 450 | 84.9 | 80 | 15.1 | 0.084 | 383 | 72.3 | 147 | 27.7 | 0.288 |
| Yes | 8 | 66.7 | 4 | 33.3 | 7 | 58.3 | 5 | 41.7 | |||
| Chronic respiratory disease | No | 454 | 84.7 | 82 | 15.3 | 0.772 | 384 | 71.6 | 152 | 28.4 | 0.160 |
| Yes | 4 | 80 | 1 | 20 | 5 | 100 | 0 | 0 | |||
| Connective tissue disease | No | 449 | 84.4 | 83 | 15.6 | 0.198 | 383 | 72 | 149 | 28 | 0.724 |
| Yes | 9 | 100 | 0 | 0 | 6 | 66.7 | 3 | 33.3 | |||
| Mild-to-severe liver disease | No | 451 | 84.6 | 82 | 15.4 | 0.339 | 385 | 72.2 | 148 | 27.8 | 0.086 |
| Yes | 5 | 71.4 | 2 | 28.6 | 3 | 42.9 | 4 | 57.1 | |||
| DM | No | 271 | 87.1 | 40 | 12.9 | 0.049 | 235 | 75.6 | 76 | 24.4 | 0.030a |
| Yes | 187 | 81 | 44 | 19 | 155 | 67.1 | 76 | 32.9 | |||
| Chronic renal disease | No | 418 | 88 | 58 | 12 | 0.001 | 351 | 74 | 125 | 26 | 0.019a |
| Yes | 40 | 60.6 | 26 | 40.4 | 39 | 59 | 27 | 41 | |||
| Malignancy | No | 448 | 84.5 | 82 | 15.5 | 0.910 | 382 | 72.1 | 148 | 27.9 | 0.680 |
| Yes | 10 | 83.3 | 2 | 16.7 | 8 | 66.7 | 4 | 33.3 | |||
| HIV/AIDS | No | 457 | 84.5 | 84 | 15.5 | 0.668 | 389 | 71.9 | 152 | 28.1 | 0.532 |
| Yes | 1 | 100 | 0 | 0 | 1 | 100 | 0 | 0 | |||
| Clinician-defined obesity | No | 342 | 86.4 | 54 | 13.6 | 0.049 | 296 | 74.7 | 100 | 25.3 | 0.017a |
| Yes | 116 | 79.5 | 30 | 20.5 | 94 | 64.4 | 52 | 35.6 | |||
Table 4.
Significance of ISARIC 4C mortality score variables according to ICU requirement and mortality of COVID-19 hospitalized patients.
| Variables | All patients (452) | Mortality |
p- value | ICU requirement |
p- value | ||
|---|---|---|---|---|---|---|---|
| Survivors (n = 458) | Non-Survivors (n = 84) | Ward (n = 390) | ICU(n = 152) | ||||
| Gender (Male)a | 362 | 299 (82.6%) | 63 (17.4%) | 0.101 | 242 (67%) | 120 (33%) | 0.001 |
| Number of comorbidities | 331 | 270 (82%) | 61 (28%) | 0.012 | 226 (68%) | 105 (32%) | 0.057 |
| Glasgow Coma Scaleb | 14.66 ± 1.14 | 14.77 ± 1.05 | 14.05 ± 1.39 | 0.001 | 14.84 ± 0.89 | 14.20 ± 1.52 | 0.001 |
| Age (years)b | 52.06 ± 16.26 | 50.06 ± 16.05 | 62.94 ± 12.84 | 0.001 | 50.67 ± 16.49 | 55.61 ± 15.14 | 0.001 |
| Respiratory rate (breaths/min)b | 25.92 ± 8.86 | 24.40 ± 7.30 | 34.17 ± 11.70 | 0.001 | 23.21 ± 6.39 | 32.87 ± 10.41 | 0.001 |
| Admission oxygen saturation (%) ( > 92)a | 246 | 174 (70.7%) | 72 (29.3%) | 0.001 | 125 (50.8%) | 121 (49.2%) | 0.001 |
| Blood urea nitrogen (BUN) (mg/dL)b | 18.24 ± 17.31 | 16.78 ± 16.91 | 26.20 ± 17.38 | 0.001 | 16.99 ± 16.88 | 21.44 ± 18.02 | 0.001 |
| C-reactive protein (mg/L)b | 95.99 ± 85.89 | 88.26 ± 83.06 | 138.11 ± 87.37 | 0.001 | 81.39 ± 75.90 | 133.44 ± 97.16 | 0.001 |
a p < 0.05 by using Chi-square test. b P < 0.05 by using t-test test.
Regarding the need for ICU admission and in-hospital mortality, statistically significant differences were observed between all three standardized risk groups (low, medium, and high) of both the CURB65 and ISARIC 4C scores between the ward/ICU groups (p < 0.05), as well as survivors/non-survivors’ groups (p < 0.05) (Table 5 ).
Table 5.
Significance of CURB-65 and ISARIC 4C mortality score values according to the standardized risk groups.
| Outcome |
(χ2, p < 0.05) | ||
|---|---|---|---|
| Discharged home (n = 458) | Expired (n = 84) | ||
| CURB65 | |||
| Low (0–1) | 377 (95%) | 21 (5%) | (128.44, <0.0001) |
| Medium (2) | 52 (64%) | 29 (36%) | |
| High (3–5) | 29 (46%) | 34 (54%) | |
| ISARIC 4C | |||
| Low (0–3) | 47 (100%) | 0 (0%) | (67.85, <0.0001) |
| Medium (4–8) | 240 (95%) | 12 (%) | |
| High (9–21) |
171 (70%) |
72 (30%) |
|
| Intensive Care requirement |
(χ2, P < 0.05) |
||
|
Ward (n = 390) |
ICU (n = 152) |
||
| CURB65 | |||
| Low (0–1) | 45 (95.70%) | 2 (4.30%) | (51.36, <0.0001) |
| Medium (2) | 206 (81.70%) | 46 (18.30%) | |
| High (3–5) | 139 (57.20%) | 104 (42.80%) | |
| ISARIC 4C | |||
| Low (0–3) | 328 (82.40%) | 70 (17.60%) | (82.53, <0.0001) |
| Medium (4–8) | 38 (46.90%) | 43 (53.10%) | |
| High (9–21) | 24 (38.10%) | 39 (61.90%) | |
ROC analysis shows an AUC value of 0.834 [95% CI; 0.800–0.865] for the CURB-65 score and 0.809 [95% CI; 0.773–0.841] for the ISARIC-4C score. The sensitivity for CURB-65 and ISARIC-4C scores was 75% and 85.71%, respectively, while the specificity of CURB-65 was 82.31% compared to 62.66% for ISARIC-4C scores (Table 6 ). Comparing the performance of both scores, the difference between AUC was 0.025 (95% CI; −0.0203-0.0704) with the z-statistic of 1.081 and p = 0.2795, the odds ratio for the ISARIC 4C mortality score was 1.4690 (95% CI; 1.3364–1.6148), and for the CURB-65 score 2.8222 (95% CI; 2.2346–3.5643), Youden's J indexes for CURB-65 and ISARIC-4C scores were 0.57 and 0.48 respectively [24] (Table 6 and Fig. 1 ).
Table 6.
Performance of the CURB-65 and ISARIC-4C mortality scores.
| Scores | CURB-65 | ISARIC 4C |
|---|---|---|
| AUC | 0.834 | 0.809 |
| 95% CI | 0.800 to 0.865 | 0.773 to 0.841 |
| Sensitivity | 75.00 | 85.71 |
| Specificity | 82.31 | 62.66 |
| YJI | 0.57 | 0.48 |
| P-value | <0.0001 | <0.0001 |
| Difference between AUCs | 0.025 | |
| 95% CI | −0.0203–0.0704 | |
| Z statistic | 1.081 | |
| Significance level | P = 0.2795 | |
Fig. 1.
ROC curves of ISARIC 4C and CURB-65 scores for prediction of mortality of COVID-19 hospitalized patients.
3. Discussion
The COVID-19 pandemic is still wreaking havoc on healthcare systems around the world, with tremendous impacts on morbidity and mortality affecting hundreds of millions of people. Globally, the massive increase in COVID-19 patients has led to resource depletion and a significant strain on the healthcare sector. Effective triage of patients presenting to hospital with suspected or confirmed COVID-19 is an important starting point for proper resource allocation and early identification of patients at increased risk of deterioration. Developing a new scoring system or using existing systems to identify patients at high risk for developing a serious or critical illness is required to apply appropriate management. However, such scores should be valid, accurate, and trustworthy to be used safely.
On exploring the literature, several studies have evaluated the performance of the CURB-65 score during the COVID-19 pandemic. In the early phase of the COVID-19 pandemic, Guo et al. found that the CURB-65 score can serve as a predictive tool for in-hospital COVID-19 mortality with 68% sensitivity and 81% specificity and supported the usefulness of the CURB-65 score as a prognostic indicator for rapid triage of COVID-19 patients [21]. The appropriate performance of the CURB-65 score in determining mortality and the need for ICU admission of COVID-19 patients has been further investigated during the pandemic; Doğanay et al. reported good performance of the CURB-65 score with 85% sensitivity, 73.96% specificity, and 0.846 AUC [20]. Demir et al. reported better performance in predicting COVID-19 mortality with 83.33% sensitivity, 90.43% specificity, and 0.942 AUC [16]. However, other authors have raised concerns about the reliance on the CURB-65 score to guide clinical decision-making in COVID-19 patient care, despite showing statistically significant differences in distinguishing between survivors and non-survivors, and there is an argument that the CURB-65 score is incapable of accurately estimating COVID-19-associated mortality due to the complexity and specific characteristics of COVID-19 disease and the presence of many risk factors [25,26]. Our study results indicates a good performance of the CURB-65 score as a prognostic tool, estimating patient mortality with a sensitivity of 75.00%, a specificity of 82.31%, YJI 0.57, AUC 0.834, and P < 0.0001, which supports its use for rapid triage of COVID-19 patients.
As the COVID-19 pandemic continues and the capacity of various health systems is depleted by a massive increase in COVID-19 patients in dire need of hospitalization, the rapid development of reliable and safe scores can aid in quick triage of COVID-19 patients, which has been a significant concern of researchers worldwide. Based on clinical observations and unique characteristics of COVID-19, many newly developed scores have been evaluated as triage tools that can identify COVID-19 patients at high risk of developing a severe or critical illness, such as QCovid population-based risk algorithm with the QResearch database in the United Kingdom linking to the death registers and hospital admissions data, the COVID-GRAM score, the COVID-19 Complication Risk Score and the rapid COVID-19 Severity Index [14,[27], [28], [29]].
One of the recently developed scores is the ISARIC 4C Mortality Score, which was developed and validated in 2020; 260 hospitals in the United Kingdom were involved, 35463 patients were enrolled in the derivation group and 22361 in the validation group. [13] The score was developed using eight parameters that are generally available at the time of COVID-19 patient admission; an appropriate data set with significant sample size was used to avoid the risk of bias, uncertainty and invalid reports. As with the CURB-65 score, several previous studies have been conducted to show the performance of the ISARIC 4C mortality score during the COVID-19 pandemic. According to Knight et al., the ISARIC 4C mortality score had high performance beyond the existing scores; it stratifies COVID-19 patients into four different risk groups (low, intermediate, high, or very high) to guide the clinical decisions with an AUC of 0.77–0.79; however, they indicated the need for further external validation to generalize the score's applicability to other populations [13]. Subsequent studies to externally validate and evaluate the performance of the ISARIC 4C Mortality Score demonstrated its applicability as a powerful triage tool that indicates COVID-19 patients at high risk of worsening and death. For example, a study conducted between March and September 2020 by the RECOVER network (Registry of Suspected COVID19 in Emergency Care) in which 99 emergency departments in the USA participated showed that the ISARIC 4C mortality score had suitable discrimination with the AUC of 0.786 in the RECOVER dataset compared to 0.763 in the original validation dataset; the authors conclude that the ISARIC 4C mortality score is a valid score to predict the risk of 30-day mortality in hospitalized COVID-19 patients [30]. Similar findings were reported by Jones et al.; they found that the ISARIC 4C Mortality Score is a valid prognostic mortality tool for COVID-19 patients that can be used in Canadian hospitals to prioritize care and resources for those patients most in need [31]. In another study, statistical significance in mortality differences between ISARIC 4C score categories was observed during the first wave of the COVID-19 pandemic [32]. Albaie et al. reported that the ISARIC 4C mortality score is useful to assess the prognosis of patients with type 2 diabetes mellitus presenting with COVID-19 with an AUC of 0.875 using a cutoff value of >14 [33]. Another study from Saudi Arabia found that the ISARIC 4C Mortality Score is an excellent tool to predict mortality in critically ill COVID-19 patients with an AUC of 0.81; the sensitivity was 70.5% and the specificity 73.97% with a cutoff value of >9 [22]. In addition, Crocker-Buque et al. support using the ISARIC 4C Mortality Score for triaging COVID-19 patients at the time of admission and as a dynamic tool that provides accurate mortality risk information at any time during the hospitalization of COVID-19 patients [34].
Our findings were consistent with the internal and external validation results of above studies; the ISARIC 4C mortality score had a sensitivity of 85.71%, a specificity of 62.66%, YJI 0.48, AUC 0.809, and p < 0.0001 in our cohort. The performance of ISARIC-4C mortality score was good in terms of the need for ICU admission and predicting in-hospital mortality of COVID-19 patients. These results support the external validation of the ISARIC 4C mortality score as a triage tool for Saudi patients presenting to the hospital with suspected or confirmed COVID-19. However, it is noteworthy that the number of comorbidities in our cohort was insignificant as a single predictor when comparing survivor/non-survivor or ward/ICU groups (p < 0.57), so adding the number of comorbidities to the predictor set may not improve the score performance, which consistent with the results of Adderley et al. study. [35].
In addition, concerns about the ISARIC 4C mortality score rose again when the COVID-19 pandemic waves with new SARS-CoV-2 variants such as Alpha (B.1.1.7) or Omicron (B.1.1.529) persisted, which is possibly the case, behave differently in terms of clinical presentation, contagiousness, and severity [36]. In addition, massive vaccination campaigns and the adaptation of new treatment options in the management of COVID-19 patients can influence disease course and presentations [30]. However, our cohort includes patients infected during the first and second COVID-19 pandemic waves. It reveals good ISARIC 4C mortality score performance, which is in agreement with the conclusion of Innocenti et al. study [37].
3.1. ISARIC-4C mortality score versus CURB-65 score
Our study comparison of the performance of the ISARIC-4C Mortality Score with the CURB-65 score showed higher sensitivity (85.71% vs. 75.00%) and significantly lower specificity (62.66% vs. 82.31) with a AUC of 0.809 versus 0.834 for the CURB-65 score. These results demonstrate the acceptable performance of both scores and support their usefulness as triage tools to predict COVID-19 patients at high risk of deterioration and death. The CURB-65 score performed better than the ISARIC 4C mortality score in terms of specificity, AUC, YJI and odds ratio. Though, the difference between the AUC for both scores was 0.025, which was not statistically significant (p = 0.2795). These results correlated with previous studies comparing the performance of both scores. Doanay et al. concluded that the CURB-65 score showed better performance on the need for ICU admission and mortality of in-hospital COVID-19 patients. They also suggest using both scores together to improve the clinical decision-making process [20]. Another study reported comparable performance for both scores, with an AUC of 0.818 for the ISARIC-4C score and 0.801 for the CURB-65 score [19]. The usefulness of both scores for predicting mortality in hospitalized COVID-19 patients has also been supported by other studies [[38], [39], [40]]. Martin et al. reported that the CURB-65 and the ISARIC-4C have the best discriminant performance in predicting in-hospital COVID-19 mortality when compared to other scores such as the Sequential Organ Failure Assessment (SOFA) [41]. However, other comparative studies showed significantly lower performance of the CURB-65 score compared to the ISARIC 4C score [42,43].
3.2. Limitations
This study has some limitations that need to be specified: The results are based on observations from a single center with a relatively small sample size that does not reflect the general Saudi population, with the possibility of selection bias. Therefore, our research results may not be generalizable to the Saudi population. Second, the study is retrospective, with the inherited limitations of such study types that may not reflect actual real-time triage. Finally, further prospective research is needed to validate the ISARIC-4C score by comparing it to a variety of other mortality scores in different medical centers in Saudi Arabia and elsewhere.
4. Conclusion
This is the first study in Saudi Arabia to demonstrate the external validity of the ISARIC-4C score at the time of admission for all hospitalized patients with confirmed SARS-CoV-2 infection and to predict their mortality risk. Further, the authors demonstrated the comparative performance of the ISARIC-4C score with the CURB-65 score as an effective tool for the correct and rational triage of COVID-19 patients within the Saudi Arabian population. Precisely, the performance of the CURB-65 score was not inferior to the ISARIC-4C score; both scores showed comparable performance with good consistent discrimination and are suitable for clinical utility as triage tools for hospitalized COVID-19 patients. However, more multicenter studies are needed to validate the ISARIC-4C score in Saudi Arabia.
Ethical statement
Document of ethical approval (IRB-2022-01-153) was obtained from the Institutional Review Board (IRB) at Imam Abdulrahman bin Faisal University, Saudi Arabia.
Funding
This research received no external funding.
Author contributions
Dr. Marwan Jabr Alwazzeh (M.A.), Dr. Arun Vijay Subbarayalu (A.S.), Dr. Batool Mohammed Abu Ali (B.A.)., Dr Reema alabdulqader (R.A.)., Dr Mashael Alhajri (M.A.H)., Dr. Sara M. Alwarthan (S.A.)., Dr. Bashayer M. AlShehail (B.A.S.)., Dr. Vinoth Raman (V.R.)., Dr. Fahd Abdulaziz Almuhanna (F.A.).
Conceptualization, M.A., and S. A.; Methodology, M. A, V.R. and A.S.; Formal Analysis, A.S., and V.R.; Investigation, B.A., R.A. and B.A.S., Data Curation, M.A.H, B.A., R.A. and B.A.S.; Writing – Original Draft Preparation, M.A.; Writing – Review & Editing, M.A., A.S. and F.A.; Supervision, F.A.; Project Administration, M.A.
Informed consent statement
Informed consent was obtained from all participants involved in the study.
Sources of funding for your research
This research received no external funding.
Data availability statements
The data presented in this study are available on request from the corresponding author.
Declaration of competing interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgement
The authors would like to thank the Institution Review Board (IRB) of Imam Abdulrahman Bin Faisal University (IRB-2022-01-153), Saudi Arabia for granting permission to conduct this study.
References
- 1.World Health Organization Geneva, Switzerland. Weekly epidemiological update on COVID-19-1. 2023. https://www.who.int/publications/m/item/weekly-epidemiological-update-on-covid-19--1-february-2023 [Internet]. [cited 2023 Mar 1]. Report No.: 128. Available from:
- 2.Zhang Y., Zhang H., Zhang W. SARS-CoV-2 variants, immune escape, and countermeasures. Front Med. 2022;16(2):196–207. doi: 10.1007/s11684-021-0906-x. Apr. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Tan L., Kang X., Ji X., Li G., Wang Q., Li Y., et al. Validation of predictors of disease severity and outcomes in COVID-19 patients: a descriptive and retrospective study. Med N Y N. 2020;1(1):128–138. doi: 10.1016/j.medj.2020.05.002. Dec 18. e3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Plečko D., Bennett N., Mårtensson J., Dam T.A., Entjes R., Rettig T.C.D., et al. Rapid Evaluation of Coronavirus Illness Severity (RECOILS) in intensive care: development and validation of a prognostic tool for in-hospital mortality. Acta Anaesthesiol Scand. 2022;66(1):65–75. doi: 10.1111/aas.13991. Jan. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Chua F., Vancheeswaran R., Draper A., Vaghela T., Knight M., Mogal R., et al. Early prognostication of COVID-19 to guide hospitalisation versus outpatient monitoring using a point-of-test risk prediction score. Thorax. 2021;76(7):696–703. doi: 10.1136/thoraxjnl-2020-216425. Jul. [DOI] [PubMed] [Google Scholar]
- 6.Carpenter C.R., Mudd P.A., West C.P., Wilber E., Wilber S.T. Diagnosing COVID-19 in the emergency department: a scoping Review of clinical examinations, laboratory tests, imaging accuracy, and biases. Acad Emerg Med Off J Soc Acad Emerg Med. 2020;27(8):653–670. doi: 10.1111/acem.14048. Aug. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Agarwal N., Biswas B., Lohani P. Epidemiological determinants of COVID-19 infection and mortality: a study among patients presenting with severe acute respiratory illness during the pandemic in Bihar, India. Niger Postgrad Med J. 2020;27(4):293–301. doi: 10.4103/npmj.npmj_301_20. Dec. [DOI] [PubMed] [Google Scholar]
- 8.Ansari K.A., Alwazzeh M.J., Alkuwaiti F.A., Farooqi F.A., Al Khathlan N., Almutawah H., et al. Early determinants of mortality in hospitalized COVID-19 patients in the eastern Province of Saudi Arabia. Int J Gen Med [Internet] 2022;15:1689–1701. doi: 10.2147/IJGM.S349598. https://www.dovepress.com/early-determinants-of-mortality-in-hospitalized-covid-19-patients-in-t-peer-reviewed-fulltext-article-IJGM Feb [cited 2022 Nov 13] Available from: [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Salameh J.P., Leeflang M.M., Hooft L., Islam N., McGrath T.A., van der Pol C.B., et al. Thoracic imaging tests for the diagnosis of COVID-19. Cochrane Database Syst Rev. 2020 Sep 30;9 doi: 10.1002/14651858.CD013639.pub2. CD013639. [DOI] [PubMed] [Google Scholar]
- 10.Fine M.J., Auble T.E., Yealy D.M., Hanusa B.H., Weissfeld L.A., Singer D.E., et al. A prediction rule to identify low-risk patients with community-acquired pneumonia. N Engl J Med. 1997;336(4):243–250. doi: 10.1056/NEJM199701233360402. Jan 23. [DOI] [PubMed] [Google Scholar]
- 11.Lim W.S., van der Eerden M.M., Laing R., Boersma W.G., Karalus N., Town G.I., et al. Defining community acquired pneumonia severity on presentation to hospital: an international derivation and validation study. Thorax. 2003;58(5):377–382. doi: 10.1136/thorax.58.5.377. May. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.King J.T., Yoon J.S., Rentsch C.T., Tate J.P., Park L.S., Kidwai-Khan F., et al. Development and validation of a 30-day mortality index based on pre-existing medical administrative data from 13,323 COVID-19 patients: the Veterans Health Administration COVID-19 (VACO) Index. PLoS One. 2020;15(11) doi: 10.1371/journal.pone.0241825. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Knight S.R., Ho A., Pius R., Buchan I., Carson G., Drake T.M., et al. Risk stratification of patients admitted to hospital with covid-19 using the ISARIC WHO Clinical Characterisation Protocol: development and validation of the 4C Mortality Score. BMJ. 2020;370 doi: 10.1136/bmj.m3339. Sep 9. m3339. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Liang W., Liang H., Ou L., Chen B., Chen A., Li C., et al. Development and validation of a clinical risk score to predict the occurrence of critical illness in hospitalized patients with COVID-19. JAMA Intern Med. 2020;180(8):1081–1089. doi: 10.1001/jamainternmed.2020.2033. Aug 1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Nguyen Y., Corre F., Honsel V., Curac S., Zarrouk V., Fantin B., et al. Applicability of the CURB-65 pneumonia severity score for outpatient treatment of COVID-19. J Infect. 2020;81(3):e96–e98. doi: 10.1016/j.jinf.2020.05.049. Sep. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Demir M.C., Ilhan B. Performance of the pandemic medical early warning score (PMEWS), simple triage scoring system (STSS) and confusion, uremia, respiratory rate, blood pressure and age ≥ 65 (CURB-65) score among patients with COVID-19 pneumonia in an emergency department triage setting: a retrospective study. Sao Paulo Med J Rev Paul Med. 2021;139(2):170–177. doi: 10.1590/1516-3180.2020.0649.R1.10122020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Song Y., Zheng S., Li L., Zhang X., Zhang X., Huang Z., et al. Deep learning enables accurate diagnosis of novel coronavirus (COVID-19) with CT images. IEEE ACM Trans Comput Biol Bioinf. 2021;18(6):2775–2780. doi: 10.1109/TCBB.2021.3065361. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Tolksdorf K., Buda S., Schuler E., Wieler L.H., Haas W. Influenza-associated pneumonia as reference to assess seriousness of coronavirus disease (COVID-19) Euro Surveill Bull Eur Sur Mal Transm Eur Commun Dis Bull. 2020;25(11):2000258. doi: 10.2807/1560-7917.ES.2020.25.11.2000258. Mar. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Citu C., Gorun F., Motoc A., Ratiu A., Gorun O.M., Burlea B., et al. Evaluation and comparison of the predictive value of 4C mortality score, news, and CURB-65 in poor outcomes in COVID-19 patients: a retrospective study from a single center in Romania. Diagn Basel Switz. 2022;12(3):703. doi: 10.3390/diagnostics12030703. Mar 13. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Doğanay F., Ak R. Performance of the CURB-65, ISARIC-4C and COVID-GRAM scores in terms of severity for COVID-19 patients. Int J Clin Pract. 2021;75(10) doi: 10.1111/ijcp.14759. Oct. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Guo J., Zhou B., Zhu M., Yuan Y., Wang Q., Zhou H., et al. CURB-65 may serve as a useful prognostic marker in COVID-19 patients within Wuhan, China: a retrospective cohort study. Epidemiol Infect. 2020;148 doi: 10.1017/S0950268820002368. Oct 1. e241. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Aletreby W.T., Mumtaz S.A., Shahzad S.A., Ahmed I., Alodat M.A., Gharba M., et al. External validation of 4C ISARIC mortality score in critically ill COVID-19 patients from Saudi Arabia. Saudi J Med Med Sci. 2022;10(1):19–24. doi: 10.4103/sjmms.sjmms_480_21. Apr. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Choudhary D., Garg P.K. 95 % confidence interval: a misunderstood statistical tool. Indian J Surg. 2013;75(5):410. doi: 10.1007/s12262-012-0555-z. Oct. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Lin S., Ma Y., Zou H. Enhanced Youden's index with net benefit: a feasible approach for optimal-threshold determination in shared decision making. J Eval Clin Pract. 2020;26(2):551–558. doi: 10.1111/jep.13311. Apr. Epub 2019 Nov 18. PMID: 31738475. [DOI] [PubMed] [Google Scholar]
- 25.Ahmed A., Alderazi S.A., Aslam R., Barkat B., Barker B.L., Bhat R., et al. Utility of severity assessment tools in COVID-19 pneumonia: a multicentre observational study. Clin Med [Internet] 2022 Jan;22(1):63–70. doi: 10.7861/clinmed.2020-1107. https://www.rcpjournals.org/lookup/doi/10.7861/clinmed.2020-1107 [cited 2022 Nov 14] Available from: [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Preti C., Biza R., Novelli L., Ghirardi A., Conti C., Galimberti C., et al. Usefulness of CURB-65, pneumonia severity index and MuLBSTA in predicting COVID-19 mortality. Monaldi Arch Chest Dis Arch Monaldi Mal Torace. 2022;4:92. doi: 10.4081/monaldi.2022.2054. Feb 22. [DOI] [PubMed] [Google Scholar]
- 27.Nafilyan V., Humberstone B., Mehta N., Diamond I., Coupland C., Lorenzi L., et al. An external validation of the QCovid risk prediction algorithm for risk of mortality from COVID-19 in adults: a national validation cohort study in England. Lancet Digit Health. 2021;3(7):e425–e433. doi: 10.1016/S2589-7500(21)00080-7. Jul. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Halalau A., Imam Z., Karabon P., Mankuzhy N., Shaheen A., Tu J., et al. External validation of a clinical risk score to predict hospital admission and in-hospital mortality in COVID-19 patients. Ann Med Interne. 2021;53(1):78–86. doi: 10.1080/07853890.2020.1828616. https://www.tandfonline.com/doi/full/10.1080/07853890.2020.1828616 Jan 1 [cited 2022 Nov 12] Available from: [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Haimovich A.D., Ravindra N.G., Stoytchev S., Young H.P., Wilson F.P., van Dijk D., et al. Development and validation of the quick COVID-19 severity index: a prognostic tool for early clinical decompensation. Ann Emerg Med. 2020;76(4):442–453. doi: 10.1016/j.annemergmed.2020.07.022. Oct. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Gordon A.J., Govindarajan P., Bennett C.L., Matheson L., Kohn M.A., Camargo C., et al. External validation of the 4C Mortality Score for hospitalised patients with COVID-19 in the RECOVER network. BMJ Open. 2022;12(4) doi: 10.1136/bmjopen-2021-054700. Apr 21. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Jones A., Pitre T., Junek M., Kapralik J., Patel R., Feng E., et al. External validation of the 4C mortalit score among COVID-19 patients admitted to hospital in Ontario, Canada: a retrospective study. Sci Rep. 2021;11(1):18638. doi: 10.1038/s41598-021-97332-1. Sep 20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Ali R., Qayyum F., Ahmed N., Haroon M.Z., Irshad R., Sajjad S., et al. Isaric 4c mortality score as A predictor of in-hospital mortality in covid-19 patients admitted in ayub teaching hospital during first wave of the pandemic. J Ayub Med Coll Abbottabad JAMC. 2021;33(1):20–25. Mar. [PubMed] [Google Scholar]
- 33.Albai O., Frandes M., Sima A., Timar B., Vlad A., Timar R. Practical applicability of the ISARIC-4C score on severity and mortality due to SARS-CoV-2 infection in patients with type 2 diabetes. Med Kaunas Lith. 2022 Jun 25;58(7):848. doi: 10.3390/medicina58070848. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Crocker-Buque T., Myles J., Brentnall A., Gabe R., Duffy S., Williams S., et al. Using ISARIC 4C mortality score to predict dynamic changes in mortality risk in COVID-19 patients during hospital admission. PLoS One. 2022;17(10) doi: 10.1371/journal.pone.0274158. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Adderley N.J., Taverner T., Price M.J., Sainsbury C., Greenwood D., Chandan J.S., et al. Development and external validation of prognostic models for COVID-19 to support risk stratification in secondary care. BMJ Open [Internet] 2022 Jan;12(1) doi: 10.1136/bmjopen-2021-049506. https://bmjopen.bmj.com/lookup/doi/10.1136/bmjopen-2021-049506 [cited 2022 Nov 12] Available from: [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Akbar S., Pan D., Ehdode A., Islam R., Abouzaid A., Balasundaram K., et al. Prognostic value of maximum NEWS-2 scores in addition to ISARIC 4C scores for patients admitted to hospital with COVID-19. J Infect. 2022;85(1):e30–e32. doi: 10.1016/j.jinf.2022.04.013. Jul. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Innocenti F., De Paris A., Lagomarsini A., Pelagatti L., Casalini L., Gianno A., et al. Stratification of patients admitted for SARS-CoV2 infection: prognostic scores in the first and second wave of the pandemic. Intern Emerg Med. 2022;17(7):2093–2101. doi: 10.1007/s11739-022-03016-7. Oct. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Yildiz H., Castanares-Zapatero D., Hannesse C., Vandermeersch D., Pothen L., Yombi J.C. Prospective validation and comparison of COVID-GRAM, NEWS2, 4C mortality score, CURB-65 for the prediction of critical illness in COVID-19 patients. Infect Dis Lond Engl. 2021;53(8):640–642. doi: 10.1080/23744235.2021.1896777. Aug. [DOI] [PubMed] [Google Scholar]
- 39.Ocho K., Hagiya H., Hasegawa K., Fujita K., Otsuka F. Clinical utility of 4C mortality scores among Japanese COVID-19 patients: a multicenter study. J Clin Med. 2022;11(3):821. doi: 10.3390/jcm11030821. Feb 3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Hassan S., Ramspek C.L., Ferrari B., van Diepen M., Rossio R., Knevel R., et al. External validation of risk scores to predict in-hospital mortality in patients hospitalized due to coronavirus disease 2019. Eur J Intern Med. 2022;102:63–71. doi: 10.1016/j.ejim.2022.06.005. Aug. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Martin J., Gaudet-Blavignac C., Lovis C., Stirnemann J., Grosgurin O., Leidi A., et al. Comparison of prognostic scores for inpatients with COVID-19: a retrospective monocentric cohort study. BMJ Open Respir Res. 2022;9(1) doi: 10.1136/bmjresp-2022-001340. Aug. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Ngiam J.N., Chew N.W.S., Tham S.M., Lim Z.Y., Li T.Y.W., Cen S., et al. Utility of conventional clinical risk scores in a low-risk COVID-19 cohort. BMC Infect Dis. 2021;21(1):1094. doi: 10.1186/s12879-021-06768-3. Oct 24. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Vedovati M.C., Barbieri G., Urbini C., D'Agostini E., Vanni S., Papalini C., et al. Clinical prediction models in hospitalized patients with COVID-19: a multicenter cohort study. Respir Med. 2022;202:106954. doi: 10.1016/j.rmed.2022.106954. Oct. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
The data presented in this study are available on request from the corresponding author.