Abstract
Objective
To evaluate the performance of the Quick COVID-19 Severity Index (qCSI) and the Brescia-COVID Respiratory Severity Scale (BCRSS) in predicting intensive care unit (ICU) admissions and in-hospital mortality in patients with coronavirus disease 2019 (COVID-19) pneumonia.
Methods
This was a retrospective cohort study of 313 consecutive hospitalized adult patients (18 years or older) with confirmed COVID-19. The area under the receiver operating characteristic curve (AUC) was used to assess the discriminatory power of the qCSI score and BCRSS prediction rule compared to the CURB-65 score for predicting mortality and intensive care unit admission.
Results
The overall in-hospital fatality rate was 32.3%, and the ICU admission rate was 31.3%. The CURB-65 score had the highest numerical AUC to predict in-hospital mortality (AUC 0.781) compared to the qCSI score (AUC 0.711) and the BCRSS prediction rule (AUC 0.663). For ICU admission, the qCSI score had the highest numerical AUC (AUC 0.761) compared to the BCRSS prediction rule (AUC 0.735) and the CURB-65 score (AUC 0.629).
Conclusions
The CURB-65 and qCSI scoring systems showed a good performance for predicting in-hospital mortality. The qCSI score and the BCRSS prediction rule showed a good performance for predicting ICU admission.
Keywords: COVID-19, SARS-CoV-2, CURB-65, Scoring systems
Introduction
Coronavirus disease 2019 (COVID-19), caused by a newly emerged betacoronavirus severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), is a rapidly spreading disease currently responsible for the fifth documented pandemic since the 1918 flu pandemic (Zhu et al., 2020, Coronaviridae Study Group of the International Committee on Taxonomy of Viruses, 2020, Liu et al., 2020). As of September 25, 2020, there had been 6 958 632 total cases reported in the United States, with 202 329 confirmed deaths (Centers for Disease Control and Prevention, 2020).
Amid a rapidly growing pandemic, appropriate resource allocation becomes essential as hospital systems are usually not designed for epidemics (Paranthaman et al., 2008). During the ongoing COVID-19 pandemic, we have seen advanced health systems stretched beyond their capacities. When the volume of a health system is exceeded, rationing decisions may not only affect patients with COVID-19 but all patients requiring acute care (Cavallo et al., 2020). After the initial peak, the United States will likely experience waves of illness as seen with previous pandemics or, at the very least, outbreaks of lingering disease for months and perhaps years to come. Hence healthcare systems and providers must be prepared (Bauchner and Sharfstein, 2020, Taubenberger and Morens, 2006).
In patients with pneumonia, clinical judgment alone may under or overestimate disease severity and lead to suboptimal decisions about whether to admit a patient to the intensive care unit (ICU) or a general medical ward (Capelastegui et al., 2006). Severity scores have been promoted as useful tools to help clinicians predict outcomes and guide decisions regarding disposition, diagnostic workup, and therapy (Jeong et al., 2013). For instance, the CURB-65 score (confusion, blood urea nitrogen >19 mg/dl, respiratory rate ≥30, low blood pressure, and age ≥65 years) was developed as a clinical prediction rule suitable for use in busy emergency departments and sought to include clinical features of prognostic importance that are easily measurable at the time of initial assessment (Lim et al., 2003).
Two prognostic clinical risk prediction scores specific for COVID-19 are of particular interest: the Brescia-COVID Respiratory Severity Scale (BCRSS) and the Quick COVID-19 Severity Index (qCSI). The BCRSS was developed in Brescia, Italy, during that nation’s COVID-19 crisis. This prediction rule uses patient examination features and the need for escalating respiratory support levels to suggest treatment recommendations. The scale radically simplifies the clinical summary of a patient’s status. It allows clinicians to compare patients, track the trend of a patient’s respiratory severity level over time, and monitor patients nearing a critical action point (Duca et al., 2020). Meanwhile, the qCSI score was derived from a dataset of hospitalized COVID-19 patients in the Northwestern United States. Its primary purpose is to predict critical respiratory illness at 24 h, as defined by high oxygen requirements, non-invasive ventilation, invasive ventilation, or death (Haimovich et al., 2020).
Validation of these prognostic clinical risk prediction scores in different patient populations is needed. Hence this study was performed to evaluate the performance of the qCSI score and the BCRSS prediction rule in predicting ICU admission and in-hospital mortality in patients with COVID-19 pneumonia.
Methods
This retrospective validation cohort study was conducted at AMITA Health Saint Francis Hospital, a teaching community hospital in Cook County, Illinois, United States, between March 2020 and May 2020. Institutional review board approval was obtained from AMITA Health to review and publish information gathered from the medical records to present clinical characteristics and outcomes of patients with COVID-19. Informed consent was waived because of the retrospective nature of the study.
Study population
Data were collected from all consecutive hospitalized adult patients (18 years or older) with COVID-19 admitted during the study period, including active cases at the date of data abstraction. A confirmed case of COVID-19 was defined by a positive result on a reverse-transcriptase polymerase chain reaction (RT-PCR) assay of a specimen collected on a nasopharyngeal swab. Only laboratory-confirmed cases were included. Patients without test results available through the electronic medical records were excluded. Patients admitted for other chief concerns and incidentally found to be COVID-19-positive were considered to have asymptomatic or mild disease and were also excluded from the analysis.
Severity scoring systems
The CURB-65, BCRSS, and qCSI scores were calculated using baseline clinical data collected retrospectively from the patient cohort. The patients were assigned to six risk strata based on the five prognostic factors of CURB-65 (Lim et al., 2003). The BCRSS prediction rule is meant to be dynamic, reassessed frequently, and rescored after interventions; nevertheless, to make the BCRSS prediction rule adequate as a screening tool for clinicians during the initial assessment, the first step of the algorithm was analyzed, which includes the following risk factors: patient wheezing or unable to speak in full sentences while at rest/with minimal effort (replaced with patient reporting shortness of breath, given the retrospective nature of this study), respiratory rate >22, oxygen saturation (SpO2) <90%, and repeat chest X-ray with significant worsening (defined as bilateral or diffuse infiltrates). The patients were then classified into five risk strata based on the four risk factors of the BCRSS (Duca et al., 2020). The qCSI is a 12-point scale that uses only three variables available at the bedside: nasal cannula flow rate, respiratory rate, and minimum documented pulse oximetry. The patients were then assigned to four risk strata (0–3) based on the following scores: 0–3 low risk, 4–6 low-intermediate risk, 7–9 high-intermediate risk, and ≥10 high risk (Haimovich et al., 2020).
Outcome variables
The primary outcome was in-hospital mortality and the secondary outcome was ICU admission. For the primary outcome, patients discharged were considered survivors, and active cases were excluded from the analysis. Patients who were assigned to hospice or full comfort care were considered non-survivors. For the secondary outcome, all patients in the cohort at the cutoff date were included in the analysis.
Statistical analysis
The data are presented as the number (percentage) for categorical variables and median (interquartile range (IQR)) for continuous variables. Categorical variables were compared with the Pearson Chi-square test or Fisher’s exact test, and continuous variables were compared using the Mann–Whitney U-test.
C statistics were used to assess the discriminatory power of the CURB-65, BCRSS, and qCSI scores for predicting outcomes. The C statistic is a summary measure of discrimination that quantifies the ability of the model to assign a high probability. C statistics are equivalent to the AUC, i.e., the area under the receiver operating characteristic (ROC) curve. C statistics range from 0.5 to 1.0; a value below 0.5 indicates a very poor model, a value of 0.5 means that the model is no better at predicting an outcome than random chance, values over 0.7 indicate a good model, values over 0.8 indicate a robust model, and a value of 1 means that the model perfectly predicts those group members who will experience the outcome and those who will not. The sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and likelihood ratio (LR) were also calculated.
The Youden index, defined as (sensitivity + specificity) − 1, was calculated at each cutoff. Only the cutoff point that showed the highest Youden index was reported, as this is considered the optimal cutoff value (Bewick et al., 2004). The discriminatory power of each score was assessed by calculating the area under each ROC curve. The estimated AUC values were compared using the Hanley–McNeil test (Hanley and McNeil, 1982). All p-value analyses were two-sided, and a p-value of less than 0.05 was considered to be significant. The data were analyzed using IBM SPSS Statistics version 23.0 (IBM Corp., Armonk, NY, USA) and MedCalc 19.2.0 statistical software (MedCalc Software, Mariakerke, Belgium).
Results
Patient baseline characteristics
A total of 313 patients were included in the study; all patients had variables for all risk score calculations with no missing values. The baseline characteristics of the cohort are shown in Table 1 . The median age was 68 years (IQR 59–78.5 years), 59.8% were male, and 61.8% were admitted from long-term care facilities (LTCF). The most common comorbidity was hypertension (70.1%), followed by diabetes (43.8%), and 41.4% were former or current smokers.
Table 1.
Baseline characteristics of patients with COVID-19 (N = 313).
| Characteristics | Number (%)/Median (IQR) |
|---|---|
| Demographics | |
| Age (years) | 68 (59–78.5) |
| Male | 182 (58.1%) |
| Caucasian | 119 (38%) |
| Long-term care facility resident | 194 (62%) |
| Comorbidities | |
| Any comorbidity | 295 (95.2%) |
| Two or more comorbidities | 266 (85%) |
| Hypertension | 222 (79.9%) |
| Diabetes | 140 (44.7%) |
| Cardiovascular | 105 (33.5%) |
| Neurocognitive | 113 (36.1%) |
| Malignancy | 33 (10.5%) |
| Smoking (former or current) | 123 (39.3%) |
| Physical examination on presentation | |
| Temperature (°C) | 37.8 (37–38.7) |
| Systolic blood pressure (mmHg) | 37.8 (37–38.7) |
| Oxygen saturation (%) | 93 (88–95) |
| Pulse (bpm) | 96 (80–110) |
| Respiratory rate (rpm) | 22 (20–28) |
| Altered mental status | 137 (43.8%) |
| Laboratory and imaging findings | |
| White blood cell count (4.0–11.0 × 109/l) | 8.1 (5.6–11.5) |
| Absolute lymphocytes (0.6–3.4 × 109/l) | 0.9 (0.6–1.3) |
| Blood urea nitrogen (7–25 mg/dl) | 27 (16–46) |
| Procalcitonin (0.20–0.49 ng/ml) | 1.05 (0.33–3.53) |
| Ferritin (11–307 ng/ml) | 479 (184–1021) |
| C-reactive protein (<1.0 mg/dl) | 10.5 (4.8–16.9) |
| Bilateral or diffuse infiltrates | 192 (61.4%) |
bpm, beats per minute; IQR, interquartile range; rpm, respirations per minute.
Comparison of in-hospital fatality rates and ICU admission rates by risk class
The overall in-hospital fatality rate was 32.3% (101/313) at the cutoff date, and the ICU admission rate was 31.3% (98/313). Table 2, shows the fatality rate and ICU admission rate across different CURB-65, BCRSS, and qCSI risk classes. The risk of in-hospital mortality and ICU admission increased along with increasing risk class in all the scoring systems. Non-survivors were more commonly classified into higher risk classes as compared with survivors (p < 0.001 for CURB-65; p < 0.001 for BCRSS; p < 0.001 for qCSI). Similarly, patients who required ICU admission were more commonly classified into higher risk classes than patients who did not require ICU admission (p = 0.008 for CURB-65; p < 0.001 for BCRSS; p < 0.001 for qCSI). Non-survivors had significantly higher CURB-65 (median 3 (IQR 2–4) vs 2 (IQR 1–3), p < 0.001), BCRSS (3 (IQR 1.5–3) vs 2 (IQR 1–3), p < 0.001), and qCSI scores (5 (IQR 2–9) vs 2 (IQR 0–5), p < 0.001) compared to survivors; this was also the case for patients admitted to the ICU compared to patients not admitted to the ICU (CURB-65: median 3 (IQR 2–3) vs 2 (IQR 1–3), p < 0.001; BCRSS: 3 (IQR 2–4) vs 2 (IQR 1–3), p < 0.001; qCSI: 6.5 (IQR 2–11) vs 2 (IQR 0–3), p < 0.001).
Table 2.
In-hospital fatality rates and ICU admission rates across risk groups.
| In-hospital fatality rate (n = 299) | ICU admission rate (n = 313) | ||||
|---|---|---|---|---|---|
| Risk scores | Number of patients | Died, n (%) | Risk scores | Number of patients | Admitted to ICU, n (%) |
| CURB-65 | CURB-65 | ||||
| 0 | 50 | 5 (10.0) | 0 | 52 | 10 (19.2) |
| 1 | 46 | 3 (6.5) | 1 | 49 | 9 (18.4) |
| 2 | 79 | 21 (26.6) | 2 | 82 | 24 (29.3) |
| 3 | 81 | 40 (49.4) | 3 | 84 | 35 (41.7) |
| 4 | 37 | 26 (70.3) | 4 | 39 | 16 (41.0) |
| 5 | 6 | 6 (100) | 5 | 7 | 4 (57.1) |
| BCRSS | BCRSS | ||||
| 0 | 36 | 5 (13.9) | 0 | 37 | 3 (8.1) |
| 1 | 82 | 20 (24.4) | 1 | 84 | 15 (17.9) |
| 2 | 72 | 21 (29.2) | 2 | 73 | 17 (23.3) |
| 3 | 67 | 33 (49.3) | 3 | 73 | 32 (43.8) |
| 4 | 42 | 22 (52.4) | 4 | 46 | 31 (67.4) |
| qCSI | qCSI | ||||
| 0 | 182 | 42 (22.2) | 0 | 193 | 30 (15.5) |
| 1 | 52 | 27 (51.9) | 1 | 53 | 19 (35.8) |
| 2 | 19 | 7 (36.8) | 2 | 23 | 17 (72.7) |
| 3 | 39 | 25 (64.1) | 3 | 44 | 32 (72.7) |
BCRSS, Brescia-COVID Respiratory Severity Scale; CURB-65, confusion, blood urea nitrogen >19 mg/dl, respiratory rate ≥30, low blood pressure, and age ≥65 years; ICU, intensive care unit; qCSI, Quick COVID-19 Severity Index.
Performance of CURB-65 and BCRSS prediction rules in COVID-19
The sensitivity, specificity, PPV, NPV, and LR for the in-hospital fatality rate and ICU admission rate at different cutoff values for each prediction rule are presented in Table 3, Table 4 . Overall, the CURB-65 score showed higher sensitivity and specificity and higher PPV and NPV for in-hospital mortality than the BCRSS and qCSI scores. For the CURB-65 prediction rule, the highest Youden index was shown at a cutoff point of ≥3 (Youden index 0.450, sensitivity 71.29%, specificity 73.74%). For the BCRSS and the qCSI scoring systems, the highest Youden index was shown at a cutoff point of ≥3 (Youden Index 0.272, sensitivity 54.46%, specificity 72.73%) and ≥1 (Youden index 0.327, sensitivity 58.42%, specificity 74.24%), respectively.
Table 3.
Sensitivity, specificity, positive predictive value, negative predictive value, and likelihood ratio for different CURB-65, qCSI, and BCRSS scores for predicting death (outcome: death, N = 299).
| Risk score | Sensitivity % (95% CI) | Specificity % (95% CI) | PPV % (95% CI) | NPV % (95% CI) | LR (95% CI) |
|---|---|---|---|---|---|
| CURB-65 | |||||
| ≥0 | 100.00 (96.4–100.0) | 0 (0.0–1.8) | 33.8 (33.8–33.8) | NA | 1 (1.0–1.0) |
| ≥1 | 95.05 (88.8–98.4) | 22.73 (17.1–29.2) | 38.6 (36.5–40.7) | 90 (78.7–95.6) | 1.23 (1.1–1.3) |
| ≥2 | 92.08 (85.0–97.5) | 44.44 (37.4–51.7) | 45.8 (42.4–49.2) | 91.7 (84.7–95.6) | 1.66 (1.4–1.9) |
| ≥3 | 71.29 (61.4–79.9) | 73.74 (67.0–79.7) | 58.1 (51.5–64.3) | 83.4 (78.5–87.4) | 2.71 (2.1–3.5) |
| ≥4 | 31.68 (22.8–41.7) | 94.44 (90.3–97.2) | 74.4 (60.5–84.7) | 73.0 (70.3–75.7) | 5.70 (3.0–10.8) |
| ≥5 | 5.94 (2.2–12.5) | 100 (98.2–100.0) | 100 | 67.6 (66.5–68.8) | NA |
| BCRSS | |||||
| ≥0 | 100 (96.4–100.0) | 0 (0.0–1.8) | 33.8 (33.8–33.8) | NA | 1 (1.0–1.0) |
| ≥1 | 95.05 (88.8–98.4) | 15.66 (10.9–21.5) | 36.5 (34.8–38.3) | 86.1 (71.3–93.9) | 1.13 (1.0–1.2) |
| ≥2 | 75.25 (65.7–83.3) | 46.97 (39.9–54.2) | 42.0 (37.9–46.2) | 78.8 (72.0–84.4) | 1.42 (1.2–1.7) |
| ≥3 | 54.46 (44.2–64.4) | 72.73 (66.0–78.8) | 50.5 (43.3–57.6) | 75.8 (71.3–79.8) | 2.0 (1.5–2.7) |
| ≥4 | 21.78 (14.2–31.1) | 89.90 (84.8–93.7) | 52.4 (38.7–65.7) | 69.3 (66.8–71.6) | 2.16 (1.2–3.8) |
| qCSI | |||||
| ≥0 | 100 (96.4–100.0) | 0 (0.0–1.8) | 33.8 (33.8–33.8) | NA | 1.0 (1.0–1.0) |
| ≥1 | 58.42 (48.2–68.1) | 74.24 (67.6–80.2) | 53.6 (46.4–60.7) | 77.8 (73.3–81.7) | 2.27 (1.7–3.0) |
| ≥2 | 31.68 (22.8–41.7) | 86.87 (81.4–91.2) | 55.2 (43.8–66.1) | 71.4 (68.4–74.2) | 2.41 (1.5–3.8) |
| ≥3 | 24.75 (16.7–34.3) | 92.93 (88.4–96.1) | 64.1 (49.3–76.6) | 70.8 (68.3–73.2) | 3.50 (1.9–6.4) |
BCRSS, Brescia-COVID Respiratory Severity Scale; CI, confidence interval; CURB-65, confusion, blood urea nitrogen >19 mg/dl, respiratory rate ≥30, low blood pressure, and age ≥65 years; LR, likelihood ratio; NA, not available; NPV, negative predictive value; PPV, positive predictive value; qCSI, Quick COVID-19 Severity Index.
Table 4.
Sensitivity, specificity, positive predictive value, negative predictive value, and likelihood ratio for different CURB-65, qCSI, and BCRSS scores for predicting ICU admission (outcome: ICU admission, N = 313).
| Risk score | Sensitivity % (95% CI) | Specificity % (95% CI) | PPV % (95% CI) | NPV % (95% CI) | LR (95% CI) |
|---|---|---|---|---|---|
| CURB-65 | |||||
| ≥0 | 100 (96.3–100) | 0.0 (0.0–1.7) | 31.3 (31.3–31.3) | NA | 1.0 (1.0–1.0) |
| ≥1 | 89.80 (82.0–95.0) | 19.53 (14.5–25.5) | 33.7 (31.7–35.8) | 80.8 (68.7–88.9) | 1.12 (1.0–1.2) |
| ≥2 | 80.61 (71.4–87.9) | 38.14 (31.6–45.0) | 37.3 (34.0–40.7) | 81.2 (73.6–87.0) | 1.30 (1.1–1.5) |
| ≥3 | 56.12 (45.7–65.12) | 65.12 (58.3–71.5) | 42.3 (36.3–48.6) | 76.5 (71.8–80.6) | 161 (1.2–2.1) |
| ≥4 | 20.41 (12.9–29.7) | 87.91 (82.8–91.9) | 43.5 (31.1–56.7) | 70.8 (68.4–73.0) | 1.69 (1.0–2.9) |
| 5 | 4.08 (1.1–10.1) | 98.60 (96.0–99.7) | 57.1 (23.3–85.4) | 69.3 (68.3–70.2) | 2.93 (0.7–12.8) |
| BCRSS | |||||
| ≥0 | 100 (96.3–100.0) | 0 (0.0–1.7) | 31.3 (31.3–31.3) | NA | 1 (1.0–1.0) |
| ≥1 | 96.94 (91.3–99.4) | 15.81 (11.2–21.4) | 34.4 (32.9–36.0) | 91.9 (78.1–97.3) | 1.15 (1.1–1.2) |
| ≥2 | 81.63 (72.5–88.7) | 47.91 (41.1–54.8) | 41.7 (37.9–45.6) | 85.1 (78.7–89.9) | 1.57 (1.3–1.8) |
| ≥3 | 64.29 (54.0–73.7) | 73.95 (67.5–79.7) | 52.9 (46.2–59.6) | 82.0 (77.5–85.7) | 2.47 (1.9–3.2) |
| 4 | 31.63 (22.6–41.8) | 93.02 (88.8–96.0) | 67.4 (53.9–78.5) | 74.9 (72.2–77.4) | 4.53 (2.6–8.0) |
| qCSI | |||||
| ≥0 | 100 (96.3–100.0) | 0 (0.0–1.7) | 31.3 (31.3–31.3) | NA | 1.0 (1.0–1.0) |
| ≥1 | 69.39 (59.3–78.3) | 75.81 (69.5–81.4) | 56.7 (49.9–63.2) | 84.5 (80.0–88.1) | 2.87 (2.2–3.8) |
| ≥2 | 50.0 (39.7–60.3) | 91.63 (87.1–95.0) | 73.1 (62.6–81.5) | 80.1 (76.7–83.1) | 5.97 (3.7–9.7) |
| ≥3 | 32.65 (23.5–42.9) | 94.42 (90.5–97.1) | 72.1 (58.9–83.2) | 75.5 (72.7–78.0) | 5.85 (3.2–10.9) |
BCRSS, Brescia-COVID Respiratory Severity Scale; CI, confidence interval; CURB-65, confusion, blood urea nitrogen >19 mg/dl, respiratory rate ≥30, low blood pressure, and age ≥65 years; ICU, intensive care unit; LR, likelihood ratio; NA, not available; NPV, negative predictive value; PPV, positive predictive value; qCSI, Quick COVID-19 Severity Index.
In the case of ICU admission, the BCRSS and qCSI scores showed higher sensitivities at higher cutoff points than the CURB-65 score. Specificity was high for the three scoring systems. PPV and NPV were similar among the scoring systems. The highest Youden index for the CURB-65 score was again shown at a cutoff point of ≥3 (Youden index 0.212, sensitivity 56.12%, specificity 65.12%). For the BCRSS prediction rule, the highest Youden index was shown at a cutoff of ≥3 (Youden index 0.382, sensitivity 64.29%, specificity 73.95%). Finally, the highest Youden index for the qCSI was found at a cutoff of ≥1 (Youden index 0.452, sensitivity 69.39%, specificity 75.81%).
The ROC curves for in-hospital mortality and ICU admission for each prognostic scoring system in COVID-19 are shown in Figure 1, Figure 2 , respectively. The CURB-65 score had the highest numerical AUC to predict in-hospital mortality (AUC 0.781) compared to the qCSI score (AUC 0.711) and the BCRSS prediction rule (AUC 0.663). When comparing the AUCs to predict in-hospital mortality with the Hanley–McNeil test among these scoring systems, the CURB-65 showed a statistically higher discriminatory power compared with the BCRSS prediction rule (p = 0.005). There was no statistical difference between the CURB-65 and the qCSI score (p = 0.060) or between the qCSI score and the BCRSS prediction rule (p = 0.051). For ICU admission, the qCSI score had the highest numerical AUC (AUC 0.761) compared to the BCRSS prediction rule (AUC 0.735) and the CURB-65 score (AUC 0.629). Both the qCSI score and the BCRSS prediction rule had a statistically higher discriminatory power to predict ICU admission than the CURB-65 score (p = 0.001 and p = 0.017, respectively). However, the discriminatory power for ICU admission was not statistically different between the qCSI score and the BCRSS prediction rule (p = 0.267).
Figure 1.
ROC curves for in-hospital mortality for the CURB-65, qCSI, and BCRSS scores. The solid line shows the ROC curve of the CURB-65 score. The dashed line shows the ROC curve of the qCSI score. The dotted line shows the ROC curve of the BCRSS score.
Figure 2.
ROC curves for ICU admission for the CURB-65, qCSI, and BCRSS scores. The solid line shows the ROC curve of the CURB-65 score. The dashed line shows the ROC curve of the qCSI score. The dotted line shows the ROC curve of the BCRSS score.
Discussion
This study evaluated the performance of the CURB-65 score, the qCSI score, and the BCRSS prediction rule in predicting in-hospital mortality and ICU admission in patients with COVID-19 illness. In this patient population, the CURB-65 and qCSI scores were good models for predicting in-hospital mortality, with the CURB-65 having the highest AUC. However, for ICU admission, the qCSI score and the BCRSS prediction rule showed the best performance, with the qCSI having the highest AUC. All of the prediction models had similar trends of increasing in-hospital mortality and rate of ICU admission with worsening risk class, except for the qCSI, which showed a higher in-hospital mortality rate in patients with a score of 1 compared to a score of 2.
Emerging viral pandemics place extraordinary demands on health systems, creating the need to ration medical equipment and interventions (Emanuel et al., 2020). The current COVID-19 epidemic has already overwhelmed health systems around the globe. Although the pandemic appears to be slowing in some areas, concerns exist for resurgence waves in the setting of relaxation of mitigation measures and the upcoming influenza season. Resources, including ICU beds, ventilators, and personal protective equipment, may again become scarce, mandating for evidence-based and ethical resource allocation strategies (Kirkpatrick et al., 2020).
For community-acquired pneumonia (CAP), several prognostic scoring systems have been developed and studied since 1982, including the British Thoracic Society rule (Anon, 1987), the modified British Thoracic Society rule (Neill et al., 1996), the Pneumonia Severity Index (Fine et al., 1997), and the CURB-65 score (Lim et al., 2003), among others. Recently, the performance of these existing CAP severity scores has been tested in patients with COVID-19 (Nguyen et al., 2020, Satici et al., 2020, Fan et al., 2020). Nevertheless, COVID-19 has proved to be more than just pneumonia, and its clinical spectrum varies from asymptomatic forms to systemic manifestations in terms of sepsis, septic shock, and multiple organ dysfunction syndromes, and affecting countries, races, and ages in different, and sometimes unpredictable, manners (Cascella et al., 2020, Wiersinga et al., 2020). Given the uncertainty around the COVID-19 illness process and prognosis and its elevated morbidity and mortality, developing specific clinical risk stratification tools may be more appropriate for these patients.
In the patient population in the present study, the CURB-65 and the qCSI scores performed well in predicting inpatient mortality. The qCSI score and the BCRSS prediction rule were superior to the CURB-65 score in predicting ICU admission. Of note, the CURB-65 showed lower AUCs as compared with previous reports. Satici et al. (2020) reported an AUC of 0.88 (95% confidence interval (CI) 0.85–0.90) in a Turkish population, while Fan et al. (2020) reported an AUC of 0.85 (95% CI 0.81–0.89) in Wuhan, China. This difference in performance may be related to the differences in clinical characteristics and demographics across populations. The community in the present study was characterized by an elevated baseline disease burden, mainly driven by old LTCF residents with several comorbidities; up to 95% of the admitted patients had one comorbidity, and 85% had at least two major underlying conditions. Compared to the study by Satici et al. (2020), our patients were 11 years older (68 vs 57 years), with a larger proportion having at least one comorbidity (95% vs 54%). Fan et al. (2020) did not present the clinical characteristics and demographics of their patients. However, in two previous retrospective cohort studies from Wuhan, China, Zhou et al. (2020) and Guan et al. (2020) reported a median age of 56 and 47 years, respectively, with 47% and 24% of the patients having at least one comorbidity, respectively.
The AUCs for the qCSI score with regards to in-hospital mortality and ICU admission in the present study (0.781 and 0.761, respectively) were similar to that described by Haimovich et al. (2020) in the independent validation cohort for their outcome of critical respiratory illness (AUC 0.81), defined as oxygenation flow rate ≥10 L/min, high-flow oxygenation, non-invasive ventilation, invasive ventilation, or death. Interestingly, they reported poor performance of the CURB-65 score to predict critical respiratory illness (AUC 0.66). In the qCSI validation cohort, a cutoff of ≥1 in the risk strata (or a score ≥4 on the 12-point scale) was associated with a specificity of 78% (95% CI 72–83%) in predicting progression to respiratory failure; in agreement, we found the highest Youden index for the qCSI at a cutoff of ≥1, with a specificity of 76%. A qCSI of 3 (or a score ≥10 on the 12-point scale) had a specificity of 99% in predicting respiratory failure with a LR of 8.36 (95% CI 7.9–8.7) in the validation cohort, while we report a specificity of 93% for the same value, with a LR of 3.5 (95% CI 1.9–6.4) for death and a specificity of 94% with a LR of 5.85 (95% CI 3.2–10.9) for ICU admission. For the BCRSS prediction rule, we found no reports assessing its performance outside the Italian population. Hence, this work should be considered as a validation study.
This study has several limitations. First, given its retrospective observational nature, there is a lack of control of variables. Second, the dataset was extracted from the electronic health records, and assumptions regarding subjective variables for score calculations had to be made when they were not explicitly specified in the chart, which may have led to an under or overestimation of the reported scores, i.e., confusion for the CURB-65 score and shortness of breath for the BCRSS prediction rule. Third, this was a single-center study, and local admission and COVID-19 management practices at this center vary from those in other institutions in the United States and other countries, which may have impacted the outcomes of mortality and ICU admission. This relates in particular to patients from LTCF with advance directives before presentation to the hospital.
In conclusion, the CURB-65 and qCSI scoring systems showed a good performance for predicting in-hospital mortality in the study population of patients. In contrast, the qCSI score and the BCRSS prediction rule showed a good performance for predicting ICU admission. Prospective research is needed to evaluate the robustness and utility of newly developed COVID-19 risk stratification models. However, given the high heterogeneity across populations affected by COVID-19, site-tailored prediction models, guided by local patient characteristics and hospital policies, may be more appropriate.
Funding
There was no financial support for this work.
Ethical approval
Approval for this work was obtained through the AMITA Health Institutional Review Board and Ethics Committee.
Consent to participate
Informed consent was waived because of the retrospective nature of the study.
Availability of data and material
Data and materials used for this work are available upon reasonable request.
Conflict of interest
The authors have no conflicts of interest to disclose.
Acknowledgements
We dedicate this work to all of the victims of COVID-19 in the City of Evanston and around the world.
References
- Bauchner H., Sharfstein J. A Bold Response to the COVID-19 Pandemic: Medical Students, National Service, and Public Health. JAMA. 2020;323(18):1790–1791. doi: 10.1001/jama.2020.6166. [DOI] [PubMed] [Google Scholar]
- Bewick V., Cheek L., Ball J. Statistics review 13: receiver operating characteristic curves. Crit Care. 2004;8(6):508–512. doi: 10.1186/cc3000. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Capelastegui A., España P.P., Quintana J.M. Validation of a predictive rule for the management of community-acquired pneumonia. Eur Respir J. 2006;27(1):151–157. doi: 10.1183/09031936.06.00062505. [DOI] [PubMed] [Google Scholar]
- Cascella M., Rajnik M., Cuomo A., Dulebohn S.C., Di Napoli R. StatPearls [Internet] Treasure Island (FL): StatPearls Publishing; 2020. Features, Evaluation, and Treatment of Coronavirus. 2020 October 4. Jan–. PMID. [PubMed] [Google Scholar]
- Cavallo J.J., Donoho D.A., Forman H.P. Hospital Capacity and Operations in the Coronavirus Disease 2019 (COVID-19) Pandemic—Planning for the Nth Patient. JAMA J Am Med Assoc [Internet] 2020 doi: 10.1001/jamahealthforum.2020.0345. (March 17 2020). https://jamanetwork.com/channels/health-forum/fullarticle/2763353 (Accessed 25 September 2020) [DOI] [PubMed] [Google Scholar]
- Centers for Disease Control and Prevention . 2020. Coronavirus Disease 2019 (COVID-19) - Cases in the US.https://www.cdc.gov/coronavirus/2019-ncov/cases-updates/cases-in-us.html [Google Scholar]
- Anon Community-acquired pneumonia in adults in British hospitals in 1982-1983: a survey of aetiology, mortality, prognostic factors and outcome. The British Thoracic Society and the Public Health Laboratory Service. Q J Med. 1987;62(239):195–220. PMID: 3116595. [PubMed] [Google Scholar]
- Coronaviridae Study Group of the International Committee on Taxonomy of Viruses The species Severe acute respiratory syndrome-related coronavirus: classifying 2019-nCoV and naming it SARS-CoV-2. Nat Microbiol. 2020;5(4):536–544. doi: 10.1038/s41564-020-0695-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Duca A., Piva S., Focà E., Latronico N., Rizzi M. Calculated Decisions: Brescia-COVID Respiratory Severity Scale (BCRSS)/Algorithm. Emerg Med Pract. 2020;22(5 Suppl):CD1–CD2. PMID: 32297727. [PubMed] [Google Scholar]
- Emanuel E.J., Persad G., Upshur R. Fair Allocation of Scarce Medical Resources in the Time of Covid-19. N Engl J Med. 2020;382(21):2049–2055. doi: 10.1056/NEJMsb2005114. [DOI] [PubMed] [Google Scholar]
- Kirkpatrick J.N., Hull S.C., Fedson S., Mullen B., Goodlin S.J. Scarce-Resource Allocation and Patient Triage During the COVID-19 Pandemic: JACC Review Topic of the Week. J Am Coll Cardiol. 2020;76(1):85–92. doi: 10.1016/j.jacc.2020.05.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Fan G., Tu C., Zhou F. Comparison of severity scores for COVID-19 patients with pneumonia: a retrospective study. Eur Respir J. 2020;56(3) doi: 10.1183/13993003.02113-2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Fine M.J., Auble T.E., Yealy D.M. A prediction rule to identify low-risk patients with community-acquired pneumonia. N Engl J Med. 1997;336(4):243–250. doi: 10.1056/NEJM199701233360402. [DOI] [PubMed] [Google Scholar]
- Guan W.J., Ni Z.Y., Hu Y. Clinical Characteristics of Coronavirus Disease 2019 in China. N Engl J Med. 2020;382(18):1708–1720. doi: 10.1056/NEJMoa2002032. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Haimovich A.D., Ravindra N.G., Stoytchev S., Young H.P., Wilson F.P., van Dijk D. Development and validation of the quick covid-19 severity index: a prognostic tool for early clinical decompensation. Ann Emerg Med. 2020;76(4):442–453. doi: 10.1016/j.annemergmed.2020.07.022. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Hanley J.A., McNeil B.J. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143(1):29–36. doi: 10.1148/radiology.143.1.7063747. [DOI] [PubMed] [Google Scholar]
- Jeong B.H., Koh W.J., Yoo H. Performances of prognostic scoring systems in patients with healthcare-associated pneumonia. Clin Infect Dis. 2013;56(5):625–632. doi: 10.1093/cid/cis970. [DOI] [PubMed] [Google Scholar]
- Lim W.S., van der Eerden M.M., Laing R. Defining community-acquired pneumonia severity on presentation to hospital: an international derivation and validation study. Thorax. 2003;58(5):377–382. doi: 10.1136/thorax.58.5.377. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Liu Y.C., Kuo R.L., Shih S.R. COVID-19: The first documented coronavirus pandemic in history. Biomed J. 2020;43(4):328–333. doi: 10.1016/j.bj.2020.04.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Neill A.M., Martin I.R., Weir R. Community acquired pneumonia: aetiology and usefulness of severity criteria on admission. Thorax. 1996;51(10):1010–1016. doi: 10.1136/thx.51.10.1010. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Nguyen Y., Corre F., Honsel V. Applicability of the CURB-65 pneumonia severity score for outpatient treatment of COVID-19. J Infect. 2020;81(3):e96–e98. doi: 10.1016/j.jinf.2020.05.049. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Paranthaman K., Conlon C.P., Parker C., Mccarthy N. Resource allocation during an influenza pandemic. Emerging Infect Dis. 2008;14(3):520–522. doi: 10.3201/eid1403.071275. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Satici C., Demirkol M.A., Sargin altunok E. Performance of pneumonia severity index and CURB-65 in predicting 30-day mortality in patients with COVID-19. Int J Infect Dis. 2020;98:84–89. doi: 10.1016/j.ijid.2020.06.038. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Taubenberger J.K., Morens D.M. 1918 Influenza: the mother of all pandemics. Emerging Infect Dis. 2006;12(1):15–22. doi: 10.3201/eid1201.050979. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Wiersinga W.J., Rhodes A., Cheng A.C., Peacock S.J., Prescott H.C. Pathophysiology, Transmission, Diagnosis, and Treatment of Coronavirus Disease 2019 (COVID-19): A Review. JAMA. 2020;324(8):782–793. doi: 10.1001/jama.2020.12839. [DOI] [PubMed] [Google Scholar]
- Zhou F., Yu T., Du R. Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study. Lancet. 2020;395(10229):1054–1062. doi: 10.1016/S0140-6736(20)30566-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Zhu N., Zhang D., Wang W. A Novel Coronavirus from Patients with Pneumonia in China, 2019. N Engl J Med. 2020;382(8):727–733. doi: 10.1056/NEJMoa2001017. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
Data and materials used for this work are available upon reasonable request.