Predicting In‐Hospital Mortality in COVID‐19 Older Patients with Specifically Developed Scores

Abstract

Background/Objectives

Several scoring systems have been specifically developed for risk stratification in COVID‐19 patients.

Design

We compared, in a cohort of confirmed COVID‐19 older patients, three specifically developed scores with a previously established early warning score. Main endpoint was all causes in‐hospital death.

Setting

This is a single‐center, retrospective observational study, conducted in the Emergency Department (ED) of an urban teaching hospital, referral center for COVID‐19.

Participants

We reviewed the clinical records of the confirmed COVID‐19 patients aged 60 years or more consecutively admitted to our ED over a 6‐week period (March 1st to April 15th, 2020). A total of 210 patients, aged between 60 and 98 years were included in the study cohort.

Measurements

International Severe Acute Respiratory Infection Consortium Clinical Characterization Protocol‐Coronavirus Clinical Characterization Consortium (ISARIC‐4C) score, COVID‐GRAM Critical Illness Risk Score (COVID‐GRAM), quick COVID‐19 Severity Index (qCSI), National Early Warning Score (NEWS).

Results

Median age was 74 (67–82) and 133 (63.3%) were males. Globally, 42 patients (20.0%) deceased. All the score evaluated showed a fairly good predictive value with respect to in‐hospital death. The ISARIC‐4C score had the highest area under ROC curve (AUROC) 0.799 (0.738–0.851), followed by the COVID‐GRAM 0.785 (0.723–0.838), NEWS 0.764 (0.700–0.819), and qCSI 0.749 (0.685–0.806). However, these differences were not statistical significant.

Conclusion

Among the evaluated scores, the ISARIC‐4C and the COVID‐GRAM, calculated at ED admission, had the best performance, although the qCSI had similar efficacy by evaluating only three items. However, the NEWS, already widely validated in clinical practice, had a similar performance and could be appropriate for older patients with COVID‐19.

1. INTRODUCTION

The novel coronavirus designated SARS‐CoV‐2, has determined an international outbreak of respiratory illness named COVID‐19. ¹ , ² Older adults and patients with previous comorbid conditions are at higher risk of developing severe disease, and death. ³ , ⁴ , ⁵

The prevalence of hypoxic respiratory failure in patients hospitalized with COVID‐19 was estimated to be about 19%, with up to 12% of patients requiring mechanical ventilation. ¹ , ² , ⁴ Indeed, based on available data, from 5% to 10% among hospitalized patients will require ICU admission, with rates even higher in older patients. ⁵ , ⁶ , ⁷ , ⁸

In this context of critically ill patients' overflow, it is mandatory to establish clear and objective criteria to stratify COVID‐19 risk for death. To date, already available national early warning scores (NEWS), and specifically developed clinical rules and scores, have been proposed for risk stratification in COVID‐19 patients. ⁹ , ¹⁰ , ¹¹ , ¹² , ¹³ Currently, even though most of developed scores include age among the factors evaluated for risk prediction, none of these tools was validated in a geriatric population, which indeed carries the highest risk of worse outcome in COVID‐19.

The aim of this study is to evaluate, in older patients with COVID‐19, the performance for death risk stratification of specifically developed scoring systems, including the International Severe Acute Respiratory Infection Consortium Clinical Characterization Protocol‐Coronavirus Clinical Characterization Consortium (ISARIC‐4C) score, the COVID‐GRAM Critical Illness Risk Score (COVID‐GRAM), the quick COVID‐19 Severity Index (qCSI). ¹¹ , ¹² , ¹³ These specifically developed scores were compared with the widely validated NEWS risk score. ¹⁴

2. METHODS

2.1. Study Design

This is a single‐center, retrospective observational study, conducted in the ED of an urban teaching hospital, which is a referral center for COVID‐19, in central Italy.

We reviewed the clinical records of all the patients 60 years or more old consecutively admitted to our ED over a six‐week period (from March 1st to April 15th, 2020). COVID‐19 was diagnosed on the basis of the WHO interim guidance. We included in the analysis only patients with positive result on real‐time reverse‐transcriptase–polymerase‐chain‐reaction assay of nasal and pharyngeal swab specimens. ¹⁵

We excluded patients already on orotracheal intubation at ED arrival, and patients for whom a do not resuscitate order was in place.

2.2. Study Variables

The following information were extracted from computerized clinical records: age, sex, clinical presentation symptoms, temperature, heart rate (HR), respiratory rate (RR), blood pressure (BP), Glasgow Coma Scale (GCS) score, oxygen supplementation, peripheral oxygen saturation (SpO₂), laboratory values, radiographic imaging, and clinical history. Physiological parameters were assessed at ED admission. Comorbidities were evaluated according to Charlson comorbidity index. ¹⁶

2.3. Early Warning Scores for COVID‐19 Risk Stratification

Four early warning scores were evaluated: three were specifically developed for COVID‐19 (ISARIC‐4c, COVID‐GRAM, qCSI), while the NEWS score was recently validated in this setting. ¹¹ , ¹² , ¹³ , ¹⁴ The qCSI assesses the respiratory function; the COVID‐GRAM, the ISARIC 4C, and the NEWS also include the assessment of cardiovascular function, level of consciousness, age, number of comorbidities, and a selection of laboratory tests (Supplementary Table S1).

All the parameters evaluated for scores calculation were obtained from ED electronic records.

2.4. Study Endpoint

The primary study endpoint was all‐causes in hospital death.

2.5. Statistical Analysis

Continuous variables are reported as median (interquartile range), and are compared at univariate analysis by Mann–Whitney U test. Categorical variables are reported as absolute number (percentage), and are compared by chi‐square test (with Fisher's test if appropriate).

For patients with incomplete dataset of parameters to calculate the scores (either vital parameters or laboratory values), we utilized a data imputation by using a multiple imputation approach. ¹⁷ We excluded patients with three or more parameters missing, since the effect on final scores calculation would have been highly unpredictable. The missing parameters were imputed by using a multiple regression model including the available parameters in the dataset, the triage code at ED admission, and patient age. The limit of imputed parameters was set according to each parameter range in the study cohort.

Once the selected scores were calculated for each patient, receiver operating characteristic (ROC) curve analysis was used to evaluate the overall performance in predicting the defined adverse outcome. Youden's index was used to estimate optimal cutoff points and corresponding sensitivity and specificity at selected score threshold values. The comparison between the ROC AUCs was made according to DeLong method. ¹⁸

A two sided P value .05 or less was regarded as significant. Data were analyzed by SPSS v25® (IBM, IL).

2.6. Statement of Ethics

The study was conducted in accordance with the Declaration of Helsinki and its later amendments, and was approved by the local Institutional Review Board (IRB #001705520).

3. RESULTS

A total of 210 patients, aged between 60 and 98 years met the inclusion criteria and were included in the study cohort (Supplementary Figure S1). Median age was 74 (67–82) and 133 (63.3%) were males (Table 1).

Table 1.

Demographic and Clinical Characteristics of Enrolled Patients

	All Population	Survived	Deceased
Variable	n = 210	n = 168	n = 42	P
Age	74 (67–82)	72 (66–80)	81 (74–85)	<.001
Sex (male)	133 (63.3)	106 (63.1)	27 (64.3)	.886
Physiological parameters at ED presentation
Peripheral oxygen saturation (%)	94 (90–96)	94 (92–96)	89 (80–92)	<.001
Respiratory rate (breaths/min)	18 (16–20)	18 (15–20)	19 (16–22)	.032
Heart rate (beats/min)	84 (72–99)	82 (71–95)	89 (76–110)	.011
Systolic blood pressure (mmHg)	130 (114–140)	129 (113–141)	128 (111–135)	.790
Diastolic blood pressure (mmHg)	78 (66–86)	78 (70–87)	73 (65–84)	.404
Axillary temperature (°C)	36.6 (36.0–37.5)	36.5 (36.1–37.3)	37.2 (36.2–38.2)	.711
Radiological findings
Negative	27 (12.9)	26 (15.5)	1 (2.4)
Interstitial/monolateral	110 (52.3)	101 (60.1)	9 (21.4)	<.001
Bilateral pneumonia	73 (34.8)	41 (24.4)	32 (76.2)
Comorbidities
Charlson comorbidity index	4 (3–5)	4 (3–5)	5 (4–6)	<.001
Hypertension	120 (57.1)	92 (54.8)	28 (66.7)	.163
Obesity	4 (1.9)	3 (1.8)	1 (2.4)	1.000
Coronary artery disease	45 (21.4)	38 (22.6)	7 (16.7)	.400
Congestive heart failure	39 (18.6)	30 (17.9)	9 (21.4)	.594
Diabetes mellitus	27 (12.9)	23 (13.7)	4 (9.5)	.471
Dementia	10 (5.7)	3 (2.2)	7 (17.5)	<.001
COPD	19 (9.0)	13 (7.7)	6 (14.3)	.186
Renal disease	27 (12.9)	13 (7.7)	14 (33.3)	<.001
Malignancy	15 (7.1)	12 (7.1)	3 (7.1)	1.000
Laboratory values
Hemoglobin (g/dl)	12.7 (9.0–14.3)	12.9 (9.0–14.2)	12.4 (9.1–14.5)	.953
Neutrophil (cells/mm3)	4,890 (3,570–6,930)	4,690 (3,540–6,610)	5,930 (3,715–8,935)	.019
Lymphocyte (cells/mm3)	940 (670–1,290)	950 (695–1,280)	825 (570–1,600)	.591
Neutrophil/lymphocyte ratio	5.3 (3.3–8.1)	5.1 (3.2–7.6)	6.5 (3.9–12.2)	.026
Creatinine (mg/dl)	0.98 (0.78–1.42)	0.96 (0.75–1.27)	1.45 (0.88–2.05)	.003
Blood urea nitrogen (mg/dl)	20 (16–33)	19 (15–25)	37 (20–58)	<.001
Sodium (mEq/L)	138 (135–140)	138 (135–140)	138 (134–141)	.692
Lactate dehydrogenase (UI/L)	324 (241–440)	306 (233–412)	511 (314–801)	<.001
Alanina transpherase (UI/L)	19 (13.5–32.5)	18.5 (13–30.5)	21 (15–46)	.248
Direct bilirubin (mg/dl)	0.6 (0.4–0.9)	0.6 (0.4–0.9)	0.6 (0.4–1.0)	.725
C‐reactive protein (mg/L)	66.8 (28.1–141.0)	53.1 (25.7–105.6)	145 (77.9–210.5)	<.001
Prothrombin time (s)	11.2 (10.7–11.9)	11.2 (10.6–11.8)	11.4 (10.8–12.4)	.220
Fibrinogen (mg/dl)	478 (392–580)	465 (390–550)	512 (405–703)	.088
D‐dimer (ng/ml)	1,230 (709–3,359)	1,228 (619–2,771)	2,071 (900–5,412)	.194
Risk scores
NEWS	3 (2–6)	3 (1–5)	6 (3–9)	<.001
ISARIC 4C	9 (7–10)	8 (6–10)	11 (9–12)	<.001
COVID‐GRAM	17.3 (8.5–34.4)	13.9 (7.1–26.9)	38.1 (23.8–56.8)	<.001
qCSI	4 (0–6)	2 (0–5)	7 (4–10)	<.001

Abbreviations: COPD, chronic obstructive pulmonary disease; COVID‐GRAM, COVID‐Gram Critical Illness Risk Score; ISARIC‐4C, International Severe Acute Respiratory Infection Consortium Clinical Characterization Protocol‐Coronavirus Clinical Characterization Consortium; NEWS, national early warning score; qCSI, quick COVID severity index.

Globally, 42 patients (20.0%) deceased (Table 1). When compared with survived patients, we found that deceased patients were significantly older (81 (74–85) vs 72 (66–80); P < .001), had worse radiological findings, and had a higher number of comorbidities (Charlson comorbidity index 5 (4–6) vs 4 (3–5); P < .001) (Table 1). In particular, deceased patients had a higher rate of dementia (17.5% vs 2.2%, P < .001), and a higher rate of renal disease (33.3% vs 7.7%, P < .001).

Among vital parameters at admission SpO₂, respiratory rate and heart rate were significantly worse in deceased patients, whereas the two groups had similar admission values in term of temperature and blood pressure (Table 1).

Laboratory values at admission associated to death were higher C‐reactive protein (CRP), blood urea nitrogen, LDH, absolute neutrophil count, and neutrophil/lymphocyte ratio (Table 1).

All the four scores evaluated showed a fairly good predictive value with respect to in‐hospital death. The ISARIC‐4C score had the highest area under ROC curve (AUROC) 0.799 (0.738–0.851), followed by the COVID‐GRAM 0.785 (0.723–0.838), NEWS 0.764 (0.700–0.819), and qCSI 0.749 (0.685–0.806) (Figure 1). However, these differences were not statistically significant.

Figure 1.

Graphical representation of the receiver operating characteristic (ROC) curve of the evaluated score.

When comparing score sensitivity, COVID‐GRAM and ISARIC‐4C had the best performance, both reaching 88.1% sensitivity for COVID‐GRAM greater than 17.7 and ISARIC‐4C greater than 8 (Table 2). However, COVID‐GRAM had a slightly higher negative predictive value (Table 2). The qCSI had the best specificity, thus having a qCSI greater than 5 the highest positive predictive value for death (43.3 (35.1–51.9)) (Table 2). The worst performer in this group was the NEWS, still keeping a fair negative predictive value of 89.2 (84.2–92.8) at selected cutoff.

Table 2.

Sensitivities, Specificities, Negative and Positive Predictive Values, Positive and Negative Likelihood Ratios for NEWS, COVID‐GRAM, ISARIC‐4C, and qCSI Scoring Systems for Predicting Death of COVID‐19 Older Patients

Score	ROC AUC	Cut off value	Sensitivity (%)	Specificity (%)	+LR	−LR	PPV	NPV
NEWS	0.764 (0.700–0.819)	>4	66.7 (50.5–80.4)	69.0 (61.5–75.9)	2.1 (1.6–2.9)	0.5 (0.3–0.7)	35.0 (28.3–42.4)	89.2 (84.2–92.8)
COVID‐GRAM	0.785 (0.723–0.838)	>17.7	88.1 (74.4–96.0)	61.3 (53.5–68.7)	2.3 (1.8–2.8)	0.2 (0.1–0.4)	36.3 (31.3–41.5)	95.4 (90.0–97.9)
ISARIC‐4C	0.799 (0.738–0.851)	>8	88.1 (74.4–96.0)	55.9 (48.1–63.6)	2.0 (1.6–2.5)	0.2 (0.1–0.5)	33.3 (29.0–38.0)	94.9 (89.1–97.7)
qCSI	0.749 (0.685–0.806)	>5	69.0 (52.9–82.4)	77.4 (70.3–83.5)	3.0 (2.2–4.3)	0.4 (0.3–0.6)	43.3 (35.1–51.9)	90.9 (86.3–94.1)

Note: Optimal cutoff was chosen according to Youden index J. Differences among area under ROC curves did not reach statistical significance.

Abbreviations: NEWS, National early warning score; COVID‐GRAM, COVID‐Gram Critical Illness Risk Score; ISARIC‐4C, International Severe Acute Respiratory Infection Consortium Clinical Characterization Protocol‐Coronavirus Clinical Characterization Consortium; qCSI, quick COVID severity index; +LR, positive likelihood ratio; −LR, negative likelihood ratio; NPV, negative predictive value; PPV, Positive predictive value.

4. DISCUSSION

The main result of present study is that among COVID‐19 older patients the specifically developed scores ISARIC‐4C, COVID‐GRAM, and qCSI, although slightly superior in terms of overall AUROC and sensitivity, do not perform significantly better than the standard NEWS. However, the qCSI gave the best results in terms of specificity by evaluating only three parameters.

The SARS‐CoV‐2 primarily infects the upper respiratory and gastrointestinal tracts, ¹⁹ binding to human angiotensin‐converting enzyme 2 for cell entry. ¹ , ² , ⁴ , ⁵ , ²⁰ Severe hypoxia and respiratory distress are common features of COVID‐19, and septic shock occurs mainly as a result of end‐stage organ failure. ¹ , ² , ⁵ Radiological findings confirm the extensive lung involvement, and up to 98% of symptomatic patients show bilateral ground glass opacity, and multiple lobular and subsegmental consolidation areas at chest imaging. ²⁰

The results of the present study are largely explained by both the underlying pathophysiological mechanisms and the clinical presentation of COVID‐19. Indeed, since the acute hypoxia is the main determinant of disease progression and severity, the evaluation of respiratory function is crucial for score prediction ability.

All the evaluated scores include an assessment of respiratory function, even if obtained in different ways. The NEWS includes both the SpO₂ and the respiratory rate in the calculation, as well as the ISARC‐4C and the qCSI. For the COVID‐GRAM calculation, the respiratory function is indirectly derivated by the assessment of X‐ray abnormalities, and directly evaluated as the presence of dyspnea, as reported by the patient. The qCSI and the NEWS both evaluate the supplemental oxygen flow given to patients, although this latter measure has a high variability, being not always directly linked to effective patient's respiratory distress.

Apart from qCSI that evaluate only respiratory distress, all the scores evaluate neurological status, by using a simplified version of GCS (normal or <15 for ISARIC‐4C), the Alert, Verbal, Pain, Unresponsive (AVPU) scale (for NEWS), and simply conscious/unconscious for COVID‐GRAM. However, although neurologic involvement is common in COVID‐19, a severe depression of consciousness is rare. ²¹ Indeed, in our cohort only seven (3.3%) patients presented with GCS less than 15 at admission, and three (1.4%) were unconscious. Hence, the contribution of this item to the score prediction was low.

The relatively low incidence of shock in COVID‐19 ¹ , ² could explain why the blood pressure in ED did not seem to be associated to worse outcome (Table 1) in our population. Indeed, none of the specifically designed scores for COVID‐19 evaluates blood pressure.

Both COVID‐GRAM and ISARIC 4C evaluate patients comorbidities. While the latter utilizes the Charlson Index adding obesity, ²² the COVID‐GRAM evaluates a selected number of conditions, including hypertension and hepatitis B. As our study was conducted in a population of COVID‐19 older patients, most of them presenting comorbidities, the influence of this item was reduced for the overall prediction. Nevertheless, our data confirmed that deceased patients showed an overall higher Charlson comorbidity index, but dementia and renal disease were significantly higher. Indeed, COVID‐19 patients with cognitive impairment are at high risk of worse outcome, and this is a major challenge in geriatric populations. ⁶ , ²³ , ²⁴

Among the scores we assessed, ISARIC‐4C and COVID‐GRAM include laboratory tests in their model. ISARIC‐4C includes blood urea nitrogen and CRP, whereas COVID‐GRAM include lactate dehydrogenase (LDH) and direct bilirubin. CRP and LDH were already described to be associated to advanced pulmonary disease in COVID‐19, ²⁵ , ²⁶ as well as kidney damage and increased blood urea nitrogen, ²⁷ as confirmed in our study. Conversely, we cannot confirm the usefulness of bilirubin evaluation since the hepatic involvement in our cohort was limited.

Both the COVID‐GRAM and the ISARIC‐4C assign an increased risk value for older age. However, in our selected population of patients above 60 years the weight of age was probably reduced because of the limited age range. This may partly explain why the PPV for hospital mortality of both COVID‐GRAM and ISARIC‐4C was lower than in the original reports. ¹¹ , ¹³ Among the scores we tested, qCSI had the highest PPV for predicting hospital mortality (43.3% for qCSI > 5). The most likely reason is that qCSI is focused on respiratory failure, which is the major cause of death in COVID‐19 patients. Despite the AUROC of qCSI was the lowest among the scores we tested in our study, this score may be preferred for a quick bedside detection of patients at higher risk of adverse events. In fact, qCSI requires only three clinical parameters (respiratory rate, pulse oximetry, and oxygen flow rate). Conversely, despite the higher complexity and the need of laboratory tests, ISARIC‐4C and COVID‐GRAM showed a high NPV (95%) and, as such, they can be used to exclude the risk of subsequent deterioration in patients destined to a non‐critical area.

Finally, only ISARIC‐4C considers the gender in risk prediction. Nonetheless, although male sex was associated to worse outcome in several reports, ³ , ⁴ , ⁵ patient gender was not significantly associated to a different outcome in our population (Table 1).

4.1. Study Limitations

As for any retrospective study some limitations are worth considering. First, our sample size is limited and therefore, the global accuracy of our ROC curve estimation could be reduced, still keeping a good reliability in ROC curve comparison. Moreover, we did not collect data about total time of eventual O₂ supplementation before ED admission, and this could affect the SpO₂ measurement at ED arrival.

4.2. Conclusions

Among the evaluated scores, the ISARIC‐4C and the COVID‐GRAM, calculated at ED admission, had the best performance in predicting death in COVID‐19 older patients. Moreover, the qCSI, although not specifically designed for death risk prediction had similar efficacy by evaluating only three items, being the best choice for a quick assessment. However, the longtime validated NEWS had a similar performance and, since it represents the standard early warning score in many institutions, could be appropriate also for older patients with COVID‐19.

Supporting information

Supplementary Figure S1: Flow‐chart of the cohort selection for the study.

Supplementary Table S1: Early warning scores for COVID‐19 risk stratification.