Abstract
Background:
Predicting mortality from COVID-19 using information available when patients present to the emergency department can inform goals-of-care decisions and assist with ethical allocation of critical care resources. The study objective was to develop and validate a clinical score to predict emergency department and in-hospital mortality among consecutive nonpalliative patients with COVID-19; in this study, we define palliative patients as those who do not want resuscitative measures, such as intubation, intensive care unit care or cardiopulmonary resuscitation.
Methods:
This derivation and validation study used observational cohort data recruited from 46 hospitals in 8 Canadian provinces participating in the Canadian COVID-19 Emergency Department Rapid Response Network (CCEDRRN). We included adult (age ≥ 18 yr) nonpalliative patients with confirmed COVID-19 who presented to the emergency department of a participating site between Mar. 1, 2020, and Jan. 31, 2021. We randomly assigned hospitals to derivation or validation, and prespecified clinical variables as candidate predictors. We used logistic regression to develop the score in a derivation cohort and examined its performance in predicting emergency department and in-hospital mortality in a validation cohort.
Results:
Of 8761 eligible patients, 618 (7.0%) died. The CCEDRRN COVID-19 Mortality Score included age, sex, type of residence, arrival mode, chest pain, severe liver disease, respiratory rate and level of respiratory support. The area under the curve was 0.92 (95% confidence interval [CI] 0.90–0.93) in derivation and 0.92 (95% CI 0.90–0.93) in validation. The score had excellent calibration. These results suggest that scores of 6 or less would categorize patients as being at low risk for in-hospital death, with a negative predictive value of 99.9%. Patients in the low-risk group had an in-hospital mortality rate of 0.1%. Patients with a score of 15 or higher had an observed mortality rate of 81.0%.
Interpretation:
The CCEDRRN COVID-19 Mortality Score is a simple score that can be used for level-of-care discussions with patients and in situations of critical care resource constraints to accurately predict death using variables available on emergency department arrival. The score was derived and validated mostly in unvaccinated patients, and before variants of concern were circulating widely and newer treatment regimens implemented in Canada.
Study registration:
Throughout the COVID-19 pandemic, health care systems around the world have been confronted with, and at times overwhelmed by, high numbers of critically ill patients.1,2 For every critically ill patient in the emergency department, several less severely ill patients present for care; in some cases, their condition deteriorates later, which places additional pressure on resources. Accurate, disease-specific mortality prediction is needed to inform shared decision-making with patients and their families around the patients’ goals of care. Disease-specific mortality prediction can also allow health care systems to allocate resources in the most transparent, objective and fair manner possible to save as many lives as possible and facilitate timely access to palliative care, if needed.3–5
Numerous models have been developed to predict mortality from COVID-19, but most are at high risk of bias.6–8 Many were developed in small or nonrepresentative patient samples; enrolled patients from early in the pandemic, before evidence-based treatments had been identified; included palliative patients (i.e., those who do not want resuscitative measures, such as intubation, intensive care unit care or cardiopulmonary resuscitation); censored outcomes; and had moderate predictive performance.6 The ISARIC 4C Mortality Score is the strongest model developed.9 However, it includes palliative patients, which limits its utility in stratifying risk among nonpalliative patients with COVID-19. In addition, the ISARIC 4C Mortality Score was developed based on data from early in the pandemic. Most other published rules involve the use of imaging or laboratory tests,9–14 which precludes their use as a first-line triage tool in the emergency department, where decisions on the appropriateness of intubation and mechanical ventilation may have to be made on arrival.
Our objective in the present study was to develop and validate a clinical score that accurately predicts mortality among nonpalliative patients with COVID-19, using clinical variables that are readily available on emergency department arrival.
Methods
Study design and setting
This study is a derivation and validation of a clinical decision rule based on observational data. Model development and reporting followed TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) standards.15 Our protocol is registered with clinicaltrials.gov (NCT04702945).
Study setting
The Canadian COVID-19 Emergency Department Rapid Response Network (CCEDRRN) is an ongoing multicentre pan-Canadian registry that enrols consecutive eligible patients with COVID-19 presenting to emergency departments in hospitals located in 8 of the 10 Canadian provinces, including the 4 most populous (only Prince Edward Island and Newfoundland and Labrador are not represented).16 The network conducts observational studies and develops clinical decision rules to inform emergency department care.17–19 The CCEDRRN patient engagement committee reviewed and provided input into the development of the research question, the choice of outcomes and the study protocol, and reviewed the submitted manuscript. Patient partners were involved in developing CCEDRRN’s website and knowledge translation tools to disseminate study results (https://www.ccedrrn.com/).
Participants
Participating sites needed to show at least 99% compliance in enrolling consecutive eligible patients for their data to be included in this study. At each site, we used trackers employing medical microbiology lists of all tested patients to identify all eligible patients. A national coordinator reviewed the lists of all sites on a weekly basis to ensure compliance. We included data from 46 of 50 CCEDRRN sites that met this criterion by the time of the data cut (Appendix 1, Table S1, available at www.cmajopen.ca/content/10/1/E90/suppl/DC1). The characteristics, resource use and outcomes of patients with COVID-19 included in the registry are posted as a preprint.20
We included patients with confirmed COVID-19 who presented to the emergency department of a participating site between Mar. 1, 2020, and Jan. 31, 2021. We defined confirmed COVID-19 as disease in patients who presented with ongoing COVID-19 symptoms and a positive result of a nucleic acid amplification test (NAAT) for SARS-CoV-2 obtained within 14 days before their arrival in the emergency department or after their arrival, which reflects the maximal incubation period.21 This allowed us to capture patients who were diagnosed in the community and subsequently presented to the emergency department, and those with early false-negative test results that became positive. We also included patients who presented with COVID-19 symptoms and were diagnosed with “confirmed COVID-19” to capture patients who were transferred to a CCEDRRN hospital and whose NAAT at the sending site could not be confirmed, and patients who were presumed by treating clinicians to have COVID-19 despite persistently negative NAAT results.
We excluded patients less than 18 years of age and those whose goals of care precluded invasive mechanical ventilation in order to not bias the study with self-fulfilling prophecy bias,22 whereby the prognostic model predicts the outcome that occurred as a result of a decision to withhold life-sustaining measures.22–24 We also excluded patients transferred to a hospital outside of CCEDRRN, as we would have been unable to ascertain their outcomes. We followed patients for 30 days if they were discharged from the emergency department or until hospital discharge if their stay lasted longer than 30 days.
Data sources
Trained research assistants abstracted data from electronic and paper-based medical records into a central, Web-based REDCap database. They captured patient demographic characteristics, vital signs, symptoms, comorbidities, COVID-19 exposure risk, diagnostic test results and outcomes. The research assistants recorded the first set of vital signs and symptoms documented in the emergency department record. The lowest oxygen saturation level was the lowest level recorded in the emergency department or by paramedics (some sites transcribed the vital signs obtained in the ambulance into their triage forms) with the patient receiving oxygen. We abstracted code status from the patients’ medical records. If a patient’s code status was not documented, we assumed that the status was full code.
We evaluated the interrater agreement of key predictor variables by comparing data collected retrospectively with prospective data.16 We developed the clinical prediction score after all chart abstraction was complete; research assistants were thus unaware of which clinical variables would be candidate predictor variables.
Outcome
The primary outcome was all-cause emergency department and in-hospital mortality. All patients had complete follow-up data at the time of the data cut. We categorized patients who were discharged from hospital as alive according to their latest hospital admission.
Predictor variables
All candidate predictor variables were recorded in the emergency department record. We selected candidate predictor variables based on a literature review of risk factors for COVID-19 mortality, review of other COVID-19 risk-prediction models, the availability of candidate predictor variables on patient arrival and the clinical knowledge of the investigator team.9,10,12,13,25–29 The candidate predictor variables included age, sex, pregnancy, type of residence, mode of arrival at the emergency department, comorbidities, symptoms, heart rate on arrival, systolic blood pressure, oxygen saturation level, respiratory rate, Glasgow Coma Scale score, oxygen delivery in the emergency department, lowest oxygen saturation level, physician or nurse impression of respiratory distress, and use of alcohol, tobacco, vaping or illicit substances (Appendix 1, Table S2).
Statistical analysis
Sample size
Assuming an event rate of less than 10%, shrinkage of 0.9 and a conservative Cox–Snell r2 of 0.1, we estimated that 8.5 events per degree of freedom were required for reliable prediction modelling in the derivation cohort.30 The 42 candidate predictor variables had 49 degrees of freedom, which indicated that 417 events were required. In the derivation cohort, there were 6758 patients who made 7420 emergency department visits. The derivation cohort had 471 events, which exceeded the number of events required.
Model development and validation
We randomly assigned participating sites to derivation or validation, with the goal of assigning 75% of eligible patients and outcome events to derivation, and the remaining to validation. We examined candidate predictors for collinearity and missing and extreme values in the derivation cohort. A few variables had missing values (systolic blood pressure had the most missing, at 4.7%) (Appendix 1, Table S2). We used 5 multiple imputations for predictors if missing categoric data could not reasonably be assumed to be absent (e.g., missing documentation of illicit substance use was classified as no substance use) and used these imputed data sets to develop the final model. The initial logistic regression model considered all candidate predictors, with continuous predictors fit with restricted cubic splines with 3 knots.
We assessed the strengths of associations between predictor variables and the outcome using an analysis of variance plot to inform the degrees of freedom to allocate to each predictor. No additional knots were allocated to continuous predictors. We used a fast step-down procedure to reduce the model to key predictors (fastbw function in the R rms package [R Foundation for Statistical Computing]). We conducted an internal bootstrap validation with 1000 bootstrap samples to provide an optimism-corrected area under the receiver-operating characteristic (AUC). We conducted this internal validation separately for each imputed data set and combined the imputation-specific AUCs using Rubin’s rules on the logit scale.31,32
We produced a nomogram from the reduced model to assign points. To enable easy clinical use, we categorized age into decades (as the nomogram points were sensitive to age), and cut points of 20 and 30 for respiratory rate on arrival based on the relation between its spline function and outcome. The resulting score ranged from −1 to 17. We calculated sensitivity and specificity at various point thresholds, along with the score’s discrimination and calibration, using one of the imputed data sets.
We validated the model in a cohort of geographically distinct sites that were not part of derivation and used a single imputation for the few missing respiratory rates (4%). We assessed outcomes independently for emergency department visits, irrespective of potential subsequent visits leading to death. We performed analyses in R using the rms and pROC packages, and used the pmsampsize package for sample size determination. To ensure patient privacy, a cell size restriction policy prohibited reporting counts of less than 5.
Validation of previously published models
We used our study cohort to externally validate other risk-prediction tools: the SEIMC score,11 the 4C Mortality Score10 and the VACO Index.33 We chose these 3 because they performed well in validation yet had critical methodologic differences that limit their translation to emergency department patients with suspected COVID-19: their cohorts included palliative patients, which may have introduced self-fulfilling prophecy bias,22 and they were from early in the pandemic, when COVID-19 testing was restricted to those with severe disease, which may have introduced spectrum bias. To facilitate comparison with our rule, we calculated their AUCs in our cohort using patients with complete data on as many predictors as possible. We compared AUCs using paired DeLong tests if the complete data for each score had the same patients and unpaired tests if patients with complete data differed by score.34
Ethics approval
This study was approved by the research ethics boards of all participating institutions with a waiver of informed consent for enrolment.
Results
We assessed 9704 consecutive patients with COVID-19 who made 10 670 emergency department visits between Mar. 1, 2020, and Jan. 31, 2021 (Figure 1). We excluded 943 patients who met 1 or more exclusion criteria, and included 8761 patients who made 9605 visits in our analyses. The follow-up time was 30 days for discharged patients and 30–229 days for admitted patients. Of the 8761 patients, 618 (7.0%) died in the emergency department or hospital. The characteristics and outcomes of patients in the derivation (n = 6758) and validation (n = 2054) cohorts are shown in Table 1.
Figure 1:
Flow diagram showing included and excluded emergency department (ED) visits. *Numbers total more than 8761 because 51 patients made visits to multiple EDs, some of which were derivation EDs and some of which were validation EDs.
Table 1:
Characteristics and outcomes of nonpalliative patients with COVID-19 presenting to emergency departments in the derivation and validation cohorts
| Characteristic/outcome | Cohort; no. (%) of visits* | |
|---|---|---|
| Derivation n = 7420 |
Validation n = 2185 |
|
| Age, mean ± SD, yr | 54.7 ± 19.8 | 53.6 ± 19.9 |
| Female sex | 3544 (47.8) | 1149 (52.6) |
| Province† | ||
| Quebec | 2992 (40.3) | 369 (16.9) |
| British Columbia | 1839 (24.8) | 291 (13.3) |
| Alberta | 1718 (23.2) | 775 (35.5) |
| Ontario | 732 (9.9) | 379 (17.4) |
| Saskatchewan | 75 (1.0) | 329 (15.1) |
| Nova Scotia | 64 (0.7) | 33 (1.51) |
| New Brunswick | < 5 | 9 (0.41) |
| Arrival from | ||
| Home | 6639 (89.5) | 2005 (91.8) |
| Institution | 631 (8.5) | 120 (5.5) |
| No fixed address | 108 (1.5) | 60 (2.8) |
| Missing | 42 (0.6) | 0 (0.0) |
| Arrival mode | ||
| Self | 4432 (59.7) | 1236 (56.6) |
| Ambulance or police | 2987 (40.3) | 948 (43.4) |
| Missing | 1 (0.0) | 1 (0.0) |
| Infection risk | ||
| Household or caregiver contact | 1064 (14.3) | 253 (11.6) |
| Institutional exposure (e.g., long-term care, prison) | 828 (11.2) | 152 (7.0) |
| Health care worker | 385 (5.2) | 107 (4.9) |
| Travel from country with known cases within 14 d | 296 (4.0) | 78 (3.6) |
| Unknown/not documented | 4847 (65.3) | 1595 (73.0) |
| Arrival heart rate, mean ± SD, beats/min | 93.4 ± 18.6 | 92.6 ± 18.1 |
| Arrival systolic blood pressure, mean ± SD, mm Hg | 131.0 ± 21.1 | 130.9 ± 20.4 |
| Arrival diastolic blood pressure, mean ± SD, mm Hg | 78.2 ± 13.0 | 78.9 ± 13.0 |
| Arrival respiratory rate, mean ± SD, breaths/min | 21.0 ± 6.2 | 20.6 ± 5.5 |
| Arrival temperature, mean ± SD, °C | 37.2 ± 0.9 | 37.0 ± 0.9 |
| Presence of respiratory distress | 1567 (21.1) | 446 (20.4) |
| 10 most common COVID-19 symptoms | ||
| Cough | 3938 (53.1) | 1247 (57.1) |
| Shortness of breath (dyspnea) | 3616 (48.7) | 1096 (50.2) |
| Fever | 3216 (43.3) | 832 (38.1) |
| Fatigue/malaise | 1992 (26.9) | 697 (31.9) |
| Chest pain (includes discomfort or tightness) | 1606 (21.6) | 497 (22.8) |
| Headache | 1263 (17.0) | 415 (19.0) |
| Nausea/vomiting | 1201 (16.2) | 468 (21.4) |
| Chills | 1194 (16.1) | 339 (15.5) |
| Muscle aches (myalgia) | 1136 (15.3) | 384 (17.6) |
| Diarrhea | 1016 (13.7) | 373 (17.1) |
| 10 most common comorbidities | ||
| Hypertension | 2179 (29.4) | 625 (28.6) |
| Diabetes | 1247 (16.8) | 319 (14.6) |
| Dyslipidemia | 1161 (15.7) | 252 (11.5) |
| Mental health diagnosis | 670 (9.0) | 270 (12.4) |
| Hypothyroidism | 547 (7.4) | 141 (6.5) |
| Asthma | 535 (7.2) | 186 (8.5) |
| Coronary artery disease | 487 (6.6) | 114 (5.2) |
| Chronic neurologic disorder (not dementia) | 423 (5.7) | 101 (4.6) |
| Chronic lung disease (not asthma or pulmonary fibrosis) | 379 (5.11) | 132 (6.0) |
| Dementia | 362 (4.9) | 86 (3.9) |
| Smoking or vaping | ||
| Current | 573 (7.7) | 190 (8.7) |
| Past or never | 6847 (92.3) | 1995 (91.3) |
| Illicit substance use | ||
| Current | 130 (1.8) | 83 (3.8) |
| Past or never | 7290 (98.3) | 2102 (96.2) |
| Oxygen required in emergency department | 1341 (18.1) | 302 (13.8) |
| Emergency department disposition | ||
| Discharged | 4488 (60.5) | 1320 (60.4) |
| Admitted | 2858 (38.5) | 848 (38.8) |
| Left against medical advice | 18 (0.2) | 6 (0.3) |
| Died | 22 (0.3) | 6 (0.2) |
| Missing | 34 (0.5) | 5 (0.2) |
| Died in emergency department or in hospital | 471 (6.4) | 147 (6.7) |
Note: SD = standard deviation.
Except where noted otherwise.
No participating site in Manitoba showed at least 99% compliance in enrolling consecutive eligible patients by the time of the data cut.
The patients in the derivation cohort made 7420 emergency department visits to 32 different sites (Appendix 1, Table S1), and those in the validation cohort made 2185 emergency department visits to 14 different sites. In the derivation cohort, 2705 visits (36.5%) occurred early during the pandemic (between Mar. 1 and June 30, 2020), and 4715 (63.5%) occurred between July 1, 2020, and Jan. 31, 2021. The corresponding values for the validation cohort were 618 (28.3%) and 1567 (71.7%).
In derivation, the step-down procedure produced a final model with 8 variables (Table 2). The derived model had an optimism-corrected AUC of 0.92. The resulting risk score ranged from −1 to 17. When the CCEDRRN COVID-19 Mortality Score was used as the only predictor in the logistic regression model, the intercept was −9.049, and the slope for the risk score was 0.652 (Table 3). The derivation cohort was well distributed across the score range and had excellent calibration (calibration intercept of 0 and slope of 1) and discrimination, with an AUC of 0.92 (95% confidence interval [CI] 0.90–0.93) (Appendix 1, Figure S1).
Table 2:
Adjusted associations between predictor variables and death, and points for the CCEDRRN COVID-19 Mortality Score
| Variable | Estimate (SE) | OR (95% CI) | Points |
|---|---|---|---|
| Age, yr* | 2.85 (0.00–41.00) | 17.30 (7.75–38.60) | |
| < 40 | 0 | ||
| 40–49 | 4 | ||
| 50–59 | 5 | ||
| 60–69 | 6 | ||
| 70–79 | 7 | ||
| ≥ 80 | 8 | ||
| Sex | |||
| Male | Ref | Ref | 1 |
| Female | −0.61 (0.12) | 0.54 (0.43–0.69) | 0 |
| Arrival from | |||
| Home or community | Ref | Ref | 0 |
| No fixed address | 0.17 (0.63) | 1.19 (0.35–4.09) | 0 |
| Institution | 0.59 (0.14) | 1.80 (1.38–2.35) | 1 |
| Arrival mode | |||
| Self | Ref | Ref | 0 |
| Ambulance or police | 0.63 (0.15) | 1.89 (1.41–2.52) | 1 |
| Chest pain | −0.80 (0.24) | 0.45 (0.28–0.72) | −1 |
| Moderate/severe liver disease | 1.94 (0.50) | 6.95 (2.61–18.50) | 2 |
| Arrival respiratory rate, breaths/min† | 0.29 (0.10) | 1.34 (1.09–1.63) | |
| < 20 | 0 | ||
| 20–29 | 2 | ||
| ≥ 30 | 3 | ||
| Mode and level of oxygen in emergency department | |||
| No oxygen | Ref | Ref | 0 |
| Nasal prongs, < 6 L/min | 0.70 (0.14) | 2.00 (1.53–2.62) | 1 |
| Face mask, simple rebreather or nasal prongs, ≥ 6 L/min | 1.94 (0.19) | 6.98 (4.79–10.16) | 2 |
| BiPAP/CPAP/HFNO | 2.56 (0.27) | 12.98 (7.62–22.08) | 3 |
| Intubation | 2.53 (0.29) | 12.50 (7.11–21.98) | 3 |
Note: BiPAP = bilevel positive airway pressure, CCEDRRN = Canadian COVID-19 Emergency Department Rapid Response Network, CI = confidence interval, CPAP = continuous positive airway pressure, HFNO = high-flow nasal oxygen, OR = odds ratio, Ref = reference category, SE = standard error.
Odds ratio was calculated for the upper quartile (70 yr) versus the lower quartile (39 yr).
Odds ratio was calculated for the upper quartile (22 breaths/min) versus the lower quartile (18 breaths/min).
Table 3:
Performance of the CCEDRRN COVID-19 Mortality Score to rule out and rule in in-hospital mortality at various cut-off values in the combined derivation and validation cohorts, and the predicted mortality risk for each value based on the CCEDRRN COVID-19 Mortality Score*
| Score | No. (%) of visits n = 9605 |
Sensitivity (95% CI), % | Specificity (95% CI), % | Negative LR | Positive LR | PPV, % | NPV, % | Mortality rate, % | Score | Predicted risk with score |
|---|---|---|---|---|---|---|---|---|---|---|
| Rule out | ||||||||||
| ≤ −1 | 178 (1.8) | 100.0 (99.4–100.0) | 2.0 (1.7–2.3) | 0.0 | 1.0 | 6.6 | 100 | 0.0 | −1 | 0.0001 |
| ≤ 0 | 825 (8.6) | 100.0 (99.4–100.0) | 9.2 (8.6–9.8) | 0.0 | 1.1 | 7.0 | 100 | 0.0 | 0 | 0.0001 |
| ≤ 1 | 1499 (15.6) | 100.0 (99.4–100.0) | 16.7 (15.9–17.5) | 0.0 | 1.2 | 7.6 | 100 | 0.0 | 1 | 0.0002 |
| ≤ 2 | 1926 (20.0) | 100.0 (99.4–100.0) | 21.4 (20.6–22.3) | 0.0 | 1.3 | 8.0 | 100 | 0.0 | 2 | 0.0004 |
| ≤ 3 | 2420 (25.2) | 100.0 (99.4–100.0) | 26.9 (26.0–27.9) | 0.0 | 1.4 | 8.6 | 100 | 0.0 | 3 | 0.0008 |
| ≤ 4 | 2910 (30.3) | 100.0 (99.4–100.0) | 32.4 (31.4–33.4) | 0.0 | 1.5 | 9.2 | 100 | 0.0 | 4 | 0.0016 |
| ≤ 5 | 3685 (38.4) | 99.8 (99.1–100.0) | 41.0 (40.0–42.0) | 0.0 | 1.7 | 10.4 | 100 | 0.0 | 5 | 0.0031 |
| ≤ 6 | 4563 (47.5) | 99.0 (97.9–99.6) | 50.7 (49.7–51.7) | 0.0 | 2.0 | 12.1 | 99.9 | 0.1 | 6 | 0.0058 |
| ≤ 7 | 5497 (57.2) | 97.4 (95.8–98.5) | 61.0 (60.0–62.0) | 0.0 | 2.5 | 14.7 | 99.7 | 0.3 | 7 | 0.0112 |
| ≤ 8 | 6410 (66.7) | 94.0 (91.8–95.7) | 70.9 (70.0–71.9) | 0.1 | 3.2 | 18.2 | 99.4 | 0.6 | 8 | 0.0212 |
| ≤ 9 | 7276 (75.8) | 88.5 (85.7–90.9) | 80.2 (79.3–81.0) | 0.1 | 4.5 | 23.5 | 99.0 | 1.0 | 9 | 0.0399 |
| ≤ 10 | 7974 (83.0) | 77.7 (74.2–80.9) | 87.2 (86.5–87.9) | 0.3 | 6.1 | 29.4 | 98.3 | 1.7 | 10 | 0.0739 |
| Rule in | ||||||||||
| ≥ 10 | 2329 (24.2) | 88.5 (85.7–90.9) | 80.2 (79.3–81.0) | 0.1 | 4.5 | 23.5 | 99.0 | 23.5 | – | – |
| ≥ 11 | 1631 (17.0) | 77.7 (74.2–80.9) | 87.2 (86.5–87.5) | 0.3 | 6.1 | 29.4 | 98.3 | 29.4 | 11 | 0.1327 |
| ≥ 12 | 1083 (11.3) | 65.0 (61.1–65.0) | 92.4 (91.9–93.0) | 0.4 | 8.6 | 37.1 | 97.5 | 37.1 | 12 | 0.2271 |
| ≥ 13 | 604 (6.3) | 46.3 (42.3–50.3) | 96.5 (96.1–96.8) | 0.6 | 13.1 | 47.4 | 96.3 | 47.4 | 13 | 0.3605 |
| ≥ 14 | 276 (2.9) | 28.2 (24.6–31.9) | 98.9 (98.6–99.1) | 0.7 | 24.8 | 63.0 | 95.2 | 63.0 | 14 | 0.5197 |
| ≥ 15 | 100 (1.0) | 13.1 (10.5–16.0) | 99.8 (99.7–99.9) | 0.9 | 62.0 | 81.0 | 94.4 | 81.0 | 15 | 0.6750 |
| ≥ 16 | 29 (0.3) | 4.4 (2.9–6.3) | 100.0 (99.9–100.0) | 1.0 | 196.3 | 93.1 | 93.8 | 93.1 | 16 | 0.7995 |
| ≥ 17 | 5 (0.0) | 0.8 (0.3–1.9) | 100.0 (100.0) | 1.0 | – | 100.0 | 93.6 | 100.0 | 17 | 0.8844 |
Note: CCEDRRN = Canadian COVID-19 Emergency Department Rapid Response Network, CI = confidence interval, LR = likelihood ratio, NPV = negative predictive value, PPV = positive predictive value.
Predicted risk = exp (−9.048 + 0.652 * CCMS)/(1 + exp [−9.049 + 0.652 * CCMS]), where CCMS = CCEDRRN COVID-19 Mortality Score.
The CCEDRRN COVID-19 Mortality Score had similar performance in validation. The validation cohort was also distributed across the score ranges, had excellent calibration (calibration intercept of 0 and slope of 1) and discrimination (AUC 0.92, 95% CI 0.90–0.93) (Figure 2).
Figure 2:
Distribution and performance of the CCEDRRN COVID-19 Mortality Score in the validation cohort (left panel) and combined derivation and validation cohorts (right panel): (A) distribution of the score, (B) observed in-hospital mortality rates across the range of the score, (C) predicted versus observed risk of in-hospital death (dashed line represents line of no difference between predicted and observed risk) and (D) receiver-operating characteristic curve with area under the curve and associated 95% confidence interval.
The score had excellent performance across a range of thresholds to rule in and rule out in-hospital mortality (Table 3; Appendix 1, Table S4). These results suggest that a score of 6 or less would categorize patients as being at low risk for in-hospital death, with a negative predictive value of 99.9%. Patients in the low-risk group had an in-hospital mortality rate of 0.1%. For scores of 15 or higher, the observed in-hospital mortality rate was 81.0%, and the CCEDRRN COVID-19 Mortality Score would categorize patients as being at high risk for in-hospital death, with a specificity of 99.8%.
Validation of previously published models
The 3 risk scores performed well for the patients with data available, yielding AUCs of nearly 0.88, less than that of the CCEDRRN COVID-19 Mortality Score. Unadjusted for multiple testing and with the use of the validation cohort, the AUC for the CCEDRRN COVID-19 Mortality Score (n = 2185) was higher than that for the SEIMC score (p = 0.04, n = 1620) and the VACO Index (p = 0.002, n = 2185); there was no evidence of a difference for the 4C Mortality Score (p = 0.07, n = 610) (Appendix 1, Figure S2, Table S5).
Interpretation
We derived and validated a parsimonious, simple score to predict in-hospital mortality among nonpalliative patients with COVID-19 presenting to emergency departments: the CCEDRRN COVID-19 Mortality Score. We found that 8 readily available clinical variables that can be ascertained at the bedside without any diagnostic testing predicted mortality accurately. The score had excellent calibration and discrimination in a geographically distinct cohort of patients who presented to other sites. It can be used as a highly sensitive score to rule out in-hospital mortality in patients at low risk with a score of 6 or less. In patients with a score of 15 or higher, the risk of in-hospital death is very high despite maximum medical therapy.
The CCEDRRN COVID-19 Mortality Score has strengths compared to prior models.6 We developed the model using simple variables that are readily available at the bedside. In contrast to prior models, we excluded patients with palliative goals of care, for whom invasive mechanical ventilation was not offered, to ensure our model did not predict risk of death among patients who were expected to die or were ineligible for the highest level of critical care.9–11,13,14,27,28,33,35,36 This avoids the potential for self-fulfilling prophecy bias.22 Prior models were derived or validated early in the pandemic, when COVID-19 testing was restricted to those with severe disease, and they did not include consecutive eligible patients; both these factors may have resulted in selection bias. The mortality rate in prior studies ranged from 13% to 30%,9,11,12,14,27,28,36 in contrast to 7% in our study. Our study was also able to capture patients who were readmitted to CCEDRRN sites and subsequently died in hospital.
Critically ill patients with COVID-19 typically require aggressive medical management in the emergency department shortly after their arrival. Being able to accurately predict mortality risk before endotracheal intubation can offer the opportunity to have discussions about patients’ goals of care and facilitate early high-quality end-of-life care for those most likely to die. Accurate mortality prediction may be essential when surging cases threaten to overwhelm critical care resources. In those rare situations, the CCEDRRN COVID-19 Mortality Score can guide allocation of scarce resources.
Our rule’s predictive ability depends mostly on age and respiratory status. The only comorbidity retained in the final model was moderate to severe liver disease. Two other rules also identified liver disease as a risk factor for death, perhaps owing to the potential for virus-induced liver inflammation. 14,33,37 Other prognostic decision rules have used similar analytic approaches27,28,33 but had lower predictive performance, with C-statistics ranging from 0.80 to 0.82, and were based on patients from early in the pandemic. Other rules incorporate measures of hypoxemia or respiratory support, which is indicative of their strong predictive power.9–13,27,28,33,35,36 Goodacre and colleagues28 identified physical performance status as a risk factor for increased mortality. In our data set, this was most closely reflected by arrival from long-term care, which indicated patients with lower performance status.
Limitations
Canada is a culturally diverse country that offers its citizens publicly funded health coverage for necessary physician and hospital care. The CCEDRRN COVID-19 Mortality Score needs to be validated externally in other health care systems. Our model could be validated externally for use in remote areas without access to laboratory testing or imaging, and in low-income countries. It predicts in-hospital mortality and may have missed deaths that occurred at home or deaths among discharged patients who returned to non-CCEDRRN hospitals. We captured readmissions to CCEDRRN sites, although it is unlikely that they had an influence on our model given their small numbers.20
Our model was based on patients with confirmed COVID-19. Although many presented to the emergency department with NAAT-confirmed COVID-19, validation in a cohort of patients with suspected COVID-19 is needed. We excluded patients if they were transferred to a non-CCEDRRN institution, as we would have been unable to ascertain their outcomes. However, the numbers of such patients were small, and their exclusion likely did not have a significant effect on the model. Our results apply to unvaccinated patients. The performance of risk tools also needs to be evaluated in patients who are fully or partially vaccinated and may have a different risk of dying from COVID-19 than unvaccinated patients.
We were unable to link our data set with vaccine registry data. However, by Jan. 31, 2021, only 0.3% of Canadians, most of whom were health care workers or residents of long-term care, had been fully vaccinated.38 Most residents of long-term care were excluded from our study owing to palliative code status. Thus, our results are applicable to unvaccinated patients. The performance of the CCEDRRN COVID-19 Mortality Score needs to be evaluated in a cohort of vaccinated patients, who may have a different risk of dying from COVID-19 than unvaccinated patients. The score also needs to be validated in cohorts infected with variants of concern and in a more recent data set, given newer treatment regimens that may affect mortality risk.
Conclusion
The CCEDRRN COVID-19 Mortality Score is a simple clinical risk score that can be applied at the bedside in the emergency department to predict unvaccinated patients’ mortality risk. There have been changes in treatment and emergence of new variants that may limit its utility. This tool can be used to inform goals-of-care decisions.
Supplementary Material
Acknowledgements
The authors gratefully acknowledge the assistance of Ms. Amber Cragg in the preparation of the manuscript. They thank the University of British Columbia clinical coordinating centre staff, the University of British Columbia legal, ethics, privacy and contract staff, and the research staff at each of the participating institutions in the network outlined in Appendix 2 (available at www.cmajopen.ca/content/10/1/E90/suppl/DC1). The network would not exist today without the dedication of these professionals. The authors thank all of their patient partners who shared their lived experiences and perspectives to ensure that the knowledge cocreated addresses the concerns of patients and the public. Creating the largest network of collaboration across Canadian emergency departments would not have been feasible without the tireless efforts of emergency department chiefs and research coordinators and research assistants at participating sites. Finally, the authors extend their most humble and sincere gratitude to all of their colleagues in medicine, nursing and the allied health care professions who have been on the front lines of the COVID-19 pandemic from day one, staffing ambulances, emergency departments, intensive care units and hospitals, and bravely facing the risks of COVID-19 to look after their fellow citizens and one another. The authors dedicate this network to them.
Footnotes
Competing interests: None declared.
This article has been peer reviewed.
Contributors: Laurie Morrison and Jeffrey Perry are co–senior authors. Corinne Hohl, Rhonda Rosychuk, Jeffrey Perry, Laurie Morrison, Patrick Archambault and Steven Brooks conceived and designed the study, with input on the design and selection of variables from all other authors. Corinne Hohl, Patrick Archambault, Steven Brooks, Éric Mercier, Michelle Welsford, Jake Hayward, Tomislav Jelic, Vi Ho and Gregory Clark managed data collection, along with other members of the Canadian COVID-19 Emergency Department Rapid Response Network. Corinne Hohl, Rhonda Rosychuk, Fiona O’Sullivan and Jeffrey Perry developed the analytic plan, and Fiona O’Sullivan analyzed the data, with assistance from Rhonda Rosychuk, Corinne Hohl and Jeffrey Perry. Corinne Hohl and Patrick Archambault drafted the manuscript. All of the authors contributed to data interpretation, revised the manuscript for important intellectual content, approved the final version to be published and agreed to be accountable for all aspects of the work.
Funding: The Canadian COVID-19 Emergency Department Rapid Response Network is funded by the Canadian Institutes of Health Research (grant 447679), the BC Academic Health Science Network Society, BioTalent Canada, Genome BC (grants COV024 and VAC007), the Ontario Ministry of Colleges and Universities (grant C-655-2129), the Saskatchewan Health Research Foundation (grant 5357) and the Fondation du CHU de Québec (grant 4007).
Data sharing: The Canadian COVID-19 Emergency Department Rapid Response Network endorses the guidance put forth by the World Health Organization to enable data sharing to optimize learning. The network accepts applications for access to data by external investigators, prioritizing data requests by network members.
Disclaimer: The funders had no role in the collection, analysis or interpretation of the data, the writing of the manuscript or the decision to submit for publication.
Supplemental information: For reviewer comments and the original submission of this manuscript, please see www.cmajopen.ca/content/10/1/E90/suppl/DC1.
References
- 1.Tan E, Song J, Deane AM, et al. Global impact of coronavirus disease 2019 infection requiring admission to the ICU: a systematic review and meta-analysis. Chest. 2021;159:524–36. doi: 10.1016/j.chest.2020.10.014. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Rosenbaum L. Facing COVID-19 in Italy: ethics, logistics, and therapeutics on the epidemic’s front line. N Engl J Med. 2020;382:1873–5. doi: 10.1056/NEJMp2005492. [DOI] [PubMed] [Google Scholar]
- 3.Satomi E, Souza PMR, Thomé BDC, et al. Fair allocation of scarce medical resources during COVID-19 pandemic: ethical considerations. Einstein (Sao Paulo) 2020;18:eAE5775. doi: 10.31744/einstein_journal/2020AE5775. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Arya A, Buchman S, Gagnon B, et al. Pandemic palliative care: beyond ventilators and saving lives. CMAJ. 2020;192:E400–4. doi: 10.1503/cmaj.200465. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Maves RC, Downar J, Dichter JR, et al. ACCP Task Force for Mass Critical Care. Triage of scarce critical care resources in COVID-19 an implementation guide for regional allocation: an expert panel report of the Task Force for Mass Critical Care and the American College of Chest Physicians. Chest. 2020;158:212–25. doi: 10.1016/j.chest.2020.03.063. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Wynants L, Van Calster B, Collins GS, et al. Prediction models for diagnosis and prognosis of COVID-19 infection: systematic review and critical appraisal. BMJ. 2020;369:m1328. doi: 10.1136/bmj.m1328. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Update to living systematic review on prediction models for diagnosis and prognosis of COVID-19. BMJ. 2021;372:n236. doi: 10.1136/bmj.n236. [DOI] [PubMed] [Google Scholar]
- 8.Gupta RK, Marks M, Samuels THA, et al. UCLH COVID-19 Reporting Group. Systematic evaluation and external validation of 22 prognostic models among hospitalised adults with COVID-19: an observational cohort study. Eur Respir J. 2020;56:2003498. doi: 10.1183/13993003.03498-2020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Knight SR, Ho A, Pius R, et al. ISARIC4C investigators. Risk stratification of patients admitted to hospital with COVID-19 using the ISARIC WHO Clinical Characterisation Protocol: development and validation of the 4C Mortality Score. BMJ. 2020;370:m3339. doi: 10.1136/bmj.m3339. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Cho SY, Park SS, Song MK, et al. Prognosis score system to predict survival for COVID-19 cases: a Korean nationwide cohort study. J Med Internet Res. 2021;23:e26257. doi: 10.2196/26257. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Berenguer J, Borobia AM, Ryan P, et al. COVID-19@Spain and COVID@ HULP Study Groups. Development and validation of a prediction model for 30-day mortality in hospitalised patients with COVID-19: the COVID-19 SEIMC score. Thorax. 2021;76:920–9. doi: 10.1136/thoraxjnl-2020-216001. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Nicholson CJ, Wooster L, Sigurslid HH, et al. Estimating risk of mechanical ventilation and in-hospital mortality among adult COVID-19 patients admitted to Mass General Brigham: the VICE and DICE scores. EClinicalMedicine. 2021;33:100765. doi: 10.1016/j.eclinm.2021.100765. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Liang W, Liang H, Ou L, et al. Development and validation of a clinical risk score to predict the occurrence of critical illness in hospitalized patients with COVID-19. JAMA Intern Med. 2020;180:1081–9. doi: 10.1001/jamainternmed.2020.2033. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Magro B, Zuccaro V, Novelli L, et al. Predicting in-hospital mortality from coronavirus disease 2019: a simple validated app for clinical use. PLoS One. 2021;16:e0245281. doi: 10.1371/journal.pone.0245281. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Moons KGM, Altman DG, Reitsma JB, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162:W1–73. doi: 10.7326/M14-0698. [DOI] [PubMed] [Google Scholar]
- 16.Hohl CM, Rosychuk RJ, McRae AD, et al. Canadian COVID-19 Emergency Department Rapid Response Network investigators and for the Network of Canadian Emergency Researchers and the Canadian Critical Care Trials Group. Development of the Canadian COVID-19 Emergency Department Rapid Response Network population-based registry: a methodology study. CMAJ Open. 2021;9:E261–70. doi: 10.9778/cmajo.20200290. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Hohl CM, Rosychuk RJ, Archambault PM, et al. Derivation and validation of a clinical score to predict death among non-palliative COVID-19 patients presenting to emergency departments: the CCEDRRN COVID Mortality Score. medRxiv. 2021 July 31; doi: 10.1101/2021.07.28.21261283. [DOI] [Google Scholar]
- 18.McRae AD, Hohl CM, Rosychuk RJ, et al. Canadian COVID-19 Emergency Department Rapid Response Network (CCEDRRN) investigators for the Network of Canadian Emergency Researchers and the Canadian Critical Care Trials Group. Development and validation of a clinical risk score to predict SARS-CoV-2 infection in emergency department patients: the CCEDRRN COVID-19 Infection Score (CCIS) BMJ Open. 2021;11:e055832. doi: 10.1136/bmjopen-2021-055832. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Davis P, Rosychuk RJ, Hau JP, et al. Diagnostic yield of screening for SARS-CoV-2 among patients admitted for alternate diagnoses. medRxiv. 2021 Sept 27; doi: 10.1101/2021.09.23.21264036. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Hohl CM, Rosychuk RJ, Hau JP, et al. Treatments, resource utilization, and outcomes of COVID-19 patients presenting to emergency departments across pandemic waves: an observational study by the Canadian COVID-19 Emergency Department Rapid Response Network (CCEDRRN) medRxiv. 2021 Aug 1; doi: 10.1101/2021.07.30.21261288. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Dugdale CM, Anahtar MN, Chiosi JJ, et al. Clinical, laboratory, and radiologic characteristics of patients with initial false-negative severe acute respiratory syndrome coronavirus 2 nucleic acid amplification test results. Open Forum Infect Dis. 2020;8:ofaa559. doi: 10.1093/ofid/ofaa559. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Turgeon AF, Lauzier F, Simard JF, et al. Canadian Critical Care Trials Group. Mortality associated with withdrawal of life-sustaining therapy for patients with severe traumatic brain injury: a Canadian multicentre cohort study. CMAJ. 2011;183:1581–8. doi: 10.1503/cmaj.101786. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Creutzfeldt CJ, Becker KJ, Weinstein JR, et al. Do-not-attempt-resuscitation orders and prognostic models for intraparenchymal hemorrhage. Crit Care Med. 2011;39:158–62. doi: 10.1097/CCM.0b013e3181fb7b49. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Wilkinson D. The self-fulfilling prophecy in intensive care. Theor Med Bioeth. 2009;30:401–10. doi: 10.1007/s11017-009-9120-6. [DOI] [PubMed] [Google Scholar]
- 25.Knight SR, Ho A, Pius R, et al. ISARIC4C investigators. Risk stratification of patients admitted to hospital with COVID-19 using the ISARIC WHO Clinical Characterisation Protocol: development and validation of the 4C Mortality Score. BMJ. 2020;370:m3339. doi: 10.1136/bmj.m3339. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Haimovich AD, Ravindra NG, Stoytchev S, et al. Development and validation of the quick COVID-19 Severity Index: a prognostic took for early clinical decompensation. Ann Emerg Med. 2020;76:442–53. doi: 10.1016/j.annemergmed.2020.07.022. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Chua F, Vancheeswaran R, Draper A, et al. Early prognostication of COVID-19 to guide hospitalisation versus outpatient monitoring using a point-of-test risk prediction score. Thorax. 2021;76:696–703. doi: 10.1136/thoraxjnl-2020-216425. [DOI] [PubMed] [Google Scholar]
- 28.Goodacre S, Thomas B, Sutton L, et al. Derivation and validation of a clinical severity score for acutely ill adults with suspected COVID-19: the PRIEST observational cohort study. PLoS One. 2021;16:e0245840. doi: 10.1371/journal.pone.0245840. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Magro B, Zuccaro V, Novelli L, et al. Predicting in-hospital mortality from coronavirus disease 2019: a simple validated app for clinical use. PLoS One. 2021;16:e0245281. doi: 10.1371/journal.pone.0245281. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Riley RD, Snell KI, Ensor J, et al. Minimum sample size for developing a multivariable prediction model: Part II — binary and time-to-event outcomes. Stat Med. 2019;38:1276–96. doi: 10.1002/sim.7992. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Marshall A, Altman DG, Holder RL, et al. Combining estimates of interest in prognostic modelling studies after multiple imputation: current practice and guidelines. BMC Med Res Methodol. 2009;9:57. doi: 10.1186/1471-2288-9-57. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Wood AM, Royston P, White IR. The estimation and use of predictions for the assessment of model performance using large samples with multiply imputed data. Biom J. 2015;57:614–32. doi: 10.1002/bimj.201400004. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.King JT, Jr, Yoon JS, Rentsch CT, et al. Development and validation of a 30-day mortality index based on pre-existing medical administrative data from 13,323 COVID-19 patients: the Veterans Health Administration COVID-19 (VACO) Index. PLoS One. 2020;15:e0241825. doi: 10.1371/journal.pone.0241825. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44:837–45. [PubMed] [Google Scholar]
- 35.Haimovich AD, Ravindra NG, Stoytchev S, et al. Development and validation of the quick COVID-19 Severity Index: a prognostic tool for early clinical decompensation. Ann Emerg Med. 2020;76:442–53. doi: 10.1016/j.annemergmed.2020.07.022. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Gupta RK, Harrison EM, Ho A, et al. ISARIC4C Investigators. Development and validation of the ISARIC 4C Deterioration model for adults hospitalised with COVID-19: a prospective cohort study. Lancet Respir Med. 2021;9:349–59. doi: 10.1016/S2213-2600(20)30559-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Demirtas CO, Keklikkiran C, Ergenc I, et al. Liver stiffness is associated with disease severity and worse clinical scenarios in coronavirus disease 2019: a prospective transient elastography study. Int J Clin Pract. 2021;75:e14363. doi: 10.1111/ijcp.14363. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.COVID-19 vaccination in Canada. Ottawa: Government of Canada; [accessed 2022 Jan. 24]. updated 2022 Jan. 21. Available: https://health-infobase.canada.ca/covid-19/vaccination-coverage/ [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.