Abstract
Objective
To risk‐stratify COVID‐19 patients being considered for discharge from the emergency department (ED).
Methods
We conducted an observational study to derive and validate a clinical decision rule to identify COVID‐19 patients at risk for hospital admission or death within 72 hours of ED discharge. We used data from 49 sites in the Canadian COVID‐19 Emergency Department Rapid Response Network (CCEDRRN) between March 1, 2020, and September 8, 2021. We randomly assigned hospitals to derivation or validation and prespecified clinical variables as candidate predictors. We used logistic regression to develop the score in a derivation cohort and examined its performance in predicting short‐term adverse outcomes in a validation cohort.
Results
Of 15,305 eligible patient visits, 535 (3.6%) experienced the outcome. The score included age, sex, pregnancy status, temperature, arrival mode, respiratory rate, and respiratory distress. The area under the curve was 0.70 (95% confidence interval [CI] 0.68–0.73) in derivation and 0.71 (95% CI 0.68–0.73) in combined derivation and validation cohorts. Among those with a score of 3 or less, the risk for the primary outcome was 1.9% or less, and the sensitivity of using 3 as a rule‐out score was 89.3% (95% CI 82.7–94.0). Among those with a score of ≥9, the risk for the primary outcome was as high as 12.2% and the specificity of using 9 as a rule‐in score was 95.6% (95% CI 94.9–96.2).
Conclusion
The CCEDRRN COVID discharge score can identify patients at risk of short‐term adverse outcomes after ED discharge with variables that are readily available on patient arrival.
Keywords: clinical decision instrument, clinical prediction score, coronavirus disease, COVID‐19, emergency medicine, prediction model
1. INTRODUCTION
1.1. Background
Most patients who present with COVID‐19 to the emergency department (ED) have no obvious need for admission and are discharged home. 1 , 2 , 3 , 4 Prior work has identified several clinical factors associated with progression to severe COVID‐19 disease. 5 , 6 , 7 , 8 , 9 , 10 The application of these findings to individual patients to support bedside decision‐making is not intuitive. Clinical decision rules (CDRs) involve the transformation of observed associations between various clinical features and outcomes into a “rule” or score that can be used to risk‐stratify patients and directly support bedside clinical decision‐making with individualized risk estimates. 11 Although COVID‐19 CDRs have been developed to support safe disposition decisions for COVID‐19 patients in the ED, 12 , 13 , 14 , 15 , 16 many have been developed using non‐representative patient cohorts, whereas some require laboratory or radiology results that are not available for most patients being considered for discharge.
1.2. Importance
Unfortunately, some patients discharged from the ED deteriorate soon afterwards. 17 This investigation, using data from a cohort of mostly unvaccinated ED patients, provides a CDR for use in unvaccinated patients to support safe disposition decisions. Our work provides insights into which clinical features are most predictive of short‐term risk after discharge from the ED, which may be used in clinical and research settings.
1.3. Goals of this investigation
The specific objectives of this study were to derive and validate a bedside CDR to predict the risk of hospital admission or death within 72 hours of ED discharge among COVID‐19 patients discharged from the ED.
2. METHODS
2.1. Design
This was a retrospective observational study using data collected in the Canadian COVID‐19 Emergency Department Rapid Response Network (CCEDRRN, pronounced “sedrin”). 18 Model development and reporting followed Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) standards. 19 The research ethics boards of all participating institutions approved this study with a waiver of informed consent for enrollment and collection of retrospective data from hospital charts. We registered CCEDRRN on clinicaltrials.gov (NCT04702945).
2.2. Setting
CCEDRRN is a pan‐Canadian collaborative research network and registry involving 49 EDs across 8 provinces. 18
2.3. Selection of subjects
We included adult patients (age 18 or older) with confirmed COVID‐19 who were discharged from the ED of a participating site between March 1, 2020, and September 8, 2021. We defined “confirmed COVID‐19” as patients with a positive nucleic acid amplification test for severe acute respiratory syndrome coronavirus‐2 (SARS‐CoV‐2) obtained within 14 days before or after their arrival in the ED. This allowed us to capture patients who were diagnosed in the community and subsequently presented to the ED and those with early false negative tests. We excluded patients from long‐term care facilities and those with a Do Not Resuscitate order.
2.4. Measurements
Research assistants abstracted data from electronic and paper‐based medical records into a central, web‐based Research Electronic Data Capture (REDCap) database (Vanderbilt University; Nashville, TN). Registry data quality measures included logic checks within the database for values out of range, audit of a sample of cases for each new research assistant, and routine monthly verification for outlying data.
We captured demographics, arrival vital signs, symptoms, comorbidities, COVID‐19 exposure risk, immunization status, diagnostic test results, and patient outcomes. Interrater agreement of key predictor variables in the registry database has been assessed and reported elsewhere. 18 The CDR was developed after all chart abstractions were complete; research assistants were thus unaware of which clinical variables would be candidate predictor variables. Telephone follow‐up was conducted to ascertain immunization status.
We coded all patients who presented to the ED in 2020 as unimmunized because the first COVID‐19 vaccination was licensed for use by Health Canada on December 14, 2020. For patients presenting in 2021, we ascertained immunization status through medical record review or telephone follow‐up of consenting patients. If the two were discrepant, we coded the patient's immunization status according to their telephone follow‐up data. If immunization status was not documented or ascertained from telephone follow up for a particular patient, we coded their immunization status according to a rule we developed based on their status as a healthcare worker, age, province of residence, and the date of the ED visit. Because vaccine eligibility was based on these factors and delivery was tightly controlled by public health authorities, we were able to use this rule with confidence to code patients as unimmunized when they were not eligible at the time of their ED visit. When patients with undocumented immunization status were deemed to be eligible for vaccination by this method, we excluded them from the analysis because we could not be certain of their actual immunization status.
We defined patients as “fully vaccinated” 7 days or more after their second dose of a 2‐dose vaccination regimen or 14 days after a single dose regimen approved by the World Health Organization. 20 We defined patients as “partially vaccinated” if their index ED visit was more than 14 days after the first dose of a 2‐dose vaccine regimen. We defined patients as “unvaccinated” if they had not received any vaccine or had received the first dose of any vaccine regimen less than or equal to 14 days before their index ED visit.
The Bottom Line
Which clinical features, attainable on history and physical examination, can predict death or a hospital admission at 72 hours in patients discharged from the emergency department (ED)with COVID‐19? We developed the Canadian COVID‐19 Discharge Score (range 0–12) in a group of mostly unvaccinated ED patients that can risk‐stratify patients based on age, sex, temperature, arrival mode, pregnancy, respiratory distress, and respiratory rate. Patients with a score of 3 or less had a risk of the outcome of 1.9% or less, whereas patients with a score of 9 or more had up to a 12.2% risk of the outcome.
2.5. Outcome
The primary outcome was a short‐term adverse event defined as hospital admission or in‐hospital death within 72 hours of ED discharge.
2.6. Data analysis
We chose candidate predictor variables available at the bedside based on literature review and clinical knowledge. The candidate variables included age, sex, pregnancy, type of residence, ED arrival mode, comorbidities, symptoms, respiratory rate, ED oxygen delivery, physician or nurse impression of respiratory distress, medication administered in ED, immunization status, and use of alcohol, tobacco, vaping, and illicit substances. Data were complete except for arrival respiratory rate and fever, which had 4.2% and 2.3% missing, respectively (Supplementary Appendix Table 1).
2.6.1. Model development and validation
We randomly assigned participating sites to derivation or validation, with the goal of assigning 75% of eligible patients and outcome events to derivation and the remaining to validation (Supplementary Appendix Table 2). We used multiple imputations with missing predictor variable data. The initial logistic regression model considered all candidate predictors. We assessed outcomes independently for each ED visit in the data set. Continuous predictors were fit with restricted cubic splines having 3 knots. We assessed the strengths of associations between predictors and the outcome using an analysis of variance plot to inform the degrees of freedom to allocate to each predictor. We used a fast step‐down procedure to reduce the model to key predictors. 21 We conducted an internal bootstrap validation with 1000 bootstrap samples and 5 multiple imputations, then combined the results using a logit transformation and Rubin's rules to provide an optimism‐corrected area under the receiver‐operating characteristic (AUC). 22 , 23 We categorized continuous predictors based on the relationship between the spline function and outcome. To enable easy clinical use, we categorized continuous variables using clinically sensible cutpoints based on the spline function. We assigned integer points for each predictor variable, which were summed to calculate the score. We used a nomogram to assign points to form a score that ranged from 0 to 12. We calculated the sensitivity and specificity at different point thresholds, along with the score's discrimination and calibration. We validated the model in a cohort of geographically distinct sites that were not part of the derivation cohort. We performed analyses in R 24 using the rms 25 and pROC packages. 26
2.6.2. Evaluation of available sample size to produce reliable prediction modeling
We used the pmsampsize package 27 for sample size determination. Assuming an event rate of less than 10%, shrinkage of 0.9, and a conservative Cox‐Snell R‐squared of at least 0.1, 4.25 events per degree of freedom were required for reliable prediction modeling in the derivation cohort. 27 The 20 candidate predictor variables had 24 degrees of freedom indicating a minimum of 102 primary outcomes of interest were required in each of the derivation and validation samples.
3. RESULTS
3.1. Characteristics of study subjects
There were 152,943 ED visits captured in the CCEDRRN registry between March 1, 2020, and September 8, 2021. Of these, 15,305 satisfied inclusion/exclusion criteria and were included in our analysis (Figure 1). The characteristics and outcomes of included patients by derivation and validation cohort can be seen in Table 1.
FIGURE 1.
Flow diagram of included and excluded emergency department visits. Abbreviations: CCEDRRN, Canadian COVID‐19 Emergency Department Rapid Response Network; DNR, do not resuscitate; ED, emergency department; SARS‐CoV‐2, severe acute respiratory syndrome coronavirus‐2.
TABLE 1.
Patient characteristics in the derivation and validation cohorts
|
Derivation cohort (n = 11,487) |
Validation cohort (n = 3818) |
|
|---|---|---|
| Age in years, mean (SD) | 46.2 (16.2) | 46.3 (16.2) |
| Female sex, n (%) | 5928 (51.6) | 1880 (49.2) |
| Province, n (%) | ||
| Quebec | 2960 (25.8) | 573 (15.0) |
| British Columbia | 4052 (35.3) | 1662 (43.5) |
| Alberta | 2733 (23.8) | 1151 (30.1) |
| Ontario | 1554 (13.5) | 208 (5.4) |
| Saskatchewan | 114 (1.0) | 175 (4.6) |
| Nova Scotia | 68 (0.6) | 49 (1.3) |
| New Brunswick | 6 (0.1) | – |
| Arrival from, n (%) | ||
| Home (community) | 11,315 (98.5) | 3655 (95.7) |
| Institution/no fixed address | 172 (1.5) | 163 (4.3) |
| Arrival mode, n (%) | ||
| Self | 8537 (74.3) | 2704 (70.8) |
| Ambulance or police | 2950 (25.7) | 1114 (29.2) |
| Arrival heart rate, beats/min, mean (SD) | 93.4 (16.9) | 94.1 (17.5) |
| Arrival respiratory rate/min, mean (SD) | 19.1 (3.4) | 19.6 (4.0) |
| Arrival temperature in degrees Celsius, mean (SD) | 37 (0.8) | 36.9 (0.8) |
| Presence of respiratory distress, n (%) | 1586 (13.8) | 653 (17.1) |
| 10 most common symptoms reported by patients with COVID‐19, n (%) | ||
| Cough | 6687 (58.2) | 2137 (56.0) |
| Shortness of breath (dyspnea) | 5359 (46.7) | 1838 (48.1) |
| Fever | 5034 (43.8) | 1526 (40.0) |
| Chest pain (includes discomfort or tightness) | 3363 (29.3) | 1132 (29.6) |
| Abdominal pain | 1097 (9.5) | 336 (8.8) |
| Dizziness/vertigo | 998 (8.7) | 341 (8.9) |
| Altered consciousness/confusion | 248 (2.2) | 115 (3.0) |
| Hemoptysis (bloody sputum) | 153 (1.3) | 63 (1.7) |
| Hemorrhage (bleeding) | 124 (1.1) | 32 (0.8) |
| Seizures | 31 (0.3) | 5 (0.1) |
| Comorbidities, n (%) | ||
| Hypertension | 2124 (18.5) | 602 (15.8) |
| Diabetes | 1332 (11.6) | 325 (8.5) |
| Mental health diagnosis | 931 (8.1) | 342 (8.9) |
| Asthma | 877 (7.6) | 286 (7.5) |
| Coronary artery disease or congestive heart failure | 335 (3.0) | 151 (4.0) |
| Rheumatologic disorder | 298 (2.6) | 112 (2.9) |
| Chronic neurologic disorder (not dementia) | 235 (2.0) | 95 (2.5) |
| Chronic lung disease (not asthma or pulmonary fibrosis) | 217 (1.9) | 103 (2.7) |
| Active malignant neoplasm (cancer) | 187 (1.6) | 32 (0.8) |
| Past malignant neoplasm (cancer) | 174 (1.5) | 49 (1.3) |
| Atrial fibrillation | 151 (1.3) | 59 (1.5) |
| Chronic kidney disease or dialysis | 142 (1.2) | 60 (1.6) |
| Obesity (clinical impression) | 131 (1.1) | 38 (1.0) |
| Dementia | 49 (0.4) | 10 (0.3) |
| Mild liver disease | 41 (0.4) | 23 (0.6) |
| Organ transplant | 38 (0.3) | 17 (0.4) |
| Moderate/severe liver disease | 23 (0.2) | 7 (0.2) |
| Smoking or vaping, n (%) | ||
| Current or past | 638 (5.6) | 213 (5.6) |
| Never or not documented | 10,849 (94.4) | 3605 (94.4) |
| Illicit substance use, n (%) | ||
| Current or past | 317 (2.8) | 203 (5.3) |
| Never or not documented | 11,170 (97.2) | 3615 (94.7) |
| Pregnant, n (%) | 201 (1.7) | 67 (1.8) |
| Oxygen required in the emergency department, n (%) | 183 (1.6) | 83 (2.2) |
| Medication administered in ED: dexamethasone, hydrocortisone, or prednisone | 525 (4.6) | 181 (4.7) |
| COVID‐19 vaccination status, n (%) | ||
| Not vaccinated | 11,278 (98.2) | 3711 (97.2) |
| Partially/fully vaccinated | 209 (1.8) | 107 (2.8) |
| Outcomes, n (%) | ||
| Admission without death | 381 (3.3%) | 125 (3.3%) |
| In‐hospital death | 23 (0.2%) | 6 (0.2%) |
Of those included, 535 (3.5%) patients experienced the primary outcome of death or hospital admission within 72 h of ED discharge. In the derivation cohort, there were 11,487 ED visits to 35 sites with 404 (3.5%) primary outcome events, 23 (0.2%) of which were in‐hospital deaths (Table 1). In the validation cohort, there were 3818 ED visits to 14 different sites with 131 (3.4%) primary outcome events, 6 (0.2%) of which were in‐hospital deaths.
3.2. Main results
In derivation, the step‐down procedure produced a final model with 7 variables (Table 2). The resulting CCEDRRN COVID discharge score ranged from 0 to 12. We combined scores 10, 11, and 12 because the observed number of cases in each category was small. The discrimination of the CCEDRRN COVID discharge score was similar across derivation, validation, and combined cohorts with an AUC of 0.70 (95% CI 0.68–0.73), 0.71 (95% CI 0.67–0.75), and 0.71 (95% CI 0.68–0.73), respectively (Figure 2 and Supplementary Figure 1). The predicted risk associated with each score is shown in Supplementary Appendix Table 3. Observed and estimated risks for admission to hospital or death within 72 h associated with each score value in the validation cohort is shown in Table 3. We observed that 37.9% of the validation cohort scored 3 or less and represented a low‐risk group (risk of outcome 1.9% or less). The sensitivity of using a score of 3 to rule out admission or death at 72 h was 89.3% (Table 4). In the validation cohort, 4.8% scored 9 or more and represented a high‐risk group (risk of outcome up to 12.2%) associated with a specificity of 95.6%. The distribution of risk across score values and the sensitivity and specificity for all cutoff values in the combined cohort can be seen in Supplementary Appendix Tables 4 and 5.
TABLE 2.
Derivation of the CCEDRRN COVID discharge score based on adjusted associations between predictor variables and the composite outcome of death or hospital admission within 72 h of ED discharge
| Variable | Categories | Estimate | Standard error | Odds ratio | 95% CI | Points |
|---|---|---|---|---|---|---|
| Age (years) a | 0.76 | 0.10 | 2.15 | (1.75–2.64) | ||
| <40 | 0 | |||||
| 40–59 | 2 | |||||
| 60–79 | 3 | |||||
| ≥80 | 4 | |||||
| Sex | ||||||
| Female | Reference | Reference | Reference | Reference | 0 | |
| Male | 0.57 | 0.11 | 1.77 | (1.43–2.19) | 1 | |
| Temperature | ||||||
| 36–37.5 | Reference | Reference | Reference | Reference | 0 | |
| <36°C | −0.36 | 0.33 | 0.70 | (0.37–1.32) | 0 | |
| >37.5°C | 0.59 | 0.11 | 1.81 | (1.46–2.24) | 1 | |
| Arrival mode | ||||||
| Self | Reference | Reference | Reference | Reference | 0 | |
| Ambulance/police | 0.41 | 0.11 | 1.51 | (1.22–1.87) | 1 | |
| Pregnant | 1.25 | 0.36 | 3.50 | (1.71–7.16) | 2 | |
| Respiratory distress | 0.4 | 0.11 | 1.50 | (1.21–1.86) | 1 | |
| Arrival respiratory rate b | 0.27 | 0.09 | 1.31 | (1.09–1.58) | ||
| <20 | 0 | |||||
| 20–29 | 3 | |||||
| 30+ | 4 | |||||
aEstimate and odds ratio for age were calculated for the upper quartile (>57 years) versus the lower quartile (<34 years).
bEstimate and odds ratio for arrival respiratory rate were calculated for the upper quartile (>20) versus the lower quartile (<18).
Abbreviations: CCEDRRN, Canadian COVID‐19 Emergency Department Rapid Response Network; CI, confidence interval; ED, emergency department
FIGURE 2.
Distribution and performance of the CCEDRRN COVID discharge score in the derivation cohort (left panel) and validation cohort (right panel): (A) distribution of the score, (B) observed death or hospital admission within 72 h of ED discharge across the range of the score, (C) predicted versus observed probability of death of hospital admission within 72 h of ED discharge, and (D) receiver operating characteristic curve with area under the curve (AUC) and associated 95% confidence intervals.
TABLE 3.
Performance of the CCEDRRN COVID discharge score to identify risk of death or hospital admission within 72 hours of ED discharge at different cutoff values in the validation cohort
| Score | Total patients | Risk category | Proportion of patients | Observed risk | Expected risk | |||
|---|---|---|---|---|---|---|---|---|
| N | % | % | N | % | N | % | ||
| 0 | 169 | 4.4 | Low | 37.9 | 1 | 0.6 | 1 | 0.7 |
| 1 | 364 | 9.5 | 1 | 0.3 | 3 | 0.9 | ||
| 2 | 378 | 9.9 | 2 | 0.5 | 5 | 1.2 | ||
| 3 | 535 | 14.0 | 10 | 1.9 | 9 | 1.7 | ||
| 4 | 603 | 15.8 | Medium | 57.3 | 15 | 2.5 | 14 | 2.3 |
| 5 | 468 | 12.3 | 22 | 4.7 | 15 | 3.1 | ||
| 6 | 438 | 11.5 | 20 | 4.6 | 19 | 4.3 | ||
| 7 | 386 | 10.1 | 18 | 4.7 | 22 | 5.8 | ||
| 8 | 294 | 7.7 | 22 | 7.5 | 23 | 7.8 | ||
| 9 | 139 | 3.6 | High | 4.8 | 17 | 12.2 | 14 | 10.4 |
| ≥10 | 44 | 1.2 | 3 | 6.8 | 6 | 13.7 | ||
Abbreviations: CCEDRRN, Canadian COVID‐19 Emergency Department Rapid Response Network; ED, emergency department
TABLE 4.
Performance metrics of the CCEDRRN COVID discharge score in the validation cohort using different score cutoffs
| Score cutoff |
Cumulative patients with cutoff score or less n (%) |
Sensitivity (%, 95% CI) | Specificity (%, 95% CI) |
|---|---|---|---|
| 0 | 169 (4.4) | 99.2 (95.8–100) | 4.6 (3.9–5.3) |
| 1 | 533 (14) | 98.5 (94.6–99.8) | 14.4 (13.3–15.6) |
| 2 | 911 (23.9) | 96.9 (92.4–99.2) | 24.6 (23.2–26) |
| 3 | 1446 (37.9) | 89.3 (82.7–94) | 38.8 (37.3–40.4) |
| 4 | 2049 (53.7) | 77.9 (69.8–84.6) | 54.8 (53.2–56.4) |
| 5 | 2517 (65.9) | 61.1 (52.2–69.5) | 66.9 (65.3–68.4) |
| 6 | 2955 (77.4) | 45.8 (37.1–54.7) | 78.2 (76.9–79.5) |
| 7 | 3341 (87.5) | 32.1 (24.2–40.8) | 88.2 (87.1–89.2) |
| 8 | 3635 (95.2) | 15.3 (9.6–22.6) | 95.6 (94.9–96.2) |
| 9 | 3774 (98.8) | 2.3 (0.5–6.5) | 98.9 (98.5–99.2) |
| 10–12 | 3818 (100) | 0 (0–2.8) | 100 (99.9–100) |
Abbreviations: CCEDRRN, Canadian COVID‐19 Emergency Department Rapid Response Network; CI, confidence interval
3.3. Limitations
Our model needs to be externally validated in other health systems. We were unable to measure natural immunity related to prior infection in our cohort and this factor may be associated with risk of short‐term outcomes. The generalizability of CCEDRRN COVID discharge score is supported by the fact that it was derived and validated using data from ED visits occurring over 4 waves of the pandemic, during a time when multiple variants evolved and while population vaccination rates were increasing. 28 , 29 However, as the virus continues to evolve and the immunity of the population changes with vaccination and natural infection, the performance of this CDR may also change. As such, the CCEDRRN COVID discharge score requires evaluation in the context of contemporary SARS‐CoV2 variants, high vaccination rates, and natural immunity. Because of the retrospective nature of our study, we were dependent on the quality of data charted in the clinical record. It is possible that data for some predictor variables were incomplete and may have led to misclassification bias. We were not able to capture deaths or admissions that occurred outside of hospitals participating in CCEDRRN, and we were not able to capture deaths that occurred in the community. Our model did not include duration of illness, and this may have influenced the risk for short‐term adverse events after ED discharge. Our tool achieved only moderate discrimination (AUC 0.70, 95% CI 0.68–0.73). This suggests there are likely to be patient, disease, or contextual factors not accounted for by the elements included in the final rule. Future research should explore these other potential factors.
4. DISCUSSION
We derived and validated a parsimonious and simple score to identify discharged COVID‐19 patients who are at risk for admission or death within 72 hours: the CCEDRRN COVID discharge score . We found that 7 bedside clinical variables can be used to risk stratify patients based on death or hospital admission at 72 h after the ED visit. Variables in the final model included age, sex, temperature, arrival mode (ambulance/police versus self), pregnancy, respiratory distress, and arrival respiratory rate.
The CCEDRRN COVID discharge score is most appropriate for application in unvaccinated individuals because 98% of the patients in our cohort were unvaccinated. Tools for use in the unvaccinated remain relevant because 15% of the Canadian population (approximately 6 million people) and 20.5% of the US population (approximately 68 million people) are unvaccinated as of September 2022. 30 , 31 There are other regions of the world with even higher proportions of the population being unvaccinated. For example, only 30% of the population of the African continent have received at least 1 dose of COVID‐19 vaccination. 32 We suspect that unvaccinated individuals will continue to be overrepresented in the ED population because they tend to experience more severe COVID‐19 after SARS‐CoV2 infection.
The CCEDRRN COVID discharge score can be used to risk‐stratify patients based on short‐term outcomes, especially at the extremes of score values. For example, among those with a CCEDRRN COVID discharge score of ≤3, the risk of adverse outcomes at 72 h is 1.9% or less and those with a score of ≥9 points had a much higher rate of the outcome (up to 12.2%). Depending on physician and patient risk tolerance and local resource availability, identifying patients in the low‐ or high‐risk CCEDRRN COVID discharge score categories could contribute meaningfully toward developing a safe disposition plan. Future research should determine whether CCEDRRN COVID discharge score improves on physician gestalt.
Other CDRs designed to support prognostication in ED COVID‐19 patients have been published. The Hospitalization or Outpatient Management of Patients with a SARS‐CoV‐2 Infection (HOME‐CoV) rule was developed through a Delphi method involving clinicians and then validated using data from 34 hospitals in Europe. 13 This before‐after study included patients with probable and confirmed COVID‐19 and had a primary outcome of invasive ventilation or death occurring within 7 days after patient assessment. This rule was used in a dichotomous way, with patients being classified as only rule positive or rule negative. Event rates in both groups were extremely low (≤3%). The rule had no meaningful impact on actual admission rates during the study. The comorbidities, obesity/body mass index, vital signs, age and sex (COVAS) score was derived and validated with data from 26,600 ED encounters across 15 hospitals in southern California. 16 Patients included those from the ED with a diagnosis of pneumonia, suspected or confirmed COVID‐19, exposure to COVID‐19, or symptoms suspicious for COVID‐19. The primary outcome was death or respiratory decompensation (including any oxygen need, including facemask, non‐invasive ventilation, or intubation) within 7 days, which occurred in 4% of patients. The final COVAS score included the 5 categories of comorbidities, body mass index, vital signs, age, and sex. The major limitation of the COVAS study was that less than 30% of the study cohort were tested for SARS CoV‐2 and only 18% were positive. It is likely that many in the cohort had infections with pathogens other than SARS CoV‐2. The COVID Home Safely Now (CHOSEN) score aimed to predict suitability for discharge from a monitored setting among patients with COVID‐19 admitted to the hospital or the ED. 14 The score was derived and validated using data from 10 hospitals in Massachusetts. Seventy‐seven percent of the cohort were patients admitted to hospital. The primary outcome was a composite of hypoxia, ICU admission, or death at 14 days. Although 2059 patients were enrolled, 733 (36%) were excluded because 14‐day follow up data were missing. A 3‐item risk score ranging from 0 to 55 points was developed consisting of age, oxygen saturation, and albumin. At a cutpoint of 30, the score had a sensitivity of 83% and specificity of 82%. The generalizability of this rule to the ED is limited because it was derived from mostly inpatient data. This rule requires laboratory investigation, which is not indicated in most patients being considered for discharge home from the ED with COVID‐19.
The CCEDRRN COVID discharge score has several advantages over other CDRs for use in the ED. 12 , 14 , 15 Our rule is simple and does not rely on the results of investigations or determination of patient comorbidities. The determination of comorbidities in the ED can be unreliable. 33 The CCEDRRN COVID discharge score uses an ultra‐short follow‐up period of 72 hours. Several prior studies used longer intervals including 7, 13 , 16 14, 14 or even 30 days. 5 , 7 , 10 , 12 We felt that a rule predicting outcomes within 72 hours of ED assessment might convey a more appropriate assessment of immediate risks that might be mitigated with decisions made in the ED, specifically around the decision to admit at the index visit. The 72‐hour interval is commonly used as a health care quality metric for patient recidivism. 34 , 35 , 36 , 37 The CCEDRRN COVID discharge score was derived and validated with data from across 49 EDs with varied academic/community status, sizes, and community types across 4 waves of the pandemic while 5 known variants of concern were circulating. 18 , 28 , 29 This supports the generalizability of our rule, especially for the Canadian ED setting. The CCEDRRN COVID discharge score was developed using data from patients with a confirmed, laboratory‐proven diagnosis of COVID‐19. In contrast to other published CDRs that were derived using much broader inclusion criteria and patients without confirmed SARS CoV‐2 infection, the CCEDRRN COVID discharge score better reflects the pathophysiology and natural history of COVID‐19. Lastly, our cohort included only those who were discharged home rather than a more heterogeneous cohort of obviously sicker patients admitted to hospital included in other CDRs. Patients with obvious indications for admission are likely to be sicker, older, and do not require the use of a CDR to identify that they are sufficiently ill to require admission.
In addition to potential clinical use, the CCEDRRN COVID discharge score may prove useful as a research tool. For example, risk stratification of subjects being considered for clinical trials with a CCEDRRN COVID discharge score could support targeted enrollment of high‐risk patients most likely to benefit from treatment and could also support subgroup analyses based on baseline risk.
In summary, the CCEDRRN COVID discharge score, which includes age, sex, temperature, arrival mode, pregnancy status, the presence of respiratory distress, respiratory rate, and pregnancy status, is associated with a patient's risk of death or hospital admission within 72 hours of ED discharge. The rule, which has potential clinical and research applications, is particularly relevant for unvaccinated individuals who present to the ED with COVID‐19.
AUTHOR CONTRIBUTIONS
Corinne Hohl conceived the idea behind the study, with input on the design and selection of variables from all other contributors. Steven Brooks, Corinne Hohl, and Laurie J. Morrison obtained funding as principal applicants on behalf of the CCEDRRN investigators. Steven Brooks, Hana Wiemer, Patrick Fok, MeganLandes, Robert Ohle, Michelle Welsford, and Corinne Hohlmanaged data collection along with other members of the CCEDRRN and verified the accuracy of underlying data. Steven Brooks, Rhonda J. Rosychuk, Jeffrey Perry, Laurie J. Morrison, Shabnam Vatanpour, and Corinne Hohldeveloped the analytic plan. Shabnam Vatanpour performed the analysis, with assistance from Rhonda J. Rosychuk, including accessing and verification of underlying data. All contributors provided input on the interpretation of our findings. Steven Brooks, Rhonda J. Rosychuk, and Shabnam Vatanpour drafted the manuscript. All authors reviewed and provided critical input to develop the final version.
CONFLICTS OF INTEREST
The authors have declared no conflict of interest.
Supporting information
Supplementary Information
Supplementary Information
Supplementary Information
ACKNOWLEDGMENTS
We thank the University of British Columbia clinical coordinating center staff; the UBC legal, ethics, privacy, and contract staff; and the research staff at each of the participating institutions in the network (Supplementary Appendix Table 6). The network would not exist today without the dedication of these professionals. Thank you to all our patient partners who shared their lived experiences and perspectives to ensure that the knowledge we co‐create addresses the concerns of patients and the public. Creating the largest network of collaboration across Canadian emergency departments would not have been feasible without the tireless efforts of emergency department chiefs, research coordinators, and research assistants at participating sites. Finally, our most humble and sincere gratitude to all our colleagues in medicine, nursing, and the allied health professions who have been on the front lines of this pandemic from day one staffing our ambulances, emergency departments, ICUs, and hospitals bravely facing the risks of COVID‐19 to look after our fellow citizens and after one another. We dedicate this network to you. The Canadian Institutes of Health Research (447679), Ontario Ministry of Colleges and Universities (C‐655‐2129), Saskatchewan Health Research Foundation (5357), Genome BC (COV024 and VAC007), Foundation du CHU de Québec (Octroi No. 4007), Sero‐Surveillance and Research (COVID‐19 Immunity Task Force Initiative), and Public Health Agency of Canada provided peer‐reviewed funding. The BC Academic Health Science Network and BioTalent Canada provided non‐peer reviewed funding. These organizations are not‐for‐profit, and had no role in study conduct, analysis, or manuscript preparation.
Biography
Steven C. Brooks, MD, MHSc, is an Associate Professor, Clinician Scientist, and emergency physician in the Department of Emergency Medicine at Queen's University in Kingston, Ontario.
Brooks SC, Rosychuk RJ, Perry JJ, et al. Derivation and validation of a clinical decision rule to risk‐stratify COVID‐19 patients discharged from the emergency department: The CCEDRRN COVID discharge score. JACEP Open. 2022;3:e12868. 10.1002/emp2.12868
Meetings: This work was presented at the Queen's University Department of Emergency Medicine R. J. Brison Research Day May 19, 2022.
Supervising Editor: Marie Vrablik, MD, MCR.
REFERENCES
- 1. Verma AA, Hora T, Jung HY, et al. Characteristics and outcomes of hospital admissions for COVID‐19 and influenza in the Toronto area. Can Med Assoc J. 2021;193(12):E410‐E418. doi: 10.1503/cmaj.202795 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Richardson S, Hirsch JS, Narasimhan M, et al. Presenting characteristics, comorbidities, and outcomes among 5700 patients hospitalized with COVID‐19 in the New York City area. JAMA. 2020;323(20):2052‐2059. doi: 10.1001/jama.2020.6775 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Gao Z, Xu Y, Sun C, et al. A systematic review of asymptomatic infections with COVID‐19. J Microbiol Immunol Infect. 2021;54(1):12‐16. (In eng). doi: 10.1016/j.jmii.2020.05.001 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Hohl CM, Rosychuk RJ, Hau JP, et al. Treatments, resource utilization, and outcomes of COVID‐19 patients presenting to emergency departments across pandemic waves: an observational study by the Canadian COVID‐19 emergency department rapid response network (CCEDRRN). Can J Emerg Med. 2022. doi: 10.1007/s43678-022-00275-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Beiser DG, Jarou ZJ, Kassir AA, et al. Predicting 30‐day return hospital admissions in patients with COVID‐19 discharged from the emergency department: a national retrospective cohort study. J Am Coll Emerg Physicians Open. 2021;2(6):e12595. (In eng). doi: 10.1002/emp2.12595 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. López‐Barbeito B, García‐Martínez A, Coll‐Vinent B, et al. Factors associated with revisits by patients with SARS‐CoV‐2 infection discharged from a hospital emergency department. Emergencias. 2020;32(6):386‐394. [PubMed] [Google Scholar]
- 7. McKay B, Meyers M, Rivard L, Stoltzfus J, Rammohan G, Stankewicz H. Predicting 30 ‐ Day outcomes in emergency department patients discharged with COVID‐19. Am J Emerg Med. 2021;50:513‐517. (In eng). doi: 10.1016/j.ajem.2021.08.077 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Menditto VG, Fulgenzi F, Bonifazi M, et al. Predictors of readmission requiring hospitalization after discharge from emergency departments in patients with COVID‐19. Am J Emerg Med. 2021;46:146‐149. (In eng). doi: 10.1016/j.ajem.2021.04.055 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Möckel M, Stegemann MS, Burst V, et al. Which parameters support disposition decision in suspected COVID‐19 cases in the emergency department (ED): a German clinical cohort study. BMJ Open. 2021;11(3):e044853. (In eng). doi: 10.1136/bmjopen-2020-044853 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. van Dam P, Zelis N, van Kuijk SMJ, et al. Performance of prediction models for short‐term outcome in COVID‐19 patients in the emergency department: a retrospective study. Ann Med. 2021;53(1):402‐409. (In eng). doi: 10.1080/07853890.2021.1891453 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. McGinn TG, Guyatt GH, Wyer PC, Naylor CD, Stiell IG, Richardson WS. Users' guides to the medical literature: XXII: how to use articles about clinical decision rules. evidence‐based medicine working group. JAMA. 2000;284(1):79‐84. (In eng). doi: 10.1001/jama.284.1.79 [DOI] [PubMed] [Google Scholar]
- 12. Azijli K, Lieveld A, van der Horst S, et al. Predicting poor outcome in patients with suspected COVID‐19 presenting to the emergency department (COVERED) ‐ Development, internal and external validation of a prediction model. Acute Med. 2021;20(1):4‐14. (In eng). [PubMed] [Google Scholar]
- 13. Douillet D, Penaloza A, Mahieu R, et al. Outpatient management of patients with COVID‐19: multicenter prospective validation of the hospitalization or outpatient management of patients with SARS‐CoV‐2 infection rule to discharge patients safely. Chest. 2021;160(4):1222‐1231.. doi: 10.1016/j.chest.2021.05.008 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Levine DM, Lipsitz SR, Co Z, Song W, Dykes PC, Samal L. Derivation of a clinical risk score to predict 14‐day occurrence of hypoxia, ICU admission, and death among patients with coronavirus disease 2019. J Gen Intern Med. 2021;36(3):730‐737. (In eng). doi: 10.1007/s11606-020-06353-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Salvatore V, Gianstefani A, Farina G, et al. CovHos, a new score to predict the need of hospitalization for coronavirus disease 2019 (COVID‐19) patients at the emergency department. Cureus. 2021;13(10):e18717. (In eng). doi: 10.7759/cureus.18717 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Sharp AL, Huang BZ, Broder B, et al. Identifying patients with symptoms suspicious for COVID‐19 at elevated risk of adverse events: the COVAS score. Am J Emerg Med. 2021;46:489‐494. (In eng). doi: 10.1016/j.ajem.2020.10.068 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Margus C, Sondheim SE, Peck NM, et al. Discharge in pandemic: suspected covid‐19 patients returning to the emergency department within 72 hours for admission. Am J Emerg Med. 2021;45:185‐191. (In eng). doi: 10.1016/j.ajem.2020.08.034 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Hohl CM, Rosychuk RJ, McRae AD, et al. Development of the Canadian COVID‐19 emergency department rapid response network population‐based registry: a methodology study. CMAJ Open. 2021;9(1):E261‐E270. doi: 10.9778/cmajo.20200290 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Moons KG, Altman DG, Reitsma JB, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162(1):W1‐W73. (In eng). doi: 10.7326/m14-0698 [DOI] [PubMed] [Google Scholar]
- 20. Status of COVID‐19 Vaccines Within WHO EUL/PQ Evaluation Process. World Health Organization. https://extranet.who.int/pqweb/key‐resources/documents/status‐covid‐19‐vaccines‐within‐who‐eulpq‐evaluation‐process
- 21. Lawless JF, Singhal K. Efficient screening of nonnormal regression models. Biometrics. 1978;34:318. [Google Scholar]
- 22. Rubin D. Multiple Imputation for Nonresponse in Surveys. John Wiley & Sons, 1987. [Google Scholar]
- 23. Snell KI, Ensor J, Debray TP, Moons KG, Riley RD. Meta‐analysis of prediction model performance across multiple studies: which scale helps ensure between‐study normality for the C‐statistic and calibration measures? Stat Methods Med Res. 2018;27(11):3505‐3522. (In eng). doi: 10.1177/0962280217705678 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. R Core Team . R: A Language And Environment For Statistical Computing. R Foundation for Statistical Computing, 2021;397‐407. 10.1007/s43678-022-00275-3 [DOI] [Google Scholar]
- 25. Harrell FE Jr, rms:Regression Modeling Strategies. R package version 6.2‐0. https://CRAN.R‐project.org/package=rms.2021
- 26. Robin X, Turck N, Hainard A, et al. pROC: an open‐source package for R and S+ to analyze and compare ROC curves. BMC Bioinform. 2011;12:77. (In eng). doi: 10.1186/1471-2105-12-77 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27. Riley RD, Snell KI, Ensor J, et al. Minimum sample size for developing a multivariable prediction model: pART II ‐ binary and time‐to‐event outcomes. Stat Med. 2019;38(7):1276‐1296. doi: 10.1002/sim.7992 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28. COVID‐19 epidemiology update . Government of Canada. October 21, 2022. https://health‐infobase.canada.ca/covid‐19/#a4
- 29. Le T. Updates on COVID‐19 Variants of Concern (VOC). National Collaborating Centre for Infectious Diseases. May 20, 2022. https://nccid.ca/covid‐19‐variants/
- 30. Canadian COVID‐19 vaccination coverage report . Ottawa: Public Health Agency of Canada, 2022. https://health‐infobase.canada.ca/covid‐19/vaccination‐coverage/
- 31. Centers for Disease Control and Prevention . COVID Data Tracker. Atlanta: US Department of Health and Human Resources, CDC, 2022. https://covid.cdc.gov/covid‐data‐tracker
- 32. Holder J. Tracking Coronavirus Vaccinations Around The World. New York Times. October 23, 2022. https://www.nytimes.com/interactive/2021/world/covid‐vaccinations‐tracker.html
- 33. Hohl CM, Rosychuk RJ, Archambault PM, et al. The CCEDRRN COVID‐19 mortality score to predict death among nonpalliative patients with COVID‐19 presenting to emergency departments: a derivation and validation study. CMAJ Open. 2022;10(1):E90‐e99. (In eng). doi: 10.9778/cmajo.20210243 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34. Keith KD, Bocka JJ, Kobernick MS, Krome RL, Ross MA. Emergency department revisits. Ann Emerg Med. 1989;18(9):964‐968. doi: 10.1016/S0196-0644(89)80461-5 [DOI] [PubMed] [Google Scholar]
- 35. Lerman B, Kobernick MS. Return visits to the emergency department. J Emerg Med. 1987;5(5):359‐362. doi: 10.1016/0736-4679(87)90138-7 [DOI] [PubMed] [Google Scholar]
- 36. Ngai KM, Grudzen CR, Lee R, Tong VY, Richardson LD, Fernandez A. The association between limited english proficiency and unplanned emergency department revisit within 72 hours. Ann Emerg Med. 2016;68(2):213‐221. doi: 10.1016/j.annemergmed.2016.02.042 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37. Pierce JM, Kellerman AL, Oster C. “Bounces”: an analysis of short‐term return visits to a public hospital emergency department. Ann Emerg Med. 1990;19(7):752‐757. doi: 10.1016/S0196-0644(05)81698-1 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Supplementary Information
Supplementary Information
Supplementary Information