Performance of point-of-care severity scores to predict prognosis in patients admitted through the emergency department with COVID-19

Abstract

Background:

Identifying COVID-19 patients at highest risk of poor outcomes is critical at emergency department(ED) presentation. Sepsis risk stratification scores can be calculated quickly for COVID-19 patients but have not been evaluated in a large cohort.

Objective:

To determine whether well-known risk scores can predict poor outcomes among hospitalized COVID-19 patients.

Design, Setting, and Participants:

Retrospective cohort study of adults presenting with COVID-19 to 156 Hospital Corporation of America(HCA) Healthcare EDs, March 2,2020-February 11,2021.

Intervention or Exposure:

Sequential Organ Failure Assessment(qSOFA), Shock Index, National Early Warning System-2(NEWS2), and quick COVID-19 Severity Index(qCSI) at presentation

Main Outcomes and Measures:

Primary outcome was in-hospital mortality. Secondary outcomes included intensive care unit(ICU) admission, mechanical ventilation, and vasopressors receipt. Patients scored positive with qSOFA≥2, Shock Index>0.7, NEWS2≥5, and qCSI≥4. Test characteristics and area under the receiver operating characteristics curves (AUROCs) were calculated.

Results:

We identified 90,376 patients with community-acquired COVID-19(mean age 64.3 years, 46.8% female). 17.2% patients died in-hospital, 28.6% went to the ICU, 13.7% received mechanical ventilation, and 13.6% received vasopressors. There were 3.8% qSOFA-positive, 45.1% Shock Index-positive, 49.8% NEWS2-positive, and 37.6% qCSI-positive at ED-triage. NEWS2 exhibited the highest AUROC for in-hospital mortality (0.593, CI 0.588–0.597), ICU admission(0.602, CI 0.599–0.606), mechanical ventilation(0.614, CI 0.610–0.619), and vasopressor receipt(0.600, CI 0.595–0.604).

Conclusions:

Sepsis severity scores at presentation have low discriminative power to predict outcomes in COVID-19 patients and are not reliable for clinical use. Severity scores should be developed using features that accurately predict poor outcomes among COVID-19 patients to develop more effective risk-based triage.

INTRODUCTION

Novel severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection, and the coronavirus disease 2019 (COVID-19) it causes, has swept through the globe and is responsible for over 580 million global cases and over 6 million attributable deaths¹ since its discovery in December 2019. The most common signs of COVID-19 in adults include fever, cough, shortness of breath, headache, and myalgias,² several of which can be assessed at emergency department (ED) presentation. The ability to quickly identify patients at highest risk of poor outcomes from COVID-19 could lead to swifter clinical management and allocation of resources in the ED setting.

Several bedside risk stratification scores rely solely on easy-to-collect vital signs data and have been readily implemented in electronic health record (EHR) systems to provide prognostic information for seriously ill patients with sepsis, including the quick Sequential Organ Failure Assessment (qSOFA), the Shock Index, and the National Early Warning System (NEWS2). These sepsis severity scores have also been used to assess risk of conditions like community-acquired pneumonia, hypercapnic respiratory failure, pulmonary embolism, chronic obstructive pulmonary disorder, and heart failure,^3–13 and may prove useful among patients with COVID-19. The benefit of bedside risk scores is that they are not subject to laboratory delay, and they can be recalculated quickly to assess a patient’s prognosis over time. qSOFA was developed to determine risk of poor outcomes from sepsis in non-ICU settings.^14–16 The Shock Index is a simple risk score (heart rate divided by systolic blood pressure) that was developed to detect changes in cardiovascular performance before the onset of shock and an elevated Shock Index is associated with poor outcomes.^17–19 NEWS2 is the official score used to identify those at risk of poor outcomes from sepsis in the United Kingdom National Health Service.^14,20–24 COVID-19-specific risk stratification scores, such as the Quick COVID-19 Severity Index (qCSI)²⁵, have been developed to incorporate hallmark symptoms of the disease into bedside risk assessment.

The predictive abilities of these commonly implemented bedside risk stratification scores have been assessed in patients with sepsis^{15,20–22,26} and other conditions,^3–13 but the performance of qSOFA, Shock Index, NEWS2, and qCSI have not been compared in a large national cohort of COVID-19 patients. Our objective was to provide a head-to-head comparison of the test characteristics of qSOFA, the Shock Index, NEWS2, and qCSI calculated at ED triage for predicting in-hospital mortality as well asadmission to the intensive care unit (ICU), receipt of mechanical ventilation, and receipt of vasopressors in patients hospitalized with COVID-19 within a large national health system.

METHODS

We conducted a retrospective cohort study of adults admitted to Hospital Corporation of America (HCA) Healthcare facilities through the ED and hospitalized due to COVID-19 disease between March 2, 2020, and February 11, 2021. HCA Healthcare is the largest private healthcare system in the United States comprised of 180 acute care facilities in the United States. The majority of these facilities are medium to large community hospitals. Hospital characteristics of the included sites appear in Supplemental Table 1. The study was approved with a waiver of informed consent by the WCG-IRB.

Data Source:

Data were collected for any encounter occurring after the date of the first positive SARS-CoV-2 test, as well as any encounter occurring 14 days prior to positive test via EHRs at participating sites (Meditech version 5.6, Medical Information Technology, Inc., Westwood, MA), and were aggregated into a central repository. This clinical data warehouse (CDW) aggregates data streams for retrospective analysis and use in real-time data decision-support tools to inform care. Certified queries were used to extract deidentified data from the CDW to develop a COVID-19 registry for HCA Healthcare. Data processing and standardization for registry development was done using Genospace, a cloud-based biomedical data ingestion, transformation, and de-identification platform.

Severity Scores:

We compared the ability of qSOFA, the Shock Index, and NEWS2 to predict key outcomes during the index COVID-19 hospitalization. We used the first vital signs collected within 24 hours to calculate severity scores. Patients with no vital signs data within the first 24 hours were excluded. Data elements used to calculate qSOFA, Shock Index, NEWS2, and qCSI can be found in Table 1. Patients were qSOFA-positive if they had a score ≥2, Shock Index-positive if they had a value of >0.7, NEWS2-positive if they had a score ≥5, and qCSI-positive if they had a score of ≥4 at least once, based on vitals from the first 24 hours of admission.^24,25,27,28 Because the “alert, verbal, confusion, pain, unresponsive” (AVCPU) scale was not consistently available in our data, we used GCS score <15 which has been used in prior studies.^15,29 Missing GCS assessments were considered normal for the primary analysis.

Table 1.

Data elements collected within the first 24 hours after emergency department presentation to identify patients meeting qSOFA, Shock Index, NEWS2, and qCSI criteria

Organ System	Quick Sequential Organ Failure Assessment Score (Range 0–3)	Shock Index (Heart rate/systolic blood pressure)	National Early Warning Score 2 (Range 0–20)	Quick COVID-19 Severity Index (Range 0–12)
Respiratory	Respiratory rate ≥ 22		Respiratory rate: 3 points: ≤ 8; 1 point: 9–11; 0 points: 12–20; 2 points: 21–24; 3 points: ≥25 Oxygen saturation Scale 1: 3 points: ≤91%; 2 points: 92–93%; 1 point: 94–95%; 0 points: ≥96% Oxygen saturation Scale 2: 3 points: ≤83%; 2 points: 84–85%; 1 point: 86–87%: 0 points: 88–92% or ≥93% on room air; 1 point: 93–94% on O2; 2 points: 95–96% on O2; 3 points ≥97% on O2 Requirement for supplemental oxygen: 2 points: Any supplemental oxygen; 0 points: No supplemental oxygen	Respiratory rate: 1 point: 23–28; 2 points: >28 Oxygen saturation: 0 points: >92%; 2 points: 89–92%; 5 points: ≤ 88% Oxygen flow rate, L/min: 0 points: ≤2; 4 points: 3–4
Cardiac	Systolic blood pressure < 100 mmHg	Ratio of systolic blood pressure to heart rate > 0.7	Systolic blood pressure: 3 points: ≤90; 2 points: 91–100; 1 point: 101–110; 0 points: 111–219; 3 points: ≥220 Heart rate: 3 points: ≤40; 1 point: 41–50; 0 points: 51–90; 1 point: 91–110; 2 points: 111–130; 3 points: ≥131
Neurologic	Glasgow Coma Scale value < 15		Glasgow Coma Scale value <15*
Other			Temperature 3 points: ≤35°C / 95°F; 1 point: 35.1–36°C / 95.1–96.8°F; 0 points: 36.1–38°C / 96.9–100.4°F; 1 point: 38.1–39°C / 100.5–102.2°F; 2 points: ≥39.1 °C / 102.3°F

^*NEWS2 utilizes level of consciousness, which was not collected. We have used the Glasgow Coma Scale value <15 as a surrogate for level of consciousness.

Outcomes:

Our primary outcome was in-hospital mortality. Secondary outcomes included admission to the ICU, receipt of mechanical ventilation, and vasopressor administration were collected from the EHR for each day of hospitalization. In-hospital mortality was defined using the hospital discharge disposition. Patients were counted as having an ICU admission if their location in the hospital was ever ICU following ED presentation. Mechanical ventilation was determined based on oxygen support data that were captured on each patient. Vasopressor receipt was identified based on intravenous receipt of dopamine, dobutamine, epinephrine, norepinephrine, phenylephrine, and vasopressin at least once during the hospital stay.

Covariates:

Demographic data included age at first positive SARS-CoV-2 test, gender, race, ethnicity, smoking status, and date of death. Patients who were with a do not resuscitate (DNR) status were included in the primary analysis and were identified using the ICD-10 code Z66 present on admission. International Classification of Diseases (ICD)-10 diagnosis codes were extracted and the R “comorbidity” package was used to identify comorbidities based on Elixhauser comorbidity groupings.³⁰ To estimate burden of comorbid illness, we used the van Walraven comorbidity index³¹ which provides an estimated risk of in-hospital death based on documented Elixhauser comorbidities.

Statistical Analysis:

Analyses were conducted using R (R Core Team 2021) and Stata 14 (StataCorp, College Station, Texas). We summarized differences in demographic and clinical characteristics between the populations meeting each severity score but elected not to conduct hypothesis testing because patients could be positive for more than one score. We calculated sensitivity, specificity, positive predictive value, and negative predictive value for each score to predict our outcomes of interest. To assess discrimination of the scores, we calculated the area under the receiver operating characteristic curve (AUROC) along with asymptotic normal 95% confidence interval (95% CI). The AUROC ranges from 0.50 to 1.00 and a score 0.70–0.80 can be considered fair.³² We assessed significant differences between AUROCs using the DeLong method³³ via Stata’s “roccomp” command.

Sensitivity Analyses:

We conducted several sensitivity analyses to explore our data further. 1) To determine whether vital signs collected earlier in the hospital admission would be more predictive of outcomes, we calculated the scores using the first vital signs collected within the first 12 hours of hospital stay. We also calculated scores using the first vital signs collected within the first 6 hours of hospital stay. 2) We excluded DNR patients to determine whether the performance improved so that we restricted our analysis population to those eligible for outcomes. 3) Given the level of missingness in the GCS measurements, we compared patients who had a documented GCS score to those who did not in order to identify differences in patients who had mental status charted in the EHR. We then calculated score performance among those who had at least one GCS value documented within 24 hours of ED admission. 4) To explore whether we would have improved prediction of poor outcomes for each score using the observed data instead of the standard cutpoints for increased risk, we calculated the Youden Index to identify the optimal cutpoint for each score based on the observed data.³⁴ 5) Finally, given that the study spanned a 12-month period in a pandemic with evolving treatments and interventions, we explored whether there was an association between month of study and score performance by outcome and by month, estimated by AUROC.

RESULTS

Our analysis dataset included 90,376 patients with community-acquired COVID-19 admitted through 156 HCA Healthcare EDs during the study period and who had at least one vital sign documented within 24 hours of ED admission. The average time to first vital sign was 3.86 hours (+/−6.2 hours; median 4.5 hours, IQR 4.0–5.3 hours). The cohort mean age was 64.3 years (+/− 6.3 years; median 66 years, IQR 53–77 years) and 46.8% were female (n=42,324). There were 15,553 (17.2%) in-hospital deaths, 25,814 (28.6%) with ICU admissions, 12,412 (13.7%) who received mechanical ventilation, and 12,326 (13.6%) who received vasopressors during the index COVID-19 admission. At ED triage, 3,418 (3.8%) were qSOFA-positive, 40,732 (45.1%) were Shock Index-positive, 45,036 (49.8%) were NEWS2-positive, and 33,951 (37.6%) were qCSI-positive. Blood pressure, heart rate, respiratory rate, and oxygen saturation were documented in ≥98% of admissions, 85% had supplemental oxygen data, and 24% had GCS documented at ED triage.

Demographic Characteristics and Clinical Course

There were differences in characteristics based on severity scores met (Table 2). Although no hypothesis testing was conducted due to overlap in patients meeting multiple scores, those qSOFA-positive compared to those positive for the Shock Index, NEWS2, and qCSI were older and had a greater degree of underlying comorbidities based on the van Walraven Comorbidity Index but were less likely to be past or current smokers. Those qSOFA-positive compared to those positive for the Shock Index, NEWS2, and qCSI were more likely to be admitted to the ICU during their hospitalization, more likely to be mechanically ventilated during their hospital stay, and were more likely to receive vasopressors as part of their clinical management. In-hospital mortality was also more common among those qSOFA-positive at triage.

Table 2.

Demographic characteristics and clinical course of patients hospitalized with COVID-19 as well as populations that met standard cutpoints of qSOFA, Shock Index, NEWS2, and qCSI within 24 hours of ED admission

Variable	Full cohort N=90,376	qSOFA n= 3,418	Shock Index n= 40,732	NEWS2 n=45,036	qCSI n=33,951
Characteristics at Presentation
Age at first positive SARS-CoV-2 test (median, interquartile range)	66 (53, 77)	73 (61, 83)	61 (47, 74)	65 (52, 76)	66 (54, 76)
Female	42,324 (46.8)	1,564 (45.8)	18,813 (46.2)	19,640 (43.6)	14,849 (43.7)
Race and Ethnicity
American Indian or Alaska Native	68 (0.1)	0 (0.0)	30 (0.1)	36 (0.1)	20 (0.1)
Asian	2,298 (2.5)	106 (3.1)	1,143 (2.8)	1,263 (2.8)	931 (2.7)
Asian Indian	483 (0.5)	21 (0.6)	240 (0.6)	247 (0.6)	168 (0.5)
Black or African American	15,426 (17.1)	514 (15.0)	7,235 (17.8)	6,942 (15.4)	4,733 (13.9)
Hispanic or Latino	26,476 (29.3)	844 (24.7)	13,121 (32.2)	14,247 (31.6)	10,916 (32.2)
Multiracial	320 (0.4)	13 (0.4)	165 (0.4)	140 (0.3)	107 (0.3)
Native Hawaiian or Other Pac Island	132 (0.2)	4 (0.1)	68 (0.2)	72 (0.2)	54 (0.2)
Other	3,085 (3.4)	136 (4.0)	1,538 (3.8)	1,719 (3.8)	1,273 (3.8)
White	40,342 (44.6)	1,689 (49.4)	16,347 (40.1)	19,387 (43.1)	15,021 (44.2)
Unknown	1,746 (1.9)	91 (2.7)	845 (2.1)	983 (2.2)	728 (2.1)
Past or current smoker	20,675 (22.9)	675 (19.8)	8,611 (21.1)	10,195 (22.6)	8,088 (23.8)
van Walraven Comorbidity Index (median, interquartile range)	6 (1, 13)	12 (6, 19)	5 (1, 13)	6 (1,13)	6 (1, 13)
Clinical Course
Admission to the intensive care unit (ICU)	25,814 (28.6)	1,863 (54.5)	13,098 (32.2)	16,636 (36.9)	12,026 (35.4)
Receipt of mechanical ventilation	12,412 (13.7)	1,107 (32.4)	6,561 (16.1)	8,630 (19.2)	6,142 (18.1)
Receipt of vasopressors	12,326 (13.6)	1,222 (35.8)	6,484 (15.9)	8,261 (18.3)	6,029 (17.8)
In-hospital mortality	15,553 (17.2)	1,407 (41.2)	7,464 (18.3)	10,134 (22.5)	7,468 (22.0)

Test Characteristics of qSOFA, Shock Index, NEWS2, and qCSI for Predicting Prognosis

NEWS2 had the highest sensitivity for predicting in-hospital mortality (65.2%, 95% confidence interval [CI] 64.4–65.9%), ICU admission (64.4%, CI 63.9–65.0%), need for mechanical ventilation (69.5%, CI 68.7–70.3%), and need for vasopressors (67.0%, CI 66.2–67.9%) but also had the lowest specificity for all outcomes (Table 3). In contrast, qSOFA had the lowest sensitivity but the highest specificity for predicting in-hospital mortality (97.3%, CI 97.2–97.4%), ICU admission (97.6%, CI 97.5–97.7%), need for mechanical ventilation (97.0%, CI 96.9–97.2%), and need for vasopressors (97.2%, CI 97.1–97.3%). The Shock Index and qCSI had sensitivities and specificities that fell between qSOFA and NEWS2 for all outcomes. All scores exhibited relatively low positive predictive values, ranging from 18.3–41.2% for in-hospital mortality, 32.2–54.5% for ICU admission, 16.1–32.4% for need for mechanical ventilation, and 15.9–35.8% for need for vasopressors. Conversely, all exhibited higher negative predictive values, ranging from 83.7–88.0% for in-hospital mortality, 72.5–79.8% for ICU admission, 87.0–91.7% for need for mechanical ventilation, and 87.2–91.0% for need for vasopressors.

Table 3.

Test characteristics of point-of-care severity scores for predicting prognosis in patients admitted through the emergency department (ED) with COVID-19 using first vital signs collected within 24 hours of ED admission

In-hospital Mortality
	Sn (95% CI)	Sp (95% CI)	PPV (95% CI)	NPV (95% CI)	AUROC (95% CI)
qSOFA ≥ 2	9.1% (8.6–9.5%)	97.3% (97.2–97.4%)	41.2% (39.5–42.8%)	83.7% (83.5–84.0%)	0.53 (0.53–0.53)
Shock Index > 0.7	48.0% (47.2–48.8%)	55.5% (55.2–55.9%)	18.3% (17.9–18.7%)	83.7% (83.4–84.0%)	0.52 (0.51–0.52)
NEWS2 ≥ 5	65.2% (64.4–65.9%)	53.4% (53.0–53.7%)	22.5% (22.1–22.9%)	88.0% (87.7–88.3%)	0.59 (0.59–0.60)
qCSI ≥ 4	48.0% (47.2–48.8%)	64.6% (64.3–64.9%)	22.0% (21.6–22.4%)	85.7% (85.4–86.0%)	0.56 (0.56–0.57)
Intensive Care Unit Admission
	Sn (95% CI)	Sp (95% CI)	PPV (95% CI)	NPV (95% CI)	AUROC (95% CI)
qSOFA ≥ 2	7.2% (6.9–7.5%)	97.6% (97.5–97.7%)	54.5% (52.8–56.2%)	72.5% (72.2–72.8%)	0.52 (0.52–0.53)
Shock Index > 0.7	50.7% (50.1–51.4%)	57.2% (56.8–57.6%)	32.2% (31.7–32.6%)	74.4% (74.0–74.8%)	0.54 (0.54–0.54)
NEWS2 ≥ 5	64.4% (63.9–65.0%)	56.0% (55.6–56.4%)	36.9% (36.5–37.4%)	79.8% (79.4–80.1%)	0.60 (0.60–0.61)
qCSI ≥ 4	46.6% (46.0–47.2%)	66.0% (65.7–66.4%)	35.4% (34.9–35.9%)	75.6% (75.2–75.9%)	0.56 (0.56–0.57)
Need for Mechanical Ventilation
	Sn (95% CI)	Sp (95% CI)	PPV (95% CI)	NPV (95% CI)	AUROC (95% CI)
qSOFA ≥ 2	8.9% (8.4–9.4%)	97.0% (96.9–97.2%)	32.4% (30.8–34.0%)	87.0% (86.8–87.2%)	0.53 (0.53–0.53)
Shock Index > 0.7	52.9% (52.0–53.7%)	56.2% (55.8–56.5%)	16.1% (15.8–16.5%)	88.2% (87.9–88.5%)	0.55 (0.54–0.55)
NEWS2 ≥ 5	69.5% (68.7–70.3%)	53.3% (53.0–53.7%)	19.2% (18.8–19.5%)	91.7% (91.4–91.9%)	0.61 (0.61–0.62)
qCSI ≥ 4	49.5% (48.6–50.4%	64.3% (64.0–64.7%)	18.1% (17.7–18.5%)	88.9% (88.6–89.1%)	0.57 (0.56–0.57)
Need for Vasopressors
	Sn (95% CI)	Sp (95% CI)	PPV (95% CI)	NPV (95% CI)	AUROC (95% CI)
qSOFA ≥ 2	9.9% (9.4–10.5%)	97.2% (97.1–97.3%)	35.8% (34.1–37.4%)	87.2% (87.0–87.5%)	0.54 (0.53–0.54)
Shock Index > 0.7	52.6% (51.7–53.5%)	56.1% (55.8–56.5%)	15.9% (15.6–16.3%)	88.2% (87.9–88.5%)	0.54 (0.54–0.55)
NEWS2 ≥ 5	67.0% (66.2–67.9%)	52.9% (52.5–53.2%)	18.3% (18.0–18.7%)	91.0% (90.8–91.3%)	0.60 (0.60–0.60)
qCSI ≥ 4	48.9% (48.0–49.8%)	64.2% (63.9–64.6%)	17.8% (17.4–18.2%)	88.8% (88.6–89.1%)	0.57 (0.56–0.57)

Sn: sensitivity; Sp: specificity; PPV: positive predictive value; NPV: negative predictive value; 95% CI: 95% confidence interval; highest values in each column appear in bold

When considering a binary cutoff, NEWS2 exhibited the highest AUROC for in-hospital mortality (0.59, CI 0.59–0.60), ICU admission (0.60, CI 0.60–0.61), need for mechanical ventilation (0.61, CI 0.61–0.62), and need for vasopressors (0.60, CI 0.60–0.60), although these AUROCs are not considered robust. AUROCs for all other scores fell in the 0.5 to 0.6 range for all outcomes, which is also considered poor performance. When evaluating the full range of possible scores measured, NEWS2 continued to exhibit the highest AUROC for all outcomes and the performance was improved, but still fell in the 0.60–0.69 range (Figure 1), with other severity scores falling in the 0.5–0.6 range of AUROCs.

Figure 1.

Receiver Operator Characteristic Curves for qSOFA, Shock Index, NEWS2, and qCSI, considering the full range of possible scores measured in the cohort. The curves display the discrimination of each score to predict (a) in-hospital mortality, (b) admission to the intensive care unit (c) need for mechanical ventilation, and (d) need for vasopressors. The area under the receiver operator characteristic curves (AUROC) and 95% confidence intervals are included.

Sensitivity Analyses

Table 4 provides the results of several of our sensitivity analyses. First, there was no difference in score performance for any outcomes after excluding the 8,725 patients (9.7%) who were DNR on admission (data not shown) Next, while our primary analysis utilized scores calculated using the first vital signs captured within 24 hours of ED presentation, we explored whether the scores calculated using the first vital signs captured within 12 hours of admission would have improved performance; however, our results were unchanged when considering first vital signs collected within the first 12 hours as well as within the first 6 hours.

Table 4.

Test characteristics of point-of-care severity scores for predicting prognosis in patients admitted through the emergency department (ED) with COVID-19, stratified by availability of vital signs and DNR status

In-hospital Mortality
	24-hour Score AUROC (95% CI) n=90,376	12-hour Score AUROC (95% CI) n=85,531	6-hour Score AUROC (95% CI) n=68,818	24-hr Score excluding DNR POA AUROC (95% CI) n=81,651	24-hr Score excluding those without GCS AUROC (95% CI) n=22,052
qSOFA ≥ 2	0.53 (0.53–0.53)	0.53 (0.53–0.53)	0.52 (0.52–0.53)	0.53 (0.53–0.53)	0.58 (0.57–0.59)
Shock Index > 0.7	0.52 (0.51–0.52)	0.51 (0.51–0.52	0.51 (0.51–0.52)	0.52 (0.51–0.52)	0.52 (0.51–0.52)
NEWS2 ≥ 5	0.59 (0.59–0.60)	0.59 (0.59–0.59)	0.59 (0.58–0.59)	0.59 (0.59–0.60)	0.61 (0.60–0.62)
qCSI ≥ 4	0.56 (0.56–0.57)	0.56 (0.56–0.57)	0.57 (0.56–0.57)	0.57 (0.56–0.57)	0.56 (0.55–0.57)
Intensive Care Unit Admission
qSOFA ≥ 2	0.52 (0.52–0.53)	0.52 (0.52–0.52)	0.52 (0.52–0.52)	0.52 (0.52–0.53)	0.57 (0.56–0.57)
Shock Index > 0.7	0.54 (0.54–0.54)	0.54 (0.53–0.54)	0.53 (0.53–0.54)	0.54 (0.53–0.54)	0.54 (0.54–0.55)
NEWS2 ≥ 5	0.60 (0.60–0.61)	0.60 (0.60–0.61)	0.60 (0.59–0.60)	0.60 (0.60–0.61)	0.61 (0.60–0.62)
qCSI ≥ 4	0.56 (0.56–0.57)	0.57 (0.56–0.57)	0.57 (0.57–0.58)	0.56 (0.56–0.57)	0.56 (0.55–0.57)
Need for Mechanical Ventilation
qSOFA ≥ 2	0.53 (0.527–0.532)	0.53 (0.53–0.53)	0.52 (0.52–0.52)	0.53 (0.53–0.54)	0.58 (0.57–0.59)
Shock Index > 0.7	0.55 (0.54–0.55)	0.53 (0.53–0.54)	0.53 (0.53–0.54)	0.55 (0.54–0.55)	0.54 (0.53–0.55)
NEWS2 ≥ 5	0.61 (0.61–0.62)	0.61 (0.61–0.62)	0.61 (0.60–0.61)	0.62 (0.62–0.62)	0.61 (0.61–0.62)
qCSI ≥ 4	0.57 (0.56–0.57)	0.57 (0.57–0.58)	0.58 (0.58–0.59)	0.57 (0.57–0.58)	0.57 (0.56–0.58)
Need for Vasopressors
qSOFA ≥ 2	0.54 (0.53–0.54)	0.52 (0.53–0.54)	0.53 (0.53–0.53)	0.54 (0.54–0.54)	0.58 (0.58–0.59)
Shock Index > 0.7	0.54 (0.54–0.55)	0.55 (0.54–0.56)	0.55 (0.54–0.55)	0.54 (0.53–0.55)	0.55 (0.54–0.56)
NEWS2 ≥ 5	0.60 (0.60–0.60)	0.60 (0.60–0.61)	0.60 (0.60–0.61)	0.60 (0.57–0.61)	0.62 (0.61–0.63)
qCSI ≥ 4	0.57 (0.56–0.57)	0.57 (0.57–0.58)	0.58 (0.58–0.59)	0.57 (0.56–0.57)	0.57 (0.56–0.58)

AUROC: Area under the receiver operator characterics curve; 95% CI: 95% confidence interval; DNR POA: Do not resuscitate present on admission; GCS: Glasgow Coma Score; highest values in each column appear in bold

Since both qSOFA and NEWS2 incorporate GCS, we explored baseline characteristics of patients with GCS documented at triage (n=22,052) and found that they were older (median age 68 years v. 65 years, p<0.0001), less likely to be past or current smokers (21.3% v. 23.4%, p<0.0001), had a significantly higher distribution of van Walraven Comorbidity Score on presentation (p<0.0001), but were less likely to be admitted to the ICU during their admission (25.6% v. 29.5%, p<0.0001). There was no significant difference in in-hospital mortality between those who did and did not have a GCS documented at presentation. There was no difference in the score performance when we restricted our analysis cohort to those who had at least one GCS measured within 24 hours of ED admission (n=22,052, Table 4).

We used the observed data and calculated the maximum Youden Index to determine the optimal cutpoint for each score within our cohort and found that the optimal cutpoint for the Shock Index and NEWS2 was quite similar to the a priori cutpoints used in the study. However, both qSOFA and qCSI had lower optimal cutpoints based on the observed data (Supplemental Table 2). There was no improvement in the AUROCs when using the optimal cutpoint calculated from the observed data for all scores and outcomes.

Finally, when evaluating the score performance over the 12-month study period, there was no statistically significant difference in AUROCs over time for most scores and outcomes (Supplemental Figure 1). Some scores performed more poorly over time for specific outcomes (qCSI for mortality; qSOFA, NEWS2, and qCSI for ICU admission; and NEWS2 and qCSI for vasopressors) but while the differences in AUROCs were statistically significantly different, they likely were not clinically significantly different.

DISCUSSION

In this retrospective cohort study of 90,376 patients with community-acquired COVID-19 that were admitted through 157 EDs within a large, geographically diverse national health system, the standard qSOFA threshold was met least frequently at triage, followed by the qCSI, Shock Index, and NEWS2. NEWS2≥5 had the highest sensitivity and AUROC but the lowest specificity for predicting all outcomes, making it a challenging bedside risk stratification score to use for identifying patients at risk of poor clinical outcomes from COVID-19. qSOFA≥2 had the highest specificity for all outcomes, although it was met infrequently (3.8%). When evaluating the performance of the 4 scores using all available cut-points, NEWS2 had the highest AUROC, followed by qSOFA, although none of the scores performed well, with AUROCs ranging in values from 0.526 to 0.656 across all outcomes.

The qSOFA, the Shock Index, NEWS2, and qCSI are already in use in EHR systems, incorporate data commonly collected at triage, and are critical to providing prognostic information for seriously ill patients; qSOFA, the Shock Index, NEWS2, and qCSI are also similar to multipurpose risk prediction tools like the Mortality Probability Model and the APACHE II score^35,36 which are regularly used for risk stratification. Given the rapid progression from organ dysfunction to death that is attributable to sepsis from infections such as COVID-19,³⁷ it is essential for providers to quickly identify patients at increased risk of poor outcomes when they present to the ED with COVID-19. Point-of-care severity scores that use readily available clinical data that are not subject to laboratory lag may allow us to identify patients at greatest risk of poor outcomes from COVID-19 immediately upon ED presentation and triage,³⁸ particularly since sepsis scores have been more broadly applied to the care of patients with community-acquired pneumonia, hypercapnic respiratory failure, pulmonary embolism, chronic obstructive pulmonary disorder, and heart failure.^3–10,12,13

We found that NEWS2≥5 had the highest sensitivity of evaluated scores for predicting poor outcomes while it had the lowest specificity. qSOFA exhibited the lowest sensitivity and the highest specificity for all outcomes measured in our study. Our results are consistent with prior studies among COVID-19 patients which demonstrate a high sensitivity and low specificity for NEWS2 as well as a low sensitivity and high specificity for qSOFA.^{39–42,43 44,45} In addition, we observed similar AUROC patterns to the published literature for both NEWS2 and qSOFA.^39,40 Other studies have demonstrated a much higher degree of predictive performance for NEWS2, with AUROCs in the moderate to excellent range for ICU admission and mortality,^46,47 perhaps due to more complete capture of the data required for score calculation, differences in the time interval over which the data were collected, and potential differences in performance within smaller patient cohorts. The Shock Index has been evaluated in sepsis and other conditions.^48,49,26 Studies show that an abnormal Shock Index is associated with increased odds of future vasopressor receipt, increased prevalence of hyperlactatemia, and increased risk of poor outcomes in the ICU.^17,18 The Shock Index did not perform favorably in our multi-center population of COVID-19 patients, a finding which is consistent with other studies of COVID-19 patients.^50,51

Several COVID-19 severity scores use data that may be subject to clinical or laboratory lag, including the BAS²IC Score⁵², 4C Mortality Score⁵³, the Brescia-COVID Respiratory Severity Scale (BCRSS)⁵⁴ and others,^55–57 with several of the models failing to perform as effectively when deployed outside of the derivation population.⁵⁸ We evaluated one COVID-specific score, the qCSI, which incorporates respiratory function, and found that it performed better than qSOFA and the Shock Index but still achieved poor performance to predict all outcomes when using the cutoff of qCSI≥4. Other studies have demonstrated improved performance of the qCSI, with AUROCs in the 0.7 range,^59,60 an improvement that may be due to more complete capture of the data required for score calculation.

None of the scores we compared performed well at predicting poor outcomes among patients hospitalized with COVID-19, which may be due to the score components themselves or issues of data availability. We also found that there was no improvement in score performance if we considered vital signs captured either within 6 or 12 hours versus 24 hours of ED admission. Our results were unaltered in our sensitivity analyses excluding patients who were DNR or who did not have a GCS score documented. Even though the study covered a one-year period early in the pandemic when new treatments and therapies were being introduced, there was no clinically significant difference in how the scores performed over time. There are some potential explanations for why existing scores may not be useful to predict outcomes for patients with COVID-19. For example, qSOFA may have suffered from poor performance because it incorporates hemodynamic instability and altered mentation, symptoms which are not as commonly reported among COVID-19 patients at ED presentation.^61,62 Scores like qSOFA and NEWS2 rely on a classic sepsis presentation, which may hinder their performance for predicting poor outcomes from COVID-19.^40,47,63 The Shock Index also prioritizes vital signs that may not be as strongly associated with poor outcomes in COVID-19 patients. Even the qCSI, the bedside COVID-19-specific score we applied in our cohort, did not perform well, perhaps due to issues of data availability around oxygen flow rate. Finally, the need to develop COVID-19-specific severity scores is highlighted by a recent study which suggests that clinical gestalt predictions were non-inferior to LOW-HARM, qSOFA, MSL-COVID-19, NUTRI-CoV, and NEWS2 risk scores for predicting COVID-19 mortality.⁶⁴

One of the major challenges of conducting EHR-based studies is the bias introduced due to inconsistent documentation of clinical data. Because researchers can only utilize data that are directly charted, assumptions must be made regarding missingness of data. In our study, we encountered considerable missingness in GCS data, which could have affected the predictive ability of qSOFA and NEWS2 in our cohort. Even though nearly 29% of our cohort were admitted to the ICU and nearly 14% received mechanical ventilation and vasopressors, only 4% of our cohort met qSOFA criteria. This is likely due to the missingness of GCS data, which was only present in 24% of patients and is one of only three qSOFA components. While imputation can be used to address missing data, we could not reliably perform imputation of GCS due to its high proportion of missingness; therefore, we assumed missing was normal. Although patients who had at least one GCS were older with a greater degree of comorbidities, there were few differences in patient outcomes. Previous studies have not seen improvement in qSOFA’s ability to predict mortality after GCS imputation in patients with sepsis.⁶⁵ Since manually collected variables like GCS are less reliably documented in the EHR, there may be limitations in their use for risk scores. Another potential limitation is that this study was conducted between March 2020 and February 2021, before vaccines and oral antiviral agents such as nirmatrelvir/ritonavir and molnupiravir were available and prior to the Delta and Omicron waves of the pandemic. Performance of the scores to predict poor outcomes may be modified by trends in viral transmission and vaccination as well as the evolving clinical management of COVID-19.

In summary, qSOFA, the Shock Index, NEWS2, and qCSI exhibited poor performance to identify patients that experienced in-hospital mortality as well as ICU admission, mechanical ventilation, and receipt of vasopressors. Score performance was consistently poor or even worse over time for some outcomes, further highlighting the potential lack of utility of existing scores for risk stratification in COVID-19 patients. Score performance might be improved through increasing completeness of key data, modification of models to gather data over a longer time period, novel metrics, or some combination of these to better represent the dynamic and rapidly changing course of COVID-19 illness.