Predicting intensive care unit admission and death for COVID-19 patients in the emergency department using early warning scores

Abstract

Aims

To identify the most accurate early warning score (EWS) for predicting an adverse outcome in COVID-19 patients admitted to the emergency department (ED).

Methods

In adult consecutive patients admitted (March 1-April 15, 2020) to the ED of a major referral centre for COVID-19, we retrospectively calculated NEWS, NEWS2, NEWS-C, MEWS, qSOFA, and REMS from physiological variables measured on arrival. Sensitivity, specificity, positive (PPV) and negative predictive value (NPV), and the area under the receiver operating characteristic (AUROC) curve of each EWS for predicting admission to the intensive care unit (ICU) and death at 48 h and 7 days were calculated.

Results

We included 334 patients (119 [35.6%] females, median age 66 [54-78] years). At 7 days, the rates of ICU admission and death were 56/334 (17%) and 26/334 (7.8%), respectively. NEWS was the most accurate predictor of ICU admission within 7 days (AUROC 0.783 [95% CI, 0.735-0.826]; sensitivity 71.4 [57.8-82.7]%; NPV 93.1 [89.8-95.3]%), while REMS was the most accurate predictor of death within 7 days (AUROC 0.823 [0.778–0.863]; sensitivity 96.1 [80.4-99.9]%; NPV 99.4[96.2–99.9]%). Similar results were observed for ICU admission and death at 48 h. NEWS and REMS were as accurate as the triage system used in our ED. MEWS and qSOFA had the lowest overall accuracy for both outcomes.

Conclusion

In our single-centre cohort of COVID-19 patients, NEWS and REMS measured on ED arrival were the most sensitive predictors of 7-day ICU admission or death. EWS could be useful to identify patients with low risk of clinical deterioration.

Introduction

After the first cases were identified in Wuhan (China) in December 2019, the novel coronavirus SARS-CoV-2 has caused a pandemic of respiratory illness named COVID-19.¹ To date, more than 29 million cases have been reported worldwide, including more than 900,000 deaths.² Typical COVID-19 patients present with fever, myalgia, fatigue, and dry cough.3, 4 Severe cases progress to severe dyspnoea and hypoxemia within one week after the onset of symptoms.3, 4, 5 In hospitalized COVID-19 patients, the prevalence of hypoxemic respiratory failure is around 20%, and more than 25% of them may require intensive care treatment.⁶

The increasing number of COVID-19 cases has challenged healthcare systems worldwide. In this context of overwhelming demand for medical assessment and triage in emergency departments (EDs), early warning scores (EWS) may be useful. EWS are physiological scoring systems based on a rapid and quantitative assessment of changes in vital signs,⁷ and were initially developed to identify and track hospitalised patients at risk of deterioration outside critical care areas in order to ensure early stabilisation and transfer to intensive care unit (ICU) when appropriate, and to prevent avoidable cardiac arrests.8, 9 However, these scores have also subsequently been investigated as a tool to predict critical illness in ED patients,¹⁰ and some, as the Rapid Emergency Medicine Score (REMS),¹¹ have specifically been developed for this purpose. In studies performed on the general ED population, the National Early Warning Score (NEWS)¹² was the most accurate in predicting both in-hospital mortality and ICU admission.¹⁰ A recent document from the Swiss Society of Intensive Care Medicine¹³ suggested using a score greater than 6 of a modified version of NEWS (hereafter referred as to NEWS-C) for deciding admission to ICU of COVID-19 patients.¹⁴ However, this recommendation is based on previous evidence on non-COVID-19 patients,15, 16 and it has not been validated so far. At present, it is not clear which EWS is the most accurate for triaging COVID-19 patients in the ED.

The aim of the present study was to compare the ability of EWS assessed on ED arrival to identify adult COVID-19 patients at risk for ICU admission or death within seven days. The EWS assessed in this study included Modified EWS (MEWS),¹⁷ NEWS, NEWS2,¹⁸ NEWS-C, Quick Sepsis Related Organ Failure Assessment (qSOFA),¹⁹ and REMS. Moreover, in order to measure the added value of EWS to the conventional ED triage process, we also compared the accuracy of EWS with that of the Regional Triage System for Infectious Diseases in use in our hospital for COVID-19 patients.

Methods

Study design and setting

This is a retrospective observational study (clinicaltrials.gov NCT04371562) conducted in the ED of the Fondazione Policlinico Universitario Agostino Gemelli, in Rome. Policlinico Gemelli is the largest teaching hospital in Rome and a referral centre for COVID-19 in central Italy. Our ED has a catchment area of 1,8 M inhabitants, with an average of 80k access/year.

The Regional Triage System for Infectious Diseases in our ED is based on a protocol issued by the Health Service of Lazio Region, where our hospital is located. The protocol, summarised on ESM Tables 1a-b, follows an ABCDE scheme (airway, breathing, circulation, disability, exposure) with additional variables such as age, blood glucose level, plus additional symptoms and signs. The triage system assigns a priority value from 1 to 5, in descending order of severity. Triage is performed by a nurse certified in a full time 7-day course followed by a 6-month supervised training.

The criteria for ICU admission of COVID-19 patients in our hospital during the study period included circulatory shock requiring vasopressors and the need for positive pressure respiratory support, including non-invasive ventilation. High-flow oxygen treatment is provided in general wards. ICU admission criteria were consistent throughout the study period. At present, no EWS is in use in our hospital.

Eligibility criteria

From March 1 to April 15, 2020 we included all adult (>18y) patients assessed in our ED for suspected COVID-19 whose diagnosis was confirmed with real-time reverse-transcriptase–polymerase-chain-reaction assay of nasal and pharyngeal swab specimens. Testing for COVID-19 was conducted according to the World Health Organization (WHO) interim guidance.²⁰ We excluded pregnant women and patients who were on circulatory shock and positive pressure support on arrival to ED. Patients with a do-not-resuscitate/do-not-intubate (DNR/DNI) order in place were also excluded.

Data collection and measurements

Two authors blinded to patients’ outcome retrieved clinical information from the electronic medical records (EMR) of the ED database. Extracted data included age, sex, clinical history and presentation, temperature, heart rate (HR), respiratory rate (RR), blood pressure (BP), Glasgow Coma Scale (GCS) score, oxygen therapy, and peripheral oxygen saturation (SpO₂). All clinical signs, including SpO₂, were measured upon ED arrival. EWS were calculated for each patient using Microsoft Excel 2010 for Windows (Microsoft Inc., Redmond, WA, USA) from available physiological measurements. Variables included in each of these scores are listed on ESM Table 2. For NEWS2 calculation, patients were considered at risk of type 2 respiratory failure (T2RF) if they had a confirmed history of chronic obstructive pulmonary disease.

Outcomes

The primary study outcomes were death and ICU admission within seven days from ED arrival. Secondary outcomes were death and ICU admission within 48 h from ED arrival. Outcomes were retrospectively assessed by reviewing EMRs of the hospital medical database.

Statistical analysis

Continuous variables were reported as median [interquartile range, IQR] and were compared with univariate analysis by Mann-Whitney U test. Categorical variables were reported as absolute number (percentage) and are compared by Chi-square test (with Fisher’s test if appropriate).

Receiver operating characteristic (ROC) curve analysis was used to estimate the performance of the evaluated scores in predicting the study outcomes. For each score threshold value, the following was calculated: sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), positive likelihood ratio (+LR), and negative likelihood ratio (-LR). The Youden index was used to estimate the optimal thresholds for sensitivity and specificity. The comparison between the areas under the ROC curve (AUROCs) was made according to the DeLong method.²¹ A p value ≤ 0.05 was regarded as significant.

To check the robustness of our analysis we imputed data in patients with less than three missing parameters using a multiple imputation approach.²² The results of imputation were added to the original study cohort with the complete dataset of vital parameters and EWS were recalculated. The resulting AUROCs were compared to the AUROC obtained in the study population by using the z-statistics (2 sided). Further data on imputation are available in Appendix 1.

Data were analysed using IBM SPSS statistics for Windows, Version 25 (IBM Corp. Armonk, NY, USA) and MedCalc Statistical Software version 19.2.1 (MedCalc Software Ltd, Ostend, Belgium; https://www.medcalc.org; 2020).

Ethical approval

This study has been approved by the local Ethics Committee (authorization #001705520) and has been performed in accordance with the ethical standards established in the 1964 Declaration of Helsinki and its later amendments. All patients gave their informed consent for clinical data analysis.

Results

Characteristics of the study subjects

During the study period, 3955 patients were seen in the ED, of whom 1801 were evaluated for suspect COVID-19 and 512 were found positive to the COVID-19 test. Of these, 336 had sufficient data on EMR for calculating EWS. Two patients with a DNR/DNI order in place were excluded, leaving 334 patients for final analysis (Fig. 1 ). Patients’ median [IQR] age was 66 [54 – 78] years (range 20 -98 years) and 215 (64.4%) were male (Table 1 ).

Fig. 1.

Flow-chart of study selection. DNR/DNI: Do-not-resuscitate/Do-not-intubate order. EMR: Electronic Medical Record; EWS: Early Warning Score.

Table 1.

Characteristics of the study population and distribution of score values according to patient outcome. Data for ICU admission and death are referred to day 7. Data are presented as median (25th – 75th percentile) or number (percentage).

	All Patients (n° 334)	Admitted to ICU		P	Survived (308 [90.9%])	Dead (26 [7.8%])	p
		No (278 [83.2%])	Yes (56 [16.8%])
Age, years	66 [54–78]	63 [51–77]	71.5 [64–78]	0.002	64 [53–77]	79.5 [74–86]	<0.001
Sex, male	215 (64.4%)	173 (62.2%)	42 (75.0%)	0.069	196 (63.6%)	19 (73.1%)	0.334
Heart rate, beats · min-1	90 [79–105]	88 [78–103]	98 [86–105]	0.058	90 [79–104]	95 [81–113]	0.265
Respiratory rate, breaths · min-1	18 [16–21]	18 [16–21]	20 [17–21]	0.027	18 [16–21]	20 [17–23]	0.068
Glasgow Coma Score	15 [15–15]	15 [15–15]	15 [15–15]	0.793	15 [15–15]	15 [15–15]	0.002
Blood pressure, mmHg
Systolic	127 [114–140]	130 [119–138]	129 [117–140]	0.838	127 [114–144]	128 [115–139]	0.728
Diastolic	80 [70–89]	80 [70–89]	77 [70–84]	0.064	80 [70–88]	73 [64–89]	0.089
Mean	96 [86–103]	96 [86–104]	93 [84–100]	0.351	94 [85–103]	92 [84–101]	0.268
SpO2, %	95 [92–97]	95 [93–97]	88.4 [81–92]	<0.001	95 [92–97]	89 [79–92]	<0.001
Body Temperature, °C	37.0 [36.7–37.5]	37.0 [36.7–37.5]	37.1 [36.9–37.4]	0.213	36.5 [36.0–37.2]	37.1 [37.0–37.5]	0.614
Comorbidities
Hypertension	140 (41.9%)	107 (38.5%)	33 (58.9%)	0.005	123 (39.9%)	17 (65.4%)	0.012
CAD, CHF	81 (24.3%)	57 (20.5%)	24 (42.9%)	<0.001	70 (22.7%)	11 (42.3%)	0.025
Diabetes	36 (10.7%)	30 (10.8%)	6 (10.7%)	1.000	34 (11.0%)	2 (7.7%)	0.597
COPD	19 (5.7%)	14 (5.0%)	5 (8.9%)	0.251	16 (5.2%)	3 (11.5%)	0.180
Dementia	12 (3.6%)	12 (4.3%)	0	0.113	9 (2.9%)	3 (11.5%)	0.023
Malignancy	14 (4.2%)	12 (4.3%)	2 (3.6%)	0.800	12 (3.9%)	2 (7.7%)	0.354
Chronic Kidney disease	11 (3.3%)	9 (3.2%)	2 (3.6%)	0.898	8 (2.6%)	3 (11.5%)	0.046
Severe Obesity	6 (1.8%)	3 (1.1%)	3 (5.4%)	0.028	6 (1.9%)	0	1.000
Living in an institution	29 (8.6%)	27 (9.6%)	2 (3.6%)	0.140	26 (8.4%)	3 (11.5%)	0.483
TRIAGE code	3 [3–4]	4 [3–4]	2 [1–3]	<0.001	3 [3–4]	2 [1–3]	<0.001
Early Warning Scores
MEWS	5 [4–7]	5 [4–7]	6 [5–7]	0.004	5 [4–7]	6 [4–8]	0.136
NEWS	3 [1–5]	2 [1–4]	6 [4–8]	<0.001	3 [1–5]	6 [3–9]	<0.001
NEWS2	2 [1–5]	2 [1–4]	6 [3–8]	<0.001	2 [1–5]	6 [2–9]	<0.001
NEWS – C	6 [4–9]	6 [4–8]	9 [6–11]	<0.001	6 [4–8]	10 [6–12]	<0.001
qSOFA	0 [0–0]	0 [0–0]	0 [0–1]	0.066	0 [0–0]	1 [0–1]	<0.001
REMS	5 [3–7]	5 [3–6]	7 [5–9]	<0.001	5 [3–7]	8 [6–11]	<0.001
Outcome
Admitted to ICU at ≤7 days	56 (16.7%)	-	-		38 (12.3%)	18 (69.2%)	<0.001
Deceased at ≤7 Days	26 (7.8%)	8 (2.9%)	18 (32.1%)	<0.001	-	-

Abbreviations: CAD Coronary Artery Disease; CHF Congestive Heart Failure; COPD Chronic Obstructive Pulmonary Disease; ICU Intensive Care Unit. TRIAGE (Triage code assigned in ED); MEWS Modified Early Warning Score; NEWS National Early Warning Score; NEWS-C: modified NEWS; qSOFA quick Sequential Organ Failure Assessment; REMS Rapid Emergency Medicine Score.

Thirty-eight (11%) COVID-19 patients were discharged from ED and quarantined. These patients were contacted daily to assess their clinical condition. The remaining 296 patients were admitted to hospital, of whom 56 (17% of the total) were admitted to ICU. Fifty patients were admitted to ICU within 48 hours, while the remaining six patients were admitted on days 3-7 after ED arrival. Patients who needed ICU admission were older, had lower SpO₂ and higher respiratory rate on ED arrival, and more frequently reported a history of hypertension, obesity, and chronic heart failure or coronary artery disease (Table 1).

Overall, 26 patients (7.8%) deceased within seven days after ED arrival, among whom 11 (3.3%) deceased by 48 hours. These patients were significantly older, had a higher respiratory rate, and a lower SpO₂ on ED arrival. As for ICU admission, in patients who died a history of hypertension and chronic heart failure or coronary artery disease was more commonly reported (Table 1).

Accuracy of warning scores

Concerning ICU admission, among all the EWS we assessed, NEWS had the highest overall accuracy both at 48 h and at 7 days (AUROC 0.802 [0.756 – 0.844] and 0.783 [0.735 – 0.826], respectively), and the highest NPV, both at 48 h and at 7 days (93.4 [90.6 – 95.4]% for NEWS ≤ 5 and 93.1 [89.8 – 95.3]% for NEWS ≤ 4, respectively) along with NEWS2 (NPV 93.7 [90.6 – 95.8] and 92.4 [89.3 – 94.7] at 48 h and 7 days, respectively). The accuracy was lower, although not significantly, than that of the triage system of our hospital (Table 2 ).

Table 2.

Sensitivity, specificity, negative and positive predictive value, and positive and negative likelihood ratio of the hospital triage score and of the EWS for predicting ICU admission within 48 hours and 7 days from ED admission. Optimal cut off values for sensitivity and specificity were chosen according to Youden index. Values of the EWS with the largest AUROC are in bold.

	AUROC	cut-off	Sensitivity (%)	Specificity (%)	+LR	-LR	PPV	NPV
Admission to ICU within 48 hours
TRIAGE	0.839 [0.795 – 0.877]	≤ 2	66.0 [51.2 – 78.8]	89.8 [85.7 – 93.1]	6.5 [4.3 – 9.6]	0.4 [0.3 – 0.6]	53.2 [43.3 – 62.9]	93.7 [91.1 – 95.7]
MEWS	0.652 [0.598 – 0.703]	> 5	70.0 [55.4 – 82.1]	64.8 [58.9 – 70.3]	2.0 [1.6 – 2.5]	0.5 [0.3 – 0.7]	25.9 [21.6 – 30.8]	92.5 [88.8 – 95.0]
NEWS	0.802 [0.756 – 0.844]	> 5	66.0 [51.2 – 78.8]	84.9 [80.2 – 88.8]	4.4 [3.1 – 6.1]	0.4 [0.3 – 0.6]	43.4 [35.3 – 51.9]	93.4 [90.6 – 95.4]
NEWS2	0.780 [0.731 – 0.823]	> 4	70.0 [55.4 – 82.1]	78.5 [73.3 – 83.2]	3.3 [2.4 – 4.3]	0.4 [0.2 – 0.6]	36.5 [30.1 – 43.3]	93.7 [90.6 – 95.8]
NEWS-C	0.771 [0.701 – 0.831]	> 8	62.9 [44.9 – 78.5]	82.7 [75.4 – 88.6]	3.6 [2.3 – 5.7]	0.5 [0.3 – 0.7]	47.8 [37.0 – 58.8]	89.8 [85.1 – 93.2]
qSOFA	0.566 [0.511 – 0.620]	> 0	34.0 [21.2 – 48.8]	79.6 [74.4 – 84.1]	1.7 [1.1 – 2.6]	0.8 [0.7 – 1.0]	22.7 [15.8 – 31.5]	87.3 [84.8 – 89.4]
REMS	0.742 [0.692 – 0.788]	> 6	60.0 [45.2 – 73.6]	75.0 [69.5 – 79.9]	2.4 [1.8 – 3.2]	0.5 [0.4 – 0.8]	29.7 [23.8 – 36.4]	91.4 [88.3 – 93.8]
Admission to ICU within 7 days
TRIAGE	0.818 [0.773 – 0.858]	≤ 2	58.9 [45.0 – 71.9]	89.6 [85.4 – 92.9]	5.6 [3.8 – 8.5]	0.5 [0.3 – 0.6]	53.2 [43.1 – 63.1]	91.5 [88.8 – 93.7]
MEWS	0.618 [0.564 – 0.671]	> 5	66.1 [52.2 – 78.2]	64.8 [58.8 – 70.4]	1.9 [1.5 – 2.4]	0.5 [0.4 – 0.8]	27.4 [22.8 – 32.6]	90.5 [86.7 – 93.2]
NEWS	0.783 [0.735 – 0.826]	> 4	71.4 [57.8 – 82.7]	77.3 [72.1 – 82.1]	3.2 [2.4 – 4.1]	0.4 [0.2 – 0.6]	38.8 [32.6 – 45.5]	93.1 [89.8 – 95.3]
NEWS2	0.762 [0.714 – 0.807]	> 4	67.8 [54.0 – 79.7]	79.1 [73.9 – 83.8]	3.2 [2.4 – 4.4]	0.4 [0.3 – 0.6]	39.6 [32.9 – 46.7]	92.4 [89.3 – 94.7]
NEWS-C	0.755 [0.684 – 0.817]	> 8	60.1 [51.9 – 81.9]	84.2 [76.9 – 90.0]	3.9 [2.4 – 6.1]	0.5 [0.3 – 0.7]	54.3 [42.8 – 65.4]	87.5 [82.6 – 91.2]
qSOFA	0.557 [0.501 – 0.611]	> 0	32.1 [20.3 – 46.0]	79.5 [74.3 – 84.1]	1.6 [1.0 – 2.4]	0.8 [0.7 – 1.0]	24.0 [16.8 – 33.0]	85.3 [82.8 – 87.5]
REMS	0.735 [0.685 – 0.782]	> 6	58.9 [45.0 – 71.9]	75.5 [70.0 – 80.5]	2.4 [1.8 – 3.3]	0.5 [0.4 – 0.7]	32.7 [26.4 – 39.6]	90.1 [86.9 – 92.6]

Abbreviations: TRIAGE (Triage code assigned in ED); MEWS Modified Early Warning Score; NEWS National Early Warning Score; NEWS-C: modi0fied NEWS; qSOFA quick Sequential Organ Failure Assessment; REMS Rapid Emergency Medicine Score; +LR Positive Likelihood Ratio; −LR Negative Likelihood Ratio; PPV Positive Predictive Value; NPV Negative Predictive Value.

Concerning in-hospital death, REMS showed the highest overall accuracy both at 48 h and 7 days (AUROC 0.882 [0.843 – 0.915] and 0.823 [0.778 – 0.863], respectively). These AUROCs were larger, although not significantly, than those of the triage system of our hospital at both time points. REMS had also the highest NPV for in-hospital death (99.6 [97.3 – 99.9]% for REMS > 6 at 48 h and 99.4 [96.2 – 99.9]% for REMS > 5 at 7 days; Table 3 ).

Table 3.

Sensitivity, specificity, negative and positive predictive value, and positive and negative likelihood ratio of the hospital triage score and of the EWS for predicting death within 48 hours and 7 days from ED admission. Optimal cut off values for sensitivity and specificity were chosen according to Youden index. Values of the EWS with the largest AUROC are in bold.

	AUROC	cut-off	Sensitivity (%)	Specificity (%)	+LR	-LR	PPV	NPV
Death within 48 hours
TRIAGE	0.849 [0.806 – 0.886]	≤ 2	81.8 [48.2 – 97.7]	83.6 [79.1 – 87.5]	5.0 [4.1 – 15.7]	0.5 [0.3 – 0.9]	14.5 [10.5 – 19.8]	99.3 [97.5 – 99.8]
MEWS	0.630 [0.576 – 0.682]	> 5	63.6 [30.8 – 89.1]	60.4 [54.8 – 65.7]	1.6 [1.0 – 2.6]	0.6 [0.3 – 1.3]	5.2 [3.3 – 8.0]	98.0 [95.7 – 99.1]
NEWS	0.829 [0.742 – 0.832]	>4	81.8 [48.2 – 97.7]	70.9 [65.6 – 75.8]	2.8 [2.0 – 3.9]	0.3 [0.1 – 0.9]	8.7 [6.5 – 11.7]	99.1 [97.0 – 99.8]
NEWS2	0.753 [0.703 – 0.798]	> 4	72.7 [39.0 – 94.0]	72.7 [67.6 – 77.5]	2.7 [1.8 – 4.0]	0.4 [0.1 – 1.0]	8.3 [5.7 – 12.0]	98.7 [96.8 – 99.5]
NEWS-C	0.773 [0.703 – 0.833]	> 9	70.0 [34.8 – 93.3]	82.3 [75.6 – 87.8]	4.0 [2.3 – 6.7]	0.4 [0.1 – 0.9]	19.4 [12.5 – 28.9]	97.8 [94.6 – 99.1]
qSOFA	0.808 [0.762 – 0.849]	> 0	81.8 [48.2 – 97.7]	79.6 [74.8 – 83.8]	4.0 [2.8 – 5.7]	0.2 [0.1 – 0.8]	12.0 [8.8 – 16.2]	99.2 [97.3 – 99.8]
REMS	0.882 [0.843 – 0.915]	> 6	90.9 [58.7 – 99.8]	71.8 [66.6 – 76.7]	3.2 [2.5 – 4.2]	0.1 [0.1 – 0.8]	9.9 [7.8 – 12.4]	99.6 [97.3 – 99.9]
Death within 7 days
TRIAGE	0.819 [0.773 – 0.858]	≤ 2	61.5 [40.6 – 79.8]	85.1 [80.6 – 88.9]	4.1 [2.8 – 6.2]	0.5 [0.3 – 0.7]	25.8 [18.8 – 34.3]	94.4 [92.8 – 95.7]
MEWS	0.586 [0.531 – 0.640]	> 5	57.7 [36.9 – 76.6]	61.0 [55.3 – 66.5]	1.5 [1.0 – 2.1]	0.7 [0.4 – 1.1]	11.1 [8.0 – 15.2]	94.5 [91.5 – 96.4]
NEWS	0.768 [0.719 – 0.812]	> 5	65.4 [44.3 – 82.8]	80.8 [76.0 – 85.1]	3.4 [2.4 – 4.9]	0.4 [0.3 – 0.7]	22.4 [16.7 – 29.3]	96.5 [94.2 – 97.9]
NEWS2	0.727 [0.676 – 0.774]	> 5	61.5 [40.6 – 79.8]	81.5 [76.7 – 85.7]	3.3 [2.3 – 4.9]	0.5 [0.3 – 0.8]	17.7 [13.3 – 23.2]	96.2 [93.9 – 97.6]
NEWS-C	0.767 [0.697 – 0.828]	> 8	65.2 [42.7 – 83.6]	79.5 [72.1 – 85.6]	3.1 [2.1 – 4.9]	0.4 [0.2 – 0.8]	32.6 [23.9 – 42.7]	93.7 [89.5 – 96.4]
qSOFA	0.673 [0.620 – 0.723]	> 0	53.8 [33.4 – 73.3]	80.2 [75.3 – 84.5]	2.7 [1.8 – 4.1]	0.6 [0.4 – 0.9]	18.7 [13.1 – 25.9]	95.4 [93.1 – 96.9]
REMS	0.823 [0.778 – 0.863]	> 5	96.1 [80.4 – 99.9]	56.8 [51.1 – 62.4]	2.3 [1.9 – 2.6]	0.1 [0.0 – 0.5]	15.8 [13.9 – 17.9]	99.4 [96.2 – 99.9]

Abbreviations: TRIAGE (Triage code assigned in ED); MEWS Modified Early Warning Score; NEWS National Early Warning Score; NEWS-C: modified NEWS; qSOFA quick Sequential Organ Failure Assessment; REMS Rapid Emergency Medicine Score; +LR Positive Likelihood Ratio; −LR Negative Likelihood Ratio; PPV Positive Predictive Value; NPV Negative Predictive Value.

For both outcomes, the overall accuracy of the NEWS-C was consistently lower than that of NEWS in our population. Among all EWS, MEWS and qSOFA showed the lowest overall accuracy. Their AUROCs were significantly lower than that of the triage system of our hospital, except for prediction of death at 48 h for qSOFA (Table 3).

ESM figures 1 and 2 include the ROC curves for prediction of ICU admission and death, respectively, along with a statistical comparison of the AUROCs of the scores included in the study.

Among the 176 patients excluded because of missing parameters, 78 (44.3%) had less than three missing parameters and were eligible for imputation. Their available parameters are shown in Appendix 1, Table A1. Among these 78 patients, six deaths (7.7%) and nine ICU admissions (11.5%) occurred within 7 days.

The EWS AUROCs in the extended population including 78 patients with imputed vital parameters (total 412 patients) were not significantly different from those of the 334 patients with complete original data. (Appendix 1, Tables A2 and A3).

Discussion

Our study showed that in COVID-19 patients NEWS and REMS assessed on arrival at the emergency department were the most accurate scores for predicting the risk of ICU admission and death, respectively, both at 48 h and at 7 days. Moreover, the accuracy of these scores was comparable with that of the comprehensive triage score used in our department.

NEWS is among the most accurate tools for predicting patient deterioration outside ICU.¹⁰ In a comparative study including 198,755 observation sets collected from 35,585 consecutive acute medical admissions in UK,²³ NEWS was more strongly associated with ICU admission or death compared to 33 other scoring systems (not including REMS). Moreover, in the specific setting of acute care patients admitted for suspected infection in ED or hospital wards, NEWS has proven to be more accurate for predicting death or ICU transfer than other tools that were purposely designed to assess infected patients, such as qSOFA and systemic inflammatory response syndrome (SIRS) criteria.²⁴

In 2017, NEWS has been updated to NEWS2 by adding a new SpO₂ scoring scale for patients with type 2 respiratory failure (T2RF),²⁵ even if the added value of this modification in improving discrimination of adverse outcomes in patients with documented T2RF has not been demonstrated yet.²⁶ NEWS2 is recommended by the Royal College of Physicians for use in COVID-19 patients.²⁷

In our study, NEWS2 did not perform better than NEWS in predicting ICU admission or death. However, the prevalence of T2RF patients in our population was very low (19/334 patients; 5.7%). This is in accordance with data from other studies showing that hypercapnia is uncommon in COVID-19 patients, even in those admitted to ICU,²⁸ while hypoxia from early pulmonary involvement is the most common presentation feature.4, 29

NEWS-C, another modified version of NEWS, has recently been suggested for assessment of COVID-19 patients.¹⁴ Age ≥65 years was added to NEWS-C as an independent risk factor based on recent reports in patients with influenza or other non-COVID infectious diseases.15, 16 Indeed, older age is associated with increased mortality in COVID-19 patients.³⁰ However, NEWS-C did not perform better than NEWS in predicting ICU admission or mortality in our study. This result can be explained considering that our population was relatively old. In fact, the median age of survivors vs. non-survivors was 64 years vs. 79.5 years in our COVID-19 population, as compared with 51 vs.68 years respectively in a recent report from China.³⁰ These data may reflect differences in demographics (median population age is 34.4 years in China vs. 42.5 years in Europe and 47.3 years in Italy³¹) and in policies for ICU admission and triage. When weighting for age, REMS may be particularly appropriate, since – unlike other EWS – it stratifies age (five groups from <45 to >74 years; see also ESM Table 2). In our COVID-19 population, REMS was the most accurate score for predicting mortality. Among the scoring systems we evaluated, REMS was the only one specifically developed to predict mortality in ED patients.11, 32

The qSOFA has been proposed as a prompt to consider possible sepsis outside the ICU. Criteria to calculate qSOFA include altered mental status (GCS < 15), tachypnoea (RR ≥ 22), and hypotension (systolic BP ≤ 100). In the original validation study, a qSOFA score of 2 or higher was associated with a significantly increased risk of hospital mortality.¹⁹ However, the accuracy of qSOFA was low in our study, confirming results from previous studies conducted in non-COVID potentially septic patients.24, 33, 34 Notably, most patients in our cohort did not have any of the three qSOFA criteria, and none of them had a qSOFA score of 2 or higher. Among the three qSOFA criteria, the prevalence of hypotension and altered mental status on ED admission in our patient population was particularly low (6.6% and 3.3%; data not shown). This is in line with recent reports showing that extrapulmonary organ dysfunction occurs later in the clinical course of COVID-19.³⁵

When compared with other scoring systems, including our institutional triage score, NEWS and REMS had the highest sensitivity and NPV for adverse outcomes. In other words, they performed particularly well in identifying COVID-19 patients who did not require ICU admission or were at low risk of death, respectively. When triaging patients in ED, having a high NPV (i.e., a low rate of false negative predictions) for adverse outcomes is paramount in order to avoid inappropriately admitting a patient at risk of deterioration to a non-critical care area. However, because of the relatively low incidence of adverse outcomes in our population, all the EWS we evaluated had a low PPV (i.e., a low rate of true positive predictions), which implies a risk of overtriage when these scores are used to decide admission to a critical care area. Although consensus suggests that overtriage is generally preferable to undertriage,³⁶ this may not be the case when meeting the demands of healthcare resources is difficult. The prognostic indications from physiological scores should be interpreted within a more comprehensive clinical approach including patient’s comorbidities, functional status, and preferences.

In our study we assessed EWS only once, on patient arrival to the ED. Since EWS have been designed as track-and-trigger systems to detect deterioration of vital signs over time, their sequential measurement in COVID-19 patients admitted to non-monitored wards may represent the logical continuation of their use in ED. The value of EWS in this context deserves further investigation.

Some limitations of our study should be acknowledged. First, this was a single-centre study conducted on a relatively small population, and its confirmation on a larger, multicentre cohort is warranted. Secondly, the EWS were retrospectively calculated from data recorded on the patients’ EMR that were not specifically collected for this purpose. This resulted in missing data necessitating imputation. Thirdly, ICU admission is an institution-dependent outcome, which may limit the generalizability of our results.

In summary, our study showed that in patients with confirmed COVID-19 assessed on ED arrival, REMS and NEWS were the most accurate indices for predicting in-hospital death and ICU admission within seven days, respectively. Due to their high NPV, these indices could be particularly useful for identifying COVID-19 patients who are at lower risk of poor outcome and can be admitted to a non-monitored ward. This supports their use as an adjunct to the process of triage and disposition of COVID-19 patients in the ED. Prospective multicentre cohort studies are needed to provide an external validation of our results.

Credit author statement

All authors have made substantial contributions to all of the following: (1) the conception and design of the study, or acquisition of data, or analysis and interpretation of data, (2) drafting the article or revising it critically for important intellectual content, (3) final approval of the version to be submitted.

The manuscript, including related data, figures and tables, has not been published previously and it is not under consideration elsewhere.

Conflict of interests

None Declared.

Acknowledgements and Funding

None Declared.

Appendix A. Supplementary data

The following is Supplementary data to this article: