Machine learning-based scoring system to predict in-hospital outcomes in patients hospitalized with COVID-19

Abstract

Background

The evolution of patients hospitalized with coronavirus disease 2019 (COVID-19) is still hard to predict, even after several months of dealing with the pandemic.

Aims

To develop and validate a score to predict outcomes in patients hospitalized with COVID-19.

Methods

All consecutive adults hospitalized for COVID-19 from February to April 2020 were included in a nationwide observational study. Primary composite outcome was transfer to an intensive care unit from an emergency department or conventional ward, or in-hospital death. A score that estimates the risk of experiencing the primary outcome was constructed from a derivation cohort using stacked LASSO (Least Absolute Shrinkage and Selection Operator), and was tested in a validation cohort.

Results

Among 2873 patients analysed (57.9% men; 66.6 ± 17.0 years), the primary outcome occurred in 838 (29.2%) patients: 551 (19.2%) were transferred to an intensive care unit; and 287 (10.0%) died in-hospital without transfer to an intensive care unit. Using stacked LASSO, we identified 11 variables independently associated with the primary outcome in multivariable analysis in the derivation cohort (n = 2313), including demographics (sex), triage vitals (body temperature, dyspnoea, respiratory rate, fraction of inspired oxygen, blood oxygen saturation) and biological variables (pH, platelets, C-reactive protein, aspartate aminotransferase, estimated glomerular filtration rate). The Critical COVID-19 France (CCF) risk score was then developed, and displayed accurate calibration and discrimination in the derivation cohort, with C-statistics of 0.78 (95% confidence interval 0.75–0.80). The CCF risk score performed significantly better (i.e. higher C-statistics) than the usual critical care risk scores.

Conclusions

The CCF risk score was built using data collected routinely at hospital admission to predict outcomes in patients with COVID-19. This score holds promise to improve early triage of patients and allocation of healthcare resources.

1. Background

Coronavirus disease 2019 (COVID-19), caused by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), has become a pandemic, leading to a public health crisis of unprecedented magnitude [1], [2], [3]. The COVID-19 clinical spectrum varies considerably, ranging from asymptomatic carriers to acute respiratory distress syndrome associated with high fatality rates. Among hospitalized patients with COVID-19, more than 15% require admission to an intensive care unit (ICU), resulting in a considerable need for intensive care beds and ventilators [4]. Whereas the main clinical characteristics and profiles of patients have been described in large cohorts from China [3], [5], Europe [6] and the USA [7], [8], [9], the clinical course of most hospitalized patients remains hard to predict, even after several months of dealing with the disease. The early identification of patients at risk of developing a severe form of COVID-19 is a major issue, to help clinicians in early triage to optimize the management of patient flow and the allocation of healthcare resources [10], [11].

So far, few specific risk scores assessing the in-hospital evolution of COVID-19 have been developed [12], [13]. Clinicians have been compelled to base their decisions on clinical experience acquired during the pandemic, and on other existing critical care scores. However, few data exist regarding the validity of these usual critical care scores in the context of COVID-19 [14]. Thus, a robust and specific COVID-19 score would help clinicians to make rapid decisions in daily practice.

Using data collected routinely at hospital admission, we aimed to identify initial predictors for developing a severe form of COVID-19 during hospitalization, and to construct a specific risk score through a nationwide multicentre observational study.

2. Methods

2.1. Study settings and population

The Critical COVID-19 France (CCF) study is a French nationwide observational multicentre study, including all consecutive adult patients admitted to hospital (24 centres) with a diagnosis of SARS-CoV-2 infection between 26 February and 20 April 2020 (ClinicalTrials.gov identifier: NCT04344327). In accordance with the World Health Organization criteria, SARS-CoV-2 infection was defined as a positive result on real-time reverse transcription polymerase chain reaction of nasal and pharyngeal swabs or lower respiratory tract aspirates, or typical imaging characteristics on chest computed tomography with a compatible clinical presentation [15]. Patients admitted directly to an ICU were not considered.

The CCF study was declared to and authorized by the French data protection committee (Commission Nationale Informatique et Liberté [CNIL]; authorization n̊2207326v0), and was conducted in accordance with the ethical standards laid down in the 1964 Declaration of Helsinki and its later amendments. The authors had full access to and take full responsibility for the integrity of the data. A complete list of the CCF investigators is provided in Appendix A. Participating centres and number of patients enrolled per centre are detailed in Table A.1.

2.2. Data collection

All data were collected by local investigators in an electronic case-report form via REDCap software (Research Electronic Data Capture; Vanderbilt University, Nashville, TN, USA), hosted by a secured server from the French Institute of Health and Medical Research at the Paris Cardiovascular Research Centre. Patient baseline information included demographic characteristics, co-existing medical conditions and chronic medications. Exhaustive data, including clinical variables, blood test results and chest computed tomography scan characteristics (when performed) were recorded at admission. Chest computed tomography scan results were assessed by a senior radiologist at the centre's local workstation, according to European guidelines [16]. The degree of scanographic lesions was based on visual assessment of parenchymal involvement, and was categorized as limited (< 25%), moderate (25–50%) or severe (> 50%). Only computed tomography scans performed during the first 24 hours were considered. Glomerular filtration rate was calculated using the Modification of Diet in Renal Disease Study equation.

2.3. Outcomes

The primary outcome was a composite of transfer to ICU or in-hospital death. Transfer to ICU was carried out from the emergency department or a conventional hospitalization ward. Data on pharmacological therapies, mode of respiratory support, complications and associated diagnoses during hospital stay were also reported. All medical interventions (including pharmacological agents to treat SARS-CoV-2) were left at the discretion of the referring medical team. Date of final follow-up for patients still hospitalized was 21 April 2020.

2.4. Statistical analysis

This report was prepared in compliance with the STROBE checklist for observational studies [17] and the TRIPOD statement for developing and validating a prediction model [18]. Categorical data are reported as counts and percentages. Continuous data are reported as means ± standard deviations for normally distributed data, and as medians (interquartile ranges [IQRs]) for non-normally distributed data. Comparisons used the χ² test or Fisher's exact test for categorical variables, and Student's t test or the Mann-Whitney-Wilcoxon test, as appropriate, for continuous variables. After exclusion of five patients with missing outcomes, the population was split into a derivation cohort (70% of the cohort, n  = 2313) and a validation cohort (30% of the cohort, n  = 560). The split was performed randomly between the different centres. Thus, the validation cohort included four centres representing the heterogeneity of the epidemic in France (one in the North, South and East of France, the most affected regions, and one in Paris). The rest of the participating centres constituted the derivation cohort (70% of the total population). The amount of available data for each variable is presented in Table 1 . Missing data were handled using multiple random forest imputation by chained equations (mice R package) before multivariable analysis. Variables with < 30% missing data were imputed. Ten unique imputed datasets were built for each cohort (derivation and validation), then penalized regression using LASSO (Least Absolute Shrinkage and Selection Operator) was used to identify predictors of severe SARS-CoV-2 [19]. As a result of multiple imputed datasets, the Stacked Adaptive Elastic Net algorithm was used [20], in which regression coefficients are assumed to be equal across imputed datasets; the datasets are stacked, and the penalized regression is performed in such a way that the same betas (coefficients) are selected at each value of lambda (shrinkage parameter) across each imputed dataset. Table A.2 shows the amount of missing data for each variable included in the final multivariable analysis. Prediction performance was evaluated using discrimination and calibration metrics by means of non-parametric bootstrap inference [21]. The C-statistic and its 95% confidence interval (CI) were estimated by a bootstrapping procedure (2000 replicates in each imputed datasets) in the derivation and validation cohorts. Model calibration was first assessed visually, then by a computational method with calibration curves and calibration slopes. Details regarding multiple imputation, stacked LASSO and bootstrap inference are provided in Appendix B.

Table 1.

Baseline characteristics of patients in the overall population, and the derivation and validation cohorts.

Characteristic	Overall population		Derivation cohort	Validation cohort	P
	(n = 2873)	Number with data	(n = 2313)	(n = 560)
Demographics
Age (years)	66.6 ± 17.0	2868	66.9 ± 16.9	65.3 ± 17.2	0.04
Male sex	1663 (57.9)	2873	1338 (57.8)	325 (58.0)	0.97
Body mass index (kg/m2)	27.8 ± 6.0	2493	27.8 ± 6.1	28.0 ± 6.0	0.52
Cardiovascular risk factors
Smoking	377 (13.4)	2805	290 (12.9)	87 (15.6)	0.33
Hypertension	1451 (50.8)	2854	1193 (52.0)	258 (46.2)	0.02
Diabetes	676 (23.7)	2855	535 (23.3)	141 (25.3)	0.34
Dyslipidaemia	798 (28.0)	2854	669 (29.1)	129 (23.2)	0.006
Co-existing conditions
Chronic kidney disease	403 (14.2)	2831	327 (14.4)	76 (13.7)	0.75
Atrial fibrillation	413 (14.5)	2847	333 (14.5)	80 (14.5)	1.00
Heart failure	329 (11.6)	2873	258 (11.3)	71 (12.9)	0.34
Coronary artery disease	362 (12.6)	2873	307 (13.3)	55 (9.82)	0.03
Immunodeficiency	147 (5.12)	2873	118 (5.10)	29 (5.18)	1.00
Treatment before hospitalization
Anticoagulation	409 (14.2)	2873	341 (14.7)	68 (12.1)	0.13
Angiotensin-converting enzyme inhibitor	505 (17.6)	2873	408 (17.6)	97 (17.3)	0.91
Angiotensin II receptor blocker	468 (16.3)	2873	379 (16.4)	89 (15.9)	0.83
Clinical characteristics
NYHA functional class		2493			0.88
I or II	1216 (48.8)		1014 (48.9)	202 (48.3)
III or IV	1277 (51.2)		1061 (51.1)	216 (51.7)
Systolic pressure (mmHg)	131 ± 22	2825	132 ± 22	129 ± 21	0.002
Diastolic pressure (mmHg)	74 ± 13	2825	74 ± 14	74 ± 13	0.24
Respiratory frequency	23 ± 7	2109	23 ± 7	23 ± 7	0.74
Temperature (°C)	37.2 ± 1.0	2825	37.2 ± 1.1	37.2 ± 1.0	0.96
Blood oxygen saturation (%)	95 ± 3.6	2849	95 ± 3.6	95 ± 3.5	0.01
Inspired oxygen (%)	29 ± 12	2778	29 ± 13	28 ± 10	0.66
Glasgow Coma Scale score < 15	193 (6.81)	2833	149 (6.54)	44 (7.91)	0.29
SIC score ≥ 4	1135 (67.9)	1672	917 (67.4)	218 (69.9)	0.44
qSOFA = 1	1295 (61.5)	2105	998 (61.0)	297 (63.2)	0.43
Laboratory
pH	7.45 ± 0.06	2004	7.45 ± 0.06	7.44 ± 0.06	0.001
PaO2:FiO2 ratio < 150	176 (9.0)	1956	147 (9.4)	29 (7.3)	0.23
Lactates (mmol/L)	1.4 ± 1.0	1754	1.4 ± 1.0	1.5 ± 1.0	0.06
Leukocytes (g/L)	7.3 ± 5.1	2822	7.4 ± 5.4	7.2 ± 3.8	0.42
Platelets (g/L)	221 ± 99	2802	218 ± 96	232 ± 112	0.007
C-reactive protein (mg/L)	90.2 ± 76.9	2753	90.5 ± 77.1	88.8 ± 76.2	0.64
GFR (mL/min/m2)	82 ± 30	2824	82 ± 29	82 ± 30	0.51
Aspartate aminotransferase (IU/L)	54 ± 69	2605	55 ± 75	49 ± 37	0.006
Alanine aminotransferase (IU/L)	46 ± 75	2610	46 ± 63	49 ± 115	0.60
D-dimer (μg/L)	1644 ± 3633	1156	1713 ± 3893	1333 ± 2091	0.05
Fibrinogen (g/L)	6.0 ± 1.7	1379	6.0 ± 1.7	6.0 ± 1.7	0.94
Ferritin (μg/L)	1092 ± 1880	722	1040 ± 1502	1250 ± 2727	0.33
Lactate dehydrogenase (IU/L)	368 ± 333	922	359 ± 348	407 ± 256	0.04
Elevated BNP or NT-proBNPa	942 (53.0)	1776	783 (54.0)	159 (48.8)	0.10
Troponin elevationb	572 (32.5)	1760	398 (28.8)	174 (46.3)	< 0.001
Positive SARS-CoV-2 RT-PCR	2591 (91.8)	2823	2107 (92.9)	484 (87.2)	< 0.001
Chest computed tomography
Parenchymal involvement		2244			0.16
Minimal or moderate (< 50%)	1815 (80.9)		1444 (80.3)	371 (83.4)
Severe (> 50%)	429 (19.1)		355 (19.7)	74 (16.6)

Data are expressed as mean ± standard deviation or number (%). BNP: B-type natriuretic peptide; FiO₂: fraction of inspired oxygen; GFR: glomerular filtration rate; NT-proBNP: N-terminal prohormone of B-type natriuretic peptide; NYHA: New York Heart Association; PaO₂: partial pressure of oxygen; qSOFA: quick Sepsis-related Organ Failure Assessment; RT-PCR: reverse-transcriptase polymerase chain reaction; SARS-CoV-2: severe acute respiratory syndrome coronavirus 2; SIC: Sepsis-Induced Coagulopathy.

^aBNP > 50 pg/mL or NT-proBNP > 300 pg/mL.

^bAbove each centre's threshold.

Sensitivity analyses were conducted in different subgroups: in the complete cases i.e. excluding patients with missing data; and in patients with positive polymerase chain reaction results.

Discrimination of the CCF risk score was compared with five usual critical care risk scores (PREDICO [prediction of severe respiratory failure in hospitalized patients with SARS-CoV-2 infection], SOFA [Sepsis-related Organ Failure Assessment], qSOFA [quick SOFA], CURB-65 [confusion, blood urea > 42.8 mg/dL, respiratory rate > 30 breaths/min, blood pressure < 90/60 mmHg, age > 65 years] and MEWS [Modified Early Warning Score]), whose C-statistics in the derivation and validation cohort are detailed in Table A.3.

A two-tailed P  < 0.05 was considered statistically significant. All data were analysed using R software, version 3.6.3 (R Project for Statistical Computing, Vienna, Austria).

3. Results

3.1. Overall population

Among 2878 consecutive adults admitted for SARS-CoV-2 infection across 24 French hospitals between 26 February and 20 April 2020, 2873 patients were analysed (57.9% men; mean age 66.6 ± 17.0 years). Patients’ baseline characteristics are presented in Table 1. Overall, the primary outcome occurred in 838 (29.2%) patients after a median delay of 3.0 (IQR 1.0–6.0) days: 551 (19.2%) were transferred to an ICU; and 287 (10.0%) died without transfer to an ICU (Fig. 1 ). Median delay before transfer to ICU was 2.0 (IQR 1.0–4.0) days, and before death without transfer to ICU was 6.5 (IQR 3.3–10) days. A total of 362 (12.6%) patients died during hospitalization. Mechanical ventilation was used in 370 (12.9%) patients, non-invasive ventilation support in 81 (2.8%) patients and high-flow oxygen therapy in 153 (5.3%) patients. Median length of hospitalization among the 1991 patients discharged alive was 8.0 (IQR 5.0–12.0) days. As of 21 April 2020, 513 (17.9%) patients were still hospitalized, including 264 patients initially transferred to an ICU and 249 patients not admitted to an ICU. Pharmacological treatments for COVID-19 included antibiotics in 2137 (74.4%) patients, corticosteroids in 214 (7.5%), immunomodulatory agents in 33 (1.2%) and immunoglobulin therapy in two (0.1%).

Fig. 1.

Flow chart for the Critical COVID-19 France study. ICU: intensive care unit; SARS-CoV-2: severe acute respiratory syndrome coronavirus 2.

3.2. Prediction model construction

Characteristics of patients in the derivation (n  = 2313) and validation (n  = 560) cohorts were mostly similar (Table 1). Univariate analyses of factors associated with the primary outcome in the derivation cohort are presented in Table 2 . Eleven variables, including demographics (sex), triage vitals (body temperature, dyspnoea, respiratory rate, fraction of inspired oxygen, blood oxygen saturation) and biological variables (pH, platelets, C-reactive protein, aspartate aminotransferase, estimated glomerular filtration rate [Modification of Diet in Renal Disease Study equation]) were included in the final model.

Table 2.

Univariate analysis (derivation cohort): Association of clinical, biological and imaging factors with primary outcome in univariate logistic regression analysis.

Variables	Presence of primary composite outcome		OR (95% CI)	P
	Yes (n = 673)	No (n = 1567)
Demographics
Age (years)	70 ± 16	66 ± 17	1.02 (1.01–1.02)	< 0.001
Male	444 (66.3)	894 (54.4)	1.65 (1.37–1.99)	< 0.001
Body mass index (kg/m2)	28.5 ± 6.4	27.5 ± 5.9	1.03 (1.01–1.04)	0.002
Cardiovascular risk factors
Smoking	95 (14.7)	195 (12.2)	1.25 (0.95–1.62)	0.12
Hypertension	395 (59.4)	798 (48.9)	1.53 (1.27–1.83)	< 0.001
Diabetes	191 (28.7)	344 (21.1)	1.51 (1.23–1.85)	< 0.001
Dyslipidaemia	229 (34.4)	440 (26.9)	1.42 (1.17–1.73)	< 0.001
Co-existing conditions
Chronic kidney disease	146 (22.2)	181 (11.2)	2.27 (1.78–2.89)	< 0.001
Atrial fibrillation	93 (14.0)	240 (14.7)	0.94 (0.72–1.22)	0.69
Heart failure	105 (16.0)	153 (9.46)	1.82 (1.39–2.37)	< 0.001
Coronary artery disease	110 (16.4)	197 (12.0)	1.44 (1.12–1.85)	0.005
Venous thromboembolic disease	53 (7.91)	128 (7.79)	1.03 (0.73–1.43)	0.35
Treatment before hospitalization
Anticoagulation	100 (14.9)	241 (14.7)	1.02 (0.79–1.31)	0.93
Angiotensin-converting enzyme inhibitor	137 (20.4)	271 (16.5)	1.30 (1.03–1.63)	0.03
Angiotensin II receptor blocker	123 (18.4)	256 (15.6)	1.22 (0.96–1.54)	0.12
Clinical characteristics
NYHA functional class				< 0.001
I or II	209 (34.7)	805 (54.7)	Ref.
III or IV	394 (65.3)	667 (45.3)	2.27 (1.87–2.77)
Heart rate (beats/min)	89 ± 19	86 ± 18	1.01 (1.00–1.01)	< 0.001
Systolic pressure (mmHg)	131 ± 24	132 ± 22	1.00 (0.99–1.00)	0.34
Diastolic pressure (mmHg)	74 ± 14	75 ± 13	0.99 (0.99–1.00)	0.14
Respiratory frequency	26 ± 7	22 ± 6	1.08 (1.07–1.10)	< 0.001
Temperature (°C)	37.4 ± 1.2	37.1 ± 1.0	1.31 (1.20–1.43)	< 0.001
Blood oxygen saturation (%)	93 ± 4	95 ± 3	0.86 (0.84–0.88)	< 0.001
FiO2 (%)	35 ± 17	26 ± 9	1.06 (1.05–1.07)	< 0.001
Glasgow Coma Scale score < 15	68 (10.4)	81 (5.0)	2.19 (1.56–3.07)	< 0.001
Heart failure signs	58 (8.8)	107 (6.6)	1.36 (0.97–1.90)	0.08
SIC score ≥ 4	317 (78.7)	600 (62.7)	2.19 (1.67–2.89)	< 0.001
qSOFA = 1	362 (74.6)	636 (55.3)	2.38 (1.88–3.01)	< 0.001
Laboratory
pH	7.45 ± 0.07	7.46 ± 0.05	0.02 (0.00–0.14)	< 0.001
PaO2:FiO2 ratio < 150	104 (18.6)	43 (4.3)	5.08 (3.52–7.44)	< 0.001
Lactates (mmol/L)	1.6 ± 1.3	1.3 ± 0.7	1.45 (1.26–1.67)	< 0.001
Leukocytes (g/L)	8.2 ± 6.1	7.0 ± 5.1	1.05 (1.02–1.07)	< 0.001
Platelets (g/L)	206 ± 96	223 ± 95	1.00 (1.00–1.00)	< 0.001
C-reactive protein (mg/L)	122 ± 86	78 ± 69	1.01 (1.01–1.01)	< 0.001
GFR (mL/min/m2)	73 ± 31	85 ± 28	0.99 (0.98–0.99)	< 0.001
Aspartate aminotransferase (IU/L)	68 ± 93	50 ± 64	1.00 (1.00–1.01)	< 0.001
Alanine aminotransferase (IU/L)	53 ± 80	43 ± 48	1.00 (1.00–1.00)	0.006
D-dimer (μg/L)	2423 ± 5982	1362 ± 2143	1.00 (1.00–1.00)	0.003
Fibrinogen (g/L)	6.4 ± 1.6	5.8 ± 1.7	1.23 (1.14–1.33)	< 0.001
Ferritin (μg/L)	1442 ± 2066	860 ± 1124	1.00 (1.00–1.00)	0.001
Lactate dehydrogenase (IU/L)	466 ± 602	314 ± 109	1.01 (1.00–1.01)	< 0.001
Elevated BNP or NT-proBNPa	306 (63.1)	477(49.4)	1.75 (1.40–2.19)	< 0.001
Troponin elevationb	180 (39.4)	218 (23.5)	2.11 (1.66–2.69)	< 0.001
Positive SARS-CoV-2 RT-PCR	626 (95.6)	1481 (91.8)	1.73 (1.15–2.69)	0.008
Chest computed tomography
Parenchymal involvement				< 0.001
Minimal or moderate (< 50%)	329 (64.3)	1115 (86.6)	Ref.
Severe (> 50%)	183 (35.7)	172 (13.4)	3.60 (2.83–4.59)

Data are expressed as mean ± standard deviation or number (%). BNP: B-type natriuretic peptide; CI: confidence interval; FiO₂: fraction of inspired oxygen; GFR: glomerular filtration rate; NT-proBNP: N-terminal prohormone of B-type natriuretic peptide; NYHA: New York Heart Association; OR: odds ratio; PaO₂: partial pressure of oxygen; qSOFA: quick Sepsis-related Organ Failure Assessment; RT-PCR: reverse-transcriptase polymerase chain reaction; SARS-CoV-2: severe acute respiratory syndrome coronavirus 2; SIC: Sepsis-Induced Coagulopathy.

^aBNP > 50 pg/mL or NT-proBNP > 300 pg/mL.

^bAbove each centre's threshold.

3.3. Prediction model performance

The CCF risk-stratification score was calculated for each patient, according to the beta-regression coefficients estimated from the stacked LASSO multivariable model. Based on this score, an online ready-to-use interface was built for clinicians to give an early estimate of a patient's individual probability of developing a severe form of COVID-19 during hospitalization (https://criticalcovidfrance.shinyapps.io/criticalcovidfrance/). The performance of the model in the derivation and validation cohorts is displayed in Fig. 2 . The corresponding C statistics were 0.78 (95% CI 0.75–0.80) in the derivation cohort and 0.75 (95% CI 0.70–0.79) in the validation cohort. Calibration plots (Fig. 2) showed good agreement between the CCF risk score-predicted probability and the observed probability of developing severe COVID-19. Calibration slope (ideally 1) and intercept (ideally 0) pooled among imputed datasets were, respectively, 1.13 and–0.05 in the derivation set, and 1.25 and–0.06 in the validation set. The CCF risk score showed significantly better discrimination compared with usual critical care risk scores specifically developed during the coronavirus pandemic (4 C [Coronavirus Clinical Characterisation Consortium], PREDICO and qCSI [quick Covid Severity Index] scores compared with CCF risk score C-statistics: P  < 0.001) or non-specific severity scores (SOFA, CURB-65 and ROX [Respiratory Rate and Oxygenation] index scores compared with CCF risk score C-statistics: P  < 0.001) (Fig. 3 ).

Fig. 2.

Calibration curves in the derivation and validation cohorts.

Fig. 3.

Receiver operating characteristic curve comparison between the Critical COVID-19 France (CCF) risk score (model) and usual critical care scores in the overall cohort. The area under the receiver operating characteristic curve (AUROC) of the CCF score (model) was higher than those of the SOFA (Sepsis-related Organ Failure Assessment) score (P < 0.001), the CURB-65 (confusion, blood urea > 42.8 mg/dL, respiratory rate > 30 breaths/min, blood pressure < 90/60 mmHg, age > 65 years) score (P < 0.001), the ROX (Respiratory Rate and Oxygenation) index (P < 0.001), the 4 C (Coronavirus Clinical Characterisation Consortium) score (P < 0.001), the PREDICO (prediction of severe respiratory failure in hospitalized patients with SARS-CoV-2 infection) score (P < 0.001) and the qCSI (quick Covid Severity Index) (P < 0.001).

3.4. Sensitivity analyses

Two sensitivity analyses were performed to test the robustness and generalizability of the model (Fig. A.1). The CCF risk score demonstrated good performance when analyses were restricted to patients with no missing data (n  = 1105; C-statistic 0.74, 95% CI 0.71–0.77) and in patients with positive polymerase chain reaction results (n  = 2591; C-statistic 0.77, 95% CI 0.75–0.79). Different patient profiles with corresponding estimated probabilities of developing a severe form of COVID-19 during hospitalization are provided in Table A.4.

4. Discussion

Using data from a multicentre observational study of 2873 patients who were hospitalized for COVID-19 across 24 French centres, we developed a risk-stratification score for the early identification of patients at risk of becoming critically ill during hospitalization. This multivariable CCF risk score combines clinical, biological and imaging data collected routinely at hospital admission, and displays good performance.

As of 2022, France is one of the most burdened countries in the world, with more than 21 million cases and 130,000 deaths related to SARS-CoV-2 infection [22]. Whereas the first case series from China [3], [5], Europe [6], [23] and the USA [8], [9], [24] reported the main characteristics and profiles of patients hospitalized with COVID-19, epidemiological data in France remain scarce. The characteristics of our French population confirmed that most patients were middle-aged or elderly men, and that cardiovascular co-morbidities were highly prevalent [25], [26]. Thirty percent of patients presented a severe form of COVID-19 in our study, defined as death or transfer to ICU. The overall mortality rate observed (12.6%) is, however, difficult to compare with other published series, given the heterogeneity among healthcare systems, the populations studied and early (but needed) reports of experiences while a significant proportion of patients are still hospitalized [6], [9], [23]. The same applies to the ICU transfer rate, estimated at 19.2% in our study. Regional, national and international disparities in the availability of critical care beds make comparison with other cohorts difficult.

Various factors were associated with the primary outcome, including co-morbidities, cardiovascular risk factors, treatment before hospitalization and clinical and paraclinical variables. Whereas co-morbidities, especially cardiac diseases, have been consistently associated with poorer outcomes [7], [23], after multiple adjustments, prediction of severe COVID-19 forms in our study was mainly captured by clinical status, and biological and chest computed tomography findings at admission. These findings highlight multiple facets of COVID-19 that combine inflammation, sepsis-like profile and rapidly progressive respiratory failure. The degree of individual systemic inflammatory response syndrome seems to drive patient prognosis to a greater extent than underlying conditions. This abnormal and amplified inflammatory or immune response is targeted by different classes of adjunctive therapies for COVID-19, such as anticytokines, immunomodulatory agents or corticosteroids, with preliminary results published from uncontrolled or non-comparative studies and on-going randomized trials [27]. Some factors already identified as prognostic factors in the literature, such as age, obesity and diabetes, were not included in the final risk score [28], [29], [30]. Inherently, machine-learning variable selection technique only retains variables with the highest impact on prognosis [19]. Thus, age may not have been retained in the final model because its impact is outweighed by other factors. Some studies have shown that the sex of the patients exerted greater influence than their age, with survival of elderly women sometimes better than that of middle-aged men with more co-morbidities [26], [28], [31]. Regarding diabetes and obesity, it is possible that the increased risk conferred by these co-morbidities is related to an inflammatory response profile to SARS-CoV-2 [30], [32]. This inflammatory excess risk is reflected in the score by the platelet level and C-reactive protein. Thus, the inclusion of multimodal variables in the score is a good reflection of the multifaceted aspect of COVID-19.

This unprecedented viral pandemic has imposed considerable resource and healthcare system reorganization in most countries throughout the world. At the first peak of the epidemic, the USA's projected needs reached approximately 17,000 ICU beds and 20,000 invasive ventilators, far beyond current availabilities [33]. Our integrative risk score, which includes variables readily available at admission, may represent a valuable tool in routine clinical practice, to improve early triage, optimize the management of patient flow and adequately readjust allocation of resources [34]. The CCF risk score may also help to predict the severity of symptoms and optimize patient selection in clinical trials [35].

Few specific risk scores assessing the in-hospital evolution of COVID-19 have been published so far [12], [13]. One of these was derived from a Chinese cohort, integrating 10 variables among 72 potential predictive variables at admission [13]; it included chest radiographic variables, which are less informative than the computed tomography scan that is primarily performed in Europe. Chest radiography is not sensitive enough for the detection of ground glass opacities, which are the main imaging features of COVID-19 pneumonia [36]. As recommended in European guidelines, chest radiography should not be used as the first-line technique, and should be restricted to the follow-up of patients admitted to an ICU or patients too fragile to be sent for computed tomography [16]. Moreover, our score provides complementary information, with possible ethnic or geographical disparities in populations in Europe or the USA.

Finally, we now face cycles of acceleration, suppression and re-emergence of the pandemic, with fast and unpredictable emergence of new SARS-CoV-2 variants [37], [38]. This risk score may be a useful tool in our arsenal to fight COVID-19 in case of new wave(s) or aggressive variants.

This study has some limitations. First, data collection was retrospective. However, the relatively short time between each patient's hospitalization and the gathering of their data (median 14 days, IQR 9–19 days) allowed investigators to easily recover a large amount of data of interest. The mean burden of missing data on variables selected in the final multivariable model was only 7.2%. Although multiple imputations were used to develop the risk-stratification score, sensitivity analyses depicted good performance, even in the non-imputed dataset. Second, our risk score was derived from variables collected at admission, to help early triage of patients. Consequently, this approach does not consider the evolution of these items or other events that occurred during hospitalization that may further improve prognostication. Third, our score was developed from a multicentre cohort, and, in the first wave of COVID-19, this potentially implies local disparities regarding the criteria for transfer to critical care and the critical care capacity of each centre. Last, while the pandemic has continued to evolve and shift, our score was constructed and validated on the first wave of patients hospitalized for COVID-19. We now have improved primary prevention and in-hospital care with proven effective treatments (i.e. corticoids) and specific vaccines that were not available at the time of the study [39], [40]. Besides, current variants, such as the Omicron variant, have been shown to be less likely to result in death or ICU admission [41], [42]. Still, the in-hospital death rate remains stable, even with recent variants Omicron BA.1 or BA.2, because hospitalized patients are older or have severe co-morbidities [43], [44]. Our death rate is consistent with recent rates described in the USA and Europe [45], [46]. Furthermore, risk factors associated with severe forms of COVID-19 still seem to be associated with outcomes in patients infected with recent SARS-CoV-2 variants [43]. Thus, despite its early development, the CCF risk score remains of value to improve the efficiency of patient triage upon admission to hospital.

5. Conclusions

Using data from a nationwide multicentre observational study of patients hospitalized for COVID-19 during the first wave, we identified independent predictors of a severe form of COVID-19, including clinical, biological and imaging variables collected routinely at admission. An accurate integrative machine learning-based risk score was developed and validated to optimize early triage of patients.

Central illustration. Critical COVID-19 France, built using machine-learning techniques: methods, selected variables and performance. CCF: Critical COVID-19 France; CI: confidence interval; COVID-19: coronavirus disease 2019; ICU: intensive care unit; LASSO: Least Absolute Shrinkage and Selection Operator.

Disclosure of interest

The authors declare that they have no competing interest.

Sources of funding

None.