Validation of Machine Learning Models to Predict Adverse Outcomes in Patients with COVID-19: A Prospective Pilot Study

Abstract

Purpose

We previously developed learning models for predicting the need for intensive care and oxygen among patients with coronavirus disease (COVID-19). Here, we aimed to prospectively validate the accuracy of these models.

Materials and Methods

Probabilities of the need for intensive care [intensive care unit (ICU) score] and oxygen (oxygen score) were calculated from information provided by hospitalized COVID-19 patients (n=44) via a web-based application. The performance of baseline scores to predict 30-day outcomes was assessed.

Results

Among 44 patients, 5 and 15 patients needed intensive care and oxygen, respectively. The area under the curve of ICU score and oxygen score to predict 30-day outcomes were 0.774 [95% confidence interval (CI): 0.614–0.934] and 0.728 (95% CI: 0.559–0.898), respectively. The ICU scores of patients needing intensive care increased daily by 0.71 points (95% CI: 0.20–1.22) after hospitalization and by 0.85 points (95% CI: 0.36–1.35) after symptom onset, which were significantly different from those in individuals not needing intensive care (p=0.002 and <0.001, respectively). Trends in daily oxygen scores overall were not markedly different; however, when the scores were evaluated within <7 days after symptom onset, the patients needing oxygen showed a higher daily increase in oxygen scores [1.81 (95% CI: 0.48–3.14) vs. -0.28 (95% CI: 1.00–0.43), p=0.007].

Conclusion

Our machine learning models showed good performance for predicting the outcomes of COVID-19 patients and could thus be useful for patient triage and monitoring.

INTRODUCTION

After initial reports in late December 2019, coronavirus disease (COVID-19) has become a worldwide pandemic.¹ Among several modalities used to treat the disease,²,³,⁴,⁵,⁶,⁷ only a few have proven to be effective.²,³ Supportive care and respiratory support are now considered as mainstays of treatment for COVID-19.⁸ As such, medical professionals must focus on patient triage to the appropriate level of care to reduce the risk of medical supply shortages.⁹

Numerous efforts have been made to establish risk factors of deterioration among patients with COVID-19.¹⁰,¹¹,¹²,¹³ However, most of these require laboratory or radiographic findings, which can be time consuming and costly to obtain. Therefore, we have developed an easy-to-use machine learning model with which to predict the risk of needing intensive care among COVID-19 patients using easily obtainable patient information (e.g., demographics, comorbidities, subjective symptoms, and body temperature).¹⁴ Another model was developed using the need for oxygen supplementation as another outcome. The models were integrated into a web-based application developed during the early phase of the pandemic.¹⁵,¹⁶ However, the original models were developed from and validated in a retrospective cohort.¹⁴ In addition, the information was uploaded by attending physicians and not by the patients themselves. Because our easy-to-use machine learning models were designed to be used directly by patients, the validity of the models in which the information is directly uploaded by patients must be investigated.

Thus, in this study, we aimed to prospectively evaluate the validity of the models for predicting the need for intensive care or oxygen supplementation among patients with COVID-19.

MATERIALS AND METHODS

Study design and patients

In this prospective observational study, we screened all adult (age≥18 years) patients with COVID-19 confirmed by polymerase chain reaction who were admitted to the Armed Forces Capital Hospital Trauma Center, Seongnam, South Korea from September 19, 2020 through November 19, 2020. We enrolled only those who volunteered to participate. The Armed Forces Capital Hospital Trauma Center is a 60-bed hospital constructed on March 5, 2020 to treat trauma patients. Because of a sudden upsurge in COVID-19 patients, a part of the center was converted into a 40-bed COVID-19 care unit. The unit is capable of providing general supportive care, including oxygen supplementation. Intensive care, such as mechanical ventilation, vasopressor use, and extracorporeal life support, is not possible in this unit, and patients needing intensive care are transferred to other hospitals.

This study was approved by the Institutional Review Board of the Armed Forces Capital Hospital (approval number: AFCH-20-IRB-037) and was conducted in accordance with the amended Declaration of Helsinki. The need for informed consent was waived because patients volunteered to directly provide their information into the application without any invasive measurements. In addition, acquiring written informed consent was considered dangerous due to the highly transmissive nature of COVID-19.¹⁷

Data collection

Patients provided their data, including demographics, smoking history, underlying comorbidities, activities of daily living, symptoms, and body temperatures, directly to an online web-based application (Supplementary Fig. 1A, only online).¹⁴ In addition to their baseline data, the patients were encouraged to provide their daily symptoms and body temperature if possible. Additional information, such as symptom onset, patient outcome, date of admission, and date of discharge, was collected by an attending physician.¹⁷

Definitions of outcomes and model development

This study included two prediction models: one predicting the need for intensive care [intensive care unit (ICU) score] and the other predicting the need for oxygen supplementation (oxygen score). The need for intensive care was defined as admission to the ICU, use of extracorporeal life support, mechanical ventilation, vasopressors, or death within the first 30 days of admission.¹⁴ This accounted for patients who could not be admitted to the ICU owing to limited hospital facilities. The need for oxygen supplementation during the first 30 days was included as another outcome because it is a useful criterion for hospitalization.¹⁸

Both models were originally derived from and validated in a separate nationwide cohort that included hospitalized patients with COVID-19 from 100 hospitals in South Korea. Patient information was uploaded to an online case report form by the attending physicians in each center, and the database was managed by the Korean Disease Control and Prevention Agency (https://icreat.nih.go.kr/).¹⁴ We used patient characteristics that could be easily provided by patients, such as demographics, smoking history, symptoms, and body temperature, to derive a machine learning model with an AutoML method.¹⁹ The details of the included variables are presented in Supplementary Table 1 (only online). Patients hospitalized from January 25, 2020 through March 20, 2020 were assigned to the model derivation group, and those hospitalized from March 21, 2020 through June 3, 2020 were assigned to the model validation group. Detailed descriptions of model derivation and validation for ICU score are provided in an earlier report.¹⁴ Results of the calculated probability of the need for intensive care based on the predefined XGBoost model are presented as numbers ranging from 0 (lowest probability) to 100 (highest probability).¹⁴ Oxygen score was derived in a similar manner using the same variables. ICU score and oxygen score to predict patient outcomes showed excellent discrimination performance in both the derivation and validation groups from the previous cohort (Supplementary Fig. 2, only online).¹⁴ Variables with high feature importance included activities of daily living, age, dyspnea, body temperature, sex, and symptoms of dyspnea (Fig. 1). Details on these variables are presented in Supplementary Tables 2 and 3 (only online).

Fig. 1. Top 20 features with high feature importance to (A) ICU score and (B) oxygen score. ICU, intensive care unit.

Statistical analysis

Probabilities of the need for intensive care and oxygen supplementation according to ICU and oxygen scores were calculated automatically after data were uploaded by patients. The probabilities are presented as numbers from 0 to 100, with 0 referring to the lowest probability of requiring intensive care or oxygen supplementation in each model and 100 referring to the highest. The attending physician could inspect these scores along with details via a web-based application for physicians (Supplementary Fig. 1B, only online). We calculated the area under the receiver operating characteristics curve (AUC) with 95% CIs to assess the discrimination performance of initial scores.

Although both scores were derived to predict 30-day outcomes using baseline scores, serial data were gathered for further analyses. We utilized linear mixed-effects models to evaluate the repeated measures of each score. The models are useful for analyzing repeated measures as they can use all data available and account for repeats within subjects.²⁰,²¹ Separate linear mixed effects models were used to estimate associations between changes in ICU and oxygen scores and the presence of each outcome (need for intensive care or oxygen) with patient-specific intercept.²⁰ The model included terms for time (hospital days), outcome, and interactions between them. Two different baseline timepoints were used in both scores: the day of hospitalization and the day of symptom onset. Scores provided on the day or after discharge were excluded in the analyses. In addition, the scores provided from the day of oxygen supplementation were excluded in the analysis of oxygen scores. All statistical analyses were performed using Stata version 16 (StataCorp. 2019. Stata Statistical Software: release 16. StataCorp LLC, College Station, TX, USA).

RESULTS

Baseline characteristics

Among 82 patients hospitalized in the COVID-19 care unit during the study period, 44 patients volunteered to participate in our study. Among those patients, 5 and 15 patients needed intensive care and supplementary oxygen, respectively. All 5 patients who needed intensive care also needed supplementary oxygen. The remaining 29 patients were discharged without the need for oxygen supplementation or intensive care (Fig. 2).

Fig. 2. Hospital course of the included patients.

The median patient age was 61 years (IQR: 53–64 years). 29 (65.9%) were female, and 35 (79.6%) were never smokers. The most common underlying comorbidities were hypertension 9 (20.5%) and diabetes mellitus 8 (18.2%). Almost all patients were able to independently perform their daily activities 43 (97.7%). The median duration between symptom onset and hospitalization was 2 days (IQR: 1–4.5 days), and the most frequent symptoms were cough 26 (59.1%), sputum 20 (45.5%), and sore throat 18 (40.9%). The baseline demographics, underlying comorbidities, and symptoms did not differ according to the requirement of oxygen or intensive care. However, patients who required intensive care had higher baseline body temperatures than those who did not need oxygen or intensive care (p=0.006) (Table 1).

Table 1. Baseline Patient Characteristics according to Level of Care.

Variables		Total (n=44)	Do not need oxygen or intensive care (n=29)	Need oxygen but not intensive care (n=10)	Need intensive care (n=5)	p-value
Age, yr		61 (53–64)	60 (43–63)	62 (58–64)	63 (62–64)	0.132
Female sex		29 (65.9)	19 (65.5)	7 (70.0)	3 (60.0)	>0.999
Smoking history						0.394
	Never smoked	35 (79.6)	24 (82.8)	8 (80.0)	3 (60.0)
	Former smoker	7 (15.9)	3 (10.3)	2 (20.0)	2 (40.0)
	Current smoker	2 (4.6)	2 (6.9)	0	0
Comorbidities
	Hypertension	9 (20.5)	5 (17.2)	2 (20.0)	2 (40.0)	0.459
	Diabetes mellitus	8 (18.2)	5 (17.2)	2 (20.0)	1 (20.0)	>0.999
	Chronic neurological disorder	3 (6.8)	3 (10.3)	0	0	>0.693
	Autoimmune disease	1 (2.3)	0	1 (10.0)	0	0.341
	Use of immune suppressants	1 (2.3)	0	1 (10.0)	0	0.341
Activities of daily living						>0.999
	Independent	43 (97.7)	28 (96.6)	10 (100.0)	5 (100.0)
	Partially dependent	1 (2.3)	1 (3.5)	0	0
Symptom
	Onset, days	2 (1–4.5)	2 (1–3)	2 (1–5)	3 (2–3)	0.936
	Cough	26 (59.1)	15 (51.7)	6 (60.0)	5 (100.0)	0.138
	Sputum	20 (45.5)	12 (41.4)	5 (50.0)	3 (60.0)	0.729
	Sore throat	18 (40.9)	14 (48.3)	2 (20.0)	2 (40.0)	0.296
	Myalgia	15 (34.1)	9 (31.0)	4 (40.0)	2 (40.0)	0.895
	Headache	15 (34.1)	9 (31.0)	3 (30.0)	3 (60.0)	0.500
	Anosmia	8 (18.2)	5 (17.2)	2 (20.0)	1 (20.0)	>0.999
	Fatigue	5 (11.4)	3 (10.3)	1 (10.0)	1 (20.0)	0.781
	Rhinorrhea	4 (9.1)	4 (13.8)	0	0	0.731
	Dyspnea	2 (4.6)	1 (3.5)	0	1 (20.0)	0.264
	Hemoptysis	2 (4.6)	1 (3.5)	1 (10.0)	0	0.571
	Arthralgia	2 (4.6)	1 (3.5)	0	1 (20.0)	0.264
	Chest pain	1 (2.3)	1 (3.5)	0	0	>0.999
	Diarrhea	1 (2.3)	0	1 (10.0)	0 (0.0)	0.341
Body temperature, °C		37.2 (36.8–37.5)	37.1 (36.6–37.4)	37.5 (37.1–37.8)	37.7 (37.6–38.0)	0.006

Data are presented as a n (%) or median (IQR). P-values were calculated using Fisher’s exact test for categorical variables and the Kruskal-Wallis test for continuous variables.

Performance of baseline scores to predict 30-day outcomes

The median ICU score at baseline was 2.59 (IQR: 2.10–5.24) in the overall population, 4.53 (IQR: 3.41–5.61) in the 5 patients who needed intensive care, and 2.39 (IQR: 2.06–5.07) in the 39 patients who did not need it (p=0.048). The patients needed intensive care within a median of 8 days (IQR: 7–9 days) from hospitalization and a median of 9 days (IQR: 8–11 days) from symptom onset. The AUC of ICU scores to predict the need for intensive care within 30 days was 0.774 (95% CI: 0.614–0.934) (Fig. 3A).

Fig. 3. ROC curve of each score for predicting 30-day outcomes. (A) ROC curve of ICU score to predict the need for intensive care within 30 days of hospitalization. (B) ROC curve of oxygen score to predict the need for oxygen supplementation within 30 days of hospitalization. ROC, receiver operating characteristics; ICU, intensive care unit; AUC, area under the receiver operating characteristics curve.

Baseline oxygen score was only assessed after oxygen supplementation in three patients, and they were excluded from the analyses of oxygen scores. The median oxygen score at baseline was 6.55 (IQR: 4.30–8.61) in 41 patients, 8.07 (IQR: 6.39–16.14) in 12 patients who needed oxygen supplementation, and 6.00 (IQR: 6.28–16.14) in 29 patients who did not need it (p=0.022). The patients needed oxygen supplementation within a median of 4 days (IQR: 3–6 days) from hospitalization and within a median of 7.5 days (IQR: 6–9.5 days) from symptom onset. The AUC of oxygen scores to predict the need for oxygen supplementation within 30 days was 0.728 (95% CI: 0.559–0.898) (Fig. 3B).

Changes in daily score according to patient outcomes

In total, 464 scores were calculated for both ICU and oxygen scores. Among those scores, 24 scores were provided on the day of and after discharge and were therefore excluded. ICU score was measured at least twice in all 44 patients, with a median of 10 measurements (IQR: 7–12.5) per patient. When baseline score was defined as that obtained on the admission day, the 5 patients who needed intensive care showed an average daily increase of 0.71 points (95% CI: 0.20–1.22), while the 39 patients who did not need intensive care showed an average daily decrease of -0.11 points (95% CI: -0.20–-0.02), with a significant difference between the two groups (p=0.002) (Fig. 4A). A similar significant difference was found when the baseline score was set as that obtained on the day of symptom onset [+0.85 points (95% CI: 0.36–1.35) vs. -0.10 points (95% CI: -0.19–-0.01), p<0.001] (Fig. 4B).

Fig. 4. Changes in scores according to disease course and 30-day outcome. Change in ICU score (A) after hospitalization and (B) after symptom onset according to the need for intensive care. Change in oxygen score (C) after hospitalization and (D) after symptom onset according to the need for oxygen supplementation. ICU, intensive care unit.

With respect to oxygen score, only 353 scores from 41 patients were included, because 87 scores were obtained on the day of oxygen supplementation. The oxygen score was measured at least twice in all 41 patients, with a median of eight measurements (IQR: 4–11) per patient. There was no significant difference in daily changes in oxygen scores starting from the day of hospitalization between the 12 patients who required oxygen and the 29 patients who did not require oxygen (p=0.113) (Fig. 4C). The difference remained insignificant between the two groups of patients when the baseline score was set as that obtained on the day of symptom onset (p=0.349) (Fig. 4D). However, when only the scores calculated within less than 7 days of symptom onset were analyzed, the patients who required oxygen supplementation showed a significantly higher increase in their daily oxygen score than those who did not need oxygen supplementation [1.81 (95% CI: 0.48–3.14) vs. -0.28 (95% CI: -1.00–0.43), p=0.007].

DISCUSSION

In this study, we aimed to validate our predeveloped machine learning models to predict the need for intensive care and oxygen supplementation among patients with confirmed COVID-19. Initial scores predicted 30-day outcomes with good discrimination performance. In addition, we found distinct patterns of changes in daily scores according to patient outcomes. To our best knowledge, this study is the first to evaluate prognostic models with real-world data provided directly by patients. Further, this study is also the first to evaluate patterns of changes in these patients.

Our models have several advantages. First, they can enable early triage with minimal resources. Early triage is crucial for achieving good outcomes in patients with COVID-19 because the time window between symptom onset to critical event is very short.²²,²³ Unlike other models,²⁴,²⁵,²⁶,²⁷,²⁸ our models do not include radiographic or laboratory findings as prediction variables. In contrast to fully equipped higher-level facilities, quarantine facilities may not have advanced medical equipment,²⁹ and underdeveloped areas may not have any medical facilities at all. Second, our web-based application can facilitate prognostic evaluation in a large group of patients. As the number of patients with COVID-19 increases, it is difficult to predict each patient’s prognosis individually given limited resources. A lack of efficient triage can lead to higher mortality rates, particularly in areas with a sudden upsurge of COVID-19 cases.³⁰,³¹ Given that our models are web based and calculate risk automatically with data provided directly from patients, large-scale risk calculation is possible. Third, our models can be used for telemedicine in the era of COVID-19.³² A previous study has reported that nearly half of all patients with COVID-19 are treated on an outpatient basis.³³ Even when patients are hospitalized, time lag exists between symptom onset and hospitalization,³⁴,³⁵ leaving the possibility of acute patient deterioration before active in-hospital management. Our models can be applied for active monitoring of patients who are on home quarantine and identify those at higher risk of deterioration and requiring early hospitalization.

Among the patients who required supplementary oxygen, oxygen scores increased within 7 days of symptom onset, but decreased thereafter. This can be partly explained by the general supportive care given after hospitalization. Among the 12 patients included in the serial analysis of oxygen score, all 12 patients (100.0%) received antipyretics, and 11 patients (91.7%) received antitussives. Even among the other 29 patients who did not need supplementary oxygen, 23 patients (79.3%) received antipyretics, and 18 patients (62.1%) received antitussives. Considering that our models do not include laboratory or radiographic findings, the variables that can change daily are limited to body temperature and subjective symptoms. Such variables are prone to change according to the extent of general supportive care. However, the median interval from symptom onset to oxygen supplementation in this study was 7.5 days, similar to other previous studies.³⁶,³⁷ Therefore, a serial measurement of oxygen score can be helpful for preemptive identification of patients who may need oxygen supplementation.

The discrimination performance of our models seemed slightly lower than those calculated from a previous retrospective cohort (Supplementary Fig. 2, only online),¹⁴ which can be attributed to the following reasons: first, there is possible selection bias in the patients included in this study. The COVID-19 unit of our center was a temporary unit capable of simple supportive care, and the number of healthcare providers was limited. Accordingly, it could not accommodate patients with severe underlying comorbidities, such as cancer, chronic lung disease, or dementia, or those dependent on others for their daily activities. Also, patients not capable of using smartphones could not participate in our study. Second, information uploaded directly by patients may differ from those defined by healthcare providers. Patients uploaded their information directly via a web-based application in this prospective cohort (Fig. 1A),¹⁵,¹⁶ unlike the data collected from the derivation cohort, which were uploaded by attending physicians.¹⁴

We can make several recommendations from this study. First, patients with higher scores should be prioritized for transfer to higher-level facilities. Although an exact score to predict short-term outcomes has yet to be established, our models have proven to be efficacious as good decision-support tools when active monitoring is impossible due to shortage of medical resources or manpower. Second, patients should record their daily status for up to 7 to 10 days after symptom onset. Our study revealed a median interval of 9 days (IQR 8–11 days) between hospitalization and ICU admission, and a median of 7.5 days (IQR: 6–9.5 days) between symptom onset and oxygen supplementation. These durations are similar to those in previous reports²²,³⁶,³⁷ and also match with trends in daily changes in ICU scores and oxygen scores in our study. If scores remain low during the first 10 days, we can carefully expect a mild disease course without the need for oxygen supplementation or intensive care. In addition, day of symptom onset seems to be a better baseline timepoint than day of hospitalization for repeated measures.

Despite these advantages, our study also has some limitations. First, the attending physician was not blinded to the calculated scores during the study period. However, all patients received standard care regardless of the calculated scores. Second, because this was a pilot study, the number of patients was limited. Further studies with larger sample sizes are needed to establish the validity of the model and to determine its usefulness for reducing overall mortality rates among patients with COVID-19.

As a pilot study, our models show fair discrimination performance for identifying patients at risk of adverse outcomes and may need intensive care and oxygen supplementation. Thus, they can be useful for predicting patient outcomes and for patient monitoring during the early disease course. Further validation studies are needed to draw a complete conclusion.

SUPPLEMENTARY MATERIALS

Supplementary Fig. 1

Example pages of the online application using predeveloped machine learning models. (A) Patient application. Patients can provide their baseline information along with daily symptoms and body temperature to the web-based application. (B) Physician application. Physicians can inspect their patients' daily status with calculated oxygen scores or intensive care unit scores, in detail.

Supplementary Fig. 2

Discrimination performance of the predeveloped machine learning models in a separate retrospective cohort. (A) ROC curve of ICU score to predict the need for intensive care within 30 days of hospitalization in the derivation group. The AUC was 0.897 (95% CI: 0.877–0.917), which was significantly higher than that of CURB-65 (0.836 with 95% CI: 0.825–0.847). (B) ROC curve of ICU score to predict the need for intensive care within 30 days of hospitalization in the validation group. The AUC was 0.885 (95% CI: 0.855–0.915), which was significantly higher than that of CURB-65 (0.843 with 95% CI: 0.829–0.857). (C) ROC curve of oxygen score to predict the need for oxygen supplement within 30 days of hospitalization in the derivation group. The AUC was 0.886 (95% CI: 0.872–0.900). (D) ROC curve of oxygen score to predict the need for oxygen supplement within 30 days of hospitalization in the validation group. The AUC was 0.865 (95% CI: 0.842–0.888). ROC, receiver-operating characteristics; ICU, intensive care unit; AUC, area under the receiver-operating characteristics curve; CI, confidence interval.

Supplementary Table 1

Variables Included for Developing the Machine Learning Models

Supplementary Table 2

Feature Importance to the ICU Score (Gradient Boosting Machine Learning Model) for the Prediction of Patients Requiring Intensive Care

Supplementary Table 3

Feature Importance to the Oxygen Score (Gradient Boosting Machine Learning Model) for the Prediction of Patients Requiring Oxygen Supplement