Development and validation of a machine learning based prediction model for near-term in-hospital mortality among COVID-19 patients

Abstract

Objectives:

To develop and validate a model for prediction of near-term in-hospital mortality among COVID-19 patients by application of a machine learning (ML) algorithm on time-series in-patient data from electronic health records.

Methods:

A cohort comprised of 567 COVID-19 patients at a large acute care healthcare system between February 10-April 7, 2020, observed until either death or discharge. Random forest (RF) model was developed on randomly drawn 70% of the cohort (training set) and its performance was evaluated on the rest of 30% (the test set). The outcome variable was in-hospital mortality within 20 to 84 hours from the time of prediction. Input features included patients’ vital signs, laboratory data, and electrocardiogram results.

Results:

Patients had a median age of 60.2 years (interquartile range 26.2 years); 54.1% were male. In-hospital mortality rate was 17.0% and overall median time to death was 6.5 days (range 1.3 to 23.0 days). In the test set, the RF classifier yielded a sensitivity of 87.8% (95% CI: 78.2%−94.3%), specificity of 60.6% (95% CI: 55.2%−65.8%), accuracy of 65.5% (95% CI: 60.7%−70.0%), area under the receiver operating curve of 85.5% (95% CI: 80.8%−90.2%), and area under the precision recall curve of 64.4% (95% CI: 53.5%−75.3%).

Conclusions:

Our ML-based approach can be used to analyze electronic health record data and reliably predict near-term mortality prediction. Using such a model in hospitals could help improve care, thereby better aligning clinical decisions with prognosis in critically ill COVID-19 patients.

Introduction

The surge in hospitalizations and ICU admissions due to patients with severe COVID-19 disease¹ has shown the need for effective prognostication, so that clinicians working with limited resources can formulate appropriate goals of care based on patients’ varying risk of deterioration. Timely and targeted delivery of palliative care services is an important component of COVID-19 management. With such data, providers could help manage symptoms of severe infection and foster shared decision-making with patients and families well before clinical deterioration and death. However, the appropriate timing of initiation of goals-of-care and/or palliative care consultations is difficult to gauge given the largely unpredictable disease trajectory of COVID-19 and how suddenly patients’ conditions can deteriorate². Age and pre-existing high-risk conditions³ can be baseline predictors of mortality. However, a number of COVID-19 patients without underlying high-risk conditions have needed hospitalization, required ICU care, or died (26%, 23%, and 5%, respectively)⁴. These data indicate that using baseline risk factors to assess mortality risk may have limited clinical utility in the context of COVID-19.

Manual assessments are performed to evaluate patients’ overall clinical condition, assess the need for interventions, and identify those at higher risk of poor prognosis. As an aid to the manual assessments, score-based approaches^5–7 have been proposed to improve the process of patient prognostication. However, validity of these approaches remains to be established for COVID-19 hospitalizations. Moreover, the repeated elicitation of scores during hospitalization can be laborious. Supervised machine learning can provide an opportunity to frequently assess large number of relevant variables, their temporal changes and the known as well as unknown interactions among variables with respect to the prognostic outcome.

We aimed to develop a novel supervised machine learning (ML)-based prediction tool to help clinical teams identify COVID-19 inpatients at higher risk of near-term in-hospital mortality, and to assist the palliative care clinicians in determining when to hold emotionally charged conversations regarding prognosis and care for these patients. We used inpatient time-series data from the institution’s electronic health record (EHR) system and applied a random forest (RF) approach. Here we describe the development, validation, and interpretation of this model.

Materials and Methods

The study was approved by the Institutional Research Board, which waived the need for informed consent.

Data Source

We compiled retrospective cohort data from the Mount Sinai Health System COVID-19 registry, which included admission-discharge-transfer events, administrative data, time-series data of clinical assessments, and laboratory and electrocardiogram (ECG) results. Our study complies with the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guideline for development and validation of prediction models.

Study population

The cohort included adults ≥18 years old with a COVID-19 diagnosis who were admitted to the Mount Sinai Hospital between February 10-April 7, 2020. We defined a COVID-19 diagnosis based on either of the following criteria: a) Positive detection of SARS-CoV-2 by reverse transcriptase-polymerase chain reaction assay, or b) Clinical diagnosis of COVID-19 made by an infectious disease specialist.

Selection of variables

Data compiled for this study included patient demographics (e.g. age, sex); relevant hospital administrative variables (e.g. admission type, source of admission); data from nursing flowsheets (e.g. vital signs, respiratory pattern); relevant laboratory results; and ECG-derived variables (e.g. P wave axis, PR interval, or QRS duration). We sought to train a model with the best discriminatory ability based on all clinically relevant variables. Variables used in this study were identified from previously published COVID-19 and critical care evidence.

Sampling strategy

For observational variables, we used the 3 most recent recorded assessments from time-series data that were available when each feature vector was created. For each patient, we generated daily feature vectors starting from admission date until the date of discharge or death.

Patterns of missingness and imputation

The missingness (i.e. all three observations within a feature vector were missing) of laboratory variables ranged between 53.5–89.0%. Missingness is largely an indicator that a particular test was not considered necessary and/or relevant by clinical judgment. For each numerical variable, we used median values of non-missing data across the cohort for imputing missing values. For discrete variables, missing values were retained.

Primary outcome

The primary outcome of interest was in-hospital death within 20–84 hours from the time of prediction. This outcome was translated into a label for training the model using discharge disposition and time of feature vector creation. The interval between the time of discharge and the time of generating each feature vector was generated daily for each patient. If the discharge disposition was “Expired (i.e. dead)” and the interval was between 20 to 84 hours, we labeled the feature vectors as positive. If the discharge disposition was “Not Expired” and the interval was between 20 to 84 hours, we labeled the feature vectors negative. We excluded the remaining feature vectors from our cohort.

The labeling strategy is illustrated in Figure 1.

Figure 1:

Feature vector labeling strategy. (a) Positive labels; (b) negative labels. V1–3: Values of observations used for creating the feature vector.

Justification of time window of outcomes

For pragmatic reasons, we defined near-term outcomes as those occurring between 20 and 84 hours from the prediction time. Approximately a day (20 hours) would allow time for providers for manual assessment by clinicians, trying interventions to prevent further deterioration, and for performing the goals-of-care and/or palliative care consultations to develop an individualized plan of care after their clinical assessment. The three days horizon (72 hours) was extended by 12 hours for operational reasons of accommodating a complete day at a hospital until evening.

Model training and development

Cohort data were randomly split into training (~ 70 percent) and test (~ 30 percent) sets. Because the distribution of positive and negative feature vectors was imbalanced (17.9% vs. 82.1%), we performed random under-sampling on the majority class in the training set until both classes became balanced. An RF algorithm was chosen for training the model with 10-fold cross-validation, using the open source Apache Spark project machine-learning library. The resultant fitted model was then applied to the test set.

Importance of variables

Features included in the final RF model were ranked using the Gini importance criteria. In this study, each continuous variable had three features, representing the three most recent observations. We calculated the overall importance of each variable by the aggregated sum of Gini importance values of all its underlying features.

Statistical analysis

A default threshold of 0.5 was used on model-derived class probabilities for assigning positive and negative predictions. Model performance statistics included assessment of sensitivity, specificity, accuracy, and positive and negative predictive values. We also plotted receiver operating characteristic (ROC) and the precision recall (PR) curves, area under the ROC curve (AUROC), and area under the PR curve (AUPRC). A 95% confidence interval of all performance statistics was obtained. Performance metrics were computed in the R environment by using custom scripts and the R packages – PRROC (v.1.3.1), pROC (v. 1.16.9000) and epiR (v. 1.0.4).

Clinical utility, need assessment, temporal validation

We applied our model to 910 individuals hospitalized with COVID-19 diagnosis at our institution during the study period and identified those who died in the hospital. RF model predictions were obtained prospectively for each day a patient stayed in the hospital. Positive predictions with high risk of near-term mortality were identified using a threshold of 0.5. Among hospitalizations that ended in death, we identified a subset of patients whom our model predicted as being at high risk of near-term mortality.

We queried the EHR system for documentation of COVID-19 patients seen by a palliative care service provider, and then categorized this subset (with patients at high model-predicted mortality risk) into two subgroups: (a) patients who received a goals-of-care and/or palliative care consultations before death, either in the Emergency Department before admission or during their inpatient stay; and (b) patients who did not receive either type of consultation before death. From group (b), we identified a subset who could have benefited from such as consultation, based on review of case notes performed by a clinician. These patients who were predicted by our tool as having high risk of mortality, are considered as having an unmet need for goals-of-care and/or palliative care consultation.

Results

The model for this cohort consisted of 1,360 feature vectors, representing in-patient data from 567 unique individuals hospitalized at our institution with COVID-19. When split into training and test sets, and after under-sampling of the training set, the two sets had 338 and 414 feature vectors, respectively.

Cohort characteristics

Patient characteristics at baseline are shown in Table 1. In both the study cohort and test set, more patients were males and the highest proportion of patients were in the 45- to 65-year age group. In-hospital mortality rates in the overall cohort and test set were similar, as was the median time to death. No significant differences were observed between the overall and test sets in distributions of race, ethnicity, relevant comorbidities, smoking habits, patients’ residential origin, and proportion of patients who received ICU care.

Table 1:

Characteristics of patients admitted in the study cohort and test set

	Overall (N=567)	Test set (N=323)
Age
Mean (SD)	59.4 (17.9)	59.3 (17.8)
Median [Min, Max]	60.2 [20.1, 97.3]	61.1 [21.2, 97.3]
Age Groups
18–45	139 (24.5%)	81 (25.1%)
45–65	203 (35.8%)	115 (35.6%)
65–80	151 (26.6%)	87 (26.9%)
> 80	74 (13.1%)	40 (12.4%)
Gender
Male	307 (54.1%)	175 (54.2%)
Female	258 (45.5%)	147 (45.5%)
Missing	2 (0.4%)	1 (0.3%)
Length of Stay
0–3 days	143 (25.2%)	54 (16.7%)
3–10 days	332 (58.6%)	207 (64.1%)
> 10 days	92 (16.2%)	62 (19.2%)
Died during hospitalization
Yes	101 (17.8%)	56 (17.3%)
No	466 (82.2%)	267 (82.7%)
ICU care received
Yes	97 (17.1%)	55 (17.0%)
No	470 (82.9%)	268 (83.0%)
Geographic origin
Manhattan	272 (48.0%)	150 (46.4%)
Queens	85 (15.0%)	52 (16.1%)
Brooklyn	89 (15.7%)	54 (16.7%)
Bronx	59 (10.4%)	30 (9.3%)
Staten Island	3 (0.5%)	1 (0.3%)
Out of New York City	46 (8.1%)	29 (9.0%)
Out of NY State	10 (1.8%)	6 (1.9%)
Missing	3 (0.5%)	1 (0.3%)
Race and Ethnicity
Non-Hispanic Black	115 (20.3%)	74 (22.9%)
Non-Hispanic White	127 (22.4%)	69 (21.4%)
Hispanic Black	18 (3.2%)	7 (2.2%)
Hispanic White	19 (3.4%)	11 (3.4%)
Asian	23 (4.1%)	14 (4.3%)
Others	223 (39.3%)	128 (39.6%)
American Indian	1 (0.2%)	0 (0%)
Pacific Islander	1 (0.2%)	0 (0%)
Unspecified	40 (7.1%)	20 (6.2%)
Smoking history
Current Smoker	21 (3.7%)	12 (3.7%)
Past smoker	133 (23.5%)	76 (23.5%)
Never smoked	315 (55.6%)	182 (56.3%)
Unknown	96 (16.9%)	52 (16.1%)
Missing	2 (0.4%)	1 (0.3%)
Hypertension
Yes	178 (31.4%)	103 (31.9%)
No	387 (68.3%)	219 (67.8%)
Missing	2 (0.4%)	1 (0.3%)
Diabetes
Yes	124 (21.9%)	74 (22.9%)
No	441 (77.8%)	248 (76.8%)
Missing	2 (0.4%)	1 (0.3%)
COPD
Yes	20 (3.5%)	13 (4.0%)
No	545 (96.1%)	309 (95.7%)
Missing	2 (0.4%)	1 (0.3%)
Obesity
Yes	65 (11.5%)	37 (11.5%)
No	500 (88.2%)	285 (88.2%)
Missing	2 (0.4%)	1 (0.3%)
Chronic kidney disease
Yes	58 (10.2%)	32 (9.9%)
No	507 (89.4%)	290 (89.8%)
Missing	2 (0.4%)	1 (0.3%)

Predictors and their importance

Data included in this study were basic demographic and hospital variables, structured clinical assessments including vital signs⁷, complete blood count, serum biochemical tests⁸, coagulation profile⁹, parameters for respiratory function and other complications¹⁰, markers of inflammation^11,12, and electrocardiography parameters¹³. Fifty-five variables (comprising 175 features) were included in the final version of trained RF model (eTable 1). The model hyperparameters used for training the best cross-validated model are listed in eTable 2.

Implicit feature selection performed by the RF algorithm reflected by non-zero Gini importance values recognized 49 variables corresponding to 99 features accounted in the final model (eFigure 1).

Predictive performance of the model

Predictive performance of the model is shown in Table 2. Receiver operating characteristics and precision recall curves of the RF classifier model are shown in Figure 2.

Table 2:

Predictive performance of near-term mortality by random forest classifier

	Number of feature vectors	# samples labeled positive for near-term mortality (%)	Sensitivity (%)	Specificity (%)	Accuracy (%)	Positive Predictive Value (%)	Negative Predictive Value (%)	Area under ROC (%)	Area under PR (%)
Test set	414	74(17.9)	87.8(78.2–94.3)	60.6(55.2–65.8)	65.5(60.7–70)	32.7(26.2–39.7)	95.8(92.2–98.1)	85.5(80.8–90.2)	64.4(53.5–75.3)

ROC: area under receiver operating characteristic curve; PR: area under precision recall curve.

Figure 2:

Receiver operating characteristics and precision recall curve of the RF classifier model in our test set.

Assessment of use and unmet need of goals-of-care/palliative care consultation

Our model predicted 95.2% of in-hospital deaths; among these, 11.6% of patients did not receive a goals-of-care or palliative care consultation. Among those who did, 65.8% received the consultation during their in-patient stay and 15.1% received it in the Emergency Department. A clinician’s manual review of case notes determined that 53% of those who did not receive a goals-of-care and/or palliative care consultation were appropriate candidates for it, representing 6.2% of the entire cohort (Figure 3).

Figure 3:

Assessment of use and potentially unmet need for goals-of-care or palliative care (GOC/PC) consultation among patients with COVID-19 who died in the hospital (6.2% of all deaths in the cohort), shown as red slice in center.

Discussion

We developed a RF-based model for predicting near-term in-hospital mortality of adult patients with COVID-19 by using time-series inpatient data. The model provided adequate discrimination (AUROC 85.5%) without the need for manual preprocessing of data. In contrast to using static variables, our model translates the variability in patients’ conditions into mortality risk predictions.

Modeling of near-term mortality with dynamic risk quantification should incorporate variables that capture the progression of COVID-19 along the common pathways underlying mortality, such as respiratory or multi-organ failure, septic shock, and cardiogenic shock from acute myocardial injury and myocarditis¹⁴. Unlike other studies^15–18 of predictive models for mortality among COVID-19 patients, ours is the first model to demonstrate feasibility of using multiple clinical variables as time-series data from the EHR. The narrow prediction time-window captured by this model is a key in scenario of acute illness with high risk of in-hospital mortality, as seen with the COVID-19. Some limitations of other predictive models of ICU and in-hospital mortality^7,19–24 include accounting for patient characteristics upon admission or access to limited variables from inpatient data. Our model addresses these limitations and accounts for the context of COVID-19 management during the pandemic.

The prediction probability generated by the model can be calibrated according to the expected mortality rate. While uncertainty of prognostication in COVID-19 can be a reason of missing the early opportunities of goals-of-care/palliative care consultations, the near-term estimated risk of mortality can be used to create alternative prognostic risk scenarios in order to build the goals-of-care and palliative care consultations with increased confidence. During the COVID-19 crisis, when critical care team may include re-deployed personnel, not specialized in critical care, the prognostication by ML-based model can assist the clinicians. The goals-of-care consultations are performed by any care team member and aim at ensuring that the care team is aware of patients’ values and preferences and that these preferences can be incorporated into hospital care, given the current clinical condition. It involves discussion of prognosis, goals, and noting any patient concerns²⁵ and shared decision on the use of life supporting interventions^26,27. The palliative care consults are performed by the palliative care specialists and aim at delivering a comprehensive, inter-disciplinary care with a focus on improving the quality of life for patients in imminent or established critical condition. These consultations can include goals-of-care discussions. In addition to helping the patients, it also helps to provide much needed to support to patient’s family members ²⁸.

The model variables of interest

Apart from conventional static prognostic variables, such as age¹ emerging as a top variable of significance, variables reflecting renal function status (e.g. blood urea nitrogen and serum creatinine) appeared significant, emphasizing the importance of pre-existing chronic kidney disease, abnormal renal function, and acute kidney injury in COVID-19^29,30 and its association with mortality²⁹. Among the arterial and venous blood gas analysis variables, the model ranked anion gap and PaCO₂ higher than markers of hypoxemia (PaO₂ and PvO₂), supporting the significance of respiratory acidosis³¹ and hypercapnia in those who died from COVID-19 despite of the improvement in oxygenation^32,33. Key markers of inflammation - C-reactive protein and complement C4 together with the markers of coagulability - D-dimer, platelet count and activated partial thromboplastin time (APTT) being predictive in our model, is consistent with the suggestion of severe COVID-19 being coagulopathy with severe immune-inflammatory state³⁴. Predictive value of ECG variables, such as the axis of T wave, PR interval, indicates importance of cardiac complications in COVID-19 in relation to patient outcome^35,36. Contradictory to the expectation, the N-terminal pro-B-type natriuretic peptide (NT-proBNP), a predictor of mortality in COVID-19³⁷, ranked low in our model. Among the hematologic variables, lymphocyte count, appeared to be predictive. Lymphocyte count has a significance in COVID-19 prognostication¹¹ and COVID-19-associated sepsis³⁸.

We chose a random forest algorithm because of its ability to handle complex multi-modal clinical data and elucidate high-order interactions among input variables without compromising the predictive accuracy³⁹. As another strength of our model, the data elements used in our model include commonly used variables in the clinical management of COVID-19.

Limitations of the model

A small subset of patients was involved in both training and test sets, introducing the potential for contamination between the two sets. However, in general, the daily feature vectors originating from the time-series data differ, even for the same patient encounter. The model requires validation in external and prospective settings, given the variability in various aspects such as patient demographics, care resources, and protocols for disease management. The treatment and intervention variables can have potential implications on the disease trajectory and mortality. However, the current version of the model does not include treatment/intervention variables and therefore, generalizability can be limited given the variability in treatment and intervention guidelines by institutions. The model was trained on COVID-19 hospitalizations regardless of patients’ current level of care; in actual practice, the frequency of obtaining clinical assessments may vary by the level of care. Due to our small sample size, we could not train separate models for each specific level of care or age group. Such customization, however, could elucidate more predictors of near-term mortality. The estimated unmet need for goals-of-care or palliative care consultations is based on a small cohort. Future work is necessary to evaluate the impact of the model’s implementation on the coverage of palliative care and other aspects of clinical workflow.

Practice implications

A tool that accurately predicts mortality risk and is incorporated in the EHR, can complement the manual clinical review and help the care providers in various ways. These include expanding the coverage of appropriately timed goals-of-care or palliative care consultations, and escalation or de-escalation of care for facilitating efficient resource utilization.

Our tool predicted 95% of all COVID-19 hospitalizations associated with in-hospital deaths in the prospective cohort. We also observed that more than half of hospitalized patients who were clinically eligible for goals-of-care or palliative care consultations did not receive them. Prognostication augmented with our model-derived predictions could potentially address this unmet need. At our institution, palliative care services are embedded within various departments and were often proactively provided to the high-risk COVID-19 patients. The extent of unmet need of these consultations could be considerably greater at centers with reduced availability of palliative care services, further highlighting the importance of ML-based mortality prediction.

Our model’s high negative predictive value (95.8%, 95% CI 92.2–98.1%) in assessing near-term mortality risk could be helpful for guiding providers when hospital or staffing resources are strained. Among other potential implications, our model could facilitate timely escalation of care, which can help to reduce the rate of invasive mechanical ventilation⁴⁰. This could be especially important in situations where such resources are limited.

In a fast-paced COVID-19 crisis, increased needs of trained manpower along with the need of critical care resources require careful planning. Acute need of critical care resources can be forecasted by using the near-term prognostication by the model together with patients’ critical care preferences. This can help hospitals plan not only for the imminent surge in demand of care but can also assist in making decisions on new admissions, transfers, logistics and staffing.

Conclusion

The ability of our proposed model to predict the near-term mortality in COVID-19 patients demonstrates its potential to be adapted according to clinical utilities, including its utility for the palliative care services.

Supplementary Material

4

eFigure 1: Model variables ranked by their non-zero Gini importance values.

Data Availability statement

Raw data were generated at Mount Sinai Health System. Derived data supporting the findings of this study are available from the corresponding author (MM) on request.