Survival models and longitudinal medical events for hospital readmission forecasting

Abstract

Background

The rate of 30-day all-cause hospital readmissions can affect the funding a hospital receives. An accurate and reliable readmission prediction model could save money and increase quality-of-care. Few projects have explored formulating this task as a survival prediction problem, where models can exploit a real-valued time-to-readmission target. This paper demonstrates the effectiveness of a survival-inspired readmission model, especially when paired with a longitudinal patient representation that is agnostic to disease-cohort and predictive task.

Methods

We forecast readmissions for a population-level cohort of 421,088 patients discharged in 2015 and 2016 from hospitals in Alberta, Canada. Clinical features and sequences of historical medical codes (calculated from at least four full years prior to discharge) from linked administrative sources serve as model inputs. We trained binary 30-day readmission models (XGBoost and a Deep Neural Network) and time-to-event readmission models (CoxPH and N-MTLR) with and without machine-learned medical knowledge at initialization, then compared against the popular LACE-based model using the AUROC score at 30 days (AUROC@30). Survival models are additionally evaluated using concordance, Integrated Brier, and L1-loss scores.

Results

All models that utilize sequence features markedly out-perform even the best models trained on only clinical features. Further, a time-to-event target improves predictive performance at 30 days, given the same model inputs and architecture. N-MTLR, using solely sequence inputs and initialized with pre-learned medical knowledge, achieves an average AUROC@30 of 0.8460 over five folds with a standard deviation of 0.003. All trained models match or out-perform the LACE baseline of 0.6587±0.003.

Conclusion

Sequences of administrative medical codes contain rich predictive information for forecasting readmissions, and embedding medical knowledge a priori using machine learning provides readmission models an advantageous foundation for training. When combined with a model that can leverage a time-to-event target, excellent performance is possible on the 30-day all-cause readmission task using only administrative data.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-024-11771-w.

Introduction

As of the early 2010s, hospital readmissions cost approximately two billion Canadian dollars per year in Canada [1] and 26 billion US dollars per year in the United States [2]. In the US, the Centers for Medicare & Medicaid Services financially penalize hospitals with high all-cause 30-day readmission rates [3]. Studies estimate that anywhere from 5-79% of readmissions are avoidable (median 27.1%) [4]. This has motivated many attempts to produce a reliable readmission prediction system, as insight into a patient’s future following a hospital stay (index admission) could help better identify who should receive expensive transitional care interventions, which could prevent rapid rehospitalization events. In other words, a dependable readmission prediction system has the potential to decrease expenditures related to readmissions and relieve suffering.

To predict whether a readmission will occur on or before 30-days post-discharge, a common practice is to capture or engineer features that are predictive of this outcome. The LACE index [5], for example, seeks to summarize a patient’s risk of hospital readmission in a single scalar; however, the simplicity of this approach (and others, such as the HOSPITAL score [6]) often leads to lackluster performance compared to more recent methods. Many other publications have shown that certain risk factors (such as hospital-acquired Clostridium difficile infection [7], cancer, pulmonary, liver, and kidney disease [8], maintenance chemotherapy, and $\ge$16 abnormal laboratory test results [9], etc.) are predictive of 30-day all-cause readmissions; however, the process of discovering, engineering, and validating these features is time-consuming and often requires domain- and target-specific knowledge.

Health Service Providers (HSPs) use a rich vocabulary of codes to track patient encounters, diagnoses and procedures. This falls under the category of “administrative” medical data, which is generally considered less sensitive than other types of electronic health information. This sequence of codes details an individual’s history with HSPs, much like natural language words convey meaning in an article or book. To improve interpretation and representation, AI researchers often use approaches like Word2Vec [10] to encode a vocabluary of words of natural language sentences as dense vectors. Word2Vec was initially applied to this medical task in 2016 [11, 12], and many medical-code-specific approaches have since followed [13, 14]. Previous researchers found that simply adding the dense vectors of recent medical codes could effectively represent patients for predicting heart-failure-specific [12] and all-cause [15] 30-day readmissions. Tools designed to deal with sequential data—such as LSTMs [16–19], GRUs [18, 20–24], Convolutional Neural Networks [25–27], attention-based architectures [18, 28, 29], and others [16, 30, 31]—have also been used to predict health outcomes from sequences of medical codes, achieving varying degrees of success. This detailed yet disease-agnostic way of representing someone’s history is applicable to many different predictive tasks, and using embeddings of even generic life events (e.g., related to education, occupation, health) is demonstrably useful for forecasting diverse outcomes [32]. Initializing deep learning sequence models with dense embeddings of medical codes has previously improved readmission prediction performance [21, 27].

The information provided by the binary target label of “30-day readmission” is coarse. If we instead consider the more general time-until-readmission as the target, a model can learn a more fine-grained relationship between the input features and the adverse outcome of interest, potentially improving performance. Any occurrence (e.g., death, leaving the study) that prevents us from measuring the adverse outcome of interest (here, a hospital readmission) is called a “censorship”, specifically right-censorship (Fig. 1). Survival analysis and survival prediction models are used to understand how the future of an “instance” (e.g., patient, machine, business) may unfold with respect to some event of interest (e.g., death, mechanical failure, bankruptcy) [33], and are formulated specifically to deal with these censoring events (Fig. 2a).

Fig. 1.

Censorship illustration with three example patients, each starting with their respective index hospitalization. Patient 1 is uncensored, as we understand their time-until-event (green-shaded area). Patients 2 and 3 are both censored, as Patient 2 moved away before experiencing an event, and Patient 3 remained event-free until the end of the study. Their respective time-until-censorships are shown using the red-shaded boxes. Some patients may never experience an adverse outcome

Fig. 2.

Survival prediction and individual survival distributions. a A survival dataset is used to train a survival model, which can in turn provide insights into the future of a new patient. t represents the time-to-event, and $\delta$ is the censoring bit. b The structure of an Individual Survival Distribution (ISD). The x axis is measured in time, and the y axis denotes the probability of reaching that time event-free. The dotted lines show the prediction that this patient has a 50% chance of no readmission for 340 days; this is their median survival time. We can use such a “median survival time” as our prediction for that patient’s time-to-event, or (if desired) use the survival probability between two different points (e.g., the times associated with the probabilities 0.25 and 0.75) to report a prediction confidence interval

There are many classes of survival algorithms, including models like Cox Proportional Hazards [34] (that simply produce a risk score, described below) and the Gail model [35] (that estimate the probability of remaining adverse-event-free until a single specific time, such as five years). However, this study focuses on algorithms that model Individual Survival Distributions (ISDs), which describe the probability that an event D will occur for an individual i—given a vector of their attributes ${\overrightarrow{x}}^{(i)}$—at or after each future time-point t: $P(D\ge t|{\overrightarrow{x}}^{(i)})$ (Fig. 2b). Multi-Task Logistic Regression (MTLR, one such ISD-generating algorithm) [36, 37] has been used to model length-of-hospital-stay for COVID-19 patients [38], time until readmission for COVID patients using ECG inputs [39], cardiovascular-related hospitalizations for hypertensive individuals [40], breast cancer onset [41], Alzheimer’s disease progression [42], and many other types of medical events.

Survival methods have been applied to the problem of readmission prediction in the past [43–46]; however, few publications have harnessed both the patient-document analogy and algorithms that model individual time-to-event probability distributions. In 2022, Kalmady and colleagues used Med2Vec [13] representations of medical events and N-MTLR [37] to model ISDs for heart-failure-specific hospital readmissions [47]. However, the Med2Vec hospital visit embeddings used to predict readmissions contained only information from the index admission, ignoring the events of an individual’s previous medical history. To address these gaps, we propose a novel architecture: an ISD-generating survival algorithm informed by a sequence model, capable of learning time-to-event information directly from a progression of past medical codes. We call this model Sequence-MTLR, or S-MTLR.

The objective of this paper is to demonstrate the following:

1. Modeling the probability of an event at all future times can improve performance on predictions of readmissions at a particular time-point—in this case, 30 days—compared to models trained specifically to predict at that time point.

2. One can effectively learn S-MTLR models that can use a patient’s past information (from many previous time points) to predict ISDs and 30-day readmission status.

3. Initializing readmission models with prior medical knowledge—extracted without the use of training labels—can improve downstream performance.

Methods

We consider multiple machine learning model types for forecasting all-cause readmissions at 30 days, differing by whether (1) sequential medical events are (versus are not) used as input (and if yes, whether a Word2Vec-based Medical Concept Embedding Dictionary is used at initialization), and (2) the target is formulated as binary (readmitted or not within 30 days) or time-to-event (time-to-event and censoring bit). We also consider various learners for different approaches.

Data

This study uses linked administrative health data collected within the province of Alberta. We accessed the following information for all individuals who interacted with Alberta Health Services from years 2011 to 2017:

1. Physician office visits from insurance data (Claims), including visit date and procedure/diagnosis codes from primary care physicians (family medicine), internal medicine specialists, and general surgery specialists

2. Drug prescriptions from Pharmaceutical Information Network (PIN), including prescription date and Anatomical Therapeutic Chemical (ATC) code

3. Ambulatory visits from National Ambulatory Care Reporting System (NACRS), including visit date, emergent status, disposition, diagnosis codes, and procedure codes

4. Hospitalizations from Discharge Abstract Database (DAD), including admit and discharge dates, discharge disposition, diagnosis codes, and procedure codes

We additionally gained access to the individual’s sex, age at discharge, and their first three postal code alpha-numerics. Diagnosis codes were ICD-9 in Claims and ICD-10-CA in DAD and NACRS. Procedure codes in Claims were Classification of Procedures Extended (CCPX) codes and were Classification of Health Interventions (CCI) codes elsewhere. All data were extracted by the Alberta Strategy for Patient Oriented Research (SPOR) SUPPORT Unit. This study and all associated protocols were approved by the Health Research Ethics Board of the University of Alberta (Study ID Pro00082041), and research was conducted in adherence to all relevant guidelines and regulations. Informed consent requirements were waived due to the data’s de-identified nature by the Health Research Ethics Board (Health) Panel.

Index admissions and target calculation

We predict readmissions for patients who were discharged from any Albertan hospital during the index period of January 1, 2015 to December 31, 2016 (Fig. 4). We select valid index episodes (each constituting an entire hospital “event”, potentially including transfers between locations), thereby determining our target population, using the following procedure (Fig. 3A).

Fig. 4.

Study and example patient timelines. The study timeline is split into three periods: the pre-index period for training Medical Concept Embeddings, the index period from which index admissions are selected, and the post-index period to examine at least one year of a patient’s future post-index-admission-discharge. The timeline for each patient is split into two periods: the lookback period of patient history (which includes all information until the index admission discharge) and the time-to-readmission (the period of readmission-free survival). Time-to-readmission is calculated from the discharge of the index admission to the day of the next admission. If time-to-readmission extends beyond the end of the study period, we instead compute time-to-censorship

Fig. 3.

Overview of procedures used for dataset procurement, model training, and model evaluation. A Procedure used to arrive at the final dataset of index admissions and study population. B Details of external cross-validation for evaluation

The set of all records from 2011 to 2017 contained 520,960 unique patients. Admissions from routine hospital admissions for baby births were not included in the initial data extraction. DAD records whose discharge date occurred within 2015 and 2016 were selected, leaving 472,339 unique patients. We removed patients who were associated with at least one record from outside of Alberta (n=21,024) or had an invalid patient identifier (n=408). Patients with only psychiatric admissions (ICD-10-CM diagnosis of F00-F99 except F10-F19, n=16,140) were also excluded (along with all other psychiatric admissions) due to characteristic patterns of readmission separating them from other hospital users. Next, we removed patients whose only hospitalization ended in death (discharge disposition code 07, n=3931). Records for each of the remaining 430,836 patients that were separated by no more than one day (regardless of discharge disposition) were collapsed into admission “episodes”. Episodes ending with further transfers (discharge disposition 01), in-hospital deaths (discharge disposition 07), or a failure to return from pass (discharge disposition 12) were removed from the selection pool, leaving 421,089 patients. Patients who reportedly experienced at least one post-mortality hospital admission ($n=1$) were also removed, leaving 421,088 unique patients. Throughout 2015 and 2016, the same individual may have been admitted and discharged multiple times. Here, our study randomly retrains only one index episode per-individual. Table 1 contains summary statistics for each patient and associated index admission.

Table 1.

Study population summary statistics, divided by 30-day readmission status

	All n = 421088	Not readmitted within 30 days n = 399716	Readmitted within 30 days n = 21372
Variable	Number (%) or Mean [Std.Dev]	Number (%) or Mean [Std.Dev]	Number (%) or Mean [Std.Dev]
Sex (Female)	263200 (62.5%)	251826 (63.0%)	11374 (53.2%)
Age
$<1$	12541 (3.0%)	10432 (2.6%)	2109 (9.9%)
$1-14$	19824 (4.7%)	19162 (4.8%)	662 (3.1%)
$15-24$	33486 (8.0%)	32545 (8.1%)	941 (4.4%)
$25-64$	238950 (56.7%)	230346 (57.6%)	8604 (40.3%)
$>65$	116287 (27.6%)	107231 (26.8%)	9056 (42.4%)
Discharge Disposition
02: Transferred to Continuing Care	10197 (2.4%)	9534 (2.4%)	663 (3.1%)
03: Transferred to Other Facility	2155 (0.5%)	2074 (0.5%)	81 (0.4%)
04: Discharged Home with Support Services	33385 (7.9%)	29730 (7.4%)	3655 (17.1%)
05: Discharged Home, No Support Services	371257 (88.2%)	354684 (88.7%)	16573 (77.5%)
06: Discharged Against Medical Advice or AWOL	4094 (1.0%)	3694 (0.9%)	400 (1.9%)
Length of Stay	7.17 [24.86]	7.02 [24.87]	9.92 [24.48]
Admission Acuity (Emergent Status)	206845 (49.1%)	193650 (48.4%)	13195 (61.7%)
Charlson Comorbidity Index	3.24 [6.56]	3.03 [6.31]	7.22 [9.41]
Emergency Department Usage (6m)	1.0 [2.41]	0.95 [2.29]	1.84 [3.94]
LACE Score	6.53 [4.62]	6.39 [4.54]	9.29 [5.25]

Variables reported correspond to clinical features used as input to machine learning systems (Model inputs section). Each statistic is calculated from one hospital admission per-individual, randomly selected for each patient from candidate admissions in the index period

We define an all-cause readmission event for a singular individual, i, as the first hospital episode whose start date falls in either the index period or the post-index period (Fig. 4) and is at least two days after the discharge date of the index episode. More concretely, this definition allows the models to learn associations between hospitalizations that are not due to psychiatric events or baby births and the first successor hospitalization that is also not due to psychiatric events or baby births.

The label used to train our 30-day binary readmission models is the readmitted bit ($r^{(i)}\in \{0,1\}$). For the readmission-free survival models, the labels required are time ($t^{(i)}\in {\Re }^{+}$, time until event) & event (${\delta }^{(i)}\in \{0,1\}$, censorship status). We begin with the set of all selected index hospital episodes. For every i’s index admission, we calculate the number of days from discharge until the next valid hospital episode from the same patient (TTR: time to readmission), and the number of days from discharge until the end of the study period (TTLTF: time to lost to followup). A standard assumption is that an individual’s TTR and TTLTF are independent when conditioned on their covariates [48]. If no future admissions exist in the study period, TTR is assigned to be $\infty$. The numeric time label $t^{(i)}$ and the binary event label ${\delta }^{(i)}$ are defined as follows:

$$\begin{array}{c}\hfill t^{(i)}=min\{\mathtt{TTR},\mathtt{TTLTF}\}\end{array}$$ 1

$$\begin{array}{c}\hfill {\delta }^{(i)}=\left\{\begin{array}{cc}1,\hfill & \text{if}\mathtt{TTR}\ne \infty \hfill \\ 0,\hfill & \text{otherwise}\hfill \end{array}\right)\end{array}$$ 2

and the binary “30-day readmission” target is defined as follows:

$$\begin{array}{c}\hfill r^{(i)}=\left\{\begin{array}{cc}1,\hfill & \text{if}{t}_i\ge 30\wedge {\delta }_i=1\hfill \\ 0,\hfill & \text{otherwise}\hfill \end{array}\right)\end{array}$$ 3

Note that there are no censorship events captured in these data before 365 days (Fig. 6).

Fig. 6.

The differing distributions of $t^{(i)}$ based on ${\delta }^{(i)}$. Note that in “Time-to-Censorship”, the x-axis starts at 365 days, which is the censoring time of individuals whose index episode discharge coincided with the last day of the index period

Model inputs

Clinical features used for model training (Table 1) include an individual’s sex (M, F, or O), age, and index discharge disposition (the code characterizing the patient’s departure from the hospital). Alongside these “raw” features, we include the LACE score and all the features that were determined to calculate LACE—the length of stay, emergent status of admission (admission acuity), Charlson Comorbidity Index (CCI) [49], and the number of times the patient used emergency services in the previous six months. The index admission “length of stay” is calculated using the admit and discharge dates of the index episode. We extract the the emergent status of the index admission by searching for emergent NACRS records (associated with MIS codes 713100000, 715130000, 715140000, 713102000) from the day before or the day of the beginning of the index episode. The comorbidipy1 library is used to calculate the CCI from DAD data. NACRS is queried to find the number of times an individual was admitted to the emergency department in the six months preceding their index admission. The singular LACE score (ranging from 0-19), is calculated for each patient’s index episode i using the above four features and mappings originally detailed by van Walraven et al. [5].

To construct each person’s “document” of historical medical codes, records with the same patient identifier were extracted from the DAD, NACRS, Claims, and PIN datasets, and sorted by timestamp. These records contained 36,834 unique codes indicating emergent and non-emergent admissions, diagnoses, procedures, prescriptions, and acute care and ambulatory discharge dispositions. Highly specific codes were “rounded” to decrease granularity; we kept only the first three alphanumerics of ICD-9 and ICD-10-CM codes and the first five alphanumerics of ATC and CCI codes, and cleft CCPX codes once place after the decimal. Codes with fewer than 100 occurrences were replaced with the generic code “RAREWORD”. Following suggestions from Choi and Nguyen [20, 21, 27], codes documented more than seven days apart for a single patient were separated by a “timecode”: “0-1m” for fewer than 31 days, “2-3m” for an interval of 31 to 90 days, and “3-6m”, “6-12m”, and “12+m” for periods of three to six, six to twelve, and more than twelve months respectively. Further, long, repeating stretches of codes were often observed in the data; for example, patients with chronic conditions may refill a single prescription on a near-weekly basis for many years. For this reason, continuous single-code repeats between timecodes were replaced with a single instance of the code. We then retrained the 200 most recent codes accumulated before and during the index episode, ending at the day of discharge.

Machine learning and evaluation

We use Word2Vec to develop a dictionary of Medical Concept Embeddings (MCEs) for every medical code (e.g., prescription) and event (e.g., time-skip) that appears with sufficient frequency in the training data patient documents. Our MCEs were trained only on information that was (1) from the pre-index period (Fig. 4), and (2) was not used to determine our final reported performance metrics. For more information about the train/test splits, see Fig. 3B.

The LACE model only uses the single LACE scalar as input, which we implement using the logistic-regression-based procedure described by Damery et al. [50]. XGBoost [51] and CoxPH [34] readmission models take tabular clinical features as input, however, XGBoost learns from the binary target, whereas CoxPH uses the time-to-event target. We extend CoxPH to generate an Individual Survival Distribution using Breslow’s estimator [52]. Our Deep Neural Network (DNN, trained using the binary target) and N-MTLR (trained using the time-to-event target) rely on a multi-pronged architecture, able to take sequence, tabular, or both types of data as input. N-MTLR intrinsically models an ISD, generating predictions for all future timepoints. See Fig. 5 for an overview of model architectures, and Appendix A for further technical details.

Fig. 5.

Architectures and inputs of models used in this study. A LACE baseline model taking the LACE score as input, B XGBoost and CoxPH models taking tabular features as input. The CoxPH risk score is combined with a baseline survival function to generate an individual survival distribution. C DNN and N-MTLR models taking tabular and/or sequence features as input

To evaluate these models’ performance at the 30-day readmission task (note that no censoring has taken place at this time), we report the Area Under the Receiver Receiver Operating Characteristic curve as measured at 30 days, AKA the AUROC@30. AUPRC@30, Brier@30, and other results are defined and reported in Appendix A. Alongside metrics used to evaluate 30-day predictions, survival models are additionally measured by their concordance (using the ISD’s negative median survival time as risk), the Integrated Brier Score (IBS) re-weighted using the Inverse Probability of Censoring Weights [53], and the L1-loss for median survival time using pseudo-observations in the place of censored values [54]. All survival evaluations are carried out using the SurvivalEVAL repository2. Please see Appendix B for more information.

Five-fold external cross validation is used to report all final numbers. The set of index episodes of interest were randomly split into five folds (containing 84,217 or 84,218 patients), each acting as the test set in turn. On each iteration, we consider the other four folds as non-test; we use three of these folds for training and the fourth as validation, which is used to halt training after no detectable decrease in loss for twenty epochs. Performances across all outer folds are averaged, and the standard deviation is examined to understand performance consistency. Fold-wise paired t-tests on AUROC@30 and concordance scores are used to statistically compare performances; claims about model rankings are made only if $p<0.05$. We do not adjust the p-value for multiple comparisons.

Results

Of the 421,088 patients meeting the study inclusion criteria, 62.5% were female and 56.7% fell into the 25-64 age category. 5.08% of individuals (n=21,372) experienced a readmission event ($r^{(i)}=1$) within 30 days. The mean time-to-event ($t^{(i)}$)—either a readmission, or a censorship—is 603.8 days (Std.Dev 296.5 days). The maximum event time is 1095 days, and the minimum event time is 2 days. 71.8% of cases are censored (${\delta }^{(i)}=0$), meaning we observed a readmission event for 28.2% of individuals within the study period. Table 1 reports statistics for the clinical features used as model inputs (broken down by 30-day readmission status) and Fig. 6 shows the time-to-event distributions by censoring status.

The across-fold mean AUROC@30 and standard deviation thereof is used to compare different models and feature sets; Table 2. For other 30-day evaluations, see Appendix B. The baseline LACE model’s AUROC@30 is $0.6587\pm 0.003$. Regarding binary models using Clinical features, XGBoost achieves $0.7546\pm 0.005$, and the Deep Neural Network’s score is $0.7147\pm 0.002$. The AUROC@30 of the Clinical CoxPH survival model is $0.6585\pm 0.002$, and for the N-MTLR survival model using Clinical features, $0.7348\pm 0.004$. Sequence features alone as input contribute to scores of $0.7787\pm 0.005$ (non-MCE DNN), $0.8114\pm 0.007$ (MCE DNN), $0.8392\pm 0.004$ (non-MCE N-MTLR), and $0.8460\pm 0.003$ (MCE N-MTLR). The addition of Clinical features to Sequence features increases the absolute score most when using the binary-target DNN (improvement of 0.0182 without the MCE initialization, and 0.006 with) compared to the time-to-event-target N-MTLR (0.0013 without MCE, 0.0007 with). The best scoring model of the study was N-MTLR with Sequence and Clinical inputs, achieving an AUROC@30 of $0.8467\pm 0.004$ (Fig. 7).

Table 2.

AUROC score evaluations of different combinations of models and feature sets at 30 days. Results are reported as AUROC@30±Std.Dev

	Feature Set
Model	LACE Index	Clinical	Seq	Clinical & Seq
Logistic Regression	$0.6587\pm 0.003$	-	-	-
XGBoost	-	$0.7546\pm 0.005$	-	-
CoxPH	-	$0.6585\pm 0.002$	-	-
DNN	-	$0.7147\pm 0.003$a	$0.7787\pm 0.005$	$0.7969\pm 0.005$
DNN + MCE Init.	-		$0.8114\pm 0.007$	$0.8174\pm 0.006$
N-MTLR	-	$0.7348\pm 0.004$a	$0.8392\pm 0.004$	$0.8405\pm 0.003$
N-MTLR + MCE Init.	-		$0.8460\pm 0.003$	$0.8467\pm 0.004$

^aMCE initialization does not affect non-sequence inputs

Fig. 7.

Comparisons of sequence-accepting models and the LACE baseline. The Deep Neural Network (DNN) uses a binary target, while N-MTLR uses a survival target to model an Individual Survival Distribution. Non-MCE Seq denotes the use of patient documents as one-hot vectors of medical codes. MCE Seq uses the unsupervised Medical Concept Embedding dictionary to represent codes as dense vectors to capture domain knowledge

To evaluate the quality of the time-agnostic readmission curves from our survival algorithms, we additionally report the time-independent concordance, Integrated Brier Score, and the L1-loss with pseudo-observations (Table 3). The Cox Proportional Hazards model only takes Clinical features as input, and achieves a concordance score of $0.6652\pm 0.001$. The concordance for N-MTLR with the same feature set is $0.6937\pm 0.003$, which increases to $0.7458\pm 0.004$ when using Sequence features alone and $0.7504\pm 0.004$ when allowed to learn from both Sequence and Clinical features. The IBS for models that only use Clinical features are $0.1526\pm 0.001$ (CoxPH) and $0.1514\pm 0.001$ (N-MTLR), whereas these scores improve to $0.1319\pm 0.001$ and $0.1313\pm 0.001$ when N-MTLR employs only Sequence inputs and Sequence+Clinical inputs respectively. N-MTLR achieves L1-loss scores around the range of 1100 and 1200 days, and the L1-loss of CoxPH is 1425.9 days with a standard deviation of 142.4 days. For discussions of 1-calibration and D-calibration tests, see Appendix B.

Table 3.

Comparisons of survival models using three metrics

		ISD-Appropriate Metric
Model	Feature Set	Concordance	IBS	L1-PO
CoxPH	Clinical	$0.6652\pm 0.001$	$0.1526\pm 0.001$	$1425.9\pm 142.4$
N-MTLR + MCE Init.	Clinical	$0.6937\pm 0.003$	$0.1514\pm 0.001$	$1183.3\pm 16.9$
	Seq	$0.7458\pm 0.004$	$0.1319\pm 0.001$	$1108.5\pm 17.9$
	Clinical + Seq	$0.7504\pm 0.004$	$0.1313\pm 0.001$	$1105.0\pm 15.4$

CoxPH can only accept tabular Clinical inputs, while N-MTLR can utilize sequence inputs. Results are reported as AUROC@30±Std.Dev

Discussion

We compare multiple models trained to predict 30-day all-cause hospital readmissions in Alberta, Canada. Using Clinical features exclusively, XGBoost achieves a higher AUROC@30 than the DNN and N-MTLR models (XGBoost $0.7546\pm 0.005$, DNN $0.7147\pm 0.003$, N-MTLR $0.7348\pm 0.004$); given XGBoost’s reputation as a classification powerhouse (especially when using categorical features as input) this result is not surprising. However, with access to medical history sequence information, the DNN (especially when initialized with the MCE dictionary) out-performs XGBoost ($0.8174\pm 0.006$ versus $0.7546\pm 0.005$). The results improve further when we exploit a survival target (training an N-MTLR model with the same feature-processing architecture as our DNN), achieving an AUROC@30 of $0.8467\pm 0.004$. Even without the use of Clinical features, this model achieves an AUROC@30 of $0.8460\pm 0.003$; this suggests that much of the information contained within our Clinical features is already encoded usefully in the Sequence inputs. All numbers already discussed far out-perform the LACE model, with an AUROC@30 of $0.6587\pm 0.003$. Note that we compare our results to LACE as Alberta’s Electronic Health Record software makes use of the LACE index, as per the ConnectCare manual3. The only model that does not out-perform LACE in AUROC@30 is CoxPH ($0.6585\pm 0.002$), which serves as a cautionary tale against choosing a time-to-event model with inadequate representational capacity for the task.

The success of transfer learning demonstrates that initializing machine learning models with prior knowledge often leads to increased performance; this is consistent with our evaluations of the Deep Neural Network and N-MTLR models utilizing the Medical Concept Embedding dictionary. Models that use lower-dimensional inputs also require tuning fewer parameters, which can decrease the risk of overfitting. Low-dimensional embedding dictionaries—like our MCE—are especially helpful when the coding system contains multiple tokens for the same concept; this is common in natural language with synonyms, and medical codes where (for example) there is a nearly analogous ICD-10-CM code (NACRS and DAD) for every ICD-9 code (Claims).

The N-MTLR model learned from sequences of historical medical codes (S-MTLR), is the most promising {model+feature} combination explored in this study. At predicting the presence of a readmission by 30 days, this combination out-performs all other models (including the deep learning approach) trained specifically to predict 30-day readmissions. This is a surprising result, as modeling readmission probability at all future points is a more demanding and general task than focusing on maximizing only the 30-day performance. We propose three potential explanations for this development. First, the survival dataset’s time-to-event target provides granular information about post-discharge progression. Imagine two patients, $P_A$ and $P_B$, where $P_A$ experiences a readmission at 31 days, and $P_B$ is readmitted nine months after discharge. According to a binary model, both are given the label $r=0$, denoting a non-readmission. In contrast, a survival model sees that $P_A$ is riskier than $P_B$ by their respective readmission times. It can therefore associate a higher probability of readmission with patients like $P_A$, who presumably carry fewer markers of relative well-being in their features. Second, specifically modeling probabilities at many future time-points (both before and after 30 days) may cause the model’s 30-day readmission prediction to consistently fall closer to the actual observation. This could happen if certain markers in a patient’s representation provide insight into (for example) 7-day readmissions and 60-day readmissions. Given that a survival curve must monotonically decrease, accurate predictions at other times may positively influence the accuracy of prediction at the timepoint of interest. Third, the combination of accessing a patient’s sequence of past events pairs well with an algorithm that models a sequence of event-free survival probabilities in the future. This could be because certain clusters of events may be highly predictive of readmission prognosis at particular points in the future—if a representation flattens the past’s sequential nature, the model may not be able to take advantage of these relationships. From these sequences, models can even learn information inherent in common tabular clinical features as well, which is made evident by the fact that adding clinical features to S-MTLR does not significantly aid in performance. This favourable combination of sequential event-related inputs and modeling event probabilities at all future time-points could likely improve performance on other tasks as well: both medical (such as modeling cancer progression) and otherwise (e.g., predicting time until an industrial machine fails given past sequences of evaluations and repairs).

There are a number of attributes that make Individualized Survival Distributions favourable for a clinical setting (e.g., implemented within an electronic health record), beyond demonstrating improved performance at the 30-day task. One is that the ISD provides meaningful information about a specific patient. By contrast, risk scores (like LACE) only provide relative information—e.g., that patient $P_A$ is more likely to be readmitted than $P_B$. (Rankings can be “correct” yet useless; if $P_A$’s chance of readmission is 0.003 and $P_B$’s is 0.001—suggesting that neither will be readmitted—a model will still be rewarded if it predicts that $P_A$ will be readmitted, and $P_B$ will not. The same can also apply when both readmission probabilities are near 1.) Second, survival curves provide insight into a far broader number of questions than other classes of models; an ISD can immediately answer “How long until we expect a readmission for patient x?”, “What is the probability patient x will be readmitted in 30 days? What about 9 weeks?”, “Is patient x more likely to be readmitted within 1 year compared to patient y? How risky is x compared to y for a readmission overall?” Finally, the information captured in an ISD is meaningfully visualized (see Fig. 2B). A physician can immediately see that this patient has around a 25% chance of reaching two years with no readmission, and that their median time-to-readmission is around 350 days. The ISD is elegant in its ability to convey comprehensive, self-contained survival information hastily.

However, the concordance statistic, as used in this study, is (roughly) a time-agnostic version of the 30-day AUROC score. The former measures how well the model ranks patients by median survival time, and the latter measures how well the model ranks patients by their 30-day readmission probability. We see that our 30-day AUROC scores are higher than the respective concordance scores for our ISD models—this suggests that the AUROCs for our ISD models drop as the timepoint we are considering increases (i.e., better performance for 30 days than for 300). One explanation is the censoring distribution—readmission events happen close to discharge with higher frequencies, and censorships start accumulating at 365 days (Fig. 6), which may make the task of distinguishing the risk of far-future prognoses more difficult. Therefore, rankings based around the tail end of the curves should be interpreted (as with all model outputs) with appropriate caution.

One strength underlying this study is the data available to our machine learners. Alberta has an integrated population-wide single-payor health system, which enables the collection of comprehensive administrative data with minimal loss to follow-up. However, certain limitations exist. One, previous studies have distinguished between “planned” and “unplanned” readmissions, or even “preventable” and “non-preventable” readmissions. However, no algorithm has been proven to reliably determine if readmissions fall under the “preventable” category, and our data did not contain a variable to distinguish with adequate acuity whether a hospital visit was “unplanned”. This means our models have the potential to capture relationships between certain discharges and planned follow-ups. Two, our data only captures within-hospital deaths; patients who were discharged and then passed away are treated identically to those who were discharged and remained alive without readmission. The former cohort are likely in some dire condition (which is likely captured in their medical history-based features), and the latter are likely associated with features that indicate relative health. The fact these two cohorts differ greatly yet are categorized in the same way could be hurting model performance by obscuring the boundary between health and sickness. Three, the entire study population residing in a particular Canadian province means that generalization of this work to different geographical areas should be performed with caution.

Future directions may involve validating S-MTLR with a publically available dataset (such as MIMIC-IV [55]) to better understand how well this approach works relative to other approaches. Another may be addressing limitations in our dataset by incorporating deaths into the definition of our “adverse outcome of interest” alongside readmissions based on information in a death registry. Exploring the potential of longitudinal event-based features and N-MTLR for other problem settings (medical and otherwise) may also prove worthwhile. We note that deep-learning architecture and hyper-parameter tuning was not performed in this study—therefore, our numbers are a lower bound of what is possible using this approach. For example, an attention layer may improve output quality, as may using a more sophisticated (e.g., transformer-based) method for generating the MCE dictionary, such as MedBERT [56]. An intensive grid-search would help elucidate this gap. In addition, adding to the multimodality of the best-performing model (e.g., by integrating a natural language model into our framework to handle clinical free-text) could potentially further enhance predictive accuracy.

Conclusions

This study seeks to advance the individualized 30-day all-cause hospital readmission prediction task. We tested and validated our proposed methods on retrospective population-wide linked administrative data from Alberta Health Services. Performance increases if our models (1) use an effective sequential patient history representation and (2) model a time-agnostic readmission prognosis using modern survival analysis tools. The S-MTLR system realizes both of these important components. Taking the step to initialize models with embedded medical knowledge further improves accuracy. While effective individually, we demonstrate that these three approaches taken together lead to the largest gains. These findings may be of interest to those who wish to operationalize a readmission prediction system, or those seeking to build (potentially multi-modal) models with both rich inputs and informative outputs.

Authors’ contributions

S.D. and R.G. defined the study’s objectives. S.D. cleaned and processed the data and designed, coded, and executed the experiments. Both authors contributed to the interpretation of results. S.D. wrote the manuscript and created all figures and tables with R.G.’s substantial feedback.

Funding

S.D. was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC). R.G. was supported by NSERC, the Alberta Machine Intelligence Institute (Amii), and the Canadian Institute for Advanced Research (CIFAR).

Data availability

Restrictions apply to the availability of the data used to support the findings of this study, which remain under a data disclosure agreement and thus are not publicly available. To inquire about the treatment and processing of data, contact S.D. at sdavis1@ualberta.ca. To request authorization to obtain data by direct access, contact research.administration@ahs.ca.

Code availability

The repository containing code for model definitions and training scripts is available from https://github.com/sacha-davis/s-mtlr.

Declarations

Ethics approval and consent to participate

This study was approved by the Health Research Ethics Board of the University of Alberta (Study ID Pro00082041); research was conducted in adherence to all relevant guidelines and regulations.

Informed consent requirements were waived due to the data’s de-identified nature by the Health Research Ethics Board (Health) Panel.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.