Discrete Time Neural Network Models to address time-varying predictor importance: An illustration in predicting mortality over different time horizons

Abstract

Clinical predictive models (CPMs) are crucial for forecasting patient outcomes using available electronic health record (EHR) data. Traditional time-to-event (TTE) models, like the Cox proportional hazards model, assume that hazard ratios remain constant over time, which may not hold in many clinical settings. In this study, we introduce a Discrete Time Neural Network (DTNN) to address these limitations by modeling time-varying predictor importance. The DTNN combines the flexibility of classification models with the advantage of handling time-to-event data, providing a single model fit across multiple time horizons. Using data from patients with end-stage kidney disease (ESKD) undergoing hemodialysis, we demonstrate that the DTNN can flexibly adjust for different risk factors across short-term and long-term mortality predictions. The model was evaluated using cumulative-dynamic area under the receiver operating characteristic (CD-AUROC) to compare patients who remained event-free up to a given time t with those who experienced the event before t. Our results show that the DTNN outperforms traditional TTE models and provides robust predictions across varying time intervals, making it an appealing choice for clinical settings where time-varying predictor importance is essential.

I. INTRODUCTION

Clinical Predictive Models (CPMs) are designed to inform clinical practice by making predictions about outcomes of interest using available clinical data. These models are often developed from Electronic Health Record (EHR) data, which provide a rich source of patient information, including demographics, medical history, treatments, and outcomes. The algorithms used in CPMs typically fall into two categories depending on the specific prediction task: classification methods and time-to-event (TTE) methods[1]. While classification methods, which seek to predict the discrete occurrence of an event, are more typically associated with the machine learning (ML) literature, TTE event methods, like the Cox proportional hazards model are often more useful in healthcare settings. In most healthcare predictive settings, one is interested in not just whether an event will occur (i.e., binary classification) but whether an event will occur by a certain time point. After fitting a TTE model one can estimate outcome probabilities for different event horizons of interest. Moreover, when considering the timing of an event, other challenges arise, most notably censoring, which TTE methods are well designed to handle. Not surprisingly a variety of ML-based TTE models have been developed [2], [3], [4], and recent reviews synthesize advances in high-dimensional survival modelling [5].

Perhaps the most common TTE approach is the Cox proportional hazards (Cox-PH) model, which is highly flexible in that the hazard rate (and consequently the event rate) may vary freely over time. This is in contrast to parametric TTE models, such as the accelerated failure time model, which assume the event rate follows a chosen parametric distribution. However, the key assumption of Cox-PH models is the proportional hazards assumption, which states that the effects of different risk factors are constant over time. This means, for example, that the effect of high blood pressure on the hazard ratio is the same if one is considering mortality risk in the next seven days, six months, or five years. As we[6] and others[7], [8] have shown, this is not an accurate assumption. Different risk factors have different effects depending on the time horizon. As we showed in a related study of predicting mortality risk for individuals undergoing hemodialysis for end stage kidney disease (ESKD), predictors that are more variable over time (e.g., blood pressure) are more useful for predicting near-term risk, while more stable predictors (e.g., comorbidities) are useful for predicting long-term risk. In these settings, a sequence of classification models that predict for different time horizons provide better performance compared to TTE models.

Although a series of discrete classification models can provide optimal performance, this approach is impractical when implementing models in clinical practice. As governance processes for CPMs mature, each individual model requires its own evaluation and monitoring plan[9]. As such managing multiple models becomes untenable. The Discrete Time Neural Network (DTNN) provides an appealing alternative by combining the benefits of a TTE (i.e., a single model to predict outcomes over multiple time horizons, natural handling of censoring) with the benefits of a series of classification models (i.e., flexible risk factors over-time). While the DTNN has been studied[10], [11], [12], [13], to our knowledge this aspect has not been explitcly explored. The goal of this paper is to illustrate how the DTNN handles non-proportional risk, highlighting its benefit compared to both traditional TTE and classification methods. We use as our use-case the task of predicting mortality for individuals with ESKD, a patient population with a high mortality rate[14].

II. Materials and Methods

A. Study Data

1). Study Environment and source data:

This study was approved by the Duke University IRB protocol #Pro00108813 with a waiver of informed consent as retrospective research and the study was conducted at the Duke University Health System (DUHS). DUHS includes three hospitals and over 100 outpatient clinics. As the only hospitals in Durham County, DUHS also serves as the primary provider for individuals living in Durham, with over 85% of residents receiving primary care through DUHS[15]. Durham County is a medium sized county in central North Carolina, with an until recently, majority-minority population. DUHS has used an integrated EPIC Electronic Health Records (EHR) system since 2014. Analytic data were extracted from a research datamart that models EHR data to the PCORnet Common Data Model[16]. The research data covered January 1^st 2014 through December 31^st 2022.

2). Analytic Cohort:

We sought to identify individuals with prevalent ESKD. To identify eligible patients, we used encounters with an ICD 9 or 10 code of 585.6 or N18.6 respectively. We required individuals to have either two outpatient encounters or one inpatient encounter with a relevant diagnosis code. Individuals were also included if they had a “end stage kidney disease” in their problem list. Additionally, to ensure that we were capturing a local patient cohort, who were likely to return to DUHS for follow-up services, we required individuals to live in one of the three surrounding counties of Durham, Wake or Orange County North Carolina. Study eligibility began on January 1^st, 2015 and ended December 31, 2021, with 2014 data serving as a burn-in period to adjudicate risk factors and 2022 data as a burn-out period to determine censoring (see below).

3). Outcome Definition and Censoring:

The primary outcome of interest was all cause mortality. DUHS uses external data vendors to ensure capture of mortality for patients who die outside of a hospital encounter. These data are integrated into the research data warehouse. We also applied a series of censoring rules to truncate follow-up time. Individuals were censored at the earliest date if they received a kidney transplant, identified via a CPT code “50360” or “50365”, or moved out of the geographic region (no longer live in Durham, Wake or Orange county). If an individual had an encounter during the burn-out period (calendar-year 2022) and was not previously censored, we used December 31, 2021 as their censoring date. Otherwise, we used their last encounter date as their censoring date. For our analyses we considered outcomes over five different time horizons: 30, 90, 180, 270 and 360 days.

4). Predictor Variables:

We abstracted a broad range of predictor variables from the data warehouse. This included socio-demographic information (e.g., age, sex, neighborhood information), service utilization, comorbidity (based the Liu comorbidity index for ESKD patients[17]), and laboratory test results on common laboratory tests. Details are provided in supplemental table 1. The most recent laboratory test was used, as well as a time indicator from when the test was drawn. We applied mean imputation if a laboratory test was never taken. For records where a laboratory test had never been ordered, we also assigned the accompanying ‘days-since-test’ feature the maximum look-back window (2,921 days), so that the model could explicitly recognize the long-missing status and exploit the informative absence of the measurement. Service utilization metrics were generated based on the previous 30 days. Comorbidities were recorded based on whether there was an encounter for that concern that month as well as whether there was ever a history of that diagnosis. In total there were 72 predictor variables.

5). Data Organization:

In order to facilitate making updated predictions, with most recent clinical data, we organized data into patient-month format. For each relevant patient month from January 1^st 2015 through December 1^st 2021, we generated a row of data for each eligible patient. We included the most recently updated clinical variables. We also generated the outcome as time to either mortality (the event of interest) or censoring.

B. Analytic Models

To assess our ability to generate time-to-event predictions over different time horizons, we considered three neural network-based modelling approaches. The first was a classification model based on a multi-layer perceptron (MLP). For this model, patient-months that had an event within the time-horizon of interest were labeled as 1 and patients months with censoring or events past the time horizon of interest were labeled as a 0. If a patient-month was censored within the time horizon of interest, they were excluded, as status unknown. A separate model with time-specific outcome labels was built for each time horizon allowing for model parameters to vary based on the time horizon. The primary advantage of an MLP is that it optimizes its fit for each time horizon while the drawback is that it requires fitting (and maintaining) multiple models.

The second was the Cox based time-to-event neural network DeepSurv [18]. For this model approach, a single model was fit and evaluated across each of the time horizons of interest. Predictions for each time horizon were made using the predicted log-hazard ratios, which were applied consistently across all time points to estimate the likelihood of the event occurring at different time intervals. Compared to the MLP, the primary advantage is that one model is fit, while the drawback is that it doesn’t allow for different parameters across different time horizons.

The final model was a DTNN model[19], [20], [21] model. The DTNN combines the advantages of an MLP with those of a single time-to-event model. Like DeepSurv, the DTNN uses one model fit over all time horizons. However, within the model, it allows for different parameters across different time horizons. This is achieved by binning the data into discrete time intervals and fitting a classification model for each bin. The architecture includes an input layer, hidden layers, and multiple output layers corresponding to each time bin.

Additionally, the model includes an extra output layer for the scenario where the event does not occur within the observed time horizon. The predicted probability of an event occurring in time bin $j$ for patient $i$ is:

$${\widehat{p}}_{ij}=\frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(z_{ij}\right)}{\sum _{k=1}^{K+1}\mathrm{e}\mathrm{x}\mathrm{p}\left(z_{ik}\right)}$$ (1)

where $z_{ij}$ is the output of the network for time bin $j$, and $K+1$ is the total number of time bins, including an additional bin for “no event within the time horizon.”

The proportion of bin $j$ completed by time $t_i$ for patient $i$ is calculated as:

$${\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{p}}_i(j)=\mathrm{m}\mathrm{i}\mathrm{n}\left(1,\mathrm{m}\mathrm{a}\mathrm{x}\left(0,\frac{t_i-{\text{bin\_start}}_j}{{\text{bin\_length}}_j}\right)\right)$$ (2)

The DTNN uses a negative log-likelihood loss function to capture the model’s performance. The event likelihood for uncensored patients is:

$$L_{\text{event},i}=\sum _{j=1}^K{d}_{ij}\cdot p_{ij}+\epsilon$$ (3)

where $d_{ij}$ is a binary indicator that is 1 if the event occurs in bin $j,\epsilon$ is a small tolerance added for numerical stability.

For censored individuals (or individuals where the event does not occur within the time horizon), the non-event likelihood is:

$$L_{\text{nonevent},i}=1-\sum _{j=1}^K{\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{p}}_i(j)\cdot p_{ij}+\epsilon$$ (4)

The final negative log-likelihood (NLL) is:

$$L(\theta )=-\sum _{i=1}^N\left[s_i\mathrm{l}\mathrm{o}\mathrm{g}\left(\sum _{j=1}^K{d}_{ij}\cdot {\hat{p}}_{ij}+\epsilon \right)+\left(1-s_j\right)\mathrm{l}\mathrm{o}\mathrm{g}\left(1-\sum _{j=1}^K\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{p}(j)\cdot {\hat{p}}_{ij}+\epsilon \right)\right]$$ (5)

where $s_i$ is the event indicator for individual $i$.

This loss function penalizes both incorrect event predictions and incorrect non-event predictions, while also accounting for the “no event within time horizon” case through the additional bin.

C. Analyses

After generating the analytic dataset, we describe the cohort based on patient demographics and comorbidities at the time of initial study eligibility. We fit a cumulative incidence plot from time from first eligibility to observe censoring mechanisms.

We divided the data into training and test sets, splitting the data at the individual level, i.e., all patients months for an individual were either in the training or test sample. We fit each of the three models on the training data. We used 5-fold cross-validation to optimize hyper-parameters (e.g., learning rate) for each modelling approach separately. We evaluated each model fit, across the different time horizons (30, 60, 90, 180, 270 and 360 days) on the test data. To account for the time-to-event nature of the data, we calculated a cumulative-dynamic area under the receiver characteristic (AUROC)[22]. Specifically, the cumulative-dynamic AUROC compares individuals who are known to be event-free through time t with those who have experienced the event before time t. This approach evaluates the model’s ability to distinguish between patients who remain event-free up to a given time and those who have already experienced the event, making it well-suited for time-to-event analysis. We used bootstrapping to calculate 95% confidence intervals for the AUROC.

In order to generate deeper insights into how the important predictor variables change based on the time horizon of interest, we used the MLP model fits to calculate the mean absolute SHAP [23] values for each input variables. We plotted the SHAP valuables, organized based on how variables they were across each of the time horizons. We also calculated the correlation of SHAP values across model fits.

All analyses were conducted in Python V 3.11.5 using (pycox, scikit-learn, scikit-survival, seaborn, shap, torch, torchtuples etc.). The DeepSurv model was available from https://github.com/havakv/pycox. The DTNN model was based on our own written code and available on our Github repository (https://github.com/engelhard-lab/collaborative-ml-notebooks/blob/main/notebooks/discrete_time_neural_survival_model.ipynb).

III. Results

A. Cohort Description

We identified 2,902 eligible patients, who had a total of 156,028 encounters and contributed a total of 85,405 person months (see Figure 1 for consort diagram). Most eligible patients were excluded based on not living locally, which is expected since DUHS serves as a larger regional referral center for kidney transplant and other advanced kidney care. Table I provides descriptive statistics for the cohort. The cohort was predominately non-Hispanic Black (66%), male (56%), and had low neighborhood socio-economic status (median rating of 3). Figure 2 presents the cumulative incidence plot, with 30.67% of the cohort experiencing the mortality event of interest. On a person month level, 0.49%, 2.06%, 4.44%, 6.75% and 8.88% had a mortality event within 30, 90, 180, 270, and 360 days respectively.

Fig. 1.

Data Extraction and Filtering Process for ESKD Patient Cohort Selection

TABLE I.

Cohort Description

Characteristic	Count (Proportion) /Median (IQR)
Age (years)	59.61 (59.09, 60.13)
Race/Ethnicity
Non-Hispanic Black	1906 (65.7%)
Non-Hispanic White	728 (25.1%)
Hispanic	156 (5.4%)
Sex
Male	1635 (56.3%)
Female	1267 (43.7%)
Area Deprivation Index*
1	442 (15.2%)
2	600 (20.7%)
3	413 (14.2%)
4	347 (12.0%)
5	323 (11.1%)
6	151 (5.2%)
7	156 (5.4%)
8	146 (5.0%)
9	142 (4.9%)
10	96 (3.3%)
Missing	86 (3.0%)
Service Utilization
Encounters	3 (1, 5)
Comorbidities
Atherosclerotic Heart Disease	798 (27.5%)
Heart Failure	885 (30.5%)
Cerebrovascular Accident	369 (12.7%)
Peripheral Vascular Disease	750 (25.8%)
Other Cardiac Disease	1177 (40.6%)
Chronic Obstructive Pulmonary Disease	436 (15.0%)
Gastrointestinal	281 (9.7%)
Liver Disease	229 (7.9%)
Dysrhythmia	892 (30.7%)
Cancer	348 (12.0%)
Diabetes	1560 (53.8%)

^*Lower ADI indicates lower (more deprived) neighborhood

Fig. 2.

Cumulative Incidence by Censor Reason

B. Predictive Modelling Results

Table II provides the model performance in an independent test set for the three modelling approaches. Models performed better for nearer versus longer time horizons. Across most time horizons, the classification based MLP had the best performance, though this was not statistically better than the DTNN. The DTNN consistently performed statistically better than the survival-based model across all evaluated time horizons.

TABLE II.

Model performance based on AUROC

Time Horizon (days)	MLP	DeepSurv	DTNN
30 (422 events)	0.877 (0.832, 0.917)	0.790 (0.727, 0.844)	0.841 (0.783, 0.891)
90 (1757 events)	0.803 (0.776, 0.831)	0.735 (0.703, 0.768)	0.790 (0.763, 0.818)
180 (3788 events)	0.745 (0.724, 0.766)	0.696 (0.669, 0.721)	0.751 (0.730, 0.774)
270 (5765 events)	0.723 (0.706, 0.742)	0.684 (0.662, 0.703)	0.731 (0.713, 0.749)
360 (7587 events)	0.691 (0.682, 0.717)	0.683 (0.666, 0.699)	0.716 (0.699, 0.732)

C. Variable Importance

Table III shows the top 4 predictors (based on SHAP value) by time horizon. Figure 3 shows the SHAP values for each variable over different time horizons, based on the distinct MLP classification models. The variable ordering (y-axis) is organized from least to greatest variance in the SHAP values. Several key features emerge. Besides albumin level, the most important predictors are primarily comorbidities. Moreover, the top predictors are consistent across time horizons. However, as Figure 3 shows, the actual importances are quite variable. In general, the more important a predictor is on average across all time horizons, the more variable its SHAP value is across all time horizons. Also noteworthy, in general (again with the exception of Albumin), the timing of when a laboratory test was performed played a more significant role in predicting mortality than the actual test results. Specifically, more recent tests were associated with higher short-term mortality risk, while older tests had less predictive power for short-term outcomes, highlighting the temporal sensitivity of lab data in mortality prediction.

TABLE III.

Top 4 predictors by time horizon

	30 Days	90 Days	180 Days	270 Days	360 Days
1	Dysrhythmia (ever/never)	Albumin (value)	Albumin (value)	Albumin (value)	Albumin (value)
2	ASHD (ever/never)	ASHD (ever/never)	ASHD (ever/never)	Dysrhythmia (ever/never)	Dysrhythmia (ever/never)
3	Diabetes (ever/never)	Heart Failure (ever/never)	Dysrhythmia (ever/never)	Cardiac (ever/never)	Heart Failure (ever/never)
4	COPD (ever/never)	Dysrhythmia (ever/never)	COPD (ever/never)	ASHD (ever/never)	Cardiac (ever/never)

Fig. 3.

Mean Absolute SHAP Values

Finally, to understand how related important predictor variables are across different time horizons, Figure 4 shows the correlation of the SHAP for the MLP models. There is relatively high correlation between adjacent time horizons (mean correlation of 0.73). However, the correlation decreases farther apart the outcome horizons are.

Fig. 4.

Correlation Matrix of SHAP Values Across Time Horizons

IV. Discussion

In this study we compared three approaches for predicting a time-based outcome: a classification (MLP), time-to-event (DeepSurv) and discrete time approach (DTNN). As expected, the classification based approach had the nominally best performance, with the TTE the worst. Importantly, the less widely used DTNN model had nearly as good performance as the classification model. As such, this works highlights the utility of such model approaches and suggests that they should be more widely used.

When one is interested in assessing risk at different time points, TTE models are appealing because they allow the developer to fit a single model. This allows one to use one model to assess the probability of the outcome, in our case mortality, at any time point of interest. The primary assumption of such an approach is that the set of effects are common across all time horizons. In Cox based models, this is referred to as the proportional hazards assumption. However, even for non-proportional hazards based models, such as accelerated failure time models, there is still an assumption that a common effect can be defined. In an AFT model, instead of assuming proportional hazards like the Cox model, the event time is modeled as being stretched or shrunk by a single factor. This factor either accelerates or decelerates the time to the event, depending on the covariates. Just like the Cox model predicts a single hazard ratio, the AFT model predicts a single value that affects the event time, but in this case, it adjusts the overall timing of the event by scaling the survival curve rather than shifting the hazard function. As shown in previous work[6], and confirmed in the present analysis, this assumption often does not hold as in our use case of modelling mortality. Depending on the time horizon of interest, different clinical factors have different degrees of importance. Ultimately, a single TTE model averages performance across all time horizons and is therefore not specifically optimized for any one of them, which can lead to inferior discrimination compared with horizon-specific approaches.

Fitting a series of classification models resolves this problem, and it is not surprising that the MLP had the nominally best performance. It reasonable to consider the MLP performance as an approximate upper-bound on model performance. However, the downside of such an approach, is that in practice, one would need to maintain multiple models. As concerns for model implementation and governance take on greater prominence [9], questions of parsimony and ease of use come to the forefront. A secondary concern, is that arguably, a series of classification models is not being most as efficient as possible with the available data. If there are shared risk factors across different time-horizons, a discrete set of models will not be able to learn from that.

It is for this reason we explored the DTNN. In our assessment it combines the best of both worlds: a single model fit that has parameters to vary based on the time horizons of interest. By predicting within risk windows, the model can easily generate a probability for risk over different time horizons. While we are not the first to use DTNN, they are also not widely used. A model like DTNN appears to have been first introduced by J. J. Hopfield[24]. Since then, they have been used to model discrete-time survival predictions in clinical settings, particularly for ICU patients [11], heterogeneous clinical data, and right-censored data[12]. While others have pointed out properties of the DTNN, such as handling time-varying hazards and incorporating complex, non-linear relationships between predictors[13], we believe we are the first to highlight and empirically validate the utility of the DTNN to properly model predictors that are non-proportional. This means that the model can perform well over both nearer and longer-term time horizons.

Beyond illustrating the utility of the DTNN, our work also presents additional insights into predicting mortality among individuals with ESKD. Given the high mortality rate among individuals with ESKD, it is not surprising that a number of mortality based models have been developed[25], [26]. These models have typically utilized data from in-center hemodialysis[27] EHRs or administrative claims[28]. Our study illustrates that a reasonably strong model for near and mid-term mortality can be generated leveraging routinely collected from health system data. It is noteworthy, that the predictive performance using just health system data is similar to that achievable with models relying on more specialized data from dialysis clinic[27]. This is a valuable clinical insight, as researchers consider developing and implementing clinical decision support tools. While patients undergoing hemodialysis receive regular care from an outpatient center (typically three times a week), they also have regular encounters with their health system via hospital and specialist (e.g., cardiologist) visits. Our results show that the information generated via encounters, comorbidities and laboratory measures is useful in predicting who is likely to die. The strong results were likely buoyed by the stringent inclusion criterion that required individuals under study to be a local patient receiving regular care at our health system.

Additionally, our work provides further insights into clinical risk factors for patients with ESKD. The top predictors were consistently the presence of comorbidities. While this was not surprising, it is noteworthy that in contrast to hospital EHR systems, information on comorbidities are typically not well captured by outpatient hemodialysis EHR systems. However, such factors are not modifiable, limiting the opportunity for intervention. Instead such a tool may indicate clinical risk and be used to promote shared decision making between patients and providers[29]. Moreover, while there was consistency in the top predictors across time horizons, the actual strength of importance of the predictors (as measured by SHAP values) was fairly variable. This confirms our hypothesis regarding non-proportional risks and helps explain why the time-to-event model has the worst performance. Finally, it was interesting to note that for most laboratory variables, aside from albumin, the time since taking the laboratory test was more informative than the test value itself. This speaks to the well documented informative presence of EHR data elements, particularly laboratory data used for diagnosis and treatment[30].

While this study highlights the utility of the DTNN, there are some limitations. First, this is an analysis in a single data set. While this was motivated from a real-world modelling task, it is worth exploring this in other settings where non-proportional hazards may be a concern. Similarly, the clinical insights, as well as the learned model, are relevant to our single center and are worth further exploring for generalizability. Second, it is worth noting that the DTNN requires more tuning than a typical MLP, leading to more computation. However, for our moderately sized dataset, we were able to fit the model on a CPU. Third, our reliance on simple mean imputation for never-measured laboratories does not fully capture the informative nature of missingness in EHR data; future work should explore approaches that model informative missing data directly[31], [32]. Finally, while this work shows that mortality for individuals with ESKD can be predicted from health system data, it is important to note that even more valuable information exists within the EHR from dialysis clinics[27]. Future work should consider how to combine data across these different care environments.

V. Conclusion

To summarize, our study highlights the effectiveness and potential of DTNNs in predicting mortality for individuals with ESKD. The DTNN combines the advantages of TTE and classification models. Moreover, it consistently outperformed DeepSurv while performing nearly as well as an MLP. This demonstrates the robustness and flexibility of DTNNs in managing complex, time-dependent interactions and heterogeneous data, making them valuable tools in clinical settings where early and precise risk assessments are critical for improving patient outcomes. Future work should continue to use and explore the DTNN into traditional time-to-event modelling settings.