Time-to-Event Pretraining for 3D Medical Imaging

Abstract

With the rise of medical foundation models and the growing availability of imaging data, scalable pretraining techniques offer a promising way to identify imaging biomarkers predictive of future disease risk. While current self-supervised methods for 3D medical imaging models capture local structural features like organ morphology, they fail to link pixel biomarkers with long-term health outcomes due to a missing context problem. Current approaches lack the temporal context necessary to identify biomarkers correlated with disease progression, as they rely on supervision derived only from images and concurrent text descriptions. To address this, we introduce time-to-event pretraining, a pretraining framework for 3D medical imaging models that leverages large-scale temporal supervision from paired, longitudinal electronic health records (EHRs). Using a dataset of 18,945 CT scans (4.2 million 2D images) and time-to-event distributions across thousands of EHR-derived tasks, our method improves outcome prediction, achieving an average AUROC increase of 23.7% and a 29.4% gain in Harrell’s C-index across 8 benchmark tasks. Importantly, these gains are achieved without sacrificing diagnostic classification performance. This study lays the foundation for integrating longitudinal EHR and 3D imaging data to advance clinical risk prediction.

1. Introduction

Foundation models for medical imaging have the potential to transform healthcare by assisting doctors in complex clinical decision making (Saab et al., 2024; Sox et al., 2024) and identifying novel pixel biomarkers predictive of future disease risk (Pai et al., 2024; Sriram et al., 2021). Such models rely on self-supervised learning (SSL) for obtaining supervision at the scale required to train on growing collections of 3D data (e.g., CT scans, MRIs). SSL captures local structural features by leveraging pretraining signal directly from images or cross-modal pairs (e.g., images and their text descriptions) (Zhang et al., 2022). While excelling at segmentation tasks and diagnostic classification of pathologies (Pinto-Coelho, 2023), SSL fails to learn prognostic biomarkers because the current pretraining regimens suffer from a missing context problem (see Figure 1). This issue arises when supervision sources are restricted to narrow time windows around the image, thus excluding long-term temporal patterns that are correlated with disease progression, which limits a model’s ability to identify prognostic biomarkers.

Figure 1:

The missing context problem in medical imaging. Existing supervision sources (red boxes) are localized to the image itself (i.e., pixel features and descriptions of those features via text) or immediate clinical context via diagnosis codes. Doing so misses future information on disease progression (black boxes), which reduces the ability to learn correlations necessary for identifying prognostic pixel biomarkers. Time-to-event pretraining provides a principled framework for incorporating the vast amount of temporal supervision available in EHR data to estimate future risk in the presence of right censorship as well as leverage a large, diverse number of clinical tasks, beyond just diagnoses, for pre-training.

These long-term temporal patterns, which we refer to as longitudinal context, are readily available in a patient’s electronic health record (EHR) which is routinely used by clinicians to guide the interpretation of images and inform treatment planning (Leslie et al., 2000; Holste et al., 2024). Longitudinal EHRs contain temporal information about the progression of disease, as well as years of patients’ health outcomes. However, the full breadth of these outcome data, in terms of temporal structure and task diversity, is rarely used as a source of supervision when training image foundation models. Current approaches for training 3D imaging models typically restrict labels to diagnosis codes sourced from the same or nearby temporal context as the image and its textual description (Blankemeier et al., 2024). Because EHR data is readily available, it offers an untapped resource for large-scale pretraining of medical image models in a manner that uses long-term temporal context. More generally, image foundation models must reflect the settings in which they will be used, which is to assist in prognosis in the clinic (Negro-Calduch et al., 2021; Yala & Hughes, 2023).

However, performing effective risk estimation using imaging models involves navigating several challenges. Developing such models requires capturing correlations between pixels and outcomes spanning years, which is difficult with current SSL methods. Direct approaches to identifying pixellevel biomarkers, such as sequential image capturing (Lu et al., 2021; Bera et al., 2022), are difficult to scale for collecting large, high-quality datasets. Moreover, conducting risk estimation requires addressing right censoring, where the outcome of interest remains unobserved by the study’s end. Naively excluding censored patients introduces bias and reduces available training data.

In medicine, time-to-event (TTE) modeling (also known as survival modeling) is commonly used to estimate future risk of an outcome at a specific time point conditioned on feature representations. Although TTE models offer many theoretical advantages, including the ability to estimate instantaneous risk at any given time point (Collett, 2023) and naturally handling right censoring (Kleinbaum & Klein, 1996), their use in image pretraining remains underexplored. Prior deep learning studies exploring TTE modeling in medical imaging have been restricted to small-scale, single-task applications, typically using 2D, end-to-end models (Zhu et al., 2016; Shu et al., 2021; Lu et al., 2021). 3D medical imaging data remains an underutilized resource, offering a wealth of biomarkers that could enhance clinical decision tools, particularly for the opportunistic detection of underdiagnosed conditions (Aali et al., 2024). However, large-scale TTE pretraining for 3D imaging has not yet been investigated, likely because multimodal medical datasets linking 3D images with longitudinal EHR data have only recently become available (Huang et al., 2024).

In this work, we propose time-to-event pretraining for medical imaging models as a way to address the missing context problem. Our central claim is that temporal supervision, defined by TTE distributions sourced from longitudinal EHRs, provides a readily available, scalable source of contextual information for pretraining that better captures prognostic pixel biomarkers. Moreover, by naturally handling right-censorship, TTE-based methods improve data efficiency and mitigate censorship bias. Our contributions are as follows:

• We present the first large-scale evaluation of time-to-event pretraining for 3D medical imaging encoders. We use a public dataset of 18,945 chest CT scans (equivalent to 4.2 million 2D images) linked to longitudinal EHR data containing 225M clinical events with a median follow-up time of 5 years.

• Our approach converts longitudinal EHR data into a source of time-to-event supervision, thus predicting not only if a clinical event will occur but also when. This richer pretraining signal goes beyond diagnostic classification explored in prior work and enables generating many pretraining tasks (8,192 in this work) that capture the temporal event structure available in longitudinal EHR data. This choice also increase per-image label density by an average of 3 times over prior approaches.

• Our approach substantially improves performance in predicting future medical outcomes, achieving on average a 23.7% increase in AUROC and a 29.4% improvement in Harrell’s C-index over baseline models for 8 benchmark tasks without negatively impacting diagnostic classification performance in 8 external tasks. Our approach also improves model calibration, measured by the Integrated Brier Score, by an average of 54%.

All our experiments are conducted using public medical datasets to ensure full reproducibility. We also make all of our experiment code and pretrained model checkpoints available for download ^{1 2}, to contribute to the community for continued pretraining of imaging foundation models with prognosis as added benefit.

2. Related work

Time-to-Event Modeling with Medical Images

Time-to-event (TTE) modeling, also known as survival analysis, predicts the distribution of time until a specific event occurs, such as death. TTE models primarily include accelerated failure time models, which assume various probability distributions, e.g., exponential (Saikia & Barman, 2017), Weibull (Breheny, 2015), and Cox proportional hazard (Cox-PH) models (Cox, 1972) which is a semi-parametric approach with a constant hazard ratio assumption. Non-parametric models like random survival forests (Ishwaran et al., 2008) capture non-linear interactions. In the deep learning era, methods such as DeepSurv (Katzman et al., 2018) and MOTOR (Steinberg et al., 2024) provide higher level feature learning, making TTE modeling easier to extend to complex inputs such as medical images. Prior TTE methods for imaging assume 2D and 2.5D model architectures and have focused on end-to-end training for small-scale, single-task models. DeepConvSurv (Zhu et al., 2016) was the first work to replace the log-partial hazard (the exponential component of a Cox model) with a CNN, enabling survival prediction directly from 2D images. Similarly, Shu et al. (2021) and Lu et al. (2021) modeled the log-partial hazard with a CNN-RNN to encode sequential images.

Pretraining for 3D Medical Image Models

Although early work in medical imaging relied on supervised pretraining using general-domain datasets such as ImageNet (Xie & Richmond, 2018; Ke et al., 2021), self-supervised learning (SSL) is now the predominate approach to scaling pretraining in medical imaging models. Popular approaches include reconstruction or de-noising objectives via masked autoencoders (MAE) (He et al., 2022), contrastive losses defined over paired samples (Chen et al., 2020; Sowrirajan et al., 2021) or leveraging multimodal pairs, such as medical images and their aligned text descriptions (Radford et al., 2021; Zhang et al., 2022) or unpaired medical images and text (Wang et al., 2022). SSL methods face added challenges when extended to 3D imaging, where instances (e.g., CT scans) contain over 100 times more pixel data than 2D counterparts like radiographs. Tang et al. (2022) combined unimodal contrastive learning, masked volume in-painting, and image rotation prediction to learn 3D structural information. Chen et al. (2023) explored 3D masked image modeling, demonstrating faster convergence compared to simple contrastive methods (SimCLR and MAE). Valanarasu et al. (2023) used a reconstruction loss to restore 3D CT volumes from tokens corrupted by noise, downsampling, and local masking. Finally, Blankemeier et al. (2024) introduced Merlin, a dual-objective framework that leverages EHR data by first using contrastive learning to align radiology reports with CT volumes, followed by training on disease phenotype classification using labels derived from diagnostic codes recorded during contemporaneous hospital visits.

Current SSL approaches excel at capturing structural features in medical images, such as organ morphology for image segmentation. However, these learned features are static and largely fail to identify dynamic patterns and biomarkers that are predictive of future health risks. Our contributions help bridge this gap in several key ways by leveraging future medical events to guide pretraining, resulting in a more powerful image encoder for outcome prediction tasks. First, we employ a TTE objective to capture future patient health dynamics while accounting for right censoring, leveraging longitudinal EHR data to greatly expand the scale, diversity, and temporal scope of pretraining supervision. Second, we evaluate pretraining using native 3D imaging architectures which are underexplored in prior TTE work. Finally, we evaluate the impact of TTE modeling in imaging at a larger scale than prior work, utilizing 18,945 CT scans—equivalent to 4.2 million 2D images—and 8,192 unique TTE tasks derived from longitudinal EHRs.

3. Preliminaries

Time-to-Event Modeling.

The objective of time-to-event modeling is to estimate the distribution of times $T$ until an event of interest occurs. Observable data is denoted as $𝒪=\left.\left\{\left({\tilde{T}}_i,{\text{Δ}}_i,X_i^T\right):\right.i=1,\dots ,n\right\}$, where $X_i$ are the features of observation $i$ and ${\text{Δ}}_i=\mathbb{I}\left(T_i\le C_i\right)$ is an event indicator function whose value is 1 when the actual event time is observed. One complexity is that medical data is often right-censored, where the survival time $T$ is not observed due to a loss of followup. With right-censoring, we do not observe $T$ and instead observe time $\tilde{T}=\min (T,C)$, where the $C$ is the censoring time. When ${\text{Δ}}_i=0$, we do not know the true survival time, but we know that it is greater than ${\tilde{T}}_i$. The mixture of known survival times and censored survival times necessitates methods that can estimate $T$ in an unbiased manner despite the censorship.

Different time-to-event models use different definitions to model the instantaneous hazard $\lambda (t)$, where $t$ represents the continuous time at which the hazard rate is evaluated. We use a piecewise exponential function (Kitchin et al., 1983) that splits the timeline into $P$ distinct intervals, or pieces, each with a constant hazard rate $\lambda (t)$. Each piece $p(t)$ has ${\lambda }_{ip(t)}=C_{ip}$, where $C$ depends on the patient’s CT image $i$ and the specific interval $p$.

Piecewise Exponential Neural Network.

To make use of deep learning models, we use a neural network to define $C_{ip}$. Using a piecewise exponential neural network (PEANN) (Fornili et al., 2014) greatly simplifies large-scale pretraining compared to Cox PH-based methods, which require creating batches where samples are paired with at least one uncensored patient (Harrell Jr et al., 1996). We first use a neural network to derive a CT image representation $M_{ip}$ for image i’s information at time piece $p$. We then apply a linear layer followed by an exponential function to define hazard ${\lambda }_{ip}=C_{ip}=\exp \left(A*M_{ip}+b\right)$ where $A$ and $b$ are learned parameters. With the hazard calculated, we can then input it into the survival function:

$$S_i(t)=\prod _{p=1}^P\exp \left[-{\lambda }_{ip}\left(\min \left(t,E_p\right)-S_p\right)\mathbb{I}\left(t\ge S_p\right)\right]$$ (1)

where each time piece $p$ has a starting point $S_p$ and end point $E_p$ and $\mathbb{I}(\cdot )$ is an indicator function. The standard survival likelihood loss function for image $i$ is then:

$${𝓛}_i={\left[S_i(t)\right]}^{1-{\Delta }_i}{\left[f_i(t)\right]}^{{\Delta }_i}$$ (2)

where ${\Delta }_i$ is event indicator whose value is 1 when the actual future event is observed for image $i$ and $f_i$ is the probability density function $f_i=-\frac{\partial S_i}{\partial t}$.

A full derivation of the loss function, including calculating the derivative and plugging in the survival function, can be found in Appendix M.

4. Time-to-Event Pretraining

In this work, we are interested in training a 3D image encoder to learn representations optimized for estimating the distribution of event times for future clinical outcomes. Using years of follow-up EHR data after image capture, we generate time-to-event pretraining tasks that provide large-scale training signal for estimating this distribution. We outline our approach for time-to-event pretraining in Figure 2. Given the computational costs of pretraining 3D image architectures from scratch, we evaluate the benefits of TTE supervision via continued pretraining of an existing neural network $f_{\theta }$, where $\theta$ represents the parameters of the pretrained backbone.

Figure 2:

Overview of the proposed time-to-event pretraining pipeline. Patients’ longitudinal EHR timelines are transformed into large-scale, time-to-event (TTE) pretraining tasks. These tasks, which reflect informative temporal patterns for medical outcome prediction, are then used for continued pretraining (full fine-tuning) of a 3D vision encoder. The resulting encoder is then frozen and adapted to downstream tasks via different task heads for classification or TTE estimation.

Creating TTE Pretraining Tasks

EHR data captures a vast amount of structured information on patient demographics, diagnoses, procedures, medications, medical devices, social determinants of health, and other aspects of medical care. These data are encoded as timestamped, standardized identifiers called medical codes that map to ontologies (e.g., ICD10, RxNorm, CPT). These ontologies collectively represent a knowledge graph in the form of a directed acyclic graph (DAG). By treating each code as a separate task and organizing EHR data chronologically, we generate extensive training signals to model longitudinal health trajectories. This approach enables tasks such as predicting when a patient might develop lung cancer or be prescribed warfarin (a blood thinner used to prevent and treat blood clots), represented as time-to-event distributions conditioned on image features. However, naively treating each medical code as a task leads to millions of candidate pretraining tasks (4.3 million in our dataset). Many of these tasks are low-frequency or otherwise redundant given an ontology structure. To select pretraining tasks, we follow Steinberg et al. (2024)’s conditional Shannon entropy selection measure, which treats ontology-aware task selection as a vertex cover problem (West et al., 2001). We select a set of medical codes that maximizes conditional entropy given a task budget, ontology DAG, and frequency distribution of observed medical codes.

We then define our TTE pretraining procedure by predicting the time until the first occurrence (if there are multiple recurring ones) of a medical code as defined above, as shown in Appendix B. The survival function in each time piece is modeled from the time piece’s starting time, where the first time piece’s starting time is the index time (here the time of the CT scan exam). When there is no event in a time piece the loss will be 0. Each TTE task (n=8,192) is modeled independently (Appendix O). We also apply censorship at patient death, which is the only competing risk. Given TTE labels for input image $X_i$, we use the loss described in Eq. 2 for full fine-tuning of $f_{\theta }$.

Task Adaptation.

Once TTE pretraining is complete, we use the frozen encoder $f_{\theta }$ to generate feature embeddings for each input image $X_i$, such that $Z_i=f_{\theta }\left(X_i\right)$. These embeddings are then passed to a task-specific head $h_{\widehat{\theta }}$, which can either be a classification head or a time-to-event head, depending on the task. We found that a CoxPH task head (DeepSurv), which directly optimizes for Harrell’s C-index, performed better in practice than fine-tuning a PEANN task head, thus we used CoxPH for all TTE task adaption. The classification head outputs prediction probabilities $p_i\left(\widehat{y}\mid Z_i\right)$ for discriminative tasks, while the survival head produces a time-dependent hazard score $H_i\left(t\mid Z_i\right)$ for time-to-event (TTE) tasks. The model’s outputs are evaluated using task-appropriate metrics, e.g., AUROC for classification or Harrell’s C-index and time-dependent C-statistics for TTE tasks.

5. Experiments

Hypotheses.

Our experiments measure the impact of TTE pretraining on an image encoder’s ability to generate representations useful for medical prognosis. We explore the following hypotheses:

1. TTE supervision improves data efficiency by utilizing temporal information from future EHRs and censored patients, increasing available task labels per-pretraining instance.

2. Existing supervised and SSL pretraining methods struggle to learn strong pixel biomarkers for disease risk due to missing temporal connections with pathologies that will be detected in the future. TTE supervision, in contrast, provides a simple approach to leveraging complex temporal information found in longitudinal EHRs for purposes of learning prognostic pixel biomarkers.

3. TTE supervision, when conducted as post hoc, continued pretraining, does not negatively impact performance on standard categorical classification tasks as used for diagnostic image labeling.

Setup.

All image and EHR preprocessing details are outlined in Appendix A. For our PEANN, we use 8 piecewise time bins and 8,192 pretraining tasks (see Appendices O,P) per the best-performing hyperparameters from Steinberg et al. (2024). Each time bin is created by uniformly dividing the range between the cohort’s earliest and latest timestamps. We utilized two compute nodes from an on-premise cluster, each with 24 Intel Xeon 2.7GHz CPU cores, 200 GB RAM and 4 Nvidia A100 GPUs or 4 H100 GPUs.

Datasets & Evaluation Tasks.

We use two 3D medical imaging datasets (see Table 1). INSPECT (Huang et al., 2024) is a multimodal dataset of paired CT scans and radiology notes where each patient is linked to their longitudinal EHR ^{3 4}. This provides an average of 5 years follow-up data post-CT scan and, in aggregate, contains 225 million medical events. RSPECT (Colak et al., 2021) is an image-only dataset of CT scans annotated by radiologists for imaging biomarkers related to pulmonary embolism and cardiac function. We evaluate 3 task categories in this work:

• Prognostic TTE: Estimate the distribution of event times for a specific outcome (e.g, mortality). We predict the first occurrence of an event, as the first occurrence (or only occurrence in cases such as mortality) is of higher clinical utility than subsequent events.

• Prognostic Classification: Binarized classification formulation of TTE tasks using bucketed time bins. We use 1, 6, and 12 month bins. Unlike in TTE, censored patients are excluded in this category.

• Diagnostic Classification: Standard classification using diagnostic image label categories.

Table 1:

Dataset summary statistics.

	INSPECT	RSPECT
# Patients	19,402	7,279
# Train	18,945	5,823
# Valid	1,089	364
# Test	3,214	1,092
Imaging	✔	✔
EHR	✔	✘
TTE Tasks	✔	✘
Diag. Tasks	✔	✔
Scan Type	Chest CT	Chest CT

INSPECT defines 3 prognostic binary tasks: hospital mortality, hospital readmission, and pulmonary hypertension (PH); and 1 binary diagnostic classification task (pulmonary embolism). Using the provided EHR data we define 5 additional prognostic TTE tasks for lung pathologies: ATX (Atelectasis), CMG (Cardiomegaly), CONS (Consolidation) EDM (Edema), and PEFF (Pleural Effusion). These tasks were selected based on their use in common chest medical imaging datasets (Irvin et al., 2019). Appendix B details how labels were assigned. RSPECT provides whole-volume labels for 9 diagnostic tasks, but no longitudinal outcome data. RSPECT does not provide a public test set, so we impose a 80/5/15 split on train (n=7,279) for our experiments.

Architectures.

We evaluate three model architectures: SwinUNETR, DenseNet, and ResNet. SwinUNETR (Tang et al., 2022) was originally designed for medical image segmentation tasks, combining elements from the Swin Transformer and the UNETR (U-Net Transformer) architectures. Weights are learned using a reconstruction loss on 10,050 3D brain/chest CT and MRI data (Valanarasu et al., 2023), a much larger pretraining dataset than used by other public 3D models, e.g., Wasserthal et al. (2023). Following Merlin, we adapt DenseNet-121 (Huang et al., 2016) and ResNet-152 (He et al., 2015) by inflating their 2D pretrained ImageNet weights (specifically the filters and pooling kernels) as described in Carreira & Zisserman (2017). Note that ResNet and DenseNet parameters were initialized using 2D weight inflation, but the architecture is fully 3D, using 3D convolutions in each Res-block or Dense-block, different from 2.5D methods (Hung et al., 2024). This process enables us to input 3D CT images into the models for training. We also evaluated Merlin’s pretrained ResNet-152 backbone, but found it performed similar to our base ResNet-152 (see Appendix E), thus we use the base weight-inflated model for consistency across experiments.

Model Baselines.

We evaluate the following continued pretraining approaches on all architectures:

• base: Baseline performance of the 3D pretrained SwinUNETR, DenseNet-121, and ResNet-152 models without continued pretraining on INSPECT. See Appendix H for a summary of the source pretraining datasets.

• base/MTL: Continued pretraining of the base model via multitask, supervised learning using the 8 INSPECT evaluation task labels. This controls for exposure to our training dataset in a consistent manner across architectures.

• base/visit: Continued pretraining using the same 8,192 tasks used for TTE supervision, but restricting label assignment to the same visit as the CT scan. This ablates the TTE component to learn temporal information and aligns more closely with the EHR-based supervision used by Merlin.

• base/TTE: Continued pretraining using TTE labels for 8,192 tasks occurring after the CT scan index timestamp.

All models except base use the same INSPECT training set examples for continued pretraining. Pretraining task labels are assigned per-CT scan and vary in density based on pretraining approach (Figure 3). Note that TTE supervision enables leveraging a patient’s entire future EHR, providing 3 times more training labels on average per CT-scan over per-visit labels. TTE also captures temporal structure and time-varying disease risk, providing supervision signal that is absent in predominant SSL methods for imaging.

Figure 3:

Label density CDF by pretraining approach.

For evaluating our pretrained encoder, we use the frozen encoder and lightweight task head strategy outlined in Section 4. For classification tasks, we use logistic regression as the classification head (linear probe) and for TTE tasks, we employ DeepSurv (Katzman et al., 2018) as the survival head. All model search hyperparameters for pretraining and adaptation are in Appendix F.

Metrics.

We evaluate discrimination performance using AUROC for time-thresholded binary classification tasks and Harrell’s C-index (Harrell et al., 1982) for TTE tasks. Harrell’s C-index is a type of C statistic that summarizes the ability of a predictive model to rank patients. Harrell’s C-index requires the predictive model to output a single risk per patient, as opposed to risk over time, thus suitable for our DeepSurv head evaluation. We use the standard formulation for Harrell’s C-index, where it first finds all possible pairs $P$ and one patient is known to have the event before another, then splits that set into $P_{\text{correct}}$ (the higher predicted risk patient has the event first), $P_{\text{tied}}$ (both patients are tied for the predicted risk), and $P_{\text{incorrect}}$ (lower predicted risk patient actually has the event first): $C_H=\frac{P_{\text{correct}}+0.5\times P_{\text{tied}}}{P_{\text{correct}}+P_{\text{tied}}+P_{\text{incorrect}}}$. Appendix D contains additional TTE evaluation metrics for time-dependent C-statistics and the integrated Brier score.

All performance results and 95% confidence intervals are reported using a test set bootstrap of $n=1000$ replicates. Statistical significance was computed using a two-tailed Z-test (p-value at 0.05 for rejecting the null hypothesis that the difference between two sample means is zero). A complete set of statistical tests are in Appendix K.

Additional Experiments.

See the appendix for further experimental ablations including: subgroup performance (Appendix R), task head capacity (Appendix L), and full fine-tuning vs. frozen backbone adaptation (Appendix J).

6. Results

Evaluating Prognostic Performance.

Table 2 reports performance for the prognostic binary classification formulations of outcome prediction for the original INSPECT tasks. This provides a simplified view of high and low-risk patients across time. We find that TTE pretrained models outperform all baselines across all architectures in our experiments. Here TTE pretraining provides an average of 22.6% performance over the base pretrained model and 15.4% average increase over base/visit. The base/MTL baseline also outperforms the base model, but underperforms TTE, highlighting the benefits of increasing pretraining tasks. Comparing base/TTE versus base/visit is more informative, as both approaches use the same number of pretraining tasks (8,192) but base/TTE substantially improves the density of the label per image during pretraining (Figure 3). Table 3 reports performance of all original INSPECT tasks and our five new outcomes using Harrell’s C-index. Here TTE pretraining largely outperforms all of our baselines across all architectures.

Table 2:

Prognostic binary classification performance for INSPECT tasks on Logistic Regression (1, 6, 12 month time horizon bins) reported as the mean AUROC of a test set bootstrap (n=1000). Bold indicates the best performer. Underlined indicates no statistically significant difference versus the _*base/TTE models.

Model	Mortality			Readmission			PH
	1M	6M	12M	1M	6M	12M	12M
SwinUNETR base	0.693	0.684	0.685	0.507	0.538	0.569	0.597
SwinUNETR base/MTL	0.676	0.700	0.697	0.502	0.543	0.551	0.606
SwinUNETR base/visit	0.693	0.716	0.670	0.560	0.554	0.528	0.560
SwinUNETR base/TTE	0.827	0.808	0.788	0.582	0.612	0.607	0.672
DenseNet-121 base	0.616	0.568	0.575	0.512	0.532	0.524	0.506
DenseNet-121 base/MTL	0.665	0.596	0.602	0.533	0.557	0.547	0.577
DenseNet-121 base/visit	0.649	0.698	0.692	0.504	0.538	0.567	0.599
DenseNet-121 base/TTE	0.770	0.730	0.725	0.629	0.643	0.637	0.689
ResNet-152 base	0.583	0.557	0.554	0.536	0.537	0.509	0.537
ResNet-152 base/MTL	0.726	0.715	0.647	0.563	0.564	0.566	0.570
ResNet-152 base/visit	0.691	0.636	0.643	0.562	0.564	0.566	0.567
ResNet-152 base/TTE	0.804	0.798	0.792	0.602	0.649	0.657	0.718

Table 3:

Prognostic TTE performance for INSPECT, measured by Harrell’s C-Index. Bold indicates the best performance. Underlined indicates no statistically significant difference versus the _*base/TTE models.

Model	Harrell’s C-Index ↑
	Mort.	Readm.	PH	ATX	CMG	CONS	EDM	PEFF
SwinUNETR base	0.717	0.653	0.696	0.558	0.662	0.549	0.697	0.641
SwinUNETR base/MTL	0.672	0.671	0.665	0.681	0.677	0.668	0.715	0.668
SwinUNETR base/visit	0.671	0.716	0.697	0.719	0.717	0.718	0.717	0.714
SwinUNETR base/TTE	0.738	0.723	0.724	0.739	0.739	0.738	0.738	0.738
DenseNet-121 base	0.505	0.505	0.541	0.505	0.505	0.506	0.505	0.536
DenseNet-121 base/MTL	0.589	0.590	0.591	0.593	0.589	0.586	0.587	0.590
DenseNet-121 base/visit	0.675	0.675	0.699	0.675	0.662	0.676	0.676	0.675
DenseNet-121 base/TTE	0.732	0.723	0.726	0.720	0.711	0.712	0.725	0.723
ResNet-152 base	0.505	0.560	0.505	0.577	0.505	0.559	0.505	0.536
ResNet-152 base/MTL	0.701	0.686	0.656	0.663	0.562	0.656	0.660	0.572
ResNet-152 base/visit	0.656	0.702	0.643	0.703	0.705	0.703	0.700	0.716
ResNet-152 base/TTE	0.732	0.739	0.735	0.728	0.727	0.727	0.737	0.737

Evaluating Diagnostic Performance.

We are also interested in assessing how TTE continued pretraining may impact standard image classification tasks, corresponding to diagnostic labeling of medical imaging for current disease biomarkers. Table 4 outlines 8 image biomarker tasks. Here, performance of all 3D model architectures is poor, especially base models. For almost all diagnostic tasks, TTE pretraining performs the same (i.e., statistically indistinguishable, detailed numbers shown in Table 17) as all other tested pretraining approaches. This aligns with the intuition that visit-level labels reflect current clinical events, should encode the same level of diagnostic information as TTE pretraining. This also aligns with the small performance gains reported in prior work when tasks derived from EHR codes to supervise image models (Blankemeier et al., 2024). Note that while these tasks reflect observable pixel biomarkers present in images, there is also overlap with pixel biomarkers indicative of future risk. For example, a RV/LV ratio ≥ 1, defined as the ratio of the right ventricular (RV) diameter to the left ventricular (LV) diameter, is indicative of increased risk of mortality (Lu et al., 2012). Here TTE pretraining yields statistically significant performance improvements across all architectures and pretraining methods.

Table 4:

Diagnostic binary classification performance for RSPECT, reported as the mean AUROC of a test set bootstrap (n=1000). Bold indicates the best performer. Underlined indicates no statistically significant difference versus the _*base/TTE models.

Model	PE						RV/LV Ratio
	Left	Cent.	Right	Chronic	Acute	Indet.	<1	≥1
SwinUNETR base	0.573	0.571	0.573	0.507	0.581	0.721	0.525	0.578
SwinUNETR base/MTL	0.583	0.633	0.591	0.504	0.594	0.732	0.517	0.598
SwinUNETR base/visit	0.545	0.664	0.562	0.556	0.595	0.741	0.521	0.546
SwinUNETR base/TTE	0.634	0.651	0.633	0.525	0.716	0.781	0.643	0.636
DenseNet-121 base	0.596	0.615	0.596	0.565	0.676	0.724	0.581	0.607
DenseNet-121 base/MTL	0.612	0.638	0.615	0.504	0.664	0.713	0.594	0.636
DenseNet-121 base/visit	0.595	0.633	0.605	0.571	0.677	0.748	0.573	0.615
DenseNet-121 base/TTE	0.647	0.716	0.644	0.586	0.665	0.762	0.623	0.686
ResNet-152 base	0.654	0.687	0.618	0.507	0.695	0.704	0.593	0.626
ResNet-152 base/MTL	0.658	0.693	0.621	0.506	0.685	0.703	0.587	0.642
ResNet-152 base/visit	0.646	0.665	0.625	0.548	0.743	0.718	0.566	0.665
ResNet-152 base/TTE	0.657	0.687	0.642	0.504	0.588	0.722	0.597	0.708

7. Discussion and Conclusion

Medical images hold significant, untapped potential as sources of imaging biomarkers to predict future disease risk. However, current SSL approaches for imaging largely fail to capture the temporal dynamics of long-term disease progression, inherently capturing only static, structural information. Building on this observation, our work explores a TTE pretraining technique that directly incorporates future temporal information at scale, yielding several insights.

Current Pretraining Struggles to Learn Prognostic Pixel Biomarkers.

Existing supervised and SSL pretraining methods consistently showed lower AUROC across prognostic tasks in our experiments. When compared against base models, TTE pretraining showed an average increase in AUROC of 0.128 (95% CI: [0.075, 0.158]), Harrell’s C-index 0.166 (95% CI [0.138, 0.490]) over three architectures on prognostic binary classification and prognostic TTE tasks respectively. This suggests that off-the-shelf pretraining methods struggle to learn pixel biomarkers associated with future disease risk, likely due to missing temporal links to future pathologies. This underscores Huang et al. (2020)’s findings that predicting disease prognosis with high accuracy and certainty remains a challenging task. Therefore, incorporating explicit temporal supervision into the pretraining process may be essential for improving prognostic tasks’ performance.

TTE Pretraining Improves Prognostic Performance.

When compared against base/MTL and base/visit , TTE supervision improves performance on Harrell’s C-index on prognostic TTE task tasks by 0.093 (95% CI [0.086, 0.554]) and, 0.038 (95% CI [0.017, 0.370]) respectively. Additionally TTE supervision does not negatively impact diagnostic binary classification for significant difference between base and TTE across RSPECT tasks, shown in Table 17.

TTE Supervision Improves Training Data Efficiency.

By leveraging temporal information from the future in EHRs and censored patients, TTE pretraining increases label density to boost AUROC for prognostic binary classification tasks by 0.093 (95% CI [0.053, 0.134]) and 0.092 (95% CI [0.051, 0.135]) when compared against base/MTL and base/visit models respectively. TTE supervision increases available task labels per-training instance by 3x on average (Figure 3). This underscores the potential of TTE pretraining as a viable objective for scaling medical imaging AI, given that expert-level annotation is time-consuming, expensive, and difficult to collect (Dgani et al., 2018; Tajbakhsh et al., 2021; Aljabri et al., 2022). Increasing the data efficiency of a given training example also contributes to reducing compute costs.

Limitations.

First, our study focuses exclusively on evaluating 3D vision encoders, which demand significant memory and computational resources—80GB memory GPUs for SwinUNETR and 40GB memory GPUs for DenseNet/ResNet architectures. Second, while our study surpasses previous work in imaging-based TTE in terms of scale, our pretraining dataset remains relatively small compared to modern, general-purpose datasets (Schuhmann et al., 2022) and recently released medical datasets (Xie et al., 2024). Furthermore, we focus exclusively on a single modality, CT scans. Expanding both the scale and diversity of the pretraining data mixtures (e.g., other imaging modalities including both 2D and 3D, historical EHR and clinical text) could enhance performance or lead to a deeper understanding of the trade-offs between different architectures. Finally, since this work focuses on evaluating encoder quality, we only evaluated frozen encoders with smaller, lightweight, supervised task heads for adaptation. Alternative adaption methods under different sample assumptions, e.g., zero/few-shot learning, may reveal different performance trade-offs.

Conclusion.

This work presents the first empirical study of using time-to-event pretraining for 3D medical vision models. By leveraging longitudinal EHRs and defining a time-to-event pretraining objective (comprising 8,192 tasks), we embed long term outcome information into the image model during pretraining. Doing so results in an average threefold increase in training labels compared to limiting just to a patient’s current EHR visit. We observe substantial improvements in prediction performance for future events, with increases of up to a 31.6% in AUROC and a 40.5% improvement in Harrell’s C-index, without negatively impacting standard binary classification tasks (e.g., image labeling for diagnostic tasks). Our results reveal a clear need for pretraining datasets for 3D medical foundation models to include tasks that capture long-term temporal structure, demonstrated here through our use of time-to-event supervision. This study on the utility of using longitudinal EHR records as a supervision source for future-guided pretraining of 3D medical imaging models lays the groundwork for innovative ways to combine EHR and imaging modalities for clinical risk prediction.