Predicting early-stage breast cancer disease-free survival and adjuvant therapy benefit from multimodal information using deep learning

Abstract

Accurate prediction of disease-free survival (DFS) is essential for tailoring adjuvant regimens and improving clinical outcomes in early-stage breast cancer (EBC). A multimodal deep learning model (Mu-model) based on deep canonical correlation analysis (DCCA) integrating multiparametric magnetic resonance imaging (MRI)—including dynamic contrast-enhanced MRI (DCE-MRI) and diffusion-weighted imaging (DWI)—with clinical text data is developed to predict DFS and identify patients likely to benefit from adjuvant therapy. This multicenter retrospective study includes 1,120 patients with EBC (training cohort, n = 459; external validation cohort, n = 661). The Mu-model achieves concordance index (C-index) values of 0.742 (95% confidence interval [CI]: 0.662–0.821) in the training cohort (TC) and 0.735 (95% CI: 0.667–0.803) in the external validation cohort (EVC) for DFS. The Mu-model score (MuS) remains an independent prognostic factor after adjustment for clinicopathologic variables (all P < 0.05). In human epidermal growth factor receptor 2 (HER2)-positive, hormone receptor (HR)-positive, and T2-stage subgroups, a significant survival benefit associated with adjuvant therapy was observed in patients with low MuS, whereas no statistically significant association was detected in patients with high MuS. Transcriptomic analysis in 19 patients indicates that high MuS is associated with immune activation and enrichment of cell-cycle and purine-metabolism pathways. The Mu-model provides non-invasive DFS prediction and recurrence risk stratification, while preliminarily exploring its potential to identify patients who may derive differential benefits from adjuvant therapy.

Introduction

Breast cancer has become the most common malignancy among women worldwide¹. It's marked biological heterogeneity substantially increases the complexity of prognostic assessment². Despite the prevalence of early-stage diagnoses³(stage I–IIIA) and efficacy of adjuvant therapy^4,5, approximately 10%–41% of early-stage patients still experience relapse⁶, which remains the predominant cause of breast cancer–related mortality⁷. Currently, clinical risk stratification and adjuvant treatment regimen decisions primarily rely on traditional factors such as molecular subtype and clinical stage. Genetic tests such as Oncotype DX⁸ and MammaPrint⁹ have been shown to effectively identify recurrence risk and predict the benefits of adjuvant chemotherapy. However, these tests are typically invasive and costly, and existing traditional clinical and pathological factors remain limited in capturing intratumoral heterogeneity, making it difficult to accurately identify patients who will truly benefit from therapy^10,11. Thus, clinicians and patients persistently encounter the challenge of overtreatment and undertreatment^12,13. A non-invasive and precise stratification tool is therefore essential to assist in identifying patients at high risk of recurrence and to facilitate the evaluation of treatment benefits across groups.

Multiparametric MRI (mpMRI) have been integrated into routine clinical practice, facilitating non-invasive evaluation of tumor biology. Research has focused on identifying imaging biomarkers that predict patient prognosis and guide personalized treatment¹⁴. Specifically, higher volume of enhancing parenchyma¹⁵ and values of kinetic heterogeneity¹⁶ observed in preoperative dynamic contrast-enhanced MRI (DCE-MRI) have been shown to correlate with poor outcomes in breast cancer patients; meanwhile, diffusion-weighted imaging (DWI) provides insights into tissue cellularity and aiding in the characterization of breast lesions, combing with DCE-MRI, it has the potential to provide extra information of tumor prognosis from both functional and anatomical perspectives^17–20. However, conventional quantitative parameter-based analyzes are constrained by predefined quantitative parameters and empirical thresholds, which insufficiently reflect the complex intratumoral heterogeneity, thereby compromising prognostic precision. Manual whole-tumor segmentation in actual clinical workflows is extremely time-consuming and difficult to standardize, limiting its large-scale application. Therefore, a clinically applicable deep learning tool integrating multimodal MRI information is likely to outperform conventional single-parameter approaches in capturing tumor aggressiveness, thereby improving risk stratification in early-stage breast cancer.

Despite the growing use of mpMRI, conventional analytic approaches, including single-parameter assessments and handcrafted radiomics, remain constrained by predefined features and insufficient multimodal integration, hindering comprehensive characterization of intratumoral heterogeneity. To address these challenges, deep learning has emerged as a transformative paradigm for synthesizing complex, high-dimensional, and multimodal data²¹. Unlike traditional radiomics or biomarker-based approaches that depend on manually engineered features, deep learning autonomously extracts hierarchical representations directly from imaging modalities and clinical data²², thereby enhancing predictive accuracy and model generalizability. While single-modality models have shown promise in prognostic prediction for breast cancer²³, their clinical utility remains limited by challenges in fully capturing the spectrum of tumor heterogeneity—a critical determinant of therapeutic resistance and clinical outcomes²⁴. Multimodal deep learning, by enabling the joint learning of complementary information across different imaging modalities and multiomics data^25–27, offers a more comprehensive and biologically meaningful characterization of tumor behaviors. However, prevalent fusion strategies—such as simple feature concatenation or attention-based pooling—often fail to explicitly model the complex non-linear correlations between modalities, potentially leading to suboptimal alignment, mutual interference, or loss of complementary information. Deep canonical correlation analysis (DCCA) addresses this limitation by projecting heterogeneous inputs into a shared latent space through nonlinear mappings that maximize cross-modal correlations^28,29. This approach-level shift allows for the extraction of highly synchronized, cross-modal representations that filter out modality-specific variations while preserving core biological signals related to heterogeneity and prognosis, thereby providing a more robust foundation for characterizing tumor heterogeneity compared with conventional feature-level integration strategies.

Therefore, our purpose was to develop and validate a multimodal deep learning model (Mu-model) that integrates preoperative DCE-MRI, DWI and clinical data to predict DFS in patients with early-stage breast cancer. We further explore the model’s potential to stratify patients for adjuvant therapy benefit, serving as a hypothesis-generating basis for future studies. In addition, we aimed to enhance interpretability by visualizing key regions contributing to the model’s predictions through heatmaps and by performing an exploratory RNA sequencing to investigate potential biological correlates.

Results

Patient’s characteristics

The clinicopathological characteristics of the patients in the training cohort (TC, n = 459) and external validation cohort (EVC, n = 661) are summarized in Table 1. Among them, 811 (72.4%) were over 45 years, and 594 (53.0%) were premenopausal. The majority of patients (n = 922, 82.3%) underwent mastectomy, and 896 (78.7%) patients received adjuvant therapy. The median follow-up time was 48.5 months (interquartile range [IQR], 23.62–61.72 months) for the training cohort, 57.3 months (IQR, 21.00–82.17 months) for the external validation cohort. A total of 103 patients experienced recurrence or metastasis during the follow-up period.

Table 1.

Clinicopathologic characteristics in the entire, training and external validation cohorts

Variables	Entire cohort (n = 1120)	Training cohort (n = 459)	External validation cohort (n = 661)	P value
Age				0.404
<45	309 (27.6%)	120 (26.1%)	189 (28.6%)
≥45	811 (72.4%)	339 (73.9%)	472 (71.4%)
Menopausal status				0.183
Premenopausal	594 (53.0%)	232 (50.5%)	362 (54.8%)
Postmenopausal	526 (47.0%)	227 (49.5%)	299 (45.2%)
Location				0.902
Left	582 (52.0%)	237 (51.6%)	345 (52.2%)
Right	538 (48.0%)	222 (48.4%)	316 (47.8%)
Histological grade				0.005
I-II	684 (61.1%)	303 (66.0%)	381 (57.6%)
III	436 (38.9%)	156 (34.0%)	280 (42.4%)
cT stage				<0.001
T1	456 (40.7%)	217 (47.3%)	239 (36.2%)
T2	664 (59.3%)	242 (52.7%)	422 (63.8%)
Ki67				<0.001
Low (<10%)	421 (37.6%)	124 (27.0%)	297 (44.9%)
High (≥10%)	699 (62.4%)	335 (73.0%)	364 (55.1%)
ER				<0.001
Negative	200 (17.9%)	34 (7.4%)	166 (25.1%)
Positive	920 (82.1%)	425 (92.6%)	495 (74.9%)
PR				<0.001
Negative	206 (18.4%)	53 (11.5%)	153 (23.1%)
Positive	914 (81.6%)	406 (88.5%)	508 (76.9%)
HER2				<0.001
Negative	443 (39.6%)	106 (23.1%)	337 (51.0%)
Positive	677 (60.4%)	353 (76.9%)	324 (49.0%)
Molecular subtype				<0.001
Luminal	970 (86.6%)	419 (91.3%)	551 (83.4%)
Others	150 (13.4%)	40 (8.7%)	110 (16.6%)
Surgery type				<0.001
Mastectomy	922 (82.3%)	307 (66.9%)	615 (93.0%)
Others	198 (17.7%)	152 (33.1%)	46 (70.0%)

P values were calculated using the t test for continuous variables and the χ2 test for categorical variables. P Values in bold indicates statistical significance. “Others” in molecular subtype refers to HER2-enriched and triple-negative breast cancer; “others” in surgery type refers to breast-conserving surgery (BCS).

ER estrogen receptor, PR progesterone receptor, HER2 human epidermal growth factor receptor 2.

Prognostic performance of the Mu-model

In the TC, the optimal cutoff for the Mu-model score (MuS) was determined to be 0.040. A significant difference in DFS was observed between patients with low and high MuS in both TC and EVC, with hazard ratio (HR) of 0.226 (95% CI: 0.121–0.420, p < 0.0001) in TC and 0.243 (95% CI: 0.147–0.400, p < 0.0001) in EVC (Fig. 1A, B). The 5-year DFS rate was 93.9% in the low MuS group versus 76.7% in the high MuS group in TC, and 92.5% versus 72.8% in EVC. The 7-year DFS rate was 90.5% in the low MuS group compared to 68.8% in the high MuS group in TC, and 90.3% versus 65.7% in EVC. We also evaluated MuS performance within subgroups defined by clinicopathologic risk factors (Fig. S4). Significant DFS differences between high and low MuS were consistently observed across all subgroups, including Age (<45, ≥45 years), menopausal status, location, histologic grade, clinical T stage, Ki67, ER, PR, HER2, and molecular type.

Fig. 1. Prognostic performance of the Mu-model.

Kaplan–Meier analyses of disease-free survival (DFS) according to the Mu-model in the A training cohort and B external validation cohort, with HRs of 0.226 (95% CI: 0.121–0.420, p < 0.0001) and 0.243 (95% CI: 0.147–0.400, p < 0.0001), respectively. CI confidence interval, HR hazard ratio.

Predictive performance of the Mu-model

We evaluated the predictive performance of the Mu-model for DFS. The C-index was 0.742 (95% CI: 0.662–0.821) in TC and 0.735 (95% CI: 0.667-0.803) in EVC. Time-dependent receiver operating characteristic (ROC) analysis further assessed the model’s accuracy at 5 and 7 years (Fig. S5A, B), yielding AUCs of 0.735 (95% CI: 0.626–0.843) and 0.757 (95% CI: 0.639-0.875) in TC, and 0.763 (95% CI: 0.688–0.839) and 0.722 (95% CI: 0.632–0.811) in EVC. The MuS was significantly higher in patients who experienced events compared to those who did not in both cohorts (all P < 0.001, Fig. S5C, D). We also performed univariate and multivariate Cox regression analyses of clinicopathologic variables (Tables S2 and 2) and identified that MuS, menopausal status and clinical T stage were significant prognostic factors. Menopausal status and clinical T stage were used to construct a clinical model for DFS prediction. The C-index of this clinical model was 0.647 (95% CI: 0.517–0.778) in TC and 0.636 (95% CI: 0.527–0.745) in EVC. To quantify the added predictive value of the Mu-model over the clinical model, we calculated the Net Reclassification Improvement (NRI), which was 0.254 (95% CI: 0.102–0.401; P < 0.001) in TC and 0.283 (95% CI: 0.167–0.373; P < 0.001) in EVC. Additionally, the prediction error curves further demonstrated lower prediction errors (Fig. S5E, F) and a smaller Integrated Brier Score for the Mu-model compared to the clinical model (0.082 vs 0.089 in TC and 0.073 vs 0.083 in EVC, respectively), indicating superior predictive performance. Moreover, the calibration curves at 3, 5, and 7 years showed good agreement between predicted probabilities and observed outcomes across both cohorts (Fig. S9). Specifically, Brier scores ranged from 0.044 to 0.111 in TC and from 0.061 to 0.108 in EVC, confirming robust and consistent calibration performance of the Mu-model. As expected for the observed probability distribution, the calibration curves were concentrated in the moderate to high predicted survival probability range, reflecting the limited number of low-probability events. Finally, two-way cross-validation and ablation experiments also demonstrated the superior performance of our model in predicting DFS (Tables S6 and S3).

Table 2.

Multivariable Cox regression analysis of disease-free survival for early-stage breast cancer in training and external validation cohorts

Variables	Training cohort (n = 459)		External validation cohort (n = 661)
	HR (95% CI)	P Value	HR (95% CI)	P Value
MuS*	1.545 (1.199–1.989)	<0.001	1.768 (1.382–2.262)	<0.001
Age (<45 y vs≥45y)	0.407 (0.124–1.333)	0.138	0.729 (0.330–1.610)	0.434
Menopausal (pre vs post)	3.387 (1.140–10.070)	0.028	1.586 (0.763–3.297)	0.217
Location (left vs right)	0.694 (0.351–1.373)	0.294	0.859 (0.472–1.562)	0.904
Histological (I-II vs III)	1.695 (0.814–3.528)	0.158	1.169 (0.594–2.300)	0.652
cT stage (T1 vs T2)	0.411 (0.186-0.909)	0.028	1.036 (0.586–1.832)	0.618
Ki67 (<10% vs≥10%)	2.292 (0.897–5.857)	0.083	1.253 (0.680–2.309)	0.470
ER (negative vs positive)	0.558 (0.131–2.377)	0.431	0.548 (0.244–1.229)	0.144
PR (negative vs positive)	1.428 (0.324–6.291)	0.638	1.268 (0.380–4.238)	0.699
HER2 (negative vs positive)	0.906 (0.4297–1.91)	0.796	1.133 (0.636–2.020)	0.671
Molecular (luminal vs others)	2.193 (0.466–10.32)	0.321	0.690 (0.155–3.078)	0.627
Surgery (mastectomy vs others)	0.475 (0.193–1.168)	0.105	1.496 (0.449–4.984)	0.512

*Continuous variable. P Values in bold indicates statistical significance. Only variables identified as significant in the univariable analyses were included in the multivariable analysis. “Others” in molecular subtype refers to HER2-enriched and triple-negative breast cancer;” others” in surgery type refers to breast-conserving surgery (BCS).

HR hazard ratio, CI confidence interval, MuS Mu-model score, ER estrogen receptor, PR progesterone receptor, HER2 human epidermal growth factor receptor type 2.

Mu-model predicts survival benefit from adjuvant therapy

To explore the potential utility of the Mu-model in identifying patients who might derive differential benefit from adjuvant therapy, defined as adjuvant chemotherapy in combination with other guideline-recommended adjuvant treatments, we analyzed the association between MuS status and survival across subgroups defined by HER2, HR status, and clinical T stage stratified by receipt of postoperative therapy. After propensity score matching (PSM), baseline characteristics were balanced between patients who did and did not receive adjuvant therapy (Tables S4 and S5). In the low MuS group, adjuvant therapy was associated with improved survival in HER2-positive (HR:0.299, 95% CI: 0.117–0.761, p = 0.008, Fig. 2A), clinical T2 stage (HR:0.150, 95% CI: 0.042–0.533, p = 0.001, Fig. 2B), and HR-positive (HR:0.326, 95% CI: 0.136–0.779, p = 0.009, Fig. 2C) patients, but not in those who were HER2-, HR-negative and clinical T1 stage (Fig. S6). In the high MuS group, adjuvant therapy showed no statistically significant association with survival across most HER2, HR, and clinical T stage subgroups, with the exception of HR-positive patients (Fig. S7). These findings should be interpreted cautiously, given the limited sample size and event rates in the high MuS group. Formal tests of interaction were performed to assess whether the effect of adjuvant therapy on DFS was modified by the MuS across those subgroups. In multivariable Cox models, the adjuvant therapy with MuS interaction was significant for HER2 status (interaction p = 0.023), for HR status (p = 0.036), and for clinical T stage (p = 0.042). These findings suggest potential heterogeneity in adjuvant therapy benefit according to those subgroups. Similar findings were observed when the analysis was repeated in the full cohort without matching in the low MuS group (Fig. S8).

Fig. 2. Association between MuS and adjuvant benefit in propensity-matched HER2 positive, cT2 stage and HR positive patients.

Kaplan–Meier curves of disease-free survival for patients in the low MuS group. In Low MuS group, patients with A HER2 positive, B HR positive, and C clinical T2-stage derived greater benefit from adjuvant therapy compared with those who did and did not receive it, with HRs of 0.299 (95% CI: 0.117–0.761, p = 0.008),0.150 (95% CI: 0.042–0.533, p = 0.001) and 0.326 (95% CI: 0.136–0.779, p = 0.009), respectively. MuS Mu-model score, HR hormone receptor, HER2 human epidermal growth factor receptor 2.

Visualization and Interpretability of the model from the image level

We first visualized the representation capability of DCCA for multimodal feature correlation. As shown in Fig. 3, the feature distributions before and after applying DCCA are displayed. Compared to Fig. 3A, the features of the two modalities in Fig. 3B exhibit substantial overlap, indicating that the model successfully captured highly similar representations across modalities. Subsequently, Grad-CAM was used to visualize four representative cases (Fig. 3C). Compared to non-recurrent patients, the model focused more on the tumor margins in DCE images of recurrent patients, potentially suggesting increased invasiveness. In DWI images, the model highlighted regions of significant signal hyperintensity in high MuS patients that correlate with restricted diffusion. These areas likely represent high cell density and microstructural complexity, consistent with the aggressive biological nature of the disease. Quantitative validation via the Pointing Game Score (PGS) further supported these observations: PGS was significantly higher in the recurrence group compared to the non-recurrence group in both DCE images (0.749 vs. 0.728; Mann–Whitney U test, p = 0.021) and DWI images (0.768 vs. 0.740; p < 0.001), demonstrating superior localization accuracy on prognostically relevant tumor subregions in recurrent/high-risk cases (Fig. S10).

Fig. 3. Visualization of the Mu-model.

t-SNE plots of high-dimensional MRI embeddings before (A) and after (B) deep canonical correlation analysis (DCCA). Each point represents a feature vector, and each color represents an image modal. C Grad-CAM heatmaps overlaid on four representative patients: two recurrent cases and two non-recurrent cases. Recurrent cases display stronger model attention at tumor margins in DCE-MRI compared with non-recurrent case. In DWI, recurrent cases show pronounced focus on peritumoral areas with pronounced diffusion restriction. DCE dynamic contrast-enhanced, DWI diffusion-weighted imaging, DCCA deep canonical correlation analysis, t-SNE t-distributed stochastic neighbor embedding, Grad-CAM gradient-weighted class activation mapping.

Interpretability of the model from the biological mechanism level

Nineteen RNA-seq patients with available data were stratified into high MuS (n = 9) and low MuS (n = 10) groups based on the Mu-model predictions. To explore transcriptomic differences between the groups, differential expression genes (DEGs) analysis was conducted. Out of 15,596 genes, 209 were upregulated and 370 downregulated (Fig. 4A; using adjusted p < 0.05 and |log2FC| > 1.00). Among these DEGs, several have been previously reported to be associated with breast cancer recurrence, metastasis or immune escape³⁰, including NDRG1, SCN1A, HSPA1B, SCGB3A1 and SPON2. GO analysis suggested enrichment of pathways related to growth signaling in high MuS group, including chromosome organization, protein DNA complex subunit organization, chromosome centromeric region, spindle and ubiquitin-like protein ligase binding (Fig. 4B). KEGG pathway analysis indicated enrichment in cell cycle and purine metabolism pathways among upregulated genes (Fig. 4C, D). Moreover, immune infiltration analysis further suggested a potentially more active immune microenvironment in high MuS patients, with increased estimated infiltration of M1 and M2 macrophages, activated T cells, and dendritic cells (Fig. 4E). This heightened immune activity may reflect a tumor-promoting immune landscape, potentially contributing to disease progression and recurrence; however, due to the small and exploratory nature of the cohort, these observations remain preliminary and require validation in substantially larger datasets to confirm biological relevance and generalizability.

Fig. 4. Biological mechanisms underlying Mu-model for disease-free survival prediction.

A Volcano plot of differential gene expression between high MuS (n = 9) and low MuS (n = 10) groups (Orange represents upregulated genes, and blue represents downregulated genes). B Visualization of the enriched Gene Ontology (GO) pathways in high MuS versus low MuS groups. KEGG pathway analyze of high MuS group. C, D Gene set enrichment analysis showed that the high MuS group had upregulation in the cell cycle and purine metabolism pathways. E Immune infiltration patterns in patients with high and low MuS. MuS Mu-model score, NES normalized enrichment score, GO gene ontology.

Discussion

In this study, we developed a multimodal deep learning model (Mu-model) that integrates multiparametric MRI with clinicopathological variables to non-invasively predict disease-free survival (DFS) and stratify recurrence risk in an independent validation cohort. After adjusting for confounding factors, the model remained a robust independent prognostic indicator, consistently identifying patient subgroups with distinct relapse risks regardless of clinicopathological status. Importantly, the Mu-model revealed heterogeneous treatment effects in the low MuS population, enabling the identification of a subset of HER2-positive, HR-positive, or clinical T-stage patients who may benefit from adjuvant therapy. Furthermore, we found that patients with high MuS exhibited upregulation of tumor progression-related genes and enhanced immune activity.

Multimodal models leverage the complementary properties of different modalities through cross-modal fusion, resulting in superior performance compared to single-modal models^24,31. However, the issue of information redundancy in multimodal fusion has often been overlooked³². In this study, we employed DCCA to explore the shared latent subspace between DCE-MRI and DWI features, effectively addressing the problem of information redundancy in multimodal fusion. We then integrated this approach to construct an early breast cancer prognosis prediction model. As anticipated, the Mu-model outperformed single-modality models (C-index: 0.735 vs. 0.602). Furthermore, it also demonstrated superior performance compared to the DC_Mu-model (C-index: 0.735 vs. 0.677), which directly concatenates DCE-MRI and DWI features for prognosis prediction. Our study emphasizes the functional dimension of tumor imaging by combining DCE-MRI and DWI, which provide complementary information on microvascular activity and tissue cellularity. This design enables the network to capture subtle spatial variations in water-molecule diffusion patterns that may be lost during apparent diffusion coefficient (ADC) computation. Previous studies³³, such as that by Yu et al., have used deep learning to integrate multimodal imaging (e.g., T1+C and T2WI) to predict DFS, achieving higher predictive performance (C-index range: 0.79–0.88) than our study. However, our research benefits from capturing the microscopic biological features of tumors via DCE-MRI and DWI, reflecting microvascular activity, blood supply, and other biological factors directly related to DFS—information that conventional sequences cannot provide. Future research integrating both structural (e.g., T2-weighted) and functional (e.g., DWI, DCE-MRI) information may further enhance model performance and interpretability, leading toward a comprehensive imaging signature of breast cancer prognosis. Additionally, other studies have integrated multi-omics data (e.g., metabolomics, transcriptomics, genomics) to enhance model performance^34–36. Yet, the cost and technical complexity of acquiring such data limit its widespread application in clinical practice. In contrast, our study enables non-invasive DFS prediction using conventional imaging techniques, offering greater practicality and scalability.

Adjuvant therapy remains the standard of care in early-stage breast cancer, aiming to reduce recurrence and optimize survival outcome³⁷. However, a substantial proportion of patients derive incremental benefit from adjuvant therapy³⁸. Identifying subgroups most likely to benefit is essential for achieving personalized management and improving prognosis. In this study, we found that a multimodal model integrating multiparametric MR images and clinical information could identify subgroups—specifically HER2-positive, T2 stage, and HR-positive patients—who were more likely to benefit from adjuvant therapy. In contrast, no statistically significant benefit was detected in the high MuS group across most subgroups. However, this observation in the high MuS group is likely constrained by limited statistical power due to its small proportion of the cohort (10.9%, n = 112), low DFS event rate (~25%, 26 events), and very sparse adjuvant therapy exposure in certain subgroups (e.g., only 2 HER2-negative and a few HR-negative cases receiving treatment). These factors increase the risk of type II error and preclude firm conclusions regarding a lack of benefit. Accordingly, these findings remain hypothesis-generating and do not contradict established evidence that high-recurrence-risk patients (e.g., per Oncotype DX criteria) generally benefit from chemotherapy in randomized settings. Despite these limitations, the results suggest that the Mu-model may endorse risk-stratified decision-making (treat vs. not treat), followed by tailored regimen or intensity selection, aligning with the sequential logic of TAILORx⁴, MINDACT³⁹, and RxPONDER⁴⁰. As a non-invasive, imaging-based tool derived from routinely acquired preoperative data, the Mu-model may serve as a parallel or alternative stratification tool in cases where genomic testing is unavailable. However, the retrospective design and modest subgroup sizes limit inferences about regimen-specific benefits (chemotherapy vs. endocrine therapy alone). Prospective validation with pre-specified MuS thresholds, larger cohorts, or integration with genomic signatures will be required to quantify absolute and relative treatment benefits and establish clinical utility.

Although deep learning models have achieved impressive prognostic performance, their “black-box” nature hinders clinical interpretability and acceptance⁴¹. Understanding the biological basis of model predictions can enhance interpretability and clinical applicability. To address this, we applied Grad-CAM visualization to elucidate the decision basis of our model. We observed that, in patients who experienced disease progression, the model predominantly focused on tumor margins and hypervascular regions, which are closely linked to vascular invasion and microenvironmental deterioration—hallmarks of poor prognosis. This observation is consistent with previous evidence showing that hypervascular tumor peripheries, characterized by elevated VEGF expression, promote angiogenesis and distant metastasis, thereby increasing recurrence risk^42,43. Furthermore, RNA-seq analysis revealed that the model-derived deep learning score was significantly associated with tumor immune infiltration and proliferation-related pathways. Patients with high scores had a higher risk of recurrence and exhibited increased infiltration of M0 macrophages, activated mast cells, and neutrophils, which may promote tumor progression through enhanced angiogenesis and immune evasion^44–46. These findings provide biological insight into the model’s predictions and strengthen confidence in its clinical interpretability. Given the limited sample size of RNA-seq sets and the inclusion of historical data, this may compromise the ability of the data to fully reflect current clinical practices. Consequently, these preliminary findings should be viewed as suggestive and warrant further validation in larger, more diverse cohorts.

Our model is primarily based on preoperative MRI images and basic clinicopathological information, which are routinely and widely used for individual tumor assessment and management. Combining this with continuous MuS as an endpoint can improve predictive reliability, particularly for HER2-positive, HR-positive and clinical T2 stage individuals. For individuals with high scores, enhanced adjuvant treatment measures should be prioritized, while enabling cautious treatment de-escalation and optimized surveillance for the low-score subgroup to minimize over-treatment and its associated toxicities. This framework offers a practical approach for integrating artificial intelligence(AI)-based risk assessment into the standard decision-making process.

Our study has several limitations. First, it was based on retrospective data from two centers, which may limit generalizability and introduce bias. Despite significant disparities in baseline characteristics and variations in adjuvant treatment protocols across different centers, the model’s prognostic performance remained consistent. To mitigate the impact of these confounding factors on therapeutic benefit analysis, we employed Propensity Score Matching (PSM). Validation in larger, prospective, and multicenter cohorts is needed. Secondly, the imaging data utilized in the model construction were limited to the maximum cross-sectional slice of the tumor rather than the full three-dimensional (3D) volume. This simplification may result in the loss of spatial heterogeneity information inherent in the entire tumor, potentially constraining the model’s ability to fully capture tumor complexity and impacting predictive performance. Additionally, although our multimodal deep learning model showed potential in guiding adjuvant therapy decisions, the study did not perform detailed stratification or subgroup analyzes based on specific auxiliary treatment regimens, such as chemotherapy or endocrine therapy. Prospective investigations with larger cohorts and more granular treatment data are needed to validate and refine the model’s clinical utility for distinct adjuvant therapy strategies.

In summary, our study demonstrates that a deep learning model based on multiparametric MRI, specifically leveraging DWI, can accurately predict DFS in patients with early-stage breast cancer. Moreover, the interpretability of the model was supported by imaging visualization and exploring RNA sequencing analyzes, which elucidated the biological underpinnings of its predictions. The integration of these imaging modalities offers a promising, non-invasive tool for individualized prognosis and treatment stratification in early breast cancer, with potential to inform adjuvant therapy decisions and improve patient outcomes.

Methods

Study patients

This study protocol was approved by the Ethics Committees of Guangdong Provincial People’s Hospital (GPPH, No.KY2023-956-03) and Yunnan Cancer Hospital (YNCH, No.KYLX2024-262). The requirement for written informed consent was waived because the data were analyzed retrospectively and anonymously. The study adhered to the Declaration of Helsinki and adheres to the Transparent Reporting of Individual Prognostic or Diagnostic Multivariate Predictive Models (TRIPOD) statement⁴⁷.

We retrospectively analyzed 1255 women with histologically confirmed early-stage⁴⁸ (stage I-IIIA) breast cancer who underwent preoperative breast MRI (2014–2020) followed by curative-intent surgery at two institutions. The exclusion criteria were (a) distant metastasis; (b) concurrent malignancy or incomplete remission with previous anticancer therapy; (c) neoadjuvant therapy prior to surgery or failure to undergo surgery; (d) lack of MRI data or insufficient image quality; (e) missing or incomplete clinical follow-up data. Inclusion criteria: (a) histopathologically confirmed breast cancer; (b) MRI performed within 1 month before puncture or surgical treatment; (c) pre-treatment clinical stage I-IIIA (cT1-2N0-2M0) based on the American Joint Committee on Cancer (AJCC) TNM Staging Manual⁴⁹ (8th Edition); (d) patients who underwent surgical treatment after MRI and had pathology available.

In total, 1120 patients were enrolled in this study. Cohort 1, 459 patients from Guangdong Provincial People’s Hospital (GPPH) were reviewed and identified as the training cohort(TC). Cohort 2, 661 patients enrolled from Yunnan Cancer Hospital (YNCH), served as the external validation cohort(EVC). Cohort 3, 19 patients with breast cancer from the GPPH with matched imaging and transcriptomic sequencing data formed a biological validation cohort for biological mechanisms analysis. Despite deriving from the same institution as cohort 1, cohort 3 was non-overlapping with it. The study design and pipeline are shown in Fig. 5. Further detail is found in Fig. S1.

Fig. 5. Workflow of the study.

A Collection of multicenter clinical characteristics and images preprocessing. B A multimodal model was developed, trained via fivefold cross-validation on the GPPH dataset, and validated on the YNCH dataset. C Model performance was assessed by C-index, Kaplan–Meier curves and prediction error curves, and model interpretability was analyzed from both imaging features and biological mechanisms. GPPH Guangdong Provincial People’s Hospital, YNCH Yunnan Cancer Hospital, DCE dynamic contrast enhanced MRI, DWI diffusion weighted imaging.

Study end point

Data were reviewed on 2025. Follow-up was performed according to the recommendations of the National Comprehensive Cancer Network guidelines. The primary endpoint was DFS, defined as the time from the date of surgery to the first documented evidence of clinical (local or regional) recurrence and/or distant metastasis (confirmed by imaging, histologic evidence, or death) for any reason⁵⁰. Recurrence status was confirmed through radiologic, histologic, or clinical evidence, and patients lost to follow-up were censored at the last available visit. Patient follow-up and recurrence status information were obtained from the medical records and imaging system, with the time of data review determined as the date of the last imaging exam or visit. Data on patients who did not recur or were lost to follow-up at the last visit were considered censored in the analysis.

Clinic-pathological data and adjuvant therapy regimens

Clinic-pathological information such as age at first diagnosis, menopausal status, histologic type, estrogen receptor (ER) status, progesterone receptor (PR) status, and human epidermal growth factor receptor type 2 (HER2) status, Ki-67 index, previous breast surgery, and adjuvant therapy were collected. The human epidermal growth factor receptor 2 (HER2) staining intensity was scored as 0, 1+, 2+, or 3+. Tumors with HER2 scores of 3+ and/or HER2 gene amplification confirmed by fluorescence in situ hybridization were regarded as HER2-positive⁵¹. In this study, adjuvant therapy was defined as adjuvant chemotherapy in combination with other guideline-recommended adjuvant treatments, including endocrine therapy, radiotherapy, and targeted therapy based on specific clinical indications and individual tumor subtypes. Treatment decisions were made by multidisciplinary teams in accordance with contemporary clinical guidelines and individualized patient risk profiles; not all patients received the full complement of these therapeutic modalities⁵².

Image acquisition and preprocessing

All patients underwent DCE-MRI and DWI before surgery using standardized protocols. Breast MRI was performed at two institutions with dedicated breast coils on 1.5T or 3.0T scanners from multiple vendors (Philips, GE HealthCare, Siemens Healthineers, and United Imaging Healthcare). Axial fat-suppressed T2-weighted images were acquired prior to contrast administration. DCE-MRI was then performed using a fat-suppressed T1-weighted gradient-echo sequence following intravenous injection of a gadolinium-based agent (0.2 mL/kg and 2 mL/s), followed by a 20 mL saline flush. DWI was acquired prior to contrast employing single-shot echo-planar imaging with fat suppression in the axial plane; diffusion gradients were applied in three orthogonal directions with b values of 0 and 800, 1000, or 1200 s/mm². Detailed acquisition parameters for each center are provided in Table S1. For analysis, DCE-MRI phase with peak tumor enhancement and the DWI images acquired at the highest b value was used.

The tumor region of interest (ROI) was automatically segmented by pretrained nnUnet on all DCE-MRI and DWI images, followed by manual verification and correction by an experienced breast radiologist (Y.F.Y, senior radiologist) using ITK-SNAP (version 4.4.2) in a blinded manner. The final ROIs were subsequently checked for quality and anatomical consistency by another senior radiologist (X.H, with 9 years of diagnostic experience), and any discrepancies were resolved by consensus. The ROI on DWI was defined with reference to the corresponding DCE-MRI and ADC maps.

Image preprocessing was performed using a standardized pipeline to ensure both spatial and intensity consistency across different centers. All MR images were resampled to an isotropic voxel spacing of 1.0 × 1.0 × 1.0 mm³ using nearest-neighbor interpolation to account for differences in voxel dimensions. To reduce inter-scanner contrast variability, image intensities were normalized by converting DCE-MRI and DWI sequences from 12-bit to 8-bit grayscale (0–255), followed by histogram matching and Z-score standardization. Subsequently, for each patient, the axial slice containing the largest tumor cross-section was selected. A rectangular bounding box enclosing the ROI was used to crop the tumor area, which was then resized to 224 × 224 pixels using bilinear interpolation to meet the input requirements of the deep learning model.

Mu-model development and validation

We developed a Mu-model based on ResNet50 architecture integrated with DCCA to predict DFS in EBC using mpMRI (DCE-MRI and DWI) and clinical data (Figs. 5 and S2). The model architecture comprises four key components: (1) a pretrained ResNet50 backbone for feature extraction from both DCE-MRI and DWI sequences, (2) a perceptual projection layer aligning extracted imaging features into a unified representation space, (3) DCCA integration to identify latent correlations between DCE-MRI and DWI feature sets by combining deep learning representations with canonical correlation analysis (CCA), and (4) a clinical data processing module featuring a single-layer perceptron for clinical feature encoding. The final prediction is generated through direct feature concatenation followed by a fully-connected layer. Employing DCCA, which combines the powerful representation capabilities of deep learning with the statistical method of CCA, enables the discovery of common features (or potential correlations) between DCE-MRI and DWI images to enhance the model’s predictive performance for DFS outcomes. The Mu-model was trained using fivefold cross-validation on a TC and validated on an EVC. The model was implemented on the TensorFlow platform using an Adam optimizer with a learning rate of 0.0001 and a batch size of 32 on NVIDIA RTX 4090 GPU. To enhance model generalizability, we employed data augmentation (including horizontal flipping, rotation, scaling, and cropping), dropout (20%), and regularization techniques to minimize the risk of overfitting. The loss function ${\mathcal{L}}_{\mathit{dfs}}$ can be given by: for DFS, we employed the Cox model to evaluate the hazard function. The Cox Proportional Hazards Loss is defined as:

$${\mathcal{L}}_{\mathit{dfs}}=-\sum _{i=1}^N\left(\frac{\exp \left({\beta }^T{x}_i\right)}{{\sum }_{j\in R\left(t_i\right)}\exp \left({\beta }^T{x}_j\right)}\right)$$ 1

Where, N is the number of samples. $x_i$ represents the network output vector of the ith sample. β is the vector of coefficients. $t_i$ is the time of event or censoring for the ith sample. $R\left(t_i\right)$ is the set of samples that are at risk at time t_i.

We employed two complementary validation schemes to assess model robustness: (1) cross-cohort validation, where the model was trained on the EVC and tested on the TC; and (2) pooled validation, where TC and EVC were combined and randomly split into training, validation, and test sets at a 6:3:1 ratio. This approach helped evaluate the impact of distribution shifts between cohorts. Furthermore, to demonstrate the superiority of the proposed Mu-model, we conducted systematic ablation studies to evaluate individual model components (Fig. S3). (1) Removal of the DCCA module: The features of DCE-MRI and DWI were Directly Concatenated instead of being fused via DCCA (denote as DC_Mu-model, Fig. S3A). (2) Without clinical branch: The clinical feature branch was removed from the network (denote as DCEDWI_Mu-model, Fig. S3B). (3-4) Single-modality models: Models based solely on DCE-MRI, DWI were constructed for comparison (denote as DCE_model, Fig. S3C, and DWI_model, Fig. S3D). (5) Backbone replacement: DenseNet121 was used to replace ResNet50 as the backbone while keeping the overall architecture unchanged (denoted as DenseNet121_Mu-model, Fig. S3E). All ablation experiments were trained using the same parameters as the Mu-model.

We evaluated the prognostic efficacy of the Mu-model for predicting DFS in TC and EVC. Using maximally selected rank statistics, we determined the optimal threshold in the TC to stratify patients into high-risk and low-risk groups based on Mu-model-derived score (MuS), which was subsequently applied to the EVC. The prognostic performance of Mu-model was evaluated using the Harrell’s concordance index (C-index). The time-dependent ROC analysis was also performed. Additionally, we developed a clinical model based on multivariable Cox regression, with variables selected through univariable and multivariable Cox analyzes. To quantify the relative improvement in predictive accuracy of the Mu-model over the clinical model, the Net Reclassification Index (NRI) was calculated.

Benefit of adjuvant therapy and subgroup analysis

We evaluated the Mu-model for its ability to predict the benefit of adjuvant therapy in EBC. To reduce selection bias and confounding, we applied a matching strategy (propensity score matching, PSM) to balance patients across MuS-defined groups. PSM was conducted using the “MatchIt” package in R, comparing patients who received therapy with those who did not, based on 1:1 nearest-neighbor matching. Matching variables included age, menopausal status, tumor location, histological grade, cT stage, Ki67, ER, PR, HER2, molecular subtype, and surgery type. Furthermore, we also evaluated the prognostic value of the Mu-model in subgroups defined by clinicopathological factors.

Visualization and molecular mechanism validation of the Mu-model

To interpret the model’s decision process and enhance clinical applicability, we first used t-Stochastic Neighbor Embedding (t-SNE) algorithm to visualize deep canonical correlation features across modalities, demonstrating the effectiveness of the DCCA modular. We then applied gradient-weighted class activation mapping (Grad-CAM)⁵³ to generate attention heatmaps that highlight salient tumor regions most contributed to the risk prediction. To quantitatively assess the reliability of these localizations across the cohort, we computed the PGS, which measures the proportion of highest-activation points falling within ground-truth tumor regions. This allowed for intuitive assessment of the model’s attention to relevant anatomical and functional imaging features.

We further performed radiogenomic validation of our model using matched MRI and RNA transcriptomic sequencing data from 19 EBC patients. After processing the MR images and clinical data to calculate MuS for each patient, we stratified them into high MuS and low MuS groups based on the optimal cutoff value determined from the TC. Differential gene expression analysis using the R package “DESeq2” identified statistically significant differentially expressed genes (DEGs) between the groups (absolute log2 fold change >1 and FDR-adjusted p-value < 0.05). To elucidate the biological significance of these molecular differences, we conducted comprehensive functional annotation through Gene Ontology (GO) analysis and pathway enrichment using the gene set enrichment analysis (GSEA). Additionally, we employed CIBERSORTx (https://cibersortx.stanford.edu/) to characterize and compare the tumor immune microenvironment composition between high MuS and low MuS groups, providing insights into potential immune-related mechanisms underlying the observed survival differences.

Statistical analysis

Statistical analyzes were performed using R software (version 4.4.2). Descriptive statistics were used to summarize the baseline characteristics of the study population. Continuous variables were expressed as mean ± standard deviation (SD) or median with IQR, and categorical variables were presented as frequencies and percentages. The differences between groups for continuous variables were assessed using the independent t-test, while categorical variables were compared using the χ² test or Fisher’s exact test where appropriate. The MuS of the model and baseline clinicopathological factors were analyzed using univariate and multivariate Cox regression analysis. To minimize the impact of confounding by indication and baseline imbalances, PSM was performed using a 1:1 nearest neighbor matching algorithm to maximize the utilization of available data and maintain statistical power. The propensity score was calculated using a logistic regression model incorporating clinical variables. A caliper width of 0.2 of the standard deviation of the logit of the propensity score was applied. Covariate balance was assessed using standardized mean differences (SMD), with an SMD < 0.1 considered indicative of negligible imbalance.

We determined the optimal threshold to stratify patient into high- and low-risk groups using the maximally selected rank statistics method from the “survminer” R package. We then calculated the DFS probability for those groups using Kaplan–Meier survival analysis and compared them with a log-rank test. In all analyzes, P <0.05 was indicative of a statistically significant difference.

Author contributions

Y.F.Y., Z.Y.X., Z.J.Z., Z.Y.L., Y.W., C.H.L., and L.W. conceived and designed the study. Y.F.Y. and Z.Y.X. collected the data. Y.F.Y., Z.Y.X., Z.J.Z., and L.W. analyzed the data and contributed to data interpretation. L.W. developed the deep learning model and processed the genomic data. Z.Y.L., Y.W., C.H.L., and L.W. critically revised the manuscript for important intellectual content. Y.F.Y., Z.Y.X., Z.J.Z., and L.W. performed the statistical analyses. Z.Y.X., Z.Y.L., Y.W., C.H.L., and L.W. acquired funding.Y.F.Y., Z.Y.X., Z.J.Z., J.Y.L., X.H., K.X.C., and L.W. drafted the manuscript. All authors read and approved the final manuscript.

Data availability

The datasets generated and/or analysed during the current study are not publicly available due to patient privacy and ethical restrictions but are available from the corresponding author on reasonable request.

Code availability

All codes used for this study can be accessed at https://github.com/WuLei-MedIA/Mumodel4DFS.git.

Competing interests

The authors declare no competing interests.

Supplementary information

The online version contains supplementary material available at https://doi.org/10.1038/s41523-026-00991-4.