A multimodal deep learning classifier for prediction of HER2-low expression in triple-negative breast cancer

Abstract

Background

Triple-negative breast cancer (TNBC) is an aggressive molecular subtype. With the new definition of HER2-low status and the availability of anti-HER2 drug conjugates, the drug regimens for HER2-low patients in TNBC could be individually tailored.

Methods

A total of 122 patients with 122 pathologically confirmed TNBCs was retrospectively investigated from 2017 to 2022. Breast US and multiparametric MRI (T2WI, DWI and DCE-MRI) were collected and analyzed. To address data scarcity, a few-shot deep learning framework (FADNET) was developed and evaluated under simulated data-constrained conditions. The model used a frozen ResNet34 backbone trained via contrastive learning and soft k-means for classification. We compared pre-treatment imaging-only models (single modalities and multimodal fusion) against a hybrid model (CLI-FADNET) that incorporated imaging and a postoperative clinical variable for post-surgical assessment.

Results

Of 122 TNBC cases, 13 (11%) patients experienced postoperative recurrence. Most clinical characteristics showed no significant differences between HER2-zero and HER2-low groups except for postoperative lymph node metastasis (≥ 4 positive nodes) (P = 0.004). Pre-treatment models based on a single imaging modality achieved AUCs ranging from 0.764 to 0.806, while the multimodal imaging fusion model improved the AUC to 0.877. The CLI-FADNET model, integrated with the postoperative clinical variable, further increased the AUC to 0.884.

Conclusion

This study validates the FADNET few-shot learning framework for classifying HER2 status under data-constrained conditions. While the imaging-only models offer pre-treatment evaluation, the CLI-FADNET hybrid model serves as a comprehensive post-surgical assessment tool. Rather than presenting a fully deployable clinical model, this methodological validation demonstrates the potential of few-shot learning in rare tumor subtypes, establishing a foundation for future large-scale external validation.

Supplementary Information

The online version contains supplementary material available at https://doi.org/10.1186/s12880-026-02419-8.

Introduction

Breast cancer (BC) is one of the most prevalent malignancies in women and a leading cause of cancer-related mortality worldwide [1]. Triple-negative breast cancer (TNBC) accounts for 15%–20% of BC and is known as the most aggressive molecular subtype, showing earlier progression and worse prognosis. Because TNBC has negative estrogen receptor (ER), progesterone receptor (PR), and human epidermal growth factor receptor 2 (HER2), it is more challenging to treat due to no targetable receptors that can be controlled with hormonal therapy or HER2-targeting dual therapy with trastuzumab and pertuzumab [2]. In recent years, it has been reported patients with low expression of HER2 could benefit from anti-HER2 drug conjugates. Traditionally, TNBC has negative HER2 expression, which was defined as immunohistochemical staining 0, 1+, or 2 + with negative fluorescence in situ hybridization (FISH) [3]. Now, the expression with IHC 1 + and 2 + with negative FISH can be classified as HER2-low, and patients are eligible for treatment using anti-HER2 drug conjugates [4]. Therefore, distinguishing TNBC patients with HER2-low expression is important for offering them better treatments to improve prognosis and survival.

Tissue biopsy remains the gold standard for HER2 status assessment [5]; however, the accurate assessment may suffer from sampling bias and temporal and spatial heterogeneity [6], particularly for most TNBC patients achieving complete pathological response after neoadjuvant chemotherapy [7]. Advances in artificial intelligence (AI) have enabled noninvasive detection technologies based on radiomics to address this limitation [8]. Ramtohul et al. integrated multiparametric MRI with radiomic features to construct a Gradient Boosted Decision Trees (GBDT) model, differentiating HER2-low from HER2-zero and HER2-positive BC [9]. Guo et al. developed a deep learning radiomics model using MRI, which fused spatiotemporal features from T2-weighted imaging and DCE-MRI to predict HER2-low status [10]. Our previous study also demonstrated that HER2-zero and HER2-low could be differentiated from HER2-positive based on preoperative MRI Breast Imaging Reports and Diagnosis System (BI-RADS) features using machine learning models [11]. Nevertheless, these approaches usually require large datasets, which are difficultly achievable in TNBC due to its lower incidence. To address this, Conditional Generative Adversarial Networks model—a framework combining transfer learning and generative adversarial networks—has demonstrated success in generating synthetic HER2 histopathological data and showed high accuracy of recognizing the HRE2 score [12]. It inspired us to find a deep learning framework to noninvasively predict the HER2-low in TNBC patients using imaging modalities.

Other than MRI, breast Ultrasound (US) is another common and important tool to diagnose and evaluate BC in clinical setting, and radiomics features based on US are reported to be associated with the biological behavior of BC [13, 14]. Models focus on single-modality imaging data which may provide restricted information because of technique limitations or individual patient differences; while multimodal imaging data could compensate for each modal’s shortcomings which can reveal tumor features across different dimensions, offering a more comprehensive perspective for clinical diagnosis and treatment decisions [15–18].

Therefore, this study aimed to integrate multimodal imaging features from breast US and multiparametric MRI with clinical data to construct a deep learning framework, providing an efficient and reliable solution for HER2-low expression prediction in TNBC [19, 20]. Our approach focused on two key objectives: First, to develop a small-sample deep learning algorithm optimized for medical imaging to achieve strong model performance even with limited datasets; Second, to systematically compare the model capabilities of single-modality imaging data versus multimodal fusion strategies on the predictive accuracy of HER2-low status.

Materials and methods

Patients and study design

The retrospective study was approved by the Institutional Review Board and written informed consent was waived. Patients with pathologically confirmed TNBC who performed breast MRI and US before surgery from January 2017 to December 2022 were identified. The exclusion criteria and the patient selection process are shown in Fig. 1. A total of 122 patients (mean age 51 years old, range: 27–75) with 122 lesions met the criteria. Of the 122 TNBC, there were 42 (34.4%) HER2-low, and 80 (65.6%) HER2-zero.

Fig. 1.

Flowchart of patients selection

The internal validation method employed in this study utilized a Monte Carlo cross-validation (repeated random sub-sampling) strategy designed to yield a robust and less biased estimate of model performance in a few-shot setting. The specific procedure was as follows: First, five independent random sub-sampling iterations were performed. In each iteration, 30 samples (15 HER2-low/15 HER2-zero) were randomly drawn without replacement from the entire cohort of to form that specific training set, while the remaining 92 samples served as the test set. Importantly, while sampling was performed without replacement within each individual iteration (ensuring 30 unique patients per training set), patients could be repeatedly selected into training sets across different iterations (i.e., sampling with replacement across the five iterations). Consequently, a given sample could appear in multiple training sets, and each sample appeared in the test sets of those specific iterations where it was not selected for training. Furthermore, every sample was guaranteed to appear in the test set.

In each iteration, a model was trained and used to generate patient-level prediction probabilities solely for the 92 samples in that specific test set. Upon completion of all five iterations, for each of the 122 samples in the entire cohort, all of its “out-of-sample” predicted probabilities (obtained exclusively from the iterations where it was in the test set) were arithmetically averaged. This produced a final, stable, and unbiased prediction probability per patient, integrating insights from multiple models. Finally, all comprehensive evaluation metrics—including the Area Under the Receiver Operating Characteristic Curve (AUC), calibration curves, and DeLong’s test—were calculated based on these aggregated out-of-sample probabilities against the true labels of the entire cohort (N = 122). Drawing from Monte Carlo and ensemble learning principles, this aggregation effectively reduces the variance inherent to evaluations based on a single small training split, providing a more reliable estimate of the framework’s diagnostic capability.

Medical imaging acquisition and processing

In this study, all participants underwent multiparametric MRI (mpMRI) examinations performed on a 3.0 Tesla GE SIGNA HDx scanner equipped with a dedicated 8-channel bilateral breast phased-array coil. The standardized imaging protocol consisted of the following sequences: T2-weighted imaging (T2WI) with short time inversion recovery (STIR); sagittal T2WI fast spin echo with fat suppression (fs-T2WI); T1-weighted imaged (T1WI) with dual-echo chemical-shift imaging; diffusion-weighted imaging (DWI) with single-shot echo planar imaging; dynamic contrast-enhanced MRI (DCE-MRI). DCE-MRI protocol comprised six phases: one pre-contrast phase and five post-contrast phases, with each phase acquired over a 90-second temporal resolution. Following the acquisition of pre-contrast images, gadopentetate dimeglumine (Magnevist; Bayer Schering Pharma, Berlin, Germany) was intravenously administered at a dose of 0.1 mmol/kg via a power injector at a flow rate of 2 mL/s, followed by a 20 mL saline flush at the same injection rate. Detailed scanning parameters are provided in Supplementary Materials S1.

Breast US examinations were performed with a LOGIQ E11 machine with a 6–15-MHz linear probe (GE Healthcare), an EPIQ 7 machine with a 5-12-MHz linear probe (Philips Medical Systems), or a HI VISION Avinus machine with a 5–13-MHz linear probe (Hitachi ALOKA). A breast sonographer with at least 5 years of experience in breast imaging performed bilateral whole-breast B-mode US. In the retrospective study, another breast sonographer and a radiologist reviewed the obtained images and reports of all cases, and determined the corresponding location on US and MRI of each lesion by consensus. The images of basic B-mode US were analyzed in the study.

Two breast radiologists with 10 and 13 years of experience in breast imaging delineated the regions of interest (ROIs) on both MRI and ultrasound (US) scans. During the annotation process, all readers were blinded to the patients’ clinical information. For both MRI (including DCE, DWI, and T2WI) and US images, radiologists reviewed the scans slice by slice using ITK-SNAP software (v3.6.0; http://www.itksnap.org) and manually delineated an initial rectangular bounding box encompassing the entire tumor. The geometric center of this initial box was then automatically calculated to serve as the lesion center. Based on this center coordinate, a standardized, fixed-size 2D bounding box was automatically generated to capture both intramammary and perimammary information. To ensure annotation reliability and eliminate inter-reader variability, all initial annotations were independently reviewed by both radiologists; any discrepancies in lesion localization were resolved through joint discussion to establish a final consensus. Subsequently, all standardized ROIs from both modalities were interpolated to a uniform resolution of 224 × 224 and subjected to Z-score normalization to standardize gray-scale intensities. Detailed image processing protocols are provided in Supplementary Material S2.

Pathological data annotation

One hundred and six patients received surgery after diagnosis and sixteen patients received neoadjuvant chemotherapy (NAT) before surgery, and pathological examinations were based on surgical samples and biopsy tissurs, respectively. HER2 status was determined by immunohistochemistry (IHC) or FISH according to the American Society of Clinical Oncology/College of American Pathologists (ASCO/CAP) 2018 guidelines [5]. HER2-low status was identified according to IHC score of 1 + or IHC score of 2 + with negative FISH. ER or PR was defined as positive with the nuclear staining of > 10% of the tumor cells. Ki-67 index was measured by IHC examination, and the cutoff value of 20% was used [21]. The expression of tumor-infiltrating lymphocytes (TILs) was evaluated on hematoxylin and eosin-stained (H&E-stained) slides, according to the recommendations by an International TILs Working Group 2014 [22]. The number of positive lymph nodes was recorded. Metastasis with the node ≤ 0.02 mm was defined as isolated tumor cells which was classified as negative lymph node. For cases that underwent NAT, the T stage and N stage were determined based on the evaluation of tumor size and axillary lymph nodes on US; for other cases, the T stage and N stage were determined by the examination of the surgical specimens in accordance with the AJCC 8th edition [23].

Deep learning network construction

In this study, a few-shot deep learning framework (FADNET) with two consecutive training phases was performed to build the models. We developed seven distinct models to validate the superiority of deep learning frameworks in predicting HER2-low status. The comparative analysis included: (1) a clinical variable-based model; (2) four single-modality deep learning models based on DCE, DWI, T2WI or US; (3) mpMRI deep learning model; (4) a multimodal imaging fusion model; (5) a hybrid model (CLI-FADNET) integrating clinical variable and multimodal imaging.

The deep learning model FADNET in this paper employed a contrastive network model. This contrastive network model effectively reduced the distance between samples of the same category while increasing the distance between samples of different categories. The network consisted of four stages: two training stages, an inference stage, and a multimodal fusion stage, as shown in Fig. 2.

Fig. 2.

The training and inference processes of FADNET, as well as the fusion process of CLI-FADNET. In the first stage, a contrast network with a 2D Residual Network (ResNet34) extracts features and utilizes contrastive loss to learn the similarity of features. In the second stage, the extracted features are concatenated to construct a feature dataset. In the inference stage, pre-trained weights and the soft k-means algorithm are employed to classify the samples. In the fusion stage, the CLI-FADNET model combines multimodal imaging data (DCE, DWI, STIR, US) with clinical variable. The outputs of the multimodal data are fused at the decision layer, and logistic regression is then used to predict the expression status of HER2-low

Training Stage 1: The contrastive network received inputs of different categories, with each sample being processed by different feature extractors (initialized with MedicalNet pre-trained weights and utilizing a 2D ResNet34 backbone in this implementation; detailed justifications for pre-training and model selection are provided in Supplementary Material S6). Crucially, the weights of the two feature extractors were shared to ensure consistent feature extraction process between them, thereby guaranteeing the consistency of the extracted features. During training, a contrastive loss function drove the optimization process, penalizing dissimilarity between samples of the same class and similarity between samples of different classes. This enabled the model to discover the inherent similarities within the data rather than relying solely on categorical labels, thereby improving generalization—particularly beneficial when training data was limited—and enhancing robustness to novel or unseen samples.

Training Stage 2: Weights learned in Stage 1 were frozen and repurposed to extract features from labeled training data across all categories. These features were then aggregated to build category-specific feature databases (e.g., two databases in this study). This database construction step consolidated discriminative feature representations for downstream inference.

Inference Stage: Test samples were first embedded using the frozen weights from Stage 1. The resulting feature vectors were compared against the precomputed category-specific databases via a soft k-means similarity metric, which quantified the affinity between test features and each database. Unlike standard k-means that makes rigid binary assignments, soft k-means assigns continuous, distance-based probability weights, providing richer discriminative scores that facilitate seamless decision-level fusion with the clinical variable via logistic regression. The category corresponding to the database with the highest similarity score was assigned as the predicted label for the test sample.

Feature Fusion Stage: To enhance performance and interpretability, the model adopted a multimodal approach. Individual deep learning models were first trained on each modality, with each model outputting per-category distance scores. These scores were fused in a post-processing step, leveraging complementary strengths across modalities. Additionally, clinical variable was concatenated with modality-specific distance scores to form an integrated feature vector. This vector was fed into a logistic regression classifier to produce the final diagnostic prediction. Detailed architectural parameters and implementation specifics were provided in the supplementary materials.

Model performance evaluation and interpretability

The performance of the model predicting HER2-low expression was evaluated using the following metrics: AUC (Area Under the Receiver Operating Characteristic Curve), accuracy, sensitivity, specificity, and F1-score. In addition, the calibration of the model was evaluated using calibration curves generated through bootstrap sampling. Besides, to explore the interpretability of the deep learning network, gradient-weighted class activation mapping (Grad-CAM [24]) and feature maps were utilized to visualize the critical regions and spatial distribution of decision-making information captured by the FADNET network during HER2-low prediction. Furthermore, to further validate the reliability of the proposed few-shot deep learning algorithm, the decision boundaries were visualized to demonstrate the effectiveness of the model’s classification decisions.

To address the inherent “black box” nature of machine learning systems, we implemented Shapley Additive explanations (SHAP values) grounded in cooperative game theory. This interpretability framework quantifies feature importance through marginal contribution allocation, thereby elucidating model decision-making patterns in HER2-low classification tasks.

Statistical analysis

All statistical analyses were performed using R (Windows 3.6.3 version) and Python (Windows 3.8.5 version). Normality of data distribution was assessed via the Kolmogorov-Smirnov test or Shapiro-Wilk test. Continuous variables with normal distribution are presented as mean ± standard deviation (SD), while non-normally distributed data are expressed as median (interquartile range, IQR). Categorical variables are reported as frequency (percentage). For continuous variables, Student’s t-test (parametric) or Mann-Whitney U test (non-parametric) was applied based on distribution normality. For categorical variables, the Chi-square test or Fisher’s exact test was used as appropriate. Variables showing significance (P < 0.05) in univariate analysis were subsequently included in multivariate logistic regression. To statistically compare the differences in AUCs (ΔAUC) between models, the DeLong test was employed.

Results

Clinical characteristics

Table 1 presents the baseline clinical and pathological characteristics of the 122 enrolled patients. Of 122 TNBC cases, 13 (11%) patients experienced postoperative recurrence, while 109 (89%) remained recurrence-free, with the median disease-free survival (DFS) 23 months (IQR 8–38) vs. 42 months (IQR 20–64) (P < 0.001). Of these variables, DFS, N stage, and metastasis lymph node (≥ 4 positive nodes) demonstrated statistical significance (P < 0.05). Patients undergoing recurrence after surgery, high percentage of TILs < 10%, Ki-67 < 20% and cT3 stage were seen more in HER2-low group, but there was no significant difference (P > 0.05). Multivariate logistic regression analyses were performed to further identify variables associated with HER2-low status. The results revealed that lymph node metastasis involving ≥ 4 nodes was independently predictor of HER2-low expression (adjusted OR = 0.24, 95% CI: 0.086 ~ 0.668; P = 0.006).

Table 1.

Comparison of clinical characteristics between HER2-low and HER2-zero groups in TNBC

	All Patients (N = 122)	HER2-low (N = 42)	HER2-zero (N = 80)	P-Value Univariate	P-Value Multivariate	OR (95% CI)
Age (years)	51.63 ± 10.22	53.19 ± 10.19	50.81 ± 10.20	0.223
DFS (month)	43.38 ± 16.73	38.98 ± 15.66	45.69 ± 16.90	0.040	0.113
Recurrence				0.346
Yes	13 (10.7%)	6 (14.3%)	7 (8.8%)
No	109 (89.3%)	36 (85.7%)	73 (91.2%)
TILs				0.489
≥ 10%	50 (41.0%)	19 (45.2%)	31 (38.75%)
< 10%	72 (59.0%)	23 (54.8%)	49 (61.25%)
Ki-67				0.185
≥ 20%	111 (91.0%)	36 (85.7%)	75 (93.75%)
< 20%	11 (9.0%)	6 (14.3%)	5 (6.25%)
cT stage				0.501
T-1	56 (45.9%)	19 (45.2%)	37 (46.25%)
T-2	61 (50.0%)	20 (47.6%)	41 (51.25%)
T-3	5 (4.1%)	3 (7.2%)	2 (2.5%)
N stage				0.003	0.990
N-0	71 (58.2%)	23 (54.7%)	48 (60.0%)
N-1	31 (25.4%)	7 (16.7%)	24 (30.0%)
N-2	14 (11.5%)	6 (14.3%)	8 (10.0%)
N-3	6 (4.9%)	6 (14.3%)	0 (0.0%)
Positive lymph node(s)				0.004	0.006	0.240 (0.086 ~ 0.668)
≥ 4	19 (15.6%)	12 (28.6%)	7 (8.8%)
< 4	103 (84.4%)	30 (71.4%)	73 (91.2%)

DFS: disease-free survival; HER2: human epidermal growth factor receptor 2; TILs: tumor-infiltrating lymphocytes; OR: odds ratio

Comparison of models based on different modalities

Predictive performance of HER2-low expression in TNBC based on different modalities is shown in Table 2. For comparison, logistic regression based on the clinical variable was used to build clinical model, which was based on the same dataset used for the deep learning network models. The clinical model achieved an AUC of 0.580 and an accuracy of 0.728. Among the models based on single MRI sequence, the DWI-based model achieved the highest AUC of 0.806 with sensitivity 0.815 and specificity 0.677, better than T2WI- and DCE-based model. AUC of 0.871 with sensitivity 0.815 and specificity 0.754 was obtained for mpMRI model, and the performance was significantly better than T2WI and DCE (both P < 0.001) but not than DWI (P > 0.05). The multImodal imaging fusion model integrating mpMRI and US modalities, achieved an AUC of 0.877 with sensitivity 0.889 and specificity 0.723, which outperformed mpMRI model (P < 0.001). When the clinical variable of positive lymph nodes ≥ 4 was integrated with the multimodal imaging fusion model, AUC of CLI-FADNET model was further increased to 0.884 with sensitivity 0.889 and specificity 0.723 (ΔAUC = 0.0068, P < 0.001; DeLong test). ROC curves of seven models based on different modalities are shown in Fig. 3. Figure 4 presents the calibration curve for CLI-FADNET, indicating concordance between the predicted probabilities and observed event frequencies.

Table 2.

Predictive performance of HER2-low expression in TNBC based on different modalities across five iterations

Modality	Accuracy	AUC(95%CI)	F1 Score	Sensitivity	Specificity
T2WI	0.804	0.783 (0.671, 0.885)	0.640	0.593	0.892
DWI	0.717	0.806 (0.689, 0.908)	0.629	0.815	0.677
DCE	0.739	0.764 (0.648, 0.875)	0.647	0.815	0.708
mpMRI	0.772	0.871 (0.774, 0.947)	0.678	0.815	0.754
Clinical	0.728	0.580 (0.501, 0.670)	0.324	0.222	0.938
US	0.750	0.797 (0.681, 0.894)	0.646	0.778	0.739
FADNET-multimodal imaging	0.772	0.877 (0.782, 0.953)	0.696	0.889	0.723
CLI-FADNET	0.772	0.884 (0.799, 0.953)	0.696	0.889	0.723

Fig. 3.

Prediction performance of different models

Fig. 4.

Calibration curve for CLI-FADNET

Interpretability of deep learning models

To interpret decision-making process of CLI-FADNET model in HER2-low prediction, gradient-weighted class activation mapping (Grad-CAM) was employed to generate heatmaps that visualize critical regions and weight distributions within the model. As shown in Fig. 5, the pixel weight distribution is visualized using distinct colors, highlighting the regions of interest prioritized by the FADNET model in predicting HER2-low expression.

Fig. 5.

Representative Grad-CAM heatmaps and original images. From top to bottom: The first row displays the original images, the second row presents the region of interest (ROI)-cropped images, and the third row illustrates the generated gradient-weighted class activation mapping (Grad-CAM) heatmaps. These visualizations demonstrate Grad-CAM heatmaps across different imaging modalities. These visualizations demonstrate Grad-CAM heatmaps across DCE、DWI、T2WI、US modalities for representative HER2-zero cases. Regions highlighted in red indicate activations strongly associated with HER2-low predictions, with darker hues denoting higher relevance to the classification outcome

Rather than merely focusing on the general tumor bulk, the Grad-CAM heatmaps highlight biologically plausible regions crucial for clinical diagnosis. Specifically, on US and T2WI, the highest activations frequently localize to the tumor margins and the tumor-stroma interface—regions deeply associated with tumor invasiveness and the microenvironment. On DCE, the network’s attention often aligns with areas of heterogeneous enhancement and peripheral neoangiogenesis. This demonstrates that FADNET’s decision-making aligns strongly with radiological intuition, extracting features from areas known to harbor aggressive biological behaviors and complex microenvironments that are highly indicative of the HER2-low subtype, rather than background imaging artifacts.

To further elucidate the network’s feature classification mechanism, the decision boundaries of the model are illustrated in Fig. 6. The well-defined separation between feature clusters demonstrates the network’s ability to effectively learn and classify discriminative characteristics.

Fig. 6.

Visualization of the network’s decision boundaries for classifying HER2-low versus HER2-zero cases. Each point represents an individual patient case. The well-defined separation between features observed in the decision boundaries demonstrates the network’s capability to effectively learn and classify features

Furthermore, to complement the interpretability analysis, SHAP (Shapley Additive explanations) values were employed to quantify the contributions of clinical and imaging features to HER2-low prediction. As shown in Fig. 7, features positioned higher on the vertical axis indicate a greater overall impact on the model’s predictions. Feature values were represented by color, with red data points denoting higher values and blue data points indicating lower values. Positive SHAP values suggested a promotive effect on the classification of HER2-low positivity. The results demonstrated that the output probability from DCE exerted the most substantial influence on the model’s predictions. Although the AUC value of the DCE modality was lower than that of other single modalities, it exhibited the highest feature importance in the multimodal fusion model. To further investigate this, we performed ablation experiments on modality fusion (results presented in Table 3). The findings showed that models without DCE modality fusion performed worse than those with DCE integration. This indicates that the information provided by DCE complementing the information from other modalities. After fusion, DCE features became critical discriminative factors. This complementary effect highlights the advantage of the multimodal approach in integrating complementary information, and its performance outperformed that of any single modality alone. This analytical approach enhanced the understanding of model behavior and identified critical features.

Fig. 7.

SHAP analysis plot for the CLI-FADNET hybrid model. The analysis is used to visualize the importance of multimodal imaging features (DCE, DWI, T2WI, US) and the selected clinical variable for predicting HER2-low status. It demonstrates the positive and negative relationships between these predictor factors and the target variable, highlighting their specific contributions to the final prediction

Table 3.

Ablation experiments on modality fusion

Modality				Metrics
DCE	DWI	T2WI	US	Accuracy	AUC (95%CI)	F1 Score	Sensitivity	Specificity
√	√	√	×	0.772	0.871 (0.774, 0.947)	0.678	0.815	0.754
√	×	√	√	0.728	0.864 (0.767, 0.949)	0.638	0.815	0.692
√	√	×	√	0.772	0.867 (0.770, 0.946)	0.677	0.815	0.754
×	√	√	√	0.728	0.854 (0.746, 0.948)	0.648	0.852	0.677

Discussion

Triple-negative breast cancer, though clinically aggressive, remains a rare subtype, restricting access to large-scale clinical datasets in practice. This scarcity poses significant challenges for developing robust predictive models, particularly in emerging areas such as HER2-low status classification. To address this limitation, we proposed FADNET, a few-shot deep learning framework designed to maximize performance in low-data environments. By integrating multimodal imaging features from MRI and US, FADNET demonstrated robust predictive capability for HER2-low status in TNBC patients, even when trained on a cohort of only 122 cases. Notably, with the fusion of imaging-derived features with key clinical variable, the model (CLI-FADNET) further enhanced model performance, underscoring the potential synergistic value of combining radiomic and clinical data. These findings highlight the potential of few-shot learning to bridge gaps in predictive modeling for rare cancer subtypes, where traditional data-driven approaches are hindered by limited sample sizes.

TNBC patients traditionally rely on aggressive therapies like NAT for early stages and postoperative adjuvant therapy for advanced stages. However, the redefinition of HER2-low status offers new hope, as TNBC patients with HER2-low expression may benefit from emerging anti-HER2 antibody-drug conjugates like t-DXd, which have shown survival improvements in metastatic HER2-low cases [25]. While approaches leveraging MRI have gained traction, achieving moderate predictive performance (AUC ~ 0.80) [9–11], our CLI-FADNET framework introduces several key methodological advancements that explain its enhanced performance. First, prior studies such as Ramtohul et al. [9] utilized handcrafted radiomic features combined with traditional machine learning algorithms (e.g., Gradient Boosted Decision Trees). While effective, these methods may not fully capture the complex, high-dimensional hierarchical semantic representations of tumors that deep learning networks can extract. Second, studies like Guo et al. [10] and our previous work [11] relied exclusively on single-modality MRI. By integrating breast US with mpMRI, our multimodal approach captures complementary biological information—combining the functional hemodynamic data and cellularity from MRI with the macroscopic morphological and tumor-stroma interface details provided by US. Finally, while recent large-scale multimodal efforts [16, 26] have shown encouraging results, reproducing such massive data collection in TNBC remains challenging due to the subtype’s relative rarity. Conventional deep learning models are notoriously data-hungry and prone to overfitting in such low-data regimes. FADNET addresses this critical bottleneck through its few-shot contrastive learning approach. By optimizing a relative distance metric rather than learning a rigid decision boundary, FADNET efficiently leverages limited data, demonstrating superior robustness and data efficiency compared to standard transfer learning approaches.

In the present study, we developed a few-shot deep learning network (FADNET) model based on imaging modalities. The models utilizing single modality and mpMRI demonstrated good predictive performance. Moreover, the multimodal imaging fusion model, which integrated features from various imaging sources, outperformed other models based on single-modality and mpMRI. Among the clinical variables, metastasis axillary lymph nodes ≥ 4 demonstrated to be the independently predictor of HER2-low status in TNBC, showing greater significance than N stage alone. After integrating the key clinical variable, CLI-FADNET model exhibited exceptional predictive performance for HER2-low status, achieving an AUC of 0.884, with a sensitivity of 0.889 and specificity of 0.723, which surpasses the previously reported multimodel deep learning model [26].

Although the AUC improvement of CLI-FADNET over the pure multimodal imaging model is numerically marginal (ΔAUC = 0.0068), its clinical relevance is substantial. In practice, integrating a robust clinical variable acts as an independent biological anchor. The inclusion of this clinical variable primarily improves model calibration and diagnostic confidence, which helps stabilize predictions in edge cases where imaging features alone are insufficient.

Furthermore, our SHAP analysis revealed a compelling phenomenon regarding the contribution of DCE-MRI. While the standalone DCE model exhibited the lowest predictive performance (AUC = 0.764), DCE-derived features emerged as the most critical contributor to the multimodal fusion model’s output. This result can be explained by the complementary biological information captured by different modalities. Biologically, DWI primarily reflects tumor cellularity and microstructural density, while US captures macroscopic morphology and the tumor-stroma interface. In contrast, DCE uniquely captures microvascular permeability and neoangiogenesis. Because TNBC is universally aggressive, both HER2-low and HER2-zero tumors generally exhibit intense and rapid baseline enhancement, making DCE a weak independent discriminator. However, when fused, the multimodal network contextualizes this data: once cellular density (DWI) and morphology (US) are established, the subtle, spatially heterogeneous variations in perfusion and hemodynamics provided uniquely by DCE become highly discriminative. As demonstrated by our ablation study (Table 3), the removal of DCE deprives the model of this critical hemodynamic context, confirming the complementary value of DCE when integrated with other imaging modalities.

Our few-shot learning framework offers distinct advantages over conventional meta-learning approaches like MAML [27], BLC-MAML [28], and MeTAL [29]. Traditional meta-learning necessitates creating a multitude of episodic tasks to mimic few-shot learning scenarios. This process not only demands significant computational resources but can also introduce biases from artificial task design. By contrast, our framework employs a contrastive learning strategy that directly enhances the feature space geometry through multimodal alignment. This approach bypasses complex task sampling, lowering computational costs and speeding up training. Moreover, the weight-sharing mechanism ensures uniform feature representation across different modalities, effectively resolving the modality heterogeneity issue common in meta-learning. Given the challenges of obtaining high-quality annotated medical data—particularly for rare cancer subtypes—FADNET proves to be an exceptionally promising solution [30].

Furthermore, our FADNET framework demonstrates superior data efficiency compared to conventional deep learning transfer learning. Conventional models typically learn rigid decision boundaries, which notoriously overfit when trained on limited medical imaging datasets with subtle inter-class differences. FADNET overcomes this by employing a contrastive learning strategy. Instead of point-by-point classification, contrastive learning constructs pairs of samples, implicitly expanding the supervisory signals from N samples to pairwise comparisons. Additionally, by optimizing a distance metric—pulling features of the same class together while pushing different classes apart—and utilizing soft k-means to match test samples against learned category databases, the model mimics the robustness of prototypical networks. This metric-based approach significantly mitigates the risk of memorizing specific training instances, rendering FADNET fundamentally more suitable for few-shot scenarios than standard linear classifiers.

The widespread application of CLI-FADNET faces several challenges, and the following limitations in the study need to be acknowledged. First, this study relies on a relatively small single-center dataset (n = 122). A model intended for clinical deployment would ideally be trained on larger, multi-center datasets using a standard training-test split (such as 8:2 or 4:1 with external validation); however, the current design was intentionally selected to evaluate the few-shot learning approach under challenging conditions, rather than to achieve the highest possible performance AI models trained on such limited data are notoriously prone to performance decline when applied to external cohorts. This degradation is primarily due to domain shifts caused by variations in imaging protocols (e.g., differences in contrast injection timings for DCE-MRI), scanner variability (e.g., variations in magnetic field strengths across MRI machines or different ultrasound probe frequencies from various manufacturers), and population heterogeneity [31]. Although our rigorous Monte Carlo cross-validation strategy acts as an internal robustness analysis—demonstrating stable performance across multiple internal data subsets—the lack of true external validation significantly limits the comprehensive assessment of the model’s generalizability. Therefore, to confirm the model’s absolute stability and clinical reliability, future research must conduct large-scale, multi-center external validation studies. Second, the model’s performance might be affected by the modality fusion method, indicating potential for further enhancement. Third, although we utilized Grad-CAM and SHAP to provide transparent diagnostic assistance, fully overcoming the inherent black-box nature of deep learning remains a challenge. Currently, these tools allow radiologists to use SHAP to determine the most predictive modality and cross-reference Grad-CAM heatmaps to verify that the model focuses on biologically plausible regions—such as infiltrative margins or heterogeneous enhancement—rather than imaging artifacts. While this level of interpretability bridges the gap between algorithms and clinical intuition to some extent, they are post-hoc explanation methods. Future studies should explore inherently interpretable network architectures to further boost diagnostic confidence.

Conclusion

This study presents FADNET, a few-shot deep learning network designed to predict HER2-low status in TNBC using limited training data. By incorporating pre-treatment multimodal imaging data (breast MRI and US), this method provides a new way to identify HER2-low patients who might benefit from emerging targeted therapies like antibody-drug conjugates. The model shows promise in differentiating HER2-low from HER2-zero cases in TNBC, which is essential for guiding precision oncology strategies. Additionally, while HER2-low patients in this study had slightly worse DFS, FADNET’s ability to identify this subgroup could help inform personalized treatment adjustments, such as optimizing drug regimens. These results highlight the potential of AI-driven multimodal analysis in advancing biomarker-driven care for TNBC and call for further validation in larger cohorts.

Supplementary Information

Below is the link to the electronic supplementary material.

Abbreviations

TNBC

Triple-negative breast cancer

US

Ultrasound

BC

Breast cancer

ER

Estrogen receptor

PR

Progesterone receptor

HER2

Human epidermal growth factor receptor 2

FISH

Fluorescence in situ hybridization

AI

Artificial intelligence

BI-RADS

Breast Imaging Reports and Diagnosis System

mpMRI

multiparametric MRI

T2WI

T2-weighted imaging

STIR

Short time inversion recovery

DCE-MRI

Dynamic contrast-enhanced MRI

ROI

Regions of interest

NAT

Neoadjuvant chemotherapy

TILs

Tumor-infiltrating lymphocytes

AUC

Area Under the Receiver Operating Characteristic Curve

Grad-CAM

Gradient-weighted class activation mapping

DFS

Disease-free survival

FADNET

Few-shot Adaptive Deep Network

CLI-FADNET

Clinical-integrated FADNET

Author contributions

Junfei Fang, Tianle Hong and Yulan Tong contributed equally to this manuscript as co-first authors. Study conception and design: Jiejie Zhou, Yezhi Lin; Model design and implementation: Junfei Fang, Cunke Miao; Data collection: Yulan Tong, Sijia Wen, Zedong Ren; Data analysis and statistical evaluation: Junfei Fang, Tianle Hong; Manuscript writing: Junfei Fang, Jiejie Zhou; Critical revision and supervision: Jiejie Zhou, Yezhi Lin. All authors read and approved the final version of the manuscript.

Funding

This study was supported by National Natural Science Foundation of China (No.82471955 and No.82572172), Lingyan Project of Zhejiang Province (No.2025C02142), the Mindray Joint Fund of Zhejiang Provincial Natural Science Foundation of China (No.LMRY26H180018) and the Scientific Research Task of Department of Education of Zhejiang Province (No.Y202352073).

Data availability

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request. To ensure reproducibility and facilitate future research, the source code for the FADNET framework will be made publicly available on GitHub upon the acceptance of this manuscript.

Declarations

Ethics approval and consent to participate

This research was conducted in accordance with the Declaration of Helsinki and was approved by the Ethics Committee of the First Affiliated Hospital of Wenzhou Medical University (approval number: YS2023-433). We confirm that all methods and experiments were performed in accordance with relevant guidelines and regulations. The requirement for written informed consent was waived by the Institutional Review Board due to the retrospective nature of the study.

Consent for publication

The requirement for informed consent was waived by the Ethics Committee due to the retrospective nature of the study.

Competing interests

The authors declare no competing interests.