Abstract
Background
Breast cancer (BC) is the most prevalent malignancy among women worldwide. The development of accurate and noninvasive diagnostic methods is essential to reduce unnecessary biopsies and surgeries. This study aims to develop a dual-modal deep learning (DL) radiomics model based on B-mode ultrasound (BUS) and contrast-enhanced ultrasound (CEUS) images. The model is designed to assist radiologists in accurately differentiating benign from malignant breast lesions.
Methods
This retrospective multicenter study included 427 female patients with breast lesions from four hospitals. Traditional radiomics models were constructed using logistic regression (LR). DL radiomics models were built on a VGG-16 network pretrained on ImageNet. An integrated model was developed through early feature fusion of BUS and CEUS features. Model interpretability was assessed with Shapley Additive exPlanations (SHAP) and heatmaps generated by gradient-weighted class activation mapping (Grad-CAM). In a two-round reader study, the integrated model provided radiologists with artificial intelligence (AI) scores and heatmaps to support diagnosis. Model performance was evaluated using the area under the curve (AUC) and decision curve analysis (DCA).
Results
In the testing cohort, the integrated model achieved the highest performance, with an AUC of 0.825 [95% confidence interval (CI): 0.744–0.907]. SHAP analysis revealed that, compared with BUS features, CEUS features had a greater impact on the model’s diagnostic performance. In the first round, the integrated model outperformed all the radiologists (model AUC: 0.825 vs. radiologists’ AUCs: 0.701–0.824). In the second round, radiologists assisted by the integrated model demonstrated improved performance. Their AUCs ranged from 0.748 to 0.869, with ΔAUCs ranging from +0.030 to +0.058. Four radiologists outperformed the model itself.
Conclusions
The integrated model provides an effective and noninvasive approach for predicting the benignity or malignancy of breast lesions. It has a strong potential to serve as a valuable clinical tool for improving radiologists’ diagnostic performance.
Keywords: Radiomics, deep learning (DL), breast cancers (BCs), contrast-enhanced ultrasound (CEUS), dual-modal
Highlight box.
Key findings
• In the first-round reader study, the integrated model demonstrated higher diagnostic accuracy than radiologists (area under the curve =0.825). In the second-round reader study, artificial intelligence (AI) assistance from the model significantly improved all radiologists’ accuracy and sensitivity.
What is known and what is new?
• Prior studies have mostly focused on single-modality radiomics models, often lacking external validation and interpretability.
• This study introduces a dual-modal deep learning radiomics approach, which has been validated across multiple centers. It also demonstrates that AI-generated scores and heatmaps can meaningfully assist radiologists in clinical diagnosis.
What is the implication, and what should change now?
• The model enhances radiologists’ diagnostic performance by improving both accuracy and efficiency in breast disease assessment.
Introduction
Breast cancer (BC) is the most prevalent malignancy among women worldwide, accounting for approximately 2.26 million new cases and 685,000 deaths annually (1,2). Early diagnosis and timely intervention are critical for improving patient prognosis and reducing mortality associated with BC (3). However, the lack of discernible clinical symptoms during the early stages often impedes the early detection of potential abnormalities through self-examination. This delay results in missed opportunities for timely intervention and treatment. The current diagnostic gold standard, biopsy-based histopathological examination, remains indispensable for confirming BC diagnosis and guiding treatment decisions (3). Biopsy is an invasive procedure and may occasionally be associated with complications such as hematoma, infection, or insufficient sampling. However, it should not be regarded as harmful. Instead, noninvasive and accurate imaging-based tools are highly valuable as supportive modalities. In this context, developing artificial intelligence (AI) systems to assist radiologists in identifying patients most likely to benefit from biopsy is of great importance. Such tools can help facilitate timely and precise diagnosis (4).
B-mode ultrasound (BUS) is widely recognized as an essential modality for BC screening owing to its cost-effectiveness, absence of radiation exposure, and ability to provide detailed morphological visualization of lesions. On the other hand, contrast-enhanced ultrasound (CEUS) offers significant advantages. It enables precise assessment of tumor blood flow, perfusion, and microvascular architecture. These capabilities substantially improve the diagnostic accuracy of BC (5). Despite these benefits, the interpretation of ultrasound images relies heavily on the subjective judgment of radiologists. This reliance can lead to considerable variability in diagnostic outcomes for the same image among different radiologists. Moreover, less experienced radiologists are more likely to produce ambiguous interpretations, which can increase false-positive rates and lead to unnecessary biopsies. In this context, AI-based radiomics models are not intended to replace biopsy. Rather, they function as supportive tools that reduce diagnostic uncertainty and support appropriate biopsy determination (6,7). Traditional radiomics extracts predefined, high-dimensional features from medical images. In contrast, deep learning (DL) autonomously identifies discriminative patterns directly from raw imaging data (8-10). Both approaches have shown promise in advancing breast disease diagnosis and characterization (11). Several studies using single-modality ultrasound have demonstrated effectiveness in distinguishing benign and malignant breast lesions (12-14). Despite this progress, several challenges remain. Most studies lack external validation, limiting assessment of model generalizability. Although diagnostic accuracy is high, model interpretability remains insufficient. The practical utility of radiomics models remains largely untested in real-world clinical practice.
In light of these insights, this study pursued three primary objectives: (I) to develop a multicenter, dual-modal ultrasound-based integrated model leveraging BUS and CEUS imaging for the accurate differentiation of benign and malignant breast lesions; (II) to investigate the interpretability of the radiomics model, thereby elucidating the decision-making logic underlying the dual-modal approach in clinical applications; and (III) to assess the model’s utility in enhancing radiologists’ diagnostic performance for distinguishing benign and malignant breast lesions. We present this article in accordance with the TRIPOD reporting checklist (available at https://gs.amegroups.com/article/view/10.21037/gs-2025-295/rc).
Methods
Patients
This study was a retrospective, multicenter analysis involving 427 female patients with breast lesions from four hospitals in different regions of China between January 2018 and May 2024. All eligible patients were consecutively enrolled according to predefined criteria. No adjustment methods, such as propensity score matching, were applied. A total of 322 patients were recruited from the People’s Hospital of Guangxi Zhuang Autonomous Region, while 31 cases were collected from the Guilin Municipal Hospital of Traditional Chinese Medicine. In addition, 28 patients were included from the Fangchenggang First People’s Hospital, and 46 patients were enrolled from the Minda Hospital of Hubei Minzu University. The study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments. The study was approved by the ethics committee of the People’s Hospital of Guangxi Zhuang Autonomous Region, China (No. KY-LW-2020-24), and individual consent for this retrospective analysis was waived.
The inclusion criteria were as follows: (I) breast lesions confirmed by pathological examination; (II) patients who underwent standard and complete BUS and CEUS examinations; and (III) imaging that met the required quality standards. The exclusion criteria included the following: (I) incomplete imaging or clinical data; (II) prior treatment with chemotherapy, radiotherapy, or targeted therapy before ultrasound examination; and (III) pregnancy or lactation at the time of imaging.
Only patients with pathologically confirmed benign or malignant breast lesions were included. Cases with normal breast findings were not used for AI model annotation or testing. In this multicenter study, data from the People’s Hospital of Guangxi Zhuang Autonomous Region, which included the largest patient cohort (n=322), were used as the training set. Data from three additional hospitals (n=105) were combined to form the independent testing set. This center-split design was adopted to evaluate model generalization across institutions and minimize center-specific bias. Radiomics features were extracted using a uniform preprocessing pipeline for all images, ensuring consistency and comparability between training and testing sets (15-17). The detailed process of patient selection is presented in Figure 1. Baseline characteristics were extracted from hospital databases for analysis.
Figure 1.
The patient recruitment workflow employed in the study. Hospital 1, People’s Hospital of Guangxi Zhuang Autonomous Region; other three hospitals, Guilin Municipal Hospital of Traditional Chinese Medicine, Fangchenggang First People’s Hospital, Minda Hospital of Hubei Minzu University. BUS, B-mode ultrasound; CEUS, contrast-enhanced ultrasound.
Acquisition of BUS and CEUS images
For ultrasound imaging, several types of devices were employed, such as the Aplio 500/i800/i900, Mindray R7/R9, and GE LOGIQ E9/E10. All were equipped with high-frequency linear array probes with a starting frequency of at least 12 MHz, consistent with the recommended standards for breast imaging. All ultrasound images were acquired following a rigorously standardized protocol with strict quality control to ensure consistency across devices and operators (18). All procedures were performed by radiologists with over five years of experience. In the BUS assessment, any suspicious lesions were scanned in multiple planes. The section displaying the maximum dimension of the lesion was chosen for subsequent CEUS evaluation. In CEUS mode, 4.8 mL of the contrast agent (SonoVue) was administered via the median cubital vein, immediately followed by a 5 mL saline flush. The timer and storage functions were initiated at the same time, and dynamic imaging was continuously acquired for no less than 2 minutes. All the ultrasound images were stored in digital imaging and communications in medicine (DICOM) format on the workstation for subsequent analysis.
Region of interest extraction and processing
The largest plane of the tumor was selected for BUS and CEUS imaging. During CEUS analysis, the time-intensity curve (TIC) was generated from the section of the tumor with the greatest dimension to locate the phase of maximal enhancement. The frame at this peak point was extracted to represent the lesion. This procedure is grounded in the clear disparity in peak enhancement patterns observed between benign and malignant tumors (19). All images were exported from the workstation and processed using ITK-SNAP software. The regions of interest (ROIs) were independently determined by two radiologists, each having over a decade of experience in breast ultrasound. To ensure accuracy and consistency, the ROIs were further examined and modified by a third radiologist with two decades of diagnostic expertise. ROIs were delineated along the tumor margins to encompass the entire lesion area.
Radiomics feature extraction and model construction
For traditional radiomics, handcrafted feature extraction from ROIs was conducted using Pyradiomics (http://pyradiomics.readthedocs.io). The extracted features were categorized into three primary groups: (I) texture; (II) intensity; and (III) geometry. In total, 107 radiomics features, including 14 shape features, 18 first-order features, and 75 texture features, were obtained from each BUS and CEUS image.
To ensure the reproducibility of ROI delineation and minimize subjective variability, intraclass correlation coefficients (ICCs) were calculated. Features with ICC values ≥0.80, both within and between observers, were retained. The Mann-Whitney U test was employed for initial feature screening, and variables with P<0.05 were preserved. Pearson correlation coefficients were then calculated to detect redundant features; if the correlation coefficient was ≥0.9, the feature demonstrating the largest mean absolute correlation was discarded.
For feature selection, least absolute shrinkage and selection operator (LASSO) regression was applied, followed by 10-fold cross-validation to identify the most representative features. The selected features were then input into a logistic regression (LR) machine learning model for radiomics model construction. To finalize the radiomics signature, five-fold cross-validation was employed.
Three traditional radiomics models were trained to evaluate predictive performance: (I) a model based on BUS images; (II) a model based on CEUS images; and (III) a dual-modal model combining BUS and CEUS images. The predictive performance of these models was assessed via the receiver operating characteristic (ROC) curve, area under the curve (AUC), decision curve analysis (DCA), accuracy, sensitivity, and specificity.
DL model construction and feature extraction
To construct the DL model, we cropped both the original and expanded ROIs, with the input image dimensions standardized to 224×224 pixels. A widely used DL architecture, VGG16, pretrained on ImageNet, was employed and subsequently fine-tuned on the training cohort via transfer learning. Deep features were derived from the block1_pool layer (fifth-to-last pooling) of the trained VGG16 model, as this layer retains detailed spatial information. The extraction resulted in 100,352 features. This intermediate layer was selected, given its superior effectiveness for our dataset.
To improve model generalizability and mitigate the risk of overfitting, principal component analysis (PCA) was applied to reduce the dimensionality of the extracted features to 32 components. These features were then standardized via z-score normalization. Spearman’s correlation analysis and LASSO regression were subsequently performed to select the most representative features. LR classifiers were trained using these selected features to construct the DL models. Three DL radiomics models were trained to evaluate predictive performance: (I) a model based on BUS images; (II) a model based on CEUS images; and (III) a dual-modal model combining BUS and CEUS images. The complete workflows for constructing the radiomics and DL models are illustrated in Figure 2.
Figure 2.
The overarching methodology employed in this research project. (A) Acquisition of BUS and CEUS images. (B) Feature extraction workflow, including the extraction of traditional radiomics features and deep learning radiomics features. (C) Feature selection process. (D) Model construction and identification of the best-performing model. AUC, area under the curve; BUS, B-mode ultrasound; CEUS, contrast-enhanced ultrasound; DCA, decision curve analysis; LASSO, least absolute shrinkage and selection operator; MSE, mean squared error; ROI, region of interest.
Integrated model construction
Feature-level fusion, also referred to as early fusion, entails combining features from multiple modalities into a single, unified feature vector to construct integrated models. In this study, the optimal radiomics and DL features extracted from the ROIs in both BUS and CEUS images were concatenated into a comprehensive feature set. These features were standardized via z-score normalization. Spearman’s correlation analysis and LASSO regression were subsequently employed to identify the most representative features. Finally, an LR classifier was trained using the selected features to develop the integrated model.
Model explanation and visualization
Following the identification of the optimal traditional radiomics, DL, and fusion models, Shapley Additive exPlanations (SHAP) analysis was employed to enhance the interpretability of the model by assessing feature importance. The features were ranked in descending order of their SHAP values to identify the most critical predictive factors within the testing cohort.
For each case, the AI model outputs a probability for both benign and malignant classes, which always sum to 1. The final classification is assigned to the category with the higher score. No additional threshold adjustment is applied beyond this 0.5 probability cutoff for classification.
Gradient-weighted class activation mapping (Grad-CAM) was applied to visualize the network’s decision-making process. Grad-CAM generates a coarse localization map that highlights the key regions contributing to the classification outcome (20). This visualization technique leverages the feature maps produced by the final convolutional layer to identify the areas of the input image most relevant to the model’s predictions.
Two-round reader study
The clinical value of the integrated model in radiologist assistance was examined through a two-round reader study (Figure 3). The study involved six radiologists, comprising two junior radiologists (<5 years of experience), two intermediate radiologists (5–10 years), and two senior radiologists (>10 years). A total of 105 breast tumors from the testing cohort were presented in a randomized sequence. To ensure unbiased evaluation, the radiologists remained unaware of the original diagnostic reports, their colleagues’ interpretations, and the definitive pathological outcomes throughout the study. During image interpretation, radiologists reviewed the AI-generated scores and Grad-CAM heatmaps alongside the ultrasound images. Final diagnostic decisions were made by the radiologists themselves, integrating AI information with their clinical expertise. The AI system served only as an auxiliary tool and did not replace human judgment.
Figure 3.
Workflow of the two-round reader study. (A) In the first round, radiologists independently interpreted images based on their clinical experience. In the second round, radiologists incorporated AI scores and heatmaps to refine their diagnostic decisions. (B) Typical cases demonstrate the effectiveness of the integrated model in assisting radiologists to achieve accurate diagnoses. The top panel illustrates a malignant lesion initially misclassified as benign by most radiologists during the first round. With the integration of AI scores and heatmaps, the diagnostic accuracy improved to 100%. The bottom panel depicts a benign lesion misclassified as malignant in the first round, where AI assistance similarly enhanced the diagnostic accuracy to 100%. AI, artificial intelligence; BUS, B-mode ultrasound; CEUS, contrast-enhanced ultrasound; ROI, region of interest.
Statistical analysis
Analytical procedures were executed on the OnekeyAI platform (v4.9.1) with Python 3.7.12, and statistical computations were carried out using Statsmodels (v0.13.2). Radiomics feature extraction was carried out with PyRadiomics (version 3.0.1). The implementation of the LR machine learning models was facilitated by Scikit-learn (version 1.0.2). DL frameworks were developed using PyTorch (version 1.11.0), with performance optimization achieved via CUDA (version 11.3.1) and cuDNN (version 8.2.1). For continuous variables, either the Mann-Whitney U test or independent t-test was applied as appropriate, while categorical variables were analyzed using Chi-squared or Fisher’s exact tests. Statistical significance was defined as a two-sided P value below 0.05.
Results
Clinical features
A total of 427 female patients with breast lesions were consecutively recruited from four medical centers. Pathological examination identified 240 malignant cases (56.2%) and 187 benign cases (43.8%). The clinical characteristics of these patients are summarized in Table 1.
Table 1. Characteristics of patients in the training and testing cohort.
| Characteristics | Training cohort (n=322) | Testing cohort (n=105) | |||||
|---|---|---|---|---|---|---|---|
| Benign | Malignancy | P | Benign | Malignancy | P | ||
| Age (years) | 43.48±11.74 | 51.67±11.40 | <0.001 | 43.15±11.57 | 48.85±9.53 | 0.003 | |
| Tumor size (mm) | 18.84±12.45 | 26.74±12.69 | <0.001 | 14.12±10.80 | 17.02±12.13 | 0.03 | |
| BUS characteristics | |||||||
| Location | 0.07 | 0.88 | |||||
| UOQ | 52 (42.62) | 115 (57.50) | 34 (52.31) | 19 (47.50) | |||
| LOQ | 28 (22.95) | 30 (15.00) | 11 (16.92) | 6 (15.00) | |||
| LIQ | 12 (9.84) | 14 (7.00) | 8 (12.31) | 7 (17.50) | |||
| UIQ | 30 (24.59) | 41 (20.50) | 12 (18.46) | 8 (20.00) | |||
| BI-RADS category | <0.001 | <0.001 | |||||
| III | 31 (25.41) | 3 (1.50) | 12 (18.46) | 1 (2.50) | |||
| Iva | 49 (40.16) | 4 (2.00) | 45 (69.23) | 15 (37.50) | |||
| IVb | 27 (22.13) | 20 (10.00) | 7 (10.77) | 10 (25.00) | |||
| IVc | 14 (11.48) | 60 (30.00) | 1 (1.54) | 9 (22.50) | |||
| V | 1 (0.82) | 113 (56.50) | Null | 5 (12.50) | |||
| Margin | <0.001 | <0.001 | |||||
| Circumscribed | 90 (73.77) | 12 (6.00) | 53 (81.54) | 5 (12.50) | |||
| Indistinct | 32 (26.23) | 188 (94.00) | 12 (18.46) | 35 (87.50) | |||
| Shape | <0.001 | <0.001 | |||||
| Regular | 98 (80.33) | 23 (11.50) | 51 (78.46) | 9 (22.50) | |||
| Irregular | 24 (19.67) | 177 (88.50) | 14 (21.54) | 31 (77.50) | |||
| Posterior features | <0.001 | 0.003 | |||||
| No attenuation | 108 (88.52) | 133 (66.50) | 60 (92.31) | 27 (67.50) | |||
| Shadowing | 14 (11.48) | 67 (33.50) | 5 (7.69) | 13 (32.50) | |||
| Microcalcification | <0.001 | <0.001 | |||||
| Absent | 107 (87.70) | 138 (69.00) | 62 (95.38) | 26 (65.00) | |||
| Present | 15 (12.30) | 62 (31.00) | 3 (4.62) | 14 (35.00) | |||
| Orientation | <0.001 | <0.001 | |||||
| Parallel | 119 (97.54) | 170 (85.00) | 64 (98.46) | 27 (67.50) | |||
| Not parallel | 3 (2.46) | 30 (15.00) | 1 (1.54) | 13 (32.50) | |||
| Enhancement intensity | 0.002 | 0.008 | |||||
| Iso- or hypo-enhancement | 58 (47.54) | 59 (29.50) | 38 (58.46) | 12 (30.00) | |||
| Hyper-enhancement | 64 (52.46) | 141 (70.50) | 27 (41.54) | 28 (70.00) | |||
| Internal homogeneity | 0.01 | 0.03 | |||||
| Homogenous | 67 (54.92) | 80 (40.00) | 40 (61.54) | 15 (37.50) | |||
| Heterogeneous | 55 (45.08) | 120 (60.00) | 25 (38.46) | 25 (62.50) | |||
| Enhancement edge | <0.001 | 0.003 | |||||
| Clear | 72 (59.02) | 69 (34.50) | 42 (64.62) | 13 (32.50) | |||
| Blurred | 50 (40.98) | 131 (65.50) | 23 (35.38) | 27 (67.50) | |||
| Perfusion defect | <0.001 | 0.01 | |||||
| Absent | 87 (71.31) | 88 (44.00) | 44 (67.69) | 16 (40.00) | |||
| Present | 35 (28.69) | 112 (56.00) | 21 (32.31) | 24 (60.00) | |||
| Radial or penetrating vessel | <0.001 | 0.002 | |||||
| Absent | 98 (80.33) | 122 (61.00) | 57 (87.69) | 24 (60.00) | |||
| Present | 24 (19.67) | 78 (39.00) | 8 (12.31) | 16 (40.00) | |||
| Enhancement scope enlarged | <0.001 | 0.008 | |||||
| Absent | 78 (63.93) | 72 (36.00) | 38 (58.46) | 12 (30.00) | |||
| Present | 44 (36.07) | 128 (64.00) | 27 (41.54) | 28 (70.00) | |||
| Lesion type | – | – | |||||
| Invasive carcinoma† | – | 193 (59.54) | – | 30 (28.57) | |||
| Carcinoma in situ | – | 3 (0.93) | – | 6 (5.71) | |||
| Other malignant‡ | – | 4 (1.24) | – | 4 (3.81) | |||
| Adenosis | 12 (3.73) | – | 23 (21.90) | – | |||
| Fibroadenoma | 53 (16.46) | – | 24 (22.86) | – | |||
| Intraductal papilloma | 6 (1.86) | – | 7 (6.67) | – | |||
| Inflammatory process | 8 (2.48) | – | 3 (2.86) | – | |||
| Fibrocystic change | 31 (9.63) | – | 3 (2.86) | – | |||
| Usual ductal hyperplasia | 8 (2.48) | – | 2 (1.90) | – | |||
| Benign phyllodes tumor | 3 (0.93) | – | 1 (0.95) | – | |||
| Other benign§ | 1 (0.31) | – | 2 (1.90) | – | |||
Data are presented as mean ± standard deviation for continuous variables, and n (%) for categorical variables. The data were expressed as the number of patients belonging to a specific category, with the percentage of the corresponding cohort presented in parentheses. †, the invasive carcinoma includes invasive ductal carcinoma, invasive lobular carcinoma. ‡, the other malignant conditions include mucinous carcinoma, intraductal papillary carcinoma, malignant breast granular cell tumor, and sarcoma. §, the other benign conditions include hamartoma and angiolipoma. BI-RADS, Breast Imaging-Reporting and Data System; BUS, B-mode ultrasound; LIQ, lower inner quadrant; LOQ, lower outer quadrant; UIQ, upper inner quadrant; UOQ, upper outer quadrant.
Radiomics and DL feature analysis
Using radiomics, 6 and 13 key features were identified from a total of 107 features extracted from the BUS and CEUS ROIs. For DL, 19 and 30 key features were selected from a total of 32 compressed features for the BUS and CEUS ROIs.
Radiomics, DL, and integrated model performance analysis
Table 2 presents the predictive diagnostic performance metrics for each model. In the training cohort, the AUCs for traditional and DL radiomics using BUS images were 0.757 and 0.850, respectively. Combining traditional and DL radiomics significantly enhanced model performance, with an AUC of 0.938. For the CEUS images, the AUCs for traditional and DL radiomics were 0.850 and 0.908, respectively, and the AUC for the integrated model reached 0.947. The final integrated model achieved an AUC of 0.943, significantly higher than the single-modality models using BUS and CEUS. The corresponding P values were as follows: BUS traditional, P<0.001; CEUS traditional, P<0.001; BUS DL, P<0.001; and CEUS DL, P=0.01. In the testing cohort, the AUCs for traditional radiomics and DL radiomics using BUS images were 0.722 and 0.754, respectively. The combination of conventional and DL radiomics improved performance, resulting in an AUC of 0.770. For the CEUS images, the AUCs for traditional radiomics and DL radiomics were 0.777 and 0.792, respectively, with the AUC for the integrated model being 0.822. The final integrated model outperformed the other models, with an AUC of 0.825. The ROC curves for all the models are shown in Figure 4. The DeLong test indicated no significant differences between the integrated model and most single-modality models in the testing cohort. The P-values were: BUS traditional, P=0.045; CEUS traditional, P=0.20; BUS DL, P=0.14; and CEUS DL, P=0.35. However, the integrated model demonstrated the greatest clinical benefit in the DCA, as illustrated in Figure 4.
Table 2. Evaluation results of the LR model in the training and testing cohort.
| Model | AUC | 95% CI | Accuracy | Sensitivity | Specificity |
|---|---|---|---|---|---|
| BUS traditional radiomics model | |||||
| Training | 0.757 | 0.703–0.811 | 0.717 | 0.710 | 0.730 |
| Testing | 0.722 | 0.615–0.830 | 0.743 | 0.600 | 0.831 |
| BUS deep learning radiomics model | |||||
| Training | 0.850 | 0.808–0.893 | 0.789 | 0.780 | 0.803 |
| Testing | 0.754 | 0.655–0.853 | 0.752 | 0.550 | 0.877 |
| BUS combined model | |||||
| Training | 0.938 | 0.913–0.964 | 0.879 | 0.900 | 0.844 |
| Testing | 0.770 | 0.665–0.874 | 0.733 | 0.725 | 0.738 |
| CEUS traditional radiomics model | |||||
| Training | 0.850 | 0.808–0.891 | 0.789 | 0.805 | 0.762 |
| Testing | 0.777 | 0.682–0.871 | 0.724 | 0.750 | 0.708 |
| CEUS deep learning radiomics model | |||||
| Training | 0.908 | 0.876–0.940 | 0.820 | 0.785 | 0.877 |
| Testing | 0.792 | 0.705–0.879 | 0.714 | 0.825 | 0.646 |
| CEUS combined model | |||||
| Training | 0.947 | 0.923–0.971 | 0.898 | 0.890 | 0.910 |
| Testing | 0.822 | 0.740–0.905 | 0.800 | 0.650 | 0.892 |
| Integrated model | |||||
| Training | 0.943 | 0.920–0.966 | 0.873 | 0.880 | 0.861 |
| Testing | 0.825 | 0.744–0.907 | 0.771 | 0.625 | 0.862 |
AUC, area under the curve; BUS, B-mode ultrasound; CEUS, contrast-enhanced ultrasound; CI, confidence interval; LR, logistic regression.
Figure 4.
Diagnostic performance comparison of different models. (A) Performance of models in the training set. (B) Performance of models in the testing set. (C) DCA in the testing set. The DCA curve demonstrates that the integrated model achieves the highest clinical benefit among all models in the testing cohort. AUC, area under the curve; BUS, B-mode ultrasound; CEUS, contrast-enhanced ultrasound; DCA, decision curve analysis.
Model interpretation
SHAP analysis was employed to interpret the integrated model and quantify the contribution of each feature to the predictive outcomes. By calculating SHAP values, local bar plots were generated to evaluate the impact of individual features on specific sample predictions, thereby establishing a framework for prioritizing feature importance. Notably, CEUS_original_firstorder_Skewness emerged as the most critical feature influencing the integrated model’s predictions. Figure 5 illustrates the overall impact of various features on model performance. Among the features incorporated into the integrated model, both BUS and CEUS features were found to play essential roles. However, the traditional radiomics features derived from CEUS demonstrated a particularly significant influence, accounting for five of the top nine features ranked by model impact. In contrast, the traditional radiomics features from BUS showed relatively limited contributions, with none ranking among the top nine most influential features.
Figure 5.
SHAP analysis of the integrated model for predicting benign or malignant breast tumors. (A) The SHAP summary plot of the integrated model reveals that the CEUS_original_firstorder_Skewness feature has the most significant impact on the prediction outcomes. This feature contributes the most to the model’s predictions across the entire dataset. (B) The waterfall plot illustrates the feature importance ranking, as determined by the SHAP values of a particular sample. BUS, B-mode ultrasound; CEUS, contrast-enhanced ultrasound; DL, deep learning; SHAP, Shapley Additive exPlanations.
The heatmaps generated by Grad-CAM illustrate regions of varying relevance to the model’s predictions, with red areas indicating high relevance to the model’s output and blue areas indicating low relevance. As shown in Figure 6, the model focuses primarily on two regions of critical importance for distinguishing between benign and malignant breast lesions. These regions are the hypoechoic region and the hyper-enhancement region inside the tumor. This finding provides further evidence supporting the model’s effectiveness and enhances its clinical interpretability.
Figure 6.
Representative patient examples are displayed, showing heatmaps for BUS and CEUS. Red areas correspond to higher weights, as indicated by the accompanying color bar. Typically, malignant lesions exhibit larger highlighted regions than benign lesions. These highlighted regions are predominantly distributed within the hypoechoic and hyper-enhanced areas inside the tumor. In contrast, the highlighted regions of benign lesions are mainly confined to the image boundaries. This may result from the lack of malignant-specific features in the center of the ROI. BUS, B-mode ultrasound; CEUS, contrast-enhanced ultrasound; ROI, region of interest.
Comparison between the integrated model and radiologists
A two-round reader study was conducted in the testing cohort to evaluate the diagnostic performance of the integrated model compared with that of the six radiologists. In the first round, the diagnostic performance of the six radiologists was compared with that of the integrated model (Figure 7). The integrated model achieved an AUC of 0.825, which was significantly greater than that of the two junior radiologists. The P values were P=0.048 versus junior radiologist 1 and P=0.03 versus junior radiologist 2. It also outperformed the intermediate and senior radiologists, although the differences were not statistically significant. The P values were P=0.64 and P=0.51 for intermediate radiologists 1 and 2, respectively, and P=0.98 and P=0.94 for senior radiologists 1 and 2, respectively. Six radiologists participated, and their AUCs were recorded based on experience: junior radiologists achieved 0.718 and 0.701, intermediate radiologists 0.799 and 0.791, and senior radiologists 0.824 and 0.821. The mean AUC across all six radiologists was 0.776, which was lower than the integrated model (0.825).
Figure 7.
Comparison of radiologists’ diagnostic performance with and without integrated model assistance. (A) Radiologists independently diagnosed breast lesions based solely on their experience, without the support of the integrated model. The results indicate that the AUC of the integrated model surpassed those of all radiologists. (B) With integrated model assistance, radiologists incorporated AI scores and heatmaps into their diagnostic process. This resulted in improved AUCs for all radiologists, with intermediate and senior radiologists achieving AUCs that exceeded the integrated model. (C-F) A detailed analysis of diagnostic performance changes demonstrated that the integrated model significantly enhanced the AUC, accuracy, and sensitivity across all radiologists. The specificity of senior radiologists improved, while slight decreases in specificity were observed among junior and intermediate radiologists. AI, artificial intelligence; AUC, area under the curve; CI, confidence interval.
In the second round, six radiologists performed diagnoses with the assistance of AI scores and heatmaps. All radiologists improved in AUC, accuracy, and sensitivity. Three also showed increased specificity. The individual AUCs after AI assistance were: junior 1 (0.748), junior 2 (0.751), intermediate 1 (0.837), intermediate 2 (0.849), senior 1 (0.864), and senior 2 (0.869). The mean AUC increased from 0.776 to 0.820 (ΔAUC =0.044), indicating a meaningful improvement in overall diagnostic performance. Four radiologists—intermediate 1, intermediate 2, senior 1, and senior 2—achieved notable AUC improvements and surpassed the AUC of the integrated model. The aggregated mean performance of radiologists approached the level of the integrated model. The performance metrics, including AUC, accuracy, sensitivity, and specificity, for the integrated model and all six radiologists across two rounds are summarized in Table 3, demonstrating the added value provided by AI support.
Table 3. Summary of the changes in the decision-making of radiologists before and after AI assistance.
| Model | Cohort | AUC | Accuracy | Sensitivity | Specificity |
|---|---|---|---|---|---|
| Integrated model | Testing | 0.825 | 0.743 | 0.725 | 0.754 |
| Junior radiologists | |||||
| Radiologist 1 | Testing | 0.718→0.748↑ | 0.752→0.771↑ | 0.575→0.650↑ | 0.862→0.846 |
| Radiologist 2 | Testing | 0.701→0.751↑ | 0.743→0.781↑ | 0.525→0.625↑ | 0.877→0.877 |
| Intermediate radiologists | |||||
| Radiologist 1 | Testing | 0.799→0.837↑ | 0.829→0.857↑ | 0.675→0.750↑ | 0.923→0.923 |
| Radiologist 2 | Testing | 0.791→0.849↑ | 0.819→0.867↑ | 0.675→0.775↑ | 0.908→0.923↑ |
| Senior radiologists | |||||
| Radiologist 1 | Testing | 0.824→0.864↑ | 0.848→0.886↑ | 0.725→0.775↑ | 0.923→0.954↑ |
| Radiologist 2 | Testing | 0.821→0.869↑ | 0.838→0.886↑ | 0.750→0.800↑ | 0.892→0.938↑ |
| Mean of six radiologists | Testing | 0.776→0.820↑ (Δ=0.044) | 0.805→0.841↑ (Δ=0.036) | 0.654→0.729↑ (Δ=0.075) | 0.898→0.910↑ (Δ=0.012) |
↑, increased. AI, artificial intelligence; AUC, area under the curve.
Discussion
This study developed an integrated model to predict whether breast lesions are benign or malignant by combining BUS and CEUS features. Compared with single-modality models, the integrated model combining DL and traditional radiomics showed better performance (AUC =0.825 vs. 0.722–0.822). SHAP analysis showed that CEUS features contributed more to model performance than BUS features. In the first-round reader study, the model outperformed radiologists (Model AUC > Radiologists AUC: 0.701–0.824). In the second-round reader study, AI scores and heatmaps from the model significantly improved diagnostic accuracy and sensitivity, especially for junior radiologists.
Most previous studies have used single-center, single-modality imaging for BC prediction (12,14). These studies often lack external validation, limiting the generalizability of their findings. Single-modality imaging provides valuable insights into tumor characteristics. However, its inherent limitations often hinder a comprehensive characterization of tumor heterogeneity, which is essential for accurate BC prediction (21). The integrated model combining BUS and CEUS features outperformed single-modality models. Limited interpretability remains a barrier to its clinical application. Despite strong diagnostic performance, this issue continues to hinder acceptance among radiologists (22). To improve model interpretability, this study uses SHAP analysis. SHAP quantifies each feature’s contribution to the model output (23). The SHAP summary plot (Figure 5) visually displays feature importance. The x-axis shows the distribution of absolute SHAP values. The color gradient reflects feature value importance. A two-round reader study evaluated the integrated model’s performance. In the first round, the model showed higher diagnostic accuracy than six radiologists. The second round assessed the model’s benefit to radiologists in clinical practice. The study showed that the model provided crucial support for accurate breast tumor diagnosis.
BC diagnosis currently depends on invasive pathological examinations. Radiomics is an emerging field that extracts imaging features invisible to the naked eye by analyzing individual modalities and integrating multimodal data with clinical information (24). Acting as a “virtual biopsy”, radiomics complements traditional imaging and aids early disease detection (25). It also offers potential to improve diagnostic accuracy, monitor treatment response, and support personalized management. Yet, many prior studies remain limited by single-center design, single-modality imaging, or a lack of comparison with radiologists (5,7,26). To address these gaps, we developed a multicenter, dual-modality integrated model. We also assessed its interpretability with SHAP analysis and validated its clinical utility in a two-round reader study. These steps demonstrate its added value in clinical practice.
The integrated model combining BUS and CEUS features outperformed single-modality models. Combining imaging techniques enables more detailed tumor analysis and detects subtle features often missed by single-modality imaging (27). BUS provides morphological information, but cannot assess blood flow perfusion. CEUS offers superior temporal resolution, allowing detailed visualization of microcirculation in tumors and surrounding tissue (18). Combining DL and traditional radiomics enables a more comprehensive tumor evaluation (28). In the testing cohort, the integrated model achieved the highest diagnostic performance (AUC =0.825). This outperformed the BUS-based (AUC =0.770) and CEUS-based combined models (AUC =0.822), as well as their respective traditional radiomics (AUC =0.722 and 0.777) and DL models (AUC =0.754 and 0.790). In the testing cohort, the integrated model’s superior performance confirms its robustness and generalizability. These results highlight the need for multicenter studies to validate integrated models. SHAP analysis identified CEUS_original_firstorder_Skewness as a key contributor to model performance. This feature measures pixel intensity asymmetry, with values below zero indicating a predominance of higher grayscale intensities (29). Malignant lesions showed lower Skewness values than benign ones. Notably, the seven most influential features in the integrated model were all derived from CEUS, underscoring CEUS’s superior predictive value over BUS in distinguishing benign from malignant tumors.
Ultrasound diagnosis requires extensive training due to its subjective nature, and diagnostic accuracy varies with experience (30). This subjectivity leads to challenges, particularly when benign and malignant features overlap (5). In the first-round reader study, radiologists achieved AUCs ranging from 0.701 to 0.824, with performance correlating positively with experience. However, diagnostic variability remained. In contrast, the integrated model showed consistently higher diagnostic accuracy. Integrated models will play an important role, but final decisions depend on experts due to limited understanding of the biological basis of model features. The second-round reader study evaluated the clinical utility of AI-assisted diagnosis. AI scores provided radiologists with computer-based quantitative analyses for individual patients. The heatmaps offered supplementary visual information by guiding radiologists to key areas. They also highlighted distinct patterns between benign and malignant lesions, thereby supporting reassessment and improving diagnostic accuracy. This assistance significantly improved diagnostic accuracy and sensitivity, especially among junior readers. Together, AI scores and heatmaps enhanced diagnostic precision. As shown in Figure 7, all radiologists improved in AUC, accuracy, and sensitivity during the second round. Junior radiologists showed the greatest improvement. The model helped mitigate limitations related to reader experience and visual perception. However, specificity declined among junior and intermediate radiologists. This may reflect a tendency to overcall malignancy under uncertainty to avoid missing cancer. Overall, the integrated model improved radiologists’ diagnostic performance and supported clinical decision-making. However, this study included only patients with benign or malignant lesions, without normal cases, and it was not specifically designed to evaluate radiologists’ reading time. Therefore, both the performance of the AI system in screening populations and its potential influence on reading efficiency remain to be validated in future prospective work.
There are several limitations in this study. First, the limited external sample size restricted assessment of the model’s generalizability, underscoring the need for larger multicenter cohorts. Second, the retrospective design introduces potential selection bias and data imbalance, which may affect the robustness of the results. Third, heterogeneity in ultrasound image quality across devices may have affected result consistency. Prospective studies are needed to further validate and generalize these findings.
Conclusions
In conclusion, this study developed an integrated model using BUS and CEUS for noninvasive differentiation of benign and malignant breast tumors. Model interpretability was enhanced through SHAP and Grad-CAM analyses. The model significantly improved radiologists’ diagnostic accuracy, supporting its potential clinical utility.
Supplementary
The article’s supplementary files as
Acknowledgments
Some of our experiments were carried out on the OnekeyAI platform. We thank OnekeyAI and their developers’ help in this scientific research work.
Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments. The study was approved by the ethics committee of the People’s Hospital of Guangxi Zhuang Autonomous Region, China (No. KY-LW-2020-24), and individual consent for this retrospective analysis was waived.
Footnotes
Reporting Checklist: The authors have completed the TRIPOD reporting checklist. Available at https://gs.amegroups.com/article/view/10.21037/gs-2025-295/rc
Funding: This work was supported by the Guangxi Key Research and Development Program (No. GuiKe AB24010145); the Innovation Project of Guangxi Graduate Education (No. YCSW2025287); the National Natural Science Foundation of China (Nos. 82160339 and 81660292); and the Guangxi Natural Science Foundation project (No. 2020GXNSFAA259014).
Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://gs.amegroups.com/article/view/10.21037/gs-2025-295/coif). The authors have no conflicts of interest to declare.
Data Sharing Statement
Available at https://gs.amegroups.com/article/view/10.21037/gs-2025-295/dss
References
- 1.Sung H, Ferlay J, Siegel RL, et al. Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 Countries. CA Cancer J Clin 2021;71:209-49. 10.3322/caac.21660 [DOI] [PubMed] [Google Scholar]
- 2.Siegel RL, Miller KD, Fuchs HE, et al. Cancer Statistics, 2021. CA Cancer J Clin 2021;71:7-33. 10.3322/caac.21654 [DOI] [PubMed] [Google Scholar]
- 3.Wang Q, Xiang Q, Yu L, et al. Corrigendum: Changes in Tumor-Infiltrating Lymphocytes and Vascular Normalization in Breast Cancer Patients After Neoadjuvant Chemotherapy and Their Correlations With DFS. Front Oncol 2021;11:661782. 10.3389/fonc.2021.661782 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Liu R, Li H, Liu C, et al. Malignant risk prediction of cystic-solid thyroid nodules using a comprehensive model integrating clinical and ultrasound features, ultrasound radiomics, and deep transfer learning. Gland Surg 2025;14:584-96. 10.21037/gs-2024-551 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Gong X, Li Q, Gu L, et al. Conventional ultrasound and contrast-enhanced ultrasound radiomics in breast cancer and molecular subtype diagnosis. Front Oncol 2023;13:1158736. 10.3389/fonc.2023.1158736 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Richman I, Tessier-Sherman B, Galusha D, et al. Breast cancer screening during the COVID-19 pandemic: moving from disparities to health equity. J Natl Cancer Inst 2023;115:139-45. 10.1093/jnci/djac172 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Yao Z, Luo T, Dong Y, et al. Virtual elastography ultrasound via generative adversarial network for breast cancer diagnosis. Nat Commun 2023;14:788. 10.1038/s41467-023-36102-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Patel H, Shah H, Patel G, et al. Hematologic cancer diagnosis and classification using machine and deep learning: State-of-the-art techniques and emerging research directives. Artif Intell Med 2024;152:102883. 10.1016/j.artmed.2024.102883 [DOI] [PubMed] [Google Scholar]
- 9.Yang M, Jin J. Radiomics model for predicting distant metastasis in soft tissue sarcoma of the extremities and trunk treated with surgery. Clin Transl Oncol 2025;27:2307-15. 10.1007/s12094-024-03746-4 [DOI] [PubMed] [Google Scholar]
- 10.Yu Q, Ning Y, Wang A, et al. Deep learning-assisted diagnosis of benign and malignant parotid tumors based on contrast-enhanced CT: a multicenter study. Eur Radiol 2023;33:6054-65. 10.1007/s00330-023-09568-2 [DOI] [PubMed] [Google Scholar]
- 11.Zwanenburg A, Vallières M, Abdalah MA, et al. The Image Biomarker Standardization Initiative: Standardized Quantitative Radiomics for High-Throughput Image-based Phenotyping. Radiology 2020;295:328-38. 10.1148/radiol.2020191145 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Shi S, An X, Li Y. Ultrasound Radiomics-Based Logistic Regression Model to Differentiate Between Benign and Malignant Breast Nodules. J Ultrasound Med 2023;42:869-79. 10.1002/jum.16078 [DOI] [PubMed] [Google Scholar]
- 13.Zheng Y, Zhou D, Liu H, et al. CT-based radiomics analysis of different machine learning models for differentiating benign and malignant parotid tumors. Eur Radiol 2022;32:6953-64. 10.1007/s00330-022-08830-3 [DOI] [PubMed] [Google Scholar]
- 14.Chen C, Wang Y, Niu J, et al. Domain Knowledge Powered Deep Learning for Breast Cancer Diagnosis Based on Contrast-Enhanced Ultrasound Videos. IEEE Trans Med Imaging 2021;40:2439-51. 10.1109/TMI.2021.3078370 [DOI] [PubMed] [Google Scholar]
- 15.Huang Z, Mo S, Wu H, et al. Optimizing breast cancer diagnosis with photoacoustic imaging: An analysis of intratumoral and peritumoral radiomics. Photoacoustics 2024;38:100606. 10.1016/j.pacs.2024.100606 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Lyu M, Yi S, Li C, et al. Multimodal prediction based on ultrasound for response to neoadjuvant chemotherapy in triple negative breast cancer. NPJ Precis Oncol 2025;9:259. 10.1038/s41698-025-01057-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Liu Z, Li Z, Qu J, et al. Radiomics of Multiparametric MRI for Pretreatment Prediction of Pathologic Complete Response to Neoadjuvant Chemotherapy in Breast Cancer: A Multicenter Study. Clin Cancer Res 2019;25:3538-47. 10.1158/1078-0432.CCR-18-3190 [DOI] [PubMed] [Google Scholar]
- 18.Tong T, Gu J, Xu D, et al. Deep learning radiomics based on contrast-enhanced ultrasound images for assisted diagnosis of pancreatic ductal adenocarcinoma and chronic pancreatitis. BMC Med 2022;20:74. 10.1186/s12916-022-02258-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Zhou X, Zhou P, Hu Z, et al. Diagnostic Efficiency of Quantitative Contrast-Enhanced Ultrasound Indicators for Discriminating Benign From Malignant Solid Thyroid Nodules. J Ultrasound Med 2018;37:425-37. 10.1002/jum.14347 [DOI] [PubMed] [Google Scholar]
- 20.Lee YW, Huang SK, Chang RF. CheXGAT: A disease correlation-aware network for thorax disease diagnosis from chest X-ray images. Artif Intell Med 2022;132:102382. 10.1016/j.artmed.2022.102382 [DOI] [PubMed] [Google Scholar]
- 21.Breen WG, Aryal MP, Cao Y, et al. Integrating multi-modal imaging in radiation treatments for glioblastoma. Neuro Oncol 2024;26:S17-25. 10.1093/neuonc/noad187 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Xu Y, Liu X, Cao X, et al. Artificial intelligence: A powerful paradigm for scientific research. Innovation (Camb) 2021;2:100179. 10.1016/j.xinn.2021.100179 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Rodríguez-Pérez R, Bajorath J. Interpretation of Compound Activity Predictions from Complex Machine Learning Models Using Local Approximations and Shapley Values. J Med Chem 2020;63:8761-77. 10.1021/acs.jmedchem.9b01101 [DOI] [PubMed] [Google Scholar]
- 24.Conti A, Duggento A, Indovina I, et al. Radiomics in breast cancer classification and prediction. Semin Cancer Biol 2021;72:238-50. 10.1016/j.semcancer.2020.04.002 [DOI] [PubMed] [Google Scholar]
- 25.Limkin EJ, Sun R, Dercle L, et al. Promises and challenges for the implementation of computational medical imaging (radiomics) in oncology. Ann Oncol 2017;28:1191-206. 10.1093/annonc/mdx034 [DOI] [PubMed] [Google Scholar]
- 26.Xu Z, Wang Y, Chen M, et al. Multi-region radiomics for artificially intelligent diagnosis of breast cancer using multimodal ultrasound. Comput Biol Med 2022;149:105920. 10.1016/j.compbiomed.2022.105920 [DOI] [PubMed] [Google Scholar]
- 27.Han W, Wang Y, Li T, et al. A CT-based integrated model for preoperative prediction of occult lymph node metastasis in early tongue cancer. PeerJ 2024;12:e17254. 10.7717/peerj.17254 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Liu Y, Wang Y, Hu X, et al. Multimodality deep learning radiomics predicts pathological response after neoadjuvant chemoradiotherapy for esophageal squamous cell carcinoma. Insights Imaging 2024;15:277. 10.1186/s13244-024-01851-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.van Griethuysen JJM, Fedorov A, Parmar C, et al. Computational Radiomics System to Decode the Radiographic Phenotype. Cancer Res 2017;77:e104-7. 10.1158/0008-5472.CAN-17-0339 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Wang W, Wu SS, Zhang JC, et al. Preoperative Pathological Grading of Hepatocellular Carcinoma Using Ultrasomics of Contrast-Enhanced Ultrasound. Acad Radiol 2021;28:1094-101. 10.1016/j.acra.2020.05.033 [DOI] [PubMed] [Google Scholar]