Abstract
Introduction
The study aims to evaluate the performance of an interpretable machine learning model in predicting preoperative axillary lymph node metastasis using primary breast cancer and lymph node features derived from contrast-enhanced mammography (CEM) and ultrasound (US) breast imaging reporting and data systems (BI-RADS).
Methods
This retrospective study included patients diagnosed with primary breast cancer. Two experienced radiologists extracted the BI-RADS features from the largest cross-section of the lesions and axillary lymph nodes based on CEM and US images, creating three datasets. Each dataset will train six base models to predict axillary lymph nodes, with pathological results serving as the gold standard. The top three models were used to train the five ensemble models. Additionally, SHapley Additive exPlanations (SHAP) was used to interpret the optimal model. The receiver-operating characteristic curve (ROC) and AUC were used to evaluate model performance.
Results
This study involved 292 female patients, of whom 99 had axillary lymph node metastasis and 193 did not. The combination of CEM and ultrasound BI-RADS demonstrated the best performance in predicting axillary lymph node metastasis. Among these, the LightGBM achieved the highest AUC (0.762) and specificity (86.67%, while the ensemble model using RF as the meta-model had an AUC (0.754) and specificity (83.33%. The most important variables identified by SHAP were the long diameters of the lymph nodes in the CEM recombined image, along with their complete morphology in the low-energy image.
Conclusion
The machine learning model using CEM and US BI-RADS features accurately predicted axillary lymph node metastasis before surgery, thereby serving as a valuable tool for clinical decision-making in patients with breast cancer.
Keywords: CEM, ultrasound, breast cancer, axillary lymph node metastasis, machine learning
Introduction
Breast cancer is the most commonly diagnosed type of cancer and the leading cause of cancer death. 1 Distant metastasis typically accounts for most fatalities, rather than primary tumor, with axillary lymph node metastasis (ALNM) being the most important prognostic factor. Assessing the status of axillary lymph nodes (ALN) aids in staging patients with breast cancer and predicting their response to treatment, thereby enabling more tailored therapy options. The assessment typically involves surgical procedures, such as sentinel lymph node biopsy and axillary lymph node dissection. However, these surgeries can result in complications such as vascular injury, upper limb sensory disturbance, and edema, which negatively affect patients’ quality of life.2,3
Ultrasound (US) is a widely accepted, convenient, and cost-effective tool for preoperative evaluation of lymph node metastasis in breast cancer. It assesses ALNM by analyzing signs such as lesion size, shape, and lymph node morphology.4–6 However, its performance is heavily dependent on the physician's experience, resulting in low repeatability and consistency. Furthermore, the evaluation of small and palpation-negative lymph nodes remains challenging.
CT and MRI can noninvasively assess the axillary lymph node status in breast cancer; however, preoperative evaluation using CT is infrequent due to its low specificity for lymph node assessment and its limitations in evaluating primary breast cancer. 6 Instead, it is more frequently used for postoperative assessment. MRI is also limited by high technical demands, long scan times, and associated costs.
Mammography is the most sensitive method for detecting breast calcifications, and microcalcification is an independent predictive factor for ALNM.7,8 Compared with US and MRI, mammography has the highest specificity and lowest sensitivity for predicting ALNM.6,9 However, conventional mammography is limited in fully displaying all axillary lymph nodes, leading to incomplete evaluation. Contrast-enhanced mammography (CEM) is an important advancement in digital mammography. CEM captures high- and low-energy images simultaneously after injecting iodine contrast. These images are then subtracted to produce the final recombined image. 10 CEM is used as a supplement to CE-MRI because it is more affordable, accessible, specific, and efficient and has better patient toleration.11–14 Previous studies have shown that CEM-based radiomics can predict lymph node metastasis in breast cancer with high sensitivity, but they exhibit low specificity.15,16
The different imaging modalities mentioned above each have their own advantages and disadvantages in evaluating lymph nodes in breast cancer, and integrating information from different modalities presents a challenge for radiologists. However, artificial intelligence offers promising solutions to this problem. Artificial intelligence can integrate information from different modalities, leveraging the advantages of various modalities to realize better predictive outcomes. Machine learning (ML), a subset of artificial intelligence, aims to replicate human learning by acquiring new information, gaining experiences, and reorganizing existing knowledge to enhance performance continuously. 17 Due to the complexity of real-life problems, a single machine learning model may not be sufficient to solve certain problems. 18 Therefore, ensemble learning was proposed to synthesize the results of several base classifiers.19,20 Previous studies have proven that ensemble models have better performance for predicting breast cancer and have introduced a new concept for the preoperative noninvasive prediction of axillary lymph node metastasis.21,22 However, the application of ML models in clinical practice is hindered because of the complexity and the “black box” nature of ML. SHAP (SHapley Additive exPlanations) is an interpretable method for explaining ML model predictions, which offers both global and local explanations, ensuring fairness and consistency in results. The application of SHAP fosters a deeper understanding of model predictions for physicians.23,24
This study aimed to create a machine-learning model that can noninvasively predict breast cancer lymph node metastasis using features from CEM and US images. The SHAP method was used to interpret the complex results of the model, providing further support for personalized breast cancer treatment.
Materials and Methods
Study Population
This retrospective study was conducted in accordance with the guidelines of the Declaration of Helsinki (2013 revision) and was approved by the Ethics Committee of Nanfang Hospital (Southern Medical University(NFEC-2017-136). Written informed consent was obtained from all participants. The study was conducted according to the TRIPOD guidelines. 25 Female patients who underwent CEM and routine ultrasound examinations at Nanfang Hospital between January 2018 and December 2021 were enrolled. The inclusion criteria were as follows: (1) histopathological and immunohistochemical confirmation of primary breast cancer, (2) pathological confirmation of lymph node status, (3) complete ultrasound and CEM data, (4) primary lesion were fully displayed in both CC and MLO view on CEM images, as well as in the 2D ultrasound with clear Doppler flow signal images, (5) at least one axillary lymph node must be clearly displayed in the MLO CEM image, (6) clear 2D ultrasound image with Doppler blood flow signals of axillary lymph nodes, and (7) all CEM and ultrasound images must be clear, artifact-free, and meet diagnostic requirements. The exclusion criteria were as follows: prior biopsy, radiotherapy, or chemotherapy before CEM, and ultrasound examination.
Instruments
CEM was performed using a Senographe Essential (GE Healthcare CESM). Before imaging, we administered a single bolus injection of non-ionic contrast medium (300 mgI/mL) through an intravenous catheter at a rate of 3 mL/s, with a dose of 1.5 mL/kg. We obtained low-energy (LE)(26-30 kVp) and high-energy (45-49 kVp) exposures in CC and MLO views for each breast. The recombined (RC) images were generated by reducing the attenuation differences between the low-energy and high-energy images and by decreasing the noise in the non-enhancing images. We first examined the abnormal breast, followed by the normal breast.
US was performed using a breast special L1 2-5 real-time linear array probe (Phillip iu 22) at a frequency of 7.5-15MHZ. The patients were positioned supine with raised arms, fully exposing the breasts and armpits. Radial continuous scanning was conducted with the nipple as the center. If lesions are detected, the scanning will focus on clinically or mammographically indicated areas using multi-sectional and multi-angle scanning. Careful observation was given to the lesions, surrounding tissues, and axillary lymph nodes.
Imaging Analysis
Two radiologists, with 9 and 10 years of experience in breast imaging, analyzed low-energy and recombined CEM images according to the fifth edition of ACR-BI-RADS 26 and ACR-BI-RADS-CEM. 10 They also analyzed ultrasound images according to the fifth edition of ACR-BI-RADS. 26 They did not know whether the patient had lymph node metastasis. Consultation was sought in case of disagreement.
We collected 10 low-energy CEM features, including fibroglandular tissue type, primary findings, maximum tumor diameter, tumor stage and number of tumors, mass density and morphology, mass margin, relationship between tumor and calcifications, and the morphology of calcifications inside and outside the tumor. For primary lesion recombined CEM images, we collected 11 features, including major enhancement types, maximum tumor diameter, T stage classification, number of enhancement sites, mass-like enhancement morphology, mass-like enhancement margin, degree of mass-like enhancement, internal features of mass-like enhancement, degree of non-mass-like enhancement, internal features, and distribution. Furthermore, we gathered 6 CEM low energy images and recombined image features related to axillary lymph nodes, including lymph node morphology, density, cortical thickening or not, cortical thickness, short and long diameter ratio, and degree of enhancement (Figure 1).
Figure 1.
A-B The Long Diameter(A) and Short dDiameter (B) of the Suspicious and Enhanced Lymph Node on the Recombined CEM Image.
Thirteen ultrasonic features of the primary lesions were collected from the US images. These features include major lesion types, maximum tumor diameter, tumor stage, number of primary lesions, lesion echo, posterior lesion echo, relationship between calcification and lesion, lesion blood signal, lesion blood signal grade, mass morphology, mass margin, lymph-node short- and long-diameter ratio, and lymph-node cortical thickening (Figure 2).
Figure 2.
A-B The Vertical Diameter(A) and Transverse Diameter (B) Diameter of the Suspicious Lymph Node on the Ultrasound Image.
To explore the effectiveness of CEM BI-RADS features, US BI-RADS features, and their combination in predicting ALNM, three datasets were generated: (1) BI-RADS features from low-energy CEM and recombined CEM images; (2) Ultrasonic BI-RADS features; (3) A combination of low-energy CEM images, recombined CEM BI-RADS features, and ultrasonic BI-RADS features.
Development of Machine Learning Models
The collected data were preprocessed prior to developing the model. Significant differences in mean, minimum, maximum, and standard deviations among features can impact model performance. Continuous variables were normalized to a range of 0-1, and categorical variables were transformed using One-Hot encoding. What's more, outliers were identified and removed using the Isolation Forest method with a threshold of 0.05, while multicollinearity was addressed using Pearson correlation with a threshold of 0.9.
Pathological histological results were used as the gold standard for feature selection. In the univariate selection process, we calculated the Pearson correlation coefficient to evaluate the relationship between each feature and the gold standard, retaining only features with absolute correlation coefficients greater than 0.2. The specification of the threshold indicates that this is a preliminary screening, and we want to retain more features. For multivariate selection, we employed the LightGBM-based SelectFromModel method from Scikit-Learn.
The machine learning models were trained using three datasets. The models were developed in Python 3.7, using packages like Scikit-Learn (version 0.23.2) and Mlxtend (version 0.18.0). The datasets were divided into a training set (70%) and a testing set (30%). The base models were trained on the training set using six machine learning classifiers: Random Forest (RF), eXtreme Gradient Boosting (XGBoost), Extra Trees Classifier (ET), Light Gradient Boosting Machine (LightGBM), Gradient Boosting Classifier (GBC), and AdaBoost. We used ten-fold cross-validation and grid search to identify the optimal hyperparameters using the AUC as the evaluation criterion. For each dataset, the top three models were selected based on their AUC scores to construct ensemble models. Each ensemble comprises three base models and one meta-model. The meta-model includes five classifiers: Logistic Regression (LR), Decision Tree (DT), RF, Support Vector Machine (SVM), and XGBoost. Finally, we compare and analyze the prediction performances of the single and ensemble machine learning models.
After obtaining the optimal ensemble model, we applied the interpretable machine learning technique SHAP (0.32.1) to explain its results. SHAP analyzes the model's results globally and locally, providing information on the overall importance of different features for predicting outcomes and their influence on individual samples, thus enhancing interpretability.
Statistical Analysis
The statistical analysis was conducted using SPSS 20.0. A two-sided test was used, with a significance level of P < .05 considered statistically significant. The model was internally verified using the bootstrapping resampling method with 1000 repetitions. The ROC, AUC, Accuracy, Recall, Precision, F1 value, specificity, and negative predictive value of each model in predicting axillary lymph node metastasis of breast cancer were calculated. Additionally, the AUCs of the different models were compared using the Delong test.
Results
General Information
This study enrolled 292 female patients with primary breast cancer, aged 49.64 ± 9.71 years (range: 28-76 years). Among these, 99 (33.9%) patients were positive for ALNM and 193 (66.1%) were negative. There were statistically significant differences in terms of palpation and family history of breast cancer between the two groups (P < .05) (Table 1). However, most cases in both groups were clinically involved (91.92% vs 79.89%), with most having no family history of breast cancer (94.95% vs 98.96%). There were significant differences in histopathologic type and grade between the ALNM (+) group and the ALNM (−) group (P < .05). The proportion of invasive carcinoma was 17.10% higher in the ALNM (+) group than in the ALNM (−) group (100.00% vs 82.90%).
Table 1.
The Clinical Features of the Breast Cancer.
| Group | ALNM(+) (N = 99) | ALNM (−) (N = 193) | P-value |
|---|---|---|---|
| Age(Years) | 49.49 ± 8.73 (30-69) | 49.72 ± 10.20 (28-76) | .855 |
| position | .474 | ||
| -Left | 49(49.49%) | 87(45.08%) | |
| -Right | 50(50.51%) | 106(54.92%) | |
| clinical features | .023 | ||
| -palpation | 91(91.92%) | 154(79.89%) | |
| -Papillary discharge | 0(0.00%) | 4(2.07%) | |
| -Other signs | 2(2.02%) | 7(3.63%) | |
| -Negative | 6(6.06%) | 28(14.51%) | |
| Menstruation | .093 | ||
| -premenopause | 66(66.67%) | 109(56.48%) | |
| -Menopause | 33(33.33%) | 84(43.52%) | |
| History of breastfeeding | .666 | ||
| -Yes | 86(86.89%) | 171(88.60%) | |
| -No | 13(13.11%) | 22(11.40%) | |
| Reproductive history | .966 | ||
| -Yes | 95(96.00%) | 185(95.85%) | |
| -No | 4(4.00%) | 8(4.15%) | |
| Family history of breast cancer | .040 | ||
| -Yes | 5(4.05%) | 2(1.04%) | |
| -No | 94(94.95%) | 191(98.96%) | |
| Histopathological types and grades | .000 | ||
| -DCIS | |||
| -Grade 1-2 | 0(0.00%) | 10(5.18%) | |
| -Grade 3 | 0(0.00%) | 23(11.92%) | |
| -DCIS with microinvasion | 1(1.01%) | 8(4.15%) | |
| -IDC with DCIS | 29(29.29%) | 40(20.73%) | |
| -IDC | |||
| -Grade 1 | 1(1.01%) | 1(0.52%) | |
| -Grade 2 | 53(53.54%) | 68(35.23%) | |
| -Grade 3 | 7(7.07%) | 17(8.81%) | |
| -ILC | 3(3.03%) | 6(3.11%) | |
| -Others | 5(5.05%) | 20(10.36%) | |
| Molecular subtypes | .179 | ||
| -luminal A | 19(19.19%) | 48(24.87%) | |
| -luminal B | 66(66.67%) | 103(53.37%) | |
| -HER2 over-express | 7(7.07%) | 21(10.88%) | |
| -triple negative | 7(7.07%) | 21(10.88%) |
ALNM, axillary lymph node metastasis; DCIS, ductal carcinoma in situ; HER2, human epidermal growth factor receptor 2; IDC, invasive ductal carcinoma; ILC, invasive lobular carcinoma.
Feature Selection
Five features of the CEM low-energy images (see Tables 2 and 4), six features of the CEM recombined images (refer to Figure 3, Tables 3 and 4), and six ultrasonic features (shown in Table 5) exhibited statistically significant differences between the ALNM (+) group and the ALNM (−) group (P < .05). We observed abnormal axillary lymph nodes that appeared enlarged and uniformly thickened on ultrasound and low-energy CEM images. Blood flow signals were detected by ultrasound. Additionally, the recombined CEM images show mild to moderate enhancement (Figure 4).
Table 2.
The CEM Low-energy Features of the Axillary Lymph-Node of Different Group.
| Groups | ALNM(+) (n = 99) | ALNM(−) (n = 193) | P-value |
|---|---|---|---|
| Breast composition | .851 | ||
| -a | 1(1.01%) | 2(1.04%) | |
| -b | 7(7.07%) | 9(4.66%) | |
| -c | 82(82.83%) | 166(86.01%) | |
| -d | 9(9.09%) | 16(8.29%) | |
| Findings | .004 | ||
| -masses | 29(29.29%) | 75(38.86%) | |
| -mass with calcification | 62(62.63%) | 85(44.04%) | |
| -calcifications | 3(3.03%) | 17(8.81%) | |
| -asymmetries | 0(0.00%) | 7(3.63%) | |
| -others | 5(5.05%) | 9(4.66%） | |
| Long diameter of tumor (L1,cm) | 3.01 ± 1.79 (0.90-10.10) | 2.69 ± 1.65 (0.70-10.10) | .123 |
| T-stage | .599 | ||
| ≤2.0 cm(T1) | 34(34.34%) | 77(39.90%) | |
| -2.0 cm＜T ≤ 5.0 cm(T2) | 54(54.55%) | 99(51.30%) | |
| -5.0 cm＜T(T3) | 11(11.11%) | 17(8.80%) | |
| Number of lesions | .154 | ||
| -1 | 74(74.75%) | 158(81.87%) | |
| ≥2 | 25(25.25%) | 35(18.13%) |
ALNM, axillary lymph node metastasis; CEM, contrast-enhanced mammography.
Table 4.
The CEM Features of the Axillary Lymph-Node Different status for Breast Cancer.
| CEM Findings | ALNM (+) (n = 99) | ALNM(−) (n = 193) | P-value |
|---|---|---|---|
| Seen on low energy images | .000 | ||
| ALN enlarged | |||
| -Not | 37(37.37%) | 153(79.27%) | |
| -Yes | 62(62.63%) | 40(20.73%) | |
| Dense ALN | .000 | ||
| -Not | 38(38.38%) | 150(77.72%) | |
| -Yes | 61(61.62%) | 43(22.28%) | |
| LN cortex | .000 | ||
| -Not thickened | 34(34.34%) | 148(76.68%) | |
| -Focal thickened | 29(29.29%) | 23(11.92%) | |
| -Uniformly thickened | 36(36.37%) | 22(11.40%) | |
| Cortical thickness of lymph nodes(cm) | 0.79 ± 0.70 (0.20-5.20 cm) | 0.34 ± 0.46 (0.20-3.60 cm) | .000 |
| Long diameter of enhanced ALN (cm) | 2.98 ± 1.66 (1.40-9.20 cm) | 2.26 ± 0.90 (1.00-4.60 cm) | .004 |
| Level of enhanced ALN | .000 | ||
| -Not | 31(31.63%) | 138(71.13%) | |
| -Mild | 35(35.71%) | 50(25.77%) | |
| -Moderate | 13(13.27%) | 5(2.58%) | |
| -Marked | 19(19.39%) | 1(0.52%) | |
| ALN short/long diameter ratio | .000 | ||
| -≤1 | 85(85.85%) | 118(61.13%) | |
| -＞1 | 14(14.15%) | 75(38.87%) |
ALN, axillary lymph nodes; ALNM, axillary lymph node metastasis; CEM, contrast-enhanced mammography; LN, lymph nodes.
Figure 3.
Main CEM Recombination Image Features of the Axillary Lymph Nodes in the Different Groups.
Table 3.
The CEM Recombined Image Features of the Axillary Lymph Node of Different Group.
| Mass-enhanced features | ALNM(+) (n = 91) | ALNM(−) (n = 160) | P-value |
|---|---|---|---|
| Shape | .022 | ||
| -Regular | 77(84.62%) | 115(71.88%) | |
| -Oval/round | 14(15.38%) | 45(28.12%) | |
| Margin | .431 | ||
| -Not circumscribed | 75(86.40%) | 135(84.47%) | |
| -Spiculated | 15(12.50%) | 20(12.42%) | |
| -Circumscribed | 1(1.10%) | 5(3.13%) | |
| Enhanced level | .904 | ||
| -Mild | 1(1.10%) | 1(0.63%) | |
| -Moderate | 8(8.79%) | 13(8.13%) | |
| -Marked | 82(90.11%) | 146(91.34%) | |
| Internal enhancement characteristics | .327 | ||
| -Homogeneous | 33(36.26%) | 69(43.13%) | |
| -Heterogeneous | 53(58.24%) | 78(48.75%) | |
| -Ring | 5(5.50%) | 13(8.12%) | |
| Long diameter of tumor (L2,cm) | 2.84 ± 1.39 (1.10-9.20) | 2.29 ± 0.96 (0.80-6.00) | .000 |
| T-stage | .027 | ||
| ≤2.0 cm(T1) | 32(35.16%) | 77(48.12%) | |
| -2.0 cm＜T ≤ 5.0 cm(T2) | 53(58.24%) | 80(50.00%) | |
| -5.0 cm＜T(T3) | 6(6.60%) | 3(1.88%) | |
| Number of lesions | .095 | ||
| -1 | 65(71.43%) | 129(80.63%) | |
| ≥2 | 26(28.57%) | 31(18.37%) |
ALNM, axillary lymph node metastasis; CEM, contrast-enhanced mammography.
Table 5.
The Ultrasound Features of the Axillary Lymph-node of Different Group.
| ALNM (+) (n = 99) | ALNM (−) (n = 193) | P-value | |
|---|---|---|---|
| Findings | .197 | ||
| -Mass | 44(44.44%) | 98(50.78%) | |
| -Mass with calcifications | 51(51.52%) | 79(40.93%） | |
| -Others | 3(3.03%) | 10(5.18%) | |
| -Negative | 1(1.01%) | 6(3.11%） | |
| Long diameter of tumor (L3,cm) | 2.59 ± 1.53 (0.00-11.30) | 1.95 ± 1.11 (0.00-8.00) | .000 |
| T-stage | .001 | ||
| ≤2.0cm(T1) | 40(40.82%) | 114(60.82%) | |
| -2.0 cm＜T ≤ 5.0 cm(T2) | 52(53.06%) | 71(38.14%) | |
| -5.0 cm＜T(T3) | 6(6.12%) | 2(1.04%) | |
| Echo pattern | .443 | ||
| -Anechoic | 0(0.00%) | 4(2.15%) | |
| -Hypoechoic | 91(92.86%) | 168(90.32%) | |
| -Hyperechoic | 1(1.02%) | 4(2.15%) | |
| -Complex cystic and solid | 6(6.12%) | 10(5.38%) | |
| Posteriors patterns | .586 | ||
| -No | 89(90.82%) | 166(89.73%) | |
| -Enhancement | 0(0.00%) | 7(1.08%) | |
| -Shadowing | 9(9.18%) | 17(9.19%) | |
| Calcifications | .256 | ||
| -Absent | 46(46.46%) | 98(52.98%) | |
| -In a mass | 52(52.53%) | 80(43.24%) | |
| -Outside of a mass | 0(0.00%) | 2(1.08%) | |
| -Intraductal | 1(1.01%) | 5(2.70%) | |
| Vascularity | .020 | ||
| -Absent | 32(32.32%) | 84(43.52%) | |
| -Internal vascularity | 39(39.40%) | 46(23.83%) | |
| -Vessels in rim | 6(6.06%) | 23(11.92%) | |
| -Internal vascularity and vessels in rim | 22(22.22%) | 40(20.73%) | |
| Masses shapes | .012 | ||
| -Oval/round | 1(1.06%) | 13(7.34%) | |
| -Regular | 93(98.94%) | 164(92.66%) | |
| Masses margins | .147 | ||
| -Circumscribed | 6(6.38%) | 19(10.73%) | |
| -Indistinct | 21(22.34%) | 58(32.77%) | |
| -Angular | 36(38.30%) | 54(30.51%) | |
| -Microlobulated | 3(3.19%) | 8(4.52%) | |
| -Spiculated | 28(29.79%) | 38(21.47%) | |
| Number of masses | .101 | ||
| -1 | 75(78.72%) | 153(86.44%) | |
| ≥2 | 20(21.28%) | 24(13.56%) | |
| ALN short/long diameter ratio | .026 | ||
| -≤1 | 63(63.63%) | 173(89.63%) | |
| -＞1 | 36(36.37%) | 20(10.37%) | |
| LN cortex | .000 | ||
| -Not thickened | 35(35.35%) | 151(78.24%) | |
| -Focal thickened | 33(33.33%) | 37(19.17%) | |
| -Uniformly thickened | 31(31.32%) | 5(2.59%) |
ALN, axillary lymph nodes; ALNM, axillary lymph node metastasis; LN, lymph nodes.
Figure 4.
(A-C) Ultrasound and CEM Images of the Metastatic Axillary Lymph Node. (A) Ultrasound Image. The Left Axillary Lymph Node Cortex is Uiformly Thickened, and the Lymphatic Hilum Disappears, with a Size of 3.7 cm × 2.5 cm and Abundant Blood Flow Signals in and Around it. (B) CEM Low Energy Image. The Left Axillary Lymph Nodes with Full Shape, High Density, and Uniform Thickness are Abnormal. (C) CEM Recombination Image. The Left Axillary Lymph Nodes with Mild-to-Moderate Enhancement are Abnormal.
Performance of Machine Learning Models
The training set comprised 193 cases, including 71 in the ALNM (+) group and 122 in the ALNM (−) group. The testing set had 99 cases, including 28 in the ALNM (+) group and 71 in the ALNM (−) group. Table 7 presents the average performance of different modes in the training set. The comparison of AUC shows that the combination of CEM and US BI-RADS features achieved the highest performance among these machine-learning models. All classifiers using CEM and ultrasonic BI-RADS features recorded AUC values above 0.750, with LightGBM achieving the highest AUC of 0.801 (Table 6). The top three models on the testing set based on these features were LightGBM, RF, and ET, with AUCs of 0.762, 0.724, and 0.720, respectively (Table 7). These three models are used as base models to construct ensemble models. The ensemble model with RF as the meta-classifier in the combination of CEM and US exhibited the best performance (Table 8), and the F1 value was higher than the RF classifier by 8.51% (65.65% vs 57.14%).
Table 7.
The Performance of Base Models in Predicting Lymph-node status for Breast Cancer Using CEM and Ultrasound Features in the Testing set.
| Classifiers | AUC | Accuracy | Recall | Precision | F1 |
|---|---|---|---|---|---|
| LightGBM | 0.7623 | 0.7386 | 0.6429 | 0.5805 | 0.6102 |
| ET | 0.7241 | 0.7273 | 0.6071 | 0.5667 | 0.5862 |
| RF | 0.7196 | 0.7273 | 0.5714 | 0.5714 | 0.5714 |
| XGBoost | 0.7190 | 0.6818 | 0.4643 | 0.5000 | 0.4815 |
| AdaBoost | 0.6845 | 0.7273 | 0.5000 | 0.5833 | 0.5385 |
| GBC | 0.6905 | 0.6818 | 0.6071 | 0.5000 | 0.5484 |
AdaBoost, ada boost Classifier; AUC, area under the curve; CEM, contrast-enhanced mammography; ET, Extra Trees Classifier; GBC, Gradient Boosting Classifier; LightGBM, Light Gradient Boosting Machine; RF, random forest classifier; US, ultrasound; XGBoost, eXtreme gradient boosting classifier.
Table 6.
Classification Performance of Different Classifiers on Different Training Dataset.
| Modes/Classifiers | AUC | Accuracy | Recall | Precision | F1 |
|---|---|---|---|---|---|
| CEM + US | |||||
| -LightGBM | 0.8011 | 0.7613 | 0.6530 | 0.6213 | 0.6240 |
| -XGBoost | 0.7902 | 0.7563 | 0.6330 | 0.6167 | 0.6122 |
| -RF | 0.7838 | 0.7608 | 0.5989 | 0.6324 | 0.6001 |
| -AdaBoost | 0.7775 | 0.7566 | 0.6206 | 0.6224 | 0.6091 |
| -GBC | 0.7599 | 0.7561 | 0.6387 | 0.6274 | 0.6131 |
| -ET | 0.7556 | 0.7192 | 0.5357 | 0.5707 | 0.5358 |
| CEM | |||||
| -LightGBM | 0.6118 | 0.6457 | 0.4320 | 0.4805 | 0.4440 |
| -XGBoost | 0.5827 | 0.6057 | 0.3934 | 0.4130 | 0.3951 |
| -RF | 0.5970 | 0.6114 | 0.3773 | 0.4172 | 0.3917 |
| -AdaBoost | 0.5881 | 0.6202 | 0.4025 | 0.4054 | 0.3971 |
| -GBC | 0.5988 | 0.6410 | 0.4650 | 0.4671 | 0.4578 |
| -ET | 0.6112 | 0.2962 | 0.3810 | 0.3274 | 0.2962 |
| US | |||||
| -ET | 0.6929 | 0.7149 | 0.5359 | 0.6005 | 0.5569 |
| -RF | 0.6526 | 0.7202 | 0.5591 | 0.6165 | 0.5701 |
| -GBC | 0.6488 | 0.6936 | 0.5287 | 0.5620 | 0.5259 |
| -XGBoost | 0.6321 | 0.6670 | 0.5287 | 0.5214 | 0.5071 |
| -LightGBM | 0.6312 | 0.6667 | 0.5305 | 0.5314 | 0.5077 |
| -AdaBoost | 0.6169 | 0.6456 | 0.4887 | 0.4886 | 0.4662 |
AdaBoost, ada boost Classifier; AUC, area under the curve; CEM, contrast-enhanced mammography; ET, Extra Trees Classifier; GBC, Gradient Boosting Classifier; LightGBM, Light Gradient Boosting Machine; RF, random forest classifier; US, ultrasound; XGBoost, eXtreme gradient boosting classifier.
Table 8.
The Performance of Ensemble Model in Predicting the Axillary Lymph-node status for Breast Cancer Based on the CEM and Ultrasound Features in the Testing set.
| Meta-classifier | models | AUC | Accuracy | Recall | Precision | F1 |
|---|---|---|---|---|---|---|
| LR | Stacking Classifier1 | 0.7464 | 0.7159 | 0.5357 | 0.5556 | 0.5455 |
| DT | Stacking Classifier2 | 0.6631 | 0.6250 | 0.3929 | 0.4074 | 0.4000 |
| RF | Stacking Classifier3 | 0.7540 | 0.7841 | 0.6429 | 0.6667 | 0.6545 |
| SVM | Stacking Classifier4 | 0.7100 | 0.7045 | 0.5000 | 0.5385 | 0.5185 |
| XGBoost | Stacking Classifier5 | 0.7321 | 0.7500 | 0.6071 | 0.6071 | 0.6071 |
AUC, area under the curve; CEM, contrast-enhanced mammography; DT, decision tree classifier; LR, logistic regression classifier; RF, random forest classifier; SVM, support vector machine classifier; XGBoost, eXtreme gradient boosting classifier.
The AUC values for the LightGBM classifier (0.762) and the ensemble RF model (0.754) were not significantly different in the testing set (P > .05). However, the ensemble RF model demonstrated 4.55% higher accuracy (0.784 vs 0.739) and 8.62% higher precision (0.667 vs 0.581). Both models exhibited comparable sensitivity and specificity in the testing set. The LightGBM classifier achieved a sensitivity of 60.71% and a specificity of 86.67%, whereas the ensemble RF model had a sensitivity of 64.29% and a specificity of 83.33%.
For the LightGBM classifier, the top three influencing factors were: (1) long diameter of ALN on CEM recombined images combined with ALN enlarged on CEM low energy images, (2) LN cortex uniformly thickened on ultrasound images, and (3) focal thickening of lymph node on low-energy CEM combined with the long diameter of the tumor on recombined CEM (Figure 5A).
Figure 5.
SHAP Value of the LightGBM (A) Model and RF-Stacking Classifier (B) for Predicting the Axillary Lymph-Node Status in Breast Cancer Using CEM and Ultrasound Features in the Testing set. CEM, Contrast-Enhanced Mammography; RF, Random Forest Classifier.
For the ensemble RF model, the top three influencing factors were: (1) long diameter of ALN on the recombined CEM image combined with ALN enlarged on low-energy CEM images, (2) LN cortex not thickened on US, and (3) LN cortex not thickened on US images combined with a solitary mass on CEM images (Figure 5B).
Discussion
Axillary lymph node metastasis is the most important prognostic factor for breast cancer. Medical imaging plays a vital role in the preoperative noninvasive evaluation of lymph node staging. In recent years, artificial intelligence has shown promising potential for detecting lymph node metastasis and assessing prognosis in breast cancer.22,27
In this study, the AUCs of machine learning models using only CEM or US BI-RADS features were lower than those of models using both CEM and US BI-RADS features. The LightGBM classifier combined the CEM and US BI-RADS features of breast cancer and axillary lymph nodes achieved the highest AUC, with an AUC of 0.801 in the training set and an AUC of 0.762 in the testing set. This suggests that the combination of mammography and ultrasound is more efficient. In addition, most models achieved AUCs exceeding 0.700, demonstrating that machine learning models using CEM and US BI-RADS features for breast cancer and axillary lymph nodes have stable predictive performance. Compared with other models, the LightGBM is an efficient gradient boosting model that uses a histogram-based algorithm to discretize continuous features, which can accelerate the computation process and reduce computational complexity. It natively supports categorical variables, avoiding feature encoding conversion and addressing most of the pain points associated with gradient boosting models. Compared to other classifier like XGBoost, RF, and DT, it offers higher memory efficiency and computational performance. In addition, LightGBM outperforms AdaBoost in anti-overfitting because of its multidimensional constraint mechanism, which balances model performance and generalization.
Similarly, Yang 28 used a clinicopathologically and mammography-based radiomics nomogram to predict ALN metastasis in patients with breast cancer. Their results showed that the AUCs of the SVM and Nomogram classifiers in the training set were 0.840 and 0.800, while those of the testing set were 0.745 and 0.730, respectively. Their AUCs in the testing set were slightly lower than those of the LightGBM classifier and the ensemble RF model in our study. Mao 15 found that a radiomics nomogram using CEM has promising potential for preoperative prediction of ALN metastasis in breast cancer. The internal validation sets had an AUC of 0.767, sensitivity of 95.00%, and specificity of 34.00%. 15 Our LightGBM classifier and ensemble RF model demonstrated higher specificity (86.67% and 83.33%), but lower sensitivity (60.71% and 64.29%). The specificities of the LightGBM and ensemble RF model were also higher than that of Tan 29 by 5.37% and 2.03%, respectively. Their radiomics nomogram showed better predictive performance compared with the independent digital mammography-reported criteria and clinical features (AUC of 0.862 vs 0.666 vs 0.657). The sensitivity, specificity, and accuracy were 75.0%, 81.3%, and 79.2%, respectively. In predicting axillary lymph nodes of breast cancer, mammography had a higher specificity (99.5%) and lower sensitivity (21.0%) than breast ultrasound. 9 However, existing studies have mainly focused on DM, US, MRI, PET, and their radiomic features, as well as the predictive performance of different US or MRI techniques for axillary lymph nodes in breast cancer. The prediction of axillary lymph node metastasis in breast cancer based solely on CEM and US BI-RADS characteristics of the primary tumor and lymph nodes is limited.
Compared to previous studies, our study compared the model performance of BIRADS features from CEM and US and integrated the two types of features to construct machine learning models. This allows for better use of different image data modalities, resulting in improved prediction outcomes. BI-RADS features align closely with the diagnostic thinking of clinical practitioners, offering better interpretability than the “black box model” of radionics. When constructing models based on BI-RADS features, there is no need for lesion segmentation and complex parameter extraction, allowing direct use of routine clinical annotation data, making it easier to integrate into existing imaging workstations and align with clinical workflows. From a modeling perspective, we not only used classic classifiers like RF and DT, but also employed more complex ensemble machine learning models, such as XGBoost, AdaBoost, LightGBM, and GBC. Furthermore, we also explored an ensemble model that integrates the results of the three trained models with the aim of achieving better prediction results. We believe that the higher specificity value in this study was because the model used both CEM and ultrasound features of the axillary lymph nodes. The low sensitivity may be attributed to the small difference between CEM for primary breast cancer lesions and BI-RADS. Our model has higher specificity, which could prevent unnecessary medical examinations and patient anxiety by reducing false positives while enhancing the reliability of negative predictions to optimize clinical decision-making. This approach may help physicians accurately stage breast cancer, develop personalized surgical and treatment strategies, and ultimately improve patient prognosis.
To better understand the important features of the predictive model, we incorporated the interpretable SHAP method. The results show that the long diameter of ALN on CEM recombined images combined with ALN enlargement on CEM low-energy images emerged as the primary shared risk factor for both the LightGBM classifier and ensemble RF model. Additionally, we found that a long diameter of ALN on recombined CEM images longer than 3 cm and ALN enlargement on low-energy CEM images are associated with an increased risk of lymph node metastasis. However, we did not find any associations between special histologic type, tumor size, cortical thickness, and mass shapes and the presence of lymph node metastasis, which have been described in other predictive models of risk.30,31 However, we did not use SHAP to combine hyperparameter tuning and feature selection due to the complexity. In future, we plan to further use SHAP to integrate feature selection and hyperparameter tuning to realize improved model performance.
We also identified several key features associated with axillary lymph node metastasis, including mass with calcification, enlarged lymph nodes, and increased lymph node density in low-energy CEM images; mass-like enhancement, irregular shape of mass-like enhancement in recombined CEM images; tumor long axis greater than 2.6 cm, T2-3 stage, and blood flow signal within the tumor lesion in US. The definition of abnormal axillary lymph nodes detected by mammography is not consistent. Enlarged axillary lymph nodes that are new, considerably larger, or rounder than those in previous examinations may be classified as abnormal FFDM lymph nodes. 11 In some studies,9,28 abnormal lymph nodes were defined as being larger than 2 cm, round, or irregular in shape, and exhibiting loss of hilar fat and increased density. In our study, we found that axillary nodes with larger size, high density, thickened cortex, and loss of fatty hilum were associated with ALN metastasis. These findings agree with previous studies.9,15,28 Additionally, we discovered that enlarged lymph nodes with moderate or high enhancement on CEM were significant indicators of axillary lymph node metastasis in breast cancer. However, after model development and interpretation using SHAP, we found that the primary factors associated with axillary lymph node metastasis were the CEM and ultrasound BI-RADS features of the lymph nodes, especially when we applied the SHARP method for interpretation. Therefore, we focus on the features that contribute most to model prediction.
Our study has several limitations. First, the model we developed and validated to distinguish ALN status was conducted in a single center with a limited sample size. Therefore, future studies should include a larger sample size and conduct multicenter external validation. Second, in our retrospective research, some ultrasound images lacked reproducibility because we did not standardize intensity and instead used default values. Third, mammography cannot capture all axillary lymph nodes (ALNs), and we were unable to match pathologically metastatic lymph nodes with suspicious ones on the ultrasound image. To improve diagnostic sensitivity, we will conduct a multicenter study and analyze the BI-RADS features, evaluate the quantitative value of enhancement in primary tumor and axillary lymph nodes, as well as the difference of intra- and extra-tumor. Finally, our study did not include comparisons with MRI or PET. We plan to collect additional data involving these modalities and compare the proposed model with these data. In addition, we leverage their features to enhance model performance.
Conclusion
In summary, incorporating interpretable machine learning models with identified clinical risk factors, CEM, and US signature shows potential as a noninvasive approach for predicting ALN metastasis in breast cancer. This could impact treatment strategies and aid in clinical decision-making.
ORCID iDs: Bowen Zheng https://orcid.org/0000-0001-8653-8060
Sina Wang https://orcid.org/0000-0002-2159-7921
Genggeng Qin https://orcid.org/0000-0002-7563-3924
Statements and Declarations
Ethical Considerations: This study was approved by the Institutional Review Board of the Nanfang Hospital, Southern Medical University (NFEC-2017-136).
Author Contributions/CRediT: Weimin Xu, Bowen Zheng, Genggeng Qin, Yingjia Li, and Weiguo Chen contribute to conception and design. Weimin Xu, Sina Wang, and Xin Liao contribute to acquisition of data. Weimin Xu,Chanjuan Wen, and Hui Zeng contribute to analysis and interpretation of data. Bowen Zheng and Zilong He contribute to build the machine learning model. Weimin Xu, Bowen Zheng,Weiguo Chen, and Yingjia Li participate in drafting the article. All authors give final approval of the version to be published.
Funding: The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Natural Science Foundation of China (82171929), the Natural Science Foundation of Guangdong Province, China (2024A1515011520), and Science and Technology Program of Guangzhou(2025A04J4117).
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Data Availability: The original data of the paper cannot be made public due to patient privacy concerns.
References
- 1.Bray F, Laversanne M, Sung H, et al. Global cancer statistics 2022: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin. 2024;74(3):229-263. [DOI] [PubMed] [Google Scholar]
- 2.Schröder L, Fricker R, Stein RG, et al. Evaluation of sentinel lymph node biopsy prior to axillary lymph node dissection: The role of isolated tumor cells/micrometastases and multifocality/multicentricity—a retrospective study of 1214 breast cancer patients. Arch Gynecol Obstet. 2018;297(6):1509-1515. [DOI] [PubMed] [Google Scholar]
- 3.McLaughlin SA, Wright MJ, Morris KT, et al. Prevalence of lymphedema in women with breast cancer 5 years after sentinel lymph node biopsy or axillary dissection: Patient perceptions and precautionary behaviors. J Clin Oncol. 2008;26(32):5220-5226. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Li J, Wang S-R, Li Q-L, et al. Diagnostic value of multiple ultrasound diagnostic techniques for axillary lymph node metastases in breast cancer: A systematic analysis and network meta-analysis. Front Oncol. 2023;12:1043185. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Alvarez S, Añorbe E, Alcorta P, et al. Role of sonography in the diagnosis of axillary lymph node metastases in breast cancer: A systematic review. Am J Roentgenol. 2006;186(5):1342-1348. [DOI] [PubMed] [Google Scholar]
- 6.Marino MA, Avendano D, Zapata P, et al. Lymph node imaging in patients with primary breast cancer: Concurrent diagnostic tools. Oncologist. 2020;25(2):e231-e242. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Tian Y, Zhao L, Gui Z, et al. Clinical and pathological features analysis of invasive breast cancer with microcalcification. Cancer Med. 2023;12(10):11351-11362. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Pálka I, Ormándi K, Gaál S, et al. Casting-type calcifications on the mammogram suggest a higher probability of early relapse and death among high-risk breast cancer patients. Acta Oncol (Madr). 2007;46(8):1178-1183. [DOI] [PubMed] [Google Scholar]
- 9.Valente SA, Levine GM, Silverstein MJ, et al. Accuracy of predicting axillary lymph node positivity by physical examination, mammography, ultrasonography, and magnetic resonance imaging. Ann Surg Oncol. 2012;19(6):1825-1830. [DOI] [PubMed] [Google Scholar]
- 10.Lee CH, Phillips J, Sung JS, et al. ACR BI-RADS® AtlasMammography Contrast Enhanced Mammography (CEM); A Supplement to ACR BI-RADS® Mammography 2013. American College of Radiology; 2022. [Google Scholar]
- 11.Gelardi F, Ragaini EM, Sollini M, et al. Contrast-Enhanced mammography versus breast magnetic resonance imaging: A systematic review and meta-analysis. Diagnostics. 2022;12(8):1890. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Lewin JM, Yaffe MJ. A history of contrast-enhanced mammography. In: Lobbes M, Jochelson MS, eds. Contrast-Enhanced Mammography. Springer International Publishing; 2019:1-21. [Google Scholar]
- 13.Houben IP, Vanwetswinkel S, Kalia V, et al. Contrast-enhanced spectral mammography in the evaluation of breast suspicious calcifications: Diagnostic accuracy and impact on surgical management. Acta Radiol. 2019;60(9):1110-1117. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Long R, Cao K, Cao M, et al. Improving the diagnostic accuracy of breast BI-RADS 4 microcalcification-only lesions using contrast-enhanced mammography. Clin Breast Cancer. 2021;21(3):256-262.e2. [DOI] [PubMed] [Google Scholar]
- 15.Mao N, Yin P, Li Q, et al. Radiomics nomogram of contrast-enhanced spectral mammography for prediction of axillary lymph node metastasis in breast cancer: A multicenter study. Eur Radiol. 2020;30(12):6732-6739. [DOI] [PubMed] [Google Scholar]
- 16.Lin F, Li Q, Wang Z, et al. Intratumoral and peritumoral radiomics for preoperatively predicting the axillary non-sentinel lymph node metastasis in breast cancer on the basis of contrast-enhanced mammography: A multicenter study. Br J Radiol. 2023;96(1143):20220068. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Erickson BJ, Korfiatis P, Akkus Z, et al. Machine learning for medical imaging. RadioGraphics. 2017;37(2):505-515. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Barragán-Montero A, Javaid U, Valdés G, et al. Artificial intelligence and machine learning for medical imaging: A technology review. Phys Med. 2021;83:242-256. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Hansen LK, Salamon P. Neural network ensembles. IEEE Trans Pattern Anal Mach Intell. 1990;12(10):993-1001. [Google Scholar]
- 20.Luo S-T, Cheng B-W. Diagnosing breast masses in digital mammography using feature selection and ensemble methods. J Med Syst. 2012;36(2):569-577. [DOI] [PubMed] [Google Scholar]
- 21.Xiao Y, Wu J, Lin Z, et al. A deep learning-based multi-model ensemble method for cancer prediction. Comput Methods Programs Biomed. 2018;153:1-9. [DOI] [PubMed] [Google Scholar]
- 22.Yurttakal AH, Erbay H, İkizceli T, et al. Diagnosing breast cancer tumors using stacked ensemble model. J Intell Fuzzy Syst. 2021;42(1):77-85. [Google Scholar]
- 23.Rodríguez-Pérez R, Bajorath J. Interpretation of compound activity predictions from Complex machine learning models using local approximations and shapley values. J Med Chem. 2020;63(16):8761-8777. [DOI] [PubMed] [Google Scholar]
- 24.Lundberg SM and Lee S-I. A unified approach to interpreting model predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS'17). Red Hook, NY: Curran Associates Inc.; 2017:4768–4777. [Google Scholar]
- 25.Collins GS, Reitsma JB, Altman DG, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): The TRIPOD statement. Br Med J. 2015;350:g7594. [DOI] [PubMed] [Google Scholar]
- 26.D’orsi C, Morris E, Mendelson E. ACR BI-RADS® Atlas, Breast Imaging Reporting and Data System. American College of Radiology; 2013. [Google Scholar]
- 27.Vrdoljak J, Krešo A, Kumrić M, et al. The role of AI in breast cancer lymph node classification: A comprehensive review. Cancers (Basel). 2023;15(8):2400. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Yang J, Wang T, Yang L, et al. Preoperative prediction of axillary lymph node metastasis in breast cancer using mammography-based radiomics method. Sci Rep. 2019;9(1):4429. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Tan H, Wu Y, Bao F, et al. Mammography-based radiomics nomogram: A potential biomarker to predict axillary lymph node metastasis in breast cancer. Br J Radiol. 2020;93(1111):20191019. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Akissue de Camargo Teixeira P, Chala LF, Shimizu C, et al. Axillary lymph node sonographic features and breast tumor characteristics as predictors of malignancy: A nomogram to predict risk. Ultrasound Med Biol. 2017;43(9):1837-1845. [DOI] [PubMed] [Google Scholar]
- 31.Cong Y, Wang S, Zou H, et al. Imaging predictors for nonsentinel lymph node metastases in breast cancer patients. Breast Care. 2020;15(4):372-379. [DOI] [PMC free article] [PubMed] [Google Scholar]