Development and External Validation of a Machine Learning Model to Predict Pathological Complete Response After Neoadjuvant Chemotherapy in Breast Cancer

Abstract

Purpose

Several predictive models have been developed to predict the pathological complete response (pCR) after neoadjuvant chemotherapy (NAC); however, few are broadly applicable owing to radiologic complexity and institution-specific clinical variables, and none have been externally validated. This study aimed to develop and externally validate a machine learning model that predicts pCR after NAC in patients with breast cancer using routinely collected clinical and demographic variables.

Methods

The electronic medical records of patients with advanced breast cancer who underwent NAC before surgical resection between January 2017 and December 2020 were reviewed. Patient data from Seoul National University Bundang Hospital were divided into training and internal validation cohorts. Five machine learning techniques, including gradient boosting machine (GBM), support vector machine, random forest, decision tree, and neural network, were used to build predictive models, and the area under the receiver operating characteristic curve (AUC) was compared to select the best model. Finally, the model was validated using an independent cohort from Seoul National University Hospital.

Results

A total of 1,003 patients were included in the study: 287, 71, and 645 in the training, internal validation, and external validation cohorts, respectively. Overall, 36.3% of the patients achieved pCR. Among the five machine learning models, the GBM showed the highest AUC for pCR prediction (AUC, 0.903; 95% confidence interval [CI], 0.833–0.972). External validation confirmed an AUC of 0.833 (95% CI, 0.800–0.865).

Conclusion

Commonly available clinical and demographic variables were used to develop a machine learning model for predicting pCR following NAC. External validation of the model demonstrated good discrimination power, indicating that routinely collected variables were sufficient to build a good prediction model.

INTRODUCTION

Neoadjuvant chemotherapy (NAC) allows unresectable tumors to become operable, whereas downstaging operable tumors allows breast conservation [1,2]. After NAC, pathological complete response (pCR), defined as no invasive nor in situ residuals in the breast and nodes, is associated with favorable outcomes compared with residual tumors [3,4,5]. For patients who have achieved pCR, less invasive surgical approaches, including breast-conserving surgery and de-escalation of axillary lymph node dissection, may be feasible while achieving comparable clinical outcomes.

Depending on the tumor subtype, approximately 20%–40% of patients receiving NAC achieve pCR; pCR rates are highest in human epidermal growth factor receptor 2 (HER2)-positive cancers and lowest in luminal A tumors [6,7]. Several predictive models have been developed to predict pCR [8,9,10,11,12,13]; however, few are broadly applicable because of radiologic complexity and variables that require additional tests. Moreover, most models focus on identifying patients who may benefit from NAC before initiating chemotherapy.

This study aimed to develop a machine learning model using simple variables to predict pCR after NAC. This model will help surgeons identify patients eligible for less-invasive surgical interventions and eventually reduce the treatment burden. We also aimed to externally validate the model using patient data from an independent hospital to demonstrate that this model could be generally applied to all patients with breast cancer who received NAC and were awaiting surgery.

METHODS

This multicenter retrospective study was performed at Seoul National University Bundang Hospital (SNUBH), a tertiary, academic, three-site hospital system. The study protocol was approved by the Institutional Review Boards of SNUBH and Seoul National University Hospital (SNUH), which waived the need for informed consent for the use of identifiable data.

Study cohort and data collection

Patients with histologically proven invasive breast carcinoma who underwent NAC followed by surgery between January 2017 and December 2019 at SNUBH and between January 2015 and December 2018 at SNUH were included in this study. The exclusion criteria were bilateral breast cancer and a personal history of breast cancer. A total of 358 patients from SNUBH and 645 patients from SNUH were included, and their clinical data were obtained from electronic medical systems.

Based on previous studies, clinical and pathological variables known to have a predictive value for NAC response in breast cancer were collected [8,9,13,14]. The clinical data included age, body mass index (BMI), clinical T stage, N stage before the initiation of NAC, and serum carbohydrate antigen 15-3 (CA15-3). Tumor size was defined as the largest tumor size measured on ultrasound or magnetic resonance imaging (MRI), and nodal staging was based on the results of physical examination, imaging tests (ultrasound, MRI, and chest computed tomography), and biopsy. cN0 was defined as negative for metastases on physical and radiological examinations. Pathological data on histological grade, estrogen receptor (ER), progesterone receptor (PR), HER2, and Ki-67 status were collected from prechemotherapy biopsy reports. The status of ER, PR, and HER2 was evaluated by immunohistochemistry and fluorescence in situ hybridization when needed. For ER and PR, ≥ 1% positive tumor cells with nuclear staining were considered positive. Finally, the mass size on post-NAC MRI was used to assess changes in tumor size after NAC [15]. Non-mass enhancement without definite features of the mass was not included in the tumor size.

Model development and validation

Before developing the prediction models, patients in SNUBH were randomly divided into training and test cohorts in a ratio of 80:20. The training set was used to develop prediction models using five machine learning algorithms: gradient boosting machine (GBM), support vector machine (SVM), random forest (RF), decision tree (DT), and neural network (NN). The outcome variable was pCR, and 10 variables selected from the literature review were used as predictors. As missing data were present in less than 2% of the total records, a simple imputation technique was used. Missing data were substituted with the mean for continuous variables and the median for categorical variables. Feature importance statistics were used to rank the variables; however, feature selection was not performed to include all the variables. To ensure the robustness of each machine learning method, a 10-fold cross-validation resampling technique was used for the training set, and the hyperparameters were optimized using a random search until the highest area under the receiver operating characteristic curve (AUC) and accuracy of each model were achieved [16].

Internal validation of each model with optimal hyperparameters was performed on the test set. The performances of the predictive models were evaluated using the AUC, accuracy, precision, recall, and F1 score. The AUCs of the different machine learning models were compared using DeLong’s test [17], and the cut-off values were calculated using the Youden index. Finally, the model with the highest AUC was selected and further tested using an external validation cohort from SNUH.

Statistical analysis

Qualitative data were presented as numbers and percentages, and quantitative data were presented as mean ± standard deviation. Student’s t-test, Fischer’s exact test, and the χ² test were used to compare continuous and discrete variables between the two groups. Statistical significance was set at p < 0.05. R software version 3.6.3 (The R Foundation for Statistical Computing, Vienna, Austria) and Python programming language (version 3.5.2; Python Software Foundation, Wilmington, USA) were used for statistical analyses and model development. The scikit-learn package was used for DT, RF, and SVM; XGBoost and CatBoost for GBM; and Keras and neuralnet for NN.

RESULTS

In total, 1,003 patients including 358 from SNUBH (287 in the training cohort and 71 in the internal validation cohort) and 645 from SNUH were included in our analyses. Overall, a pCR was observed in 348 (36.3%) patients. The incidence of pCR was 37.4% (134/358) for SNUBH and 33.2% (214/645) for SNUH. Patient demographic and clinicopathological characteristics in both the training and test sets are presented in Table 1.

Table 1. Patient characteristics of the training and test cohorts.

Characteristics		Training set (n = 287)	Test set (n = 71)	p-value
Age (yr)		49.7 ± 10.4	48.8 ± 9.9	0.507
BMI (kg/m2)		23.8 ± 3.7	23.6 ± 3.6	0.687
Clinical tumor stage*				0.607
	T1	24 (8.4)	5 (7.0)
	T2	184 (64.1)	43 (60.6)
	T3	57 (19.9)	19 (26.8)
	T4	22 (7.7)	4 (5.6)
Clinical node stage*				0.619
	N0	71 (24.7)	14 (19.7)
	N1	121 (42.2)	36 (50.7)
	N2	51 (17.8)	11 (15.5)
	N3	44 (15.3)	10 (14.1)
Histologic grade				0.527
	Low	6 (2.1)	1 (1.4)
	Intermediate	99 (35.2)	30 (42.3)
	High	176 (62.6)	40 (56.3)
ER status				0.737
	Negative	165 (57.5)	43 (60.6)
	Positive	122 (42.5)	28 (39.4)
HER2 status				0.440
	Negative	162 (57.4)	45 (63.4)
	Positive	120 (42.6)	26 (36.6)
Ki-67 index (%)		38.4 ± 18.5	39.4 ± 19.8	0.700
CA15-3 (U/mL)		14.2 ± 15.4	16.6 ± 15.8	0.245
Lesion size at post-NAC MRI (cm)		1.7 ± 1.8	1.6 ± 1.8	0.663

Data are presented as means ± standard deviations or number (%).

BMI = body mass index; ER = estrogen receptor; HER2 = human epidermal growth factor receptor 2; CA15-3 = carbohydrate antigen 15-3; NAC = neoadjuvant chemotherapy; MRI = magnetic resonance imaging.

^*Stratified according to the American Joint Committee on Cancer (AJCC) 7th TNM stage.

The associations between pCR and 10 variables were assessed (Table 2). Univariate analysis revealed that the factors that significantly affected the possibility of pCR were negative ER status, positive HER2 status, serum CA15-3 levels, and lesion size on post-NAC MRI. A variance importance plot for the GBM is shown in Figure 1. The lesion size on post-NAC MRI ranked first, followed by HER2, ER, and CA15-3 status.

Table 2. Univariate analysis of pathological complete response in relation to clinicopathological characteristics (all cohort).

Characteristics		Non-pCR (n = 224)	pCR (n = 134)	p-value
Age (yr)		49.4 ± 10.5	49.7 ± 10.0	0.812
BMI (kg/m2)		24.0 ± 3.9	23.5 ± 3.2	0.173
Clinical tumor stage*				0.194
	T1	15 (6.7)	14 (10.4)
	T2	138 (61.6)	89 (66.4)
	T3	51 (22.8)	25 (18.7)
	T4	20 (8.9)	6 (4.5)
Clinical node stage*				0.200
	N0	48 (21.4)	37 (27.6)
	N1	99 (44.2)	58 (43.3)
	N2	37 (16.5)	25 (18.7)
	N3	40 (17.9)	14 (10.4)
Histologic grade				0.104
	Low	6 (2.7)	1 (0.8)
	Intermediate	87 (39.7)	42 (31.6)
	High	126 (57.5)	90 (67.7)
ER status				< 0.001
	Negative	106 (47.3)	102 (76.1)
	Positive	118 (52.7)	32 (23.9)
HER2 status				< 0.001
	Negative	160 (72.7)	47 (35.3)
	Positive	60 (27.3)	86 (64.7)
Ki-67 index (%)		37.5 ± 18.8	40.4 ± 18.7	0.166
CA15-3 (U/mL)		17.0 ± 18.8	10.8 ± 5.1	< 0.001
Lesion size at post-NAC MRI (cm)		2.1 ± 1.7	0.9 ± 1.7	< 0.001

Data are means ± standard deviations or number (%).

pCR = pathological complete response; BMI = body mass index; ER = estrogen receptor; HER2 = human epidermal growth factor receptor 2; CA15-3 = carbohydrate antigen 15-3; NAC = neoadjuvant chemotherapy; MRI = magnetic resonance imaging.

^*Stratified according to the American Joint Committee on Cancer (AJCC) 7th TNM stage.

Figure 1. Variance importance plot of the gradient boosting machine model in predicting pathological complete response.

NAC = neoadjuvant chemotherapy; MRI = magnetic resonance imaging; HER2 = human epidermal growth factor receptor 2; ER = estrogen receptor; CA15-3 = carbohydrate antigen 15-3; BMI = body mass index.

Performance of machine learning models

The optimal hyperparameters found in a 10-fold cross-validation and the performance of each machine learning model in the test set are shown in Table 3 and Figure 2. GBM showed the largest AUC (0.903, 95% confidence interval [CI], 0.833–0.972) and the highest accuracy (81.7%) in the test cohort. The AUC of RF was 0.898 (95% CI, 0.824–0.972), SVM 0.801 (95% CI, 0.696–0.906), and NN 0.870 (95% CI, 0.746–0.956). DT showed the smallest AUC (0.758; 95% CI, 0.640–0.876) and SVM showed the lowest accuracy (71.8%). DeLong’s test on the AUC of each model compared to the AUC of GBM showed that the performance of GBM was better than that of RF, but the difference was insignificant. However, the GBM model was significantly more accurate than the DT, SVM, and NN models. The details of the model performance in terms of accuracy, precision, recall, and F1 score are provided in Supplementary Table 1.

Table 3. Optimal hyperparameters of each machine learning model and DeLong’s test comparing area under the receiver operating characteristic curve to gradient boosting machine in the test set.

Characteristics	Optimal hyperparameter	AUC	Accuracy	p-value
Decision tree	Maximum depth: 5	0.758	0.746	0.009
Support vector machine	Kernel: radial bias function	0.801	0.718	0.002
	C: 1.0
	Gamma: 0.05
Random forest	No. of trees: 150	0.898	0.803	0.406
	Maximum depth: 5
Neural network	No. of hidden layers: 3	0.870	0.746	0.037
	No. of noes in a layer: 5
Gradient boosting machine	No. of trees: 500	0.903	0.817	-
	Interaction depth: 3

AUC = area under the receiver operating characteristic curve.

Figure 2. Comparison of area under the receiver operating characteristic curves among the machine learning models to predict pathological complete response after neoadjuvant chemotherapy (1: gradient boosting machine, 2: random forest, 3: neural network, 4: support vector machine, 5: decision tree).

AUC = area under the receiver operating characteristic curve.

External validation of the final model

Internal validation of the test set demonstrated the superior performance of the GBM compared to that of other machine learning models. Therefore, we selected the GBM model to assess external generalizability and real-world performance. The external validation of the SNUH cohort is shown in Figure 3. Satisfactory results were obtained with an AUC of 0.833 (95% CI, 0.800–0.865), sensitivity of 72.8%, and specificity of 77.7% with a cut-off value of 0.318. The AUC was slightly lower than that of the internal validation set; however, the difference was not significant (p = 0.387).

Figure 3. External validation of the gradient boosting machine model to predict pathological complete response after neoadjuvant chemotherapy.

PV = predictive value; Sens = sensitivity; Spec = specificity.

DISCUSSION

We developed and compared the ability of five machine learning models to predict pCR after NAC in patients with breast cancer. Ten routinely available clinicopathological variables were used as predictive variables, and the GBM model resulted in the largest AUC of 0.903 and highest accuracy of 81.7%. External validation using patient data from an independent hospital demonstrates the generalizability of our prediction model.

Previous studies have attempted to predict pathological responses to NAC in patients with breast cancer. Several researchers proposed nomograms using preoperative clinicopathological variables (e.g., age, NAC cycles, ER, HER2, and status of lymphovascular invasion) or just simple laboratory indexes (e.g., clinical tumor stage, lymphocyte-to-monocyte ratio, fibrinogen level, and D-dimer level) to predict pCR after NAC [10,12]. The AUCs ranged from 0.77 to 0.90 [8,11,18], and these nomograms helped stratify patients who would benefit more or less from NAC. However, the small patient size limited the validation power, and the results of the external validation were not as good when tested in other institutions.

Recently, machine learning-based prediction models have been developed with better performance (AUCs ranged from 0.83 to 0.93) [14,19,20]. For example, Kim et al. [14] compared six machine learning algorithms (logistic regression, LASSO regression, SVM-linear, SVM-rbf, RF, and LightGBM) and demonstrated that LightGBM had the highest AUC for predicting pCR in patients with breast cancer treated with NAC, followed by curative surgery. Asri et al. [21] compared four machine learning algorithms (SVM, DT, naïve Bayes, and k-nearest neighbors) on the Wisconsin Breast Cancer dataset to assess the best algorithm for breast cancer risk prediction and diagnosis. Although the SVM was the best in terms of precision and error rate, all four types of algorithms showed an accuracy above 90%. Delen et al. [22] compared two data mining methods (artificial NN and DT) with a statistical method (logistic regression) to develop a prediction model for breast cancer survivability and demonstrated that artificial NN and DT performed better than logistic regression models. Machine learning algorithms can incorporate an unlimited number of input variables, and the optimal set of variables can be improved using various training methods. By including extensive clinicopathological variables that might have been omitted while developing logistic regression-based nomograms, the predictive performance of machine learning models tends to be better. However, machine learning models are more complicated in nature and difficult to apply in other institutes because of their complexity. Therefore, we intended to develop a prediction model using a machine learning algorithm to increase accuracy while using routinely available variables to maximize generalizability.

We performed a literature review to select clinical and demographic variables known to be associated with pCR. To create an accurate but generalizable model, we excluded institution-specific clinical or radiologic variables and finally selected 10 variables: age, BMI, clinical T and N stages, histologic grade, ER and HER2 status, Ki-67 index, CA15-3, and MRI lesion size after NAC. Previous studies have reported that age, BMI, tumor stage at diagnosis, histologic grade, tumor subtype, and Ki-67 expression are associated with pCR [8,13,23]. Most studies showed that molecular subtype was the most important predictive marker for pCR [24,25], and another study showed that patients with higher BMI were associated with higher tumor characteristics, such as hormone-negative status and higher TNM stage, and were less likely to achieve pCR after NAC [26]. The baseline CA15-3 level before NAC also correlated with pCR and was added to our model as an independent predictor [14]. Our study showed that HER2-positive and ER-negative cancers tended to have higher pCR rates. Breast cancer subtypes are categorized based on ER, PR, HER2, and Ki-67 expression, and chemotherapeutic agents are selected according to the cancer subtype. These results reflect a higher response to trastuzumab in HER2 positive subtype and a lower response to chemotherapy in hormone receptor-positive breast cancer cells. Feature importance in the GBM model showed that in addition to radiologic tumor size after NAC and tumor subtype, CEA 15-3 and BMI contributed more than other tumor characteristics in predicting pCR. Unfortunately, it is difficult to provide a statistical interpretation of the feature importance score, which is a limitation of this study.

To our knowledge, this is the first study to externally validate a machine learning model for predicting pCR following NAC. We demonstrated that when using machine learning algorithms, simple clinical variables were sufficient to create a good prediction model. With the aid of this prediction model, surgeons can perform more breast-conserving surgeries and avoid unnecessary wider excisions. However, to allow real-time predictions, the model should be deployed as a web service in the near future.

Our study had some limitations. Many studies have demonstrated that the response to NAC is largely dependent on the breast cancer subtype. Triple-negative or HER2-positive breast cancer is more sensitive to chemotherapy, with pCR rates of up to 70%, whereas luminal A breast cancer is less sensitive to chemotherapy and less likely to achieve pCR. Nonetheless, approximately half of our patients were of the luminal type, and the pCR rate was less than one-fourth, whereas more than half of patients with triple-negative or HER2-positive breast cancer achieved pCR. Although we included ER and HER2 statuses as predictive variables, it would be more useful to develop independent prediction models for different subtypes. Finally, our predictive model did not provide information on axillary pCR. However, previous studies have shown that breast pCR strongly correlates with axillary pCR [27]. Although this correlation is more evident in HER2-positive or triple-negative breast cancer than in hormone-positive breast cancer, axillary surgery may be de-escalated in patients with predicted breast pCR.

In conclusion, we demonstrated that when using machine learning algorithms, routinely available clinical and demographic variables were sufficient to predict pCR following NAC. This simple model enables surgeons to minimize the extent of surgery and allows for more breast-conserving surgery. External validation of the model using independent validation samples confirmed its good discriminatory power, demonstrating that this model is applicable in different clinical settings.

SUPPLEMENTARY MATERIAL

Supplementary Table 1

Comparison of machine learning models to predict pathologic complete response following neoadjuvant chemotherapy