Evolution of Breast Cancer Recurrence Risk Prediction: A Systematic Review of Statistical and Machine Learning-based Models

Abstract

Background:

Selection of appropriate adjuvant therapy to ultimately reduce the risk of breast cancer recurrence is a challenge for medical oncologists. Several automated risk prediction models have been developed using retrospective clinical data and have evolved significantly over the years in terms of predictors of recurrence, data usage, and predictive techniques (statistical/machine learning).

Methods:

Following PRISMA guidelines, we performed a systematic literature review of the aforementioned statistical and machine learning (ML) models published between January 2008 and December 2022 through searching 5 digital databases - PubMed, Science Direct, Scopus, Cochrane and Web of Science. The comprehensive search yielded a total of 163 papers and after a screening process focusing on papers that dealt exclusively with statistical/ML methods, only 23 papers were deemed appropriate for further analysis. We benchmarked the studies based on development, evaluation metrics, and validation strategy with an added emphasis on racial diversity of patients included in the studies.

Results:

30.4% of the included studies use statistical techniques while 69.6% are ML-based. Among these, traditional machine learning models (Support Vector Machines, decision tree, logistic regression, Naive Bayes) are the most frequently used (26.1%) along with deep learning (26.1%). Deep learning and ensemble learning provide the most accurate predictions (AUC = 0.94 each).

Conclusions:

Machine learning-based prediction models exhibit outstanding performance, yet their practical applicability might be hindered by limited interpretability and reduced generalization. Moreover, predictive models for breast cancer recurrence often focus on limited variables related to tumor, treatment, molecular, and clinical features. Imbalanced classes and the lack of open-source datasets impede model development and validation. Furthermore, existing models predominantly overlook African and Middle Eastern populations, as they are trained and validated mainly on Caucasian and Asian patients.

1. Introduction

Breast cancer (BC) is the most common cancer among women and is second in terms of cancer mortality [1]. Breast cancer mortality could potentially be reduced by providing physicians with precise risk prediction models to guide patient treatment and management pathways [2, 3]. Numerous statistical mod- els have been proposed to predict either risk of developing breast cancer [4], risk of carrying a genetic mutation [5], risk of recurrence or survival estimation after treatment [6]. These models are arguably limited since they rely on traditional hypotheses and associations between dependent clinical predictors, which may not fully capture the complexity and variability of breast cancer recurrence. As a result, the predictive accuracy and utility of such models may be constrained by their narrow focus on a limited set of variables that may not fully reflect the diverse range of factors influencing breast cancer recurrence. Recent advances in artificial intelligence (AI) have demonstrated the ability to utilize multimodal data which might be prognostic of future disease going beyond predictive models based on static and hand selected structured data elements [7]. In breast cancer research, machine learning (ML) models have been shown to provide accurate risk predictions using features engineered from comprehensive electronic healthcare records (EHR). Furthermore, recent ML-based models have shown incremental improvement in performance when also incorporating other forms of data, typically breast images and free-text clinical notes [8–10].

In this systematic review, we focused on benchmarking models for predicting BC recurrence as only few systematic reviews have focused on BC recurrence prediction over the recent years. These reviews either may have not been comprehensive in their inclusion and exclusion criteria or did not have depth in terms of describing types of AI models that have evolved over time. As an example, a review published in 2014, focuses solely on statistical models for predicting recurrence of early-stage breast cancer and published between 1982 and 2011 [6]. It provides an overview of the prognosticators incorporated in models and reports their clinical validity. AI-based models emerged starting from 2012. As such, the second review conducted in 2016 evaluates the performance of machine learning techniques applied to breast cancer recurrence prediction published between 1997 and 2014 [11]. However, the selection and exclusion criteria used may not have been comprehensive, which may have led to the omission of some prominent models. Equally important, new models have been developed thus necessitating an updated literature review [11]. The most recent survey from 2022 summarizes models published between 2011 and 2021 and provide an overview of AI techniques and the employed datasets in a descriptive way [12].

To the best of our knowledge, this review is the first to cover a large range of publication years (2008 – 2022) in a systematic way and includes both traditional and AI models. Interestingly, in clinical practice, physicians tend to use statistical models rather than machine learning due to the black box nature of the models (provide predictions without giving any insight into their internal functioning) [6]. We aim for this review not only to provide the reader a superficial view of models, but also analyze their evolution through years (the incorporation of new predictors and the improvement of model performance over time). Moreover, this review provides insight into patient disparities captured within the current modeling framework and provides insight toward required improvements to ameliorate generalizability and reduce known racial disparities reported in breast cancer outcomes. We separate the models into statistical-based and machine learning-based. The former often predict recurrence using Cox proportional hazard regression with traditional predictors: clinical and tumor-related factors. The latter (ML models) provide predictions based on deep learning, ensemble learning or traditional machine learning models. The ML models incorporate factors beyond traditional ones, such as psychological, demographic, and genomic signatures.

Benchmarking models, reporting underlying incorporated predictive features, and the targeted patient population could allow clinicians and researchers alike to have greater awareness as they consider clinical deployment or development of new models. This review offers a thorough and detailed overview of the existing models used to assess the risk of breast cancer recurrence. It provides a summary of the most notable models that have been published within the last fifteen years, and aims to address the essential goal of tracing the development and advancement of breast cancer recurrence risk prediction models.

2. Methods

Following PRISMA guidelines [13], we considered studies published between 2008 and 2022. We chose this timeframe to observe and analyze the progress of both traditional statistical and AI based predictive models through recent years.

To identify relevant studies, we utilized the Covidence software (Veritas Health Innovation, Melbourne, Australia) to manage our systematic review. Covidence offers various features to help researchers collaborate within a team for systematic review, screen and select relevant studies, and extract data in a streamlined and efficient manner. We examined five digital databases: PubMed, Science Direct, Scopus, Cochrane and Web of Science. During the first phase, we limited research to titles only (not articles’ key words or abstracts) and we queried each database using the following key words: (Breast cancer) AND (recurrence) AND (prediction OR risk assessment).

After removing duplicates, two authors (H.E.H and I.B) independently selected studies to be retained from screening based on relevancy using titles and abstracts. Thereupon, the screened studies were assessed based on full- texts. We included only original studies that satisfied all the inclusion criteria (e.g., full-text availability, original research, peer-reviewed). We primarily excluded comparative and validation studies as well as studies outside of our scope (see Appendix Fig. 1 for detailed exclusion criteria). In case of disagreement about eligibility, two other authors (B.N.P and A.S) independently reviewed the study to finally include or exclude it by consensus.

Afterwards, two appraisers (H.E.H and I.B) reviewed the included studies independently. For each model, we retrieved details on the statistical/ML techniques, the development process (feature selection, validation method) and the evaluation metrics. Moreover, we extracted characteristics of training and validation datasets (follow-up time of patients, distribution of recurrent and non-recurrent cases, incorporated predictors) with emphasis on patient population demographics (countries included, race, age), and we reported the availability of models as online risk calculators or nomograms. We additionally examined the progress of performance in the included studies.

3. Results

From a total of 163 papers, 23 were subjected to the second phase of review and comprehensive analysis. Table 1 contains all the references for the studies included. Appendix Fig. 1 encapsulates the systematic review process and succinctly delineates the specific inclusion and exclusion criteria that were employed at each stage of the PRISMA protocol. Without exception, all of the studies incorporated into the present review have prioritized the development of novel prediction models through the application of statistical or machine learning algorithms, as opposed to the validation of pre-existing models using independent datasets. The trends in publications per year according to the type (statistical/machine learning) are shown in Appendix Fig. 2. Machine learning resurgence started circa 2012 and continues to date. The healthcare field has witnessed a significant surge in the adoption of machine learning techniques, which has, in turn, inspired researchers to explore its applicability in predicting breast cancer recurrence. This trend has been prevalent for the past decade, from 2012 to 2022, during which numerous studies have been conducted to investigate the feasibility and effectiveness of machine learning-based approaches. How- ever, in 2022, statistical modeling has regained attention, with two notable studies utilizing competing-risks regression [14] and Cox regression [15]. We hypothesize that the resurgence in the use of statistical models in breast cancer prediction can be attributed to two key factors: interpretability and data size requirements. Firstly, traditional statistical models offer interpretable results, enabling researchers to gain insight into the intricate relationships between input variables and prediction outcomes. This interpretability holds significant importance in the medical domain, where a comprehensive understanding of the underlying factors influencing predictions is critical for informed decision- making and optimal patient care. Secondly, due to the limited availability of data for breast cancer recurrence prediction, statistical models prove to be effective in managing small datasets. A grouping of the employed techniques is summarized in Fig. 1a. From our analysis, we found machine learning mod- els to be more popular than statistical models in BC recurrence prediction despite arguably more difficulty in interpreting ML model results and in implementation compared to simple statistical models. In fact, 69.6% of studies in this review are ML-based while 30.4% use statistical techniques (see Fig. 1b). Traditional machine learning models (SVM, DT, LR, NB) are the most used (26.1%), along with deep learning (26.1%), then ensemble learning (13%) as illustrated by Fig. 1b.

Table 1.

Dataset details for each model

	Dataset
Study	Size	Median follow-up time (years)	Age range		Distribution	Source	Availability
[16]	586	12	39 88	-	-	Istituto Nazionale dei Tumori of Milan, Italy (INT)	-
[17]	1868	5.6	25 89	-	R:11%, NR:89%1	Memorial Sloan-Kettering Cancer Center	-
[18]	378	9	-		-	Central Pathology Office in Milan, Italy	-
[19]	679	8	-		R:71%, NR:29%	Korean tertiary teaching hospital	-
[20]	579	16	37 57	-	R:19%, NR:81%	Isfahan Sayed-o-Shohada cancer research center	-
[21]	679	7.1	21 83	-	R:29%, NR:71%	Korean tertiary teaching hospital	-
[22]	15314	6.09	19 98	-	R:12%, NR:88%	University of Texas MD Anderson Cancer Center	-
[23]	217	10	32 85	-	27.64%, 23.50%, 23.50%, 25.34%2	Omid Hospital of Mashhad	-
[34]	4735	9.8	57 71	-	R:7%, NR:93%	ATAC (Arimidex, Tamoxifen, Alone or in Combination) dataset	Dataset link
[24]	192	5	25 67	-	LR:22%, IR:64%, HR:14%3	Single institute in Korea	-
[25]	217	-	-		R:40%, NR:60%	Grade-A3 hospital in eastern China	-
[26]	6447	3.99	-		R:7%, NR:93%	Northwestern Memorial Hospital	-
[27]	320	6	30 84	-	LR:45%, IR:44%, HR: 11%	3 French hospitals: Besançon, Belfort and Dijon	-
[35]	1984	6	-		R:24%, NR:76%	UCI Machine Learning Repository	Dataset link
1R: Recurrent cases, NR: Non recurrent cases.
2Four classes distributed according to the time to recurrence (0–149 months).
3LR: Low risk of recurrence, IR: Intermediate risk, HR: High risk.
4The study used two datasets for model training, but we only considered the dataset on which the model performed the best.
	Dataset
Study	Size	Median follow-up time (years)	Age		Distribution	Source	Availability
[28]	575	7	-			Mizoram Cancer Institute	-
[29]	13117	6	43 55	-	R:9%, NR:91%1	Samsung Medical Center	-
[36]	127	8	-		R:25%, NR:75%	Cancer Hospital of the Chinese Academy of Medical Sciences	Dataset link
[30]	8956	7.46	41 67	-	-	Oncoshare breast cancer database from Stanford University	-
[31]	138	-	27 86	-	LR:60%, HR:40%2	Taipei Veterans General Hospital	-
[14]	3532	9	45 70	-	-	Five North American institutions	-
[32]	4757	4	20 70+	-	R:4%, NR:96%	Japanese Association for Theranostics	-
[15]	64044		15 85+	-		SEER database, U.S. National Cancer Institute	Dataset link
[33]	6486	4.75	20 90	-	R:8%, NR:92%	29 hospitals affiliated with the Chinese Society of Breast Surgery hospitals	-
1R: Recurrent cases, NR: Non recurrent cases.
2LR: Low risk of recurrence, HR: High risk.

Fig. 1.

(a) Techniques employed for the prediction of breast cancer recurrence; (b) Distribution of publications according to these techniques. Abbreviations: ML = Machine Learning; SVM = Support Vector Machines; DT = Decision Tree; LR = Logistic Regression; NB = Naive Bayes

3.1. Benchmarking models based on developmental datasets

We analyzed the training datasets used with a particular focus on countries of origin for the samples and recurrence predictors used. Table 1 summarizes details in terms of size (number of patients), follow-up time, age range, distribution of recurrent cases, source of dataset, as well as online availability. From the 23 reviewed works, the majority used non-public databases that are created in partnership with specialized breast cancer centers in different countries making such studies difficult to benchmark according to performance [14, 16–33]. Only 4 studies used publicly available datasets [15, 34–36] - (i) Wisconsin prognostic breast cancer (WPBC) dataset, University of California [35], (ii) Surveillance, Epidemiology, and End Results (SEER) database, U.S. National Cancer Institute [15], (iii) Arimidex, Tamoxifen, Alone or in Combi- nation (ATAC) dataset [34] and (iv) dataset shared by the Cancer Hospital of the Chinese Academy of Medical Sciences [36]. The majority of the datasets reported a median follow-up time of no less than four years subsequent to the diagnosis of breast cancer. Notably, most of the datasets exhibited a substantial degree of class imbalance, characterized by an uneven distribution of cases between the ‘Recurrent’ and ‘Non-recurrent’ classes (see Table 1).

After analyzing the datasets, we provided a categorization of the candidate predictors into 8 main categories (Appendix Table 1). Then, in accordance with this categorization, we represented the predictors considered by each study in Table 2. Several predictors have been found to be associated with an elevated risk of recurrence; however, the strength and significance of these predictors differed across the studies included in the analysis. It remains unclear as to what extent the combination of multiple factors might improve the accuracy of recurrence risk prediction.

Table 2.

Incorporated recurrence predictors

		Recurrence predictors
Study	Data format	Tumor-related	Molecular	Clinical	Demographic	Treatment-related	Lifestyle and psychological	Genomic	Cell nuclei-related
[16]	Tabular	✓	✓	✓		✓
[17]	Tabular	✓	✓	✓	✓	✓
[18]	Tabular	✓	✓	✓		✓
[19]	Tabular	✓	✓	✓		✓
[20]	Tabular	✓	✓	✓	✓	✓
[21]	Tabular	✓	✓
[22]	Tabular	✓	✓	✓	✓	✓	✓
[23]	Tabular	✓	✓	✓	✓	✓	✓
[34]	Tabular	✓	✓	✓		✓
[24]	Tabular	✓	✓	✓		✓
[25]	Tabular	✓	✓	✓	✓	✓	✓
[26]	Free-text notes			✓	✓		✓
[27]	Tabular	✓	✓	✓		✓
[35]	Tabular	✓	✓			✓			✓
[28]	Tabular	✓	✓	✓	✓	✓
		Incorporated recurrence predictors
Study	Data format	Tumor-related	Molecular	Clinical	Demographic	Treatment-related	Lifestyle and psychological	Genomic	Cell nuclei-related
[29]	Tabular	✓	✓	✓	✓	✓
[36]	Tabular + Images	✓	✓	✓
[30]	Free-text notes	✓	✓	✓	✓
[31]	Images	✓
[14]	Tabular	✓	✓	✓		✓
[32]	Tabular	✓	✓	✓		✓		✓
[15]	Tabular	✓	✓	✓	✓	✓
[33]	Tabular	✓	✓	✓		✓

According to our review, the majority of predictive models have been developed using either US or Asian population (Fig. 2). Such training data reflects a selection bias for the recurrence prediction models. Such selection bias could potentially affect the clinical adoption of the models since the model performance outside these countries could be poor. None of these studies report their performance outside the countries of the training dataset.

Fig. 2.

Heat map of countries based on nationality of women included in training and validation sets. From dark to light, the color gradient indicates the importance of the number of studies published per country

A detailed analysis of the 23 statistical and ML models that were included is summarized in Table 3. Each model targets different patients based on the type and primary treatment of breast cancer. Hence, we specify the target patients and we report the employed statistical/ML techniques, the validation method as well as the availability of the model (online risk calculator or nomogram). In some articles, the authors of the concerned studies did not explicitly indicate any specific category of breast cancer patients, suggesting that the proposed model is potentially applicable to different categories. The following subsections (3.1.1 and 3.1.2) divide studies into statistical and machine learning-based, respectively.

Table 3.

A concise summary of the 23 recurrence risk prediction models ranked by year

Study	Year	Authors	Targeted patients	ML/Statistical techniques	Validation	Risk calculator (Online/Nomogram)
[16]	2008	Boracchi et al.	Node-positive, estrogen–receptor-positive breast cancer inpostmenopausal women	Piecewise regression model	External on a cohort of 175 patients from the General Hospital of Venice	Nomogram [16]
[17]	2010	Rudloff et al.	Ductal carcinoma in situ for patients having undergone breast-conserving therapy	Cox proportional hazard multivariable regression	Internal (bootstrap resampling)	Nomogram [17]
[18]	2012	Dellapasqua et al.	Lymph node-negative endocrine responsive breast cancer in premenopausal patients	Cox proportional hazard multivariable regression + Kaplane-Meier method	External on a cohort of 1005 patients from a single institution	Nomogram [18]
[19]	2012	Kim et al.	Patients who had undergone surgery	SVM + feature selection based on Kaplan-Meier analysis and univariate Cox regression	Internal	Online risk calculator
[20]	2016	Mohebian et al.	-	Ensemble learning (Bagged Decision Tree) + Statistical Feature Selection methods + Particle Swarm Optimization (PSO)	Internal (holdout and 4-fold crossvalidation)	Online risk calculator
[21]	2016	Kim et al.	Patients who had undergone surgery	Naïve Bayesian classifier + Pearson chi-square test and univariate logistic regression for features selection	Internal	Nomogram [21]
[22]	2016	Wu et al.	Stage I to III invasive primary breast cancer	Backward stepwise Cox proportional hazards regression	Internal	-
[23]	2018	Vazifehdan et al.	-	C4.5 classifier + Combination of Bayesian network with tensor factorization to impute missing data	External on Wisconsin (569 cases) and Cleveland (303 cases) datasets	-
[34]	2018	Dowsett et al.	Estrogen Receptor–Positive breast cancer treated With 5 years of Endocrine Therapy	Cox proportional hazard multivariable regression + Kaplane-Meier method	External on ‘BIG 1–98’ dataset of 6,711 patients	-
[24]	2019	Yoo et al.	Hormone Receptor-positive, lymph node-negative early breast cancer	Multivariable logistic regression	External on a dataset of 264 patients of a different time period	Nomogram [24]
[25]	2020	Gu et al.	-	Ensemble learning (XGBoost) + Case-based reasoning (CBR)	Internal (10-fold cross-validation)	-
[26]	2020	Wang et al.	-	Conventional machine learning classifiers and Knowledge-guided Convolutional Neural Network (K-CNN)	Internal (5-fold cross-validation)	-
Study	Year	Authors	Patients targeted	ML/Statistical techniques	Validation	Risk calculator (Online/Nomogram)
[27]	2020	Baltres et al.	Estrogen receptor-HER2-negative	Deep multi-layer perceptrons (DMLP) + t-SNE data visualization algorithm for feature selection	Internal	-
[35]	2021	Alwohaibi et al.	Invasive breast cancer	SVM + Statistical Feature Selection methods + Brain Storming Optimization algorithm (BSO)	Internal (stratified 10-fold cross-validation)	-
[28]	2021	Dawngliani et al.	-	Ensemble learning (voting: J48 combined with Naϊve Bayes)	Internal (10-fold cross-validation)	-
[29]	2021	Kim et al.	Adjuvant breast cancer	Deep Learning-Based Survival Algorithm (Weibull Time To Event Recurrent Neural Network (WTTE-RNN))	Internal	-
[36]	2021	Yang et al.	HER2-positive breast cancer	CNN + multimodal model	External on the Cancer Genome Atlas dataset (TCGA) of 123 patients	-
[30]	2021	Sanyal et al.	-	Recurrent neural network (RNN) with long short-term memory (LSTM)	External on a dataset of 224 patients	-
[31]	2021	Phan et al.	-	CNN (Xception model) + Laplacian algorithm for detecting blurry images	Internal	-
[14]	2022	Sittenfeld et al.	Pathologic stage T1–2N1M0 breast cancer	Competing-risks regression	Internal	Online risk calculator
[32]	2022	Osako et al.	Patients who underwent SN (Sentinel lymph node) biopsy	Logistic regression + optimization based on the Akaike information criterion	Internal (10-fold cross-validation)	-
[15]	2022	Wang et al.	Invasive hormone receptor positive breast cancer	Model in [34] based on Cox regression + Oncotype DX1 recurrence score (RS)	Internal	Nomogram [15]
[33]	2022	Xin et al.	Early-stage invasive breast cancer with low-positive HER2	Random forest model + feature selection based on Kaplan-Meier analysis and univariate cox regression	Internal	-

¹Oncotype DX: a clinical tool (blood test) based on genetic profile for the estimation of BC recurrence risk score [37].

3.1.1. Statistical models

In this category, we included the models that applied linear mathematical function to model the relationship between independent predictor and dependent variables. The majority of the statistical models utilize Cox proportional hazards regression as their basis [15, 17, 18, 22, 34], which requires the proportional hazards (PH) assumption to hold true for various covariates. The studies mainly focus on investigating the variation in the hazard function of breast cancer recurrence over time. As a result, they determine the timeline for breast cancer recurrence probability and develop a comprehensive risk profile over time.

Among the remaining models, there are two noteworthy ones: one utilizes piecewise regression [16], and the other employs competing-risks regression [14]. Piecewise regression and competing-risks regression are two statistical techniques that can be used to model complex relationships between variables in a dataset. Piecewise regression involves dividing the dataset into different segments and fitting a separate regression model to each segment, providing greater flexibility in modeling [38]. On the other hand, competing-risks regres- sion accounts for the possibility of multiple outcomes that compete with each other, such as death from different causes, and estimates the probabilities of each outcome while examining the effects of different covariates on these probabilities [39].

Four studies propose nomograms as risk calculators [15–18]. Nomograms have emerged as a predictive tool for assessing the likelihood of a particular outcome. In the context of cancer prognosis, these tools have proven particularly useful. Recent research endeavors have sought to further enhance the clinical utility of nomograms by developing online risk calculators. Sittenfeld et al. put forth such a calculator with the aim of providing clinicians and patients alike with an accessible and accurate means of evaluating BC recurrence risk [14].

3.1.2. Machine learning-based models

In the machine learning category, we grouped the models that applied linear/non-linear learning strategy from input features to predict the dependent variables. The majority of studies include an internal validation procedure [14, 15, 17, 19–22, 25–29, 31–33, 35], often utilizing k-fold cross-validation, in which the same dataset is divided into separate partitions for both training and validation purposes (see Table 3). Frequently, such models could be overfitted to the training data, meaning that they may perform well on the training data but fail to generalize effectively to new, unseen data. Only seven studies have engaged in external validation of the model they have developed [16, 18, 23, 24, 30, 34, 36]. Three models are accessible in the form of web- based risk calculators, which can be used by both physicians and patients [14, 19, 20]. Two models are introduced as nomograms that are exclusively designated for utilization by physicians [21, 24]. Machine learning techniques are heterogeneous (Fig. 1b) but there was a predominance of traditional ML models [19, 21, 23, 24, 32, 33, 35]. Three types of ML-based studies are dis- cussed separately in the following subsections: traditional models, ensemble models, and deep learning models.

Traditional machine learning methods

Four traditional machine learning techniques are used for recurrence pre- diction, viz., SVM [19, 35], logistic regression [24, 32], decision tree [23] and Naive Bayesian classifier [21]. SVM and logistic regression were the most commonly applied. They both show good accuracy and precision in prediction (AUC no less than 0.85). Naive Bayesian classifier and decision tree were used in only two studies. However, it’s worth noting that decision trees are efficient and characterized by their ease of interpretability, which can be considered extremely important for clinicians as they may increase trust-ability of the model.

Ensemble learning methods

In an effort to enhance breast cancer recurrence prediction, researchers attempted various combinations of machine learning techniques to harness the potential of complementary models. Mohebian et al. use bagged decision tree [20]. Gu el al. use XGBoost combined with Case-based reasoning [25]. Dawngliani et al. use majority voting and the optimal model is the one combining J48 with Naive Bayes [28]. Finally, after evaluating different ML models, Xin et al. opt for random forest due to its effectiveness in mitigating errors in datasets with imbalanced data [33].

Deep learning methods

We grouped the neural network models with multiple non-linear activation layers as deep learning methods. Deep learning for BC recurrence emerged in 2020. Primarily, such models consider histopathology images or electronic health records (EHR). For EHR, two studies use natural language process- ing (NLP) techniques to analyze free-text EHR data [26, 30]. Two other studies use neural networks to analyze tabular EHR [27, 29]. For histopathology images, the studies primarily apply end-to-end 2D convolutional neural networks (CNN) [31, 36].

3.2. Performance evolution of models over years

The included studies have reported different performance measures as shown in Appendix Fig. 3a and Appendix Fig. 3b. The area under the receiver-operating characteristic curve (AUC) is the most frequently mentioned metric for evaluating the performance of machine learning models. It is a comprehensive measure of accuracy that takes into account the balance between sensitivity and specificity. The c-index is the most commonly used metric for evaluating the performance of statistical-based models, and it measures the probability that a patient who survived longer would be predicted to have a longer survival time by the model, compared to a randomly selected patient [40]. The AUC across ML-based studies ranged between 0.75 and 0.94 while for statistical models, an average c-index of 0.70 is reached by the majority of studies.

Fig. 3a displays the progress of machine learning-based studies from 2008 to 2022 according to the reported AUC. Two studies were excluded from the analysis because they did not provide information on the AUC metric [23, 35]. Similarly, Fig. 3b displays the evolution of statistical-based studies in terms of C-index. Concerning ML models, the highest AUC was achieved by Gu et al. [25] who used ensemble learning with case-based reasoning and Sanyal et al. [30] using deep learning (weak supervised learning on free-text electronic health records). As for statistical models, Wu et al. [22] reported the highest c-index (0.81) using backward stepwise Cox proportional hazards regression. Along with classical predictors of recurrence (tumor-related, molecular, clinical, demographic, and treatment-related), lifestyle and psychological data form a new category of predictors considered by the three aforementioned studies.

Fig. 3.

Performance evolution: (a) of Machine Learning-based studies over years; (b) of Statistical modeling-based studies over years

4. Discussion

Breast cancer recurrence can occur months or years after the initial diagnosis and treatment, leading to increased morbidity and mortality [41]. The cancer may recur on the same side of the breast as the original cancer (local recurrence), or it may spread to other areas of the body beyond the breast such as the bones, liver, lungs or brain (metastatic disease) [42]. Early prediction of breast cancer recurrence may help inform a more individualized treatment plan and could also inform patients about their future risks, which may guide their life decisions. With the advent of new digital technologies, several predictive models have been proposed to aid breast cancer diagnosis and treatment. However, the accurate prediction of a disease outcome (e.g., BC recurrence) is one of the most challenging tasks since it should discover and identify patterns and relationships between the current and historic EHR to provide precise predictions [43]. When these models effectively predict future outcomes in clinical care, they will contribute to advancing individualized, precision care-based cancer treatment.

In this study, we performed a systematic review of all the prominent auto- mated breast cancer recurrence prediction models published between 2008 – 2022. The models used both traditional statistical and complex deep learning techniques, and often produced performance > 0.9 AUC on the internal validation set. The existing studies, however, either use cross-validation or random splitting of the dataset for internal validation, which makes it difficult to draw a ‘true’ comparative performance. Knowing this limits the confidence on the validation results. Given the growing number of studies, we find that there is an urgent need for standardized validation of these models not only to rank the comparative performance on the same population, but to also comprehend the importance of the heterogeneous predictors (e.g., tabular EHR, clinical notes, radiology and pathology images) and inform their further use in clinical practice. Significantly, recurrence prediction models are generally trained and validated only on Caucasian and Asian populations and largely ignore African and Middle Eastern patients. This may introduce the selection bias and the models may fail to generalize outside the US, Europe and Asia.

Concerning predictors, tumor-related, treatment-related, molecular and clinical variables (especially cancer detection mode and family history of BC) were considered by the majority of predictive models and produced high pre- diction performance (see Appendix Table 1). Thus, these factors likely contain prognostic information for recurrence of breast cancer. We believe that socio-economic and behavioral type variables (including a wide area of social determinants of health) are largely being ignored by the current breast cancer prognosis modeling – primarily due to sparse recording of such data in EHR. However, they may have a major impact in improving the performance of prognosis models. In current days, such variables are gradually being discovered and recorded in EHR (mostly in non-regular fashion), such as education level, living in a disadvantaged area, unemployment, experiencing racial prejudice, having strong social connections and support, immigrant status, poor housing conditions, stress, smoking, insurance coverage and proximity to healthcare services [44]. Given this fact, future models should incorporate both clinical and disease related factors as well as socioeconomic factors to improve the prediction performance and generalizability.

As for prediction performance, in machine learning studies, the AUC ranged between 0.75 and 0.94, and ensemble learning had significantly improved the performance. Currently, ensemble learning and deep learning reported the highest AUC (0.94 each) [30, 45].

For Statistical-based studies, the discriminatory index was generally around 0.70 but in 2016, it increased to 0.81 [34]. Although we noticed that some models were using several highly correlated predictors and the large number of correlated candidate predictors may affect model performance. Therefore, candidate predictor selection is particularly important while maintaining pre- diction quality. A reduction in the number of predictors can reduce the fitting and complexity of the model as well as will help clinicians interpret and under- stand the model. However, studies often did not report feature selection and curation process.

Generally, prediction models based on machine learning show excellent performance, but may be limited in their application due to lack of interpretability and decreased generalization. Practically speaking, the selected predictors must be appropriate according to previously established clinical knowledge. Expectedly, variables that are selected purely by statistical or machine learning methods without a clear clinical context are not easily accepted by the clinicians. This may also play a role in why prediction models based purely on statistical modeling or machine learning are not widely used, despite what is shown by their superior prediction performance. Currently, the clinical tool that appears to have been embraced by clinicians is Oncotype Dx (Genomic Health, Redwood, CA) [37]. It is a genetic profile-based blood test that was first introduced in 2004 [46] and later validated by a number of research studies [47, 48]. The test, however, is prohibitively expensive and may not be eligible for large-scale BC patients since it is suggested based on the patient’s insurance type and availability of the test.

Limitations.

Our study has several limitations. First, in order to provide an in-depth analysis, our survey only considered the studies published between 2008 – 2022 and searched 5 digital databases. However, the prominent machine learning-based studies on the topic barely began in the late 2000s and these 5 digital databases are among the most noteworthy of digital libraries. In addition, the analysis is primarily focused on novel automated statistical and ML models for breast cancer recurrence risk prediction, thus we didn’t include investigation that pertained to novel biomarkers since it does not reflect the real contribution of the statistical and ML models. Thus, it is out of the scope of this review. Then, our comparative performance analysis is based on the reported performance on different internal datasets which may not be the ‘true’ performance of the models. Given the limited availability of open-source models and data, the models were not verified using the same test set. The purpose of this study was not to validate the models on external datasets, but we suggest that this is a promising area for future research. Finally, we recognize clinical tools for BC recurrence prediction (e.g., Oncotype Dx) are available yet not referenced as these tools do not explicitly include machine learning usage. Our future will acknowledge these clinical tools and next steps will compare performances between these tools and AI models to see if they are complementary or not.

Supplementary Material

PV - Appendix Figure 2

Appendix Fig. 2 Trends in publications by year according to the type (statistical/machine learning

PV - Appendix Figure 3

Appendix Fig. 3 Distribution of performance measures: (a) reported by Machine Learning-based studies; (b) reported by Statistical modeling-based studies.

PV - Appendix Figure 1

Appendix Fig. 1 PRISMA flow diagram for article selection; n represents the number of studies included in each phase

CONTEXT SUMMARY.

Key objective:

Tracing the development and advancement of breast cancer recurrence risk prediction models published between 2008 and 2022 in a systematic way that includes both statistical and AI models.

Knowledge generated:

Predictive models utilizing easily accessible, low-cost data, such as clinical, pathological, and demographic information, are promising tools that may address the limited availability of expensive tests like Oncotype DX for breast cancer recurrence prediction, particularly in low-income countries.

Relevance:

This review article serves as a guide for practitioners, offering valuable insights on how to effectively incorporate these tools into clinical practice. Moreover, it provides researchers with valuable feedback and recommendations to further improve and refine the predictive models.