Abstract
Background
Breast cancer (BC) surveillance primarily aims to detect locoregional recurrence (LRR) and contralateral breast cancer (CBC), but current strategies are largely one-size-fits-all and not tailored to individual risks. The Dutch INFLUENCE 3.0 prediction model provides personalised 5-year LRR and CBC risks to support shared decision-making on surveillance. In this study, INFLUENCE 3.0 is externally validated in a Belgian population.
Methods
All female BC patients curatively treated between 2000 and 2018 at the University Hospitals of Leuven were included. Calibration was assessed using the Integrated Calibration Index (ICI), E50, E90 metrics, and calibration plots. Discrimination was evaluated by calculating the Area Under the Curve (AUC) for each year, based on cumulative events up to that year.
Results
The final Leuven cohort included 8548 patients, of whom 7549 were not treated with neoadjuvant systemic therapy (non-NST) and 999 were (NST). Calibration metrics were generally acceptable, with ICI and E50 values below 0.01 in most subgroups. Calibration plots showed overestimation of predicted risks, particularly at lower predicted risk levels. Discrimination for LRR risk was good to very good (AUC 0.72–0.86 in the non-NST group; >0.82 in the NST group). For CBC risk, discrimination was poor in the non-NST group (AUC ≈ 0.5) and moderate-to-very good in the NST group (AUC 0.68–0.88).
Conclusion
INFLUENCE 3.0 showed acceptable calibration and discrimination for LRR risk, and although CBC risk discrimination in the non-NST group was poor. The overall risk estimates – primarily the LRR risks – may still help in guiding tailored surveillance schemes.
Keywords: Breast cancer, Prediction, Recurrence, Contralateral breast cancer, External validation
Graphical abstract
Highlights
-
•
Breast cancer follow-up is currently not tailored to individual risks.
-
•
INFLUENCE3.0 predicts 5-year locoregional and contralateral breast cancer recurrence.
-
•
INFLUENCE supports decision-making on personalised breast cancer follow-up.
-
•
INFLUENCE shows strong external validation for estimation of locoregional recurrence in Belgian data.
-
•
Discrimination was poor for estimation of contralateral breast cancer.
1. Introduction
In 2023, approximately 11,600 women in Belgium were diagnosed with invasive breast cancer (BC), most with early-stage disease [1]. Following curative treatment, patients enter surveillance primarily aimed at early detection of locoregional recurrence (LRR) and contralateral breast cancer (CBC). European guidelines recommend at least five years of follow-up with annual mammography and regular clinical visits [2]. Similarly, American guidelines recommend annual mammography and regular clinical visits for 3–10 years after treatment, with the duration of follow-up varying according to risk categories largely defined by tumor stage and subtype [3]. However, patients have substantial variation in recurrence risk related to additional factors such as age, tumor size, receptor subtype and treatment characteristics. With rising BC incidence and improved survival, this one-size-fits-all approach increases burden on healthcare and patient stress, while only a minority of recurrences are detected through scheduled surveillance [[4], [5], [6], [7], [8]]. A more risk-based personalised follow-up strategy to improve efficiency and reduce unnecessary burden for healthcare systems and patients is needed [[7], [8], [9]].
This demand led to the development of the Dutch prediction model INFLUENCE in 2015, subsequently updated to INFLUENCE 2.0 and most recently to INFLUENCE 3.0 [4,10,11]. INFLUENCE 3.0 provides individualised yearly risks of LRR and CBC within five years after surgery. In contrast to earlier versions, it is also applicable to patients treated with neoadjuvant systemic therapy (NST) and those with T4 disease. The model incorporates a wide range of clinicopathologic and treatment-related variables including age, mode of detection, histology, sublocalisation, grade, pT, pN, hormonal receptor status ± endocrine treatment, HER2 status ± targeted treatment, surgery ± immediate reconstruction ± radiation therapy, chemotherapy, presence of ductal carcinoma in situ, axillary lymph node dissection and pathologic complete response (in case of NST). It is available as a CE-certified online tool via the Evidencio platform [12]. INFLUENCE supports shared decision-making on personalised surveillance and is currently used in clinical practice within the multicenter NABOR study evaluating the (cost-)effectiveness of personalised follow-up [13]. The INFLUENCE 3.0 model was previously internally validated on the Dutch BC population and demonstrated adequate predictive performance [8].
Before broader clinical implementation, INFLUENCE 3.0 requires external validation in independent populations [[14], [15], [16], [17]]. Therefore, the objective of this study is to externally validate INFLUENCE 3.0 on patient data from the University Hospitals Leuven, Belgium. The results provide insight into the robustness of INFLUENCE 3.0 and thus will indicate whether the model could be suitable for broader use across different populations and healthcare settings.
2. Methods
2.1. Ethical considerations
This study was approved by the Ethics Committees of the University of Twente (240722) and the University Hospitals Leuven (S68978), with a non-commercial Data Transfer Agreement in place.
2.2. Data acquisition
Data were obtained from the University Hospitals Leuven, a tertiary referral centre in Belgium, where clinical and treatment data, including follow-up information, are prospectively collected through the multidisciplinary breast cancer centre. Patients with breast cancer treated at the University Hospitals Leuven continue their follow-up at our institution for up to 10 years after treatment. During the first five years, follow-up consists of biannual clinical evaluations, annual mammography, and CA 15-3 (tumor marker) testing. During the subsequent five years, clinical examinations are performed every nine months, while mammography and CA 15-3 testing are performed every 18 months. This comprehensive follow-up protocol resulted in extensive treatment and follow-up data with limited missingness. Female patients with unilateral BC treated with curative intent between 2000 and 2018 were included, yielding 8742 patients. Data were pseudonymized and securely transferred to the University of Twente. Exclusion criteria from INFLUENCE 3.0 development were applied (synchronous BC, positive surgical margins, unknown surgery date or incomplete follow-up).
2.3. Outcomes and definitions
“The primary outcomes were LRR and CBC, which were defined as reappearance of breast cancer on the ipsilateral side or regional nodes (LRR) or the contralateral breast (CBC) [18].” Individual follow-up was calculated from definitive surgery to the first event, distant metastasis, death, or last observation. Events occurring within 30 days were counted jointly; otherwise, only the first event was considered. Patients were censored at the time another event occurred (distant metastasis or death), with these events treated as competing events [19], and follow-up was truncated at five years.
Missing data were assumed missing at random and imputed five times using the mice package in R (Appendix A) [20].
2.4. Validation
For each imputed dataset, quarterly individual LRR and CBC risks were estimated using the appropriate INFLUENCE 3.0 submodel. Calibration and discrimination of the four submodels (LRR and CBC, analyzed separately for patients with and without NST (hereafter referred to as NST and non-NST) were evaluated and compared with internal validation results.
Calibration was assessed using the Integrated Calibration Index (ICI) (weighted difference between predicted and observed probabilities), E50 (median absolute difference), and E90 (90th percentile of the absolute difference), calculated annually and compared against internal validation values [10]. Lower values indicate better calibration, with ICI <0.01 considered good. Calibration was assessed using a Cox model with three restricted cubic splines and complementary log–log–transformed predicted risks, following Austin et al. [21].
In addition, calibration plots were generated at five years using smoothed curves with bootstrap CI's and decile-based Kaplan–Meier estimates [17,22]. Discrimination was assessed annually using time-dependent area under the curves (AUCs) at 1, 2, 3, 4 and 5 years from date of definitive surgery, and were classified as poor (0.5–0.6), moderate (0.6-0.7), good (0.7–0.8), or very good (≥0.8), with corresponding CIs [23].
Results from the five imputed datasets were pooled using Rubin's Rules [24,25].
3. Results
3.1. Patient characteristics
After the exclusion of several patients (Fig. 1), the final cohort consisted of 8548 patients: 7549 who did not receive neoadjuvant systemic therapy (non-NST group) and 999 who did (NST group). Baseline and treatment-related characteristics for both groups are summarized in Table 1.
Fig. 1.
Flowchart of patient selection and exclusion
* Decided in consultation with a clinician. Among these 18 cases, most had intervals much longer than 3 months (e.g. years) between BCS and mastectomy, suggesting prior breast cancer. Also, irradical margins may cause mastectomy after BCS is performed, but this should occur shortly after the initial BCS. ** Decided in consultation with a clinician. A lead time exceeding 400 days was considered implausible for both non-NST and NST patients. *** Initially, approximately 300 patients had irradical surgery. In the INFLUENCE 3.0 development, patients with focally irradical margins and those receiving re-intervention resulting in clean margins were included. A random sample of 50 patients was reviewed; only 2 had more than focal irradical margins without appropriate re-intervention. Based on this, and in agreement with a clinician, the remaining 250 patients were assumed to have either focal irradicality or adequate re-intervention. **** Second primary breast tumour diagnosed within 90 days of the first.
Abbreviations: LRR = locoregional recurrence, CBC = contralateral breast cancer, NST = neoadjuvant systemic treatment, BCS = breast conserving surgery, p = pathological (post-operative), c = clinical (pre-treatment), T = tumour, N = nodal, is = in situ, x = unknown/could not be assessed.
Table 1.
Overview of patient, tumour and treatment characteristics in the Leuven cohort divided into two groups: patients treated without neoadjuvant systemic treatment (non-NST) and patients treated with neoadjuvant systemic treatment (NST).
| No neoadjuvant systemic treatment (non-NST) (n = 7549) | Neoadjuvant systemic treatment (NST) (n = 999) | |
|---|---|---|
| Year of diagnosis | ||
| 2000/2001 | 730 (9,7%) | 51 (5,1%) |
| 2002/2003 | 802 (10,6%) | 60 (6,0%) |
| 2004/2005 | 840 (11,1%) | 68 (6,8%) |
| 2006/2007 | 879 (11,6%) | 81 (8,1%) |
| 2008/2009 | 879 (11,6%) | 99 (9,9%) |
| 2010/2011 | 779 (10,3%) | 105 (10,5%) |
| 2012/2013 | 824 (10,9%) | 127 (12,7%) |
| 2014/2015 | 718 (9,5%) | 169 (16,9%) |
| 2016/2017 | 735 (9,7%) | 159 (15,9%) |
| 2018 | 363 (4,8%) | 80 (8,0%) |
| Age (years) | ||
| median (IQR) | 58 (50 – 68) | 53 (44 - 64) |
| < 40 | 411 (5,4%) | 160 (16,0%) |
| 40 – 49 | 1434 (19,0%) | 245 (24,5%) |
| 50 – 59 | 2151 (28,5%) | 260 (26,0%) |
| 60 – 69 | 1974 (26,1%) | 167 (16,7%) |
| 70 – 79 | 1108 (14,7%) | 128 (12,8%) |
| > 79 | 471 (6,2%) | 39 (3,9%) |
| Mode of detection | ||
| clinically detected | 4465 (59,1%) | 834 (83,5%) |
| screen-detected | 2763 (36,6%) | 91 (9,1%) |
| unknown | 321 (4,3%) | 74 (7,4%) |
| Sublocalisation | ||
| outer quadrants | 3425 (45,4%) | 448 (44,8%) |
| inner quadrants | 1527 (20,2%) | 146 (14,6%) |
| central parts | 213 (2,8%) | 42 (4,2%) |
| overlapping lesions | 1959 (26,0%) | 244 (24,4%) |
| unknown | 425 (5,6%) | 119 (11,9%) |
| Histological tumour type | ||
| ductal | 5898 (78,1%) | 875 (87,6%) |
| lobular | 995 (13,2%) | 72 (7,2%) |
| mixed ductal lobular | 87 (1,2%) | 9 (0,9%) |
| other | 569 (7,5%) | 43 (4,3%) |
| Differentiation grade | ||
| grade 1 | 1295 (17,2%) | 42 (4,2%) |
| grade 2 | 3550 (47,0%) | 411 (41,1%) |
| grade 3/4 | 2659 (35,2%) | 526 (52,7%) |
| unknown | 45 (0,6%) | 20 (2,0%) |
| Clinical tumour stage | ||
| cTIS | 100 (1,3%) | - |
| cT0 | 20 (0,3%) | - |
| cT1 | 3599 (47,7%) | 95 (9,5%) |
| cT2 | 3214 (42,6%) | 363 (36,3%) |
| cT3 | 429 (5,7%) | 209 (20,9%) |
| cT4 | 72 (1,0%) | 332 (33,2%) |
| unknown | 115 (1,5%) | - |
| Clinical nodal stage | ||
| cN0 | 6245 (82,7%) | 296 (29,6%) |
| cN1 | 1176 (15,6%) | 399 (39,9%) |
| pN2 | 26 (0,3%) | 97 (9,7%) |
| pN3 | 15 (0,2%) | 207 (20,7%) |
| unknown | 87 (1,2%) | - |
| Pathological tumour stage (ypT in case of NST) | ||
| pT0 | - | 219 (21,9%) |
| pTIS | - | 95 (9.5%) |
| pT1 | 3815 (50,5%) | 238 (23,8%) |
| pT2 | 3141 (41,6%) | 244 (24,4%) |
| pT3 | 545 (7,2%) | 166 (16,6%) |
| pT4 | 48 (0,6%) | 22 (2,2%) |
| unknown | - | 15 (1,5%) |
| Pathological nodal stage | ||
| pN0 | 4758 (63,0%) | 481 (48,1%) |
| pN1 | 2116 (28,0%) | 244 (24,4%) |
| pN2 | 462 (6,1%) | 122 (12,2%) |
| pN3 | 213 (2,8%) | 97 (9,7%) |
| unknown | - | 55 (5,5%) |
| Hormonal receptor status + treatment | ||
| positive + endocrine therapy | 6302 (83,5%) | 562 (56,3%) |
| positive – endocrine therapy | 252 (3,3%) | 17 (1,7%) |
| negative | 976 (12,9%) | 415 (41,5%) |
| unknown | 19 (0,3%) | 5 (0,5%) |
| HER2 status + treatment | ||
| negative | 6585 (87,2%) | 683 (68,4%) |
| positive + targeted therapy | 549 (7,3%) | 269 (26,9%) |
| positive – targeted therapy | 314 (4,2%) | 38 (3,8%) |
| unknown | 101 (1,3%) | 9 (0,9%) |
| Ductal carcinoma in situ present | ||
| no | 2037 (27,0%) | 493 (49,3%) |
| yes | 5512 (73,0%) | 506 (50,7%) |
| Axillary lymph node dissection | ||
| no | 3250 (43,1%) | 163 (16,3%) |
| yes | 4299 (56,9%) | 836 (83,7%) |
| Type of surgery + RT + immediate breast reconstruction | ||
| BCS + RT | 4049 (53,6%) | 234 (23,4%) |
| BCS – RT mastectomy + RT + direct reconstruction | 57 (0,8%) | 5 (0,5%) |
| mastectomy + RT + direct reconstruction | 2254 (29,9%) | 691 (69,2%) |
| mastectomy – RT – direct reconstruction | 80 (1,1%) | 11 (1,1%) |
| mastectomy – RT + direct reconstruction | 1008 (13,4%) | 55 (5,5%) |
| 101 (1,3%) | 3 (0,3%) | |
| Chemotherapy | ||
| no | 4768 (63,2%) | 94 (9,4%) |
| yes | 2781 (36,8%) | 905 (90,6%) |
| Pathologic complete response | ||
| no | NA | 720 (72,1%) |
| yes | NA | 275 (27,5%) |
| unknown | NA | 4 (0,4%) |
| Vital status at end of follow-up | ||
| alive | 5556 (73,3%) | 673 (67,4%) |
| deceased (all causes) | 1993 (26,4%) | 326 (32,6%) |
| Locoregional recurrence as first eventa | ||
| no | 7324 (97%) | 960 (96,1%) |
| yes | 225 (3%) | 39 (3,9%) |
| Contralateral breast tumour as first eventa | ||
| no | 7234 (95,8%) | 971 (97,2%) |
| yes | 315 (4,2%) | 28 (2,8%) |
Abbreviations: NST = neoadjuvant systemic treatment, IQR = interquartile range, c = clinical (pre-operative), p = pathological (post-operative), T = tumour, N = nodal, IS = in situ, HER2 = human epidermal growth factor receptor 2, NA = not applicable, BCS = breast-conserving surgery, RT = radiation therapy.
In the first 5 years after surgery, there were 93 and 28 LRR cases in the non-NST and NST group, and 86 and 13 CBC cases respectively.
Within five years, LRR occurred in 121 (1.4%) and CBC in 99 (1.2%) patients, with slightly higher LRR and CBC rates in the NST group (1.2% non-NST versus 2.8% NST and 1.1% non-NST versus 1.3% NST, respectively).
3.2. Calibration
Yearly ICI, E50, and E90 values by subgroup, which generally increased over time (Table 2). All ICI and E50 values were <0.01 except for LRR in the NST group; E90 values were naturally higher but remained close to zero. Calibration was slightly less favorable than in internal validation [10].
Table 2.
Overview of the ICI, E50 and E90 for every year, for each subgroup.
| Year | LB | ICI | UB | LB | E50 | UB | LB | E90 | UB | ||
|---|---|---|---|---|---|---|---|---|---|---|---|
| LRR | non-NST | 1 | 0.0001 | 0.0005 | 0.0013 | 0.0000 | 0.0003 | 0.0007 | 0.0000 | 0.0004 | 0.0021 |
| 2 | 0.0005 | 0.0013 | 0.0025 | 0.0002 | 0.0009 | 0.0018 | 0.0002 | 0.0020 | 0.0047 | ||
| 3 | 0.0007 | 0.0018 | 0.0036 | 0.0003 | 0.0013 | 0.0025 | 0.0004 | 0.0024 | 0.0058 | ||
| 4 | 0.0010 | 0.0025 | 0.0047 | 0.0006 | 0.0019 | 0.0035 | 0.0008 | 0.0040 | 0.0082 | ||
| 5 | 0.0012 | 0.0030 | 0.0056 | 0.0007 | 0.0022 | 0.0042 | 0.0011 | 0.0048 | 0.0096 | ||
| NST | 1 | 0.0018 | 0.0045 | 0.0110 | 0.0017 | 0.0047 | 0.0085 | 0.0000 | 0.0055 | 0.0264 | |
| 2 | 0.0032 | 0.0105 | 0.0201 | 0.0034 | 0.0073 | 0.0127 | 0.0000 | 0.0248 | 0.0606 | ||
| 3 | 0.0049 | 0.0135 | 0.0244 | 0.0050 | 0.0101 | 0.0164 | 0.0000 | 0.0277 | 0.0639 | ||
| 4 | 0.0075 | 0.0177 | 0.0298 | 0.0078 | 0.0139 | 0.0212 | 0.0000 | 0.0316 | 0.0688 | ||
| 5 | 0.0091 | 0.0194 | 0.0316 | 0.0099 | 0.0164 | 0.0239 | 0.0016 | 0.0322 | 0.0692 | ||
| CBC | non-NST | 1 | 0.0005 | 0.0012 | 0.0019 | 0.0003 | 0.0011 | 0.0018 | 0.0013 | 0.0021 | 0.0028 |
| 2 | 0.0004 | 0.0011 | 0.0023 | 0.0001 | 0.0007 | 0.0021 | 0.0009 | 0.0029 | 0.0051 | ||
| 3 | 0.0013 | 0.0027 | 0.0045 | 0.0008 | 0.0029 | 0.0049 | 0.0022 | 0.0049 | 0.0079 | ||
| 4 | 0.0021 | 0.0040 | 0.0063 | 0.0018 | 0.0045 | 0.0069 | 0.0029 | 0.0060 | 0.0099 | ||
| 5 | 0.0031 | 0.0052 | 0.0079 | 0.0025 | 0.0058 | 0.0087 | 0.0042 | 0.0083 | 0.0132 | ||
| NST | 1 | 0.0000 | 0.0015 | 0.0064 | 0.0000 | 0.0013 | 0.0040 | 0.0000 | 0.0029 | 0.0145 | |
| 2 | 0.0011 | 0.0030 | 0.0083 | 0.0017 | 0.0024 | 0.0069 | 0.0000 | 0.0049 | 0.0161 | ||
| 3 | 0.0021 | 0.0058 | 0.0117 | 0.0031 | 0.0061 | 0.0089 | 0.0000 | 0.0083 | 0.0241 | ||
| 4 | 0.0031 | 0.0047 | 0.0104 | 0.0029 | 0.0037 | 0.0104 | 0.0044 | 0.0088 | 0.0168 | ||
| 5 | 0.0030 | 0.0079 | 0.0161 | 0.0038 | 0.0088 | 0.0135 | 0.0015 | 0.0110 | 0.0305 | ||
| Year | LB | ICI | UB | LB | E50 | UB | LB | E90 | UB | ||
| LRR | non-NST | 1 | 0.0001 | 0.0005 | 0.0013 | 0.0000 | 0.0003 | 0.0007 | 0.0000 | 0.0004 | 0.0021 |
| 2 | 0.0005 | 0.0013 | 0.0025 | 0.0002 | 0.0009 | 0.0018 | 0.0002 | 0.0020 | 0.0047 | ||
| 3 | 0.0007 | 0.0018 | 0.0036 | 0.0003 | 0.0013 | 0.0025 | 0.0004 | 0.0024 | 0.0058 | ||
| 4 | 0.0010 | 0.0025 | 0.0047 | 0.0006 | 0.0019 | 0.0035 | 0.0008 | 0.0040 | 0.0082 | ||
| 5 | 0.0012 | 0.0030 | 0.0056 | 0.0007 | 0.0022 | 0.0042 | 0.0011 | 0.0048 | 0.0096 | ||
| NST | 1 | 0.0018 | 0.0045 | 0.0110 | 0.0017 | 0.0047 | 0.0085 | 0.0000 | 0.0055 | 0.0264 | |
| 2 | 0.0032 | 0.0105 | 0.0201 | 0.0034 | 0.0073 | 0.0127 | 0.0000 | 0.0248 | 0.0606 | ||
| 3 | 0.0049 | 0.0135 | 0.0244 | 0.0050 | 0.0101 | 0.0164 | 0.0000 | 0.0277 | 0.0639 | ||
| 4 | 0.0075 | 0.0177 | 0.0298 | 0.0078 | 0.0139 | 0.0212 | 0.0000 | 0.0316 | 0.0688 | ||
| 5 | 0.0091 | 0.0194 | 0.0316 | 0.0099 | 0.0164 | 0.0239 | 0.0016 | 0.0322 | 0.0692 | ||
| CBC | non-NST | 1 | 0.0005 | 0.0012 | 0.0019 | 0.0003 | 0.0011 | 0.0018 | 0.0013 | 0.0021 | 0.0028 |
| 2 | 0.0004 | 0.0011 | 0.0023 | 0.0001 | 0.0007 | 0.0021 | 0.0009 | 0.0029 | 0.0051 | ||
| 3 | 0.0013 | 0.0027 | 0.0045 | 0.0008 | 0.0029 | 0.0049 | 0.0022 | 0.0049 | 0.0079 | ||
| 4 | 0.0021 | 0.0040 | 0.0063 | 0.0018 | 0.0045 | 0.0069 | 0.0029 | 0.0060 | 0.0099 | ||
| 5 | 0.0031 | 0.0052 | 0.0079 | 0.0025 | 0.0058 | 0.0087 | 0.0042 | 0.0083 | 0.0132 | ||
| NST | 1 | 0.0000 | 0.0015 | 0.0064 | 0.0000 | 0.0013 | 0.0040 | 0.0000 | 0.0029 | 0.0145 | |
| 2 | 0.0011 | 0.0030 | 0.0083 | 0.0017 | 0.0024 | 0.0069 | 0.0000 | 0.0049 | 0.0161 | ||
| 3 | 0.0021 | 0.0058 | 0.0117 | 0.0031 | 0.0061 | 0.0089 | 0.0000 | 0.0083 | 0.0241 | ||
| 4 | 0.0031 | 0.0047 | 0.0104 | 0.0029 | 0.0037 | 0.0104 | 0.0044 | 0.0088 | 0.0168 | ||
| 5 | 0.0030 | 0.0079 | 0.0161 | 0.0038 | 0.0088 | 0.0135 | 0.0015 | 0.0110 | 0.0305 | ||
Abbreviations: E50 = median absolute difference. E90 = 90th percentile of the absolute difference. ICI = Integrated Calibration Index. LB = 95% confidence interval lower bound. UB = 95% confidence interval upper bound.
Predicted LRR risks were generally low. Median LRR risk was 0.95% (IQR 0.54–1.70%) in non-NST and 3.08% (IQR 2.57–4.71%) in NST patients. For the majority of predictions up to ∼5%, calibration curves lay below the diagonal, with CIs widening at higher risks, especially in the NST group (Fig. 2).
Fig. 2.
Calibration plots for 5-year LRR risk estimation (non-NST group and NST group).
Most predicted CBC risks were 1–2% (Fig. 3). Median risk was 1.70% (IQR 1.43–1.90%) in non-NST and 1.60% (IQR 1.23–1.95%) in NST patients. In non-NST, the calibration curve stayed below the diagonal, while in NST it rose above 1.5–2.3% before falling again. CIs were wider in the NST group.
Fig. 3.
Calibration plots for 5-year CBC risk estimation (non-NST group and NST group).
3.3. Discrimination
Overall, AUCs declined slightly over time (Fig. 4). For LRR, AUCs were >0.8 in years 1–2 and 0.7–0.8 in years 3–5 in the non-NST group. In NST patients, AUCs remained >0.8 each year, exceeding internal validation values (Table 3).
Fig. 4.
Yearly AUCs for every subgroup. In grey the internal validation results of INFLUENCE 3.0 [6] are shown.
Table 3.
Discrimination results per year.
| Time (years) | LB | AUC | UB | ||
|---|---|---|---|---|---|
| LRR | non-NST | 1 | 0.751 | 0.857 | 0.963 |
| 2 | 0.741 | 0.816 | 0.890 | ||
| 3 | 0.724 | 0.788 | 0.851 | ||
| 4 | 0.667 | 0.731 | 0.796 | ||
| 5 | 0.668 | 0.726 | 0.784 | ||
| NST | 1 | 0.772 | 0.881 | 0.990 | |
| 2 | 0.769 | 0.853 | 0.938 | ||
| 3 | 0.760 | 0.838 | 0.915 | ||
| 4 | 0.751 | 0.829 | 0.906 | ||
| 5 | 0.751 | 0.825 | 0.899 | ||
| CBC | non-NST | 1 | 0.347 | 0.529 | 0.711 |
| 2 | 0.407 | 0.499 | 0.591 | ||
| 3 | 0.422 | 0.496 | 0.570 | ||
| 4 | 0.439 | 0.508 | 0.578 | ||
| 5 | 0.464 | 0.528 | 0.592 | ||
| NST | 1 | 0.786 | 0.879 | 0.972 | |
| 2 | 0.771 | 0.854 | 0.937 | ||
| 3 | 0.611 | 0.722 | 0.833 | ||
| 4 | 0.552 | 0.680 | 0.808 | ||
| 5 | 0.582 | 0.692 | 0.802 |
Abbreviations: LB = 95% confidence interval lower bound. AUC = (mean) area under the receiver operating characteristic curve. UB = 95% confidence interval upper bound.
For CBC, non-NST AUCs remained around 0.5 over five years, lower than internal validation. In NST, AUCs were >0.8 in years 1–2 and ∼0.7 thereafter.
4. Discussion
4.1. Evaluation of the results
Calibration of the four INFLUENCE 3.0 submodels was generally good, with the ICI and E50 below 0.01 for all models except the LRR risk estimation in the NST group, where the ICI slightly exceeded this threshold, indicating somewhat reduced calibration. However, ICI, E50, and E90 are best interpreted comparatively, as no universally established reference values exist [21]. The threshold of 0.01 was therefore used as a point of reference, in line with the internal validation of INFLUENCE 3.0 [8]. Moreover, calibration measures depend on the distribution of predicted risks. As the LRR submodel in the NST group generated higher predicted risks, similar degrees of miscalibration may result in larger absolute differences, leading to higher ICI, E50, and E90 values.
For the LRR model in the NST group, calibration plots indicated general overestimation of predicted risks compared to observed outcomes. The bootstrap-based CI did not include the diagonal in the low predicted risk range, suggesting significant miscalibration in this region. As bootstrap-based calibration curves avoid information loss due to risk grouping and are considered more robust, they were used as the primary representation of calibration performance [17,21]. The complementary decile-based plot showed wider CIs, reflecting greater uncertainty, likely due to the small number of events (n = 28) [17,26]. For the CBC risk estimation in the NST group, the calibration curve was irregular and non-monotonic, diverging from the diagonal across different regions, again likely driven by the very small number of events (n = 13).
In the non-NST group, calibration metrics for the CBC model were good, although the calibration curve consistently underestimated observed risks, with the CI falling below the diagonal over a substantial range. However, absolute deviations from perfect calibration were small, remaining below 0.01 across the entire risk range. Calibration of the Cox proportional hazards model for LRR risk estimation in the non-NST group was best. This may be partly explained by Cox models being generally less prone to overfitting and yielding more stable risk estimates than random survival forest models [8,27,28].
For external validation studies, some degree of miscalibration is expected [14,17,29]. In this study, differences in case-mix and incidence rates likely contributed to the observed miscalibration [14,17]. Specifically, the generally lower five-year event rates in the Leuven cohort (1.4% LRR and 1.2% CBC) compared to the Dutch development population (2.2% LRR and 1.4% CBC) may explain part of the risk overestimation [14,[29], [30], [31], [32]].
Discriminative performance was the most important aspect of the model evaluation. Overall, the AUCs for LRR were good in both groups (range 0.73–0.88). As expected, AUCs decreased over time, consistent with internal validation results and the declining predictive ability further from baseline [33].
CBC risk estimation showed markedly poorer performance, particularly in the non-NST group, with AUCs around 0.5, indicating discrimination no better than chance [17]. In the NST group, CBC discrimination was good at three years but declined to moderate performance at four and five years.
The reasons for this weak performance are difficult to pinpoint. Case-mix variation in the external cohort appeared adequate, but CBC risk prediction is inherently challenging due to the limited predictive power of established risk factors and the influence of lifestyle variables [34]. Other known CBC predictors—germline mutations, family history, BMI, and young-age RT—are not included in INFLUENCE 3.0 [35]. However, even models incorporating some of these factors, such as PredictCBC-2.0, achieved only moderate AUCs (∼0.65) in large cohorts [34]. A sensitivity analysis, in a more homogeneous and less historical cohort, did not improve performance, suggesting that changes in treatment policy do not explain the findings (Appendix B).
The original INFLUENCE 1.0 model, which predicted five-year LRR in non-NST–treated patients only, was externally validated in a German cohort. Of course, a direct comparison with our validation is not possible because the updated INFLUENCE 3.0 model additionally predicts for CBC, is applicable to both NST and non-NST patients and used different methodology. Nevertheless, the German validation showed that the original model had good total discrimination (AUC 0.73; 95% CI 0.69–0.77), but performance declined annually, reaching no meaningful discrimination by 5 years (AUC 0.50; 95% CI 0.41–0.60). Calibration analyses showed that the original model slightly underestimated LRR risk in that population [30].
In our validation of the INFLUENCE 3.0 model, AUCs for LRR were somewhat higher (range 0.76–0.88) and, although they also declined over time, discrimination remained good at 5 years: AUC 0.723 (95% CI 0.67–0.78) in the non-NST group and AUC 0.825 (95% CI 0.75–0.90) in the NST group. Calibration plots similarly showed overestimation of LRR risk at lower predicted probabilities.
4.2. Comparison of the Belgian and Dutch cohorts
The results could be related to differences in patient populations. The cohorts were similar in age, tumor sublocalisation, histology, HER2 status with targeted therapy, and chemotherapy, but differed on other factors. The Dutch cohort included patients diagnosed 2012–2016, while 65% of Leuven patients were diagnosed before 2012. Screen-detected cases were lower in Leuven (36.6% non-NST, 9.1% NST vs. 47.1% and 15.2%) and ductal carcinoma in situ (DCIS) was more frequent (73% vs. 50.7% non-NST; 50.7% vs. 35.6% NST). Leuven patients also had higher-grade tumors, more advanced stage, higher mastectomy rates, more endocrine therapy (83.5% vs. 53.6% non-NST), and more frequent axillary lymph node dissection (56.9% vs. 15.2% non-NST; 83.7% vs. 36.4% NST).
4.3. Clinical implications
Some miscalibration was observed, mainly overestimation of LRR and CBC risks, but absolute deviations were small due to low event rates [14,29,31]. Although this may demand for slightly intensified surveillance, the clinical impact is likely limited, and modest overestimation may be preferable to underestimation when tailoring follow-up.
In practice, discrimination is considered the more important performance aspect for INFLUENCE 3.0, as it identifies patients who may be candidates for less intensive surveillance [36]. In that regard, the model showed adequate discrimination for LRR in both NST and non-NST groups.
In contrast, CBC discrimination in the non-NST group was poor and near random, largely due to the narrow range of low predicted risks. However, because calibration was acceptable, these predictions should not be considered unreliable or misleading, they simply show limited variation. Therefore, while tailoring surveillance schemes based on CBC risk predictions is not recommended, clinicians and patients can be guided by the LRR risk estimates, which demonstrate better discrimination. This LRR risk, together with the relatively constant CBC risk, may still provide meaningful guidance in optimizing individual surveillance strategies.
4.4. Limitations
Some limitations are the use of historical (2000–2018), single-center data, with a relatively small sample size (particularly the NST subgroup) and few events, resulting in less stable estimates [10].
Furthermore, INFLUENCE 3.0 includes a broad set of established clinical predictors of LRR and CBC [35,[37], [38], [39], [40], [41], [42]]. However, other studies also link LRR to positive/close margins, lymphovascular invasion, and premenopausal status [43]. While these factors are not included in the model, INFLUENCE 3.0 indirectly accounts for margins and menopausal status through exclusion of patients with positive/close margins and inclusion of age, which largely reflects menopausal status.
Another limitation is incomplete information on irradical surgical margins for a subset of patients. Of approximately 300 patients with initially irradical margins, 50 were manually reviewed; only two had more-than-focally irradical margins without appropriate re-intervention and were excluded. Based on this low proportion, it was assumed that similar cases among the remaining patients were rare, and they were retained in the analysis. Nevertheless, this likely resulted in fewer exclusions than in the Dutch cohort (0.02% vs. 0.7%). As positive margins increase LRR risk, their inclusion may have led to underestimation of model performance [44,45].
Lastly, the definition of our primary outcome LRR needs some explanation. We used the same definition as the one that was used for the development of INFLUENCE, which is in line with consensus-based event definitions [18]a. However, it should be noted that in clinical practice the definitions of recurrent events vary and that there are multiple different classification systems to classify a recurrent event as a true recurrence or second primary breast cancer [46]. We chose to embrace this practical variation because treatments are likely to be based on the definition as reported in the patient files. A sensitivity analysis of our data in which we excluded the ipsilateral new primary breast cancers (which were only few events) from this definition did not alter the results.
4.5. Future perspectives
INFLUENCE 3.0 currently estimates 5-year recurrence risks, although LRR and CBC are known to continue occurring beyond 10 years of follow-up [47,48]. In the Leuven cohort, median follow-up was 9.8 years in the non-NST group and 6.5 years in the NST group. Over this period, 225 non-NST patients (3.0%) developed LRR and 315 (4.2%) CBC, while 39 LRRs (3.9%) and 28 CBCs (2.8%) occurred in the NST group. The number of LRR and CBC events nearly doubled with extended follow-up, underscoring the clinical relevance of extending INFLUENCE 3.0 predictions to at least 10 years in future updates.
An additional important future improvement is extending INFLUENCE 3.0 to predict distant recurrence. Although no randomized BC-specific evidence shows that earlier systemic treatment of asymptomatic metastatic disease improves outcomes—as such trials are ethically challenging—identifying patients at high risk for distant recurrence remains clinically valuable, as it reflects aggressive tumor biology. Moreover, evidence shows that local stereotactic RT in oligometastatic disease improves overall survival (OS) [49]. Earlier identification of distant recurrences could therefore support earlier symptom management preserving quality of life, and detection of oligometastatic disease, potentially improving OS [50,51]. Since INFLUENCE 2.0 already included distant recurrence prediction, integrating all recurrence types into a single, unified INFLUENCE model would be valuable to clinical decision-making and support personalised follow-up [11].
Furthermore, INFLUENCE 3.0 provides individualised LRR and CBC risk estimates, which should be interpreted by clinicians and discussed with patients to guide follow-up. However, translating absolute percentages into clinical decisions is challenging. Therefore, expanding the model to compare predicted risks with population-level benchmarks, such as the yearly LRR and CBC incidence rates summarized by Spronk et al. could allow the establishment of clinically meaningful cut-offs [47]. Patients below such thresholds might follow less intensive follow-up, while higher-risk patients could be considered for a more intensive surveillance [4,52]. Nonetheless, developing and validating such clinical guidance would require separate dedicated research, which is partly addressed by the ongoing NABOR study [53].
5. Conclusion
In this study, the INFLUENCE 3.0 model was externally validated using data from the University Hospitals Leuven, Belgium. While the model showed acceptable calibration and good to very good discrimination for LRR risk estimation, its discriminative ability for CBC risk estimation in the non-NST group was moderate to poor. However, because CBC risk predictions were narrowly distributed and showed acceptable calibration, the overall risk estimates from INFLUENCE 3.0 – primarily the LRR risks – could still be valuable for tailoring surveillance strategies. These findings highlight the need for further model updates such as recalibration or alternative approaches to improve CBC risk estimation, and emphasize the importance of additional external validation – ideally performed on larger datasets – to confirm model suitability across different populations and healthcare settings.
CRediT authorship contribution statement
Aline van der Vorst: Writing – review & editing, Writing – original draft, Resources, Project administration. Carlijn Grimberg: Writing – review & editing, Writing – original draft, Investigation, Formal analysis. Marissa C. van Maaren: Writing – review & editing, Resources, Project administration, Data curation, Conceptualization. Tom A. Hueting: Writing – review & editing, Methodology. Caroline Weltens: Writing – review & editing, Supervision, Conceptualization. Sabine Siesling: Writing – review & editing, Supervision, Conceptualization. Catharina G.M. Groothuis-Oudshoorn: Writing – review & editing, Methodology.
Ethics declaration
The authors have NOT obtained informed consent from participants or their legal representatives. This retrospective observational study was based on routinely collected clinical data from a registry that uses an opt-out procedure. Patients are automatically included in the registry, and their data may be used for research purposes unless they object to such use. Therefore, separate informed consent for the present study was not required.
This study was performed in compliance with relevant laws, regulatory frameworks and guidelines where the research took place. This study was approved by the BMS Ethical review committee, University of Twente, and Ethics Committee Research UZ/KU Leuven. (Approval No. 240722 (University of Twente) and S68978 (UZ Leuven))
Data availability
Due to privacy regulations, the dataset used in this study is not publicly available. Aggregated data is available on reasonable request by contacting the corresponding author.
Declaration of generative AI and AI-assisted technologies in the writing process
During the preparation of this work the author(s) used ChatGPT for linguistics solely. After using this tool/service, the author(s) reviewed and edited the content as needed and take(s) full responsibility for the content of the publication.
Funding
During the preparation of this manuscript, Aline Van der Vorst received funding from the Emmanuel van der Schueren Starter Fund of Kom op tegen Kanker [KOTK_KUL/2024/13905]. The funder did not play any role in the design, analysis and conclusions of this study.
Declaration of competing interest
The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Prof. Sabine Siesling is specialty editor Epidemiology at the Breast. Given her role as specialty editor, Sabine Siesling will have no involvement in the peer-review of this article and have no access to information regarding its peer review. Dr. Marissa C van Maaren, Dr. Catharina GM Groothuis-Oudshoorn and Tom A Hueting have been involved in the development of the INFLUENCE 3.0 model. If there are other authors, they declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgements
The authors gratefully acknowledge the contribution of Chantal Remmerie to the data search.
Footnotes
Supplementary data to this article can be found online at https://doi.org/10.1016/j.breast.2026.104944.
Appendix A. Supplementary data
The following is the Supplementary data to this article.
References
- 1.Cancer fact sheet breast cancer, 2023 | Belgian Cancer Registry [Internet]. [cited 2025 Nov 26]. Available from: https://kankerregister.org/en/publicaties/cancer-fact-sheet-breast-cancer-2023.
- 2.Loibl S., André F., Bachelot T., Barrios C.H., Bergh J., Burstein H.J., et al. Early breast cancer: ESMO clinical practice guideline for diagnosis, treatment and follow-up. Ann Oncol. 2024 Feb 1;35(2):159–182. doi: 10.1016/j.annonc.2023.11.016. PubMed PMID: 38101773. [DOI] [PubMed] [Google Scholar]
- 3.Nahleh Z., Alfano C.M., Somerfield M.R., Lee J.M., Hieken T.J., Magnuson A., et al. Breast cancer Follow-Up and surveillance after primary treatment: ASCO guideline update. J Clin Oncol. 2026 doi: 10.1200/JCO-26-01700. PubMed PMID: 42462191. [DOI] [PubMed] [Google Scholar]
- 4.Witteveen A., Vliegen I.M.H., Sonke G.S., Klaase J.M., Ijzerman M.J., Siesling S. Personalisation of breast cancer follow-up: a time-dependent prognostic nomogram for the estimation of annual risk of locoregional recurrence in early breast cancer patients. Breast Cancer Res Treat. 2015 Aug 31;152(3):627–636. doi: 10.1007/S10549-015-3490-4. PubMed PMID: 26162567. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Pennery E., Mallet J. A preliminary study of patients' perceptions of routine follow-up after treatment for breast cancer. Eur J Oncol Nurs. 2000 Sep;4(3):138–145. doi: 10.1054/EJON.2000.0092. PubMed PMID: 12849643. [DOI] [PubMed] [Google Scholar]
- 6.Kiebert G.M., Welvaart K., Kievit J. Psychological effects of routine follow up on cancer patients after surgery. Eur J Surg. 1993;159(11–12):601–607. http://www.ncbi.nlm.nih.gov/pubmed/8130301 [Internet] [PubMed] [Google Scholar]
- 7.Geurts S.M.E., de Vegt F., Siesling S., Flobbe K., Aben K.K.H., van der Heiden-van der Loo M., et al. Pattern of follow-up care and early relapse detection in breast cancer patients. Breast Cancer Res Treat. 2012;136(3):859–868. doi: 10.1007/s10549-012-2297-9. [DOI] [PubMed] [Google Scholar]
- 8.Van Maaren M.C., Hueting T.A., van Uden D.J.P., van Hezewijk M., de Munck L., Mureau M.A.M., et al. The INFLUENCE 3.0 model: updated predictions of locoregional recurrence and contralateral breast cancer, now also suitable for patients treated with neoadjuvant systemic therapy. Breast. 2025 Feb 1:79. doi: 10.1016/j.breast.2024.103829. PubMed PMID: 39541608. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Voets M.M., Hassink N.S., Veltman J., Slump C.H., Koffijberg H., Siesling S. Opportunities for personalised follow-up in breast cancer: the gap between daily practice and recurrence risk. Breast Cancer Res Treat. 2024 Jun 1;205(2):313–322. doi: 10.1007/S10549-024-07246-5. PubMed PMID: 38409613. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Van Maaren M.C., Hueting T.A., van Uden D.J.P., van Hezewijk M., de Munck L., Mureau M.a.M., et al. The INFLUENCE 3.0 model: updated predictions of locoregional recurrence and contralateral breast cancer, now also suitable for patients treated with neoadjuvant systemic therapy. Breast. 2025;79 doi: 10.1016/j.breast.2024.103829. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Völkel V., Hueting T.A., Draeger T., van Maaren M.C., de Munck L., Strobbe L.J.A., et al. Improved risk estimation of locoregional recurrence, secondary contralateral tumors and distant metastases in early breast cancer: the INFLUENCE 2.0 model. Breast Cancer Res Treat. 2021 Oct 1;189(3):817–826. doi: 10.1007/S10549-021-06335-Z. PubMed PMID: 34338943. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Influence 3.0: risk of locoregional recurrence, secondary contralateral tum - evidencio [Internet]. [cited 2025 Sep 3]. Available from: https://www.evidencio.com/models/show/2238.
- 13.Klaassen-Dekker A., Drossaert C.H.C., Van Maaren M.C., Van Leeuwen-Stok A.E., Retel V.P., Korevaar J.C., et al. Personalized surveillance and aftercare for non-metastasized breast cancer: the NABOR study protocol of a multiple interrupted time series design. BMC Cancer. 2023 Dec 1;23(1) doi: 10.1186/S12885-023-11504-Y. PubMed PMID: 37964214. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Ramspek C.L., Jager K.J., Dekker F.W., Zoccali C., van Diepen M. External validation of prognostic models: what, why, how, when and where? Clin Kidney J. 2021;14(1):49–58. doi: 10.1093/ckj/sfaa188. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.van Maaren M.C., Hueting T.A., Völkel V., van Hezewijk M., Strobbe L.J., Siesling S. The use and misuse of risk prediction tools for clinical decision-making. Breast. 2023;69:428–430. doi: 10.1016/j.breast.2023.01.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Van Calster B., Steyerberg E.W., Wynants L., van Smeden M. There is no such thing as a validated prediction model. BMC Med. 2023;21(1):70. doi: 10.1186/s12916-023-02779-w. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Riley R.D., Archer L., Snell K.I.E., Ensor J., Dhiman P., Martin G.P., et al. Evaluation of clinical prediction models (part 2): how to undertake an external validation study. Br Med J. 2024;384 doi: 10.1136/bmj-2023-074820. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Moossdorff M., Van Roozendaal L.M., Strobbe L.J.A., Aebi S., Cameron D.A., Dixon J.M., et al. Maastricht Delphi consensus on event definitions for classification of recurrence in breast cancer research. J Natl Cancer Inst. 2014 Dec 1;106(12) doi: 10.1093/JNCI/DJU288. PubMed PMID: 25381395. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Putter H., Fiocco M., Gekus R.B. Tutorial in biostatistics: competing risks and multi-state models. Stat Med. 2007 May 20;26(11):2389–2430. doi: 10.1002/SIM.2712. PubMed PMID: 17031868. [DOI] [PubMed] [Google Scholar]
- 20.Buuren S van, Groothuis-Oudshoorn K., Vink G., Schouten R., Robitzsch A., Rockenschaub P., et al. 2024. Mice: multivariate imputation by chained equations [Internet]https://cran.r-project.org/web/packages/mice/index.html Available from: [Google Scholar]
- 21.Austin P.C., Harrell Jr FE., van Klaveren D. Graphical calibration curves and the integrated calibration index (ICI) for survival models. Stat Med. 2020;39(21):2714–2742. doi: 10.1002/sim.8570. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Sawyer S. The greenwood and exponential greenwood confidence intervals in survival analysis [Internet]. Available from:: https://www.math.wustl.edu/∼sawyer/handouts/greenwood.pdf.
- 23.Hond AAH de, Steyerberg E.W., Calster B van. Interpreting area under the receiver operating characteristic curve. Lancet Digit Health. 2022;4(12):e853–e855. doi: 10.1016/S2589-7500(22)00188-1. [DOI] [PubMed] [Google Scholar]
- 24.Van Buuren S. second ed. 2021. Flexible imputation of missing data; p. 444.https://stefvanbuuren.name/fimd/sec-nutshell.html [Internet] [Google Scholar]
- 25.Chapter9 Rubin's Rules | Book_MI. 2025. https://bookdown.org/mwheymans/bookmi/rubins-rules.html knit [Internet] Available from: [Google Scholar]
- 26.Riley R.D., Collins G.S., Ensor J., Archer L., Booth S., Mozumder S.I., et al. Minimum sample size calculations for external validation of a clinical prediction model with a time-to-event outcome. Stat Med. 2022;41(7):1280–1295. doi: 10.1002/sim.9275. [DOI] [PubMed] [Google Scholar]
- 27.Charilaou P., Battat R. Machine learning models and over-fitting considerations. World J Gastroenterol. 2022;28(5):605–607. doi: 10.3748/wjg.v28.i5.605. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Montesinos López O.A., Montesinos López A., Crossa J. In: Multivariate statistical machine learning methods for genomic prediction [Internet] Montesinos López O.A., Montesinos López A., Crossa J., editors. Springer International Publishing; Cham: 2022. Overfitting, model tuning, and evaluation of prediction performance; pp. 109–139. [DOI] [PubMed] [Google Scholar]
- 29.Steyerberg E.W. Statistics for biology and health [Internet] Springer International Publishing; Cham: 2019. Clinical prediction models: a practical approach to development, validation, and updating.http://link.springer.com/10.1007/978-3-030-16399-0 Available from: [Google Scholar]
- 30.Voelkel V., Draeger T., Groothuis-Oudshoorn C.G.M., de Munck L., Hueting T., Gerken M., et al. Predicting the risk of locoregional recurrence after early breast cancer: an external validation of the Dutch INFLUENCE-nomogram with clinical cancer registry data from Germany. J Cancer Res Clin Oncol. 2019;145(7) doi: 10.1007/S00432-019-02904-4. PubMed PMID: 30927074. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Van Calster B., McLernon D.J., van Smeden M., Wynants L., Steyerberg E.W., Bossuyt P., et al. Calibration: the achilles heel of predictive analytics. BMC Med. 2019;17(1):230. doi: 10.1186/s12916-019-1466-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Siesling S., Meijer J., van Maaren M.C., de Munck L., Koppert L., van den Bongard D. Abstract P3-12-06: locoregional and contralateral breast tumor events within 5 years after diagnosis of primary breast cancer in the Netherlands: a population-based study including 121,426 breast cancer patients. Clin Cancer Res. 2025 Jun 15;31(12_Supplement) doi: 10.1158/1557-3265.SABCS24-P3-12-06. P3-12–06. [DOI] [Google Scholar]
- 33.Kamarudin A.N., Cox T., Kolamunnage-Dona R. Time-dependent ROC curve analysis in medical research: current methods and applications. BMC Med Res Methodol. 2017 Apr 7;17(1) doi: 10.1186/S12874-017-0332-6. PubMed PMID: 28388943. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Giardiello D., Hooning M.J., Hauptmann M., Keeman R., Heemskerk-Gerritsen B.a.M., Becher H., et al. PredictCBC-2.0: a contralateral breast cancer risk prediction model developed and validated in ∼ 200,000 patients. Breast Cancer Res. 2022;24(1):69. doi: 10.1186/s13058-022-01567-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Akdeniz D., Schmidt M.K., Seynaeve C.M., McCool D., Giardiello D., van den Broek A.J., et al. Risk factors for metachronous contralateral breast cancer: a systematic review and meta-analysis. Breast. 2019 Apr 1;44:1–14. doi: 10.1016/j.breast.2018.11.005. PubMed PMID: 30580169. [DOI] [PubMed] [Google Scholar]
- 36.Sangen M.J.C. van der, Poortmans P.M.P., Scheepers S.W.M., Lemaire B.M.D., Berlo C.L.H. van, Tjan-Heijnen V.C.G., et al. Prognosis following local recurrence after breast conserving treatment in young women with early breast cancer. Eur J Surg Oncol. 2013;39(8):892–898. doi: 10.1016/j.ejso.2013.05.004. [DOI] [PubMed] [Google Scholar]
- 37.Nolan L., Davey M.G., Calpin G.G., Ryan É.J., Boland M.R. Risk of locoregional recurrence after breast cancer surgery by molecular subtype-a systematic review and network meta-analysis. Ir J Med Sci. 2024 Dec 1;193(6):2965–2974. doi: 10.1007/S11845-024-03809-Z. PubMed PMID: 39331262. [DOI] [PubMed] [Google Scholar]
- 38.Merino T., Ip T., Domínguez F., Acevedo F., Medina L., Villaroel A., et al. Risk factors for loco-regional recurrence in breast cancer patients: a retrospective study. Oncotarget. 2018 Jul 1;9(54):30355–30362. doi: 10.18632/ONCOTARGET.25735. PubMed PMID: 30100994. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Valachis A., Mamounas E.P., Mittendorf E.A., Hayashi N., Ishitobi M., Natoli C., et al. Risk factors for locoregional disease recurrence after breast-conserving therapy in patients with breast cancer treated with neoadjuvant chemotherapy: an international collaboration and individual patient meta-analysis. Cancer. 2018 Jul 15;124(14):2923–2930. doi: 10.1002/CNCR.31518. PubMed PMID: 29723396. [DOI] [PubMed] [Google Scholar]
- 40.Starikov A., Askin G., Blackburn A., Lu C.M., Reznik E., Kim J., et al. Mode of detection matters: differences in screen-detected versus symptomatic breast cancers. Clin Imag. 2021 Dec 1;80:11–15. doi: 10.1016/j.clinimag.2021.06.032. PubMed PMID: 34218078. [DOI] [PubMed] [Google Scholar]
- 41.Colleoni M., Zahrieh D., Gelber R.D., Holmberg S.B., Mattsson J.E., Rudenstam C.M., et al. Site of primary tumor has a prognostic role in operable breast cancer: the international breast cancer study group experience. J Clin Oncol. 2005;23(7):1390–1400. doi: 10.1200/JCO.2005.06.052. PubMed PMID: 15735115. [DOI] [PubMed] [Google Scholar]
- 42.Vrieling C., Van Werkhoven E., Maingon P., Poortmans P., Weltens C., Fourquet A., et al. Prognostic factors for local control in breast cancer after long-term Follow-up in the EORTC boost vs no boost trial: a randomized clinical trial. JAMA Oncol. 2017 Jan 1;3(1):42–48. doi: 10.1001/JAMAONCOL.2016.3031. PubMed PMID: 27607734. [DOI] [PubMed] [Google Scholar]
- 43.Peng G., Zhou Z., Jiang M., Yang F. Can a subgroup at high risk for LRR be identified from T1-2 breast cancer with negative lymph nodes after mastectomy? A meta-analysis. Biosci Rep. 2019 Sep 20;39(9) doi: 10.1042/BSR20181853. PubMed PMID: 31484798. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Moran M.S., Schnitt S.J., Giuliano A.E., Harris J.R., Khan S.A., Horton J., et al. Society of surgical oncology–american society for radiation oncology consensus guideline on margins for breast-conserving surgery with whole-breast irradiation in stages I and II invasive breast cancer. Int J Radiat Oncol Biol Phys. 2014;88(3):553–564. doi: 10.1016/j.ijrobp.2013.11.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Houssami N., Macaskill P., Luke Marinovich M., Morrow M. The association of surgical margins and local recurrence in women with early-stage invasive breast cancer treated with breast-conserving therapy: a meta-analysis. Ann Surg Oncol. 2014 Mar;21(3):717–730. doi: 10.1245/S10434-014-3480-5. PubMed PMID: 24473640. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Blacker S., Boyle J.M., Wang L., Withrow D.R., Delon C., Dodwell D., et al. Evaluation of methods to classify ipsilateral breast tumour recurrences as local recurrence or new primary tumour. NPJ Breast Cancer. 2026 Dec 1;12(1) doi: 10.1038/S41523-025-00850-8. PubMed PMID: 41688485. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Spronk I., Schellevis F.G., Burgers J.S., de Bock G.H., Korevaar J.C. Incidence of isolated local breast cancer recurrence and contralateral breast cancer: a systematic review. Breast. 2018 Jun 1;39:70–79. doi: 10.1016/j.breast.2018.03.011. PubMed PMID: 29621695. [DOI] [PubMed] [Google Scholar]
- 48.Mancini S., Bucchi L., Biggeri A., Giuliani O., Baldacchini F., Ravaioli A., et al. Incidence and temporal patterns of true recurrences and second primaries in women with breast cancer: a 10-year competing risk-adjusted analysis. Breast. 2025 Apr 1;80 doi: 10.1016/j.breast.2025.103883. PubMed PMID: 39889470. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49.Palma D.A., Olson R., Harrow S., Gaede S., Louie A.V., Haasbeek C., et al. Stereotactic ablative radiotherapy for the comprehensive treatment of oligometastatic cancers: long-term results of the SABR-COMET phase II randomized trial. J Clin Oncol. 2020;38(25):2830–2838. doi: 10.1200/JCO.20.00818. PubMed PMID: 32484754. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Lu W.L., Jansen L., Post W.J., Bonnema J., Van De Velde J.C., De Bock G.H. Impact on survival of early detection of isolated breast recurrences after the primary treatment for breast cancer: a meta-analysis. Breast Cancer Res Treat. 2009 Apr;114(3):403–412. doi: 10.1007/S10549-008-0023-4. PubMed PMID: 18421576. [DOI] [PubMed] [Google Scholar]
- 51.Cardoso F., Senkus E., Costa A., Papadopoulos E., Aapro M., André F., et al. 4th ESO–ESMO international consensus guidelines for advanced breast cancer (ABC 4) Ann Oncol. 2018;29(8):1634–1657. doi: 10.1093/annonc/mdy192. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 52.Witteveen A., de Munck L., Groothuis-Oudshoorn C.G.M., Sonke G.S., Poortmans P.M., Boersma L.J., et al. Evaluating the age-based recommendations for long-term Follow-Up in breast cancer. Oncologist. 2020 Sep 1;25(9):e1330–e1338. doi: 10.1634/THEONCOLOGIST.2019-0973. PubMed PMID: 32510767. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 53.Klaassen-Dekker A., Drossaert C.H.C., Van Maaren M.C., Van Leeuwen-Stok A.E., Retel V.P., Korevaar J.C., et al. Personalized surveillance and aftercare for non-metastasized breast cancer: the NABOR study protocol of a multiple interrupted time series design. BMC Cancer. 2023 Dec 1;23(1) doi: 10.1186/s12885-023-11504-y. PubMed PMID: 37964214. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
Due to privacy regulations, the dataset used in this study is not publicly available. Aggregated data is available on reasonable request by contacting the corresponding author.