Performance of three breast cancer risk assessment tools in US Black women

Abstract

Background

Breast cancer risk prediction models aid identification of high-risk women for earlier or more frequent screening. The two most commonly used U.S. models appear to perform less well in Black women, possibly because Black women have a lower proportion of estrogen-receptor positive breast cancer. We recently developed and externally validated a model for use in Black women (BWHS model). Here, we compare performance metrics of that model with the other two models using data from a large cohort of Black women.

Results

We assessed the NCI Breast Cancer Risk Assessment Tool (BCRAT) using the option for Black women, the IBIS model, including clinical variables only, and the BWHS model in data from a cohort of 50,235 Black women followed over four sequential 5 year periods. Predictors were updated at the start of each 5 year period, and 2041 invasive breast cancers occurred. Calibration metrics, expected over observed number of cancers, were 0.99 (0.94–1.04), 0.97 (0.93–1.02), and 1.13 (1.08–1.18) from the BWHS, BCRAT, and IBIS models, respectively. The metrics for discriminatory accuracy, age-adjusted area under the curve (AUC), were 0.58 (0.56–0.59), 0.56 (0.55–0.57), and 0.56 (0.55–0.57), from the BWHS, BCRAT, and IBIS models, respectively.

Conclusions

In this comparison, the BWHS model had better calibration and discrimination than BCRAT and IBIS, including among women age < 40, indicating a benefit to using the BWHS model for Black women. While models that incorporate mammographic features may have higher AUCs, models based on clinical factors are beneficial for young women and those without available mammography data.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13058-026-02274-z.

Introduction

Absolute breast cancer risk prediction models estimate a woman’s risk of developing breast cancer over a given period (e.g. 5 years) in the presence of competing risks and are useful for recommending screening strategies and risk-reduction approaches [1]. However, many primary care providers do not use risk prediction calculators due to time constraints during primary care visits; thus, adding a new calculator for a specific population requires evidence of its relative benefit. To this end, we compared performance metrics of the BWHS calculator [2, 3], developed specifically for breast cancer risk assessment in Black women, with metrics from the two calculators most often used in U.S. clinical practice, the National Cancer Institute’s Breast Cancer Risk Assessment Tool, BCRAT or “Gail model” [4–6] and the IBIS or “Tyrer-Cuzick” model [7, 8].

Methods

Validation study population

The Black Women’s Health Study is a prospective cohort study of 59,000 self-identified Black women from all regions of the U.S. [9]. Participants, aged 21–69 in 1995, completed a lengthy baseline questionnaire at enrollment and have updated their data on exposures and outcomes in biennial questionnaires since then. Follow-up has been successful for 85% of potential person-years through 14 follow-up cycles. Incident breast cancers are identified by self-report and by linkage to state cancer registries and the National Death Index. The study protocol was approved by the Boston University Institutional Review Board.

BWHS model

The BWHS model [2] was developed using data from three large case-control studies and age-and race-specific breast cancer incidence and competing mortality rates from the NCI’s Surveillance, Epidemiology, and End Results (SEER) program. It was validated in prospective data from the Black Women’s Health Study. Model predictors are first-degree family history of breast and prostate cancer, breast biopsy, age at menarche, ever parous, breastfeeding, body mass index (BMI) (current and at age 18), menopausal status, and bilateral oophorectomy. Age interactions with family history of breast cancer, breast biopsy, and age at menarche, as well as an interaction between menopausal status and current BMI, are included in the model.

BCRAT model

In 2007, Gail et al. modified their original BCRAT model using data from Black women in the Women’s CARE study and population-based age- and race-specific incidence rates from SEER [5]. Covariates in the BCRAT “CARE” model include number of breast biopsies, atypical hyperplasia, age at menarche, and number of first-degree relatives with breast cancer. An interaction between age and the number of breast biopsies is also included. The online version of the original BCRAT model includes an option for race; if Black is selected, the calculator uses the BCRAT “CARE” model.

IBIS model

The IBIS model (TC 8) [7] includes age at menarche, number of live births, age at first birth, menopausal status, age at menopause, height, weight, history of hyperplasia (usual and/or atypical), history of and age at diagnosis of ovarian cancer, Ashkenazi Jewish heritage, hormone replacement therapy use, BRCA1&2 test results, extensive family history of breast and ovarian cancer, breast density, and a polygenic single nucleotide polymorphism score (PRS). The model incorporates population-based incidence and mortality rates from the general UK female population. We used a version of the model without mammographic density or PRS because these are not available in many clinical situations.

Statistical methods

To assess model performance for 5 year breast cancer risk estimates, we constructed four 5 year follow-up intervals: 2001–2005, 2006–2010, 2011–2015, and 2016–2020. For each interval, validation cohort participants aged 30 through 70 were included if they had not been diagnosed with invasive breast cancer at the start of that interval. Age- and time-dependent predictors were updated at the start of each interval. For each model, we computed separate 5 year predictions for each interval to which a woman contributed.

To handle variables that are in specific models but not available in validation data, we used instructions provided for each model. For the BCRAT model, number of biopsies was set to one if the participant reported biopsy and hyperplasia was set to none for all participants. For the IBIS model, the following variables were set to none: history of hyperplasia or atypical hyperplasia, Ashkenazi Jewish heritage, genetic testing, family history of bilateral breast cancer. The IBIS model included exact age of relative’s breast cancer diagnosis, whereas the validation cohort provided data only on whether the diagnosis was before or after age 50. For relatives diagnosed before age 50, we assigned an age from 20 to 49 using a shifted beta distribution with α = 5 and β = 1. For relatives diagnosed at age 50 or older, we assigned age at diagnosis using a uniform distribution.

To evaluate calibration, we estimated the ratio of expected to observed numbers of invasive breast cancer cases (E/O), where E was calculated by summing the predicted 5 year risks over the four time periods. We estimated E/O ratios overall and within strata of age, breast cancer family history, and history of biopsy. We assessed discriminatory accuracy by computing the area under the curve (AUC) within 5 year age groups and used inverse-variance weighting to calculate the overall age-adjusted AUC. We used a bootstrap procedure wherein we resampled women with replacement and used the quantiles of the bootstrap distribution to obtain 95% confidence intervals for AUCs and two-sided bootstrap p-values for the differences between AUCs for different models.

We created cross-classification tables between each pair of models to compare assignments of cases and non-cases as “high risk” (≥ 1.67% risk over 5 years).

Results

Over 20 years of follow-up, 2041 incident invasive breast cancers developed among 50,235 Black participants. The expected number of breast cancers relative to observed (E/O) was 0.99 (95% confidence interval 0.95–1.04) from the BWHS model, 0.97 (0.93–1.02) from BCRAT, and 1.13 (1.08–1.18) from IBIS (Table 1).

Table 1.

Calibration of three breast cancer risk prediction calculators in 20 years of follow-up among 52,901 participants in the Black Women’s Health Study

	BWHS model				BCRAT model				IBIS model
	E	O	E/O (95% CI)		E	O	E/O (95% CI)		E	O	E/O (95% CI)
Overall	2027	2041	0.99 (0.95–1.04)		1989	2041	0.97 (0.93–1.02)		2299	2041	1.13 (1.08–1.18)
Quintiles of risk
1 (low)	148	155	0.95 (0.81–1.12)		131	158	0.83 (0.71–0.97)		153	160	0.96 (0.82–1.12)
2	288	317	0.91 (0.82–1.02)		296	322	0.92 (0.82–1.03)		297	348	0.86 (0.77–0.95)
3	389	363	1.07 (0.97–1.19)		371	395	0.94 (0.85–1.04)		398	410	0.97 (0.88–1.07)
4	491	515	0.95 (0.87–1.04)		498	500	1.00 (0.91–1.09)		522	469	1.10 (1.01–1.21)
5 (high)	711	691	1.03 (0.95–1.11)		693	666	1.04 (0.96–1.12)		929	654	1.42 (1.32–1.54)
Age at start of follow-up
30–39	124	131	0.95 (0.80–1.12)		107	131	0.81 (0.69–0.96)		139	131	1.06 (0.89–1.26)
40–49	580	524	1.11 (1.02–1.21)		519	524	0.99 (0.91–1.08)		658	524	1.26 (1.15–1.37)
50–59	747	776	0.96 (0.90–1.03)		780	776	1.01 (0.94–1.08)		895	776	1.15 (1.07–1.24)
60–70	576	610	0.94 (0.87–1.02)		584	610	0.96 (0.88–1.04)		608	610	1.00 (0.92–1.08)
Family history of breast cancer
No	1530	1580	0.97 (0.92–1.02)		1483	1580	0.94 (0.89–0.99)		1597	1580	1.01 (0.96–1.06)
Yes	497	461	1.08 (0.98–1.18)		507	461	1.10 (1.00-1.20)		703	461	1.52 (1.39–1.67)
Breast biopsy
No	1290	1290	1.00 (0.95–1.06)		1341	1290	1.04 (0.98–1.10)		1587	1290	1.23 (1.17–1.30)
Yes	737	751	0.98 (0.91–1.05)		648	751	0.86 (0.80–0.93)		712	751	0.95 (0.88–1.02)

E, expected number of invasive breast cancers; O, observed number of invasive breast cancers; CI, confidence interval

Age-adjusted AUCs were 0.58 (0.56–0.59) for the BWHS model, 0.56 (0.55–0.57) for BCRAT, and 0.56 (0.55–0.57) for IBIS (Table 2). These differences were statistically significant, with p = 0.013 for BWHS versus BCRAT and p = 0.028 for BWHS versus IBIS. BWHS model AUCs were higher than AUCs from the other models in every age group.

Table 2.

Discriminatory accuracy of three breast cancer risk prediction calculators in data from 52,901 women in the Black Women’s Health Study followed for 20 years

Age at risk		AUC (95% CI)
	Cases	BWHS model	BCRAT model	IBIS model
30–39	131	0.64 (0.59–0.68)	0.62 (0.58–0.67)	0.62 (0.58–0.67)
40–44	221	0.58 (0.55–0.62)	0.56 (0.52–0.60)	0.56 (0.53–0.60)
45–49	303	0.56 (0.52–0.59)	0.54 (0.51–0.57)	0.54 (0.51–0.57)
50–54	388	0.58 (0.55–0.61)	0.58 (0.55–0.61)	0.56 (0.53–0.59)
55–59	388	0.57 (0.54–0.60)	0.56 (0.53–0.58)	0.56 (0.53–0.59)
60–64	337	0.57 (0.53–0.60)	0.54 (0.51–0.58)	0.55 (0.52–0.58)
65–70	273	0.59 (0.55–0.62)	0.56 (0.53–0.60)	0.55 (0.51–0.59)
Weighted average	2041	0.58 (0.56–0.59)	0.56 (0.55–0.57)	0.56 (0.55–0.57)
Difference from BWHS model AUC			δ = 0.014, p = 0.013	δ = 0.015, p = 0.028

AUC, area under the ROC curve; CI, confidence interval

Overall, among the observed breast cancer cases, 31.1% were classified as high-risk by the BWHS model, 27.1% by BCRAT, and 36.1% by the IBIS model (Online Appendix Fig. 1). Among non-cases, 17.8% (BWHS), 16.2% (BCRAT), and 23.3 (IBIS) were classified as high risk. Concordance among the models for individual women was poor: 15.5–25.6% of cases that were classified as high-risk in one model were not classified that way in another.

Discussion

This performance comparison of three absolute breast cancer risk prediction models in data from 50,235 Black women followed for 20 years indicated that both calibration and discriminatory accuracy metrics were better for the BWHS model than either the BCRAT or IBIS models. The differences in discriminatory accuracy were modest but statistically significant, with AUCs of 0.58 from BWHS versus 0.56 in each of BCRAT and IBIS.

Strengths of this research include the updating of exposures every 5 years and the large validation cohort sample size. The primary limitation is that several predictors used in the IBIS model and two used in the BCRAT model were not available in the validation study. This is a limitation in most validations of risk calculators because no validation cohort has data on predictors from all models [10–12]. Those variables are probably only minor contributors to absolute breast cancer risk due to their low prevalence (e.g. family history of ovarian cancer, BRCA1/2 gene mutations, Ashkenazi Jewish heritage). We set these predictors to their lowest risk level, which could explain the slight underestimation for BCRAT, but cannot explain the overestimation for IBIS. Of note, more detailed data on family history of breast cancer would likely have led to greater overestimation of risk in the IBIS model.

Paige et al. recently tested the BCRAT, IBIS, and a third model (Breast Cancer Surveillance Consortium) [13, 14] in data from women age 40–74 presenting for screening mammography in a multisite California health system [15]. Overall, the proportions of women with estimated 5 year risk ≥ 1.67% were similar across the models, but risk estimates for individual women varied, with 21–26% discordance in pairwise comparisons of the models. Similarly, in the present study, classification according to the 1.67% risk cut point was discordant for 15.5–25.6% of breast cancer cases in pairwise model comparisons.

Breast cancer risk prediction models using only questionnaire-based predictors have been shown to have limited discriminatory accuracy, with age-adjusted AUCs ranging from 0.54 to 0.61 [10–12, 15, 16]. Adding a PRS to such models markedly improves discrimination [3, 16–18]. We previously demonstrated that adding a PRS derived and externally validated in data from Black women improved discriminatory accuracy of the BWHS model [3]. PRS are not currently available for clinical use at the primary care level but could be incorporated when preventive medications are being considered for women already identified as high risk.

Prediction models incorporating mammographic features have shown great promise, with additional work underway [19–22]. Such models may become more important, given that U.S. women are now recommended to begin mammographic screening as early as age 40 [23]. However, breast cancer risk calculators using only clinical information are still very useful for women under age 40 to identify those who would most benefit from initiating screening early and among previously unscreened older women and those for whom mammography data are not available. Based on the present findings, we suggest that the BWHS calculator be used for Black patients.

Supplementary Information

Below is the link to the electronic supplementary material.

Author contributions

GZ carried out all analyses, prepared the tables, and co-wrote the manuscript with JRP. RMP participated in interpretation of the results, made suggestions for additional analyses, and provided edits for the manuscript. JRP conceived the hypothesis, designed the analysis, and co-wrote the manuscript with GZ.

Funding

Funding for this research was provided by NIH grants U01CA164974 and R01CA228357. Dr. Palmer also receives support from the Breast Cancer Research Foundation, the Susan G. Komen Foundation, and the Karin Grunebaum Cancer Research Foundation. The content of this study is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Data availability

The de-identified data used in this article may be shared upon approval of a data request by Black Women’s Health Study Data Access Committee and with appropriate human subjects approvals and data transfer agreements. For more information contact the corresponding author.

Declarations

Ethics approval and consent to participate

The Black Women’s Health Study (BWHS) has been approved by the Institutional Review Board of the Boston University Medical Center.

Consent for publication

BWHS participants give implied consent by completing biennial questionnaire about their health and habits and returning the questionnaires to the study offices. Each questionnaire is accompanied by a document that explains the elements of informed consent.

Competing interests

The authors declare no competing interests.