Impact of Artificial Intelligence System and Volumetric Density on Risk Prediction of Interval, Screen-Detected, and Advanced Breast Cancer

PURPOSE

Artificial intelligence (AI) algorithms improve breast cancer detection on mammography, but their contribution to long-term risk prediction for advanced and interval cancers is unknown.

METHODS

We identified 2,412 women with invasive breast cancer and 4,995 controls matched on age, race, and date of mammogram, from two US mammography cohorts, who had two-dimensional full-field digital mammograms performed 2-5.5 years before cancer diagnosis. We assessed Breast Imaging Reporting and Data System density, an AI malignancy score (1-10), and volumetric density measures. We used conditional logistic regression to estimate odds ratios (ORs), 95% CIs, adjusted for age and BMI, and C-statistics (AUC) to describe the association of AI score with invasive cancer and its contribution to models with breast density measures. Likelihood ratio tests (LRTs) and bootstrapping methods were used to compare model performance.

RESULTS

On mammograms between 2-5.5 years prior to cancer, a one unit increase in AI score was associated with 20% greater odds of invasive breast cancer (OR, 1.20; 95% CI, 1.17 to 1.22; AUC, 0.63; 95% CI, 0.62 to 0.64) and was similarly predictive of interval (OR, 1.20; 95% CI, 1.13 to 1.27; AUC, 0.63) and advanced cancers (OR, 1.23; 95% CI, 1.16 to 1.31; AUC, 0.64) and in dense (OR, 1.18; 95% CI, 1.15 to 1.22; AUC, 0.66) breasts. AI score improved prediction of all cancer types in models with density measures (P_LRT values < .001); discrimination improved for advanced cancer (ie, AUC for dense volume increased from 0.624 to 0.679, Δ AUC 0.065, P = .01) but did not reach statistical significance for interval cancer.

CONCLUSION

AI imaging algorithms coupled with breast density independently contribute to long-term risk prediction of invasive breast cancers, in particular, advanced cancer.

INTRODUCTION

Multiple artificial intelligence (AI) algorithms based on deep learning have been developed, which improve accuracy of breast cancer detection on mammography.^1-4 More recently, these algorithms have been considered for their use in near-term risk prediction and preliminary data suggest improved discrimination of breast cancer compared with conventional clinical risk models.^1,2

CONTEXT

• Key Objective

• Does an artificial intelligence (AI) algorithm designed for improved breast cancer detection on mammography contribute to long-term breast cancer risk prediction, in particular, for interval and advanced cancers, and in combination with breast density?

• Knowledge Generated

• In a nested case-control study of 2,412 invasive breast cancers and 4,995 matched controls, an AI score assessed on mammograms 2-5.5 years before cancer was associated with invasive breast cancer and similarly predictive of interval and advanced cancers and in dense and nondense breasts. Furthermore, the score improved prediction of all cancer types in models with density measures. AI imaging algorithms coupled with breast density independently contribute to long-term risk prediction of invasive breast cancers, including advanced cancers.

• Relevance (P.H. Frankel)

• This effort reports a computer-aided/AI mammogram-based risk model to improve risk estimates excluding breast cancer diagnosis within 2 years from a negative mammogram. This adds further evidence to support computer-aided imaging assessments and will help motivate prospective studies to disentangle improved earlier diagnosis from an improved risk model for subsequent cancer. Such prospective studies will also provide estimates that can more easily guide clinical practice.*

• *Relevance section written by JCO Associate Editor Paul H. Frankel, PhD.

There are limited data, however, for the performance of these AI algorithms in predicting long-term risk of invasive cancer and risk of advanced and interval cancers (defined as screening failures), which would most benefit from early detection and intervention. Interval cancers are known to be increased among women with dense breasts,⁵ because of decreased mammogram sensitivity.^6,7 Wanders et al⁸ recently validated an AI cancer detection system integrating area-based breast density to improve discrimination of interval cancer within 2 years of a negative mammogram. They did not have information on screen-detected cancers, or consider advanced cancers, or long-term risk. Understanding long-term risk prediction models that incorporate AI for advanced cancer will provide insight into their ability to guide screening strategies to reduce breast cancer morbidity and mortality.^9,10

In this article, we extend prior work to evaluate the contribution of an AI cancer detection system on the basis of deep convolutional neural networks combined with volumetric density measures to interval, screen-detected, advanced, and nonadvanced cancer risk. We hypothesize that the AI detection score, which reflects suspicious mammographic findings, will improve long-term risk prediction of invasive cancer above volumetric density measures and for advanced and interval breast cancers. We focus on assessing mammograms taken at least 2 years but not more than 5.5 years before the cancer to evaluate longer-term risk prediction.

METHODS

Study Sample

Study participants were from two retrospective case-control studies nested within large breast imaging cohorts from US-based breast screening practices (Fig 1). The San Francisco Mammography Registry (SFMR) participates in the National Cancer Institute–funded Breast Cancer Surveillance Consortium (BCSC).¹¹ The SFMR obtains annual institutional review board approval and passive permission for data collection and participant enrollment and data linkages for research purposes. The Mayo Clinic institutional review board approved a waiver of informed consent and Health Insurance Portability and Accountability Act authorization for participants from the breast screening practice who did not refuse permission to use their medical records for research (according to Minnesota Research Authorization).

FIG 1.

Study schema. ^aInvasive breast cancers were classified as advanced cancer and interval cancer, two primary end points for the study. These are not mutually exclusive categorizations. ^bUnable to classify because of missing clinical data (as seen in the Data Supplement [Supplemental Table 1]). DCIS, ductal carcinoma in situ.

The SFMR obtained for-processing digital screening examinations from Hologic Selenia machines at four facilities since 2006, which served as the underlying SFMR imaging cohort. Annual linkage to the California Cancer Registry identified 3,580 incident breast cancers reported from 2007 through 2017. Up to two control participants (n = 6,841) without previous breast cancer or breast implants were selected from the SFMR imaging cohort and matched to each case participant by age within 5 years, race, date of screening examination within 1 year, mammography machine, and facility.

For the Mayo Clinic screening cohort, digital images from Hologic Selenia machines were collected from women in the tristate region of Minnesota, Iowa, and Wisconsin seen at the Rochester, Minnesota facility from 2009 through 2017. Annual linkage to the Mayo Clinic tumor registry identified cases of incident breast cancer reported through December 2017 (n = 1,176). Up to three control participants (n = 3,239) without previous breast cancer or breast implants were selected from the Mayo Clinic cohort and matched on the same criteria as above.

Primary analyses were restricted to invasive cases only with mammograms between 2 and 5.5 years before diagnosis or last follow-up, resulting in 1,700 cases and 3,109 matched controls from SFMR and 712 invasive cancers and 1,886 matched controls from Mayo (Fig 1). For participants with multiple images within the time window of 2-5.5 years before cancer, only one mammogram (closest to 5 years before) was used for all cases and their matched controls from both studies.

Interval cancer was defined as invasive breast cancer occurring within 12 months of a negative mammography result (Breast Imaging Reporting and Data System [BI-RADS] 1 or 2). Screen-detected cancer was defined as invasive cancer occurring within 12 months of a positive screening mammography result (BI-RADS 0) with a final BI-RADS of 3, 4, or 5. Advanced cancer was defined using American Joint Committee on Cancer (AJCC) prognostic pathologic stage II or higher since it most accurately predicts breast cancer death.¹⁰ All advanced cancers were included regardless of mammography BI-RADS assessment. Women without mammograms within the appropriate time window (for defining interval and screen-detected cancers) or tumor characteristics (for defining advanced cancers) were classified as missing (Fig 1).

AI Cancer Detection System

A validated AI detection system (Transpara, version 1.7.0; ScreenPoint Medical) was used to provide a score from 1-10, representing an increasing probability of malignancy at the time of breast screening.^3,4,12,13 The system is based on deep convolutional neural networks and automatically detects signs of malignancy on digital mammography and digital breast tomosynthesis.⁴ All four mammogram views were deidentified and processed in a blinded fashion without knowledge of clinical information to arrive at the malignancy score.

BI-RADS Breast Density and BI-RADS Assessment

Clinical BI-RADS density was classified into four categories as part of routine clinical practice: (a) almost entirely fatty, (b) scattered fibroglandular densities, (c) heterogeneously dense, and (d) extremely dense. Volumetric density was assessed in a blinded fashion using Volpara, version 1.5.4, a fully automated method that uses the measured breast thickness and x-ray attenuations in the for-processing image to create estimates of dense volume (DV) and volumetric percent density (VPD) measures for each woman, incorporating all four mammogram views,¹⁴ as performed in the clinical setting.

In addition, clinical BI-RADS assessment that describes mammogram interpretation (0 = incomplete, 1 = negative, 2 = benign findings, 3 = probably benign, 4 = suspicious abnormality, and 5 = highly suspicious of malignancy) was available from the clinical review of mammograms from the Mayo Clinic cohort only.

Statistical Analyses

We used an existing study sample designed to examine the association of automated density measures with breast cancer risk.^14,15 Descriptive characteristics were compared by case-control status, for both studies combined and separately; Kruskal-Wallis and chi-squared tests were used to examine differences by case status.

Because of the matched study design, conditional logistic regression was used to examine primary associations of AI score, breast density measures (BI-RADS density, and log-transformed VPD%, DV), and breast cancer risk, including invasive breast cancer, advanced/nonadvanced cancer, and interval/screen-detected cancer and secondarily, ductal carcinoma in situ (DCIS). Odds ratios (ORs) and 95% CIs were estimated. C-statistics (and 95% CIs) were also calculated as a measure of discriminatory accuracy or AUC. Models assumed a linear trend for Transpara, VPD%, and DV and an ordinal trend for BI-RADS density. The C-statistics accounted for the matched data by taking the ratio of the total number of concordant pairs and the total number of pairs, considering only pairs within matched sets.¹⁶

Subgroup analyses were performed to examine differences in AI score and breast cancer associations by age (<60 and 60+ years), dense breasts (BI-RADS a, b v c, d), BMI (<25, 25-30, and 30+), and race/ethnicity (Asian, Black, Hispanic, White, and Others). We also examined models with AI score and BI-RADS assessment, using similar approaches, among the Mayo Clinic studies, where data were available. All models were adjusted for continuous age and BMI at the time of mammogram. Stratified analyses of BI-RADS density required breaking the matched data and were analyzed using unconditional logistic regression, adjusted for the matching factors.

We compared the performance of nested models (ie, addition of AI to breast density models) using the likelihood ratio test (LRT; P_LRT) to evaluate improvement in model fit, as previously described.¹⁷ We also evaluated improvement in discriminatory accuracy by comparing AUCs between models on the basis of 1,000 bootstrap samples. Each bootstrap sample was constructed by selecting n-matched sets of individuals chosen at random with replacement, with n equal to the number of cases in the specific analysis. Matched C-statistics were calculated as described above for each model using each bootstrap sample. For each model comparison, the difference in model C-statistics was calculated over all 1,000 samples; 95% confidence limits and normal distribution P values (P_AUC) were calculated using the standard deviation of this difference as the SE estimate. We used the same bootstrapping approach to test differences in AUCs across subgroups (P_AUC). SAS version 9.4 (Cary, NC) was used for analyses, and two-sided P values < .017 (0.05 ÷ 3) were considered statistically significant to account for multiple testing of invasive, interval, and advanced cancer.

RESULTS

A total of 2,412 participants with invasive breast cancer and 4,995 matched control participants were eligible from the two screening cohorts (Table 1; Fig 1; and Data Supplement [Supplemental Table 1], online only). Of the invasive cases, 286 were interval cancers, 1,670 were screen-detected, and 456 were missing, most of whom (83%) did not have imaging within the required time points for classification; there were 289 advanced cancers, 2,012 nonadvanced cancers, and 111 missing an essential tumor characteristic required for AJCC prognostic stage classification (Data Supplement [Supplemental Table 2]). The median follow-up time was 4.1 years, and 47% of participants were older than age 60 years. BMI, breast density, and AI scores were higher among cases versus controls (Table 1). AI detection scores were weakly correlated with breast density measures, age, and BMI (Data Supplement [Supplemental Table 3]). Mayo participants were more likely to be older, be White, have a higher BMI, and have nondense breasts than SFMR participants (Data Supplement [Supplemental Table 1]).

TABLE 1.

Demographics, Breast Density, and AI Score Distributions by Invasive Breast Cancer (case-control) Status

AI Malignancy Score and Breast Cancer Risk

Adjusted for age and BMI, AI score was associated with invasive breast cancer (OR, 1.20; 95% CI, 1.17 to 1.22 per unit increase; AUC, 0.63; 95% CI, 0.62 to 0.64). ORs and discriminatory accuracy were similar for interval and screen-detected cancers (P_AUC = .93; Table 2), advanced and nonadvanced cancers (P_AUC = .67; Table 2), and dense and nondense breasts (P_AUC = .19; Fig 2). There was an increased cancer risk and higher discrimination associated with AI score among women age 60 years and older compared with those age younger than 60 years (ΔAUC, 0.04; 95% CI, 0.01 to 0.07; P_AUC = .01; Fig 2) but no statistically significant difference in discrimination by obesity (BMI > 30 relative to normal or underweight BMI < 25 [ΔAUC, 0.03; 95% CI, 0.00 to 0.07; P_AUC = .05]) or race (all P_AUC > .37; Fig 2).

TABLE 2.

Association of Transpara Artificial Intelligence Malignancy Score With Breast Cancer Risk

FIG 2.

Associations (ORs, 95% CI) of Transpara AI malignancy score with breast cancer by age, density, BMI, and race/ethnicity. AI, artificial intelligence; OR, odds ratio.

Association of Breast Density Measures With Breast Cancer Risk

Adjusted for age and BMI, all breast density measures were associated with invasive breast cancer (Table 3). Unlike the AI score, there were stronger associations and discrimination of interval (v screen-detected) cancer for all density measures (P_AUC values < .02; Table 3). For advanced versus nonadvanced cancer, the ORs and AUCs were similar for all density measures (P_AUC values > .14; Table 3).

TABLE 3.

Association of Transpara AI Score and Breast Density Measures With Invasive Breast Cancer and With Screen-Detected and Interval Cancers and Advanced and Nonadvanced Cancers

Contribution of AI to Breast Density Measures

The inclusion of AI scores combined with either clinical or volumetric density measures improved prediction of all cancer types compared with models with density alone (Table 3; All P_LRT < .001). For advanced cancers, inclusion of AI scores with DV also resulted in statistically significant improvement in discriminatory accuracy relative to models with DV alone (ΔAUC, 0.065; 95% CI, 0.015 to 0.114; P_AUC = .01). Similar improvement was observed with addition of AI scores to VPD (ΔAUC, 0.067; 95% CI, 0.018 to 0.116; P_AUC = .007) but did not reach statistical significance with BI-RADS (ΔAUC, 0.045; 95% CI, –0.006 to 0.097; P_AUC = .09; Table 3). Improvement in discrimination was also seen with AI scores added to density models for nonadvanced cancers, for all three density measures (VPD, DV, and BI-RADS) compared with models with density alone (P_AUC values < .001; Table 3).

As expected, all AUCs for screen-detected cancer improved with inclusion of AI scores for models with VPD, DV, and BI-RADS density (P_AUC values ≤ .001; Table 3). However, for interval cancers, the contribution of AI scores resulted in smaller gains in AUC, which did not reach statistical significance for models with any density measure (Table 3).

Secondary Analyses

We examined the association of the AI score with 811 in situ cancers and 1,663 matched controls with mammograms 2-5.5 years before breast cancer (Fig 1). The associations of AI score and in situ breast cancer were like invasive cancer (OR, 1.16; 95% CI, 1.13 to 1.20; AUC, 0.62; 95% CI, 0.59 to 0.64) and showed similar improvement in risk prediction (P_LRT < .001) and discrimination (P_AUC values < .01) with the addition of AI to all density measures (Data Supplement [Supplemental Table 4]).

We also examined the contribution of AI score to the BI-RADS assessment and breast cancer association. As expected, BI-RADS assessment was associated with breast cancer (Data Supplement [Supplemental Tables 5 and 6]), with increased odds of breast cancer among all categories relative to BI-RADS 1 or negative findings; the AUC was 0.566 (95% CI, 0.543 to 0.587). However, across all BI-RADS assessments, the mean AI score was higher among cases than controls (Data Supplement [Supplemental Figure 1]). The addition of the AI score resulted in attenuation of the risk estimates for BI-RADS assessment and a higher AUC (0.643; 95% CI, 0.621 to 0.664). Importantly, the AI score and breast cancer association did not change in models with (OR, 1.20; 95% CI, 1.16 to 1.24) or without BI-RADS assessment (OR, 1.19; 95% CI, 1.15 to 1.23).

In addition, we examined how Transpara performed in our study population closer to the cancer for comparison with prior studies. Of our invasive case-control sample, 1,010 cases and 2,140 controls also had mammograms within 2 years of the cancer (cases) or follow-up date (controls). We found statistically significant, but stronger association and higher discrimination of AI models (without density measures) within 2 years of cancer (OR, 1.30; 95% CI, 1.26 to 1.34; AUC, 0.70; 95% CI, 0.68 to 0.72) relative to AI models assessed on mammograms 2-5.5 years before the cancer (OR, 1.20; 95% CI, 1.17 to 1.24; AUC, 0.63; 95% CI, 0.61 to 0.65).

DISCUSSION

The Transpara AI algorithm assessed on mammograms 2-5.5 years before breast cancer showed similar associations with invasive cancer and discriminatory accuracy in dense and nondense breasts and for interval versus screen-detected and advanced versus nonadvanced cancers. These findings imply that the AI score is robust for long-term risk prediction by severity of cancer and extent of density. Furthermore, the incorporation of AI score into models combined with breast density showed improved prediction for all invasive cancer types, with the highest discriminatory accuracy seen for advanced and interval cancers, suggesting that the combination will be important for prediction of these cancers with worse prognosis.

Previous studies have developed and validated AI algorithms, including Mirai and iCAD, that show high discrimination of breast cancer^1,2,18-20 (invasive cancer and DCIS combined) with AUCs above 0.75, surpassing that of established clinical risk models, such as Gail and Tyrer-Cuzick/IBIS.^1,2 iCAD was developed for a near-term 2-year risk prediction, whereas Mirai was developed for risk prediction up to 5 years. Although our AUCs for longer-term risk are lower than those reported by Mirai, it is difficult to directly compare for multiple reasons. First, we excluded mammograms within 2 years of the cancer, where the algorithms show optimal performance, whereas these previous studies using Mirai and iCAD did not. When we investigated the performance of Transpara only using mammograms within 2 years of the cancer, we show higher discrimination (AUC, 0.70), comparable with these other studies. Second, our case-control study design matched closely on age, mammogram date, and race/ethnicity, all of which limit the ability to examine their contributions to AUC and direct comparison across other study designs. Finally, we evaluated invasive breast cancer that may benefit from targeted screening, whereas Mirai and iCAD algorithms include invasive cancer and DCIS. Ensuring accurate prediction of the cancers with worst prognosis, that is, advanced cancer defined as pathologic prognostic stage, should be a priority of AI risk models to allow early intervention or prevention. In fact, the TMIST clinical trial is using study-defined definition of advanced cancer as its primary end point²¹ although their definition has been shown to be less predictive of breast cancer death than prognostic pathologic stage.⁹ We found that the Transpara AI score assessed on mammograms 2-5.5 years before cancer was similarly predictive of advanced and nonadvanced cancers. When AI score was incorporated with breast density measures, the AUC improved for both cancer types. It will be important to evaluate the contribution of the AI scores to the new BCSC model for advanced cancer²² that includes age, breast density, race/ethnicity, first-degree family history of breast cancer, history of benign breast biopsy, BMI, menopausal status, and screening interval, with the goal of improving identification of those most needing supplemental or alternative screening strategies to biennial mammographic screening. With the already high AUCs for advanced cancer seen in our study (ie, AUC = 0.679) for AI score with only DV, BMI, and age (albeit with matching), we expect strong discrimination of advanced cancer with the addition of these other model factors.

Although the AI score significantly increased prediction across all types of cancer, there was less evidence for improvement in discrimination of interval cancers with AI score added to density measures. The smaller gains in AUC likely reflect the strong association of breast density with interval cancers, even when assessed years before the cancers.¹⁵ A prior report⁸ showed that inclusion of the AI score with density measures assessed within 2 years of the cancer resulted in improved discrimination of interval cancer but differed from our study in its focus on near versus long-term prediction, its definition of interval cancers over a 2-year versus 1-year period, and use of area-based density²³ versus clinical BI-RADS density or volumetric measures, which may show stronger associations with interval cancer than area-based density measures. Both reports suggest, though, that breast density measures may be complementary to AI detection and risk and need to be considered for prediction of all invasive cancer types.

With the advent of new AI and other imaging risk factors for breast cancer risk, it is important to consider clinically versus statistically significant improvement in discrimination for risk prediction. We define clinically important changes in AUC using prior work on the addition of new risk factors, including benign breast disease (BBD) and the polygenic risk score (PRS), to established risk models for invasive breast cancer. Inclusion of BBD to the BCSC risk model resulted in a change in AUC of 0.65 to 0.66,²⁴ and inclusion of the breast cancer PRS to the BCSC model resulted in a change in AUC from 0.66 to 0.69.²⁵ Thus, we propose a change in AUC of 0.01 to 0.03 as clinically significant. Importantly, the inclusion of the Transpara AI score to all models with breast density surpasses this threshold across all invasive cancer types.

Finally, our study cannot disentangle whether the AI score on mammograms 2-5.5 years before the cancer is reflective of suspicious findings underlying breast cancers missed at review or identifying suspicious changes that may develop into a detectable breast cancer in the future. Our models that incorporate both BI-RADS assessment and the AI score show that the AI score is robust and contributes additional risk information not contained in the BI-RADS assessment. Importantly, some of the high AI scores, reflective of suspicious changes, are subsequently associated with a diagnosis of advanced cancer or aggressive tumors that are not detectable until rapid growth into a detectable mass. It is an advantage of AI algorithms to identify suspicious areas that could develop into advanced cancer since it provides the opportunity to apply screening strategies among those at high risk of these cancers to potentially increase detection of these tumors at an earlier stage.

We acknowledge several limitations in our study including focus on two-dimensional mammography instead of digital breast tomosynthesis or three-dimensional mammography, the focus on one AI algorithm, and the limited power to evaluate groups by race or ethnicity. In particular, our statistical power for comparisons within advanced cancer and interval cancer groups might have limited our ability to detect smaller contributions to discriminatory accuracy of imaging-based AI score to density models. Although racial and ethnic differences have not been seen to date for Mirai,¹ there have been differences seen in volumetric density by race,²⁶ and it will be important to consider future analyses within multiple races and ethnicities. In addition, the retrospective and case/control nature of the study design limits absolute risk estimation and enriches for cancer, not allowing for actual estimates for the impact of AI when in wider use; prospective cohort studies will allow the estimation of absolute as opposed to relative risk and evaluate the impact of the AI score on relevant clinical thresholds.

In conclusion, the Transpara AI score improves long-term risk prediction when combined with clinical risk factors including breast density for overall invasive cancers, screen-detected, advanced, and nonadvanced cancers. For interval cancers, density measures remain of greatest importance for discrimination, even years before the cancer. Incorporating both density and AI scores into long-term risk models will be important for accurate prediction of invasive cancer outcomes, in particular, advanced cancer.

AUTHOR CONTRIBUTIONS

Conception and design: Celine M. Vachon, Aaron D. Norman, Stacey J. Winham, Karla Kerlikowske

Financial support: Celine M. Vachon, Karla Kerlikowske

Administrative support: Aaron D. Norman, Karla Kerlikowske

Provision of study materials or patients: Celine M. Vachon, Karla Kerlikowske

Collection and assembly of data: Celine M. Vachon, Christopher G. Scott, Aaron D. Norman, Matthew R. Jensen, Karla Kerlikowske

Data analysis and interpretation: Celine M. Vachon, Christopher G. Scott, Sadia A. Khanani, Matthew R. Jensen, Carrie B. Hruska, Kathleen R. Brandt, Stacey J. Winham, Karla Kerlikowske

Manuscript writing: All authors

Final approval of manuscript: All authors

Accountable for all aspects of the work: All authors

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

Impact of Artificial Intelligence System and Volumetric Density on Risk Prediction of Interval, Screen-Detected, and Advanced Breast Cancer

The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated unless otherwise noted. Relationships are self-held unless noted. I = Immediate Family Member, Inst = My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO's conflict of interest policy, please refer to www.asco.org/rwc or ascopubs.org/jco/authors/author-center.

Open Payments is a public database containing information reported by companies about payments made to US-licensed physicians (Open Payments).