Artificial Intelligence in Breast Cancer Screening: Evaluation of FDA Device Regulation and Future Recommendations

Abstract

Importance:

Contemporary approaches to artificial intelligence (AI) based on deep learning have generated interest in the application of AI to breast cancer screening (BCS). The U.S. Food and Drug Administration (FDA) has approved several next-generation AI products with an indication for BCS in recent years. However, questions related to AI’s accuracy, appropriate use, and clinical utility remain.

Objectives:

To describe FDA’s current regulatory process for AI products, summarize the evidence used to support FDA clearance and approval of AI products indicated for BCS, consider the advantages and limitations of current regulatory approaches, and suggest ways to improve on the current system.

Evidence:

We reviewed premarket notifications and other publicly available FDA documents used in the clearance and approval of AI products with an indication for BCS from January 1, 2017 through December 31, 2021.

Findings:

Nine AI products indicated for BCS suspicious lesion identification and mammogram triage were included. The majority of products were cleared through the 510(k) pathway, and all clearances were based on previously collected, retrospective data. Six products used multicenter designs. Enriched data were used for eight devices, and four devices lacked details on whether products were externally validated. Test performance measures, including sensitivity, specificity, and area under the curve, were the main outcomes reported. The majority of devices used tissue biopsy as their gold standard for BCS accuracy evaluation. Other measures of clinical utility, including cancer stage at detection, interval cancer detection, or other outcomes, were not reported for any of the devices.

Conclusions and Relevance:

Important gaps in reporting of data sources, dataset type, validation approach, and clinical utility assessment were identified. As AI-assisted reading becomes more widespread in BCS and other radiologic exams, strengthened FDA evidentiary regulatory standards, development of post-marketing surveillance, a focus on clinically meaningful outcomes, and stakeholder engagement of will be critical for ensuring safety and efficacy of these products.

Introduction

Screening mammography has a number of limitations including imperfect sensitivity and specificity and inter-reader variability. Artificial intelligence (AI) applications have been proposed both for improving accuracy (sensitivity and specificity) and for reducing radiologist reading time and cognitive load by, for example, identifying regions of interest or triaging images with abnormal findings. While AI-based technologies may hold promise, there are still many unanswered questions about AI’s accuracy, efficacy, appropriate use, and impact on health outcomes.^1–3 Despite these questions, a number of AI-based products for breast cancer screening (BCS) have already been authorized by the US Food and Drug Administration (FDA) and are available for clinical use to assist radiologists.^4,5

We characterized FDA’s regulatory evaluation process for marketed AI products and summarized the publicly available evidence supporting AI product clearance and approval for BCS, with particular attention to study design, population/dataset, validation, gold standards, and outcomes. Additionally, we consider the advantages and limitations of current regulatory approaches. Finally, we suggest ways to strengthen the current system to ensure that products ultimately used in routine BCS contribute to improved health.

A Brief History of Automated Image Analysis in Breast Cancer Screening

The development of computer software for radiographic image analysis began as early as the 1960s in the US.^6,7 However, it was not until three decades later, in 1998, that FDA approved the first computer-aided detection (CAD) system for screening mammography, which used early AI algorithms to identify suspicious lesions for second review by radiologists.^8,9 Although evidence supporting CAD was limited to small studies focused on test characteristics, Congress mandated that Medicare cover CAD.¹⁰ By 2016, about 92% of mammography facilities in the United States had adopted CAD into their clinical practice.¹¹ However, once adopted, these CAD systems did not appear to improve diagnostic accuracy, generating more false positives and subsequent unnecessary clinical workup.^11,12 CAD also increased detection of ductal carcinoma in situ (DCIS), a precancerous lesion with low risk of mortality, and increased the biopsy rate compared to interpretation without CAD.^13,14 Importantly, CAD has not been shown to reduce mortality from breast cancer and has been associated with rising screening costs.¹² Despite the historical shortcomings of CAD, interest in next-generation AI for BCS has grown substantially in recent years with the development of convolutional neural networks (CNNs), which use a filter methodology to identify image features and have greater accuracy than other methods.^15,16 CNNs have been successfully applied to a wide range of image analysis applications and have renewed enthusiasm for automated mammography interpretation.^11,16–19

Current FDA Regulation of AI

Although AI-based diagnostic products are distinct from typical therapeutic medical devices, FDA has recognized that AI has the potential to impact human health. FDA has adopted a standard known as Software as a Medical Device (SaMD) for identifying software that may be regulated under its authority. SaMD is defined as “software intended to be used for one or more medical purposes that performs these purposes without being part of a hardware medical device.”²⁰

Products classified as SaMD may be reviewed through three existing FDA medical device pathways: 510(k), De Novo, and Premarket Approval (PMA). The specific pathway chosen for review depends on whether there is a predicate device and the device’s risk (Figure).²¹ PMA is used for the highest risk devices and requires “valid scientific evidence” to inform safety and effectiveness. This evidence can comprise a range of study designs, from randomized, controlled trials to single arm studies.²² The De Novo pathway provides a regulatory approach to clearance for novel low to moderate risk devices without an established predicate on the market.²³ The majority of AI software products reviewed by FDA have been cleared through 510(k) review.⁵

Figure. Overview of U.S. FDA Medical Device Regulatory Process with AI in BCS Product Examples.

Most FDA-approved AI products to date have gone through 1510(k) clearance. Abbreviations: AI, artificial intelligence; AUC, area under the curve; BCS, breast cancer screening; DBT, digital breast tomosynthesis; MRMC, multi-reader, multi-case; MRI, magnetic resonance imaging

FDA Standards under the 510(k) Pathway

Products submitted for clearance through the 510(k) pathway must demonstrate “substantial equivalence” to a predicate device that is already FDA approved, cleared, or grandfathered in. For this to occur, the agency must deem the intended use of the new device and predicate to be the same. The technological characteristics of the devices must either also be deemed the same or not raise questions pertaining to safety and effectiveness.²⁴

FDA has also provided specific guidance for demonstrating substantial equivalence for AI applications for computer-aided detection (CADe), for which assessment of performance is central.²⁵ A range of performance metrics may be used, with a receiver operating characteristic curve summary metric (e.g., area under the curve) recommended as the primary endpoint, and sensitivity and specificity as secondary endpoints. Performance may be assessed through a variety of study designs including retrospective reader studies, prospective reader studies, and stress tests, which include difficult-to-read images. Multi-reader, multi-case (MRMC) study design, in which radiologists read a number of mammograms with AI and then, after a washout period, read the same mammograms without (or vice versa) is cited as commonly used to assess CADe clinical performance. Importantly these guidelines do not apply to computer-aided diagnosis (CADx) devices, which are intended to provide more detailed diagnostic information.

Other Regulatory Approaches under Consideration:

Although FDA has used its medical device regulatory pathways for SaMD, AI products have features that create unique challenges for regulation. In particular, AI may use adaptive algorithms, which are dynamic and can change in response to new information. Adaptive algorithms are difficult to regulate in the current framework because the product initially reviewed may be different from the one a consumer ultimately encounters.²⁶ In addition, the current regulatory framework is cumbersome even for locked algorithms because it does not allow for rapid iterative updates in response to new information about use or performance.

FDA has proposed a voluntary program known as the Software Pre-Cert Pilot Program, which is designed to address the specific challenges of regulating SaMD. The Pre-Cert program would focus on vetting software developers and processes, and would allow for streamlined review of some products from participating developers.²⁷ The program also calls for developers to engage in “real world performance analytics” to evaluate product performance once in use. Drawing on this framework, FDA has also proposed an approach to regulating adaptive algorithms in which FDA would similarly evaluate the quality of the software developer and require a priori plans for re-review by regulators when there are significant changes in intended use and performance.^4,28,29

Characteristics of Evidence Used in FDA Clearance of AI Products for Breast Cancer Screening

Because AI products for BCS may come to market via several different pathways, data supporting the efficacy of these products may be varied. We assessed the strength of evidence supporting AI products for BCS that have been cleared and to identify systematic or consistent gaps in evidence. We specifically focused on AI products with an indication for suspicious lesion identification or mammogram triage. Understanding and assessing the evidence supporting these products will be critical as more products come to market, existing products gain market share, and the presence of next-generation AI in BCS grows.

To this end, we reviewed all products cleared or approved by FDA Center for Devices and Radiological Health via radiology panels for the 510(k), De Novo, and PMA review pathways between January 1, 2017 and December 31, 2021 to focus our review on next-generation AI products. BCS indication determination was made from product name or from the product summary if not apparent from product name. We considered products to use AI if the device classification name included “radiological computer assisted detection” or “diagnosis software for lesions suspicious for cancer.” If the device classification name included “radiological image processing system,” the product summary was consulted for possible inclusion as an AI product. We cross-referenced our product list with published databases of FDA-approved AI products.^2,4,5,30 We also searched PubMed, arXiv and medRxiv for published reports of AI products and cross-referenced products with FDA listings.³ If multiple versions existed for a product, we included only the initial version of the product.

We abstracted information about the evidence used to support product clearance from publicly available data including FDA premarket notifications, decision summaries, and summaries of safety and effectiveness. We specifically evaluated the clearance or approval pathway used, study characteristics including study design, data source and type, gold standards, and outcomes. We evaluated whether products were clearly noted to be externally validated during reader testing, in which diagnostic accuracy is assessed on a sample independent from that used in development.

Product Overview:

We identified 9 products with applications in BCS (Table; eTable). Of these products, seven were cleared through the 510(k) pathway, one was approved through the De Novo pathway, and one was approved through full premarket review. Products were designed for use on varied imaging modalities including full-field digital mammography (FFDM), digital breast tomosynthesis (DBT), and magnetic resonance imaging (MRI). Importantly, all products are indicated for use by radiologists in an assistive or concurrent reading capacity rather than a diagnostic capacity.

Table.

Summary of Data Used for FDA Clearance or Approval of AI Products Indicated for Suspicious Lesion Identification or Triage during Breast Cancer Screening

Product name (Company)	Year of FDA clearance or approval	FDA pathway	Predicate device (Company)	Population	Dataset type	Validation type	Outcomes
PowerLook Tomo Detection (iCAD, Inc.)	2017	Pre-market approval	--	Multicenter	Enriched	External	Sensitivity, specificity, AUC, recall rate, radiologist reading time
QuantX (Quantitative Insights, Inc.)	2017	De Novo	--	Multicenter	Enriched	External	Sensitivity, specificity, AUC
Transpara (ScreenPoint Medical BV)	2018	510(k)	OsteoDetect (Imagen Technologies, Inc.)	Multicenter	Enriched	Not reported	Sensitivity, specificity, AUC, radiologist reading time
cmTriage (CureMetrix, Inc.)	2019	510(k)	ContaCT (Viz.AI)	Multicenter	Enriched	External*	Sensitivity, specificity, AUC, recall rate, processing time
Genius AI Detection (Hologic, Inc.)	2020	510(k)	PowerLook Tomo Detection V2 Software (iCAD, Inc.)	Not reported	Enriched	Not reported	Sensitivity, AUC, recall rate, radiologist reading time
HealthMammo (Zebra Medical Vision, Ltd.)	2020	510(k)	cmTriage (CureMetrix, Inc.)	Multicenter	Enriched	Not reported	Sensitivity, specificity, AUC, processing time
Mammoscreen (Therapixel)	2020	510(k)	Transpara (ScreenPoint Medical BV)	Single center	Not reported	Not reported	Sensitivity, specificity, AUC, radiologist reading time
Lunit INSIGHT MMG (Lunit, Inc.)	2021	510(k)	Transpara (ScreenPoint Medical BV)	Not reported	Not reported	External	Sensitivity, specificity, AUC, recall rate
Saige-Q (DeepHealth, Inc.)	2021	510(k)	cmTriage (CureMetrix, Inc.)	Multicenter	Enriched	External	Sensitivity, specificity, AUC, processing time

Asterisk (*) denotes testing in populations or datasets that were likely to be independent from development datasets, based on developers’ descriptions. Abbreviation: AUC, area under the curve

Characteristics of Studies Used to Support Clearance:

All nine products were cleared or approved on the basis of clinical studies, all of which used previously collected datasets. Of these, all products with an indication for suspicious lesion identification (n=6) used a MRMC design. The remaining three products were intended for triage (i.e., identification of potentially abnormal lesions to improve workflow) and used a blinded multicenter study design in which performance was compared to historical controls. Six products were evaluated using data from two or more clinical sites; one product was evaluated using data from one clinical site, and the remaining (22%) products did not have information reported on the number of sites. Seven products were examined with datasets enriched for breast cancer cases, and the remainder (22%) did not have information reported on dataset type. Five products appeared to be externally validated.

Characteristics of Reported Outcomes:

Publicly available FDA documents that provided quantitative outcome measures primarily reported test performance measures and radiologist reading time. All products reported sensitivity, specificity, and/or area under the curve (AUC). Seven reported processing or radiologist reading time. Three products also reported recall rate (i.e., the proportion of exams for which follow-up imaging was recommended). Eight reported biopsy as the gold standard for cancer cases, and the remainder (11%) did not report information on how cancer cases were ascertained. Additionally, six reported a follow-up period for non-cancer cases, and the remainder (33%) did not report information on non-cancer case ascertainment. None of the studies examined clinical outcomes, such as long-term cancer outcomes, cancer stage, or interval cancer detection.

Limitations:

We note that although we reviewed publicly available data, companies may have collected and/or submitted more extensive materials supporting product clearance with greater detail on study design, validation, populations studied, etc.

Advantages and Limitations of Current Standards for Device Approval

Currently, the majority of FDA-approved AI products for BCS have reported test accuracy for identifying breast cancer as the key metric for demonstrating substantial equivalence. Accuracy is an essential feature for any AI product designed to detect breast cancer, and a product that is less accurate than the current standard would clearly be detrimental. Evaluating accuracy also has other advantages. Test characteristics are relatively simple to measure, can be captured in a short time horizon, and are simple to compare between a device and the predicate technology or status quo approach. A focus on performance can be aligned with outcomes that improve care for patients. For example, improved specificity reduces false positives, thereby limiting the number of women who have to return for additional workup.

Current approaches to demonstrating substantial equivalence, though, have several important weaknesses. First, studies supporting the approval of AI products currently on the market for BCS incorporate features that increase the risk of bias when assessing performance. For example, all data submitted for product approval relied on previously assembled annotated datasets. This approach risks potential ascertainment bias if breast cancer is only detected in cases when a human reader has recommended biopsy.¹ A number of studies also used enriched datasets, which contain more true positive cases than would be seen in a typical screening population. While they are convenient and increase statistical power, enriched datasets do not reflect the typical spectrum and prevalence of breast cancer and can lead to an AUC higher than what would be observed in real-world conditions.^31–33 Thus, a number of common features of study design may lead to bias that could result in overestimation or underestimation of test accuracy.

Second, studies of test performance may incorporate design features that limit generalizability. Ideally, AI products are externally validated using a distinct dataset. Products that reuse data for development and validation may be overfit and perform poorly when applied to other populations. Relatedly, not all products were studied using data from multiple clinical sites. Reliance on a single center or population may mean that results do not generalize across different patient populations with different demographics, or different sites where clinical practice differs.⁴

Lastly, and perhaps most importantly, focusing on cancer detection does not necessarily translate to improved health. First, increased sensitivity may mean that some breast cancers are diagnosed earlier, but simply diagnosing a breast cancer earlier does not always improve outcomes. Second, greater sensitivity could actually result in harm by way of more false positive tests, resulting in more callbacks, biopsies, costs, and associated distress. AI may also contribute to overdiagnosis of indolent breast cancers that may not affect women in their lifetimes. Third, AI may miss clinically important, aggressive breast cancers while finding slower-growing tumors or DCIS—a change that would not be noticed if only focusing on performance. Lastly, AI may influence radiologist practice patterns in unpredictable ways that ultimately change diagnostic outcomes, but are not captured in limited, controlled studies of test accuracy.

Recommendations and Future Directions

Improving Evidentiary Standards

In the next decade, AI products are likely to become more common for BCS applications and many other diagnostic and therapeutic uses in radiology and other fields. It is crucial to consider whether current regulatory standards offer sufficient protection for patients against products that ultimately provide limited utility or are even harmful. Within its current statutory authority, FDA has the ability to alter evidentiary standards for AI product clearance by including specific requirements for study design, outcomes, populations included in studies, and approaches to validation. FDA could also consider modifying its voluntary guidance to which AI product manufacturers are not bound, but have strong incentive to adhere.

Should FDA require randomized trials of AI products for breast cancer screening prior to clearance or approval? Prospective randomized trials would address many of the shortcomings of current approaches to evaluation of AI products. Trials, however, are typically expensive and complex, and require significant expertise and engagement from multiple groups. In addition, trials may not be well suited to products like AI, which may have frequent updates or change over time. Proposals from FDA for alternate approaches to clearance, such as the Pre-Cert program or the proposed framework for regulating AI/machine learning, shift emphasis toward evaluating processes rather than products in response to concerns that AI products are not well suited to trials and traditional evaluation frameworks.^27,28,34

However, without trials, we may implement technology based on evidence that incorporates bias and without a full understanding of potential risks and benefits. Pragmatic trial designs, such as stepped wedge trials, point-of-care trials, or other designs emphasizing provider or practice-level randomization, can address some clinical trial challenges. Health system networks that leverage common data models for electronic health record data infrastructure, such as PCORnet or NESTcc, might be useful to support trials in real-world clinical environments to assess effectiveness, given questions about how AI products might perform once adopted. However, even if feasible, clinical trials would require significant coordination, along with substantial investment from developers.

In addition, FDA could consider strengthening requirements for and reporting of other study design features. Importantly, greater diversity of study settings and participants is needed. The majority of the data that supported FDA clearance of AI products for BCS did not report the race and ethnicity of the patients included. However, most available training datasets used in AI development contain mammograms largely from White women.^35–37 The inclusion of diverse populations when developing AI products is critical to ensure performance is maintained across populations.^38,39 Clinical diversity is also important. Mammographic performance changes with age and breast density, and including women across the spectrum of age, breast density, and breast cancer risk will be essential. Recommending or requiring diverse populations for algorithm training and testing would improve generalizability. In addition, reporting the characteristics of these populations would allow end users to fully evaluate products.

The Role of Post-Marketing Surveillance

In addition to strengthening evidentiary standards, a robust system of post-marketing surveillance is needed. Post-marketing surveillance could help detect unintended consequences of AI when applied by physicians, deviations in performance compared to what was reported in controlled studies, or changes in intended use. Post-marketing surveillance by developers is already a central feature of proposed FDA frameworks for regulating AI.^27,28,34 Post-marketing surveillance could leverage existing registries, such as the Breast Cancer Surveillance Consortium, registries developed by professional societies, or reporting structures used for quality assurance. Health system networks also have the potential to contribute to post-marketing surveillance infrastructure. Developing post-marketing surveillance systems will rely on engagement and investment from developers. However, public-sector investment will also be important for developing and maintaining infrastructure for surveillance. Post-marketing surveillance will be increasingly important as adaptive products are adopted into clinical practice, as the initial product approved may change over time.

The Role of Modeling Studies

In addition to empirical studies, modeling studies can also play a role in evaluation. Modeling studies, using established simulation models such as those maintained by CISNET, have historically been used to evaluate new BCS technologies. Simulation models can project the long-term impact of a screening technology on outcomes like morbidity, mortality, and cost, based on short-term parameters like sensitivity and specificity. Models have some important limitations—they make assumptions about the natural history of breast cancer and the modeled population. Modeling results are also dependent on the parameters associated with the screening technology. Thus, models can be useful but require accurate inputs to be informative. Improving the strength of evidence around AI for BCS would help improve the value of modeling approaches.

Focusing on Patient-Centered Outcomes

Across the spectrum of evidence generation, a key question is what outcomes should be evaluated. No single outcome measure is paramount, and different outcomes may suit different purposes. While mortality is the most meaningful long-term outcome for studies of BCS, its use as an endpoint is challenging and increasingly impractical given the rapid pace of new technology development, especially in the field of AI. Mortality is also a relatively rare outcome and requires very large sample sizes to evaluate. More contemporary studies of BCS have focused on surrogate endpoints that can be measured in a shorter time scale, such as interval cancers and late-stage diagnosis.^40–43 However, whether reducing interval cancers improves mortality is uncertain, and important questions remain about the value of focusing on these surrogate outcomes. Quality of life measures have also been noted as helpful, patient-centered adjunctive outcome measures.⁴⁴ Finally, although focusing on test accuracy alone may not be sufficient to understand whether a product improves health outcomes, measuring characteristics of performance will still be important and meaningful. These measures could be extended in the post-marketing surveillance phase to understand whether products perform as expected with diverse patients and in varied settings.

The Role of Other Interested Parties

Increasing regulatory requirements for approval has important tradeoffs. More stringent regulatory standards would also likely mean that fewer products will enter the market and the pace of new product entry would be slower. Fundamentally, FDA’s goal is to ensure the safety and efficacy of medical devices. Demonstrating that an AI product has similar performance to the predicate may be sufficient for current regulatory standards but may fall short of ultimately ensuring a product improves health, reflective of a broader criticism of the 510(k) pathway by the National Academy of Medicine.⁴⁵ If FDA is viewed to be responsible only for maintaining the minimum bar for AI product approval, other parties, including payers, the medical and scientific community, and patients, may play a role in either encouraging or constraining AI adoption. Payers can set coverage criteria and determine reimbursement rates for the use of medical technologies including AI in practice. Currently, CAD is not separately reimbursed for screening mammography. Going forward, if reimbursement were to be considered, payers could request additional evidence from AI product manufacturers to better inform coverage decisions. The professional community can also influence practice and adoption of AI through guidelines, opinion leaders, and use by influential organizations. Finally, last but not least, patients are of central importance and should be involved in discussions about new BCS technologies. Patients deserve to know about changes in medical technology such as AI influencing their care, and more work must be done to inform them of these changes and empower them to make informed decisions.

Conclusions

The role of AI in breast cancer screening is only likely to grow over the next decade. Increased FDA evidentiary regulatory standards, development of improved post-marketing surveillance and trials, a focus on clinically meaningful outcomes, and engagement of key stakeholders all can help ensure that new AI products meet the goal of improving breast cancer screening outcomes and, ultimately, advancing health for all women.