Artificial Intelligence for Mammography and Digital Breast Tomosynthesis: Current Concepts and Future Perspectives

Abstract

Although computer-aided diagnosis (CAD) is widely used in mammography, conventional CAD programs that use prompts to indicate potential cancers on the mammograms have not led to an improvement in diagnostic accuracy. Because of the advances in machine learning, especially with use of deep (multilayered) convolutional neural networks, artificial intelligence has undergone a transformation that has improved the quality of the predictions of the models. Recently, such deep learning algorithms have been applied to mammography and digital breast tomosynthesis (DBT). In this review, the authors explain how deep learning works in the context of mammography and DBT and define the important technical challenges. Subsequently, they discuss the current status and future perspectives of artificial intelligence–based clinical applications for mammography, DBT, and radiomics. Available algorithms are advanced and approach the performance of radiologists—especially for cancer detection and risk prediction at mammography. However, clinical validation is largely lacking, and it is not clear how the power of deep learning should be used to optimize practice. Further development of deep learning models is necessary for DBT, and this requires collection of larger databases. It is expected that deep learning will eventually have an important role in DBT, including the generation of synthetic images.

© RSNA, 2019

Online supplemental material is available for this article.

Learning Objectives:

After reading the article and taking the test, the reader will be able to:

• ■ Describe the limitations of the classic computer-aided detection (CAD) systems

• ■ Describe how neural networks enable deep learning systems to improve their predictions of the likelihood of malignancy for lesions detected at digital mammography and digital breast tomosynthesis (DBT)

• ■ Identify the challenges of developing deep learning models for DBT

Accreditation and Designation Statement

The RSNA is accredited by the Accreditation Council for Continuing Medical Education (ACCME) to provide continuing medical education for physicians. The RSNA designates this journal-based SA-CME activity for a maximum of 1.0 AMA PRA Category 1 Credit^™. Physicians should claim only the credit commensurate with the extent of their participation in the activity.

Disclosure Statement

The ACCME requires that the RSNA, as an accredited provider of CME, obtain signed disclosure statements from the authors, editors, and reviewers for this activity. For this journal-based CME activity, author disclosures are listed at the end of this article.

Summary

Because of the advances in deep learning, the quality of artificial intelligence is rapidly improving for breast imaging and it will likely play an important role for mammography and digital breast tomosynthesis in all steps—from image generation and denoising to risk prediction, cancer detection, and, ultimately, therapy selection and outcome prediction.

Essentials

• ■ In clinical practice, the use of computer-aided diagnosis (CAD) does not improve diagnostic accuracy because the many false prompts lead to higher false-positive rates, recall rates, and biopsy rates.

• ■ Neural networks are capable of learning intermediate, more abstract, representations of the data before classifying the entire image.

• ■ The difference in the appearance of the normal breast parenchyma with digital breast tomosynthesis images obtained with machines from different vendors is much greater than that with full-field digital mammography; this is an important consideration when training deep learning models.

• ■ Previous mammograms and images obtained with different imaging modalities can be exploited to improve the quality of prediction of neural networks.

• ■ The performance of deep learning–based systems is better than that of classic CAD systems based on manually crafted features, approaching that of radiologists for specific tasks.

Introduction

Multiple randomized clinical trials have demonstrated that screening mammography reduces the mortality from breast cancer by 20%–22% (1,2). As a result, mammography is the cornerstone of breast cancer screening (3,4). In addition, mammography is the initial examination for many women with breast symptoms (5,6). In 2015, 22.6 million mammograms were obtained in the United States alone (7). The evaluation of mammograms thus demands a large number of dedicated radiologists. Unfortunately, there is an increasing shortage of qualified readers in many countries (8). Even for women screened with mammography, as many as one in three cancers manifests as interval cancer; a large proportion of these cancers were, in retrospect, visible on the previous screening mammograms (9,10). Missed cancers at mammography are, therefore, one of the most common reasons for malpractice lawsuits in radiology (11,12). The recent introduction of digital breast tomosynthesis (DBT), in which multiple projections of the breast are obtained over a limited angular range to reconstruct a three-dimensional data set of mammography images (13,14), is only a partial solution. Although DBT depicts 30%–40% more cancers than full-field digital mammography (15,16), the reading time is approximately doubled (17,18) and cognitive and perception errors still occur (19). Consequently, there is a need for assistance with the evaluation of mammography and DBT, both to maximize the cancer detection rate and to address the workload issues.

Automated analysis (ie, artificial intelligence [AI]) of mammograms and DBT images may address these needs. Computer-aided diagnosis (CAD) for mammography has been under development since the late 1960s (20). Its primary aim is to assist radiologists in identifying subtle cancers that might otherwise be missed. CAD programs mark focal areas of increased density and microcalcifications. The first CAD software for screening mammography received U.S. Food and Drug Administration approval in 1998 (21). Early results were promising (22–25), and CAD has been widely adopted into clinical practice—with approximately 92% of all mammography facilities in the United States using this technology by 2016 (26). However, its clinical value is uncertain (27,28), mainly due to the large number of false-positive findings.

The success of deep convolutional neural networks (CNNs) in the 2012 ImageNet Large Scale Visual Recognition Challenge (29) triggered new interest in the development of better automated image analysis methods. In the past few years, similar deep neural networks were shown to be highly effective in tasks ranging from face recognition to self-driving cars (30–33). Recent studies have shown that CNNs can also be highly successful in various tasks in the health care industry, ranging from retina analysis to digital pathology (34–36), and in multiple applications in radiology (37–39). Several excellent reviews have been published on the general use of AI in these fields (40–42). Figure 1 illustrates the hierarchy of terms used in AI, as they are not completely interchangeable. It is foreseeable that deep learning will also lead to a major change in the automated analysis of images from mammography and DBT. In this review, we discuss the potential of deep learning techniques for mammography and DBT. In addition, we discuss the current technical approaches to improve on the available CAD systems for mammography and the potential use of these techniques in clinical practice.

Figure 1:

Diagram illustrates the relationship between artificial intelligence (AI), machine learning (ML), neural networks (NN), deep learning (DL), and convolutional neural networks (CNN). AI is the most general of these terms, as it includes systems that aim to mimic human intelligence by learning from data (machine learning) and by applying manually defined decision rules. Machine learning includes neural networks but also pertains to many other methods, such as kernel methods (eg, support vector machines) and decision tree–based methods. Among neural networks, deep learning, which involves study of neural networks consisting of many layers, is currently the most successful in practical applications and the subject of the most intense research. Finally, the type of deep neural networks most frequently applied in medical image analysis are the convolutional neural networks.

Conventional AI

Most available CAD systems perform, in essence, two separate tasks. In the first task, potential lesions that stand out from the normal fibroglandular tissue are detected. The second task entails the reduction in the number of false-positive findings. In this step, the potential lesions are classified and obvious false-positive findings are removed from the list of potential lesions. To perform this task, classic CAD systems depend on human-designed features. For example, masses are detected by using their gray level (how white it is), gradient (if it stands apart from its surroundings), texture (how homogeneous it is), and shape (if it resembles a mass) (43); microcalcifications are detected by actively searching for rod-like high-intensity pixels within the mammogram (Fig 2) (44–46). To reduce the number of false-positive findings, candidate voxels are clustered into possible lesions and analyzed with use of additional features such as distribution, shape, margin, and texture (47–49). To reach a final classification about whether a finding should be flagged, the CAD systems combine the most discriminative features by using a classifier (eg, support vector machines, random forests), and lesions above a predefined threshold are subsequently marked (50).

Figure 2:

Left, mediolateral oblique view from screening mammogram in 54-year-old asymptomatic woman. A computer-aided diagnosis (CAD) prompt is present (arrowhead). Right, magnification view of area of interest. The CAD system identified a small group of calcifications (arrow), which were sampled for biopsy and yielded grade 2 ductal carcinoma in situ.

CAD systems are classified into two groups: computer-aided detection systems and computer-aided diagnosis systems. Computer-aided detection systems focus on the localization task (ie, detection of a suspicious abnormality). They serve as a second reader to radiologists and leave subsequent patient management decisions to the radiologist (51,52). Computer-aided diagnosis systems, however, characterize an abnormality that is identified by a radiologist or a computer. The computer-aided diagnosis system estimates an abnormality’s probability of disease and classifies it as benign or malignant. The radiologist then decides whether the abnormality warrants further evaluation and determines its clinical significance (51,52). The advances in the design of classifying features over the years has resulted in a substantial improvement in both the sensitivity and specificity of CAD (51–54). The performance of the top systems reported in the literature approached that of humans when using feature-based classification, albeit only on specific tasks (eg, mass detection) in selected data sets (55,56).

Most conventional CAD systems present their findings in the form of prompts on the mammogram, which requires the radiologist to decide whether the prompts represent an underlying malignancy. Because of the limited specificity of these systems, this is a challenging task. Ikeda et al (57) reported that, when using a feature-based CAD system, approximately 1000 prompts must be analyzed to detect one additional cancer. It is therefore not surprising that the use of CAD in mammography leads to a slightly higher detection rate (range, 1%–19%) when combined with single reading, but at the cost of a lower specificity (incremental recall rate, 6%–36%) and longer evaluation times. Double reading still seems to be significantly better than single reading with CAD (58–65). In clinical practice, the use of CAD does not improve diagnostic accuracy (27,28) because the many false prompts lead to higher false-positive rates, recall rates, and biopsy rates. In addition, the use of CAD does not appear to be cost-effective (66). Several studies concluded that CAD applications require substantial improvement to really be beneficial for patient care. Table 1 summarizes most of the literature on the implementation of CAD into the clinical workflow.

Table 1:

Summary of Landmark Decisions or Studies on CAD

Note.—AUC = area under the receiver operating characteristic curve, CAD = computer-aided diagnosis, FDA = Food and Drug Administration.

Why Deep Learning?

AI, powered by the recent advances in machine learning, may make CAD for mammography more valuable in clinical practice. The most promising of these advances is deep learning—a family of machine learning methods focusing on developing multilayered neural networks (67,68). Like conventional CAD systems, neural networks are mostly trained by using supervised learning, in which every training example comes with an expected output. Logistic regression, decisions trees, and support vector machines, which are used for conventional CAD, are examples of supervised learning models not based on neural networks. However, what these methods have in common is that, although the decision process they use to arrive at the classification decision might be very complex, they do not learn any intermediate representations of the data. That implies that these methods can only work well if the input features they are presented with are very predictive to begin with. However, as apparent from the false-positive findings in conventional CAD, in mammographic evaluation it is very difficult to design features on the level of the input pixels that would allow the classifier to accurately predict the label for the entire image. Neural networks, on the other hand, are capable of learning intermediate, more abstract, representations of the data before classifying the entire image (68). CNNs only combine information from voxels that are spatially close to each other and are therefore especially suited for image evaluation. This is key to understanding why neural networks work so well for image analysis in comparison to other methods. A more in-depth explanation of the functioning of neural networks and, in particular, CNNs can be found in Appendix E1 (online).

How Deep Learning Works in Mammography

Deep learning models appear to be successful in evaluating mammograms. In the Digital Mammography Dialogue for Reverse Engineering Assessments and Methods, or DREAM, Challenge, held between November 2016 and May 2017, many teams competed in developing machine learning models to classify screening mammograms according to whether cancer was present. Every team used the same data set, which consisted of 640 000 images from more than 86 000 women. The most successful teams used deep learning models (69,70), achieving a sensitivity of up to 87%. This is on par with the 88% sensitivity achieved by radiologists with the same data set. However, only the leaderboard teams achieved a specificity of 82%, a performance that approaches the specificity of the radiologists in the Breast Cancer Surveillance Consortium. The overwhelming majority of deep learning models developed in this challenge were based on relatively simple variations of the CNNs described earlier. A basic way of classifying the various models is according to whether they are trained by using only the examination-level labels (indicating whether the patient under examination has cancer) or both examination-level labels and pixel-level labels (annotations of malignant or benign lesions). The models trained with only examination-level labels are trainable end to end, whereas the models trained with both examination-level and pixel-level labels need a more complex training procedure.

Models trained with examination-level labels are usually the most similar to the standard deep CNNs (71–73). They are sometimes modified by taking into account multiple mammographic views simultaneously (71,72) or by adding a multiple-instance layer (73). On the other hand, models that also use pixel-level labels are trained as two separate models in different variations (70,74). Some of these models are also fine-tuned end to end after the two-stage training (74). Models that learn from both examination-level and pixel-level labels generally exhibit higher performance and/or require fewer cases because they learn from a more detailed supervision. However, the data collection is much more laborious, and performance is dependent on the quality of the annotations—which is a difficult problem as there is no real ground truth and interreader variability is substantial (75).

Technical Challenges Unique to Mammography

Not all deep learning algorithms have equal performance. Fine-tuning algorithms to specific tasks in mammography and DBT requires more effort than just the use of a very general CNN on a large data set. For one, it is difficult to train a CNN that is good in the detection of both masses and calcifications. Consequently, separate CNNs are often trained for both lesion types and the outcomes are only combined in the final output of the AI support system. Furthermore, algorithms must be consistent and reproducible over mammograms obtained by different technologists using mammography machines from various vendors. Validation of the deep learning algorithms across different vendors is substantial because all vendors use their own proprietary postprocessing data to make the mammograms ready for presentation, and the raw data are usually not stored. This has a large influence on image appearance and implies that a CNN trained on mammograms obtained with a machine from one vendor may not be applicable to mammograms obtained with a machine from another vendor (Fig 3) (76,77). Consequently, normalization of mammograms is an important task that must be carried out with machine learning techniques.

Figure 3:

Images in 58-year-old asymptomatic woman who presented for screening mammography. Arrows indicate cancer. A, Right mediolateral oblique screening mammogram shows an asymmetry with associated architectural distortion (arrow). B, The asymmetry (arrow) is better seen on the magnification view. Subsequent biopsy yielded a grade 2 invasive ductal carcinoma. C, Mammogram obtained 2 years earlier, with the output of an artificial intelligence (AI)–based computer-aided diagnosis system (red circle; 93 represents a 93% likelihood of malignancy). Arrow shows same asymmetry seen in A. D, Magnification view of the cancer-containing area in C. Arrow indicates the same asymmetry seen in B. Because of the high likelihood of cancer predicted by the AI system, the lesion is prompted even when missed by the evaluating radiologist. Consequently, this cancer would likely have been detected earlier if an AI system had been used in the original reading. Note that the mammograms were obtained with machines from different vendors and look different mainly due to the applied postprocessing. (Image courtesy of Nico Karssemeijer, PhD.)

Most efforts in deep learning focused on applying existing techniques to mammography rather than proposing new ones specifically suited to the domain. Medical images have properties that make them very different from images from natural scenes (eg, images of a tree or dog) that are not synthetic or artificial (70). For example, although the objects of interest that determine the class usually occupy a large fraction of natural images, objects of interest in medical images are often relatively small. The standard well-known network architectures were designed for the natural images and do not take these peculiarities into account. Therefore, research is necessary to understand how these architectures can be optimized for medical images—mammograms and DBT in particular. If enough computational capacity was available, this problem could be largely solved by using an automated neural architecture search (78).

Furthermore, to integrate deep learning into clinical practice, it is necessary to explain their predictions in a form understandable to humans. The simplest form of such an explanation points to the input pixels that influenced its predictions. This is often referred to as an attention or saliency map (Fig 4). Multiple powerful methods that draw attention to the locations in an image that contribute to the decision for a particular case have recently been proposed for natural images (78–81). Adaptation of these methods to mammography and DBT data will be technically challenging because the data are of much higher dimensionality than are data from typical natural images. However, the benefit that these methods could bring may extend beyond aiding in interpretation. A neural network can learn from millions of images in a few days; this is impossible for radiologists. Therefore, it is conceivable that neural networks may eventually be used as a knowledge discovery tool when their ability to explain predictions improves.

Figure 4a:

Examples of saliency maps for screening mammography examination classification in 67-year-old asymptomatic woman. Images are left craniocaudal mammograms without (a) and with (b, c) overlying heat maps. There is a 3.6-cm irregular round dense mass (black arrow in a) in upper central left breast and a 5-mm cluster of calcifications (white arrow in a) in medial inferior left breast. From a probability of malignancy score of 0–1, the maximum value of the benign green heat map is 0.71. The maximum value of the malignant red heat map is 0.881. Both values indicate that the classifier predicts with high certainly that the mass is malignant and the calcifications are benign. At pathologic examination, the mass was an invasive ductal carcinoma and the calcifications were benign fat necrosis. (Image courtesy of Nan Wu, PhD.)

Figure 4b:

Examples of saliency maps for screening mammography examination classification in 67-year-old asymptomatic woman. Images are left craniocaudal mammograms without (a) and with (b, c) overlying heat maps. There is a 3.6-cm irregular round dense mass (black arrow in a) in upper central left breast and a 5-mm cluster of calcifications (white arrow in a) in medial inferior left breast. From a probability of malignancy score of 0–1, the maximum value of the benign green heat map is 0.71. The maximum value of the malignant red heat map is 0.881. Both values indicate that the classifier predicts with high certainly that the mass is malignant and the calcifications are benign. At pathologic examination, the mass was an invasive ductal carcinoma and the calcifications were benign fat necrosis. (Image courtesy of Nan Wu, PhD.)

Figure 4c:

Examples of saliency maps for screening mammography examination classification in 67-year-old asymptomatic woman. Images are left craniocaudal mammograms without (a) and with (b, c) overlying heat maps. There is a 3.6-cm irregular round dense mass (black arrow in a) in upper central left breast and a 5-mm cluster of calcifications (white arrow in a) in medial inferior left breast. From a probability of malignancy score of 0–1, the maximum value of the benign green heat map is 0.71. The maximum value of the malignant red heat map is 0.881. Both values indicate that the classifier predicts with high certainly that the mass is malignant and the calcifications are benign. At pathologic examination, the mass was an invasive ductal carcinoma and the calcifications were benign fat necrosis. (Image courtesy of Nan Wu, PhD.)

Clinical Applications

Increase the Cancer Detection Rate and Reduce the Recall Rate

The most important task of CAD systems so far has been the detection of cancer on digital mammograms, the quality of which has improved with the implementation of deep learning (69–72,82–87); a few AI systems are now performing at the level of radiologists (Fig 5) (82). An open question is how to use this strong performance to optimize the current diagnostic and screening processes. Recently, reimbursement for the use of CAD in the United States was bundled into the price of a mammogram; thus, it is no longer possible to charge directly for the use of CAD. Hence, CAD should improve the quality and/or efficiency of mammography reading to be profitable. As an initial step, deep learning–based systems may be used for cancer detection in a very similar way to the classic CAD systems, pointing out abnormalities. There are two ways to implement this: using prompts for all findings and, in an interactive setting, showing findings only when specific areas of the mammogram are queried. Although the second approach proved more effective when using conventional CAD (86), this poses ethical problems when using systems with human-like performance because evident cancers detected by the AI system might go unnoticed when the human reader does not click on the right spot. Consequently, hybrid systems have been proposed that function as a decision aid that provides interactive feedback and prompts for the most evident findings (Fig 3). The use of such an AI system increased reader performance significantly (82,85).

Figure 5:

Receiver operating characteristic curves for radiologists reading mammograms unaided and stand-alone artificial intelligence (AI) computer system (Transpara; Screenpoint, Nijmegen, the Netherlands). Circles indicate the radiologists’ operating points at Breast Imaging Reporting and Data System category 3 thresholds. (Reprinted, with permission, from reference 82.)

Because AI systems are much more specific than previous CAD systems (83,84), they may be used to reduce the recall rate—for example, by identifying specific mammographic features to differentiate recalled benign images from malignant and negative cases (87). Recent advances with the availability of large sets of annotated data used to train complex neural networks with many layers have shown a decrease in the number of false-positive prompts and a reduction in recall rates by 10%–20% (Fig 6). Another important feature of the AI systems is the feedback provided when, according to the system, the likelihood of cancer is very low. This may increase the confidence of the reporting radiologist and, hence, expedite the reading of normal cases while allowing more time for potential cancer-containing cases. Rodriguez-Ruiz et al (85) recently found that an AI case score, a metric ranging from 1 to 10 that describes the likelihood of malignancy, was significantly associated with a reduction in reading time. Readers reduced their reading time in cases with a low score, leading to a potential reduction of overall reading time of 4.5% for a screening data set (albeit the general reading times in that study were longer than those normally needed in clinical practice). Alternative approaches that classify mammograms as negative by AI systems alone, without human input, have been proposed. These approaches would have a much larger effect on workflow efficiency.

Figure 6:

Bilateral mediolateral oblique mammograms in two women with breast cancer (arrow). Mammograms were obtained, A, C, without and, B, D, with the output of a convolutional neural network–based cancer detection system. The likelihood of cancer presence is given as a heat map. A, B, Images in one patient with a relatively obvious spiculated mass found to be a grade 2 invasive ductal breast cancer. C, D, Images in another patient with a much more subtle asymmetry found to be an invasive lobular carcinoma. (Images courtesy of Beomseok Suh, PhD.)

A human-like AI system could, for example, be used as a fully independent second reader of screening mammograms. A second human reader would only arbitrate discrepancies between the first human reader and the AI system, thus halving the workload for any screening program in which double reading is standard. While tempting, it should be noted that the actual effect of such an approach on recall policy and positive predictive values for recall and biopsy is still unknown. Following earlier studies with conventional CAD systems, a single reading plus CAD approach was not adopted due to a slightly lower sensitivity and higher recall rate (65). The increased performance of the AI system might not completely overcome this, as it is also dependent on the behavior of the human reader who eventually determines whether the finding is suspicious and who will recall the patient for additional imaging. The idea to dismiss mammograms that are categorized as very likely normal without any human reader interpretation is the logical next step. Such preselection of normal cases may be based on case-based AI scores as described earlier but will likely result in dismissal of a small fraction of cancers by the computer alone. Ethical considerations and cost-effectiveness will determine whether such an approach might be viable in the future. Table 2 lists the differences and potential use of deep learning–based AI systems compared with conventional CAD systems in the detection of cancer. Table 3 provides an overview of these CNN-based AI systems and their current clinical applications.

Table 2:

Potential Use of CAD for Cancer Diagnosis

Note.— AI = artificial intelligence, CAD = computer-aided diagnosis.

Table 3:

Summary of Recent Results for Digital Mammography and DBT and AI Applications

Note.—AI = artificial intelligence, AUC = area under the receiver operating characteristic curve, CAD = computer-aided diagnosis, CNN = convolutional neural network, DBT = digital breast tomosynthesis, FFDM = full-field digital mammography.

Quantitative and Reproducible Assessment of Breast Density to Stratify Risk for Breast Cancer

Another important task for CAD systems is to provide an accurate and reproducible assessment of mammographic breast density (88–91). Mammographic breast density may mask an underlying cancer. In addition, dense breast tissue is an independent risk factor for the development of breast cancer (92,93). Consequently, breast density assessment is commonly used for stratification of women for supplemental screening examinations. Conventional CAD systems may standardize the reporting of breast density by using either the projected white areas from the processed mammograms directly or a volumetric calculation of the amount of fibroglandular tissue from raw mammograms (94,95). Several studies showed that automated quantitative assessment of breast density is more robust than human evaluations, especially when evaluated over time (96,97).

However, the prediction of risk for the development of breast cancer with use of automated measures seems to be inferior to that of visual assessment (95,98,99). This might have to do with the subconscious incorporation of fibroglandular tissue distribution and texture features by radiologists. Automated risk prediction becomes better when incorporating texture features (100–103), and the integration of texture features with deep learning may strongly improve their discriminatory power (104). Several studies have shown that deep learning–based classification of fibroglandular density categories is closer to radiologist observations than classic feature-based techniques (88,105–107). Moreover, Lehman et al (89) showed that the vast majority (94%) of deep learning–based density classifications is accepted by reporting radiologists. Although the predictive value of AI-based density estimations still must be assessed in modeling studies (108), it is anticipated that the risk assessment with such density estimations, when trained on sufficiently large databases, is similar to that of radiologists.

Applying Deep Learning Algorithms to DBT

The issue of image normalization is even more important in DBT than in mammography. DBT images acquired with machines from different vendors have differences in angular range, acquisition technique, pixel binning, and reconstruction technique (14). Therefore, the difference in the appearance of the normal breast parenchyma with DBT images from different vendors is much larger than that with full-field digital mammography. This is an important consideration for training deep learning models. In addition, available training data sets for DBT are much smaller, which implies that other techniques that work with a relative paucity of data must be used to improve performance. To manage this issue, transfer learning can be applied. Transfer learning is based on the assumption that if two learning tasks are similar, a network trained to solve a task with more data available can be reused for a task with fewer training data available (109). Most commonly, transfer learning is implemented by copying the parameters of the network trained with a lot of data into the network that is intended to solve the task for which fewer data are available. Subsequently, the second network is only trained for a very short time to prevent overfitting. In the context of breast imaging, this technique was, for example, used for classifying breast density by using a network originally designed for performing Breast Imaging Reporting and Data System classification (88). Current cancer detection systems for DBT are largely based on adaptations of networks originally trained on mammograms to allow the image patterns learned from mammography to be transferred to the analysis of DBT images. However, the depth dimension in tomosynthesis has a poor spatial resolution and therefore only a limited influence on the detection accuracy per anatomic slice (110). It is therefore to be expected that the performance of AI for DBT is somewhat behind the performance for mammography (110).

Current CNN-based systems for DBT already improve upon features that are manually identified and labeled by humans (111,112). With larger training data sets, these algorithms will improve and become indispensable in the evaluation of DBT because the potential gain in workflow efficiencies will be much higher due to the longer reading times of DBT examinations. Proposed detection systems largely work with conventional prompts placed on the synthetic mammogram. These prompts guide the reader to the most suspicious section in the DBT volume when clicked. More advanced integration of AI with DBT is expected, and potential applications are listed in Table 4. These applications start at the level of scatter correction and denoising to reduce the radiation dose (113,114). Basic reconstruction of a DBT volume is based on (filtered) back-projection, a technique that is commonly used for CT. However, studies have shown that more sophisticated iterative techniques considerably improve image quality (115–117), which also improves subsequent automated cancer detection with CNNs (117). It seems that deep learning–based techniques may further optimize the quality of the reconstructed images (118,119). In the future, synthetic mammograms will be generated from the tomosynthesis data by using deep learning techniques, as current synthetic mammograms may, at best, be comparable to full-field digital mammograms (120,121). The use of machine learning to generate synthetic mammograms may enhance suspicious findings in the DBT volume so that they become more conspicuous. In addition, they may even remove normal tissue that may mask eventual relevant findings. The use of a multiplanar reconstruction fitted through the most suspicious lesions detected by a conventional CAD system in a DBT examination improved reader performance compared to that with full-field digital mammography (122). A commercially available synthetic mammography system on which lesions detected in the DBT volume are enhanced has also been evaluated (Fig 7). In an initial reader study, readers performed equally well with and without CAD, but the average reduction in reading time was 23.5% (123). In addition, James et al (124) found that radiologist performance increased substantially when they compared CAD-enhanced synthetic mammograms with conventional synthetic mammograms.

Table 4:

AI Solutions for Issues in DBT

Note.—AI = artificial intelligence, DBT = digital breast tomosynthesis.

Figure 7:

Examples of artificial intelligence (AI)–enhanced synthetic mammograms. A, Normal synthetic craniocaudal mammogram of right breast. B, AI-enhanced craniocaudal acquisition clearly shows an invasive ductal carcinoma (arrow) that is hardly visible in A. C, Normal synthetic mediolateral oblique mammogram and, D, AI-enhanced version. The invasive ductal carcinoma in D (arrow) is hardly visible in C. (Image courtesy of Corinne Balleyguier, PhD.)

Radiomics

Radiomics, an expansion of CAD, is defined as the conversion of images to minable data by means of digital decoding of radiologic images into quantitative features (125). In radiomics analysis, the tumor is segmented from its background and various tumor features (eg, intensity, shape, size or volume, and texture patterns) are extracted. Once large high-quality and well-curated data sets are available, they can be used for data mining, which refers to the process of discovering patterns in large data sets. This process can use AI, machine learning, or statistical approaches (126). The goal of quantitative radiomics is to yield predictive image-based phenotypes of breast cancer with the aim of better classifying the tumor to improve treatment and prognosis, in line with precision medicine. Furthermore, radiogenomics (ie, imaging genomics) aims to find associations between imaging data and clinical data, molecular data, genomic data, and outcome data (127). Most radiomic studies extract data from breast MRI to determine the cancer phenotype and, in particular, heterogeneity (128,129). However, several studies have shown correlations between mammographic characteristics and biologic profiles of breast cancers (130,131). Consequently, mammographic data may be used to gain insight into breast cancer phenotypes. In a recent study, Shi et al (132) showed that a CNN detected occult invasion in patients with ductal carcinoma in situ, achieving an area under the receiver operating characteristic curve of 0.70 in a very small database of digital mammograms. This is slightly better than that achieved by Li et al (133), who used more conventional radiomics feature extraction techniques. Another recent study using such feature extraction (134) showed that parenchymal texture features of the contralateral breast may be used to improve the differentiation between benign and malignant lesions. Using a similar approach, Yang et al (135) achieved a classification accuracy of 84% in predicting lymph node involvement from mammographic characteristics of the primary tumor. Another recent study (136) reported that radiomics features of the parenchyma from DBT in women with occult breast cancer in dense breasts differ from those in women without cancer, thus yielding the possibility to predict breast cancer risk estimation. It may be possible to further optimize therapy by using automated extraction of mammographic features of cancer, although it remains to be seen whether these features are complementary to clinical and histopathologic information alone.

Future Directions

A shortcoming of currently used neural network models is that they only evaluate the most recent examination. Although it is possible to make a reasonably good assessment this way, it is evident that this does not take into account all the information a radiologist would rely on to evaluate a difficult examination. Previous mammograms, and images obtained with different imaging modalities, can be exploited to improve the quality of prediction of neural networks. A network that could learn by using these data would be especially useful in diagnosing very early stages of cancer, as even subtle changes in the breast tissue are difficult for a radiologist to perceive. In addition, nonimaging-based patient characteristics, such as demographic information, history of cancer, and genetic information, may be integrated into the model. Given a sufficiently large data set, neural networks could use these pieces of information in conjunction with the image data to identify women at high risk of cancer. Similarly, in patients with breast cancer, AI may allow for highly personalized therapy, commonly referred to as precision medicine, using deep learning–based radiomics assessment. Data on the effect of AI systems on clinical performance and patient outcome are limited. Studies evaluating such end points are vital for the positioning of these techniques in health care, especially because policy-level issues such as reimbursement and liability have yet to be defined.

As AI becomes an important tool for radiologists, it will become fully integrated in the different imaging modalities (137). To be efficient in this role, a deep neural network must be able to explain its decision in a form that is comprehensible to humans. Clinical implementation of AI systems is limited by the machine’s current inability to explain its decisions and actions to human users. This must be addressed. Beyond improving the understanding of the predictions made by the neural networks, indicating important parts of the mammogram could be used for planning and analysis of subsequent imaging examinations such as US or MRI.

Conclusions

The development and implementation of artificial intelligence (AI) for mammography has been ongoing for several decades. Because of the advances in deep learning, the speed of implementation and the clinical value of AI have markedly increased. AI will play an important role for mammography and digital breast tomosynthesis (DBT) in all steps—from image generation and denoising to risk prediction, cancer detection, and, ultimately, therapy selection and outcome prediction. Compared with classic computer-aided detection systems based on manually crafted features, deep learning–based systems have a better performance—approaching that of radiologists for specific tasks. Still, there are also residual shortcomings of the novel AI solutions. These include the need for very large and well-curated data sets to train and validate algorithms and a necessity to devise continuous quality control systems as the algorithms are versatile and may evolve over time when more data become available. External validation studies are urgently needed. Although many recent studies are promising and report strong results, we must look at them critically and recognize their limitations in several aspects. First, almost all works only report the area under the receiver operating characteristic curve in detecting malignancy. Although the area under the receiver operating characteristic curve is the most widely applied metric for measuring a classifier’s performance, it is sensitive to class distribution. Studies that use test data of different class distributions should not be compared by using the area under the receiver operating characteristic curve. Second, very few studies explain the data distribution used for training and testing in enough detail. Little is known about how accurate these different networks are for different types of findings. We also do not know how well different networks would work when applied to data acquired with different machines or to data acquired for a population of different demographic characteristics. Finally, few studies have been performed to evaluate how the advances in AI can be implemented in a manner that maximizes their clinical impact, which must be the ultimate target. Even with these limitations, it is expected that AI will play a major role in the evaluation of mammography and DBT in the near future, particularly in the screening setting.