Impact of Transfer Learning Using Local Data on Performance of a Deep Learning Model for Screening Mammography

Abstract

Purpose

To investigate the issues of generalizability and replication of deep learning models by assessing performance of a screening mammography deep learning system developed at New York University (NYU) on a local Australian dataset.

Materials and Methods

In this retrospective study, all individuals with biopsy or surgical pathology-proven lesions and age-matched controls were identified from a South Australian public mammography screening program (January 2010 to December 2016). The primary outcome was deep learning system performance—measured with area under the receiver operating characteristic curve (AUC)—in classifying invasive breast cancer or ductal carcinoma in situ (n = 425) versus no malignancy (n = 490) or benign lesions (n = 44). The NYU system, including models without (NYU1) and with (NYU2) heatmaps, was tested in its original form, after training from scratch (without transfer learning), and after retraining with transfer learning.

Results

The local test set comprised 959 individuals (mean age, 62.5 years ± 8.5 [SD]; all female). The original AUCs for the NYU1 and NYU2 models were 0.83 (95% CI: 0.82, 0.84) and 0.89 (95% CI: 0.88, 0.89), respectively. When NYU1 and NYU2 were applied in their original form to the local test set, the AUCs were 0.76 (95% CI: 0.73, 0.79) and 0.84 (95% CI: 0.82, 0.87), respectively. After local training without transfer learning, the AUCs were 0.66 (95% CI: 0.62, 0.69) and 0.86 (95% CI: 0.84, 0.88). After retraining with transfer learning, the AUCs were 0.82 (95% CI: 0.80, 0.85) and 0.86 (95% CI: 0.84, 0.88).

Conclusion

A deep learning system developed using a U.S. dataset showed reduced performance when applied “out of the box” to an Australian dataset. Local retraining with transfer learning using available model weights improved model performance.

Keywords: Screening Mammography, Convolutional Neural Network (CNN), Deep Learning Algorithms, Breast Cancer

Supplemental material is available for this article.

© RSNA, 2024

See also commentary by Cadrin-Chênevert in this issue.

Summary

Reduced performance obtained when applying a United States–based deep learning system for breast cancer screening to South Australian mammograms was mitigated by retraining with transfer learning on local data.

Key Points

• ■ Performance of deep learning models for breast cancer classification developed at New York University (original area under the receiver operating characteristic curve [AUC] values: NYU1, 0.83; NYU2, 0.89) was reduced when evaluated on an external test set of mammograms from a South Australian public mammography screening program (AUCs: NYU1, 0.76 [P < .001]; NYU2, 0.84 [P = .001]).

• ■ Retraining the models with transfer learning using original model weights—which are not generally available—improved model performance, achieving AUCs closer to those of the original models (AUCs after local training without transfer learning: NYU1, 0.66 [P < .001]; NYU2, 0.86 [P = .04]; AUCs after retraining with transfer learning: NYU1, 0.82 [P = .54]; NYU2, 0.86 [P = .04]).

• ■ Provision of model weights may be necessary to enable deep learning systems to be adapted to local clinical environments.

Introduction

Delays in breast cancer diagnosis and treatment, compared with earlier detection, are associated with adverse clinical outcomes and treatments that can decrease quality of life (1–3). Participation in mammographic screening has been associated with a relative reduction in breast cancer mortality of 20%–41% (4–9).

Convolutional neural networks (CNNs) are a family of deep learning methods that are especially suited to analyzing complex data such as images (10). Training occurs via updates to the numerical parameters (or weights) of a CNN. Rather than starting training with random weights (random initialization, or training “from scratch”), weights can be transferred from another learned task and retrained (transfer learning) (11).

The diagnostic accuracy of single human radiologists interpreting screening mammograms has been reported as a partial area under the receiver operating characteristic curve (AUC), sensitivity, and specificity of 0.88, 86.9% (95% CI: 86.3%, 87.6%), and 88.9% (95% CI: 88.8%, 88.9%), respectively (12). Recent studies have reported human-level performance with CNNs for screening mammography (13–17). However, one study found that the performance of a commercial CNN system (18) was reduced—and inferior to that of humans—when tested in a different country and population from that of its training (19). Some authors have reported inconsistent performance of deep learning models in mammogram classification, with AUCs ranging from 0.71 to 0.95 (20). CNN performance is sensitive to dataset characteristics, image idiosyncrasies, and the distribution of the training data (21–23), which may account for the reduced performance in some situations. Adjusting CNNs with local data may improve performance (14).

The performance of a deep learning algorithm depends on the training data distribution because the algorithm assumes that the distribution of the test data will be similar (24). The original training data distribution may be idiosyncratic to the local environment and, relative to the global population, subject to sampling bias. The performance of deep learning algorithms may also “shift” over time due to changes in technology, clinical practice, population structure, or other factors (25). For example, CNN performance may vary based on the vendor of the mammography equipment used to acquire the images used as model input (26). The capacity of CNNs to maintain performance on independent data, including from new geographical locations, is referred to as generalizability. Internal testing on a subset of the original training dataset is often performed in medical machine learning, while external testing on a completely independent dataset is less often undertaken. External testing, called replication in other fields, is essential for assessing both the validity and generalizability of deep learning models.

Wu et al published a CNN-based breast cancer classifier developed at New York University (NYU) with two different variants, one that accepts only images as input (NYU1) and a second that accepts images and benign and malignant heatmaps (NYU2) generated by a patch-level network (17). The best single model results for this system were an AUC of 0.83 (95% CI: 0.82, 0.84) for NYU1 and an AUC of 0.89 (95% CI: 0.88, 0.89) for NYU2.

To investigate the limits of generalizability in the context of deep learning for screening mammography, we tested this publicly available system (17,27) in a public mammography screening dataset from South Australia. Then we adjusted this system for local data using two approaches: training from scratch and retraining with transfer learning.

Materials and Methods

The current study was approved by the local institutional review board (HREC/16/229, R20160601), with a waiver of consent for the retrospective use of de-identified clinical data.

Dataset

The South Australian public mammography screening program, BreastScreen SA (BSSA), follows rigorous national accreditation standards, including needle biopsy sensitivity greater than 90% and a rate of recall for further assessment of less than 10% for individuals aged 50–69 years (28). Independent double reading with arbitration is the standard in BreastScreen Australia public mammography programs, which may increase detection and decrease recall rates (29).

All individuals with biopsy or surgical pathology-proven lesions (n = 3160) and age-matched controls (n = 3240) were identified from all individuals screened in BSSA with full-field digital mammography (craniocaudal and mediolateral oblique views) from January 1, 2010, to December 31, 2016 (n = 207 691; Fig 1). The study sample was randomly split into training (70%), validation (15%), and test (15%) subsets, stratified by the dominant finding of the dominant lesion, based on further lesion characterization with mammographic and US imaging. The primary local test set (LTS), LTS1, included individuals with invasive breast cancer or ductal carcinoma in situ (n = 425), individuals with benign lesions (n = 44), and age-matched controls (n = 490; Figs 1, S4; Table S1). All analyses were conducted on a per-individual level.

Figure 1:

Dataset flow diagram. A “round” refers to one episode of screening, consisting of at least four standard views (craniocaudal [CC] and mediolateral oblique [MLO] for each breast). 1 = excluding assessment mammograms, 2 = stratified patient-wise by dominant finding, BSSA = BreastScreen SA, LM = lateromedial, NYU = New York University, Val. = validation.

All case subsets were age-matched (on average, within 1 year) from a pool of control individuals. Individuals without the four standard views—craniocaudal and mediolateral oblique views of each breast—or with previous malignancy, biopsy, implants, or breast symptoms were excluded from analysis. A potential control round was included if the individual had no concerning abnormalities reported at both the index round and at least one subsequent screening round (Fig 1). The images used to train the original NYU model were acquired using Mammomat Inspiration (22.81%; Siemens Healthineers), Mammomat Novation DR (12.65%; Siemens Healthineers), Lorad Selenia (40.92%; Hologic), or Selenia Dimensions (23.62%; Hologic) (17) devices, whereas BSSA images in the training set were acquired using equipment from Sectra Imtec (46.03%) and Philips (53.97%) (Table 1).

Table 1:

BreastScreen SA Study Dataset Characteristics

The NYU models were fitted on all available biopsy-proven benign cases, each with their own label for model training and hyperparameter validation (Fig 1). To match the clinical task of interpreting screening mammograms, the primary outcome of interest was differentiating individuals with pathology-confirmed invasive breast cancer or ductal carcinoma in situ from age-matched controls and individuals with biopsied benign lesions (see Table S1 for pathology details).

Testing NYU Models without Retraining

Our initial replication analyses focused on applying the original, static NYU models to local data without any retraining on local data. The preprocessing and NYU system inference code were run as published (27) in LTS1, although some additional preprocessing was necessary due to differences in image acquisition and image size (see Figs S5, S6). Wu et al (17) reported on two models, one that used only the four-view mammograms as input (NYU1) and another that included heatmaps as well as the four-view mammograms (NYU2). Patch-wise benign and malignant heatmaps were generated for each BSSA image (Fig 2) with inference of the NYU CNN trained on NYU radiologists’ hand-drawn benign (n = 3158) and malignant (n = 855) segmentations, which are freely available (27). This network outputs a probability heatmap of each patch belonging to a benign or malignant segmentation. For both model variants, the maximum image-wise score was used for the patient-wise malignancy score.

Figure 2:

Wu et al (17) heatmaps on BreastScreen SA images. Images show left (top) and right (bottom) mediolateral oblique (MLO) mammograms (left column) with overlaid benign (middle column) and malignant (right column) patch-level heatmaps in a single individual with biopsy-proven invasive ductal carcinoma. Heatmaps were produced using a convolutional neural network trained by Wu et al (17) (the model was not retrained). Note the high probability of malignant segmentation for densities in the malignancy-free right breast and bilaterally in the axillae. Right-sided images were horizontally flipped for model training, validation, and testing.

Retraining on South Australian Local Data

Both the NYU1 and NYU2 CNNs were (a) trained from scratch on local data (with randomly initialized weights) and (b) retrained on the local training subset with weights initialized from the NYU weights (transfer learning). In all experiments, the network architecture was unchanged. The heatmap-generating CNN that produced input data for NYU2 was not retrained due to the lack of availability of hand-drawn segmentations in the BSSA dataset. We selected the best-performing hyperparameters on the validation data (see Model Training in Appendix S1), and we report the AUCs of each model after training from scratch with no transfer learning and after retraining with transfer learning (using the NYU weights).

Additional Experiments

We assessed the effect of disease prevalence on performance with a second local test set, LTS2 (Fig 1). We randomly sampled a larger number of age-matched control individuals to approximate the prevalence of malignancy in the work by Wu et al (17) (8.4% breast-wise cancer prevalence, 16.8% individual-wise prevalence). LTS2 included the same individuals with malignant and benign lesions as LTS1 and none of the control individuals used in the training, validation, or LTS1 subsets. The individual-wise malignancy prevalence in LTS2 was 13.06% (see Fig 1). We also present results stratified by individual characteristics and radiologic findings. We were not able to assess performance with respect to breast density, as it is not routinely collected in BSSA (30).

Statistical Analysis

For all experiments, receiver operating characteristic (ROC) curves and the areas under these curves (AUCs) were calculated. CIs were calculated using the bootstrap method (31), resampling test set individuals with replacement for 10 000 iterations. Independent ROC curves from the NYU and BSSA datasets were statistically compared using the methods of Hanley and McNeil (32). Dependent ROC curves within the BSSA dataset were statistically compared using the methods of DeLong et al (33).

All model development and analysis used Python version 3 (Python Software Foundation) and PyTorch Lightning (34), a PyTorch wrapper. Statistical significance was set at P < .05.

Code Availability

All code used for this study is available at https://github.com/jamesjjcondon. This repository is not currently operational for replication and relies on private BSSA data. It is made available for transparency of model retraining and evaluation.

Results

Dataset Characteristics

There were 959 individuals evaluated in LTS1 (mean age, 62.5 years ± 8.5 [SD]; all female): 425 individuals with malignancy including ductal carcinoma in situ (mean age, 62.5 years ± 8.6), 44 individuals with benign lesions (mean age, 61.7 years ± 7.2), and 490 controls (mean age, 62.6 years ± 8.5). There were 3254 individuals evaluated in LTS2 (mean age, 62.4 years ± 8.5; all female): the same 469 individuals with malignant and benign lesions as in LTS1 and 2785 controls (mean age, 62.4 years ± 8.5; see Tables 1, S1, and S2 for detailed dataset characteristics).

Model Performance on External Test Set

For comparison of performance, we refer to the results of the single models described by Wu et al (17) (rather than the results of ensembles of models). As shown in Figure 3 and Table 2, the NYU1 model without retraining achieved an AUC in LTS1 of 0.76 (95% CI: 0.73, 0.79), which, compared with the AUC of 0.83 (95% CI: 0.82, 0.84) on NYU data (17), was a drop of 0.07 (P < .001). The NYU2 model without retraining achieved an AUC of 0.84 (95% CI: 0.82, 0.87) when applied to LTS1, which, compared with the original AUC of 0.89 (95% CI: 0.88, 0.89) on NYU data (17), was a drop of 0.05 (P = .001).

Figure 3:

Receiver operating characteristic (ROC) analysis of the New York University (NYU) models, without (static) and with retraining, for breast cancer detection. Graphs show the effect of retraining (A) the image-only model (NYU1) and (B) the image and heatmap model (NYU2) with transfer learning. AUC = area under the ROC curve.

Table 2:

Malignancy Differentiation in Test Set 1

Model Performance after Retraining with and without Transfer Learning

Retraining of NYU1 and NYU2 on LTS1 without transfer learning resulted in AUCs of 0.66 (95% CI: 0.62, 0.69; P < .001 vs performance of original model on NYU data) and 0.86 (95% CI: 0.84, 0.88; P = .04), respectively.

Retraining of NYU1 and NYU2 on LTS1 with transfer learning resulted in AUCs of 0.82 (95% CI: 0.80, 0.85; P = .001 vs performance of original model on LTS1, P = .54 vs performance of original model on NYU data) and 0.86 (95% CI: 0.84, 0.88; P = .28 and .04), respectively (Fig 3, Table 2).

For the best-performing model, retrained NYU2 with transfer learning—with a validation operating point chosen for validation subset sensitivity above 90%—the LTS1 sensitivity and specificity were 93% (95% CI: 91%, 96%) and 54% (95% CI: 50%, 58%), respectively. For comparison with a previously published partial AUC of 0.88 for human radiologists (with a maximum false-positive rate of 40%, ie, a specificity of 60%) (12), the partial AUC of retrained NYU2 on the LTS1 dataset was 0.81 (95% CI: 0.79, 0.84).

Additional Analyses

On LTS2, the AUC of the NYU2 model retrained with transfer learning was 0.88 (95% CI: 0.86, 0.90; Fig S2), close to the AUC of 0.89 observed in the original NYU screening test set (P = .23 vs performance of original model on NYU data). For results by radiologic and individual characteristics, see Table 3 and Figures S1 and S2.

Table 3:

Performance Results by Individual and Radiologic Characteristics for NYU2 Model Retrained with Transfer Learning

Discussion

To our knowledge, this is the first study to investigate the validation of an internationally trained deep learning system in an Australian screening mammography dataset. We demonstrate incomplete replication and hence reduced generalizability of the NYU models in local data. By retraining with transfer learning on local data, we were able to achieve a level of performance similar to that originally reported by Wu et al (17) on NYU data. The original AUCs of the NYU1 and NYU2 models were 0.83 (95% CI: 0.82, 0.84) and 0.89 (95% CI: 0.88, 0.89), respectively. The AUCs of the NYU1 and NYU2 models after retraining with transfer learning were 0.82 (95% CI: 0.80, 0.85; P = .54 vs performance of original model on NYU data) and 0.86 (95% CI: 0.84, 0.88; P = .04), respectively. We note that the additional input of patch-level heatmaps improved performance in our dataset, even without local retraining of the heatmap-generating model, supporting the findings reported in Wu et al (17). However, we also note that the overall performance for all the models evaluated was below an acceptable clinical standard in our dataset.

Recent external testing shows that while some models do appear to generalize well across populations and mammography system vendors, as demonstrated in one study (external testing AUC of 0.96) (16), many do not (external testing AUC of 0.44–0.65) (20). While Salim et al (16) did not investigate the cause of the preserved performance in their study, they did note that the best-performing model in external testing was trained on more data, utilized pixel-level annotations for training, and was a higher-capacity model than the other two models tested. It is worth noting that the best-performing model in our dataset (NYU2) also made use of pixel-level information during training. Regardless, our results show that testing models on local data prior to clinical use is critical to avoid potentially significant decreases in diagnostic accuracy.

There are several possible explanations for the reduced original model performance observed in our study. There were subtle visual differences when comparing BSSA and NYU images, despite preprocessing (Figs 4, S5). Performance across the devices from different vendors used in our local dataset was similar, but we could not assess the impact of the differences in vendors and devices between NYU and BSSA. This difference may have influenced the drop in performance. The differences in image size across datasets (Fig S6) may have also played a role in the reduced performance in local testing. Some variation in size is unavoidable in mammography; the acquisition process can introduce variation in the size of breast tissue and its contents relative to the pressure applied to the breast, and there is natural variation in breast and lesion size. The NYU system, like many CNNs, accepts a fixed image size, necessitating the resizing of images prior to processing.

Figure 4:

BreastScreen SA (left) and New York University (right) mammograms in the mediolateral oblique (MLO) view (not to scale) show subtle differences in image contrast and opacity that may affect model generalizability.

It is important to recognize that the observed improved performance with retraining was dependent on the availability of the original model and weights, as without access to the weights, retraining with transfer learning would not be possible. Access to model details and weights also expedited retraining and transferred expertise (hand-drawn segmentations) (see Significance of Transfer Learning in Appendix S1).

Our study had several limitations. One limitation was that data on interval cancers were not available in the BSSA dataset. The lack of interval cancer data could produce a bias in our results, as a subset of these cancers will be visibly present on the original images (missed cancers) as opposed to cancers that developed between screenings and were not visible on the original images. This could inflate the false-positive rate of the deep learning models, which may correctly identify early or subtle cancers that were not detected at screening. This effect is likely to be very small and unlikely to substantially influence our findings.

Another limitation is that ground truth may potentially be a source of variation in diagnostic accuracy when attempting to replicate mammography deep learning systems across populations and datasets. Except for a small number of individuals who did not undergo surgery, our ground truth for individuals with malignancy was the postsurgical specimen evaluation, where all excised tissue underwent pathologic examination and diagnosis with Systematized Nomenclature of Medicine (SNOMED) classification. The ground truth in Wu et al (17) was based on text mining of biopsy pathology reports, and a significantly larger portion of NYU biopsies were for benign lesions (76.2% vs 10.2% in the BSSA dataset; see Table S3). The pathologic subtype terminology used in BSSA is different from that presented in Wu et al (see Tables S3–S6). In screening mammography datasets, it is possible that individuals with a screening round labeled as normal subsequently went on to develop a tumor that was diagnosed and treated in a separate medical system, although this is unlikely to have a large impact on our modeling. Despite differences in ground truth labeling, both models were “ignorant” of tumor subclass labels and were trained at a higher level of abstraction: malignant or not. While different demographic and lifestyle factors are associated with the prevalence of different tumor subtypes (35), the absolute number of subtypes and the magnitude of any differing radiologic appearances is unlikely to have any significant effect on performance. In addition, some proportion of invasive cancers will likely be completely mammographically occult, only detectable with other modalities such as US or MRI (based on current technology). If this proportion differs significantly across populations, this may affect the performance of externally tested deep learning systems. Of note, due to our exclusion criteria, the task of differentiating malignancy in this dataset was less difficult compared with clinical practice. The visual detection of malignancy in individuals with only one breast or with previous biopsies and/or surgery, for example, can be more challenging.

In conclusion, we replicated, with retraining, the performance results of two breast cancer classification models developed in the United States on South Australian data. Transfer learning with access to complete model details and weights—which are not generally available—allowed the deep learning system developed in New York to be validated in a public screening mammography dataset from South Australia. However, we note that neither model would be clinically useful as a stand-alone reader in the South Australian population, as specificity was considerably lower than the current clinical standard for screening mammography. Our findings suggest that (a) model performance may be reduced in a different clinical environment and (b) provision of complete model details and weights is beneficial to enabling deep learning systems to be adapted to local clinical environments. Local testing of deep learning systems is likely to be necessary for predeployment model assessment, regulatory approval, and safe clinical translation, as the reduced performance of deep learning systems in new environments poses a substantial barrier to the widespread use of artificial intelligence in radiology.