Evaluation of Computer-aided Diagnosis on a Large Clinical Full-Field Digital Mammographic Dataset

Abstract

Rationale and Objectives:

To convert and optimize our previously developed computerized analysis methods for use with images from full-field digital mammography (FFDM) for breast mass classification in order to aid in the diagnosis of breast cancer.

Materials and Methods:

An institutional review board approved protocol was obtained, with waiver of consent for retrospective use of mammograms and pathology data. Seven hundreds and thirty-nine full-field digital mammographic images, which contained 287 biopsy-proven breast mass lesions, of which 148 lesions were malignant and 139 lesions were benign, were retrospectively collected. Lesion margins were delineated by an expert breast radiologist and were used as the truth for lesion-segmentation evaluation. Our computerized image analysis method consisted of several steps: 1) identified lesions were automatically extracted from the parenchymal background using computerized segmentation methods; 2) a set of image characteristics (mathematical descriptors) were automatically extracted from image data of the lesions and surrounding tissues; and 3) selected features were merged into an estimate of the probability of malignancy using a Bayesian artificial neural network classifier. Performance of the analyses was evaluated at various stages of the conversion using receiver operating characteristic (ROC) analysis.

Results:

An AUC value of 0.81 was obtained in the task of distinguishing between malignant and benign mass lesions in a round-robin by case evaluation on the entire FFDM dataset. We failed to show a statistically significant difference (P value=0.83) as compared with results from our previous study in which the computerized classification was performed on digitized screen-film mammograms (SFM_D).

Conclusion:

Our computerized analysis methods developed on digitized screen-film mammography can be converted for use with FFDM. Results show that the computerized analysis methods for the diagnosis of breast mass lesions on FFDM are promising, and can potentially be used to aid clinicians in the diagnostic interpretation of FFDM.

INTRODUCTION

Breast cancer is the most frequently diagnosed cancer in women in the United States (1). An estimated 178,480 new cases of invasive breast cancer and 62,030 new cases of in situ breast cancer are expected to occur among women during 2007. An estimated 40,460 breast cancer deaths are expected in 2007 (1). Screening mammography has been the most effective tool for early cancer detection over the past several decades (2, 3), and it has been shown to reduce the cancer mortality by as much as 40% (4, 5). In addition, computer-aided detection (CADe) methods have been shown to improve the detection of more cancers in mammography screening (6, 7).

Once a lesion is detected, diagnostic imaging workup is performed in order to determine if a biopsy is warranted. Computer-aided diagnosis (CADx) has been proposed to aid the radiologist during diagnostic mammography interpretation (8). Most of the computerized analysis methods have been developed using databases of digitized screen-film mammograms (SFM_D) (9-13). In recent years, full-field digital mammography (FFDM) has been approved by the Food and Drug Administration (FDA) for clinical use. There were 13,559 mammography units, in which 13.8% were FFDM units, as of October 1 of 2006 in the United States (14). Because of the digital nature of FFDM, it offers many advantages such as image storage, image transmission and retrieval, and digital image processing. With the easy access to digital images, computerized image analyses can be directly applied to FFDM without the need for film digitization, as is needed with screen-film mammography.

We have previously developed computerized analysis CADx methods for the interpretation of mammographic mass lesions in order to aid clinicians in the diagnosis of breast cancer (9, 15-19). Our initial development and evaluation were performed on digitized screen-film mammograms (20). It is important to note that the purpose of our current study is not to report on novel computerized analysis methods, but rather to convert and optimize our previously developed methods for the analysis of SFM_D to those for FFDM. At the various stages of our conversion to FFDM, we evaluated the performance of the computerized methods in the task of distinguishing between malignant and benign mass lesions.

MATERIALS AND METHODS

Database

An institutional review board (IRB) approval was obtained for retrospective collection of mammograms and pathology data at the University of Chicago Hospitals. Data collection and usage were compliant with the Health Insurance Portability and Accountability Act (HIPAA) regulations. The full-field digital mammograms used in this study were acquired with a GE (Waukesha, WI) Senographe 2000D FFDM system in the Department of Radiology at the University of Chicago Medical Center. The FFDM images were acquired at 12-bit quantization with a pixel size of 100 μm.

A total of 739 FFDM images were obtained. There were 287 biopsy-proven mass lesions, of which 148 lesions (412 images) were malignant and 139 lesions (327 images) were benign. These FFDM images were diagnostic exams and performed between 2002 and 2005 from 190 patients. The number of images per lesion varied from one to thirteen, including both standard views and special views. Most lesions had two to three images available for the study. All lesions were outlined by an expert breast imaging radiologist. The distribution of breast density for these cases in terms of Breast Imaging Report and Data System (BI-RADS) is shown in Figure 1.

Figure 1.

The distribution of breast density in terms of BI-RADS for the lesions in the entire dataset.

Computerized classification methods

The computerized analysis method for mammographic lesions was initially developed and evaluated on digitized screen film mammograms, and has been reported elsewhere (9,16,18-20). The method consists of several steps: (a) automated extraction of the lesion from the surrounding parenchymal background; (b) automatic lesion feature extraction in terms of mathematical descriptors; and (c) merging of lesion features into an estimate of the probability of malignancy (PM).

For a given lesion, the analysis proceeds as follows. First, the center of the lesion is manually indicated, and then automatic lesion segmentation is performed. The automatic lesion segmentation method is based on a multi-transition-point, gray-level, region-growing technique, and has been described in detail elsewhere (9). It is important to note that the same segmentation parameters from our previous SFM_D study were applied on the FFDM images. Segmentation performance is assessed using an area overlap measure (21), which is calculated by the ratio of the area within the intersection of the human-delineated margin and the computer-determined margin to the union of these two regions. Once segmented, various features (mathematical descriptors) are then automatically extracted to quantify the characteristics of the lesion and its local environment (surrounding tissue). The detailed descriptions of these features can be found elsewhere (16,22). The features are then merged using a Bayesian artificial neural network (BANN) classifier to generate an estimate of probability of malignancy (23).

In this study, we investigated the computerized image analysis scheme at various stages in the conversion from SFM_D to FFDM, as shown in Figure 2. The first evaluation (Evaluation #1) was performed on FFDM images without any retraining and/or calibration of the computerized image analysis method that was previously developed using digitized screen-film mammograms (SFM_D). The same five image features used in our previous SFM_D study (20) were extracted. These features were used to quantify spiculation, margin sharpness, texture, shape, and gray level in the mammographic lesions and surrounding tissues. The same neural network classifier weights generated with the previous SFM_D training database were applied in this FFDM evaluation. Basically, this was an independent testing of the SFM_D-developed CADx on the FFDM dataset. The next evaluation (Evaluation #2) consisted of retraining the BANN using FFDM images. The only difference from Evaluation #1 was the classifier weights. In this approach, the same five features used in the SFM_D study were extracted from the FFDM images, and the neural network classifier was retrained with these five features to generate new classifier weights. The third evaluation (Evaluation #3) included both reselecting features and using these reselected features to retrain the BANN on the FFDM images. In this approach, all fifteen features that were previously developed on SFM_D were extracted. Stepwise feature selection was employed using the Wilks lambda criterion (24,25) to select a subset of features for the classification task. The selected features were merged with a BANN classifier.

Figure 2.

Evaluations of the computerized analysis at various stages of conversion from SFM_D to FFDM.

To further optimize the computerized image analysis method, fourteen new image features (22) were also extracted from the FFDM images. Linear stepwise feature selection was performed on all twenty-nine features including both previously developed features on SFM_D (16) and new image features (22). The BANN was retrained with the selected features to generate an estimate of probability of malignancy (Evaluation #4).

Performance evaluation and statistical analysis

The performance of each classifier, in the task of distinguishing between malignant and benign mass lesions, was evaluated using receiver operating characteristic (ROC) analysis (26-29), with the area under the ROC curve (AUC) used as a figure of merit. The leave-one-out (round-robin) by lesion method (30) was used in the performance evaluation for each classifier. Leave-one-out by lesion requires all images of a lesion to be removed while training with all other images. The trained classifier is then run on images of the lesion removed. The level of statistical significance among different classifiers was calculated using the ROCKIT computer program (31).

RESULTS

The performance of lesion segmentation in terms of percentage of lesion images accurately segmented at an overlap threshold cutoff of 0.4, are 68.9%, 66.0%, and 72.5% for all lesions, malignant lesions, and benign lesions, respectively.

Performance results of the CADx method at various stages of conversion are given in Table 1. As expected, due to the difference acquisition detectors for SFM_D and FFDM, in Evaluation #1, a low AUC value of 0.74 was obtained in the task of differentiating between malignant and benign lesion on the FFDM dataset. It is important to emphasize that in this independent evaluation, the five lesion features used and the trained neural network classifier weights were obtained on a prior SFM_D training dataset, i.e., without any retraining or recalibration.

Table 1.

Performances for the CADx method at various stages of conversion in the task of distinguishing between malignant and benign lesions. Here, “by case” round-robin analysis was performed on the FFDM dataset (N= 287 breast mass lesions). “SFM_D” refers to features or classifier weights obtained previously with the screen-film trained method.

Classifiers	Training Cases	Feature Selection	Classifier Weights	Testing Cases	Testing Methods	AUC ± SE	95 % C.I.
Evaluation #1	SFMD	SFMD	SFMD	FFDM	independent	0.74 ± 0.03	[0.69, 0.80]
Evaluation #2	FFDM	SFMD	FFDM	FFDM	round-robin by case	0.77 ± 0.03	[0.72, 0.82]
Evaluation #3	FFDM	FFDM (15 features)	FFDM	FFDM	round-robin by case	0.78 ± 0.03	[0.73, 0.83]
Evaluation #4	FFDM	FFDM (29 features)	FFDM	FFDM	round-robin by case	0.81 ± 0.03	[0.76, 0.86]
Prior Evaluation for comparison*	SFMD	SFMD	SFMD	SFMD	independent	0.81 ± 0.05	[0.69, 0.88]

^*reference #20

For Evaluation #2, an AUC value of 0.77 were obtained from ROC analyses in the task of differentiating between malignant and benign lesions on FFDM dataset from the leave-one-out by case analysis. Recall that only the artificial neural network was retrained on the FFDM images, but the features were those selected from our previous SFM_D study. Classifier retraining is necessary for the conversion of our computerized image analysis methods from SFM_D to FFDM, since there are intrinsic differences in the physical image quality between SFM_D and FFDM systems (32).

For Evaluation #3, three features were selected from all fifteen previously developed image features. These features were shape, texture, and contrast and they were used to quantify the characteristics of the lesions and local environment. The neural network classifier was retrained with these three lesion features. An AUC value of 0.78 was obtained from ROC analyses for the leave-one-out by case evaluation method.

For Evaluation #4, from the 29 features, five image features were selected from the feature selection. These five features were margin sharpness, shape, size, texture, and gray level that were used to characterize mass lesions and local tissues. The BANN was retrained with these five features. An AUC value of 0.81 was obtained from ROC analyses with the leave-one-out by case analysis.

In our previous study on SFM_D with an independent set of 97 lesions (20), an AUC value of 0.81 was achieved from the ROC analysis in the task of distinguishing between malignant and benign lesions.

Statistical assessments were performed on the differences in the performance measures in terms of AUC values obtained from the classifiers at the various conversion steps. The results are given in Table 2. Interestingly, we failed to show a statistically significant difference (overall α_T = 0.05) between the independent testing (Evaluation #1) and the retrained methods on FFDM (Evaluation #2, #3, #4) and our prior evaluation on SFM_D in the task of distinguishing between malignant and benign lesions.

Table 2.

Statistical analysis results for differences in the performance among different neural network classifiers that were used in SFM_D and FFDM studies. From ROCKIT, P values were calculated for differences in the classification performance for a pair of neural network classifiers. The significance level α for the individual test was calculated using Holm's procedure (overall α_T =0.05) for multiple tests of significance.42

	Evaluation #1 (FFDM)	Evaluation #2 (FFDM)	Evaluation #3 (FFDM)	Evaluation #4 (FFDM)	Prior evaluationa (SFMD)
Evaluation #1 (FFDM)	–	0.20	0.17	0.016 (α = 0.005)	0.32
Evaluation #2 (FFDM)	–	–	0.70	0.09	0.66
Evaluation #3 (FFDM)	–	–	–	0.16	0.77
Evaluation #4 (FFDM)	–	–	–	–	0.83
Prior evaluation* (SFMD)	–	–	–	–	–

^aReference #20

The low classification performance with Evaluation #1 on FFDM was expected since there were no retraining and recalibration for this approach. However, we failed to show a statistically significant difference between the reoptimization approach (Evaluation #4) and the SFM_D method (Evaluation #1) applied to FFDM in the task of distinguishing between malignant and benign lesions (p-value = 0.016 and α = 0.005). It is very encouraging that by just retraining the previous developed SFM_D classifier on FFDM, a similar classification performance was achieved. The classification performance comparison of computerized image analysis methods performed on FFDM in this study and our prior evaluation on SFM_D (20) in terms of ROC curves is shown in Figure 3.

Figure 3.

ROC curves of computerized image analysis methods performed on FFDM in this study and on SFM_D from previous study. Evaluation on SFM_D was reported in reference #20. Confidence intervals are given in Table 1.

The probability of malignancy distributions from various BANN classifiers at the various stages of conversion is shown in Figure 4. The separation between malignant and benign lesion gradually increased from Evaluation #1 to Evaluation #4, showing a trend towards improved classification performance.

Figure 4.

The distributions of the output PMs (probability of malignancy) obtained with the various Bayesian artificial neural network classifiers: (a) Evaluation #1; (b) Evaluation #2; (c) Evaluation #3; and (d) Evaluation #4. Output PMs are those from round-robin by case analyses.

DISCUSSION

In this study, we progressively evaluated our computer-aided diagnosis method on FFDM images in the classification of mammographic mass lesions. Our computerized image analysis methods were previous developed and evaluated on digitized screen-film mammograms. However, by retraining and recalibrating those existing computerized methods, similar classification performance was achieved on FFDM images. Hence, the results from this study are encouraging. It is very important to note that by simply retraining the previous developed CADx, we can achieve a similar classification performance on FFDM as on SFM_D. Our results indicate that computer-aided diagnosis methods developed for SFM_D can be converted for analysis of FFDM. It is apparent from this study that computerized image analysis techniques for SFM_D and FFDM are similar. However, with our database of 287 actual lesions, the statistical power for demonstrating the statistical significance of the difference between the AUC values for Evaluation #1 (AUC = 0.74) and for Evaluation #4 (AUC = 0.81) is 76% at α = 0.05, so we believe that reoptimization is still necessary to warrant the high classification performance. Other studies have also found that reoptimization is necessary for converting computer image analysis methods from SFM_D images to FFDM images (33, 34). However, one study showed similar performance in classifying calcifications in FFDM as malignant or benign without requiring optimizing from its initial development on SFM_D (35). Also, we want to point out here that similar breast mass lesion classification performance was achieved both on FFDM and SFM_D images in our study, the similar results were also reported by several other groups (36-41), no statistically significant difference in diagnostic performances between FFDM and SFM_D were detected. We believe that further research is needed on the assessment of CAD system performance on both SFM_D and FFDM images.

The results from this study are very similar to those that we found previously on SFM_D. This is very exciting and important since the prior results were on screen-film mammography and these are on FFDM. By simply retraining the previously developed computerized classification methods, we can achieve a similar classification performance on FFDM as on screen-film mammograms. The results from this study demonstrate the robustness of our breast mass lesion classification method, and move CADx for FFDM one step closer to clinical incorporation.

The lower classification performance from Evaluation #1 is expected, since there are differences in the physical image quality between the SFM_D and FFDM systems. The SFM_D exhibits higher spatial resolution, increased noise, and lower contrast as compared to FFDM system (32). In addition, there was no retraining and recalibrating on previous developed computerized image analysis methods from SFM_D study for Evaluation #1. However, we failed to show a statistically significant difference between Evaluation #1 and our prior evaluation on SFM_D. This may due to the small dataset in our previous SFM_D study, which resulted in larger standard error from ROC analysis. We expect to observe a statistically significant difference if the larger dataset was used.

Several ROI examples are shown in Figure 5. The computer-generated lesion contours are superimposed on mammographic lesions. The estimate of the probability of malignancies (PM) for each individual lesion was generated from the neural network classifier in Evaluation #4. The correctly-segmented malignant or benign lesions contours yield correct lesion features (mathematical descriptors), and thus reliable PM values (Figure 5a, 5b, 5d, 5e). On the other hand, the overlap of mammographic lesion and background parenchyma resulted in an under-segmented contour (Figure 5c) and an over-segmented lesion contour (Figure 5f), thus yielding erroneous computer-extracted features and unreliable PM values. Further improved computerized lesion segmentation methods may improve our computerized lesion classification performance.

Figure 5.

Selected ROI examples with computer generated lesion contours. PM values are the estimate of probability of malignancies generated from computerized image analysis methods used in Evaluation #4.

The digital nature of FFDM allows us to manage digital data more efficient, as for screen-film mammograms, digitization is needed before any computerized methods can be applied. Thus, computerized image analysis methods may be easily incorporated into existing FFDM systems in the diagnostic breast imaging area. Further research work will be needed to perform extensive training and independent testing on an even larger dataset to ensure the robustness of our computer-aided diagnosis methods.