Comparison of Segmentation-Free and Segmentation-Dependent Computer-Aided Diagnosis of Breast Masses on a Public Mammography Dataset

Abstract

Purpose:

To compare machine learning methods for classifying mass lesions on mammography images that use predefined image features computed over lesion segmentations to those that leverage segmentation-free representation learning on a standard, public evaluation dataset.

Methods:

We apply several classification algorithms to the public Curated Breast Imaging Subset of the Digital Database for Screening Mammography (CBIS-DDSM), in which each image contains a mass lesion. Segmentation-free representation learning techniques for classifying lesions as benign or malignant include both a Bag-of-Visual-Words (BoVW) method and a Convolutional Neural Network (CNN). We compare classification performance of these techniques to that obtained using two different segmentation-dependent approaches from the literature that rely on specific combinations of end classifiers (e.g. linear discriminant analysis, neural networks) and predefined features computed over the lesion segmentation (e.g. spiculation measure, morphological characteristics, intensity metrics).

Results:

We report area under the receiver operating characteristic curve (A_Z) values for malignancy classification on CBIS-DDSM for each technique. We find average A_Z values of 0.73 for a segmentation-free BoVW method, 0.86 for a segmentation-free CNN method, 0.75 for a segmentation-dependent linear discriminant analysis of Rubber-Band Straightening Transform features, and 0.58 for a hybrid rule-based neural network classification using a small number of hand-designed features.

Conclusions:

We find that malignancy classification performance on the CBIS-DDSM dataset using segmentation-free BoVW features is comparable to that of the best segmentation-dependent methods we study, but also observe that a common segmentation-free CNN model substantially and significantly outperforms each of these (p<0.05). These results reinforce recent findings suggesting that representation learning techniques such as BoVW and CNNs are advantageous for mammogram analysis because they do not require lesion segmentation, the quality and specific characteristics of which can vary substantially across datasets. We further observe that segmentation-dependent methods achieve performance levels on CBIS-DDSM inferior to those achieved on the original evaluation datasets reported in the literature. Each of these findings reinforces the need for standardization of datasets, segmentation techniques, and model implementations in performance assessments of automated classifiers for medical imaging.

Graphical Abstract

INTRODUCTION

Breast cancer is the most deadly cancer for women in developing countries and the second most deadly cancer for those in developed nations.¹ Mammograms are an essential component in early detection of breast cancer, and interpretation sensitivity greatly affects patient survival rates.² On the other hand, imperfect specificity of breast lesion diagnosis by mammography causes physical and psychological discomfort to false positive patients who are subjected to further, possibly invasive tests.^3,4 As a result, a variety of Computer Aided Diagnosis (CADx) systems designed to provide quantitative, objective mass classification have been developed.⁵

Existing CADx techniques for mass classification generally fall into two categories: segmentation-dependent methods, which require a detailed outline of the lesion upon which to compute features for classification, and segmentation-free methods, which do not. Segmentation-dependent methods tend to rely on predefined sets of features, while segmentation-free approaches leverage representation learning techniques to learn explanatory features directly from the available data. Segmentation-free approaches could provide substantial value in clinical practice by obviating the need for detailed segmentations, but have only recently been able to achieve results similar to those of segmentation-dependent methods.^6–9 Though recent studies of segmentation-free approaches have shown promising results, CADx systems for mass classification have rarely been evaluated on the same datasets, making the type of comparative performance analysis that should precede clinical deployment difficult to perform.⁸ Major limitations to the evaluation of different mammography CADx systems on common datasets have ranged from insufficient descriptions of model implementation in the original literature to challenges with data availability and provenance.

In this work, we leverage the recent Curated Breast Imaging Subset of the Digital Database for Screening Mammography (CBIS-DDSM) to compare segmentation-free and segmentation-dependent approaches to automated mass classification on a standard, public dataset. To perform this comparison, we implement four different techniques from the literature: a segmentation-free Bag-of-Visual-Words (BoVW) mass classification algorithm inspired by traditional computer vision,¹⁰ a segmentation-free Convolutional Neural Network (CNN) trained using commodity deep learning software,¹¹ a segmentation-dependent algorithm based on the Rubber-Band Straightening Transform (RBST) of Sahiner et al.,⁶ and the segmentation-dependent approach of Huo et al. that combines predefined features with an artificial neural network.⁷ Evaluating the performance of each of these techniques on the CBIS-DDSM test set after training and tuning was performed using the standard CBIS-DDSM training set yields several useful conclusions. First, we observe that malignancy classification performance on the CBIS-DDSM dataset obtained using the segmentation-free BoVW method is comparable to that of the best segmentation-dependent methods, but also find that the segmentation-free CNN model substantially and significantly (p<0.05) outperforms each of these by 11 points of area under the receiver operating characteristic curve (A_Z). These results reinforce recent findings suggesting that representation learning techniques such as BoVW and CNN can obviate the need for precise or method-specific lesion segmentation while maintaining high levels of performance. Second, we find that our re-implementations of existing segmentation-free methods yield performance levels on CBIS-DDSM inferior to those achieved on the original evaluation datasets reported in the literature. We propose that these discrepancies result from some combination of differences in the segmentation techniques used, parameter tuning on small datasets in the original work, and implementation choices.

BACKGROUND AND RELATED WORK

Due to the importance of characteristics of the lesion margin in differentiating benign and malignant tumors, many existing CADx methods have been based on obtaining mathematical descriptions of the tumor outline.^7,12–20 Such segmentation-dependent techniques require accurate segmentation of the lesion margin in order to extract image features. Methods that require hand-drawn segmentation of lesions, a process not usually performed in clinical practice, can make resultant CADx systems inefficient for clinical use. CADx systems utilizing automated lesion segmentation have, however, been studied with promising results. For example, Mudigonda et al. obtained an A_Z of 0.85 for binary classification of breast masses using hand-drawn Regions-of-Interest (ROIs) as a basis for their automated segmentation method.¹³ Likewise, Sahiner et al.¹⁴ and Huo et al.⁷ developed CADx systems using segmentation methods requiring only a general bounded region identified by the radiologist and achieved A_Z results of 0.91 and 0.94, respectively. An extensive analysis of existing methods for automated mass segmentation methods that support segmentation-dependent CADx can be found in the review of Oliver et al.²¹

More recent segmentation-free CADx approaches have attempted to attain high levels of diagnostic performance without any lesion segmentation requirements. For instance, multiple workers such as Jamieson et al.²² (A_Z = 0.71), Liu and Jiang²³ (A_Z = 0.72), and Li et al.²⁴ (A_Z = 0.796) have demonstrated the promise of segmentation-free algorithms, most of which have been based on various types of learned features. A particularly large body of work has arisen that applies deep learning algorithms such as Convolutional Neural Networks (CNNs) to automated mass analysis. The recent review of deep learning techniques in mammography by Hamidinekoo et al. found that nine major mass classification studies performed between 1996 and 2017 using deep learning approaches yielded A_Z values between 0.71 and 0.97.⁹ However, because each of these studies used different datasets, several of which are not public, it is difficult to fairly compare the performance of these different algorithms.

The focus of this work is on analyzing the performance of multiple different techniques on the relatively recent CBIS-DDSM dataset. We leverage this dataset because it (a) is provided with standard train/validation splits (b) contains cases verified by pathology (c) contains images curated by expert mammographers from multiple institutions and (d) contains a large amount of data in addition to the raw images – e.g. lesion segmentation, Breast Imaging Reporting And Data System (BI-RADS) descriptors, BI-RADS abnormality ratings, and severity ratings – that make it amenable to analysis via a wide variety of machine learning approaches. Note that BI-RADS describes both a classification system and lexicon for reporting breast imaging results that includes both descriptors of imaging features that were shown to correlate with high predictive values associated with either benign or malignant disease, and a classification system to describe the likelihood that the imaging findings represent malignancy.²⁵

For the sake of completeness, we provide additional background on recent work in mass classification and the datasets on which these studies were evaluated here. Wang et al. extracted five manually-designed image features and applied logistic regression for classification, achieving A_Z of 0.806±0.025 on an internal dataset.²⁶ Zhu et al. developed a deep multi-instance network and achieved A_Z of 0.859 ± 0.03 on the public INbreast dataset.²⁷ Arevalo et al. used a combination of CNN-extracted features and hand-crafted features and obtained A_Z of 0.826 on the public BCDR-FM dataset.²⁸ Kim et al. applied ResNet to a large-scale mammography dataset from 5 institutions, and achieved A_Z of 0.906.²⁹ Ribli et al. applied Faster R-CNN to the INbreast database, which achieved A_Z of 0.95.³⁰ Al-masni et al. applied a YOLO model for this task and achieved 5-fold cross-validation A_Z of 0.965 on a subset of the DDSM dataset.³¹ Lotter et al. developed a curriculum training method with a multi-scale CNN and obtained 0.901 ± 0.031 on a subset of DDSM; these authors also note “as full image mammogram classification lacks standardized evaluation framework, it is somewhat difficult to directly compare our results to other work.”³² In other words, because most studies are evaluated on different, incompletely reported, or private datasets, it is difficult even for researchers in the field to understand how the performance of their methods compares to that reported by other work. Ting et al. used hierarchical features from a CNN and achieved AUC 0.92 ± 0.02 on a subset of DDSM.³³ Chougrad et al. applied various CNNs including VGG, ResNet-50 and Inception networks, and achieved A_Z of 0.99 on MIAS dataset and 0.98 on a subset of the DDSM dataset.³⁴

Many studies have also used the CBIS-DDSM dataset we leverage here, albeit in slightly different ways. Ragab et al.³⁵ and Li et al.³⁶ applied convolutional neural networks on segmented ROIs, and achieved A_Z of 0.94 and 0.85 on CBIS-DDSM respectively. Tsochatzidis et al. examined multiple popular CNNs, and achieved A_Z of 0.859 and 0.804 on DDSM-400 and CBIS-DDSM respectively.³⁷ Chougrad et al. proposed a multi-label classification setting and fine-tuned a pre-trained CNN, achieving mean A_Z of 0.89 ± 0.08 for 5-fold cross-validation on CBIS-DDSM.³⁸ Chen et al.³⁹ and Falconi et al.⁴⁰ fine-tuned CNNs and achieved A_Z of 0.86 and 0.844 on CBIS-DDSM respectively. Alkhaleefah et al. fine-tuned VGG-19 and applied data augmentation techniques on their own data splits of CBIS-DDSM, and achieved A_Z of 0.961.⁴¹ Shu et al. proposed different pooling structures for CNNs and obtained A_Z of 0.838±0.0001 on CBIS-DDSM.⁴² Samala et al. aimed to assess the generalization errors of CNNs, and found A_Z of 0.83±0.03 on internal and CBIS-DDSM combined data.⁴³ Gossmann et al. investigated the performance deterioration of deep neural networks for lesion classification due to distribution shift, and achieved A_Z of 0.833 on CBIS-DDSM.⁴⁴ Beltran-Perez et al. developed a three-step pipeline to extract image features using a multiscale generalized radial basis function and the discrete cosine transform, and achieved 93.99% accuracy on CBIS-DDSM.⁴⁵ Ansar et al. applied transfer learning of MobileNet and obtained 74.5% accuracy on CBIS-DDSM.⁴⁶ De Vriendt et al. proposed a all-in-one graph-based deep semi-supervised learning framework, and obtained A_Z of 0.811 with only 40% labeled data on CBIS-DDSM.⁴⁷

A small number of existing studies compare multiple computer assisted detection or diagnosis techniques on the same datasets. The work of Oliver et al. implements multiple approaches for mass detection techniques on a single dataset, but does not address mass classification.²¹ The recent work of Kooi et al. compares a CNN-based approach to a single reference system based on extracted features, but does so on a non-public dataset.⁸ They find that the segmentation-free CNN approach (A_Z = 0.93) slightly outperforms their segmentation-dependent, feature-based method (A_Z = 0.91).

MATERIALS AND METHODS

In order to compare multiple segmentation-free representation learning approaches to several segmentation-dependent predefined feature methods using a standard, public dataset, we implemented four high-performing methods from the literature that are both trained and evaluated using standard splits from the CBIS-DDSM mass classification dataset.⁴⁸ Segmentation-free techniques analyzed include both a BoVW approach adapted from traditional computer vision as well as a standard CNN-based approach from the field of deep learning.^10,11 Segmentation-dependent techniques analyzed include two high-performing approaches from the literature: linear discriminant analysis of features computed on the lesion margin as described by Sahiner et al.⁶ and hybrid rule-based neural network classification of a small number of salient features as described by Huo et al.⁷ We obtained existing code for each technique where possible, and re-implemented the remainder as faithfully as possible following descriptions in the literature; specific implementation decisions are described in detail below. We note that it was not possible to obtain code for the exact methods for lesion segmentation implemented in previous studies, and we therefore use the segmentations provided as part of the public CBIS-DDSM dataset. The technique used by Lee et al.⁴⁹ to compute these segmentations was based on the Chan-Vese local level set framework,⁵⁰ where the coarse annotation from the original DDSM dataset was used for initialization. The process of computing and validating these segmentations is described in detail by Lee et al.⁴⁹ We describe each method implemented for this study in detail below; implementations are built using MATLAB (v. R2011) and Python 3.6 unless otherwise noted. Note that each method uses only the mammogram image, and no other descriptors included in the CBIS-DDSM dataset.

Segmentation-Free Representation Learning Methods

Each of the segmentation-free methods described below relies on classification of a set of features learned directly from training data, and does not require lesion segmentation.

Bag-of-Visual Words

Figure 1 outlines the procedure for the BoVW method. Each image was first preprocessed using the method developed by Chan et al., which involves filtering a bounded portion of the image around the ROI in order to smooth out structures in the background tissue that may obscure the mass margin.^14,51 Figure 2 shows an example of an ROI before and after this preprocessing. We then utilize the Scale-Invariant Feature Transform (SIFT) to compute the primitive feature set on which our BoVW method is based using a bounding box around the entire ROI; all SIFT features were computed using the VLFeat open source library in MATLAB (v. R2011).⁵² Subsequent image classification is based on a feature histogram created by counting the number of image patches assigned to each individual visual word by a consensus clustering approach described in detail in the Supplementary Material. Finally, we train an L₁-regularized logistic regression classifier, also known as Least Absolute Shrinkage and Selection Operator (LASSO), utilizing the glmnet (v. 2.0–16) software package.⁵³ Hyperparameters, such as the number of clusterings upon which to evaluate consensus clustering and the regularization parameter for LASSO, were tuned using 20% of the training data as a held-out validation set.

Figure 1.

Flowchart of a Bag-of-Visual-Words (BoVW) method, which relies on sequential steps including image processing, feature extraction, formation of visual words dictionary, and ultimate classification.

Words (BoVW) classifier.

Figure 2.

Example of a Region-of-Interest (ROI) before (a) and after (b) preprocessing for the Bag-of-Visual-Words (BoVW) classifier.

Convolutional Neural Network

We evaluate the performance of a standard Convolutional Neural Network (CNN) architecture implemented using the Keras (v. 2.2.0) Python software package with a Tensorflow backend.^54–56 Due to its superior performance on a variety of image classification tasks both inside and outside of medicine, we utilize a 121-layer Densely Connected CNN (DenseNet-121) of Huang et al. for this comparison.^11,57,58 The DenseNet-121 model was trained on a random 90% sample of the DDSM-CBIS training set and validated on the remaining 10% for 100 epochs using a single Tesla P100 GPU with a batch size of 32 images, a dropout rate of 0.2, a learning rate of 0.001, and the Adam optimizer. Hyperparameter values were determined using coarse grid search in the vicinity of default parameters. The network was initialized using weights from a model pre-trained on the ImageNet dataset, all parameters were assumed learnable, and the learning rate was decreased by a factor of $\sqrt{0.1}$ when validation accuracy had not increased for more than 10 epochs.⁵⁷ All images were mean-standard-deviation-normalized, cropped to 750 × 750 pixels around the segmentation centroid, and further downsampled to 224 × 224 pixels before entering into the CNN. Data augmentation was achieved via random application of mild zooms (in the range of [0.8, 1.2]), horizontal flips, and random rotations. While such augmentations are important in most state-of-the art image classification results, note that due to the physical particulars of mammography, augmentations such as contrast and brightness enhancements were specifically not applied.⁵⁹

Segmentation-Dependent Predefined Feature Methods

Each of the segmentation-dependent methods described below is drawn from the CADx literature, and relies on classification of a set of features defined a priori over a provided lesion segmentation.

Linear Discriminant Analysis of Rubber-Band Straightening Transform Features (LDA-RBST)

The CADx system of Sahiner et al. relies on linear discriminant analysis performed on a variety of predefined features, including those derived using the Rubber-Band Straightening Transform (RBST) for which Sahiner has generously provided the code.⁶ The RBST converts the margin of the image into a straight line as shown in Figure 3, and is accomplished by determining the normal direction to each ROI edge pixel and taking 40 pixels along that direction to compose each line of the RBST. Several texture features were computed from the RBST image, including features from the gray-level co-occurrence matrix (GLCM) at ten pixel differences (d = 1, 2, 3, 4, 6, 8, 10, 12, 16, 20) and four directions (θ = 0°, 45°, 90°, 135°) as well as run-length statistics (RLS) features in four directions ( θ = 0°, 45°, 90°, 135°). Additionally, various morphological features were obtained from the original ROI image. Each set of features is listed in Table 1.

Figure 3.

Example Region-of-Interest (ROI) and resulting Rubber-Band Straightening Transform (RBST) image.

Table 1.

List of features employed by method of Sahiner et al.

Morphological	GLCM	RLS
Fourier descriptor	Difference average	Long run emphasis
Convexity	Difference entropy	Run percentage
Rectangularity	Inverse difference moment	Gray level non-uniformity
Perimeter	Difference variance	Run length non-uniformity
NRL mean	Inertia	Short run emphasis
Contrast	Correlation
NRL entropy	Inf. Measure of correlation 1
Circularity	Inf. Measure of correlation 2
NRL area ratio	Energy
NRL standard deviation	Entropy
NRL zero-crossing count	Sum variance
Perimeter-to-area ratio	Sum entropy
Area	Sum average

Neural Network Classification of Hand-Designed Features (NN-HDF)

Finally, we re-implemented the feature extraction method from Huo et al. that supports a hybrid rule-based neural network classifier.⁷ Five features were included in this method: spiculation measure, sharpness, average gray level, contrast, and texture. The spiculation measure was the key feature for the original work of Huo et al.⁷ It is found using radial edge gradient analysis over the four different neighborhoods shown in Figure 4, where Sobel filters were used to obtain the gradient magnitude and orientation of the ROI.⁶⁰ The orientation was then normalized based on the radial direction, and a gradient-magnitude-weighted histogram of the normalized orientation was found. The spiculation measure is the Full-Width at Half Maximum (FWHM) of this histogram. As Huo et al. do not describe the exact method for determining the FWHM required for their method, we utilize the following straightforward computation. We first smooth the histogram with an averaging filter of length two. We then find the first and last bins at which the histogram exceeded the half maximum. Finally, we use linear interpolation to determine the exact bin position and convert to degrees. We use 24 bins, allowing for a bin size of 15° as in the original paper. We note here that Huo et al. report that only an approximately correct outline of the mass lesion was required for the purposes of this analysis.

Figure 4.

Neighborhoods used in NN-HDF method (excluded areas in black). Panel (a) represents the segmented mass, panel (b) represents the mass margin, panel (c) represents the mass plus the surrounding periphery, and panel (d) represents the surrounding periphery.

Evaluation

Each method was trained (where appropriate) and tested using the CBIS-DDSM data set with the provided train and test splits.⁴⁸ The data set includes 691 training cases (355 benign, 336 malignant) and 200 test cases (117 benign, 83 malignant). We assess the performance of each method by analyzing A_Z values and associated 95% confidence intervals.⁶¹

Statistical Techniques

Model performance was assessed using A_Z on a held-out test set, computed using either the scikit-learn (v 0.19) Python library or MATLAB (v. R2011). A statistical test of non-inferiority implemented in the rocNIT (v. 1.0) R library was used to compare different classifiers characterized by similar performance levels. The method of Hanley and McNeil was used to compute 95% confidence intervals on A_Z values, and p-values less than 0.05 were considered statistically significant throughout the analysis.⁶² Statistical computations performed by J.A. D., A.H., and R.S.L..

RESULTS

Table 2 contains the results for each method described above on CBIS-DDSM. Most noticeably, the CNN substantially and significantly outperforms all other approaches with an A_Z of 0.86 [0.83, 0.89] (p<0.05); note that these results are on par with the best segmentation-free results of which we are aware on the standard CBIS-DDSM dataset. Amongst the remaining methods, the LDA-RBST technique of Sahiner et al. yielded the best classification performance results with an A_Z of 0.75 [0.69, 0.81], followed closely by the BoVW method with an A_Z of 0.73 [0.66, 0.79]. A non-inferiority test performed on the results from these two techniques results in a p-value of 0.01 for a δ of 0.15 and an α of 0.05, demonstrating significant non-inferiority of the BoVW method with respect to that LDA-RBST.⁶³ The NN-HDF method performed significantly worse than all other methods on CBIS-DDSM, with an A_Z value of 0.58 [0.51, 0.65] (p<0.05). We present additional results specific to each technique in detail below.

Table 2.

Results for different classification methods on CBIS-DDSM dataset.

Method	AZ [95% Confidence Interval]
BoVW	0.73 [0.66, 0.79]
CNN	0.86 [0.83, 0.89]
LDA-RBST	0.75 [0.69, 0.81]
NN-HDF	0.58 [0.51, 0.65]

Segmentation-Free Representation Learning Methods

Bag-of-Visual-Words

As described in the Supplementary Material, the BoVW method requires parameter optimization with respect to the number of clusterings used in consensus clustering and the regularization parameter in LASSO. Based on the empirical findings shown in Figure 5, we chose 10 clusterings and a regularization parameter value of 0.014. With these optimized parameters, we achieved A_Z of 0.73 using the BoVW method.

Figure 5.

Results per parameter using Bag-of-Visual-Words (BoVW) method: (a) area under the receiver operating characteristic curve (A_Z) versus number of clusterings using Scale Invariant Feature Transform (SIFT) features, (b) mean-squared error versus logarithm of regularization parameter λ.

Convolutional Neural Network

The fundamental difference between the DenseNet-121 deep learning approach and other methods in this paper is the fact that deep neural networks are able to learn their own feature maps in a manner that best explains the data available. Thus, the trained neural network is itself a feature extractor, and the combination of a fully connected linear layer and a softmax operator is responsible for classification. Performing the training procedure described above with ten different random seeds yielded best-case test set A_Z of 0.88 [0.85, 0.90], worst-case test set A_Z of 0.85 [0.82, 0.89], and median A_Z of 0.86 [0.83 0.89], which represents the best performance of any method described in this manuscript.

In order to ensure that this classification performance is not caused by anomalies within the data or training process, we compute class activation maps (CAMs) to assess whether the CNN classifications are leveraging appropriate spatial regions of the image.⁶⁴ Pertinent visualizations can be found in Figure 6, where we observe that correct classifications result when the network activations are directly over the lesion, while errors in both directions occur when weights for the correct class are high in spatial areas outside of the mass itself.

Figure 6.

Images and class activation maps (CAMs) from the convolutional neural network (CNN) model for (a) true positive, (b) false positive, (c) false negative, and (d) true negative examples. CAMs presented depict areas most responsible for the correct classification in red and those least responsible in blue– i.e. for examples (a) and (c), weights are those for the malignant class while in examples (b) and (d) these weights are for the benign class.

Segmentation-Dependent Predefined Feature Methods

Linear Discriminant Analysis of Rubber-Band Straightening Transform Features

Our analysis is similar to that of Sahiner et al.,⁶ where we choose the top ten features from morphological and texture features separately as well as together for use in linear discriminant analysis. Feature selection was accomplished using Wilks’ lambda. The resulting A_Z from each of these scenarios was 0.62, 0.70, and 0.75, respectively. Table 3 lists the top ten features found for each category. Note that Sahiner et al.⁶ specifically mention that they expect the selected features to change based on the particular training dataset used, so our procedure represents an intended implementation of this technique.

Table 3.

List of top features from method of Sahiner et al.

Top 10 Morphological	Top 10 Texture	Top 10 Combined
Perimeter	RLS Long Run Emphasis of RBST, Offset: 45°	Perimeter
Contrast	GLCM Entropy of RBST, Offset: −2, 2, Distance: 2	Contrast
Mean of RDSa)	GLCM Entropy of RBST, Offset: −3, −3, Distance: 3	GLCM Difference Entropy of RBST, Offset: −16, −16, Distance: 16
Area Ratio of RDS a)	GLCM Difference Variance of RBST, Offset: 0, 12, Distance: 12	GLCM Difference Average of RBST, Offset: −12, 12, Distance: 12
Perimeter to Area Ratio	GLCM Difference Variance of RBST, Offset: −12,−12, Distance: 12	GLCM Difference Variance of RBST, Offset: −4, 0, Distance: 4
Convexity	GLCM Difference Variance of RBST, Offset: −6, −6, Distance: 6	GLCM Difference Variance of RBST, Offset: 0, 8, Distance: 8
Area	GLCM Entropy of RBST, Offset: −8, 0, Distance: 8	GLCM Difference Entropy of RBST, Offset: 0, 16, Distance: 16
Entropy of RDS a)	GLCM Energy of RBST, Offset: −16, −16, Distance: 16	GLCM Entropy of RBST, Offset: −2, 0, Distance: 2
Standard Deviation of RDS a)	GLCM Difference Average of RBST, Offset: −2, 2, Distance: 2	GLCM Entropy of RBST, Offset: 0, 4, Distance: 4
Zero-crossing Count of RDS a)	GLCM Inverse Difference Moment of RBST, Offset: −6, −6, Distance: 6	GLCM Sum Entropy of RBST, Offset: 0, 1, Distance: 1

^a)Radial Distance Signal

Neural Network Classification of Hand-Designed Features

Huo et al. developed a hybrid rules-based neural network classifier in their work.⁷ The rule pertains to the spiculation measure, automatically concluding that any mass with a spiculation measure higher than 160° was malignant, and using the rest of the features as input to a neural network to determine the malignancy of the lesions with lower spiculation measure. The results of our re-implementation of this method evaluated on CBIS-DDSM are shown per feature in Table 4 along with the results reported by Huo et al. on their original dataset. The resulting A_Z for the hybrid classifier using the 160° threshold was 0.51. With a threshold of 320° optimized for our dataset, the A_Z was 0.58.

Table 4.

Per-feature Results of NN-HDF Method on CBIS-DDSM and Literature Datasets

Feature	Literature AZ	CBIS-DDSM Az
Spiculation Measure	0.88	0.53
Sharpness	0.53	0.53
Average Gray Level	0.65	0.52
Contrast	0.59	0.52
Texture Measure	0.54	0.51

DISCUSSION

Our analysis supports several distinct conclusions. First, we find that the two segmentation-free methods are able to classify benign vs. malignant masses as well as or better than segmentation-based methods that use predefined features. In particular, while BoVW performs similarly to the best predefined feature method, the deep learning method (A_z = 0.86) improves A_z by 11 points over the best competing segmentation-based method (A_z = 0.75). These results support the conclusion that the two segmentation-free mass classification methods that leverage representation learning, BoVW and CNN, can obviate the need for accurate segmentations while improving performance with respect to traditional segmentation-based CADx methods that use predefined features. While our observation that segmentation-free deep learning models can outperform segmentation-based models is consistent with expectations of previous work,⁸ our study is the first to demonstrate this across multiple different techniques on a standard, public mammography dataset. A robust finding that mammography CADx could safely move to segmentation-free methods would benefit busy clinical workflows, as providing precise lesion segmentation or region-of-interest outlines can be laborious and time consuming in practice. Thus, further evaluation studies in larger patient cohorts and more diverse image sets would be useful to confirm the performance improvement we have observed on CBIS-DDSM from segmentation-free representation learning techniques.

Our second important finding is that our re-implementation of existing segmentation-dependent methods yielded performance levels on CBIS-DDSM inferior to those reported on the original evaluation datasets in the literature. For instance, Huo et al. reported an A_Z of 0.88 using only the spiculation measure feature, while our re-implementation achieved an A_Z of only 0.53 on CBIS-DDSM using that same feature. Additionally, while Sahiner et al. reported an A_Z of 0.91, we find an A_Z of only 0.75 on CBIS-DDSM using our re-implementation of this method. There exist several possible explanations for these discrepancies. First, the technique used to provide mass lesion segmentations for DDSM was not the same as that originally used in any of the segmentation-dependent algorithms, which could affect their efficacy. We propose that further investigating this sensitivity would be a productive direction for future work. Second, as described in the Methods section, the literature does not always describe existing methods in sufficient detail to ensure an exact re-implementation, and the original code is rarely available, meaning that there likely exist differences in implementation between our study and the original work. Third, segmentation-dependent techniques in the literature are often tuned on small datasets, such as 95 images from 68 patients for Huo et al.⁷ and 168 mammograms from 72 patients for Sahiner et al.⁶ While we do tune salient parameters for these methods as described in Methods, other preprocessing choices made in the literature (e.g. neighborhood size for Huo et al.) could affect these models’ ability to transfer to new datasets. These results reinforce the importance of performing algorithm assessment on large public datasets and exerting consistent effort to publicly release new datasets for evaluation as acquisition hardware and software change.

Our study has several important limitations, some of which have been previously mentioned. First, the comparison we have performed between our CBIS-DDSM results and those reported in the literature is imperfect, as we were not able to acquire the complete code for every method. This being said, we use standard implementations of CNN and BoVW methods,^10,11 use code provided by Sahiner for LDA-RBST,⁶ and had two separate researchers implement NN-HDF to ensure repeatability; code for each method can be made available upon request. Furthermore, our results in Figure 4 indicate that our implementation for isolating the margin and periphery – two key parameters of NN-HDF – yields appropriate results (cf. the original paper⁷). Another limitation of our work is that we utilize the segmentations provided by CBIS-DDSM because code for methods used in Sahiner et al. and Huo et al. was unavailable; while we believe this to be a reasonable approach, the difference in segmentation methods could affect the performance of these two techniques. A final limitation of our study is that the DDSM data set is itself an old collection comprising scanned film mammography. Modern mammography is digital, and the results of the methods described in this paper could be different if we used a digital mammography collection. Note that this caveat is not confined to segmentation-dependent methods, as segmentation-free deep learning methods in particular carry the potential to focus on features that are semantically nonsensical as a result of data-driven feature learning, and rigorous evaluation procedures should be utilized to ensure that clinically reasonable features and spatial regions are being utilized. Given the scarcity of public, freely available collections of digital mammography images, and because many prior works have used the DDSM for evaluation, we have chosen to use the CBIS-DDSM collection as the basis of the present work. In the future, it would be helpful to evaluate all CADx methods on digital mammography data sets should they become available, and to remain keenly attuned to the potential for confounding variables such as image quality, latent subsets in the data, and label inconsistency to result in flawed assessments of classifier performance.

CONCLUSIONS

In this work, we use the public CBIS-DDSM dataset to compare the performance of multiple segmentation-free and segmentation-dependent CADx algorithms using a common evaluation standard. We find that segmentation-free representation learning techniques such as BoVW and CNN are able to equal or outperform re-implementations of segmentation-dependent CADx algorithms on CBIS-DDSM. If verified on larger populations, the use of segmentation-free techniques could increase the positive impact of CADx systems on clinical workflows by minimizing the amount of clinician time and precision required to utilize them effectively. We also observe that segmentation-dependent CADx algorithms do not perform as well on CBIS-DDSM as on the original evaluation datasets in the literature, implying that some combination of differences in segmentation approach, variations in implementation, or an underlying lack of generalizability are affecting algorithm performance. It is our hope that this work provides motivation for further study of different mass classification algorithms using public datasets, which would greatly benefit both clinical and scientific communities.

Highlights.

• Segmentation-based and segmentation-free breast mass analysis approaches are compared

• Comparison on common, public datasets is crucial for comparative algorithm analysis

• Modern representation learning techniques often outperform manual feature engineering

• More public evaluation data -- e.g. for digital mammograms -- would be beneficial