External validation of a publicly available computer assisted diagnostic tool for mammographic mass lesions with two high prevalence research datasets

Abstract

Purpose:

Lesions detected at mammography are described with a highly standardized terminology: the breast imaging-reporting and data system (BI-RADS) lexicon. Up to now, no validated semantic computer assisted classification algorithm exists to interactively link combinations of morphological descriptors from the lexicon to a probabilistic risk estimate of malignancy. The authors therefore aim at the external validation of the mammographic mass diagnosis (MMassDx) algorithm. A classification algorithm like MMassDx must perform well in a variety of clinical circumstances and in datasets that were not used to generate the algorithm in order to ultimately become accepted in clinical routine.

Methods:

The MMassDx algorithm uses a naïve Bayes network and calculates post-test probabilities of malignancy based on two distinct sets of variables, (a) BI-RADS descriptors and age (“descriptor model”) and (b) BI-RADS descriptors, age, and BI-RADS assessment categories (“inclusive model”). The authors evaluate both the MMassDx (descriptor) and MMassDx (inclusive) models using two large publicly available datasets of mammographic mass lesions: the digital database for screening mammography (DDSM) dataset, which contains two subsets from the same examinations—a medio–lateral oblique (MLO) view and cranio–caudal (CC) view dataset—and the mammographic mass (MM) dataset. The DDSM contains 1220 mass lesions and the MM dataset contains 961 mass lesions. The authors evaluate discriminative performance using area under the receiver-operating-characteristic curve (AUC) and compare this to the BI-RADS assessment categories alone (i.e., the clinical performance) using the DeLong method. The authors also evaluate whether assigned probabilistic risk estimates reflect the lesions’ true risk of malignancy using calibration curves.

Results:

The authors demonstrate that the MMassDx algorithms show good discriminatory performance. AUC for the MMassDx (descriptor) model in the DDSM data is 0.876/0.895 (MLO/CC view) and AUC for the MMassDx (inclusive) model in the DDSM data is 0.891/0.900 (MLO/CC view). AUC for the MMassDx (descriptor) model in the MM data is 0.862 and AUC for the MMassDx (inclusive) model in the MM data is 0.900. In all scenarios, MMassDx performs significantly better than clinical performance, P < 0.05 each. The authors furthermore demonstrate that the MMassDx algorithm systematically underestimates the risk of malignancy in the DDSM and MM datasets, especially when low probabilities of malignancy are assigned.

Conclusions:

The authors’ results reveal that the MMassDx algorithms have good discriminatory performance but less accurate calibration when tested on two independent validation datasets. Improvement in calibration and testing in a prospective clinical population will be important steps in the pursuit of translation of these algorithms to the clinic.

1. INTRODUCTION

Computer assisted diagnostic (CADx) algorithms for mammography interpretation have received substantial attention in the past. Generally, these algorithms can be divided into those that employ imaging features that are automatically extracted from the mammographies and those that employ descriptors that are assessed by radiologists; the latter can be referred to with the term semantic CADx. Elter and Horsch provide a comprehensive review of available mammography CADx algorithms.¹

Many promising semantic CADx algorithms with good to very good diagnostic performance have been proposed; the statistical techniques employed include artificial neural networks,^2,3 Bayesian networks,^4–6 decision trees,⁷ and logistic regression.^8,9 However, before it is acceptable to actually apply a semantic CADx algorithm in clinical routine, an external validation of the algorithm’s diagnostic performance is mandatory.^10,11 External validation is defined as evaluation of the performance of a classification algorithm on data that were not used to generate the algorithm.¹¹

Recently, a publicly available semantic CADx algorithm for mammographic masses (MMs) was published by Benndorf and colleagues¹² and we refer to the algorithm with mammographic mass diagnosis (“MMassDx”). The MMassDx algorithm takes into account the qualitative breast imaging-reporting and data system (BI-RADS) mass lesion descriptors, patient age, and optionally BI-RADS assessment categories. The MMassDx algorithm can be accessed at www.ebm-radiology.com/nbmm/index.html; the algorithm implements a naïve Bayes network to derive diagnostic decisions. The MMassDx algorithm showed good diagnostic performance in the development study; the area under the curve was 0.935 with BI-RADS assessment categories included and area under the curve was 0.876 without assessment categories included. A validation with different cases from the same practice was performed by the authors in their study.

External validation should, to demonstrate robustness of the MMassDx algorithm, also be performed on patients from different practices.¹¹ Second, it is worthwhile to investigate how the MMassDx algorithm performs in a variety of clinical circumstances to identify potential areas of use and potential limitations of the approach. We therefore aim at the external validation of the MMassDx algorithm proposed by Benndorf and colleagues.¹² We process two large, publicly available datasets of mammographic mass lesions^7,13–15 and apply the MMassDx algorithm to them. The two datasets come from different practices and are thus independent from the data used to train the original MMassDx algorithm. Confirmation of high performance on these two datasets (i.e., external validation) would be a step toward demonstrating potential clinical efficacy of the algorithm.

2. MATERIALS AND METHODS

2.A. The BI-RADS lexicon

The BI-RADS offers a standardized terminology for the description of lesions detected on x-ray mammography.¹⁶ BI-RADS was established in 1992 to facilitate an unequivocal communication between radiologists, even at different practicing sites, and referring clinicians.¹⁷ The BI-RADS lexicon has been refined over the last 30 yr, and currently, it is in its fifth edition. In 1999, the Mammography Quality Standards Act (MQSA) made it mandatory by law that every mammographic report in the United States includes the language of the BI-RADS for final assessment.¹⁸

In case of mass lesions, the following descriptors are assessed according to BI-RADS: shape (round, oval, lobulated, or irregular), margin (circumscribed, microlobulated, obscured, ill-defined, or spiculated), and density (fat-dense, low, isodense, or high). Furthermore, each lesion is assigned an assessment category ranging from 0 to 6. This assessment category can be regarded as a probabilistic stratification of the risk of malignancy of the described lesion. BI-RADS 0 denotes a lesion for which further work-up is required and BI-RADS 1 denotes a mammography without a detected lesion. BI-RADS 2 assigns a risk of malignancy of 0%, BI-RADS 3 of <2%, BI-RADS 4 of 2%–95%, and BI-RADS 5 assigns a risk of >95% (almost certainly malignant). BI-RADS 6 denotes lesions which are histopathologically proven malignant. For pictorial examples of the BI-RADS descriptors, refer to the paper by Balleyguier and colleagues.¹⁹

Up to now, no guidelines exist to link certain combinations of BI-RADS descriptors to certain BI-RADS assessment categories. This is why multivariate prediction models, like the MMassDx algorithm investigated herein, could help to quantify the risk and thus help to more accurately make further management decisions.

2.B. The classifier at ebm-radiology.com

The development and testing of the MMassDx algorithm are detailed in the paper by Benndorf and colleagues.¹² The MMassDx algorithm can be accessed at www.ebm-radiology.com/nbmm/index.html. At this homepage, an interface is presented that allows the radiologist to enter a combination of observed BI-RADS mass lesion descriptors, patient age, and optionally a BI-RADS assessment category. The following notation will be used throughout the paper:

• —The algorithm that takes into account age and BI-RADS descriptors is referred to as “descriptor model.”
• —The algorithm that takes into account age, BI-RADS descriptors, and BI-RADS assessment categories is referred to as “inclusive model.”
• —The diagnostic performance of the BI-RADS assessment categories alone is referred to as “clinical performance.”

The MMassDx algorithm is based upon a naïve Bayes network. For each mammographic mass lesion, the network calculates a post-test probability of malignancy based on the values of the entered predictive variables. The post-test probability is calculated with Bayes’ theorem and assumed conditional independence of the single predictive variables. The MMassDx algorithm automatically bins the calculated post-test probability into a risk group analogously to the BI-RADS assessment categories. For this step, the method proposed by Zadrozny and Elkan is used.²⁰ Figure 1 illustrates the binning technique used. We highlight that the calibration technique is applied to overcome a shortcoming of the naïve Bayes in assignment of the actual probabilities of malignancy.²¹ For detailed information about Bayesian networks and especially naïve Bayes networks, refer to Refs. ²² and ²³.

FIG. 1.

Binning method for naïve Bayes classifiers by Zadrozny and Elkan (Ref. ²⁰). For example, 15 instances are regarded with classes Y (=yes) and N (=no). Step (1): the naïve Bayes classifier calculates a probability of malignancy for each instance. Step (2): the instances are ordered according to the calculated probability, starting with the lowest probability. Step (3): three equally sized bins are formed, with 5 members each. The probability of malignancy in each bin is the number of Y-instances/all instances in this bin. New instances classified by the naïve Bayes classifier can be sorted into one of the bins, and the probability of malignancy assigned is the probability from step 3.

2.C. The external validation datasets

We first process the data from the digital database for screening mammography (DDSM) to be used as a validation dataset for the MMassDx algorithm.¹⁵ The DDSM is publicly available at http://marathon.csee.usf.edu/Mammography/Database.html. It contains information of 1220 mammographic mass lesions, described in either medio–lateral oblique (MLO) or cranio–caudal (CC) view, or described in both views. For each lesion, morphological BI-RADS descriptors (shape and margin) and patient age are given. Information regarding the DDSM can be found in Ref. 15, and we process the database as provided in Ref. 14. The data there are provided as a tab-delimited text file, the content is identical to the source homepage, and processing was performed with R 2.15.3. The dataset allows for different descriptors in the MLO and CC views. We thus generate a dataset for each view, referred to by “DDSM MLO” and “DDSM CC.”

We second process the mammographic mass dataset (referred to as “MM”) provided by Elter and colleagues⁷ at the University of California, Irvine, Machine Learning Repository,¹³ http://archive.ics.uci.edu/ml/datasets/Mammographic+Mass, which contains information of 961 mammographic mass lesions. For each lesion, morphological BI-RADS descriptors and patient age are given. Table I gives the characteristics of the two validation datasets and compares them with the dataset that was used in the original publication by Benndorf and colleagues.

TABLE I.

Comparison of the original dataset for the classifier at www.ebm-radiology.com/nbmm/index.html with the datasets for the current external validation.

	Original dataset, Benndorf et al.	Mammographic mass (MM) dataset, Elter et al. (Ref. 7)	DDSM MLO,a Heath et al. (Ref. 15)	DDSM CC,a Heath et al. (Ref. 15)
Data acquired between	2005–2011	2003–2006	1988–1999	1988–1999
n proven mass lesions	9629	961	1212	1159
Contains missing values	Yes	Yes	Yes	Yes
n mass lesions with complete descriptors	2453b (25.5%)	821 (85.4%)	1108 (91.4%)	1054 (91.4%)
n malignant (%)	313 (12.8%)	396 (48.2%)	562 (50.7%)	519 (49.2%)
Multiple descriptors allowedc	Yes	No	Yes	Yes
n with multiple descriptors for mass shape	0	0	6	7
n with multiple descriptors for mass margin	95	0	78	75
n with multiple descriptors for mass density	0	0	Not applicable, density not assessed	Not applicable, density not assessed

^aIn the DDSM database, separate information is given for MLO view and CC view mammographies. We process both views as separate datasets, since descriptors are allowed to differ between the two views for a given lesion.

^bFor classifier construction, these data are split into training data and validation data. Pretest probability in the training data was 10.8%.

^cFor cases with multiple descriptors, the descriptor most suspicious for malignancy is chosen for application of the MMassDx algorithm.

2.D. Statistical analysis

We apply the MMassDx algorithm to the DDSM MLO, DDSM CC, and MM datasets. For each lesion, the MMassDx algorithm reports (a) a post-test probability and (b) a diagnostic risk group as explained in Ref. 12. We apply both the descriptor model and the inclusive model to the datasets. For probability calculations, the pretest probability was set to 10.8%, i.e., the pretest probability in the training data in the work by Benndorf and colleagues.¹²

2.D.1. Analysis of classifier discrimination

We perform receiver-operating-characteristic (ROC) analysis with the calculated post-test probabilities and the obtained diagnostic risk groups. We employ the area under the ROC curve (AUC) as a measure for discriminative performance.²⁴ We compare the AUC of the ROC curves from the MMassDx algorithm with the AUC of the BI-RADS assessment categories (i.e., the clinical performance) in the respective validation dataset using the method developed by DeLong.²⁵ We consider a P-value <0.05 to denote a statistically significant difference.

Figure 2 provides a diagram of the analyses performed with numbers given to the consecutive steps. These numbers will be used throughout the remaining paper for referral to the scenarios investigated. For example, scenario 2c is the application of the MMassDx (inclusive) algorithm to the DDSM CC dataset. Scenario 2d stratifies the resulting post-test probabilities from 2c into the diagnostic groups as described in Ref. 12, i.e., the results are calibrated.

FIG. 2.

External validation of the MMassDx algorithm at www.ebm-radiology.com/nbmm/index.html. DDSM: digital database for screening mammography (Ref. ¹⁵), MM: mammographic mass dataset (Ref. ⁷). (A) MMassDx algorithm without BI-RADS assessment categories used. (B) MMassDx algorithm with BI-RADS assessment categories used. In each numbered scenario, the MMassDx algorithm (descriptor, inclusive) is applied to the respective dataset. First, post-test probabilities of malignancy are derived (e.g., scenario 2c), which are then calibrated into diagnostic groups (scenario 2d). ROC analyses are performed for all scenarios. The numbers 1a–2f will be used throughout the remaining paper.

Discriminative performance of the BI-RADS assessment categories alone, i.e., the clinical performance, also was assessed with ROC analysis. As detailed in Sec. 2.A of the paper, the BI-RADS assessment categories can be regarded as a risk stratification tool themselves. For ROC analysis, the assessment categories were considered an ordinal variable with sequence 2, 3, 0, 4, and 5.

2.D.2. Analysis of classifier calibration

The MMassDx algorithm at www.ebm-radiology.com/nbmm/index.html returns a probabilistic estimate how likely the lesion is malignant. Since the management of mammographic mass lesions depends on this risk (i.e., benign lesions will undergo routine screening follow-up; probably, benign lesions will undergo short term follow-up and suspicious lesions will undergo tissue sampling), we are also interested in how accurately the risk estimates of the classifier are compared to the actual risk of the lesions considered.

For this purpose, we employ the technique of calibration plots.^10,26 For scenarios 1b, 1d, 1f, 2b, 2d, and 2f in Fig. 2, we plot on the x-axis the predicted probability of the pre-specified bin as defined in Table 3 in Ref. 12, and on the y-axis the percentage of how many lesions in the allocated bin actually proved to be malignant. In an ideally calibrated classifier, the predicted and actual risks should match. If the classifier underestimates the risk of malignancy, the resulting points will be above the identity line. If the classifier overestimates the risk, the resulting points will be below the identity line, compare Fig. 3 for an illustration of this concept.

FIG. 3.

Principle of a calibration plot: on the x-axis, the predicted probabilities for the pre-specified bins derived by classifier calibration (Ref. ²⁰) are given. On the y-axis, the corresponding rate of outcomes among all cases that were assigned a certain bin is plotted. A perfectly calibrated classifier results in points that lie on the identity line. The areas for under- and overestimation of the probability for the outcome are highlighted in the plot.

3. RESULTS

Table II lists the results of the ROC analyses for the different scenarios as defined in Fig. 2. With the exception of scenario 1f (MM dataset, calibrated classifier), all applications of the MMassDx algorithm perform better than the clinical performance alone (P < 0.05 for each comparison). In all investigated scenarios, the MMassDx algorithm shows good discriminative performance with AUC values >0.80.

TABLE II.

Performance of the MMassDx algorithm at www.ebm-radiology.com/nbmm/index.html when applied to the DDSM MLO, DDSM CC, and MM datasets.

Model	External dataset	Analysis according to Fig. 1	MMassDx area under the ROC curve (AUC)	P-value in comparison to clinical performance
Descriptor model	Benndorf 2015a	n.a.	0.876	0.799
	DDSM MLO	1a	0.876b	<0.001
	DDSM MLO	1b	0.838b	<0.01
	DDSM CC	1c	0.895b	<0.001
	DDSM CC	1d	0.856b	<0.01
	MM	1e	0.862b	0.019
	MM	1f	0.842	0.294
Inclusive model	Benndorf 2015a	n.a.	0.935b	<0.001
	DDSM MLO	2a	0.891b	<0.001
	DDSM MLO	2b	0.819b	<0.01
	DDSML CC	2c	0.900b	<0.001
	DDSM CC	2d	0.832b	<0.01
	MM	2e	0.900b	<0.001
	MM	2f	0.868b	0.001

^aClassifier performance from Ref. 12 added for comparison.

^bDenotes statistical significant difference from the performance of the BI-RADS assessment categories alone, i.e., the clinical performance. No statistically significant difference was found in the clinical performance between DDSM MLO, DDSM CC, and MM (P > 0.05).

Figure 4 exhaustively provides the diagnostic performance of the MMassDx algorithm in the datasets analyzed (ROC curves). Again, notation of the investigated scenarios follows Fig. 2. Figure 5 provides the information about classifier calibration in the scenarios where applicable. The calibration points do not follow the identity line, and the MMassDx algorithm systematically underestimates the true probability of malignancy in the diagnostic bins. Multiple points result at the 0% risk of malignancy value (x-axis) because multiple bins with an estimated risk of 0% are defined in Ref. 12. These bins proved to have a higher cancer yield in the validation datasets.

FIG. 4.

Receiver-operating-characteristic (ROC) plots for the classifier at www.ebm-radiology.com/nbmm/index.html when applied to the DDSM MLO, DDSM CC, and MM datasets. The numbers 1a–2f follow the notation given in Fig. 2. In each diagnostic scenario, the ROC curves for the descriptor model, the inclusive model, and the BI-RADS assessment category alone (clinical performance) are given. Binormal fitting is performed for plotting.

FIG. 5.

Calibration plots for the classifier at www.ebm-radiology.com/nbmm/index.html when applied to the DDSM MLO, DDSM CC, and MM datasets. The numbers follow the notation given in Fig. 2. In each plot, the identity line is given, and a perfectly calibrated CADx algorithm would result in points that lie on this line.

To further investigate calibration behavior of the MMassDx algorithm, we provide in Table III a comparison of how many lesions comprised the diagnostic bins in the study by Benndorf and colleagues¹² and the number of lesions in the bins in the DDSM and MM data. Here, we observe that bins with a low probability of malignancy are only marginally populated in comparison to the development study in the DDSM datasets, and in the MM dataset, bins 3–7 are only marginally populated.

TABLE III.

Comparison of how many lesions are assigned to the diagnostic bins in the DDSM MLO, DDSM CC, and MM datasets in comparison to the development study (Ref. ¹²). Results are for the validated descriptor model (i.e., scenarios 1b, 1d, and 1f from Fig. 2). For comparison, the distribution in the validation data from the development study (Ref. ¹²) is given.

Bin	DDSM MLO (scenario 1b)	DDSM CC (scenario 1d)	MM (scenario 1f)	Benndorf (validation data)
1	0 (0%)	0 (0%)	163 (19.9%)	82 (7.0%)
2	40 (3.6%)	32 (3.0%)	106 (12.9%)	95 (8.1%)
3	23 (2.1%)	20 (1.9%)	10 (1.2%)	196 (16.6%)
4	45 (4.1%)	43 (4.1%)	25 (3.0%)	104 (8.8%)
5	79 (7.1%)	74 (7.0%)	25 (3.0%)	143 (12.1%)
6	108 (9.7%)	107 (10.2%)	45 (5.5%)	108 (9.2%)
7	62 (5.6%)	60 (5.7%)	24 (2.9%)	94 (8.0%)
8	124 (11.2%)	116 (11.0%)	99 (12.1%)	105 (8.9%)
9	219 (19.8%)	216 (20.5%)	220 (26.8%)	127 (10.8%)
10	408 (36.8%)	386 (36.6%)	104 (12.7%)	123 (10.5%)

4. DISCUSSION

Diagnostic performance of prediction algorithms has two separate dimensions: first, discrimination measures the ability of the algorithm to differentiate between lesions with different outcomes. Discriminatory performance can be visualized with ROC curves. Second, calibration measures the agreement between predicted and observed risks. Calibration performance can be visualized with calibration plots. It has been demanded that both discrimination and calibration of prediction algorithms should be examined in studies that evaluate prediction algorithms.^11,27

In our study, we demonstrate that the MMassDx algorithm proposed at www.ebm-radiology.com/nbmm/index.html shows good discriminative performance when applied to two research datasets of mammographic mass lesions. The AUC values, ranging between 0.83 and 0.88 for the descriptor model and between 0.83 and 0.90 for the inclusive model, are in accordance with AUC values reported before for semantic mammographic CADx algorithms based on machine learning algorithms. The neural network approach by Baker and colleagues yielded an AUC of 0.89,³ the Bayesian network by Fischer and colleagues yielded an AUC of 0.88,⁵ and the tree-augmented Bayesian network by Burnside and colleagues yielded an AUC of 0.96.⁴ Elter and colleagues reported AUC values of 0.87 and 0.89 for their decision tree model and case-based learning algorithm, respectively.⁷

We second demonstrate that the MMassDx algorithm is not well calibrated when applied to the two validation datasets, especially in lesions for which a low probability of malignancy is calculated, compare with Fig. 5. Thus, even if the MMassDx algorithm performed well discriminationwise, its actual application to the datasets would not have resulted in an adequate management for a substantial proportion of patients.

The reason for the insufficient calibration has to be searched for in the nature of the datasets, we believe. When compared to the dataset, the classifier was developed with, it is obvious that the pretest probability of malignancy in the two validation datasets is much higher (13% versus 50%, compare with Table I). Furthermore, the DDSM and MM datasets do not come with detailed descriptions of how the data were collected—i.e., there is no information whether cases were consecutively observed mass lesions, consecutively histopathologically verified mass lesions, or deliberately sampled such that a case-control ratio of 1:1 obtained. The population examined with a diagnostic test may affect diagnostic (discriminatory) performance of prediction algorithms, a phenomenon which is known as spectrum effect;²⁸ we reason that classifier calibration might likewise be affected.

If we regard the pretest probability of malignancy as a surrogate parameter for the characterization of the study population, it is worthwhile to derive an estimate of the pretest probability in the population the MMassDx algorithm is developed for. As a premise, we take that the algorithm is applied to all mass lesions observed in a given period of time at a given practice. Obviously, the prevalence of disease in the screening population [around 1% (Refs. ²⁹ and ³⁰)] is too low, since in the majority of patients, no abnormality is detected at all at mammography screening [i.e., rating is BI-RADS 1 (Ref. ³¹)]. On the other hand, taking into account only lesions with histopathological verification will overestimate the pretest probability [estimates range between 30% and 50% (Refs. ³² and ³³)], since clearly benign findings will only undergo biopsy in rare occasions. A reliable estimate for the pretest probability among consecutively observed mammographic mass lesions may be derived as follows.

Zonderland and colleagues³⁴ report on a consecutive hospital population of patients referred to mammography (35% diagnostic, 65% screening). The distribution of BI-RADS assessment categories among their study population was 1,542 BI-RADS 1, 935 BI-RADS 2, 154 BI-RADS 3, 74 BI-RADS 4, and 57 BI-RADS 5. An explanation of the BI-RADS categories is given in Subsection 2.A of this paper. BI-RADS 1 ratings do not describe a lesion per definition, and we are thus left with 1220 cases with a described lesion. Of these 1220 cases, 115 had a malignant outcome. Therefore, the pretest probability in the study by Zonderland and colleagues was, given a lesion had been observed, 115/1220 = 9.4%. This value is close to the 10.8% in the development study of the MMassDx algorithm,¹² indicating that the algorithm was developed under sensible clinical premises.

We hypothesize that differences in the study populations, as indicated by the differing pretest probabilities, at least partly explain the insufficient calibration of the MMassDx algorithm. The decision to perform histopathological sampling of a lesion detected at mammography also depends on variables not considered by the MMassDx algorithm, e.g., the presence of dense fibroglandular tissue³⁵ or a genetic risk profile to actually develop breast cancer.³⁶ Since we do not know about case selection in the validation datasets, we cannot exclude the possibility that the presence of such risk factors influenced the case distribution in the validation datasets. A lesion that we would call “probably benign” in an overall low-risk patient might be rendered “suspicious” if we come to know that the patient has an increased lifetime risk of breast cancer >50%. This is one possible explanation of the high cancer yield in the low probability bins in the calibration curves in Fig. 4.

The visualization of classifier calibration in Fig. 4 demonstrates a positive result, though. If the MMassDx algorithm assigns a high probability of malignancy, then the lesion under consideration will likely be malignant. Underestimation of the probability here does not play a clinically significant role: a threshold of 2% of suspicion for malignancy is generally regarded the threshold at which to perform biopsy.³⁷ The assignment of a 40% risk of malignancy thus has the same clinical consequence as the assignment of a risk of 50%, for example.

Semantic CADx algorithms in general depend on the assessment of the employed descriptors by a physician (a radiologist in our case). Although the BI-RADS lexicon offers a highly standardized diagnostic approach,¹⁶ interobserver disagreement may lead to statistical noise introduced into classification aids using these descriptors. Kappa values for the assignment of mass lesion descriptors in mammography reading range between 0.48 (Ref. ³⁸) and 0.60 (Ref. ³⁹) in general, although lower values have been reported as well.⁴⁰ For BI-RADS assessment categories, fair to moderate agreement has been reported⁴¹ with formal training in lexicon usage improving on agreement rates. Both the DDSM and the MM dataset do not come with detailed information about training of the involved readers and an analysis of interobserver disagreement. How the performance of different readers affects semantic CADx algorithms appears as a promising direction for future research.

We have not performed a dedicated analysis of the optimal classification threshold of the Bayesian classifier taking into account expected utility of the classification,⁴² but regarded the entire AUC as parameter for discriminatory performance. The optimal classification threshold is known to be prevalence dependent.^42,43 The definition of actual utilities (that is, costs and benefits for true positive, false negative, false positive, and true negative classifications) and the evaluation of classifier performance in different clinical scenarios (with differing pretest probability), are non-trivial tasks. Taking these issues into account also presents as possible future research topic.

To sum up, from the results of our study, we infer that the MMassDx algorithm might be of clinical use in the future, since it links combinations of BI-RADS descriptors to probabilistic risk estimates and achieves a high discriminatory performance. If the MMassDx algorithm reports a high probability of malignancy, the estimate will be clinically meaningful. However, the MMassDx algorithm shows insufficiencies regarding calibration when it reports a low probability of malignancy. We recommend (and plan) to perform the logical next step of external validation: the application of the MMassDx algorithm to a consecutive clinical cohort of mammographic mass lesions. This experiment will show whether the insufficiencies in calibration are due to spectrum effects or a true insufficiency of the algorithm. Additionally, it should be investigated whether incorporation of further risk factors as predictive variables into the MMassDx algorithm can improve on the discriminatory and calibration performance.