A decision support system for mammography reports interpretation

Abstract

Purpose

Mammography plays a key role in the diagnosis of breast cancer; however, decision-making based on mammography reports is still challenging. This paper aims to addresses the challenges regarding decision-making based on mammography reports and propose a Clinical Decision Support System (CDSS) using data mining methods to help clinicians to interpret mammography reports.

Methods

For this purpose, 2441 mammography reports were collected from Imam Khomeini Hospital from March 21, 2018, to March 20, 2019. In the first step, these mammography reports are analyzed and program code is developed to transform the reports into a dataset. Then, the weight of every feature of the dataset is calculated. Random Forest, Naïve Bayes, K-nearest neighbor (K-NN), Deep Learning classifiers are applied to the dataset to build a model capable of predicting the need for referral to biopsy. Afterward, the models are evaluated using cross-validation with measuring Area Under Curve (AUC), accuracy, sensitivity, specificity indices.

Results

The mammography type (diagnostic or screening), mass and calcification features mentioned in the reports are the most important features for decision-making. Results reveal that the K-NN model is the most accurate and specific classifier with the accuracy and specificity values of 84.06% and 84.72% respectively. The Random Forest classifier has the best sensitivity and AUC with the sensitivity and AUC values of 87.74% and 0.905 respectively.

Conclusions

Accordingly, data mining approaches are proved to be a helpful tool to make the final decision as to whether patients should be referred to biopsy or not based on mammography reports. The developed CDSS may also be helpful especially for less experienced radiologists.

Introduction

Breast cancer is the most prevalent cancer among women in the majority of countries all over the world [1]. It is also highly rated as the leading cause of cancer death [2]. Therefore, early detection strategies are of the utmost importance to reduce the cancer burden. One of these strategies is non-invasive imaging techniques such as mammography [3].

Currently, mammography is the most effective method for the detection of early-stage breast tumors [4]. Radiologists provide a narrative report based on their observations from mammography breast images. This free-text report includes information about breast composition, calcification, mass, asymmetry, and other findings. American College of Radiology developed a widely accepted reporting system entitled Breast Imaging-Reporting and Data System (BI-RADS) [5] as a scoring tool to assess the risk of breast cancer. BI-RADS presents standard terminology, report structures, and classification abnormalities; thus, it enables health providers to effectively communicate with each other. Based on the mammography findings, BI-RADS suggests a category that ranges from 0–6, namely BI-RADS score. Therefore, mammography reports usually include one or two categories for the assessment of findings. According to the BI-RADS score, a decision about referring the case for further investigation or biopsy will be made.

Despite the simplicity of using BI-RADS, the number of patients referred to biopsy – which poses serious physical and psychological threats to the human body – is unreasonably high [4]. This is due to a great deal of uncertainty in the interpretation of mammography results [6] that might cause overdiagnosis [7], one of the most important problems in breast cancer screening. In [8] the authors reported that about 25% of detected breast cancer cases through mammography are overdiagnosed. In other words, if these women were not screened, they would live with no problem. Also, these interpretations could be various due to the radiologists’ experiences [9]. For reducing these problems, utilizing computer systems could be helpful.

Clinical Decision Support Systems (CDSS) is one of the computer systems that are beneficial for experts to make decisions more accurately [10]. These systems could employ decision rules [11] or data mining techniques to create predictive models based on previous data to diminish human errors [12]. Data Mining represents the process of extracting knowledge from data to discover hidden relationships [13]. One of the most applicable areas of data mining in the healthcare industry. In healthcare, these systems could be used for decision-making regarding diagnosis and treatment [14–16], management [17] and detecting fraud in insurance [18]. Some studies applied the CDSS for breast cancer detection [19, 20]. Some of these systems [21, 22] utilized data mining techniques to make models based on datasets to predict breast cancer. Most of these datasets are publicly available. It seems that there is a need to evaluate these techniques on other datasets rather than public ones.

In this study, a CDSS using data mining approaches is proposed to decrease the uncertainty in the interpretation of mammography reports. The dataset is extracted from obtained mammography free-text reports of a referral hospital in Iran. The CDSS can recommend whether women should be referred for breast biopsy for further examinations or not.

Methods

This study is conducted in the Imam Khomeini Hospital Complex, the largest educational-medical hospital in Tehran, Iran and a referral hospital for breast cancer in the country.

The overall steps of performing the current project are presented in Fig. 1.

Fig. 1.

Overall steps of performing current research

Analyzing reports

In the first step, 2441 mammography reports belonging to the women who had mammography during a year from March 21, 2018, to March 20, 2019, at Imam Khomeini Hospital Complex, were considered. The reports were extracted from the internal reporting system website of the hospital. The reports were recorded in a free-text format. An example of these reports is as follows:

A 69 years old woman without family history and chief complaint

This is screening mammography and previous mammography is not available. Scattered areas of glandular density are noted in both breasts (breast composition b). There is no evidence of suspicious mass, microcalcification or any definite sign of malignancy in the breasts. Intramammary lymph node is seen at UOQ of left breast Accessory breast is noted at right axillary. Asymmetry is seen in left inner part. US exam is helpful in this kind of breast tissue.

BIRADS:2

Benign finding(s).

Annual screening mammogram is recommended.

Creating dataset

In the second step, based on the reports and reporting items for BI-RADS [23], a dataset is created with 14 features shown in Table 1.

Table 1.

Features of the mammography reports dataset

Feature name	Feature short name	Value description
Age	age	< 100
Mammography type	mtype	1 = screening 2 = diagnostic
Breast cancer history	hist	0 = no (default value) 1 = yes
Sign	sign	0 = without chief complaint 1 = mass palpation or mass sensation 2 = pain or itching 3 = mass palpation or mass sensation or stiffness 4 = bloody secretion or nipple discharge 5 = at least 2 items 6 = at least 3 items
Breast composition	comp	1 = a 2 = b 3 = c 4 = d
Shape (of mass)	shape	0 = none (default) 1 = round 2 = oval 3 = irregular
Margin (of mass)	margin	0 = none (default value) 1 = circumscribed or well-defined 2 = obscured or partially obscured 3 = micro lobulated 4 = Indistinct or ill-defined 5 = spiculate
Density (of mass)	dens	0 = none (default value) 1 = low or fat-containing 2 = equal 3 = high
Asymmetries	asym	0 = none (default value) 1 = asymmetry 2 = global asymmetry 3 = focal asymmetry 4 = developing asymmetry
Morphology (of calcifications)	morph	0 = none (default value) 1 = skin, vascular, coarse, large rod-like, round or punctate (< 1 mm), rim, dystrophic, milk of calcium, suture 2 = amorphous, coarse heterogeneous, fine pleomorphic, fine linear or fine-linear branching
Distribution (of calcifications)	dist	0 = none (default value) 1 = diffuse or scattered 2 = regional 3 = clustered or grouped 4 = segmental 5 = linear
Associated features	af	0 = none (default value) 1 = skin retraction, nipple retraction, skin thickening, trabecular thickening, axillary adenopathy, architectural distortion, calcifications
Special cases	special	0 = None (Default Value) 1 = intramammary lymph node, a wart on the skin
BI-RADS (referral label)	BI-RADS (REFERRAL)	1, 2, 3 = no referral for biopsy 4a, 4b, 4c, 5, 6 = referral to biopsy

The transformation of the reports to the dataset with the features (mentioned in Table 1) has been done by a program written in Visual Studio 2019 C# [24]. This program detects the features’ values by matching the different expressions. The matching rules are created based on expressions seen in the reports.

Weight of factors analysis

In the third step, the weight of each factor is calculated based on Information Gain [25]; then, they are normalized and presented using a bar chart. The information gain metric is expected to determine the more informative features. It is helpful to build predictive models depending on the weight of features to reduce entropy.

Building and evaluating models

In the fourth step, to build a predictive model, data preprocessing is required. Then the predictive models are built using three classification methods. Finally, the evaluation of these models with well-known metrics is performed. This step is taken using RapidMiner Studio 9.3 software [26].

Data preprocessing

The records missing age or BI-RADS scores are removed. These records indicate that the report file is incomplete. A target feature named REFERRAL as a label is generated based on the BI-RADS score, which is indicative of the need for referral for biopsy. The target feature is set to 0 if BI-RADS is less than 4; otherwise, if BI-RADS is greater than 3, the target value is set to 1. Also, records with a BI-RADS score of 0 indicating incomplete assessment are removed. Due to data imbalance between two classes, the Synthetic Minority Over-sampling Technique (SMOTE) [27] is applied to the training dataset. In real-world problems, mostly, the proportion of classes in a dataset is not the same, thus, it has a negative effect on classifier performance. SMOTE solves this problem by synthesizing new instances of the minority class.

Creating models

A common strategy in developing a predictive model is classification. In classification learning methods, a given dataset with a predefined label is utilized to create a classifier [28]. To support the decision in our problem, we employ Random forest [29], Naïve Bayes [30] and K-nearest neighbor (K-NN) [31] as classifiers. The K-NN and the Naïve Bayes classifications are usually selected in basic studies to achieve some induction about data relationships. Besides, they are relatively fast and simple. The random forest classification is more powerful due to combining multiple decision tree classifiers and voting among their results. The definition of these classifiers is presented as follows:

Random Forest: Random forest classification comprises various decision tree classifiers based on the assumption that combining multiple classifiers improves accuracy [32]. It creates several decision tree classifiers with different parameters. finally, to determine the label for each new instance it votes among all acquired results from each decision tree. The combination and voting system make it a powerful method [33].

Naïve Bayes

Naïve Bayes classification is a probabilistic approach based on Bayes theory. It assumes all features are independent [34]. While, in fact, many datasets especially medical datasets do not support it. This assumption makes it a simple method with acceptable speed and accuracy; however, the performance becomes lower in cases with dependency among features [28]. This classification can be used as a primary method in studies due to its fast induction [34].

K-NN

K-nearest neighbor is another simple and basic classification that is broadly applied in studies. The most important parameter in this classifier is a distance metric that is determined relevant to a problem [35]. This model classifies each new instance with calculating the most frequent class among its k-nearest neighbors in the dataset. This algorithm is also simple and fast but noisy data could reduce its performance [28].

Deep learning

Deep learning classifiers automatically extract features of raw data in large datasets with multiple computational layers that represent features in different abstract levels [36]. This classifier had shown great performance in various areas in processing multimedia and unstructured text data.

Evaluating models

The test dataset instances categorized into four categories shown in a matrix called the confusion matrix (Table 2). The performance of classification algorithms is evaluated with the well-known indices like accuracy (Eq. 1), sensitivity (Eq. 2), specificity (Eq. 3) that are calculated using the instances frequency in each category. Besides, The Area Under Curve (AUC) of a receiver operating characteristic (ROC) is evaluated to measure the performance of the classifiers.

$$Accuracy=\frac{TP+TN}{TP+FP+FN+TN}$$ 1

$$Sensitivity=\frac{\mathit{TP}}{TP+FN}$$ 2

$$Specificity=\frac{\mathit{TN}}{FP+TN}$$ 3

Table 2.

Confusion matrix of a binominal classifier

	Actual positive	Actual negative
Predictive positive	TP	FP
Predictive negative	FN	TN

Results

1204 records were extracted from the reports and included in the dataset. The dataset contained 1138 records, which were related to the women who had not needed to be referred to biopsy (REFFERRAL = 1), and 66 records, which were for those who had needed to be referred to biopsy for further tests (REFFERRAL = 0).

The weight of the features based on their importance is shown in Fig. 2.

Fig. 2.

The weight of the features for detecting the need for referral to biopsy

The comparison between the models is presented in Table 3. The best value for each index is shown in bold text. The K-NN model is the most accurate and specific classifier with the values of 84.06% and 84.72% respectively. The Random Forest classifier has the best sensitivity and AUC with the values of 87.74%, 0.905 respectively.

Table 3.

Comparison between classifiers for detecting the need for referral to biopsy

Model	Accuracy (S.D%)	Sensitivity (S.D%)	Specificity (S.D%)	AUC (S.D)	Execution time(s)
Random Forest	82.06% (5.12)	87.74% (14.24)	81.72% (5.40)	0.905 (0.084)	14
Naïve Bayes	80.65% (3.62)	77.68% (19.10)	80.84% (3.01)	0.867 (0.097)	1
K-NN	84.06% (3.32)	72.98% (18.25)	84.72% (3.91)	0.729 (0.188)	1
Deep Learning	77.16% (6.92)	76.55% (17.99%)	77.26% (8.05)	0.859 (0.098)	23

Discussion

In this study, classifier models have been proposed for helping clinicians to decide about referring the woman to biopsy or not. Although the K-NN model has been shown to have the best accuracy (84.06%), it seems more sensitive models such as Random Forest—with the sensitivity of 87.74% in this case—might be recommended to avoid misdiagnosing breast cancer. The results revealed the mammography type (diagnostic or screening), mass and calcification features mentioned in the reports were the most important features for decision making about referring to biopsy. Other studies [37] have also mentioned these important factors. Also, Ferreira et al. proposed a Support Vector Machine (SVM) classifier with an accuracy of 85.6% to predict malignancy mainly based on the mass and density properties [38]. The deep learning classifier proposed in this paper showed not such a high performance in comparison with other classifiers. One justification for this issue could be explained as the labeled dataset for training accurate deep learning classifiers should include enormous samples [39]. Since our dataset is relatively small, the deep learning classifiers didn’t perform well. In addition, the deep learning classifier’s power is feature extraction from raw data.

Several studies reported a higher accuracy for predicting breast cancer, this could be due to the difference in chosen datasets. As mentioned earlier, they mostly used datasets that are publicly available. In this study, the dataset was obtained by converting free-text mammography reports to dataset format. Chaurasia and Pal [40] used Naïve Bayes, RBF Network, and Decision Tree techniques to predict breast cancer on Wisconsin dataset provided by the University of California Irvine machine learning repository [41]. They achieved the best accuracy of %97.36 related to the Naïve Bayes classifier. Asri et al. [42] used Decision Tree, Naïve Bayes, KNN, and SVM techniques to predict breast cancer risk, they achieved an accuracy of %97.13 related to SVM classifier. Another study [43] classified breast tissue based on mammography reports on public datasets have reached to the accuracy of > 99%.

Diz et al. [44] proposed a CAD for breast cancer diagnosis. the best accuracy reported for the proposed system was 89.3%. the researchers used naive Bayes, Random Forest, and SVM classifiers. Another study by Kaushik and Kaur [45] utilized data mining techniques for the prediction of breast tissue biopsy results. The researchers proposed a model with an accuracy of 83.5% and a ROC of 0.907.

Chaudhuri et al. [46] in their study predicted the recurrence of breast cancer with an accuracy of 68.5% by using discriminant analysis and decision tree techniques. In addition to modeling the prediction of breast cancer with datasets and texts, other data types such as gene expression data are used to classify breast cancer [47]. Also, there is a trend to use deep learning techniques for breast cancer prediction through histopathologic images [48] and mammography images [49].

The mammography reports in this study were formatted in a free-text format. Many Studies used natural language processing (NLP) and machine learning techniques to extract data from mammography [50–54] and ultrasound mammography reports [55]. A review is conducted on text mining approaches for cancer-related information [56]. This review showed that many studies use symbolic methods based on dictionary lookup and information extraction with the use of pattern matching as the current study.

Banerjee et al. [57] used NLP techniques for automatic inference of BIRADS from mammography reports. They achieved an average precision of 91% for their proposed method. Another study by Bulu et al. [58] proposed terminology of the radiology domain entitled New RadLex based on NLP techniques. The proposed model’s overall accuracy was reported to be 80%.

It is recommended to standardize a structure for reporting to eliminate the possibility of missing information in the free-text format and to make processing easier. For example, missing a feature in the report could be confusing since the absence might be interpreted either as a missing value or an unimportant value from the operator’s view.

This standard might be formed by defining a semi-structural format based on features that have the most effect on the decision-making process. These features could be obtained from the models presented in this and similar papers.

The results revealed that data mining techniques could be used for breast cancer mammography reports interpretation. Further studies might focus on automated CDSS with the input of mammography images and learn from biopsy results as feedback. Moreover, the dataset sample size might be extended to contain more samples especially positive samples to achieve more reliable models. In the case of adequate dataset size, the classifiers’ performance might exceed radiologists’ performance in cancer detection [59]. Besides, the use of ensemble methods might improve the classifier performance [60].

Limitations

The limitations of this study were as follows: The dataset of this project includes mammography reports of women under the supervision of just one clinician (N. A.). Therefore, the dataset size is limited and results might be biased. Also, some reports were written in Persian and, therefore, we excluded them from the analysis.

Conclusions

Overall, we have developed and evaluated a CDSS based on data mining methods for interpreting mammography reports. The input of this system is free-text mammography reports. Afterward, using presented predictive models, the system could suggest the need for biopsy referral as an output. The proposed system could be helpful especially for lower-experienced radiologists as an aid system. Thus, utilizing these CDSS systems could help clinicians to save time in the diagnosis of regular cases and spend more time for patients with more severe problems. Also, the mentioned systems could reduce the uncertainty of diagnosis and unnecessary further examinations—alleviating the negative effects such as overdiagnosis, cost burden, and patient worries.

Compliance with ethical standards

Conflict of interest

The authors report no conflict of interest.