Application of computer-aided diagnosis to predict malignancy in BI-RADS 3 breast lesions

Abstract

Purpose

To evaluate the ability of computer-aided diagnosis (CAD) system (S-Detect) to identify malignancy in ultrasound (US) -detected BI-RADS 3 breast lesions.

Materials and methods

148 patients with 148 breast lesions categorized as BI-RADS 3 were included in the study between January 2021 and September 2022. The malignancy rate, accuracy, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and area under the curve (AUC) were calculated.

Results

In this study, 143 breast lesions were found to be benign, and 5 breast lesions were malignant (malignancy rate, 3.4 %, 95 % confidence interval (CI): 0.5–6.3). The malignancy rate rose significantly to 18.2 % (4/22, 95 % CI: 2.1–34.3) in the high-risk group with a “possibly malignant” CAD result (p = 0.017). With a “possibly benign” CAD result, the malignancy rate decreased to 0.8 % (1/126, 95 % CI: 0–2.2) in the low-risk group (p = 0.297). The AUC, sensitivity, specificity, accuracy, PPV, and NPV of the CAD system in BI-RADS 3 breast lesions were 0.837 (95 % CI: 77.7–89.6), 80.0 % (95 % CI: 73.6–86.4), 87.4 % (95 % CI: 82.0–92.7), 87.2 % (95 % CI: 81.8–92.6), 18.2 % (95 % CI: 2.1–34.3) and 99.2 % (95 % CI: 97.8–100.0), respectively.

Conclusions

CAD system (S-Detect) enables radiologists to distinguish a high-risk group and a low-risk group among US-detected BI-RADS 3 breast lesions, so that patients in the low-risk group can receive follow-up without anxiety, while those in the high-risk group with a significantly increased malignancy rate should actively receive biopsy to avoid delayed diagnosis of breast cancer.

1. Introduction

Breast cancer is the most common form of cancer in women and has been threatening women worldwide [1]. Promoting early detection, followed by timely and appropriate treatment, is key to improving breast cancer survival [2,3]. Ultrasound (US) has been proven to detect breast cancer and has been found to detect additional breast cancers in comparison with conventional mammography [4,5]. The American College of Radiology (ACR) published the second version of the Breast Imaging Reporting and Data System (BI-RADS) lexicon for US in the fifth edition of the BI-RADS atlas in 2013 to standardize reports and make management recommendations [6,7].

The ACR suggests the malignancy rate of BI-RADS 3 is ≤ 2 %, so short-interval follow-up is recommended instead of biopsy [8]. This management strategy is to avoid unnecessary biopsies in a large number of cases, but patient cooperation is required [9]. However, patient compliance with short-interval follow-up recommendation is often not high, with up to 33 % of patients not returning for recommended 6-month follow-up [10]. Solid oval circumscribed masses with parallel orientation, isolated complicated cysts, and clustered microcysts are suggested to be appropriate for BI-RADS 3 category in the second version of BI-RADS lexicon for US [8]. The BI-RADS lexicon also declares that radiologists with personal experience may use category 3 for other lesions with features that they personally consider to have a likelihood of malignancy≤2 % [8]. This may lead to make the evaluation of stability difficult [11]. Previous studies have shown that the malignancy rate of BI-RADS 3 could be higher than 2 %, resulting in a delay in cancer diagnosis for a considerable number of patients, and there is still controversy how best to manage such lesions [[11], [12], [13], [14]]. Immediate biopsy or surgery is recommended for BI-RADS 3 lesions in some institutions because it's considered safer [15]. Therefore, a method to predict malignancy in BI-RADS 3 lesions is of clinical value.

Nowadays, deep learning-based computer-aided diagnosis (CAD) system (S-Detect, Samsung Ultrasound RS80A, Samsung Medison Co. Ltd.) has become commercially available and has been shown to provide objective opinion to assist radiologists in diagnosis of the breast and thyroid lesions [16]. Previous studies have reported that CAD system (S-Detect) could provide decision-making support for radiologists to diagnose breast cancer by improving specificity and reducing unnecessary biopsies in clinical practice [[16], [17], [18]]. However, few studies have discussed the value of CAD system in identifying malignant lesions in BI-RADS 3 category, and we believe that it is possible to further improve the use of this assessment category with CAD assistance [19].

Therefore, the purpose of this study was to evaluate the ability of CAD system (S-Detect) to identify malignancy in US-detected BI-RADS 3 breast lesions.

This study differs from our previous two studies. One of the studies evaluated the usefulness of CAD software on the diagnostic performance of radiologists in the differentiation of benign and malignant breast lesions measuring up to 2.0 cm on US, including BI-RADS 3 breast lesions upgrade and BI-RADS 4A downgrade [20]. Another study compared whether less-experienced radiologists combined with CAD results could achieve the level of experienced radiologists, and explored how to select the planes for CAD analysis [21]. In addition, the present study included data from 9 hospitals, whereas the two previous studies included data from only 8 hospitals.

2. Materials and Methods

2.1. Patient

This prospective multicenter study was approved by the institutional review board of all participating centers (no. M2021304), and written informed consent was obtained from all patients. Patients with US-visible BI-RADS 3 breast lesions were recruited from 9 hospitals in China, who presented with symptom, palpable abnormality, or prior exams abnormality (US, mammography, or MRI), or no complaints. Patients who are pregnant or breastfeeding, with prior biopsy of the same breast lesion, with neoadjuvant chemotherapy or radiotherapy, with maximum diameter of the lesion >5 cm (beyond the probe collection range) were excluded. For patients with multiple breast lesions, only the largest one was included. The pathological findings, obtained by core-needle biopsy or surgical excision, referred to as the gold standard of final diagnosis. Finally, 148 patients with 148 breast lesions categorized as BI-RADS 3 were included in the study between January 2021 and September 2022 (Fig. 1).

Fig. 1.

Flowchart of the study.

2.2. Ultrasound image acquisition and analysis

US examinations were performed using an RS80A US system (Samsung Medison Co., Ltd.) equipped with a 3–12 MHz or 4–18 MHz linear probe by one of 16 radiologists with 1–10 years of experience in breast US from 9 hospitals. These radiologists were mixed body/breast radiologists who either lacked subspecialty training in breast imaging or for whom the number of annual breast ultrasounds accounted for less than 10 % of the total number of annual ultrasounds. The radiologists, who knew the patient's clinical information and previous examination findings (US, mammography or MRI), independently evaluated the lesion and made a diagnosis of the lesion according to the fifth edition of BI-RADS lexicon after the dynamic scanning [8]. Then, CAD analysis (S-Detect) was performed on the gray-scale image in long-axis plane by the same radiologist, who performed the gray-scale ultrasound. After clicking the lesion center on the selected image, the software automatically calculated the contour of the lesion and evaluated its US features in long-axis plane. Manual adjustment could be made when the segmentation inaccurately defined the contour of the lesion. The lesion was finally diagnosed in a dichotomous format, in real time, as “possibly benign” or “possibly malignant” [16,22].

2.3. Statistical analysis

The SPSS v26.0 (IBM Corporation, New York, USA) was used for statistical analysis. Continuous variables were described as mean ± standard deviation or median; categorical variables were presented in the form of frequencies and proportions. The diagnostic value was evaluated by calculating the accuracy, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and area under the curve (AUC). The t-test (or Mann–Whitney U test) was used to evaluate for variability between continuous variables, while the chi-square test (or McNemar test) was used to evaluate for a relationship between categorical variables. A p value＜0.05 was considered statistically significant.

3. Result

3.1. Basic characteristics of patients and lesions

Finally, 148 patients (mean age, 39.9 ± 11.4 years (range, 14–70 years)) with 148 BI-RADS 3 breast lesions were collected in this study. 143 breast lesions were found to be benign, and 5 breast lesions were malignant (malignancy rate, 3.4 %, 95 % confidence interval (CI): 0.5–6.3). The mean diameter of the lesions was 1.7 ± 0.8 cm (range, 0.5–4.7 cm). A summary of the patients and breast lesions’ characteristics is shown in Table 1.

Table 1.

The characteristics of patients and breast lesions.

Characteristics	All lesions	Benign	Malignant
Mean age ± SD (range), years	39.9 ± 11.4（14–70）	41.8 ± 10.9	39.8 ± 13.9
Mean lesion size ± SD (range), cm	1.7 ± 0.8（0.5–4.7）	1.6 ± 0.8	2.0 ± 1.0
Pathological results
Fibroadenoma		76
Proliferative disease		56
Inflammatory lesions		6
Intraductal papilloma		4
Phyllodes tumor		1
Ductal carcinoma in situ			3
Invasive ductal carcinoma			1
Lymphoma			1

3.2. Stratification of the malignancy risk in BI-RADS 3 lesions with computer-aided diagnosis system

The malignancy rate rose significantly to 18.2 % (4/22, 95 % CI: 2.1–34.3) in the high-risk group with a “possibly malignant” CAD result (p = 0.017). With a “possibly benign” CAD result, the malignancy rate decreased to 0.8 % (1/126, 95 % CI: 0–2.2) in the low-risk group (p = 0.297).

The AUC of the CAD system was 0.837 (95 % CI: 77.7–89.6). The sensitivity of the CAD system was 80.0 % (95 % CI: 73.6–86.4), meaning that the CAD system could identify four out of five malignancies in BI-RADS 3 lesion group. The specificity was 87.4 % (95 % CI: 82.0–92.7), and accuracy was 87.2 % (95 % CI: 81.8–92.6). The positive and negative predictive values were 18.2 % (95 % CI: 2.1–34.3) and 99.2 % (95 % CI: 97.8–100.0), respectively (Table 2).

Table 2.

Diagnostic performance of CAD system.

	AUC (95 % CI)	Accuracy% (95 % CI)	Sensitivity% (95 % CI)	Specificity% (95 % CI)	PPV% (95 % CI)	NPV% (95 % CI)
CAD system	0.837 (77.7–89.6)	87.2 (81.8–92.6)	80.0 (73.6–86.4)	87.4 (82.0–92.7)	18.2 (2.1–34.3)	99.2 (97.8–100.0)

CAD = computer-aided diagnosis; AUC = area under the curve; PPV = positive predictive value; NPV = negative predictive value.

Among the five malignant lesions, one lesion was diagnosed as possibly benign and four lesions were correctly diagnosed as possibly malignant by CAD system (Fig. 2 (A, B) and Fig. 3 (A, B)). Table 3 shows the characteristics of the five malignant breast lesions.

Fig. 2.

A breast lesion of a 35-year-old woman which proved to be lymphoma. (A) The long-axis gray-scale image showed a hypoechoic lesion, which was classified as BI-RADS 3 by the radiologist. (B) The corresponding CAD result suggested “possibly benign”.

Fig. 3.

A breast lesion of a 64-year-old woman which proved to be low-grade ductal carcinoma in situ. (A) The long-axis gray-scale image showed a hypoechoic lesion, which was classified as BI-RADS 3 by the radiologist. (B) The corresponding CAD result suggested “possibly malignant”.

Table 3.

Characteristics of the five malignant breast lesions.

	Age, years	Menopause	Family history	Pathological results	Largest diameter, cm
True positive cases (“possibly malignant” CAD result)	64	Yes	No	Ductal carcinoma in situ, low-grade	1.2
	36	No	No	Ductal carcinoma in situ, moderate grade	1.9
	36	No	No	Ductal carcinoma in situ, high-grade, with small focal invasion	3.8
	28	No	No	Ductal carcinoma in situ, low-grade	1.8
False negative cases (“possibly benign” CAD result)	35	No	No	Lymphoma	1.3

CAD = computer-aided diagnosis.

Among the 143 benign lesions, the CAD results of 125 lesions were possibly benign and the CAD results of 18 lesions were possibly malignant. Thus, CAD system revealed a false-positive result in 12.6 % (18/143) of benign BI-RADS 3 lesions. The 18 breast lesions misdiagnosed by CAD system included nine cases of proliferative disease, seven cases of fibroadenoma, one case of intraductal papilloma, and one case of plasma cell mastitis (Fig. 4 (A, B)). There were no significant differences between the true-negative and false-positive groups with regard to age, menopause, family history, and largest diameter of lesion (p＞0.05) (Table 4).

Fig. 4.

A breast lesion of a 25-year-old woman which proved to be fibroadenoma. (A) The long-axis gray-scale image showed a hypoechoic lesion, which was classified as BI-RADS 3 by the radiologist. (B) The corresponding CAD result suggested “possibly malignant”.

Table 4.

Characteristics of the 143 benign breast lesions.

	True negative group (“possibly benign” CAD result)	False positive group (“possibly malignant” CAD result)	P
N	125	18
Mean age ± SD, years	40.4 ± 11.1	36.5 ± 13.4	0.172
Menopause			0.177
Yes	30	7
No	95	11
Family history			0.589
Yes	2	0
No	123	18
Largest diameter ± SD, cm	1.6 ± 0.8	2.0 ± 0.8	0.053

CAD = computer-aided diagnosis; SD = standard deviation.

4. Discussion

The risk of malignancy in BI-RADS 3 lesions is relatively low, but can exceed 2 % in distinct patient populations [8,12,23,24]. In our study, malignancy rate of BI-RADS 3 lesions categorized by radiologists was slightly higher (3.4 %). CAD system contributed to classify patients into a low-risk group with a malignancy rate of 0.8 % or a high-risk group with a significantly increased malignancy rate of 18.2 %.

In this study, the CAD system successfully identified four out of five malignancies in BI-RADS 3 lesion group. The four true positive cases diagnosed by CAD system included three cases of ductal carcinoma in situ and one case of ductal carcinoma in situ with small focal invasion. This would allow 80.0 % of cancers to be directly indicated for biopsy at an early stage during the first US examination, rather than at the follow-up examinations many months or even years after the tumor has progressed. Missed or delayed diagnosis of cancer may have a decisive impact on a patient's survival. Our results suggest that CAD system is able to identify a high-risk group with a more than five-fold increase (18.2 %/3.4 %) in the risk of malignancy and a low-risk group with a reduced risk of malignancy in US-detected BI-RADS 3 breast lesions. Further studies with larger samples are needed to demonstrate this. In our study, a case of lymphoma was incorrectly classified as possibly benign by CAD system. After learning the data of nearly 10,000 lesions, CAD system (S-Detect) can distinguish malignant lesions from benign lesions [25]. However, breast lymphoma is very rare, so it is understandable that CAD system (S-Detect) has limited diagnostic ability of breast lymphoma.

In our series, 18 out of 143 benign lesions were misdiagnosed as possibly malignant by CAD system, resulting in 12.6 % (18/143) of benign BI-RADS 3 lesions recommended for biopsy. In this study, the cases misdiagnosed by CAD system were mainly proliferative diseases and fibroadenomas, which were suggested by CAD system to be possibly malignant because they were irregular and/or microlobulated. The reasons may be as follows [26]: (1) Proliferative disease may sometimes appear as a microlobulated or irregular solid mass. (2) As fibroadenomas grow, they are more likely to develop more than three lobulations or microlobulations. (3) The margins of complex fibroadenomas with peripherally located sclerosing adenosis may appear as irregularity. However, radiologists with personal experience classified these lesions as BI-RADS 3 because they considered these lesions had a likelihood of malignancy ≤2 %. The BI-RADS lexicon recommends short-term follow-up for BI-RADS 3 lesions rather than immediate biopsy [8]. However, at least 16.6 % of these patients often require a biopsy at one of follow-ups because of tumor progression or morphological changes [12]. In addition, ultrasound recommendation is not the only basis for making biopsy decisions. Other imaging findings, clinical history and patient's wishes will be also taken into consideration [12,25]. Therefore, there are still many patients with BI-RADS 3 lesions who undergo biopsy instead of short-term follow-up. Based on this experience, if CAD system was used as an adjunct to routine US, it would not increase the number of biopsies. Meanwhile, four-fifths of the cancers in this group could have been diagnosed immediately on the first US examination. In addition, it may reduce patient anxiety and medical costs associated with periodic surveillance in a low-risk group.

This study has several limitations. First, the interpretation of 16 radiologists was treated as a single reading interpretation, leading to a certain bias. Second, since all the BI-RADS 3 lesions included in the study were from the population with pathological results, there may be a selection bias. Third, some benign breast lesions were only confirmed by biopsy without surgical excision.

In conclusion, CAD system (S-Detect) enables radiologists to distinguish a high-risk group and a low-risk group among US-detected BI-RADS 3 breast lesions, so that patients in the low-risk group can receive follow-up without anxiety, while those in the high-risk group with a significantly increased malignancy rate should actively receive biopsy to avoid delayed diagnosis of breast cancer.

Funding

This study has received funding from Horizontal Project Foundation of Peking University Third Hospital (Grant No. H75502-21).

Declarations

Ethics statement: This prospective multicenter study was approved by the institutional review board of all participating centers (no. M2021304), and written informed consent was obtained from all patients. We confirm that the study complies with all regulations.

Data availability statement

The data associated with our study has not been deposited into a publicly available repository. The data that has been used is confidential.

CRediT authorship contribution statement

Ping He: Writing – original draft, Supervision, Software, Formal analysis, Conceptualization. Wen Chen: Writing – review & editing, Investigation. Ming-Yu Bai: Supervision, Data curation. Jun Li: Resources, Data curation. Qing-Qing Wang: Resources, Data curation. Li-Hong Fan: Resources, Data curation. Jian Zheng: Resources, Data curation. Chun-Tao Liu: Resources, Data curation. Xiao-Rong Zhang: Resources, Data curation. Xi-Rong Yuan: Resources, Data curation. Peng-Jie Song: Resources, Data curation. Li-Gang Cui: Writing – review & editing, Visualization, Validation, Supervision, Project administration, Investigation, Funding acquisition, Conceptualization.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.