Abstract
As thyroid and breast cancer have several US findings in common, we applied an artificial intelligence computer-assisted diagnosis (AI-CAD) software originally developed for thyroid nodules to breast lesions on ultrasound (US) and evaluated its diagnostic performance. From January 2017 to December 2017, 1042 breast lesions (mean size 20.2 ± 11.8 mm) of 1001 patients (mean age 45.9 ± 12.9 years) who underwent US-guided core-needle biopsy were included. An AI-CAD software that was previously trained and validated with thyroid nodules using the convolutional neural network was applied to breast nodules. There were 665 benign breast lesions (63.0%) and 391 breast cancers (37.0%). The area under the receiver operating characteristic curve (AUROC) of AI-CAD to differentiate breast lesions was 0.678 (95% confidence interval: 0.649, 0.707). After fine-tuning AI-CAD with 1084 separate breast lesions, the diagnostic performance of AI-CAD markedly improved (AUC 0.841). This was significantly higher than that of radiologists when the cutoff category was BI-RADS 4a (AUC 0.621, P < 0.001), but lower when the cutoff category was BI-RADS 4b (AUC 0.908, P < 0.001). When applied to breast lesions, the diagnostic performance of an AI-CAD software that had been developed for differentiating malignant and benign thyroid nodules was not bad. However, an organ-specific approach guarantees better diagnostic performance despite the similar US features of thyroid and breast malignancies.
Keywords: Breast neoplasms; Thyroid nodule; Diagnosis, Computer-assisted; Artificial intelligence
Introduction
Ultrasound (US) is a major imaging modality used to diagnose both thyroid and breast cancer. Imaging features on US are very important in differentiating benign and malignant lesions in the thyroid gland and breast, and help radiologists decide which lesions need to undergo aspiration or biopsy [1]. Hypoechogenicity or marked hypoechogenicity, microlobulated or irregular margins, microcalcifications, and taller-than-wide shape suggest malignancy in thyroid nodules [2–5]. According to the American College of Radiology Breast Imaging Reporting and Data System (ACR BI-RADS), irregular shape, not-circumscribed margins, microcalcifications, and not-parallel orientation are also suspicious features for breast nodules [6].
US examinations are known to be highly performer dependent in their imaging acquisition and interpretation [7]. Quantitative analysis by artificial intelligence computer-assisted diagnosis (AI-CAD) may be a solution for this variability and lack of reproducibility regardless of the type of organ being examined. The convolutional neural network (CNN), a popular deep learning structure, has been widely used for AI-CAD in thyroid and breast nodules alike. It has shown a diagnostic accuracy of 82–98% for thyroid cancer, which is not inferior to radiologists [8–13]. Commercial software based on CNN is already used for breast US in clinical practice [14–16].
Until now, AI-CAD systems have been specifically developed for certain medical fields or organs, and we wondered if these systems could be integrated together or if they needed to be applied independently due to innate differences. Using 13,560 US images of thyroid nodules labeled benign and malignant, we recently developed an AI-CAD algorithm (Severance Artificial intelligence program, SERA) [17]. Concentrating on several suspicious features on US that are common to thyroid and breast cancer, we investigated whether SERA developed from thyroid nodules could work on breast nodules with similar performance.
Methods
This retrospective study was approved by the institutional review board (IRB) of Severance Hospital, Seoul, Korea, with a waiver for informed consent.
Study Population
From January 2017 to December 2017, 1520 patients who underwent US-guided core-needle biopsy on breast lesions in our institution were included. Among them, 469 patients who had breast nodules less than 1 cm in size were excluded since the AI-CAD was originally developed with thyroid nodules larger than 1 cm. We also excluded 18 patients who were lost to follow-up after being diagnosed with uncertain malignant potential (B3, high-risk) lesions [18], 14 patients diagnosed with other malignancies including hematologic malignancy or metastasis from other organs, 14 patients who underwent re-biopsy for proven malignant lesions, and 4 male patients diagnosed with gynecomastia were excluded. Finally, 1042 breast lesions of 1001 patients were enrolled in our study (Fig. 1).
Fig. 1.
Study population
The standard reference was the pathologic result of US-guided 14-gauge core-needle breast biopsy in 973 breast lesions. For the 69 high-risk lesions, the standard reference was the pathologic results of vacuum-assisted biopsy for 15 lesions, surgical pathology for 52 lesions, and stability on follow-up US examinations for 2 lesions.
All 1001 patients were included in two previously published domestic articles about the medical audit for US guided biopsy and positive predictive values of BI-RADS 4 and 5 lesions [19, 20]. Among cancer patients, 192 patients diagnosed between January 2017 and April 2017 were included in a study on the diagnostic performance of synthetic mammography applied with AI-CAD [21].
In addition, we separately collected 1084 breast lesions that were biopsied during 2018 in the same institution to fine-tune the model. The newly collected cases consisted of 508 malignant lesions (47%, 508/1084) and 576 benign lesions (53%, 576/1084).
Image Analysis
US examinations followed by biopsy were performed by one of eleven radiologists specializing in breast imaging with 1–22 years of experience using two different US machines (iU22; Phillips Medical Systems, Bothell, WA, LOGIQ E9; GE Healthcare, Milwaukee, WI). The radiologists prospectively recorded lesion size, image findings, and final assessment according to BI-RADS at the time of examination. One radiologist (S.E.L with 2 years of experience in breast imaging) retrospectively reviewed the US examinations and selected one representative image for each biopsied lesion. The images were stored as JPEG images. Using the Paint program of Window 7, square regions of interest (ROIs) covering the whole nodule were drawn by a researcher and confirmed by the radiologist (S.E.L).
AI-CAD Application
The SERA is a deep learning-based computer-aided diagnosis. A research team in the department of radiology/research institute of radiological science and the school of mathematics and computing in Yonsei University developed SERA that had been trained with 13,560 US images of thyroid nodules that were either surgically confirmed or cytologically proven as benign (category II) or malignant (category VI) on the Bethesda system and larger than 1 cm in size [17]. The pre-trained convolutional neural network VGG-16 was used for transfer learning and modified through fine-tuning. In the choice of pre-trained net for transfer learning, “AlexNet,” “VGG-16,” “VGG-19,” “SqueezeNet,” “GoogLeNet,” “Inception-v3,” “DenseNet201,” “ResNet-18,” “ResNet-50,” “ResNet-101,” “Xception,” and “Inception-Resnet-v2” were tested with Matlab2019b and “VGG-16” was selected because it provided the best validation performance. The team optimized network parameters, weights, and biases, through a stochastic gradient descent with momentum algorithm with a mini-batch size 64. The initial learning late was set to 3 × 10–4, and 20 epochs were conducted in the training process with 13,560 data (6780 benign and 6780 malignant) (Fig. 2). The homepage of SERA is “http://seracse.yonsei.ac.kr” and it is only available through membership registration. When users upload an US image, SERA presents continuous numbers between 0 and 100, which correspond to the probability of a given test image being malignant (Fig. 3). In an external validation set for diagnosing thyroid nodules, SERA showed comparable diagnostic performance to expert radiologists [17].
Fig. 2.
Schematic figure of convolutional neural networks for developing and fine-tuning SERA
Fig. 3.
Image of the SERA (Severance Artificial intelligence) program which is a deep learning-based computer-aided diagnosis software developed for diagnosing thyroid nodules. When we upload an US image, SERA shows the binary result (benign or malignant) with a malignant probability score
For this study, we additionally fine-tuned SERA with 1084 breast lesions, consisting of 508 malignant lesions (47%, 508/1084) and 576 benign lesions (53%, 576/1084).
Statistical Analysis
We used the continuous number generated by SERA that indicated the probability of malignancy to evaluate its diagnostic performance. The diagnostic performance of SERA was assessed using a receiver operating characteristic (ROC) curve analysis, and the area under the curve (AUC) with a 95% confidence interval (CI) was calculated. The optimal cut-point for calculating sensitivity and specificity was estimated based on the Youden index. We compared diagnostic performance between the initial version and fine-tuned version of SERA using the generalized estimating equation.
Final assessments made by the radiologist were dichotomized as positive or negative using two different BI-RADS categories as cutoffs: (1) BI-RADS 4a: negative for category 2 and 3 lesions, positive for category 4a to 5 lesions; and (2) BI-RADS 4b: negative for category 2, 3, and 4a lesions, and positive for category 4b to 6. Statistical analyses were performed using Medcalc (MedCalc Software Ltd, Belgium). P values of less than 0.05 were considered to indicate statistical significance.
Results
Table 1 summarizes the clinical and pathological characteristics of the 1001 patients (mean age 45.9 ± 12.9 years) and the 1042 breast lesions that consisted of 665 (63.0%) benign findings and 377 (37.0%) malignancies. The mean size of the total 1042 breast nodules was 20.2 ± 11.8 mm (range, 10–100 mm). Nearly half of lesions were assessed as BI-RADS 4a (484/1042, 46.4%). The pathologic results from core-needle biopsy are listed in Table 1. Most of the benign breast lesions were fibroadenoma (405/665, 60.9%), and most of the malignant breast lesions were invasive ductal carcinoma (263/377, 70.0%).
Table 1.
Demographics of the 1001 patients and 1042 breast lesions
| N (%) | |
|---|---|
| Mean age (years) | 45.8 ± 12.9 |
| Mean size (mm) | 20.1 ± 11.7 |
| US BI-RADS category | |
| 2 | 3 (0.3) |
| 3 | 158 (15.2) |
| 4a | 484 (46.4) |
| 4b | 79 (7.6) |
| 4c | 121 (11.6) |
| 5 | 197 (18.9) |
| Pathologic diagnosis | |
| Benign | 665 |
|
Fibroadenoma Fibroadenomatoid hyperplasia Intraductal papilloma Stromal fibrosis Ectatic duct Adenosis Inflammation Fibrocystic change Epithelioid granuloma Fat necrosis Benign phyllodes tumor Atypical ductal hyperplasia Apocrine metaplasia Others |
405 (60.9) 64 (9.6) 40 (6.0) 32 (4.8) 30 (4.5) 14 (2.1) 12 (1.8) 11 (1.7) 11 (1.7) 9 (1.4) 8 (1.2) 7 (1.1) 7 (1.1) 15 (2.3) |
| Malignancy | 377 |
| Invasive ductal carcinoma | 263 (70.0) |
| Ductal carcinoma in situ | 71 (18.8) |
| Invasive carcinoma with lobular feature | 22 (5.8) |
| Papillary carcinoma | 8 (2.1) |
| Mucinous carcinoma | 6 (1.6) |
| Others | 7 (1.9) |
US ultrasonography, BI-RADS Breast Imaging Reporting And Data System
Diagnostic Performance of SERA Trained by Thyroid Lesions
When we applied SERA to all breast lesions, the median cancer probability score was 0.85 (range 0.06–0.99). Overall AUC for the score to differentiate breast lesions was 0.678 (95% CI: 0.649, 0.707) with a sensitivity of 47.5%, specificity of 79.7%, positive predictive value (PPV) of 57.0%, and negative predictive value (NPV) of 72.8% (Table 2). Representative cases of a correct and wrong diagnosis by SERA are shown in Figs. 4 and 5, respectively.
Table 2.
Comparison of diagnostic performance for breast cancer between SERA and radiologists
| SERA | SERA after fine-tuning | P-valuea | Radiologists (BI-RADS 4a criterion) | P-valueb | Radiologists (BI-RADS 4b criterion) | P-valuec | |
|---|---|---|---|---|---|---|---|
| Sensitivity | 47.5 (42.3, 52.7) | 73.7 (69.0, 78.1) | < 0.001 | 100 (99.0, 100) | < 0.001 | 90.2 (86.7, 93.0) | < 0.001 |
| Specificity | 79.7 (76.4, 82.7) | 81.1 (77.9, 84.0) | 0.5355 | 24.2 (21.0, 27.7) | < 0.001 | 91.4 (89.0, 93.4) | < 0.001 |
| PPV | 57.0 (52.4, 61.5) | 68.8 (65.1, 72.3) | < 0.001 | 42.8 (41.7, 43.8) | < 0.001 | 85.6 (82.3, 88.5) | < 0.001 |
| NPV | 72.8 (70.7, 74.8) | 84.5 (82.1, 86.6) | < 0.001 | 100 (NA) | < 0.001 | 94.3 (92.4, 95.7) | < 0.001 |
| Accuracy | 68.0 (65.1, 70.9) | 78.4 (75.8, 80.9) | < 0.001 | 51.6 (48.5, 54.7) | < 0.001 | 91.0 (89.1, 92.6) | < 0.001 |
| AUC | 0.678 (0.649, 0.707) | 0.841 (0.817, 0.863) | < 0.001 | 0.621 (0.591, 0.651) | < 0.001 | 0.908 (0.889, 0.925) | < 0.001 |
The number in parentheses indicates the 95% confidence interval
SERA Severance Artificial intelligence program, BI-RADS Breast Imaging Reporting And Data System, PPV positive predictive value, NPV negative predictive value, AUC area under the receiver operating characteristic curve
aComparison between SERA and SERA after fine-tuning
bComparison between SERA after fine-tuning and radiologists (BI-RADS 4a)
cComparison between SERA after fine-tuning and radiologists (BI-RADS 4b)
Fig. 4.
A 59-year-old woman diagnosed with invasive ductal carcinoma. Extremely dense mammography shows no correlating lesions except the benign mass opacity in the left lower medial breast (a–b), and US shows a 17-mm hypoechoic, microlobulated mass in the left 9 o’clock position, which was diagnosed as BI-RADS 4a (c, arrow). The SERA correctly diagnosed the lesion as malignant (score 0.98) from the input image drawn in the region of interest (d). BI-RADS: Breast Imaging Reporting And Data System, SERA: Severance Artificial intelligence program
Fig. 5.
A 66-year-old woman diagnosed with invasive ductal carcinoma. Mammography shows a suspicious mass opacity in the right breast (a-b, arrow), and US shows a 10-mm hyperechoic, spiculated mass in the right 12 o’clock position, which was diagnosed as BI-RADS 4c (c). The SERA misdiagnosed the lesion as benign (score 0.30) from the input image drawn in the region of interest (d)
Diagnostic Performance of SERA After Fine-tuning
After fine-tuning, the diagnostic performance of SERA for breast lesions improved. Overall AUC to differentiate breast lesions was 0.841 (95% CI: 0.817, 0.863) with a sensitivity of 73.7%, specificity of 81.1%, PPV of 68.8% and NPV of 84.5%, with all diagnostic values significantly higher than those of the initial version of SERA, except for specificity that did not show statistical significant differences (Table 2). A representative case of changed diagnosis after fine-tuning is shown in Fig. 6.
Fig. 6.
A 58-year-old woman diagnosed with invasive ductal carcinoma. Mammography shows suspicious mass opacity in the right breast (a–b, arrow), and US shows a 12-mm isoechoic, microlobulated mass in the right 11 o’clock position, which was diagnosed as BI-RADS 4a (c). Initial SERA misdiagnosed the lesion as benign, but after fine-tuning, correctly diagnosed the lesion as malignancy (score 0.75) from the input image drawn in the region of interest (d). BI-RADS: Breast Imaging Reporting And Data System, SERA: Severance Artificial intelligence program
Figure 7 compares diagnostic performance between SERA and radiologists. The overall AUC of SERA was significantly higher than that of radiologists when the cutoff category was set at BI-RADS 4a (radiologists: 0.621, P ≤ 0.001) with all diagnostic parameters being higher including sensitivity, specificity, PPV, and NPV. However, when the cutoff was set at BI-RADS 4b, the overall AUC of SERA was significantly lower than that of radiologists (Radiologists: 0.908, P < 0.001).
Fig. 7.
Area under the receiver operating characteristic curves (AUROCs) for the diagnostic performance of SERA after fine-tuning and radiologists with both BI-RADS 4a and 4b criteria
Discussion
We applied SERA that was originally trained with thyroid lesions to breast lesions in the hope that it could be adopted to diagnose breast cancer because thyroid and breast cancers share several suspicious features on US. Overall AUC of SERA to differentiate 1042 breast lesions was 0.678, and this increased to 0.841 after fine-tuning with 1084 breast lesions. This was comparable with previous results of SERA, which presented AUCs of 0.821–0.885 to differentiate malignant thyroid nodules [17].
Prior studies on CNN models have shown AUCs of 0.73–0.93 or accuracies of 82–98% for differentiating thyroid nodules [8–13], and several have even shown better performances than radiologists [10, 17]. Despite originating in different organs, cancers can share US features due to common malignant characteristics such as infiltrating growth, rapid growth rate, or tendency to invade surrounding tissues [22]. It is common for breast and thyroid cancers to present with not-circumscribed margins, vertical orientation, hypoechogenicity, and internal microcalcifications on US. The thyroid nodules used to develop the AI-CAD program mainly consisted of papillary thyroid carcinoma (96.5% of malignant lesions), which could have resulted in more consistent imaging findings. However, the pathologies of the breast nodules included in this study were heterogeneous as shown in Table 1. Even invasive ductal carcinoma, the most common malignant breast cancer (70.1%), is known to present distinctive imaging features depending on molecular subtype [23]. Besides, ductal carcinoma in situ usually manifests as a “non-mass-like lesion” on US, which is not a subtype defined on thyroid US [24]. This might explain why the AI-CAD program did not show comparable diagnostic performance for breast lesions compared to thyroid nodules.
When comparing the diagnostic performance of SERA with radiologists, we selected two cutoff categories, BI-RADS 4a and 4b, to evaluate the diagnostic performance of radiologists. SERA showed significantly higher AUC with all diagnostic parameters being higher than radiologists when the cutoff category was BI-RADS 4a. However, it showed significantly lower AUC with decreased diagnostic parameters when the cutoff was BI-RADS 4b. A previous study showed similar results when breast nodules were diagnosed with an initial version of commercial breast CAD based on feature extraction techniques combined with a classifier [14]. CAD also showed better diagnostic performance than radiologists when the cutoff was BI-RADS 4a, but not BI-RADS 4b, although radiologists currently consider BI-RADS 4a as the cutoff for biopsy rather than the differentiation of benign and malignant lesions. However, recently developed AI-CAD programs based on CNN for breast lesions have improved the diagnostic performance of radiologists, and this improvement is especially more pronounced in inexperienced readers [16, 25, 26].
A previous study suggested that an AI-CAD program based on deep learning may be more influenced by nodule size compared to a program based on other machine learning techniques of classifiers following feature extraction because the deep learning model seems to automatically incorporate information on nodule size from the input itself to maximize performance [9]. Breast lesions are generally larger than thyroid lesions; therefore, a thyroid AI-CAD might judge breast lesions more suspiciously, resulting in the high median score of cancer probability seen in our study.
There are several limitations to this study. First, this was a retrospective study performed in a single institution. Second, although we did fine-tune the CNN model with more breast lesions, the number might not be enough to show better results. However, we wanted to see if a CNN model trained with thyroid nodules could be applied to breast lesions. Third, the method of drawing ROI as a square around the lesion might not be fully compatible with breast lesions because the background parenchyma of the breast is generally more heterogeneous than the background parenchyma of the thyroid gland. Fourth, we used an in-house AI-CAD program developed for thyroid nodules larger than 1 cm, and the results from this program might not be immediately generalized to other AI-CAD and breast nodules less than 1 cm.
In conclusion, the diagnostic performance of the thyroid AI-CAD was not bad for breast lesions. However, an organ-specific approach accompanied with specific training is required to maximize the performance of the AI-CAD program for the breast despite US features common to both breast and thyroid cancer.
Acknowledgements
The authors thank Hanpyo Hong, PhD for providing the expertise needed to perform the additional statistical analysis for revision and Medical Illustration and Design, part of the Medical Research Support Services of Yonsei University College of Medicine for their help in designing the figures.
Author Contribution
SEL and JYK contributed to the design and implementation of the research, to the analysis of the results, and to the writing of the manuscript. EL developed the AI-CAD program and obtained the results from the program. E-KK, JHY, VYP, and JHY participated in revision of the manuscript.
Funding
This study was supported by the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIT) (2019R1A2C1002375 and 2021R1A2C2007492).
Availability of Data and Materials
Additional documents related to this study are available on request to the corresponding author. However, the datasets from Severance Hospital were used under license for the current study and are not publicly available.
Code Availability
The homepage of SERA is “http://seracse.yonsei.ac.kr,” and it is only available through membership registration.
Declarations
Disclaimer
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Ethics Approval
This retrospective study was approved by the institutional review board of Severance Hospital, Seoul, Korea, with a waiver for informed consent.
Consent to Participate
Not applicable.
Consent for Publication
Not applicable.
Conflict of Interest
The authors declare no competing interests.
Footnotes
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Frates MC, Benson CB, Charboneau JW, Cibas ES, Clark OH, Coleman BG, Cronan JJ, Doubilet PM, Evans DB, Goellner JR. Management of thyroid nodules detected at US: Society of Radiologists in Ultrasound consensus conference statement. Radiology. 2005;237(3):794–800. doi: 10.1148/radiol.2373050220. [DOI] [PubMed] [Google Scholar]
- 2.Haugen BR, Alexander EK, Bible KC, Doherty GM, Mandel SJ, Nikiforov YE, Pacini F, Randolph GW, Sawka AM, Schlumberger M. 2015 American Thyroid Association management guidelines for adult patients with thyroid nodules and differentiated thyroid cancer: the American Thyroid Association guidelines task force on thyroid nodules and differentiated thyroid cancer. Thyroid. 2016;26(1):1–133. doi: 10.1089/thy.2015.0020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Kim E-K, Park CS, Chung WY, Oh KK, Kim DI, Lee JT, Yoo HS. New sonographic criteria for recommending fine-needle aspiration biopsy of nonpalpable solid nodules of the thyroid. American Journal of Roentgenology. 2002;178(3):687–691. doi: 10.2214/ajr.178.3.1780687. [DOI] [PubMed] [Google Scholar]
- 4.Kwak JY, Han KH, Yoon JH, Moon HJ, Son EJ, Park SH, Jung HK, Choi JS, Kim BM, Kim E-K. Thyroid imaging reporting and data system for US features of nodules: a step in establishing better stratification of cancer risk. Radiology. 2011;260(3):892–899. doi: 10.1148/radiol.11110206. [DOI] [PubMed] [Google Scholar]
- 5.Moon W-J, Jung SL, Lee JH, Na DG, Baek J-H, Lee YH, Kim J, Kim HS, Byun JS, Lee DH. Benign and malignant thyroid nodules: US differentiation—multicenter retrospective study. Radiology. 2008;247(3):762–770. doi: 10.1148/radiol.2473070944. [DOI] [PubMed] [Google Scholar]
- 6.Mendelson EB B-VM, Berg WA, et al.: ACR BI-RADS® Ultrasound. In: ACR BI-RADS® Atlas, Breast Imaging Reporting and Data System. 5th ed. VA: Reston, 2013.
- 7.Kim SH, Park CS, Jung SL, Kang BJ, Kim JY, Choi JJ, Kim YI, Oh JK, Oh JS, Kim H. Observer variability and the performance between faculties and residents: US criteria for benign and malignant thyroid nodules. Korean journal of radiology. 2010;11(2):149–155. doi: 10.3348/kjr.2010.11.2.149. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Ko SY, Lee JH, Yoon JH, Na H, Hong E, Han K, Jung I, Kim EK, Moon HJ, Park VY. Deep convolutional neural network for the diagnosis of thyroid nodules on ultrasound. Head & neck. 2019;41(4):885–891. doi: 10.1002/hed.25415. [DOI] [PubMed] [Google Scholar]
- 9.Park VY, Han K, Seong YK, Park MH, Kim E-K, Moon HJ, Yoon JH, Kwak JY. Diagnosis of thyroid nodules: performance of a Deep Learning convolutional neural network Model vs. Radiologists. Scientific reports. 2019;9(1):1–9. doi: 10.1038/s41598-019-54434-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Lee E, Ha H, Kim HJ, Moon HJ, Byon JH, Huh S, Son J, Yoon J, Han K, Kwak JY. Differentiation of thyroid nodules on US using features learned and extracted from various convolutional neural networks. Scientific Reports. 2019;9(1):1–11. doi: 10.1038/s41598-019-56395-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Gao L, Liu R, Jiang Y, Song W, Wang Y, Liu J, Wang J, Wu D, Li S, Hao A. Computer-aided system for diagnosing thyroid nodules on ultrasound: a comparison with radiologist-based clinical assessments. Head & neck. 2018;40(4):778–783. doi: 10.1002/hed.25049. [DOI] [PubMed] [Google Scholar]
- 12.Ma J, Wu F, Zhu J, Xu D, Kong D. A pre-trained convolutional neural network based method for thyroid nodule diagnosis. Ultrasonics. 2017;73:221–230. doi: 10.1016/j.ultras.2016.09.011. [DOI] [PubMed] [Google Scholar]
- 13.Chi J, Walia E, Babyn P, Wang J, Groot G, Eramian M. Thyroid nodule classification in ultrasound images by fine-tuning deep convolutional neural network. Journal of digital imaging. 2017;30(4):477–486. doi: 10.1007/s10278-017-9997-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Kim K, Song MK, Kim EK, Yoon JH: Clinical application of S-Detect to breast masses on ultrasonography: a study evaluating the diagnostic performance and agreement with a dedicated breast radiologist. Ultrasonography 36(1):3–9, 2017. 10.14366/usg.16012 [DOI] [PMC free article] [PubMed]
- 15.Cho E, Kim EK, Song MK, Yoon JH. Application of computer-aided diagnosis on breast ultrasonography: evaluation of diagnostic performances and agreement of radiologists according to different levels of experience. J Ultrasound Med. 2018;37(1):209–216. doi: 10.1002/jum.14332. [DOI] [PubMed] [Google Scholar]
- 16.Choi JS, Han BK, Ko ES, Bae JM, Ko EY, Song SH, Kwon MR, Shin JH, Hahn SY. Effect of a deep learning framework-based computer-aided diagnosis system on the diagnostic performance of radiologists in differentiating between malignant and benign masses on breast ultrasonography. Korean J Radiol. 2019;20(5):749–758. doi: 10.3348/kjr.2018.0530. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Koh J, Lee E, Han K, Kim EK, Son EJ, Sohn YM, Seo M, Kwon MR, Yoon JH, Lee JH, Park YM, Kim S, Shin JH, Kwak JY. Diagnosis of thyroid nodules on ultrasonography by a deep convolutional neural network. Sci Rep. 2020;10(1):15245. doi: 10.1038/s41598-020-72270-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Houssami N, Ciatto S, Bilous M, Vezzosi V, Bianchi S. Borderline breast core needle histology: predictive values for malignancy in lesions of uncertain malignant potential (B3) British journal of cancer. 2007;96(8):1253–1257. doi: 10.1038/sj.bjc.6603714. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Lee SE, Park VY, Yoon JH, Moon HJ, Kim Mj, Kim E-K: Sonographically guided 14-guage core needle biopsy: medical audit for one year of 2017 Journal of Korean Society of Breast Screening 16:70–76, 2019.
- 20.Lee SE, Park VY, Yoon JH, Moon HJ, Kim Mj, Kim E-K: Positive predictive value of breast ultrasonography BI-RADS category 4 and 5 lesions in one institution at 2017 Journal of Korean Society of Breast Screening 16:53–59, 2019.
- 21.Lee SE, Han K, Kim E-K: Application of artificial intelligence–based computer-assisted diagnosis on synthetic mammograms from breast tomosynthesis: comparison with digital mammograms. European Radiology:1–9, 2021. [DOI] [PubMed]
- 22.Baba AI, Catoi C. Comparative oncology. Bucharest (RO): The Publishing House of the Romanian Academy; 2007. [PubMed] [Google Scholar]
- 23.Cho N. Molecular subtypes and imaging phenotypes of breast cancer. Ultrasonography. 2016;35(4):281. doi: 10.14366/usg.16030. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Park JW, Ko KH, Kim E-K, Kuzmiak CM, Jung HK. Non-mass breast lesions on ultrasound: final outcomes and predictors of malignancy. Acta Radiol. 2017;58(9):1054–1060. doi: 10.1177/0284185116683574. [DOI] [PubMed] [Google Scholar]
- 25.Park HJ, Kim SM, La Yun B, Jang M, Kim B, Jang JY, Lee JY, Lee SH. A computer-aided diagnosis system using artificial intelligence for the diagnosis and characterization of breast masses on ultrasound: added value for the inexperienced breast radiologist. Medicine. 2019;98(3):e14146–e14146. doi: 10.1097/MD.0000000000014146. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Lee J, Kim S, Kang BJ, Kim SH, Park GE: Evaluation of the effect of computer aided diagnosis system on breast ultrasound for inexperienced radiologists in describing and determining breast lesions. Medical ultrasonography 21(3):7, 2019. 10.11152/mu-1889 [DOI] [PubMed]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
Additional documents related to this study are available on request to the corresponding author. However, the datasets from Severance Hospital were used under license for the current study and are not publicly available.
The homepage of SERA is “http://seracse.yonsei.ac.kr,” and it is only available through membership registration.