Artificial Intelligence Improves Detection of Supplemental Screening Ultrasound-detected Breast Cancers in Mammography

Abstract

Despite recent advances in artificial intelligence (AI) software with improved performance in mammography screening for breast cancer, insufficient data are available on its performance in detecting cancers that were initially missed on mammography. In this study, we aimed to determine whether AI software-aided mammography could provide additional value in identifying cancers detected through supplemental screening ultrasound. We searched our database from 2017 to 2018 and included 238 asymptomatic patients (median age, 50 years; interquartile range, 45–57 years) diagnosed with breast cancer using supplemental ultrasound. Two unblinded radiologists retrospectively reviewed the mammograms using commercially available AI software and identified the reasons for missed detection. Clinicopathological characteristics of AI-detected and AI-undetected cancers were compared using univariate and multivariate logistic regression analyses. A total of 253 cancers were detected in 238 patients using ultrasound. In an unblinded review, the AI software failed to detect 187 of the 253 (73.9%) mammography cases with negative findings in retrospective observations. The AI software detected 66 cancers (26.1%), of which 42 (63.6%) exhibited indiscernible findings obscured by overlapping dense breast tissues, even with the knowledge of magnetic resonance imaging and post-wire localization mammography. The remaining 24 cases (36.4%) were considered interpretive errors by the radiologists. Invasive tumor size was associated with AI detection after multivariable analysis (odds ratio, 2.2; 95% confidence intervals, 1.5–3.3; p < 0.001). In the control group of 160 women without cancer, the AI software identified 19 false positives (11.9%, 19/160). Although most ultrasound-detected cancers were not detected on mammography with the use of AI, the software proved valuable in identifying breast cancers with indiscernible abnormalities or those that clinicians may have overlooked.

INTRODUCTION

Early detection by mammographic screening plays a crucial role in reducing breast cancer mortality [1]. However, mammography has limitations with an overall sensitivity of 75% to 85%, which can decrease to 30–50% in women with dense breast tissue [2,3]. Dense breasts on mammography are important factors contributing to the missed early diagnosis and an increased incidence of interval or advanced breast cancer [2,4,5].

Thus, the American College of Radiology (ACR) recommends supplemental breast cancer screening with ultrasound for women at high risk who would qualify for but cannot undergo magnetic resonance imaging (MRI) [6]. In a prospective Japan Strategic Anti-Cancer Randomized Trial (J-START) study, breast cancer detection improved in a group of asymptomatic Japanese women, aged 40–49 years, who underwent mammography with ultrasound compared to the group that underwent only mammography (91.1% vs. 77.0%), regardless of their breast density [7].

Recent advancements in artificial intelligence (AI) have shown promising results in mammography screening performance [8,9,10]. For instance, mammography readings obtained using AI software can improve cancer detection and reduce recall rates [8]. It can enhance diagnostic performance [11] and decrease the need for supplementary mammographic views [12]. In addition, a standalone AI application in mammographic screening workflows can achieve or even exceed human reader detection performance and improve efficiency [13]. Considering that AI software can reduce mammographic misdiagnosis [8,14], supplemental screening of ultrasound-detected cancers that were initially overlooked or misinterpreted on mammography can be correctly identified using AI. However, the performance of AI software in detecting cancers found through supplemental screening ultrasound still needs to be adequately assessed.

Therefore, in this study, we aimed to analyze whether AI-aided mammography readings could detect breast cancers that were initially missed during screening mammography but later detected by supplemental ultrasound. In addition, we investigated the clinical and pathological characteristics associated with cancer detection using AI software.

METHODS

Patient inclusion

This retrospective study was approved by the Institutional Review Board of our hospital (IRB. No. H-2006-066-1131). Patient approval and written informed consent were not required to review images or records.

We retrospectively searched our hospital’s cancer registry from January 2017 to December 2018 and included asymptomatic patients diagnosed with breast cancer through ultrasound screening who underwent surgery as an initial treatment without neoadjuvant chemotherapy (n = 884). We excluded patients diagnosed with mammographic abnormalities (n = 609) to include only supplemental ultrasound-detected breast cancer, mammography with skin markers (n = 30), and breast cancer with apparent architectural distortion due to previous surgical excision or biopsy (n = 7). A total of 238 patients with ultrasound-detected breast cancer were included in this study. Among them, 14 had multicentric breast cancers, and one had bilateral cancers; thus, 253 were finally included for the analysis (Supplementary Figure 1).

In addition to the study group, the control group included patients who underwent screening mammography and ultrasound, along with subsequent breast biopsy due to ultrasound-detected suspicious lesions that were later confirmed as benign during the same study period at our healthcare center (n = 210). Among them, we excluded 50 patients who did not have at least 1 year of follow-up record. Consequently, 160 benign cases were included in this study.

Image acquisition

All imaging data and AI output results were obtained prospectively in routine clinical practice and stored in the institutional picture archiving and communication system. All mammography images were acquired using a full-field digital mammography unit. Whole-breast ultrasound examinations were performed by one of the eight breast radiologists with 1–30 years of experience (Supplementary Data 1). MRI examinations were performed using a 3-T scanner (MAGNETOM Skyra [Siemens Healthineers, Erlangen, Germany], Ingenia [Philips Healthcare, Amsterdam, Netherlands]) with a dedicated breast coil. Detailed imaging sequences and parameters have been described previously [15].

AI software

The AI algorithm used in this study was developed using deep convolutional neural networks (Lunit INSIGHT MMG, ver. 1.1.4.0; Lunit, Seoul, Korea) employing the ResNet-34-based neural network. The process of developing and configuring this commercially available AI system for breast cancer detection has been previously described [8,16]. The system displays a heat map for areas suspicious of breast cancer in each breast from all four mammographic images. It assigns an assessment score of 0–100 for tumor presence, where 100 represents the highest level of suspicion. The AI system provides per image four-view heat maps and a representative abnormality score for each breast, based on the maximum abnormality score from the craniocaudal and mediolateral oblique views.

Data collection and reference standard

Two breast radiologists (Chang JM and Yoen H, with 16 and 3 years of experience in breast imaging, respectively) reviewed all imaging examinations, including mammography with AI software, ultrasound, and, if available, MRI, along with the medical records and the final surgical pathology. They evaluated whether known malignancies were accurately depicted and categorized as Breast Imaging Reporting & Data System 4A–5 for each modality. In the AI reports, maximum pixel-level abnormality scores of ≥10 points were considered as test-positive, and an abnormality scores < 10 were displayed as “low,” representing test-negative [8,12,17,18,19]. In cancer cases, maximum pixel-level abnormality scores of ≥ 10 points within the corresponding mammographic location in at least one view were considered accurate. Preoperative breast MRI and/or post-wire localization mammograms were used as a reference standard for determining the cancer location. In multifocal, multicentric, or bilateral cancer cases, we assessed whether each type of cancer could be identified using different imaging modality and AI software. In the negative control group, maximum pixel-level abnormality scores of ≥ 10 points for at least one view were considered false-positive.

Clinical findings and biopsy or surgical results, including tumor size, histological type, nuclear grade, estrogen receptor, progesterone receptor, human epidermal growth factor receptor type 2, and Ki-67 status, were obtained from pathology reports.

Statistical analyses

The number of breast cancer and false-positive cases detected by the AI software was calculated. The Student’s t-test or Mann–Whitney U test was used for continuous variables to compare AI-detected and AI-undetected cancers. The χ² and Fisher’s exact tests were used to analyze categorical variables. Univariate and multivariate logistic regression analyses were used to determine factors associated with AI-detected cancers. Variables with a p-value of < 0.1 were entered into the multivariable analysis. All statistical analyses were performed using SPSS (Released 2020, IBM SPSS Statistics for Windows, Version 27.0.; IBM Corp., Armonk, USA).

RESULTS

The median age of the patients was 50 years (Interquartile range: 45–57 years). Among the 238 patients, 59.2% (141/238) had heterogeneously dense breasts and 33.2% (79/238) had extremely dense breasts; additionally, 84.5% of them underwent breast-conserving surgery (201/238) and 15.5% (37/238) underwent total mastectomy. Axillary surgery was not performed in 21 patients (8.8%), whereas 86.6% (206/238) of them underwent sentinel lymph node biopsy and 4.6% (11/238) underwent axillary lymph node dissection. Among the 253 cancers analyzed, the mean invasive tumor size was 0.9 ± 0.8 cm. Following surgical resection, invasive ductal carcinoma was the most common histopathological type (65.6%, 166/253), followed by ductal carcinoma in situ (22.1%, 56/253) and invasive lobular carcinoma (9.5%, 24/253). The details are summarized in Table 1.

Table 1. Patient and lesion characteristics of artificial intelligence-detected and artificial intelligence-undetected cancers.

Characteristics			Total	AI-undetected	AI-detected	p-value
Patient information (n = 238)
	Age (yr)		50 [45–57]	50 [46–56]	50 [45–57]	0.979
	Breast density					0.259
		Almost entirely fatty	1 (0.4)	0 (0)	1 (1.6)
		Scattered fibroglandular density	17 (7.1)	11 (6.3)	6 (9.5)
		Heterogeneously dense	141 (59.2)	103 (58.9)	38 (60.3)
		Extremely dense	79 (33.2)	61 (34.9)	18 (28.6)
Surgery (n = 238)
	Breast surgery					0.374
		Breast conserving surgery	201 (84.5)	150 (85.7)	51 (81.0)
		Total mastectomy	37 (15.5)	25 (14.3)	12 (19.0)
	Axillary surgery					0.331
		No	21 (8.8)	18 (10.3)	3 (4.8)
		Sentinel lymph node biopsy	206 (86.6)	150 (85.7)	56 (88.9)
		Axillary lymph node dissection	11 (4.6)	7 (4.0)	4 (6.3)
Histopathology results (n = 253)
	Invasive tumor size (cm)		0.9 ± 0.8	0.8 ± 0.7	1.3 ± 1.0	< 0.001
	Histopathologic type					0.197
		IDC	166 (65.6)	116 (62.0)	50 (75.8)
		DCIS	56 (22.1)	47 (25.1)	9 (13.6)
		ILC	24 (9.5)	19 (10.2)	5 (7.6)
		Etc.	7 (2.8)	5 (2.7)	2 (3.0)
	Nuclear grade					0.157
		Grade 1	23 (11.7)	18 (12.9)	5 (8.8)
		Grade 2	140 (71.4)	102 (73.4)	38 (66.7)
		Grade 3	33 (16.8)	19 (13.7)	14 (24.6)
		N/A	57	48	9
	Histologic grade					0.107
		Grade 1	62 (31.6)	45 (32.4)	17 (29.8)
		Grade 2	111 (56.6)	82 (59.0)	29 (50.9)
		Grade 3	23 (11.7)	12 (8.6)	11 (19.3)
		N/A	57	48	9
	Ki-67		4.0 ± 8.2	3.8 ± 8.1	4.4 ± 8.4	0.589
	Molecular subtype					0.214
		Luminal	223 (88.1)	161 (86.1)	62 (93.9)
		HER2	11 (4.3)	9 (4.8)	2 (3.0)
		Triple negative	19 (7.5)	17 (9.1)	2 (3.0)
	Lymphovascular invasion					0.011
		No	239 (94.5)	181 (96.8)	58 (87.9)
		Yes	14 (5.5)	6 (3.2)	8 (12.1)
	Axillary lymph node metastasis					0.050
		No	236 (93.3)	178 (95.2)	58 (87.9)
		Yes	17 (6.7)	9 (4.8)	8 (12.1)

Values are presented as median [interquartile range], mean ± standard deviation or number (%).

AI = artificial intelligence; IDC = invasive ductal carcinoma; DCIS = ductal carcinoma in situ; ILC = invasive lobular carcinoma; Etc., mucinous carcinoma (n = 5), mixed invasive ductal and lobular carcinoma (n = 2); N/A = not available; HER2 = human epidermal growth factor receptor 2.

The AI software detected 66 cancers in 63 patients (26.1%), whereas 187 cancers in 175 patients (73.9%) remained undetected. In contrast, it detected 19 false-positive lesions (11.9%) in 160 women without cancer. Upon performing univariate analysis, invasive cancer size was observed to be associated with cancer detection by the AI software (odds ratio [OR], 2.4; 95% confidence interval [CI], 1.6–3.5; p < 0.001), indicating that larger cancers were more likely to be detected by the AI software than smaller cancers. Furthermore, cancers with lymphovascular invasion (OR, 4.2; 95% CI, 1.4–12.5; p = 0.011) and positive axillary lymph node metastasis (OR, 2.7; 95% CI, 1.006–7.4; p = 0.049) were also more likely to be detected by the AI software than cancers without lymphovascular invasion or axillary lymph node metastasis. The other pathological and demographic characteristics were not associated with cancer detection. After multivariate analysis, tumor size was the only factor significantly associated with cancer detection (OR, 2.2; 95% CI, 1.5–3.3; p < 0.001) (Table 2).

Table 2. Factors associated with cancer detection by artificial intelligence software.

Variables			Univariate			Multivariate
			OR	95% CI	p-value	OR	95% CI	p-value
Patient information (n = 238)
	Age		1	0.97–1.03	0.979
	Breast density				0.745
		Almost entirely fatty
		Scattered fibroglandular density	1	1
		Heterogeneously dense	1	1
		Extremely dense	1	1
Surgery (n = 238)
	Breast surgery				0.373
		Breast conserving surgery
		Total mastectomy	1.4	0.7–3.0
	Axillary surgery				0.348
		No
		Sentinel lymph node biopsy	2.2	0.6–7.9	0.210
		Axillary lymph node dissection	3.4	0.6–19.4	0.163
Histopathology results (n = 253)
	Invasive tumor size (cm)		2.4	1.6–3.5	< 0.001	2.2	1.5–3.3	< 0.001
	Histopathologic type				0.208
		IDC
		DCIS	0.4	0.2–0.98	0.043
		ILC	0.6	0.2–1.7	0.352
		Etc.	0.9	0.2–4.9	0.930
	Nuclear grade				0.165
		Grade 1
		Grade 2	1.3	0.5–3.9	0.587
		Grade 3	2.7	0.8–8.9	0.113
		N/A
	Histologic grade				0.119
		Grade 1
		Grade 2	0.9	0.5–1.9	0.854
		Grade 3	2.4	0.9–6.5	0.079
		N/A
	Ki-67		1.01	0.98–1.04	0.527
	Molecular subtype				0.244
		Luminal
		HER2	0.6	0.1–2.7	0.490
		Triple negative	0.3	0.7–1.4	0.120
	Lymphovascular invasion
		No
		Yes	4.2	1.4–12.5	0.011
	Axillary lymph node metastasis
		No
		Yes	2.7	1.006–7.4	0.049

OR = odds ratio; CI = confidence interval; IDC = invasive ductal carcinoma; DCIS = ductal carcinoma in situ; ILC = invasive lobular carcinoma; Etc. = mucinous carcinoma (n = 5), mixed invasive ductal and lobular carcinoma (n = 2); N/A = not available; HER2 = human epidermal growth factor receptor 2.

We retrospectively reviewed 66 cancers that were initially missed on screening mammography but were detected using AI software. Among them, 42 tumors were accurately marked by the AI software but were not clearly visible to the human eye due to the overlapping dense breast tissue (Figure 1). The remaining 24 cases were identified as interpretive errors by the radiologists. These cases included masses in 12 (50.0%), focal asymmetries in 8 (33.3%), grouped calcifications in 3 (12.5%) (Figure 2), and architectural distortion in 1 (4.2%) patient.

Figure 1. Images of a 50-year-old patient with invasive ductal carcinoma of the right upper outer breast. (A) Screening digital mammography reveals extremely dense breasts without suspicious findings in the right breast. (B) Supplemental screening ultrasound image reveals an irregular mass with angular margins and posterior shadowing in the right upper outer breast. Color Doppler image reveals vascularity adjacent to the mass (arrow). (C) The AI software provides an abnormality score of 40% on both craniocaudal and mediolateral oblique views of the right breast. (D) The patient underwent ultrasound-guided wire localization before lumpectomy, targeting the ultrasound-detected mass. Subsequent digital mammography confirms that the wire is located at the same site as the area previously marked by the AI software (arrowheads). A skin marker is then applied to the wire entry site. AI = artificial intelligence.

Figure 2. Images of a 56-year-old patient with invasive ductal carcinoma of the right upper outer breast. (A) Screening digital mammography reveals heterogeneously dense breasts with grouped calcifications in the right upper outer breast (arrows). The calcifications were so subtle that they were overlooked by the radiologists during initial interpretation. (B) Supplemental ultrasound screening image reveals an irregular mass in the right upper outer lobe (arrowheads). (C) Grouped amorphous calcifications are visualized with 200% zoom (open arrow). (D) The patient underwent ultrasound-guided wire localization targeting an ultrasound-detected mass prior to lumpectomy. Subsequent digital mammography confirms that the wire is located at the same site as the grouped calcifications. A skin marker is then applied at the wire entry site. (E) The retrospective application of the artificial intelligence software provides an abnormality score of 66% at the site of calcification in both craniocaudal and mediolateral oblique views of the right breast.

DISCUSSION

In our analysis of 253 supplemental screening ultrasound-detected cancers, the AI software failed to detect 187 cancers (73.9%, 187/253) in 175 patients (73.5%, 175/238). However, the AI software correctly detected 66 cancers (26.1%, 66/253) in 63 patients (26.5%, 63/238). The AI software proved helpful in detecting 42 cancers (16.6%, 42/253) that were challenging for the radiologists to identify. Interestingly, although the AI software made some false-positive marks in patients without cancer, it demonstrated the potential to prevent interpretive errors and reduce the risk of missed cancer in 24 cases (9.5%, 24/253). Invasive tumor size was the most important factor associated with cancer detection by the AI software (OR, 2.4; 95% CI, 1.6–3.5; p < 0.001). Lymphovascular invasion (OR, 4.2; 95% CI, 1.2–12.5; p = 0.011) and axillary lymph node metastasis (OR, 2.7; 95% CI, 1.006–7.4; p = 0.049) demonstrated association with cancer detection on univariate analysis but not on multivariate analysis.

Our findings are consistent with those of a previous study in which AI software additionally identified 31% of occult breast cancers and contributed to a 2.1% increase in the cancer detection rate [18]. Another study demonstrated that among the 204 missed cancers on mammography, 67% were true negatives; however, 33% of them could also be detected with the use of AI-computer-aided diagnosis [19]. Calcification, focal asymmetry, and asymmetry [19] are common findings in patients with missed cancer. Kim et al. [18] reported that AI-detected cancers are frequently associated with advanced-stage cancers exhibiting positive axillary lymph node metastases, and these patients are more likely to undergo intensive treatment or wider surgical resections than those with AI-undetected cancers.

Screening mammography can miss detecting 10%–30% of breast cancers [20], and breast density poses one of the major obstacles to cancer detection as it can mask lesions [21,22,23]. In our study, 92.4% of the patients had dense breasts (220/238), making supplemental ultrasound screening crucial for these patients. Additionally, when we retrospectively reviewed 66 AI-detected cancers, 42 (63.6%) had indiscernible findings obscured by dense breast tissue, whereas the remaining 24 were considered interpretative errors. Previous studies have shown that fatigue [24,25], hours awake and asleep [26], and annual and cumulative reading volumes [27] affect radiologists’ reading performance. As the interpretation process involves search, perception, and decision-making [28,29], faults in any of these processes can result in interpretative errors [30]. Since the AI software is free from limitations related to reader fatigue or work overload, the occurrence of reading errors caused by these factors can be reduced.

This study had several limitations. First, this was a retrospective study conducted at a single tertiary hospital with a relatively small sample size. The classification of “ultrasound-detected cancers” can be subjective and variable based on radiologists’ experience and skills in interpreting mammograms. Second, we focused only on the diagnostic performance of commercially available AI software provided by one vendor. Thus, the results of this study are confined, although our findings are consistent with those in published literature. Data obtained using different AI software programs are required to obtain more generalized results. Finally, post-biopsy mammograms were included in the study. Although we excluded mammograms with definite biopsy or scar changes, we did include mammograms without biopsy-related changes. It is essential to consider that changes resulting from the biopsy procedure itself could potentially impact the performance of the AI software.

In conclusion, the AI software was able to detect 26% of ultrasound-detected cancers, particularly those with a large size. AI software has the potential to assist in identifying indiscernible or overlooked breast cancers, thereby preventing interpretive errors and reducing the risk of missed cancers.

SUPPLEMENTARY MATERIALS

Supplementary Data 1

Image acquisition

Supplementary Figure 1

Study flowchart