Analysis of Specimen Mammography with Artificial Intelligence to Predict Margin Status

Abstract

Background:

Intra-operative specimen mammography is a valuable tool in breast cancer surgery, providing immediate assessment of margins for a resected tumor. However, the accuracy of specimen mammography in detecting microscopic margin positivity is low. We sought to develop an artificial intelligence model to predict the pathologic margin status of resected breast tumors using specimen mammography.

Methods:

A dataset of specimen mammography images matched with pathologic margin status was collected from our institution from 2017-2020. The dataset was randomly split into training, validation, and test sets. Specimen mammography models pre-trained on radiologic images were developed and compared with models pre-trained on non-medical images. Model performance was assessed using sensitivity, specificity, and area under the receiver operating characteristic curve (AUROC).

Results:

The dataset included 821 images and 53% had positive margins. For three out of four model architectures tested, models pre-trained on radiologic images outperformed non-medical models. The highest performing model, InceptionV3, showed a sensitivity of 84%, a specificity of 42%, and AUROC of 0.71. Model performance was better among patients with invasive cancers, less dense breasts, and non-White race.

Conclusions:

This study developed and internally validated artificial intelligence models which predict pathologic margins status for partial mastectomy from specimen mammograms. The models’ accuracy compares favorably with the published literature on surgeon and radiologist interpretation of specimen mammography. With further development, these models could more precisely guide the extent of resection, potentially improving cosmesis and reducing re-operations.

Introduction

Breast-conserving surgery with radiation is often the preferred approach for early-stage breast cancer, balancing oncologic resection and cosmesis.¹ In this approach, obtaining negative margins is critical for reducing local recurrence. However, reported re-operation rates for positive margins remain high, at about 15%.^2-6 Multiple interventions have attempted to identify positive margins intra-operatively, including specimen mammography.

Specimen mammography is a widely used technique that provides immediate feedback on the quality of resection and may assist surgeons with identifying suspicious margins and directing targeted removal of additional tissue.⁷ However, specimen mammography can be inaccurate, with sensitivity ranging from 20% to 58% and area under the receiver operating characteristic curve (AUROC) of ~0.7.^8-11 A tool to assist clinicians with interpretation of specimen mammography for partial mastectomy would be beneficial for improving identification of positive margins and reducing rates of re-operation. This tool would be particularly useful for low-volume and low-resource centers, which see higher rates of positive margins.¹²

In addition, in part because of the limitations of specimen mammography and other intra-operative margin assessment techniques, universal use of cavity shave margins has been adopted as a highly effective means of reducing positive margins.¹³ However, this technique does result in a higher volume of tissue resected and higher costs for pathology.^14,15 An automated method of assessing the primary specimen could assist surgeons with selective use of cavity shave margins.

Deep learning, commonly referred to as artificial intelligence (AI), is an emerging field that has enabled computational interpretation of medical images.¹⁶ It has been extensively applied to screening mammography, with a recent systematic review identifying 82 relevant studies describing high accuracy for identifying breast cancer.¹⁷ Deep learning also shows promise for intra-operative use, with recent applications to the interpretation of laparoscopic video.^18-20 However, deep learning has not yet been applied to intra-operative specimen mammography.

The goal of this study is to develop a computer vision model that can predict margin status for partial mastectomy based on specimen mammography alone. We hypothesized that this model could demonstrate clinically actionable accuracy and outperform human accuracy in predicting pathologic margin status.

Methods

Dataset

Prior to data collection, approval from the University of North Carolina Institutional Review Board (#20-1820) was obtained. We included all consecutive patients undergoing partial mastectomy for breast cancer, including invasive lobular carcinoma, invasive ductal carcinoma, and ductal carcinoma in situ (DCIS), for whom specimen mammography and pathologic margin status were available. Data was collected in two phases. First, from 7/2017 to 6/2020, specimen mammograms were prospectively collected as part of a single surgeon’s quality assurance process. Second, from 8/2020 to 4/2022, specimen mammograms from four surgeons were collected specifically for this project. In the second phase of data collection, chart review of pre-operative mammography and pathology records was used to obtain additional clinical information including age, race/ethnicity, BMI, breast density,²¹ tumor type, and tumor grade.

A single 2D, anterior-posterior image was selected for each specimen. All images were obtained using a Faxitron^® Trident^® HD Specimen Radiography System (Hologic, Marlborough, Massachusetts), intra-operatively, immediately after resection of the specimen. Our institution’s process for margin processing/analysis is to obtain a specimen mammogram, which is interpreted intra-operatively by a breast radiologist, routinely obtain cavity shave margins, and submit the primary specimen and margins for formal pathology.

Primary outcome

Using retrospective chart review, specimen mammograms were matched with pathology reports and categorized into positive and negative classes based on National Comprehensive Cancer Network (NCCN) guidelines.²² For specimens with invasive cancer, a positive margin was defined as “ink on tumor.” For specimens with DCIS, a positive margin was defined as DCIS within 2mm of the margin. For specimens containing both DCIS and invasive cancer, but with DCIS within 2mm of the margin or mixed pathology, we elected to categorize this pathology as positive, acknowledging that, clinically, this result is treated as negative. We used this approach for model training to maximize the sensitivity of the model for DCIS margins and avoid training the model to ignore DCIS in specimens with invasive cancer. In addition, we used specimen mammograms and pathologic margin status of the main specimen, excluding cavity shave margins, which were routinely used, but not routinely imaged.

Data processing

The dataset was divided randomly into training, validation, and test sets in a 60/20/20 ratio. A single anterior-posterior image was used for each specimen and resized to 512 x 512 pixels. The training set underwent standard data augmentation including random flipping (vertically and horizontally), zoom, shifts (horizontal and vertical), and rotation.²³

Pre-training Datasets

We compared two different pre-training strategies. First, we used models from the RadImageNet project, which developed pre-trained models based on 1.35 million annotated medical images.²⁴ In contrast, prior medical computer vision projects have often relied on ImageNet, which is a large database of >14 million images. However, ImageNet includes largely images of non-medical objects, such as balloons or strawberries.²⁵ We compared the performance of “radiology-specific models,” pre-trained on RadImageNet, with “general models,” pre-trained on ImageNet.

Modeling

We developed models based on the architectures used by the RadImageNet project, including ResNet-50, InceptionV3, Inception-ResNet-v2, and DenseNet-121.²⁴ These are different types of convolutional neural networks, a type of AI model specifically used for computer vision. To this end, we implemented eight models: one of each base architecture pre-trained on RadImageNet and one of each pre-trained on ImageNet. After the highest performing model type and pre-training strategy was identified, the model was further optimized. Additional details on model development are available in the GitHub repository referenced at the end of this section. A diagram of the model is shown in Figure 1a.

Figure 1.

A - Diagram of model structure, B - Flowchart of study design

Model evaluation

Model performance for predicting pathologic margin status was assessed primarily using area under the receiver operating curve (AUROC). This is a classification metric that assesses a model’s ability to distinguish positive from negative cases and ranges from 0.5 at the worst, to 1 at the best. For the highest performing model (based on AUROC), evaluation metrics were calculated for categories within tumor type, breast density, and race/ethnicity because screening mammography has previously been shown to have different accuracy within these groups.^26,27 For this analysis, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and area under the precision-recall curve (AUPRC) were also calculated. In addition, Grad-CAM was used to create saliency maps to assess pixel importance for 10 randomly selected images from the test set and these were visually assessed.²⁸ Figure 1b shows the overall workflow for model development and evaluation.

Models were implemented and evaluated using the Python (version 3.8) libraries scikit-learn and Tensorflow/Keras.^29,30 An NVIDIA RTX A4500 graphics card (NVIDIA, Santa Clara, CA) was used for model training and validation. For characterization of the cohort, Chi-squared test was used to compare categorical variables and T-test to compare continuous variables, using the tableone package.³¹ Code to reproduce this work and additional details on methods are available at github.com/gomezlab/cvsm.

Results

The dataset included 806 images. Of these, 450 were collected in the first, single surgeon phase, while 356 were collected in the second, multiple surgeon phase. 431 (52.5%) images had positive margins in the main specimen without considering cavity shave margins. Representative images for each classification are shown in Figure 3. Within the positive margin classification, 128 had mixed pathology with both invasive cancer and DCIS 2mm from the margin. The average age was 60 and most patients had IDC (70.0%) or DCIS (21.3%). Non-Hispanic White patients comprised 70.1% of the cohort, compared with 18.1% Non-Hispanic Black and 7.8% Hispanic. Most patients had a breast density of B (scattered fibroglandular densities) (45.8%) or C (heterogeneously dense) (36.9%). Patients with infiltrating ductal carcinoma who had a tumor grade of 2 were more likely to have positive margins (Table 1). After splitting into training, validation, and testing groups, 485 images were used for training, 160 for validation, and 161 for testing.

Figure 3.

Receiver operating characteristic and precision-recall curves for models predicting margin status

Table 1.

Demographic and clinical characteristics

			Overall	Margin (−)	Margin (+)	P-Value
n			357	139	218
Age, mean (SD)			60.5 (13.1)	59.2 (12.9)	61.4 (13.2)	0.109
Race/Ethnicity, n (%)		Asian	11 (3.1)	6 (4.3)	5 (2.3)	0.082
		Hispanic	27 (7.6)	7 (5.0)	20 (9.2)
		Non-Hispanic Black	63 (17.6)	22 (15.8)	41 (18.8)
		Non-Hispanic White	253 (70.9)	101 (72.7)	152 (69.7)
		Other/Unknown	3 (0.8)	3 (2.2)
Density, n (%)		A	17 (4.8)	6 (4.3)	11 (5.0)	0.651
		B	164 (45.9)	59 (42.4)	105 (48.2)
		C	130 (36.4)	56 (40.3)	74 (33.9)
		D	46 (12.9)	18 (12.9)	28 (12.8)
Tumor Type, n (%)		DCIS	76 (21.3)	30 (21.6)	46 (21.1)	0.925
		IDC	250 (70.0)	96 (69.1)	154 (70.6)
		ILC	31 (8.7)	13 (9.4)	18 (8.3)
Grade, n (%)	DCIS, n (%)	1	8 (11.9)	5 (17.9)	3 (7.7)	0.399
		2	31 (46.3)	13 (46.4)	18 (46.2)
		3	28 (41.8)	10 (35.7)	18 (46.2)
	IDC, n (%)	1	69 (28.6)	25 (27.2)	44 (29.5)	<0.001
		2	99 (41.1)	26 (28.3)	73 (49.0)
		3	73 (30.3)	41 (44.6)	32 (21.5)
	ILC, n (%)	1	7 (22.6)	3 (23.1)	4 (22.2)	0.482
		2	23 (74.2)	9 (69.2)	14 (77.8)
		3	1 (3.2)	1 (7.7)	0 (0.0)

BMI – Body mass index, DCIS – Ductal carcinoma in situ, IDC – Invasive ductal carcinoma, ILC – invasive lobular carcinoma

We first compared the performance of models pre-trained on radiologic images (RadImageNet), compared with models pre-trained on general images (ImageNet). Overall, radiology-specific models showed higher accuracy compared with general models with AUROC 0.63 to 0.71 vs. 0.46 – 0.68, respectively. Each of the model types tested showed higher performance with radiology-specific pre-training, except for one (DenseNet121), and performance was similar between the two pre-training strategies in this case (AUROC 0.66 vs 0.68). The highest performing model was InceptionV3 with radiology-specific pre-training. Receiver operating characteristic and precision-recall curves are shown for all models in Figure 4.

Figure 4.

Analysis of pixel importance for radiology-specific and general models. Red/yellow indicates higher importance

Based on this analysis, InceptionV3 with radiology-specific pre-training was further optimized. This model showed an AUROC of 0.71, AUPRC of 0.73, sensitivity of 85%, specificity of 45%, positive predictive value of 62%, and negative predictive value of 70%. In subset analysis, we found that model performance was worse for DCIS (AUROC 0.65) compared with invasive ductal carcinoma (AUROC 0.71) and invasive lobular carcinoma (AUROC 0.75). We also found that model performance was worse for extremely dense breast tissue (category D) compared with less dense breast tissue (Table 2). For race/ethnicity, model performance was worse for Non-Hispanic White patients compared with non-White patients, although this difference appeared to be partially driven by breast density, with 14% of Non-Hispanic White patients having category D breast density, compared with 9% of non-White patients (Table 3).

Table 2.

Model accuracy metrics by breast tissue density category

Density Category	AUROC	AUPRC	Sensitivity	Specificity	PPV	NPV
A	0.758	0.877	0.818	0.333	0.692	0.500
B	0.682	0.773	0.875	0.339	0.700	0.606
C	0.756	0.787	0.865	0.357	0.640	0.667
D	0.542	0.614	0.786	0.278	0.629	0.455

Add breast density categories. AUROC – area under the receiver operating characteristic curve, AUPRC – area under the precision-recall curves, PPV – positive predictive value, NPV – negative predictive value

Table 3.

Model accuracy metrics by race/ethnicity category

Race/Ethnicity Category	AUROC	AUPRC	Sensitivity	Specificity	PPV	NPV
Asian	0.800	0.832	0.800	0.333	0.500	0.667
Hispanic	0.793	0.892	0.900	0.714	0.900	0.714
Non-Hispanic Black	0.737	0.845	0.829	0.364	0.708	0.533
Non-Hispanic White	0.667	0.728	0.861	0.307	0.65	0.596

AUROC – area under the receiver operating characteristic curve, AUPRC – area under the precision-recall curves, PPV – positive predictive value, NPV – negative predictive value

To improve model interpretability, or understanding what parts of the image contributed the most to model decision-making, we assessed where the attention of the model was focused. In most cases, models pre-trained on radiologic images had attention focused on relevant parts of the image, such as localization wires, biopsy clips, and areas of tumor, while models pre-trained on non-medical images did not (Figure 5).

Discussion

This project developed an artificial intelligence model that predicts the pathologic margin status of partial mastectomy specimens based on specimen mammograms. Pre-training with radiology images was found to improve model predictions compared with pre-training with non-medical images. With an internal test set, the model showed a sensitivity of 85%, a specificity of 45%, and AUROC of 0.71. Analysis of pixel importance suggested that model attention was focused on relevant portions of the image.

Despite advances in intra-operative margin assessment, the rate of positive margins after partial mastectomy remains high.³ In fact, due to limitations in visual, tactile, and radiographic intraoperative assessment of margins, cavity shave margins have been widely adopted to reduce the rate of positive margins.^13,14 Still, specimen mammography is a widely used method that benefits from its availability within the operating room and its ability to provide immediate feedback. Because of this, specimen imaging for non-palpable lesions is recommended by the Collaborative Attempt to Lower Lumpectomy Reoperation rates (CALLER) toolbox.³² However, the diagnostic accuracy of specimen mammography is highly variable and relatively low, with reported sensitivity ranging from 20-58%.^8-11 A meta-analysis of nine studies showed a pooled sensitivity of 53% (95% CI 45 – 61%) with a pooled specificity of 84% (95% CI 77 - 89%).³³ AUROC is similarly variable, but low, ranging from 0.60 to 0.73.^33-37 Our models’ accuracy metrics are higher than many previous results published in the literature, demonstrating the potential of this approach.

Another interesting finding from our study was that our model showed lower accuracy in predicting margin status among patients with the highest breast density (category D). This is expected, given similar findings for screening mammography, but has not been previously reported for specimen mammography.²⁷ In contrast to previous studies applying artificial intelligence (AI) to mammography, our models show higher accuracy among non-White patients compared with White patients.²⁶ This difference may be because non-White patients are well-represented within our dataset and because of the higher percentage of category D breast density among White patients in this study. In addition, our study agrees with previous literature showing that DCIS margins are particularly difficult to assess using specimen mammography.^7,38 Use of a larger training set focused on DCIS alone may be necessary to overcome this limitation.

More generally, the overall performance of our models agrees with other recent advances in computer vision classification of mammograms that show improved accuracy when radiologists are assisted by AI.^39-41 Recently, AI has also been successfully applied to laparoscopic video, demonstrating its potential to assist with real-time, intra-operative decision-making.^42-44 AI-assisted interpretation of specimen mammography may function similarly., Low-volume or low-resource centers that lack access to dedicated breast surgeons or radiologists have the highest rates of positive margins and may benefit most from AI-assistance, raising the potential for these systems to improve equity in surgical outcomes.^12,45 AI-assisted clinical decision-support systems would be most useful to these clinicians and thus have the potential to improve equity in surgical outcomes for their patients.⁴⁶

There are two clinical scenarios that would specifically benefit from our models. First, if a positive margin was predicted, this could guide additional resection. Alternatively, identifying negative margins on the primary specimen could allow precise patient-specific omission of cavity shave margins which could decrease the overall volume of resection, improve cosmesis, and decrease costs related to pathology.¹⁵ Because of cavity shave margins, we were not able to develop a model for margin positivity based on the final margin, so we focused on the second scenario, confidently identifying negative margins. To maximize the negative predictive value, we trained the model on the widest margin guideline, DCIS. We chose this approach to maximize the negative predictive value, and because false positives in this setting are less problematic as the patient would simply receive routine cavity shave margins. The threshold of NPV for clinical use is an open question and dependent on individual surgeon judgment. However, further development of these models is warranted, including collection of a larger dataset to ensure reliability across different imaging hardware and institutions and mammography-specific transfer learning to improve model accuracy.

This project has several limitations. First, the accuracy and generalizability of the model is most significantly limited by the size and single-institution nature of the dataset. A larger, multi-institutional dataset would likely result in a more accurate and robust model and could verify its external validity. Second, the rate of positive margins is higher than previously reported because of our exclusion of cavity shave margins and use of DCIS margins in mixed invasive cancer/DCIS pathology.³ This approach maximizes the sensitivity of the model for DCIS margins, reduces false negatives, and results in a single model which can be used for all cases. Third, our model does not identify which side of the specimen may have a positive margin. Leveraging model attention techniques to automatically localize image features associated with a positive margin is possible and represents a direction for future research.^47,48 Finally, we did not assess radiologist or surgeon accuracy for margin prediction on our dataset, although this is likely to be similar to previous literature.

More recent imaging techniques, such as 3D tomosynthesis or optical coherence tomography, may also be more accurate compared with 2D specimen mammography and result in improved models.^49-51 In addition, trials of new technologies for margin assessment such as fluorescence and spectroscopy have shown promising results .^52-57 However, these devices require purchase of costly hardware and carry the risk of allergic reaction. In contrast, the current approach has the advantage of using widely available systems.

Conclusion

In conclusion, we developed a prototype model that predicts the pathologic margin status of partial mastectomy specimens based on intra-operative specimen mammography. The model’s predictions compare highly favorably with human interpretation. Optimized and externally validated versions of this model could assist surgeons with predicting margin status intra-operatively, guide selective use of cavity shave margins, and ultimately reduce the need for re-operation in breast-conserving surgery.

Figure 2.

Example specimen mammograms by margin status classification

Synopsis.

Specimen mammography can assess specimen adequacy following partial mastectomy for breast cancer. This study developed and internally validated an artificial intelligence model to predict pathologic margin status from specimen mammograms, with comparable accuracy to surgeons and radiologists.