A Multi-Task Learning Approach for Segmentation of Breast Arterial Calcifications in Screening Mammograms

Abstract

Screening mammogram is a standard and cost-efficient imaging procedure to measure breast cancer risk among 45+ year old women. Quantifying breast arterial calcification (BAC) from screening mammograms is a non-invasive and cost-efficient approach to assess the future risk of adverse cardiovascular events among women, such as heart attack and stroke. However, segmentation of breast arterial calcification is an involved task and poses several technical challenges such as extremely small BAC finding, low breast arteries to breast area ratio in the mammogram images, tissue features such as breast folds and heterogeneous density, have very similar imaging appearance. In this work, we aim to address the shortcomings of existing SOTA methods, e.g., SCUNet, and analyze the comparative performance. Given the fact that we will not be able to simply resize mammogram to preserve the microscopic BAC details, we adopted a patch-based methodology for segmentation using the original resolution which may hinder the model understanding of whole mammogram. We propose a multi-task learning approach for patch-based BAC segmentation by adding an auxiliary task of patch position prediction which forces the model to learn breast anatomy to comprehend the locations where BAC will not occur, such as breast boundary. The proposed method achieves state-of-the-art performance compared to the baselines. To demonstrate the utility, we also validate our method on external data and provide survival analysis for adverse cardiac events based on difference in BAC score and provide a comparison with coronary calcium score (CAC).

Introduction

Cardiovascular disease (CVD) is the leading cause of death for women in the United States, responsible for 1 in 5 deaths annually – greater than all cancer subtypes¹. Understanding and overcoming gaps in knowledge to improve early diagnosis of cardiovascular disease is the first step toward improving the cardiovascular health of women. Arterial calcification is readily visualized on routine screening mammography, which is ideally suited to image vascular calcifications because of its high resolution and contrast between arteries and surrounding breast tissue. As shown in the existing literature, the prevalence of breast arterial calcification (BAC) correlates with cardiovascular disease^2–4. Annual mammograms are currently recommended for women aged 40 until the 7th decade of life. Mammography screening rates in women 40 years old and older are high (67.5% in 2019) ^5,6. Likewise, mammography rates are high in racial and ethnic minorities (e.g., 70.8% non-Hispanic Blacks) ⁷. Thus, re-purposing mammography images for use in prediction of diseases other than breast cancer could be clinically impactful given the high percentage of women with images available in the general population.

To date, assessment of BAC from mammograms has been manual which requires extensive expert engagement for outlining BAC from images; therefore limits its application for the regular screening exams and restricting the research studies only to small populations ^8,9. Evaluation of BAC effectiveness to a large population requires an automated process that, to be feasible for prospective clinical applications, must be sufficiently rapid to provide information in near real-time. Previous work ^10–12 demonstrates the ability for automated BAC segmentation, but these works either focus only on prevalence, require raw scanner data, or perform segmentation without focusing on clinical outcomes ¹³. Existing work primarily focused on detecting BAC presence and none of these works provided automated quantification of BAC, which is important for translating BAC as a robust clinical and research tool.

Segmentation and quantification of BAC is a technically challenging task because of - (i) variation of breast tissue appearance based on tissue density; (ii) microscopic BAC presence and complex structural representation (similar to branched tree) in the large mammogram image makes it challenging to extract – even by expert annotator; and (iii) the extreme class imbalance between “True BAC” pixels and “No BAC” pixels. We propose a multi-task learning ¹⁴ patch-based architecture for accurate BAC segmentation from high-resolution images and automated quantification of BAC area. We not only validated the model on the external dataset that is significantly different in racial distribution (internal data: 48% white, 52% black; external data: 92% white, 8% Asian), but we also successfully correlated it with proportional hazards of future CVD diseases. We further performed a thorough ablation study to justify the relevance of each component in the proposed architecture. We also examined the association of BAC with coronary calcium score (CAC).

Methodology

Given a high-resolution mammogram image (3328 × 2560), our goal is to detect and segment calcifications in the breast arteries. This is a challenging task because of the extreme class imbalance between “true BAC” pixels and “No BAC” pixels. Figure 1 shows our proposed model architecture. Given a mammogram image I ∈ R^W×H×C with width W, height H, and channels C, we generate equal sized image patches from the image I. Following Guo et al. ¹⁵, each image patch I_p is of the size 512 × 512. This patch-wise processing helps us to perform segmentation on the input image without losing minute details that often fade due to image resizing for managing high computational requirements.

Figure 1.

Our multi-task learning CNN-based architecture for BAC segmentation–pixel-wise label prediction and patch location detection.

Inspired by Guo et al. ¹⁵, we proposed a multi-task learning approach for patch-based BAC segmentation by adding an auxiliary task of patch position prediction which forces the model to learn breast anatomy to comprehend the locations where BAC will not occur, such as breast boundary. This strategy may ultimately help to reduce false positive rate for the BAC.

The proposed model is a UNet-based architecture and comprises an image encoder and two decoders. The first decoder, denoted by d_seg upsamples the encoded image feature x_e ∈ R^w×h×d to the input image size I_p and produces a single-channel binary segmentation mask for the input image; where w and h denote the width and height of the encoded spatial features, and d is the feature dimension. A second auxiliary decoder d_aux is added to decode the encoded image feature x_e into the patch position (see Figure 1). More specifically, this decoder predicts the image patch position in the original high-resolution image I.

One limitation of patch-based processing is the unawareness of the global spatial positioning. Given the fact that BAC appears only in the location of the arteries which are anatomically present within the middle layer of the breast arterial wall, learning the auxiliary task of patch position prediction parallel to the segmentation task should help the model to avoid generating false positives for image patches that do not usually contain BAC, e.g. breast tissue folds. For instance, breast tissues next to the chest wall are least expected to have BAC or in the outer quadrant. Thus, making the model position-aware should help in predicting better segmentation masks.

Our learning objective is, therefore, a weighted sum of the two loss terms given as:

$$L_{total}={\lambda }_1{L}_{seg}+{\lambda }_2{L}_{aux}$$ (1)

where, L_seg and L_aux are the cross-entropy loss functions for segmentation loss and the patch position loss, respectively; λ₁ and λ₂ are the regularization parameters and the values are decided empirically for these but could also be treated as hyperparameters and optimized on the validation set.

Results and Analysis

Datasets: With the approval of the Emory Institutional Review Board (IRB), we use 398 screening mammograms available with a 50:50 ratio from two groups: subjects with BAC, and subjects that do not have BAC. Heavy BAC in mammograms were annotated by three expert radiologists using md.ai platform ¹ and small BAC depositions were excluded from annotation due to time constraints. To have a standard train-val-test setup, we split these images into 80%-10%-10% ratio resulting in 318 training images, 40 validation images, and 40 test images. Finally, each image is divided into 512 × 512 sized patches with an overlap of 64 pixels between adjacent pairs. This yields 9643 training patches, 1218 validation patches, and 1241 patches for testing the models. Patches from a single image are only considered either in training, validation or test sets to preseve data leak.

For external validation of our trained model, we use mammograms for 7,264 subjects from Mayo Clinic BioBank which comprise of screening mammograms captured from three sites (Arizona, Florida and Rochester) with age 40-79 and no prior history of CVD. Note that mammogram images from Mayo Biobank were not annotated with BAC outlines.

Label generation: On the Emory dataset, radiologists mark heavy BAC in mammograms in the form of points (or vertices) and generated splines without noting the difference in BAC deposition. Predicting the exact coordinates for these splines in each mammogram is an extremely challenging task as there is a high inter-rater variability for identifying the exact coordinates. Therefore, we transform it into a segmentation task by generating segmentation masks for those annotations. We do so by applying a fixed width to those line segments. To find the best-suited width for the segmentation task, we generate BAC labels with 12 pixels, 16 pixels, and 20 pixels widths as shown in Figure 2. We find that the baseline (SCUNet) performs best with width=16 pixels. Hence, we use ground truth masks with 16 pixels for all our experiments.

Figure 2.

Segmentation masks generation with different widths.

Patch position generation: To generate patch position labels, each patch is assigned a unique position in a column-major layout. To formulate patch position prediction as a regression task, we consider the normalized top-left coordinates x/W, y/H as labels for each patch, where W is the width and H is the height of the full image.

Comparison with baselines: On the Emory test dataset, we consider two baseline methods: deeplab-v3-R50¹⁶ - a deep segmentation model, and SCUNet ¹⁵ - a lighter segmentation model designed for BAC. Evaluation metrics are recall, precision, F-score, Jaccard score, and dice score at patch level. Our final model is based upon SCUNet ¹⁵ with an added auxiliary decoding branch for patch position prediction. All models are trained on both “true BAC” patches and “No BAC” patches. We train the model in two stages unless specified: first, optimize the model for the segmentation task, and second, train the full model with both losses L_seg and L_aux. Training epochs for stages 1 and 2 are 50 epochs and 40 epochs respectively. Other training parameters are as follows: batch size= 32, lr = 1e − 4, λ₁ = 0.99, and λ₂ = 0.01. All experiments are performed on two 24GB NVIDIA Quadro RTX 6000 GPUs.

We find that our model is able to perform on par with the deeplab-v3R50 despite being ≈ 70× smaller in size (Table 1). When compared to SCUNet, we achieve a recall of 61.23% with an improvement of 6.3% ↑, and ≈ 3% ↑ improvement for F-score, Jaccard score, and the dice score respectively. False positive rate (FPR) for “true BAC” patches is also slightly improved for our model: 1.9% vs SCUNet: 2.13%. To compare our model with the baseline on “No BAC” subjects while making sure that none of the compared models have a significant disadvantage, we consider image patches from the exams with the detected regions’s area to be less than a certain threshold (connected region area ≤ 10,000). We obtain the false positive rate of 0.72% compared to 1.59% from SCUNet. Figure 3 presents qualitative examples of the patches stitched into the whole image. Our model is even able to detect narrow BAC for cases that were not properly marked by the radiologists due to time constraints. However, low dice and jaccard scores are obtained due to these incomplete annotations - see Figure 3, where the dice score is as low as 19.62 and 39.32 when being compared with the radiologists’ defined annotation, but interestingly the model is able to segment most of the BAC region, even when it was not annotated by the radiologists.

Table 1.

Performance comparison on the internal test set. Ours-MTL-Cls is the proposed multi-task model with patch position classification as an auxiliary task. Ours-MTL-Reg uses regression-based loss for the patch position branch and trained end-to-end.

Method	#params	Recall (%)	Precision (%)	Pixel Acc. (%)	F-score (%)	Jaccard (%)	Dice (%)
Deeplab-R50	47M	60.94	57.67	97.42	56.95	42.64	56.95
SCUNet	0.22M	55.10	57.25	98.63	54.50	39.66	54.50
Ours-MTL-Cls	0.57M	61.23	57.77	97.44	57.39	42.53	57.39
Ours-MTL-Reg	0.55M	60.40	56.90	98.64	56.99	42.12	56.99

Figure 3.

Qualitative examples from our model on Emory test set. ‘GT’ shows the radiologist-defined BAC outline and the predicted heatmap is a pixel-wise probability score computed by the model. The dice scores are: Example 1 (up): 19.62; and Example 2 (down): 39.32.

Relationship between BAC scores and survival time: As mentioned previously, radiologists’ defined BAC annotation only includes heavy depositions and ignores the narrow BAC due to time constraints (though such findings have similar imaging appearance only narrow thickness/partial deposition). Thus, validating the model’s performance only against the ground truth is not conclusive and may result in low dice and jaccard scores while the accuracy is high.

On the external Mayo Clinic dataset, instead of segmentation accuracy, we explore the correlation between BAC scores and actual adverse cardiac event occurrences (e.g. heart failure, Myocardial Infraction, stroke, cardiac death). The events are curated in Mayo Biobank using ICD9/10 diagnosis codes and manual chart review. We first normalize the BAC scores w.r.t the corresponding breast tissue area. The survival time is the duration between the screening date and the adverse events, i.e., 0 for no event, and 1 for CVD events. We then dichotomized subjects w.r.t their BAC scores into three groups: 1) subjects with BAC scores <= 0.001, termed as low BAC; 2) mid BAC– 0.001 < BAC <= 0.005; and 3) high BAC with BAC scores > 0.005. These groups were obtained for both left MLO view and right MLO view mammogram images separately. We observe a direct correlation between the BAC score and the survival probability. As shown in Figure 5, the probability of not developing CVD for subjects with low BAC in left and right is ,: 93% even after 15 years. On the contrary, the probability of not developing CVD events for patients with high BAC in the left breast drops to ≈ 68% and ≈ 75% for the right breast respectively in 15 years. When performing a t-test to compare survival curves of low BAC to mid BAC and lowBAC to high BAC in both breasts, we observe p < 0.005 which gives strong evidence that the BAC score has a major contribution to estimating the survival time of the patients without any CVD event. We observed significant overlap between mid and high BAC CVD estimation up to the first 10 years of the study, but a clear difference is observed for the long-term CVD estimation (10+ years).

Figure 5.

(a) Kaplan-Meier survival curves for subjects with CAC: survival curves for CVD events in patients with CAC scores (p − value <0.005), (b) Correlation between BAC scores for right (y-axis) and left (x-axis) breasts. N indicates the number of subjects in each group.

Association between BAC and CAC scores: Coronary artery calcium is measured from the expensive coronary CT imaging exam which highlights calcium deposits in the heart arteries and is predictive of future CVD events ¹⁷. Thus, we also investigated the association between BAC and CAC scores computed from 392 subjects who have CT scans within 1 year of mammogram. We first divide this cohort into two groups based on their CAC scores. In this population, CAC score is measured to be low (≤ 100) for 340 subjects, and CAC score > 100 is considered to be a moderate (medium plaque) or high (high plaque) CAC score. Each of these groups is further split into sub-groups based on BAC scores, where, BAC≤= 1e − 4 is used as low BAC and BAC> 1e − 4 is used as high BAC. This results in 4 sub-groups for each side of the breast i.e., left (L) and right (R). In Table 2, N denotes the number of subjects in each final subgroup. We report the correlation coefficient r between BAC scores and the calcium scoring for each subgroup in Table 2. We observe the expected trend that low CAC scores are correlated with low BAC scores, i.e., for low CAC (CAC <= 100 ), correlation is higher with low BAC cases (BAC<= 1e − 4: r=0.1687 (L), r=0.1000) whereas subjects with high CAC scores are more correlated to high BAC scores for both L and R sides ¹⁸. However, based on the overall smaller coefficient values, we do not observe a significantly strong association between BAC and CAC scores.

Table 2.

Correlation coefficients for CAC against BAC. CAC score is used to split population into two groups: 1) CAC <= 100 (low CAC scores), and 2) CAC > 100 (high CAC scores). Each of these groups are further divided into sub-groups based on the BAC presence in these subjects. Here, L and R denote CAC scores for the left breast and the right breast respectively. N denotes the number of samples in each subgroup.

Group	Sub-group	Variable	N	r
CAC <= 100	BAC ≤ 1e − 4	L	27	0.1687
		R	27	0.0974
	1e − <4 BAC <= 5e − 4	L	285	0.0523
		R	285	-0.0398
	BAC > 5e − 4	L	27	0.1475
		R	27	0.6837
CAC > 100	BAC ≤ 1e − 4	L	6	0.0641
		R	4	-0.0964
	BAC > 1e − 4	L	46	0.1276
		R	48	0.2120

Ablations and Analyses: We summarize the ablation results in Table 3.

Table 3.

Model ablations and hyperparameters search results for our model. Our base model is SCU-Net trained for segmentation task only. Cls denotes patch position classification, Reg denotes patch position regression.

Method	Recall (%)	Precision (%)	Pixel Acc. (%)	F-score (%)	Jaccard (%)	Dice (%)
base model	55.10	57.25	98.63	54.50	39.66	54.50
Patch Position Prediction
Ours-MTL-Cls	61.23	57.77	97.44	57.39	42.53	57.39
Ours-MTL-Reg	60.40	56.90	98.64	56.99	42.12	56.99
Loss Regularization Parameters
λ1=1.0, λ2=0.01	56.19	60.40	97.46	55.77	40.98	55.77
λ1=0.99, λ2=0.01	60.40	56.90	98.64	56.99	42.12	56.99
λ1=0.9, λ2=0.1	58.02	56.31	98.63	55.44	40.64	55.44
Training Protocol
e2e	53.99	58.66	98.43	53.99	39.19	53.99
two-stage	61.23	57.77	97.44	57.39	42.53	57.39
Patch Position Regression
(x, y)	60.40	56.90	98.64	56.99	42.12	56.99
patch no.	53.47	54.20	98.82	51.40	37.04	51.40
Label Width vs Model’s Performance
12px	51.30	48.55	97.82	48.00	33.53	48.00
16px	55.10	57.25	98.63	54.50	39.66	54.50
20px	46.20	66.95	97.98	51.21	37.10	51.20

Patch position prediction task: There are two possible ways to formulate the patch position prediction task: i) patch position classification–where the model is trained to predict the unique patch position within the grid of the corresponding image; ii) patch position regression to predict the top-left coordinates (x, y) of each patch. We obtain our best results with the patch position classification task (see section Patch Position Prediction in Table 3).

Loss regularization parameters: For balancing the auxiliary loss, we experimented with different values for λ₁ and λ₂. We attempted using equal weights (λ₁ = λ₂ = 1), as well as decreasing value of λ₁ ∈ {1., 0.1} (step size = 1/10) while increasing value of λ₂ ∈ {0.1, 0.9} such that λ₁+λ₂ = 1. We find no notable improvement in results, particularly in terms of F-score, jaccard score, and dice score. When experimenting with other possibilities, we empirically find that λ₁=0.99 and λ₂=0.01 helps in the model training. See rows 4, 5, and 6 in Table 3 for selected combinations of λ₁ and λ₂.

Training protocol: For the patch classification task, when training end-to-end for BAC segmentation and patch classification simultaneously, we obtained suboptimal performance than the base model without any auxiliary task (see e2e). Training the model in two stages yields the best results, i.e., two-stage training where the model for the segmentation task will start training first for a specific number of epochs followed by adding the patch position prediction task in the second stage. This suggests that the auxiliary task helps better when the model has the knowledge of BAC features.

Regression task: When formulating patch prediction as a regression task, we considered two possibilities: i) predicting patch x, y coordinates in the full image and ii) regressing the patch number in total number of image patches. We find that true patch coordinates regression always worked better compared to regressing the patch’s unique position (patch number) in the image. Results for both experiments are reported in rows 9 and 10 of Table 3.

Label width for ground-truth generation: The radiologists annotate a set of coordinates over the BAC area. Therefore, we have to generate segmentation masks from the spline coordinates provided by radiologists. We empirically find that using 16 pixels width for the segmentation mask yields the best results. Results for choosing optimal label width are obtained using base model as shown in the last section of Table 3.

Qualitative Results Figure 3 shows qualitative results on Emory internal hold-out test set. We can see that our model is able to detect BAC correctly in regions that were not annotated by radiologists. When evaluating quantitatively, this means the ground-truth annotations are inadequate to conclude the model’s actual performance. Figure 6 shows qualitative analysis on Mayo Clinic test set used for external validation. We show random cases of patients with “mid BAC” scores in the top row and “high BAC” scores in the bottom row respectively. We can see that the model is robust enough to perform well on external dataset and extract BAC from mammograms with various tissue densities.

Figure 6.

Qualitative results for external validation on Mayo BioBank dataset. Results are randomly selected for subjects with “mid BAC” scores and “high BAC” scores for varying tissue density.

Conclusion

This work presents a patch-wise multi-task learning method for BAC segmentation with an auxiliary task of patch position prediction. Our model is generalizable to cases without BAC deposition which contains the most of the screening population, and provide more accurate measures with less false positives than the baseline methods ¹⁵. We further performed external data validation of the model on screening mammograms from Mayo Clinic BioBank data. Though our study showed only a moderate relation between BAC and CAC due to presence of only a limited number of women with cardiac CT exams, we found a strong correlation between BAC scores of the patients and their adverse cardiac outcomes over a period of 15 years.

Figures & Tables

Figure 4.

Kaplan-Meier survival curves for CVD events for subjects with BAC scores (Low BAC) ≤ 0.001, 0.001 < (Mid BAC) BAC scores <= 0.005, and (High BAC) BAC scores ≥ 0.005. (a) survival curves for patients with BAC in the left breast, and (b) survival curves for patients with BAC in the right breast. N indicates the number of subjects in each group.