Enhancing Brain Metastases Detection and Segmentation in Black-Blood MRI Using Deep Learning and Segment Anything Model (SAM)

Abstract

Purpose

Black-blood (BB) magnetic resonance images (MRI) offer superior image contrast for the detection and segmentation of brain metastases (BMs). This study investigated the efficacy and accuracy of deep learning (DL) architectures and post-processing for BMs detection and segmentation with BB images.

Materials and Methods

The BB images of 50 patients were collect to train (40) and test (10) the DL model. To ensure consistency, we implemented piecewise linear histogram matching for intensity normalization and resampling. Modified U-Net, including combination with generative adversarial network (GAN), was applied to enhance the segmentation performance. The U-Net-based networks generated bounding boxes indicating regions of interest, which were then processed in a post-processing using the Segment Anything Model (SAM). We quantitatively assessed the three U-Net-based models and their post-processed counterparts in terms of lesion-wise sensitivity (LWS), patient-wise dice similarity coefficient (DSC), and average false-positive rate (FPR).

Results

The modified U-Net with GAN yielded a patient-wise DSC of 0.853 and a LWS of 89.19%, which outperformed the standard U-Net (patient-wise DSC of 0.815) and modified U-Net only (patient-wise DSC of 0.846). Combining GAN architecture with modified U-Net also reduced the FPR, less than 1 on average. Post-processing with SAM further did not affect LWS and FPR, but effectively enhanced the patient-wise DSC by 2%–3% for the U-Net-based models.

Conclusion

The modifications to standard U-Net notably improves the detection and segmentation of BMs in BB images, and applying SAM as post-processing can further enhance the precision of segmentation results.

Graphical Abstract

INTRODUCTION

Brain metastases (BMs) are the most common intracranial tumors and comprise over half of all clinically diagnosed brain tumors in adults.¹,² Palliative whole-brain radiotherapy (WBRT) has been previously recommended. However, with advances in systemic therapy, the treatment perspectives for BMs have evolved. Recently, the American Society of Clinical Oncology Society for Neuro-Oncology-American Society for Radiation Oncology (ASCO-SNO-ASTRO) published guidelines for BMs in solid tumors.³ The evolution of radiotherapy (RT) modalities has significantly altered treatment strategies regarding RT. Firstly, the decreased role of WBRT is presented. Secondly, stereotactic radiosurgery (SRS) is increasingly used to treat BM. SRS offers the advantage of less cognitive decline compared to WBRT, which is particularly recommended for patients with a small number of BMs.⁴ Therefore, with the increasing use of SRS in radiation oncology, accurate detection and delineation of BMs are essential.⁵

Contrast-enhanced magnetic resonance imaging is the preferred imaging modality for detecting and delineating BMs,⁵,⁶,⁷ allowing the detection of tiny metastases with high spatial resolution and low partial volume effects. Several studies have attempted to automatically detect and segment BMs on T1 weighted gadolinium enhanced (T1Gd) MR images using deep learning (DL) algorithms.⁸,⁹,¹⁰,¹¹,¹²,¹³ However, T1Gd MR simultaneously enhances BMs and intracranial vessels, causing difficulties in the auto-segmentation of BMs. Black-blood (BB) imaging, on the other hand, is known to selectively suppress blood signals, enhancing the visualization of small lesions compared to T1Gd.¹⁴,¹⁵ Thus, once DL-based auto-segmentation is involved in BB images, more accurate and reliable BMs detection and segmentation can be achieved.¹⁶,¹⁷,¹⁸,¹⁹

Although several DL-based algorithms have been developed for the detection and delineation of BMs,⁸,⁹,¹⁰,¹¹,¹²,¹³ only a few studies have attempted to extend them to BB images.¹⁶,¹⁷,¹⁸,¹⁹ Several studies that applied DL-based algorithms to BB images have reported improved BMs detection performance, particularly for small metastases. For instance, Kottlors, et al.¹⁶ demonstrated that a neural network architecture trained on BB images achieved significantly higher accuracy, with an area under the curve of 0.87, compared to 0.53 on T1Gd images. Oh, et al.¹⁷ also evaluated BMs detection using BB images, achieving a sensitivity of 87.95% and an average false-positive rate (FPR) of 14.48. Additionally, Kikuchi, et al.¹⁸ showed that their BMs detection model achieved the sensitivity of 91.7%, with a FPR of 1.5, which was slightly higher than that of the observer test. However, these studies did not directly address the segmentation of BMs on BB images. Another study Park, et al.¹⁹ investigated BMs segmentation on BB images, but their network required combining BB images with other imaging to achieve improved segmentation performance.

In this study, we investigated methods to enhance the performance of BMs auto-segmentation through DL algorithms on BB images. Specifically, we propose 1) modifying the neural network architecture to improve sensitivity and reduce false positives and 2) employing post-processing using the Segment Anything Model (SAM) to refine the segmentation performance.

MATERIALS AND METHODS

Patient cohort

The patient cohort comprised 50 patients diagnosed with BMs from lung cancer who underwent either WBRT or SRS between January 2019 and December 2020 at the Yonsei Cancer Center. The dataset consisted of BB images with manual delineation of the BMs by an expert radiation oncologist. The study was approved by the Institutional Review Board of Severance Hospital (4-2021-0306), and the requirement for informed consent was waived due to the retrospective nature of the study. Table 1 summarizes the baseline characteristics of the patients with BMs. In the training set, five patients had a single tumor, two had two tumors, two had three tumors, and 31 had four or more tumors. In the test set, two patients had a single tumor, one had two tumors, two had three tumors, and five had four or more tumors. Tumor volumes in the training set ranged from 0.003 cc to 101.672 cc, with a median of 0.049 cc. In the test set, volumes ranged from 0.007 cc to 16.642 cc, with a median of 0.069 cc.

Table 1. Characteristics of 50 Patients with BM.

Characteristic		Total (n=50)	Train-set (n=40)	Test-set (n=10)
Age (median, yr)		62.5 (range 31–81)	64.5 (range 40–81)	60 (range 31–76)
Sex
	Male	25	19	6
	Female	25	21	4
Number of patients per tumor
	1 tumor	7	5	2
	2 tumors	3	2	1
	3 tumors	4	2	2
	≥4 tumors	36	31	5
Tumor volume in cc
	Maximum	101.672	101.672	16.642
	Minimum	0.003	0.003	0.007
	Q1	0.018	0.018	0.024
	Q2 (median)	0.049	0.049	0.069
	Q3	0.203	0.023	0.254

BM, brain metastases.

Data pre-processing

The MR images in our datasets had different pixel spacing resolutions ranging from 0.195 to 0.417. To ensure consistency, we adjusted the data for pixel spacing by setting it to 0.4. The slice thickness for all the scans was consistently 1 mm. The intensity distribution of the MR images showed slight variations across patient scans, although the same simulator and pulse sequence were employed, which necessitated the use of intensity normalization to constrain the variations.²⁰,²¹,²² Among various methods, we adopted piecewise linear histogram matching,²³ called Nyul normalization, which is a data-driven approach that establishes a standard intensity scale from a set of representative images, which is subsequently applied to each MR image. This process aligns the histogram of the MR images with the landmarks of the standard scale, thereby enhancing the consistency across different images.

DL models

The U-Net architecture,²⁴ which is the most widely used architecture for semantic segmentation, consists of an encoding phase to capture context and a symmetric decoding phase for precise localization. In this study, the encoder had four layers, each comprising a convolutional layer, followed by batch normalization, ReLU activation, and a pooling layer. Similarly, the decoder had four layers, each incorporating a transposed convolutional layer for upsampling, followed by concatenation with the corresponding feature map from the encoder through skip connections. In this study, the network input and output were aggregated into three consecutive 2D slices, capturing partial volume information during the training of the network.

The major concern of the standard U-Net architecture is the loss of detailed image gradient information, which is mostly attributed to pooling operations in the encoding phase. A skip connection was devised to mitigate this loss by introducing the imaging features in the encoder to the newly upsampled features in the decoder. The modified U-Net provides additional amendments to conventional skip connections²⁵ as illustrated in Fig. 1A. The image features before applying the max-pooling operation (downsampling) were transmitted to the skip connection. The downsampled images resulting from pooling were then processed through the transpose convolution (deconvolution) operator, and the result was subtracted from the image features in the skip connection. This subtraction process allowed for identifying the imaging information lost during the pooling operation. The image-loss information was plugged into the newly upsampled features in the decoder, which helped preserve the image gradient information more effectively than the conventional skip connection. The modified U-Net, which was initially proposed by Seo, et al.,²⁵ includes an object-dependent adaptive filter that is activated depending on the object size. However, we chose not to incorporate this modification in the present study since our dataset predominantly consisted of small BMs (median volume, 0.049 cc).

Fig. 1. A deep learning approach enhancing BMs detection and segmentation. (A) A modified U-Net. (B) A modified U-Net with GAN. (C) The post-processing step using SAM. BM, brain metastases; GAN, generative adversarial network; SAM, Segment Anything Model.

To enhance the BMs detection and segmentation performance, the modified U-Net architecture uses a generative adversarial network (GAN)²⁶ consisting of a generator and discriminator (Fig. 1B). The generator consisted of a modified U-Net. The discriminator distinguishes between real and generated segmentation maps. The basic idea of a GAN is to compete with the generator against the discriminator, which makes it challenging to discern which of the true and generated segmentation maps is genuine. The architecture of the discriminator consisted of a series of four layers, each comprising a convolutional layer, followed by batch normalization and PReLU activation.

SAM as post-processing

SAM is an open-source software provided by Meta AI (New York City, NY, USA) comprising three main components: an image encoder, a prompt encoder, and a mask decoder. The image encoder utilizes a vision transformer (ViT)²⁷ pre-trained with a mask autoencoder²⁸ to capture detailed image features. The prompt encoder processes manual annotations using positional encoding. The mask decoder efficiently maps the image and prompts embeddings, along with an output token, to produce a mask through self-attention and cross-attention. We opted to use SAM for post-processing, focusing its application on refining segmentation within predefined bounding boxes generated by the three U-Net-based networks. The three U-Net-based networks generated different bounding boxes around the BM-segmented regions, with selected the regions of interest (ROIs) serving as the SAM prompt encoder input. Finally, SAM refines the segmentation within these boxes to enhance accuracy (Fig. 1C).

Evaluation

All training tasks in this study utilized a single graphic processing unit (NVIDIA TITAN RTX). The investigated networks were implemented using the PyTorch framework.²⁹ Preprocessing steps, including resampling and intensity normalization, were implemented using scikit-learn³⁰ and an intensity-normalization library.²² During the training process, a batch size of four was selected, and the models underwent early stopping after 100 epochs. The networks were optimized using the Adam optimizer and DICE loss function. The learning rate was initially set to 2e-5, and the scheduler dynamically adjusted it based on the validation loss. Furthermore, it reduced the rate by a factor of 0.5 if there was no improvement throughout two epochs; otherwise, the rate was reduced by a factor of zero.

To evaluate the robustness of our method, we set aside a test set comprising data from 10 randomly selected patients out of the total of 50 patients. The patient scans were excluded from the training process. The performances of the investigated networks, both with and without post-processing, were assessed using the dice similarity coefficient (DSC), lesion-wise sensitivity (LWS), and average FPR. The DSC values represent the accuracy of BMs segmentation, while the LWS and average FPR are associated with the detection capability of BMs on BB images. Statistical analyses were performed using SPSS ver. 27 (IBM Corp., Armonk, NY, USA), and significance was assessed using the Wilcoxon signed-rank test for non-normally distributed data.

RESULTS

Performance of three U-Net-based models

The comparison metrics for evaluating the performances of the three U-Net-based models are listed in Table 2. The modified U-Net equipped with a specialized skip connection yielded a patient-wise DSC of 0.846, a LWS of 89.19%, and an average FPR of 1.4. These results outperformed those of the standard U-Net with a conventional skip connection. Additionally, as shown in Fig. 2, the modified U-Net with the GAN, including the discriminator, enhanced the segmentation performance, achieving a patient-wise DSC of 0.853, while the LWS remained unchanged. Notably, the modified U-Net with GAN significantly reduced the average FPR to less than 1 compared to the standard U-Net (p=0.024). While the improvement over the modified U-Net did not reach statistical significance (p=0.059), it still demonstrated a consistent trend toward enhanced performance.

Table 2. Comparison of the Results of Three U-Net-Based Models for BMs Detection and Segmentation.

		Standard U-Net	Modified U-Net	Modified U-Net with GAN
LWS (%)		87.84	89.19	89.19
Average FPR		2.2	1.4	0.9* (p=0.024)
Patient-wise DSC		0.815 (±0.09)	0.846 (±0.06)	0.853 (±0.06)
Tumor volume range (lesion-wise DSC)
	≥0.1 cc	0.724 (±0.27)	0.712 (±0.28)	0.715 (±0.28)
	0.06–<0.1 cc	0.754 (±0.05)	0.761 (±0.08)	0.763 (±0.08)
	0.04–<0.06 cc	0.540 (±0.34)	0.635 (±0.27)	0.674 (±0.17)
	0.02–<0.04 cc	0.567 (±0.22)	0.553 (±0.20)	0.575 (±0.17)
	<0.02 cc	0.515 (±0.34)	0.558 (±0.38)	0.556 (±0.37)

BM, brain metastases; GAN, generative adversarial network; LWS, lesion-wise sensitivity; FPR, false-positive rate; DSC, dice similarity coefficient.

Asterisk (^*) indicates the statistical significance of differences between the modified U-Net with GAN and the standard U-Net, as determined by p-values.

Fig. 2. The comparison of three U-Net-based models. The figure demonstrates the improvements in DSC, sensitivity, and false positives across different U-Net model variations. (A) True positives are shown, highlighting DSC improvements across the standard U-Net, modified U-Net, and modified U-Net with GAN. (B) False negatives are presented, illustrating the increase in sensitivity achieved by the modified models. (C) False positives are displayed, indicating the reduction in false positives with both modified models. Yellow bounding boxes indicate zoomed-in images to make differences clearer. DSC, dice similarity coefficient; GAN, generative adversarial network.

The lesion-wise DSC across varying BMs volumes is also presented in Table 2. For BMs with volumes in the range of 0.04–0.06 cc, the modified U-Net and modified U-Net with GAN achieved notable improvements, yielding lesion-wise DSC of 0.635 and 0.674, respectively, as compared to the standard U-Net’s 0.540. This indicates enhanced segmentation performance for smaller BMs. For BMs with volumes over 0.1 cc, the lesion-wise DSC across all three models was consistently around 0.715–0.724. However, for BMs with very small volumes, below 0.04 cc, the lesion-wise DSC was lower across all models.

Table 3 demonstrates that the LWS remained consistently high at 100% across all models when the number of BMs per patient was less than four, aligning with the ASCO-SNO-ASTRO guidelines 4 recommending SRS for such cases. Performance differences across models became apparent as the number of BMs per patient equaled or exceeded four. Regarding the volume of BMs, the largest variation across the models occurred in the range of 0.04–0.06 cc. Within this range, the standard U-Net yielded a relatively low LWS (88.89%), whereas the other two models achieved a LWS of 100%.

Table 3. LWS Depends on the Number of BMs Per Patient and the Volume Range of BMs.

		Standard U-Net	Modified U-Net	Modified U-Net with GAN
Patients with tumor count <4		100	100	100
Patients with tumor count ≥4		84.75	86.44	86.44
Tumor volume range
	≥0.1 cc	93.55	93.55	93.55
	0.06–<0.1 cc	100	100	100
	0.04–<0.06 cc	88.89	100	100
	0.02–<0.04 cc	92.31	92.31	92.31
	<0.02 cc	66.67	66.67	66.67

LWS, lesion-wise sensitivity; BMs, brain metastases; GAN, generative adversarial network.

In Table 4, regardless of the number of BMs, a consistent trend is shown, in which the average FPR decreases in the order of standard U-Net, modified U-Net, and modified U-Net with GAN. With fewer than four BMs, the modified U-Net only and the modified U-Net with GAN achieved an average FPR of less than 1. When the number of BMs was equal to or exceeded four, the gaps among the models increased. Importantly, the modified U-Net with GAN constrained the FPR to approximately one under such conditions.

Table 4. Average FPR Depending on the Number of BMs Per Patient.

	Standard U-Net	Modified U-Net	Modified U-Net with GAN
Patients with tumor count <4	1.17	0.83	0.67
Patients with tumor count ≥4	3.75	2.25	1.25

FPR, false-positive rate; BMs, brain metastases; GAN, generative adversarial network.

Performance of post-processed models

Table 5 lists the numerical performance of SAM-based post-processing models. The post-processing tasks enhanced segmentation performance, increasing the patient-wise DSC by 2% for the U-Net-based models, with statistical significance observed for the standard U-Net (p=0.017), the modified U-Net (p=0.019), and the modified U-Net with GAN (p=0.037). The modified U-Net with GAN achieved a patient-wise DSC of 0.873. For lesion-wise DSC, performance improved across all volume ranges. Notably, in the volume range of over 0.1 cc, all three models exhibited statistically significant improvements with p<0.001, while in the 0.02–0.04 cc range, the improvements were also significant with p<0.01. The enhancement in segmentation performance throughout post-processing was also evident in the delineations of BMs, as shown in Fig. 3. The predicted segmentation maps after post-processing closely resembled the ground-truth contours compared to the segmentation maps before processing. The post-processing, which utilized the bounding boxes from the U-Net-based approaches, did not affect LWS or the average number of FPR, but segmentation performance was improved.

Table 5. The Numerical Performance of Three Post-Processed Models for BMs Detection and Segmentation.

After post-processing		Standard U-Net	Modified U-Net	Modified U-Net with GAN
LWS (%)		87.84	89.19	89.19
Average FPR		2.2	1.4	0.9
Patient-wise DSC		0.834 (±0.08)* (p=0.017)	0.868 (±0.05)* (p=0.019)	0.873 (±0.05)* (p=0.037)
Tumor volume range (lesion-wise DSC)
	≥0.1 cc	0.754 (±0.27)* (p<0.001)	0.746 (±0.28)* (p<0.001)	0.752 (±0.28)* (p<0.001)
	0.06–<0.1 cc	0.775 (±0.05)	0.788 (±0.08)	0.787 (±0.08)
	0.04–<0.06 cc	0.576 (±0.34)	0.667 (±0.27)	0.718 (±0.17)
	0.02–<0.04 cc	0.638 (±0.22)* (p=0.002)	0.607 (±0.20)* (p=0.002)	0.630 (±0.17)* (p=0.003)
	<0.02 cc	0.534 (±0.34)	0.575 (±0.38)	0.586 (±0.37)

BMs, brain metastases; GAN, generative adversarial network; LWS, lesion-wise sensitivity; FPR, false-positive rate; DSC, dice similarity coefficient.

Asterisks (^*) indicate the statistical significance of differences between the post-processed model and the corresponding model before post-processing, as determined by p-values.

Fig. 3. The cases achieved enhancement through post-processing. The top row (A) displays under-segmentation (center) and its refinement (SAM post-processed), while the bottom row (B) shows over-segmentation (center) and its refinement (SAM post-processed). In the SAM post-processed images, the green box represents the bounding box used as input for the SAM’s prompt encoder, generated by the U-Net-based model. In the comparison column, red contours denote the ground truth, green contours indicate the model’s initial segmentation, and yellow contours show the post-processed segmentation, illustrating refinement in both under- and over-segmentation cases. Yellow bounding boxes indicate zoomed-in images to make differences clearer. SAM, Segment Anything Model.

DISCUSSION

The primary objective of this study was to enhance the automatic detection and segmentation of BMs in BB images. Detecting and segmenting BMs has been challenging due to their intrinsic features such as irregular shapes, small volumes, and unpredictable locations of occurrence. BB imaging suppresses the vessel elements that can facilitate BMs detection. To enhance the performance of DL-based models in terms of BMs detection and segmentation, we focused on two key aspects: modifications to the DL network architecture and post-processing using a foundation model (SAM)-based prompt.

For the network architecture, the standard U-Net was modified by incorporating a new concept of skip connection (modified U-Net) and adding a discriminator to the generator (modified U-Net with GAN). The enhanced skip connections were designed to mitigate information loss during the encoding phase, allowing more effective retention of gradient information in BMs detection and segmentation. Additionally, the inclusion of a discriminator in the modified U-Net improved segmentation performance by encouraging the model to generate more realistic and accurate segmentation maps. The modified U-Net and modified U-Net with GAN models demonstrated substantial performance improvements, both achieving an LWS of 89.19%. Notably, the modified U-Net with GAN effectively minimized false positives, reducing the average FPR from 2.2 to 0.9 (p=0.024), demonstrating its robustness in enhancing detection reliability, as shown in Table 2.

In the post-processing stage, we used SAM to further refine the segmentation maps. While SAM alone can provide segmentation results for given medical images, it may not produce qualified segmented maps when the model is trained with different types of images. Furthermore, since SAM was designed for a broad range of segmentation tasks, applying it to our dataset posed challenges, as it segmented not only the GTV but also other structures, potentially leading to difficulty in interpreting multiple GTVs as distinct labels. Alternatively, adjusting the model to improve BMs detection and segmentation predictions could be beneficial, but fine-tuning may be challenging due to the limited dataset size used for tuning compared to the original SAM model’s training data. Given these considerations, we employed SAM as a post-processing step, focusing on narrow ROIs. As shown in Tables 2 and 5, this approach contributed to improved segmentation performance, with the modified U-Net with GAN achieving a significant increase in patient-wise DSC from 0.853 to 0.873. Furthermore, lesion-wise DSC significantly improved after post-processing (p<0.001 for over 0.1 cc, p<0.01 for 0.02–0.04 cc), demonstrating enhanced segmentation performance across different lesion sizes and reinforcing the robustness of our approach.

A significant commonality between our study and previous studies¹⁶,¹⁷,¹⁸,¹⁹ is the recognition of the potential benefits of employing BB images. Previous studies demonstrated the efficacy of utilizing BB images by achieving notable improvements in BMs detection performance. A previous study¹⁹ reported that the average FPR reached 0.59. However, the network was designed for training using both T1Gd and BB images. Additionally, the datasets used in the previous study may have differed from our datasets in terms of the proportion of patients with a small volume of BMs. Notably, our proposed network detected BMs at 100% with an average FPR of 0.67 when the number of BMs was less than four. This implies that the proposed network has the potential to be applied to patients with SRS according to the ASCO-SNO-ASTRO guidelines.⁴ In general, studies on BMs from lung cancer cases reported a median volume of 0.1 cc to 4 cc,³¹ whereas our dataset focused on tumors with a significantly smaller median volume. Specifically, our dataset included a total of 787 BMs, with approximately 20% (165 BMs) containing both necrosis and viable tumor tissue. Notably, around 80% of the BMs in this study were very small and did not contain necrosis. This aspect underscores the challenge of detecting small BMs, highlighting the effectiveness of our approach in identifying and segmenting these difficult-to-detect cases.

Despite the various advantages stated in this study, there were certain limitations. First, the patient cohort included datasets from a single institution and consisted of only 50 patients, a relatively small sample size compared to previous studies on BMs segmentation. Although this size is modest, our study focused on enhancing the segmentation and detection accuracy of small-volume BMs in BB images by amending network architecture and employing foundation model with relatively limited sample size. The modifications of the skip connection of the U-Net and foundation model-based post-processing methods were demonstrated to effectively function in this work. Second, while convolutional neural network-based approaches have been employed for current automated BMs segmentation, recent advancements such as ViTs³²,³³,³⁴ may offer potential improvements. ViTs are believed to overcome the drawbacks of convolution-based networks, which mostly rely on local image information. BMs segmentation requires a very small volume of BMs, and the degree of improvement from ViTs may be insignificant. Third, this study utilized SAM as a post-processing method, which successfully enhanced the segmentation performance, as evidenced by the increase in patient-wise DSCs. However, the post-processing method with bounding boxes from the U-Net inferences was unable to identify false-negative BMs (related to sensitivity) or suppress false-positive BMs (related to average FPR). From our observations, applying the given SAM model to our datasets did not perform well because the given model was trained with numerous types of natural images and not solely BB images. Another approach for fine-tuning the given network failed to achieve better performance than the U-Net-based approach. As a further study, expanding the dataset to include multiple institutions and using federated transfer learning could improve the generalizability of DL-based models for BMs segmentation. Additionally, developing a domain-specific SAM model trained on BB images specifically for BMs segmentation may further enhance the accuracy in identifying and segmenting BMs, addressing the limitations noted with the current SAM model.

In conclusion, the proposed work developed DL-based auto-segmentation networks to enlarge the BMs detection and segmentation performance in BB images. Modifying the skip connection in the standard U-Net notably enhanced the sensitivity of BMs in BB images, and incorporating the discriminator helped reduce false positives in BMs detection. Additionally, it was demonstrated that applying SAM as a post-processing step further improved the segmentation performance of BMs in BB images.