Extended nnU-Net for Brain Metastasis Detection and Segmentation in Contrast-Enhanced Magnetic Resonance Imaging With a Large Multi-Institutional Data Set

Abstract

Purpose:

The purpose of this study was to investigate an extended self-adapting nnU-Net framework for detecting and segmenting brain metastases (BM) on magnetic resonance imaging (MRI).

Methods and Materials:

Six different nnU-Net systems with adaptive data sampling, adaptive Dice loss, or different patch/batch sizes were trained and tested for detecting and segmenting intraparenchymal BM with a size ≥2 mm on 3 Dimensional (3D) post-Gd T1-weighted MRI volumes using 2092 patients from 7 institutions (1712, 195, and 185 patients for training, validation, and testing, respectively). Gross tumor volumes of BM delineated by physicians for stereotactic radiosurgery were collected retrospectively and curated at each institute. Additional centralized data curation was carried out to create gross tumor volumes of uncontoured BM by 2 radiologists to improve the accuracy of ground truth. The training data set was augmented with synthetic BMs of 1025 MRI volumes using a 3D generative pipeline. BM detection was evaluated by lesion-level sensitivity and false-positive (FP) rate. BM segmentation was assessed by lesion-level Dice similarity coefficient, 95-percentile Hausdorff distance, and average Hausdorff distance (HD). The performances were assessed across different BM sizes. Additional testing was performed using a second data set of 206 patients.

Results:

Of the 6 nnU-Net systems, the nnU-Net with adaptive Dice loss achieved the best detection and segmentation performance on the first testing data set. At an FP rate of 0.65 ± 1.17, overall sensitivity was 0.904 for all sizes of BM, 0.966 for BM ≥0.1 cm³, and 0.824 for BM <0.1 cm³. Mean values of Dice similarity coefficient, 95-percentile Hausdorff distance, and average HD of all detected BMs were 0.758, 1.45, and 0.23 mm, respectively. Performances on the second testing data set achieved a sensitivity of 0.907 at an FP rate of 0.57 ± 0.85 for all BM sizes, and an average HD of 0.33 mm for all detected BM.

Conclusions:

Our proposed extension of the self-configuring nnU-Net framework substantially improved small BM detection sensitivity while maintaining a controlled FP rate. Clinical utility of the extended nnU-Net model for assisting early BM detection and stereotactic radiosurgery planning will be investigated.

Introduction

Brain metastasis (BM) is a common manifestation of cancer, which occurs in 10% to 20% of all patients with cancers and increases patient mortality and morbidity with low 5-year survival, despite multimodal treatment options of surgery, radiotherapy, and chemotherapy.¹ Stereotactic radiosurgery (SRS), which delivers high radiation doses to BMs with a tight margin, is an effective therapy for achieving long-term local control of multiple BMs.² Magnetic resonance imaging (MRI) is commonly used to diagnose BM and delineate these lesions for SRS planning. Failure to identify metastases early in their course of progression could lead to patients not receiving adequate or appropriate therapy, potentially leading to overall decreases in quality of life and survival and the need for additional procedures with associated costs. Given the number of metastases potentially present in the brain, detecting and accurately delineating all lesions of relevance for treatment selection and implementation involves significant human effort, and as such, automated tools that assist radiologists, surgeons, and radiation oncologists in their respective roles in detection and delineation of multiple metastases can positively impact both the efficiency as well as the efficacy of management of affected patients.

Automated algorithms based on artificial intelligence and especially deep learning techniques have been developed for the detection and segmentation of BMs from MRI with promising results, particularly for large-size BMs.^3–20 However, there are still challenges in the development of such automated approaches for detecting BMs. One of these is how to achieve high sensitivity of small BM detection while controlling the FP detection rate. The structural and intensity similarities between small BM and vasculature on 2 Dimensional (2D) Magnetic Resonance (MR) slices can make BM detection difficult,⁵ posing a challenge to techniques that work well for larger lesions. In addition, early works are largely based upon relatively small data sets from a single institute,^5,9–11,19 for which generalizability of the resulting models is uncertain.

The nnU-Net is a state-of-the-art medical image segmentation framework that uses the U-Net²¹ with automatic pipeline configuration.²² It makes automated adaptations including the patch size, batch size, data augmentation, pre-processing, network topology, and postprocessing and provides a competitive out-of-the-box solution for various medical image segmentation tasks. It also shows superior performance on the segmentation tasks compared with other U-Net variants.

In this study, we aimed to extend the nnU-Net framework for detecting and contouring BMs from 3D post-Gd MRI scans with particular efforts to enhance the detection of small brain metastatic lesions while reducing false positives. We trained and tested the extended nnU-Net using a large multi-institutional data set and performed tests on a second publicly released data set.

Methods and Materials

Patient data

MRI scans of 2092 patients with BMs who were irradiated by SRS were retrospectively collected from 7 institutions after approval of local institutional review boards of each institute. All patients had 3D volumetric post-Gd T1w MR images pre-SRS, which were mostly the MPRAGE sequence.²³ Some patients had other types of T1w MR imaging sequences such as SPGR, SPACE, PETRA, BRAVO, and so on. Gross tumor volumes (GTVs) of BMs for SRS treatment planning that were delineated by treating physicians at each institute were retrospectively collected and curated by local teams of radiation oncologists, radiologists, and medical physicists. The GTVs were used as ground truth for training, validation, and testing the BM detection and segmentation models. Considering that missing the GTVs of the untreated BMs reduces the accuracy of ground truth and that not all BMs visible on a given MRI are to be treated with a given plan, additional central efforts were carried out by 2 radiologists to review the cases. GTVs of the uncontoured BMs were created when the 2 radiologists reached the consensus on diagnosis. The intraparenchymal metastases that had a dimension greater than 2 mm in all 3 axes (a volume <0.004 cm³) and were not adjacent to a surgical bed were included and curated in this study. This threshold was chosen due to observations that the uncertainty of diagnosis of BM that had 1 to 2 mm diameters was considerably high by the physicians on our team. Finally, this data set was randomly divided into subsets of 1712, 195, and 185 patients for training, validation, and testing, respectively. To increase the variety of BM sizes and shapes in our training data set, the training data set was augmented with synthetic BMs of 1025 MRI volumes using a 3D-2D generative pipeline,²⁴ which was shown to be effective in enhancing overall BM detection performance while maintaining the FP rate per case.^24,25 This testing data set was considered Testing Data Set 1. an additional testing data set (Testing Data Set 2), which is a subset of recently released public data set (TCIA Gamma Knife MR sets) that consists of 206 patients with BMs, was used for further testing of the developed models. Additional data curation of this data set was carried out to identify uncontoured BMs by 1 radiologist. BM size distributions within the training, validation, and testing data sets (Testing Data Set 1 And Testing Data Set 2) are shown in Figure 1, and other details of the data sets are provided in Supplementary Materials.

Figure 1.

Brain metastasis volume distributions of the training and validation data set, Testing Data Set 1, and Testing Data Set 2. The tumor volume is based upon the ground-truth contour volumes (in cm³).

Detection and segmentation models

We employed the nnU-Net²² as a framework for brain metastasis detection and segmentation from 3D post-Gd T1w MR images, but with several modifications to maximize detection and segmentation performance. For the default nnU-Net, we trained with the 3D U-Net trainer²² in a 5-fold cross-validation, resulting in a training configuration with patch size 128 х 128 х 128 and batch size 2. To optimize the detection performance of small BMs, we incorporated tumor volume adaptive 3D patch sampling into the nnU-Net framework, in which each lesion had an identical sampling probability regardless of its size, using pre-computed BM bounding-boxes for computation efficiency. This adaptive sampling prevents over-sampling of large BMs during network training. We investigated if this adaptive data sampling (ADS) strategy could mitigate the severe data imbalance of lesion size between small and large BMs within the nnU-Net model. Recent work showed that a smaller 3D patch size (64 х 64 х 64) for U-Nets could be more efficient for detecting smaller BMs.⁶ We trained the same training configuration with a smaller patch size of 64 х 64 х 64 and compared it with the default patch size of 128 х 128 х 128 to analyze the impact of the network field of view. A study showed that training the nnU-Net with a large batch size could enhance the performance of brain tumor segmentation from MRI scans.²⁶ We trained the nnU-Net with ADS configuration, patch size 128 х 128 х 128, and a batch size of 5 and compared with the default batch size 2 to determine the influence of batch size on BM segmentation. Recent studies have demonstrated that using a tumor volume adaptive Dice loss (ADL) could improve the sensitivity of small BM detection while keeping a low FP rate.^7,8 We integrated ADL into the nnU-Net framework along with ADS. Overall, our experiments included 6 different models: a nnU-Net with the default self-adaptive mechanism (patch size 128 х 128 х 128 and batch size 2), a nnU-Net with ADS, a nnU-Net with patch size 64 х 64 х 64 and ADS, a nnU-Net with batch size 5 and ADS, a nnU-Net with ADL, and a nnU-Net with ADS and ADL. All configurations employed patch size 128 х 128 х 128 and batch size 2 unless otherwise specified.

Evaluation

The detection and contouring performance of the trained nnU-Net models were evaluated on 2 testing data sets. We applied the connected component analysis algorithm²⁷ to the output volumes of the models to isolate each prediction structure. A BM was considered detected if the ground-truth contour overlapped with a predicted structure (called true positive [TP]) and otherwise was counted as false negative. A predicted structure was considered a false positive (FP) if it had no overlap with any BM ground-truth contour. A brain mask was applied to the final segmentation mask of BMs as a postprocessing step before measuring performance. For detection performance, we measured overall lesion-level sensitivity, average patient-level sensitivity, and average case-level FP rate. For segmentation performance, we measured lesion-level Dice similarity coefficient (DSC), 95-percentile Hausdorff distance (HD95), and average Hausdorff distance of the TP BMs. We also stratified BMs based on size to analyze the detection and segmentation performances of small and large BMs. To analyze the sensitivity of the models at different operating points, we generated Free-Response Operating Characteristic (FROC) curves by varying the threshold value of the model output to produce the final segmentation mask. The detection metrics were computed at each operating point to reveal the relationship between the sensitivity and average FP per patient. These metrics were computed per patient first and then averaged over all testing patients.

Results

We evaluated the 6 nnU-Net models on the 2 different testing data sets (Testing Data Set 1 And Testing Data Set 2) as described in Patient Data Section.

Performance on Testing Data set 1

Testing Data Set 1 consisted of 185 patients with 952 curated BMs (424 lesions [44.5%] <0.1 cm³). The average and median numbers of BMs per patient were, respectively, 5.3 and 4 (range: 1–30), and the average and median BM sizes were, respectively, 1.1 cm³ and 0.12 cm³ (range: 0.008–83.7 cm³). The BM volume distribution is shown in Figure 1.

Performance assessment of BM detection

The FROC curves of the 6 models of the patient-wise average FP rate and 3 sensitivity computations, overall lesion-wise sensitivity, average patient-wise sensitivity, and overall lesion-wise sensitivity for small BMs only (0.004–0.1 cm³), are shown in Figure 2. The models using ADS except with a small patch size (64 х 64 х 64) increased the areas under FROC curves for both large and small BMs compared with the default nnU-Net. The model incorporating ADL substantially increased the area under the FROC curve particularly for small BMs, compared with the default model and the model with ADS. The model incorporating both ADS and ADL slightly decreased the area under the FROC curve compared with the model with ADL. In our nnU-Net models, the effect of batch size on BM detection was not noted, and a smaller patch size degraded detection performance greatly. At an approximated average FP rate of 1.0 per patient, the model with ADL, the best model, achieved an overall sensitivity of 0.925 for all size BMs and 0.870 for small BMs (0.004–0.1 cm³).

Figure 2.

Free-response operating characteristic curves of overall lesion-wise sensitivity of all size brain metastases (left column), average patient-wise sensitivity of lesions (middle column), and overall sensitivity of small BMs only (right column) of Testing Data Set 1 by the 6 models (top row) and of Testing Data Set 2 by the model with ADL (bottom row).

To further evaluate BM detection, we fixed operation points of the models to approximately have 0.9 overall lesion-level sensitivity. The resulting operation points for each model are described in Supplementary Materials. Both models with ADL alone and with ADS and ADL together achieved significantly lower average patient-level FP rates of 0.65 ± 1.17 (95% CI: 0.48–0.82) and 0.66 ± 1.14 (95% CI: 0.49–0.83) than the other 4 models (pairwise Wilcoxon signed rank test: P < .001), see Table 1. Patient-level average sensitivity was 0.930 ± 0.156 (95% CI: 0.907–0.952) for the model with ADL and 0.918 ± 0.175 (95% CI: 0.893–0.943) for the model with ADS and ADL. The models with ADS except using a small patch size significantly reduced average FP rates compared with the default nnU-Net model (pairwise Wilcoxon signed rank test: P < .001), but the model with ADS did not significantly reduce FP rate compared with the model using ADS and ADL together.

Table 1.

Detection and segmentation performance comparisons of the 6 nnU-Net models on Testing Data Set 1

Model	Overall sensitivity (all lesions)	Average Sensitivity (per patient)	FP rate	HD95/Avg. HD	DSC
nnU-Net	0.903	0.937 ± 0.136 95% CI: 0.917–0.956	1.79 ± 1.79 95% CI: 1.52–2.04	2.20 ± 3.70/ 0.40 ± 0.91	0.675 ± 0.168
nnU-Net with ADS	0.894	0.934 ± 0.139 95% CI: 0.914–0.954	0.91 ± 1.27 95% CI: 0.73–1.10	1.47 ± 1.26/ 0.25 ± 0.29	0.726 ± 0.149
nnU-Net with ADS and 643 patch size	0.894	0.925 ± 0.151 95% CI: 0.903–0.947	3.26 ± 2.07 95% CI: 2.96–3.57	1.67 ± 1.98/ 0.29 ± 0.56	0.710 ± 0.157
nnU-Net with ADS and batch size 5	0.906	0.940 ± 0.130 95% CI: 0.921–0.958	1.21 ± 1.51 95% CI: 0.99–1.43	2.00 ± 2.90/ 0.36 ± 0.78	0.678 ± 0.162
nnU-Net with ADL	0.904	0.930 ± 0.156 95% CI: 0.907–0.952	0.65 ± 1.17 95% CI: 0.48–0.82	1.45 ± 1.91/ 0.23 ± 0.42	0.758 ± 0.145
nnU-Net with ADS and ADL	0.895	0.918 ± 0.175 95% CI: 0.893–0.943	0.66 ± 1.14 95% CI: 0.49–0.83	1.46 ± 1.54/ 0.23 ± 0.33	0.758 ± 0.149

Abbreviations: Average sensitivity = average of metastasis-level sensitivity per patient over all patients; Avg HD = the average Hausdorff distance of detected true positive brain metastases in mm; DSC = Dice similarity coefficient of detected true positive brain metastases; FP rate = false-positive rate; HD95 = the 95 percentile Hausdorff distance of detected true positive brain metastases in mm; mean ± standard deviation; overall sensitivity = lesion-level sensitivity.

Performance assessment of BM segmentation

We evaluated BM segmentation performances of the 6 nnU-Net models using the same operation points to have 0.9 overall lesion-level sensitivity. Table 1 shows segmentation performance metrics of DSC, HD95, and average HD at the chosen operation points. The model with ADL and the model with ADS and ADL achieved significantly higher mean TP DSCs of 0.758 (STD = 0.145 and STD = 0.149, respectively) than the other models (pairwise Wilcoxon signed rank test: P < .001). The models with ADS, with ADL, or with both ADS and ADL with the default patch size and default batch size achieved similar and not significantly different HD95 values of 1.47 ± 1.26 mm, 1.45 ± 1.91 mm, and 1.46 ± 1.54 mm, respectively, which were significantly lower than the default model and the models with ADS but using a small patch size or a large batch size (pairwise Wilcoxon signed rank test: P < .001).

Segmentation performances stratified for small BMs 0.004–0.1 cm³ (2–6 mm ESD [Estimated Sphere Diameter]) and medium/large BMs ≥0.1 cm³ at the same operation points used before are shown in Table 2 and Table E1, respectively. Note that the overall sensitivity values of all the models were decreased to 0.807−-0.829 for small BMs and increased to 0.958–0.974 for medium/large BMs compared with those for all sizes of BMs. For detected TP small BMs 0.004–0.1 cm³ (2–6 mm ESD), the model with ADL and the model with ADS and ADL achieved significantly higher DSC values of 0.707 and 0.710 (STD = 0.139 and STD = 0.141, respectively) than the other models (pairwise Wilcoxon signed rank test: P < .001). These 2 models achieved similar HD95 values of 1.09 ± 0.48 mm and 1.10 ± 0.50 mm, respectively, which were significantly lower than the other models (pairwise Wilcoxon signed rank test: P < .001). The differences in DSC and HD95 between the 2 models were not statistically significant. For detected TP medium/large BMs (≥0.1 cm³), the model with ADL and the model with ADS and ADL achieved significantly higher DSC values of 0.793 and 0.791 (STD = 0.139 and STD = 0.145) than the other models (pairwise Wilcoxon signed rank test: P < .05). The model with ADS achieved the lowest HD95 value of 1.59 ± 1.2 mm, which was significantly lower than the other models (pairwise Wilcoxon signed rank test: P < .05). For large BMs (≥0.5 cm³, Table E2), the models with ADS, with ADL, and with both ADS and ADL achieved similar DSC values of 0.831 ± 0.095, 0.827 ± 0.129, and 0.83 ± 0.132, respectively, which were significantly higher than the other models (pairwise Wilcoxon signed rank test: P < .001). The model with ADS achieved the lowest HD95 value of 1.80 ± 1.49 mm, which was significantly lower than the other models (pairwise Wilcoxon signed rank test: P < .001).

Table 2.

Segmentation performance comparisons of the 6 nnU-Net models on small BMs (0.004–0.1 cm³, ESD 2–6 mm) of Testing Data Set 1

Model	Overall sensitivity (all lesions)	HD95/Avg. HD	DSC
nnU-Net	0.820	1.83 ± 3.41 (0.44 ± 0.99)	0.591 ± 0.178
nnU-Net with ADS	0.807	1.32 ± 1.33 (0.30 ± 0.38)	0.646 ± 0.149
nnU-Net with ADS and 643 patch size	0.807	1.56 ± 2.73 (0.37 ± 0.83)	0.629 ± 0.172
nnU-Net with ADS and batch size 5	0.829	1.66 ± 2.77 (0.39 ± 0.82)	0.603 ± 0.170
nnU-Net with ADL	0.824	1.09 ± 0.48 (0.23 ± 0.15)	0.707 ± 0.139
nnU-Net with ADS and ADL	0.814	1.10 ± 0.50 (0.23 ± 0.15)	0.710 ± 0.141

Operating points were set at the same as all size brain metastases in Table 1.

Abbreviations: Avg HD = the average Hausdorff distance of detected true positive brain metastases in mm; DSC = Dice similarity coefficient of detected true positive brain metastases; HD95 = the 95 percentile Hausdorff distance of detected true positive brain metastases in mm; mean ± standard deviation; overall sensitivity = lesion-level sensitivity.

We further stratified testing performances of smaller BM sizes in the ranges of 0.0005–0.014 cm³ (1–3 mm ESD) and 0.014–0.1 cm³ (3–6 mm ESD) with the nnU-Net model with ADL (Table E3). For the 1 to 3 mm ESD BMs, the overall sensitivity achieved by the nnU-Net model with ADL was 0.622 with a FP rate of 1.06, and the overall sensitivity was 0.875 for 3 to 6 mm ESD BMs. For detected TP 1 to 3 mm ESD BMs, the model achieved a DSC value of 0.602 and an HD95 value of 1.05 mm. The DSC value was 0.692, and the HD95 value was 1.12 mm for detected TP 3 to 6 mm ESD BMs. Table E4 shows the testing performance of BMs <0.014 cm³ (<3 mm ESD), which was measured without the minimum BM size threshold. The overall sensitivity decreased to 0.566, and the FP rate increased to 1.35. For detected TP BMs, the model achieved a DSC value of 0.570 and an HD95 value of 1.09 mm.

False positives analysis

The nnU-Net model with ADL produced 127 FP BMs using the same operation point, for which the volume distribution is shown in Figure E1. In total, 83% of FP BMs had a volume under 0.1 cm³. The FP-predicted structures were visually examined by a neuroradiologist, and subsets were identified as vascular entities (15%) and leptomeningeal metastases (15%) that were not included in the study design or a few entities involved in the ventricles. Other FPs were indiscernible by the manual investigation. Examples of TP and FP BMs are illustrated in Figure 3.

Figure 3.

The top row shows examples for brain metastases correctly identified by the automated system. Green line depicts ground-truth contour and blue line depicts AI-generated contour. The bottom row shows lesions detected by the AI system, which were not manually contoured. Visual investigation by a neuroradiologist indicated that (a) is likely leptomeningial metastasis, (b) is a suspected true positive intra parenchymal metastasis, (c) is a vessel (better appreciated by studying this and surrounding slices), and (d) is a leptomeningeal, likely subarachnoid, metastasis.

Performance on Testing Data Set 2

The Testing Data Set 2 consisted of 206 patients with 2070 curated BMs (1140 lesions (55.1%) <0.1 cm³). The average and median numbers of BMs per patient were, respectively, 10.0 and 7 (range: 1–99), and the average and median BM sizes were, respectively, 0.49 cm³ and 0.08 cm³ (range: 0.009–31.6). The BM size distribution is shown in Figure 1. The model with ADL that achieved the overall best performance on Testing Data Set 1 was evaluated by Testing Data Set 2.

Performance assessment of BM detection

The model with ADL achieved overall sensitivity 0.917 for all size BMs and 0.90 for small BMs (<0.1 cm³) at an approximated FP rate of 1.0 per patient, and overall sensitivity of 0.86 for all size BMs and 0.80 for small BMs (0.004–0.1 cm³) at an approximated FP rate of 0.45, as shown in Figure 2. To further evaluate BM detection, we fixed the operation point of the model to have 0.90 of overall lesion-level sensitivity. The resulting operation point is described in Supplementary Materials. The model achieved an average patient-level FP of 0.57 ± 0.85 (95% CI: 0.46–0.69) (Table 3).

Table 3.

Detection and segmentation performance of the nnU-Net with ADL on Testing Data Set 2

Overall sensitivity (all lesions)	Average Sensitivity (per patient)	FP rate	HD95/Avg. HD	DSC
0.907	0.904 ± 0.151 95% CI: 0.883–0.925	0.57 ± 0.85 95% CI: 0.46–0.69	1.91 ± 2.01/0.33 ± 0.47	0.705 ± 0.176

Abbreviations: Average sensitivity = average of metastasis-level sensitivity per patient over all patients; Avg HD = the average Hausdorff distance of detected true positive brain metastases in mm; DSC = Dice similarity coefficient of detected true positive brain metastases; FP rate = false-positive rate; HD95 = the 95 percentile Hausdorff distance of detected true positive brain metastases in mm; mean ± standard deviation; overall sensitivity = lesion-level sensitivity.

Performance assessment of BM segmentation

For segmentation performance evaluation, we used the original GTV contours of the TCIA data set because our data curation process identified but did not manually contour the untreated BMs. Table 3 shows that the model with ADL achieved a mean DSC of 0.705 ± 0.176, HD95 of 1.91 ± 2.01 mm, and average HD of 0.33 ± 0.47 mm. Segmentation performances stratified for BM volumes using threshold values of 0.1 cm³ and 0.5 cm³ are shown in Table E5. Compared with the average value of all sizes of BMs in Table 3, the overall sensitivity was decreased to 0.887 for small BMs 0.004–0.1 cm³ and increased to 0.931 for medium/large BMs ≥ 0.1 cm³ and to 0.953 for BMs ≥ 0.5 cm³ measured at a FP rate of 0.57, which was selected as described in Section 3.2.1. Similarly, the mean DSC was decreased to 0.639 ± 0.148 for small BMs and increased to 0.748 ± 0.179 for medium/large BMs and to 0.810 ± 0.175 for large BMs. The mean HD95 was decreased to 1.45 ± 0.88 mm for small BMs, increased to 2.22 ± 2.44 mm for medium/large BMs, and further increased to 2.49 ± 3.18 mm for large BMs.

We further stratified testing performances of smaller BM sizes to 0.0005–0.014 cm³ (1–3 mm ESD) and 0.014–0.1 cm³ (3–6 mm ESD) in Table E6 with the nnU-Net model with ADL. For BMs 0.0005–0.014 cm³, the overall sensitivity achieved by the nnU-Net model with ADL was 0.836 with a FP rate of 1.12, DSC was 0.487, and HD95 was 1.99 mm for detected TP BMs. For BMs 0.014–0.1 cm³, the overall sensitivity was 0.890, and the model achieved a DSC value of 0.600, and an HD95 value of 1.70 mm for detected TP BMs. Table E7 shows the testing performance of BMs < 0.014 cm³ (<3 mm ESD), which was measured without the minimum BM size threshold. The overall sensitivity decreased to 0.781, and the FP rate increased to 2.14. For detected TP BMs, the model provided a DSC value of 0.450 and an HD95 value of 3.44 mm.

Discussion

In this study, we investigated whether extended nnU-Net models with recent advanced training methods could substantially improve the detection and segmentation of BM, particularly for small BMs (<0.1 cm³) that are more challenging to detect but highly clinically relevant, using a large multi-institute data set. Although the nnU-Net framework is designed to automatically determine network architecture and training strategy for optimal performance, we observed that the extensions with ADS and ADL had significant improvements in BM detection and segmentation performances, which was validated using 2 testing datasets. The nnU-Net model with ADL achieved the overall best detection sensitivity and segmentation accuracy compared with other models. Using ADL that is designed to guide the network to focus on small BMs during training and mitigate severe data imbalance between small BMs and healthy tissue areas enhanced the detection and segmentation performances for small BMs but did not deteriorate the performance for large BMs compared with the default nnU-Net model.

Studies on developing and testing deep-learning-based models for automated BM detection and segmentation on post-contrast T1w or multi-contrast MRI images have been growing rapidly from early feasibility-type studies based upon a small size of data from a single institute^9,10 to recent investigations of the challenges in detection and segmentation of small BMs^4,7,8,11,12 and testing with a large data set from multi-institutes.⁶ A recent study showed BM size stratified performance analysis in which detection sensitivities were 1.0, 0.9, 0.70, and 0.93 for BMs >6 mm, 3 to 6 mm, <3 mm, and all sizes with a mean FP rate per case of 3.0 on 118 BMs from 27 patients achieved by the 3D DeepMedic model trained using 445 BMs from 108 patients.²⁸ Overall, detection of BM greater than 0.1 cm³ (>6 mm ESD) could achieve sensitivity of approximately 90% or higher at a patient-wise FP rate of 1.0.¹¹ For detection and segmentation of small BMs, there were a few investigations with data sets containing more than 500 scans from a single institute.^4,11,29 One that trained a 3D patch-based U-Net model on a data set of 509 patients achieved 0.70 sensitivity and 0.74 DSC at an estimated FP rate per patient of 2.4 for the BM size of 0.06 cm³.^4,11 Another that trained the self-adaptive nnU-Net model using 1051 patients achieved 0.683 and 0.698 per-lesion sensitivity, and 0.30 and 0.71 DSC in segmentation for respective lesion sizes of <3 mm and 3 to 6 mm at 0.47 estimated FP rate per patient and 0.64 estimated false negative rate per patient.⁴ The study by Rudie et al²⁹ that developed a 3D U-Net trained with 463 MRI cases achieved detection sensitivities of 0.7 for all BMs, 0.964 for BMs > 6 mm, 0.767 for BMs 3 to 6 mm, and 0.147 for BMs < 3 mm with a mean FP rate per case of 0.46 validated on 100 testing cases. Detection and segmentation performances for small sizes of BMs using our extended nnU-Net model with ADL trained and tested using multi-institute data compared favorably with these studies (see FROC curves in Fig. 2 and Tables E4-E7). Overall performance of our model was also superior compared with another study that trained a 3D network using multi-institute data of 407 patients and which achieved 0.91 sensitivity and 0.73 DSC at a FP rate of 1.9 for all sizes of BMs.⁶

Prior studies investigated the sensitivity of BM detection from MRI by radiologists performing as expert observers. Sunwoo et al³⁰ showed that 4 observers (2 radiology residents and 2 neuroradiologists) achieved an average BM detection sensitivity of 0.893 with 0.18 FP rate per case on 134 BMs from 30 patients with BMs and 30 patients without BMs when BMs smaller or equal to 2 mm diameter were excluded. When BMs smaller or equal to 2 mm in diameter were included, the sensitivity dropped to 0.776. Another study showed that 2 observers (neuroradiologists) achieved sensitivity values of 0.632, 0.908, 0.984, and 0.879 for BMs smaller than 3 mm, 3 to 6 mm, larger than 6 mm, and all sizes of 1416 BMs from 100 testing patients.²⁹ Further studies are needed to directly compare performances of AI tools and human experts using the same data from multiple institutes and expert observers.

This AI tool could assist radiologists in detecting multiple BMs accurately and also improve the efficiency and reproducibility of clinicians in delineating multiple BMs to support the selection of therapeutic strategies and planning of treatments. Given the relatively large number of BMs potentially present in the brain, such assistance could have a large-scale impact on diagnosis, treatment, and longitudinal therapy response assessment. Such a tool can have a significant impact in supporting appropriate clinical treatment decisions, resulting in the improvement of quality of life and survival of patients with multiple intracranial metastases. A sufficiently accurate tool may significantly reduce the time and complexity associated with both the review of images as well as delineation of anatomy for radiosurgery treatment planning. A full understanding of clinical usefulness requires testing and investigation in the clinical setting, which will be a focus of future investigations.

There are several technical limitations that were not addressed in this study. The impact of our data augmentation through metastasis pattern synthesis by the Generative Adversarial Network (GAN)-based networks was not analyzed. A separate systematic study would be needed to understand the usefulness of the data augmentation approach. Another limitation is that the ground-truth BM contours may be subject to intra- and inter-observer variations. Although we performed 2 levels of systematic data curation, the effect of imperfect ground-truth BM contours on our model performance could be investigated further. For example, some of the FP BMs may be identified later as TP BMs on subsequent follow-up imaging. We used 2 mm ESD as a main target minimum BM size, and major data curation was performed mostly for BMs larger than 2 mm ESD. Although we further stratified the system performance with a 1 mm ESD cutoff or without any cutoff at all to include BMs smaller than 2 mm ESD in the analysis as shown in Supplementary Materials, the quality of our ground-truth GTV contours for BMs smaller than 2 mm ESD may be limited compared with that for BMs larger than 2 mm. For future work, we will investigate the clinical utility with multiuser studies in which the trade-off between sensitivity and FP rate and operation point selection will be considered.

Conclusion

In this study, we have developed an automated system for BM detection and segmentation on 3D post-contrast T1-weighted MRI by extending the nnU-Net framework. Although the nnU-Net is designed to provide competitive out-of-the-box solutions for many medical image segmentation problems based on its self-adaptive mechanism, we observed that BM detection and segmentation performance can be substantially further enhanced through task-specific extensions. With a high small BM detection sensitivity and a controlled rate of FP per patient, such a system could improve early-stage BM identification and investigation. Further impact analysis on real-world utility of the developed system for improving the efficiency and consistency of clinical practice is warranted.

Supplementary Material

Supplementary material associated with this article can be found in the online version at doi:10.1016/j.ijrobp.2024.07.2318.

Data Sharing Statement:

Research data from University of Michigan, New York University and University of North Carolina will be available for public after August 31, 2026.