TumorSynth: Integrated Brain Tumor and Tissue Segmentation on Brain MRI Scans of Any Resolution and Contrast

Abstract

Purpose

To develop and validate a deep neural network that simultaneously segments brain tumors and anatomic structures, regardless of the contrast and resolution of the input scans, and can effortlessly adapt to unseen modalities.

Materials and Methods

The authors included various MRI scans from patients with and without brain tumors from four different datasets. Patient data were divided into a training set and a test set. The authors’ method, TumorSynth, combines a Bayesian generative model and a deep learning segmentation model. The generative model creates paired synthetic labels and images with simulated tumors and brain tissues, providing a rich dataset for training the segmentation model. The authors quantitatively compared its performance with that of other widely used methods by calculating Dice similarity coefficients (DSCs).

Results

A total of 1971 patients with and without tumors were included in the study (training set, n = 351 patients; test set, n = 1620 patients). The median DSCs for segmentation (authors’ method vs reference standard) were 0.89 (IQR, 0.83–0.95; P < .001) for the unaffected brain volume and 0.89 (IQR, 0.84–0.94; P < .001) for the tumor region. There were no differences in parcellation performance when an MRI sequence was missing (P = .07). In cross-modality validation, the authors’ method achieved DSC values of 0.88 for apparent diffusion coefficient, 0.85 for diffusion-weighted imaging, 0.80 for susceptibility-weighted imaging, and 0.79 for fractional anisotropy images. The authors observed a 4% false-positive rate when processing tumor-free MR images.

Conclusion

The authors developed a deep neural network for brain tumor and tissue segmentation, validated its performance across standard structural MRI sequences, and determined its generalizability to unseen data.

Keywords: Segmentation, Neuro-Oncology, CNS, Deep Learning, Neurosurgery

Supplemental material is available online for this article.

© RSNA, 2026

Summary

We developed TumorSynth, a three-dimensional network for simultaneous segmentation of brain structures and tumors, which demonstrated high accuracy across standard structural MRI sequences and robust generalizability on unseen data.

Key Points

• ■ TumorSynth, trained on 1404 presurgical brain MRI sequences (351 patients), achieved a median Dice similarity coefficient of 0.89 for tumor and normal tissue segmentation on a held-out test set of 1620 patients (14 408 sequences), with a low false-positive rate of 4% on healthy scans.

• ■ Volume differences between reference standard segmentation and TumorSynth predicted volumes in unaffected brain and tumor areas were 0.007 and 0.08, respectively.

• ■ Dice similarity coefficients between TumorSynth and reference standard segmentations for common clinical sequences ranged from 0.80 (susceptibility-weighted imaging) to 0.93 (fluid-attenuated inversion recovery), with particularly high Dice similarity coefficient values for fluid-attenuated inversion recovery and T2-weighted sequences (0.93 and 0.92, respectively).

Introduction

Gliomas are the most common primary brain tumors and are associated with a poor prognosis. Accurate tumor delineation at MRI is an essential step for effectively planning surgery or radiation therapy. Joint segmentation of tumors and surrounding tissue structures can further improve strategies such as maximal safe resection, radiation therapy planning (1), and treatment monitoring. The development of deep learning models trained on the Multimodal Brain Tumor Image Segmentation (BraTS) has improved the accuracy of glioma segmentation (2). However, the presence of a space-occupying lesion distorts the normal architecture and connectivity of the nearby brain regions, as well as remote areas linked through white matter tracts (3,4), which poses great challenges to the segmentation and parcellation of other healthy tissues. The absence of effective and convenient (nontumorous) whole-brain parcellation models is hampering comprehensive patient therapy planning.

One of the reference standards with which researchers are trying to address this issue is virtual brain grafting, which attempts to segment brain tissue in tumor-adjacent regions by first generating a lesion-free T1-weighted image through extensive nonlinear registration to template the brain and then performing lesion filling followed by conventional parcellation tools (eg, FreeSurfer, GIF) (5). Despite its utility, virtual brain grafting is computationally intensive because of its complex lesion filling pipeline, and its multiple registration processes are prone to accumulating errors. Artificial intelligence–based models have sought to address these limitations through domain adaptation, which transfers knowledge from separate tasks (6) or generates realistic pathology data to support segmentation (7,8). However, the performance of these models often heavily depends on specific imaging sequences, restricting generalizability across diverse modalities.

In this article, we introduce TumorSynth, a novel framework that simultaneously segments both healthy brain regions and tumor subregions in a three-dimensional space. TumorSynth uses a Gaussian mixture model–based data augmentation generative model within a Bayesian segmentation framework that is trained on dynamically sampled synthetic scans with minimal and limited real data exposure. We adopted a generative method similar to SynthSeg (9,10), which applies Gaussian mixture model–based data augmentation and uses domain randomization by randomly sampling domain attributes during the dataset creation (11), but expanded it into space-occupying lesions to achieve robust tissue segmentation across diverse scanners (9,12).

Materials and Methods

Our retrospective, Health Insurance Portability and Accountability Act–compliant study was approved by the institutional review board of each site (identification no. KY2022-078-04). The requirement to obtain patient informed consent was waived owing to the retrospective nature of the study.

Study Patients and Datasets

In this study, we collected four distinct datasets—the BraTS, University of California, San Francisco, Tiantan Hospital, and a Tumor-free dataset—comprising data from a total of 1971 patients, including pretreatment patients with glioma and patients without tumors (see Table 1). Of these, 30 patients from Tiantan Hospital and 90 patients without tumors were previously studied in a brain parcellation study (13). Inclusion and exclusion criteria were listed in the corresponding publications. Data from only 351 patients, randomly sampled from the BraTS dataset, were used for training. The remaining data were used to test the performance of TumorSynth across various scenarios.

Table 1:

Breakdown of Data Used in Our Study by Dataset

Parameter	BraTS			UCSF		Tiantan Hospital		Tumor-free
	Grade II-IV Gliomas			Grade II-IV Gliomas		Grade II-IV Gliomas		Nontumor
	Training	Complete Test	Incomplete Test	Incomplete Test	Complete Test	Incomplete Test	Complete Test	Complete Test
No. of total sequences	1404	3600	2880	3187	4000	329	252	160
No. of patients	351	900	900	500	500	60	42	160
No. of patients with T1-weighted sequences	351	900	751	382	500	52	42	160
No. of patients with T1-weighted contrast-enhanced sequences	351	900	702	425	500	50	42	0
No. of patients with T2-weighted sequences	351	900	738	389	500	57	42	0
No. of patients with FLAIR sequences	351	900	689	422	500	55	42	0
No. of patients with ADC sequences	0	0	0	403	500	43	42	0
No. of patients with SWI sequences	0	0	0	392	500	48	42	0
No. of patients with DWI sequences	0	0	0	387	500	12	0	0
No. of patients with FA sequences	0	0	0	387	500	12	0	0

Note.—A complete test refers to all modalities being used as input, and an incomplete test simulates the absence of certain modalities. ADC = apparent diffusion coefficient, BraTS = Multimodal Brain Tumor Image Segmentation, DWI = diffusion-weighted imaging, FA = fractional anisotropy, FLAIR = fluid-attenuated inversion recovery, SWI = susceptibility-weighted imaging, UCSF = University of California, San Francisco.

The BraTS 2021 dataset (14) included scans from 900 patients with T1-weighted, T1-weighted contrast-enhanced, T2-weighted, and fluid-attenuated inversion recovery scans. Tiantan Hospital dataset is a clinical dataset of brain tumor MRI scans obtained from Tiantan Hospital that included data from 60 patients (37 with grade II [60%] and 23 with grade IV gliomas [40%]), with different MRI sequences available for each patient.

University of California, San Francisco Preoperative Diffuse Glioma MRI dataset included scans from 500 patients (15) (55 with grade II [11%], 42 with grade III [9%], and 403 with grade IV [80%] diffuse gliomas), including T1-weighted, T1-weighted contrast-enhanced, T2-weighted, and fluid-attenuated inversion recovery as well as diffusion-weighted imaging, fractional anisotropy maps, susceptibility-weighted imaging, and apparent diffusion coefficient maps. The Tumor-free dataset comprised 160 tumor-free T1-weighted scans from the NeuroMorphometrics database.

Reference-Standard Segmentation

Our reference-standard segmentation encompassed two main components: tumors and tumor-free brain tissue. For tumor segmentations, we adhered to the BraTS three-class criteria (2), which define three distinct tumor regions: the enhancing tumor, representing the actively growing tumor; the tumor core, which includes the enhancing tumor, nonenhancing tumor, and necrotic tumor but excludes edema; and the whole tumor, which is the entire tumor mass, including the enhancing tumor, tumor core, and surrounding edema, as visible on fluid-attenuated inversion recovery scans. For the BraTS and University of California, San Francisco Preoperative Diffuse Glioma MRI datasets, we used the provided reference standards. For clinical data from Tiantan Hospital, we employed a pretrained supervised model based on nnU-Net (16) to generate initial tumor segmentations. These segmentations were blinded, subsequently reviewed, and manually corrected by senior radiologists (including J.L., with 5 to 10 years of experience) using all available MRI sequences.

For segmentation of the normal brain tissue, we first applied virtual brain grafting to inpaint the tumor region (5). We then obtained automated tissue segmentations by applying three well-known methods: FreeSurfer (17), GIF (13), and SynthSeg (9). The three masks were fused using majority voting to enhance accuracy/robustness. Finally, the tumor region labels were reintegrated into the whole-brain parcellation labels. All final segmentations were thoroughly reviewed and corrected by a board-certified neuroradiologist (J.L., with 5 to 10 years of experience).

Synthetic Data Generation

In this work, we trained our segmentation network on synthetic data generated by using a two-step generative approach (9), as shown in Figure 1. The process involved initial label map augmentation followed by synthetic image generation. Notably, we adopted a domain-randomization strategy (11), where all the parameters of the generative model including MRI contrast, resolution, shape, and artifacts were fully randomized on the fly (9).

Figure 1:

Synthetic data generation process overview. Synthetic images were generated from a set of randomly selected templates. The resulting synthesized images and their corresponding segmentations were then used to train our network (12). L_avt-k = average top-K loss, L_dice+ce = sum of the cross-entropy (CE) and Dice loss.

Label Map Augmentation

The label map augmentation process starts with the random selection of a segmentation template from the training dataset. The selected label map is then spatially deformed in three-dimensional space by using a combination of scaling, rotation, shearing, translation, and nonlinear warping transforms to increase morphologic variability. The nonlinear transforms are implemented by using a diffeomorphic stationary velocity field, which is created by sampling a small Gaussian-distributed vector field, resampling it to the full image size, and integrating it with the scale-and-square approach (18). The deformed label map is then used to initiate the synthetic image generation process.

Synthetic Image Generation

This process is guided by a Gaussian mixture model, which randomly assigns a predetermined mean and SD distribution to each label from a label map on the basis of prior knowledge from various MRI sequences. If S is the deformed label map with C classes, and θ_G = {μ_c,Σ_c}_1≤c≤C groups all the parameters of the Gaussian mixture model, the preliminary image G can be expressed as $p\left(G\mid S,{\theta }_G\right)={\displaystyle {\prod }_{j=1}^{J}N\left(G_j;{\text{μ}}_{S_j},{\text{Σ}}_{S_j}\right)}$, where j is each voxel of the image and J the total number of voxels of the image, such that each obtained image G presents a random contrast (10). Although the synthesized images may appear unrealistic, this strategy boosts the generalizability of the downstream segmentation network by forcing it to learn domain-independent features. To complete our generative model, we further augmented the resulting volumes with random blurring to simulate different orientations (axial/sagittal/coronal), section thickness, section spacing, bias field corruption, and brightness augmentation.

TumorSynth Method Description

TumorSynth is a contrast-agnostic cascaded network designed for simultaneous tumor segmentation and brain parcellation. Recent research has shown that nnU-Net consistently outperforms transformer-based vision transformer models in medical imaging applications (19). Building on this, we adapted the nnU-Net version 2 as the segmentation backbone for our approach (20). We implemented a two-stage strategy with a cascaded network architecture, as shown in Figure 2. The primary network segment focuses on identifying normal brain tissue and the outer boundary of the tumor, and the secondary cascaded network is dedicated to segmenting the more intricate inner structures of the tumor.

Figure 2:

Cascaded architecture for tumor and tissue segmentation. For each input image, the first network segments the brain anatomy and identifies potential tumors. The second network further classifies the tumors into three specific classes. If multimodal data are available, postprocessing will be applied to refine the tumor structure.

For the primary network, we used the average top K loss (21) as the loss function, which is calculated as the average of the top K largest losses within the sample set z. Here, L_{z} represents the individual loss function, L_{avt-k} denotes the average top K loss, and f is the prediction function that maps input samples to predicted labels. The loss function ι is the sum of the cross-entropy and Dice loss: $L_{avt\text{-}k}\left(L_z(f)\right))\frac{1}{K}{\displaystyle {\sum }_{k=1}^K{\iota }_{[k]}\left(f\right)}$. This approach helps to protect minority class samples and improves robustness against noise. For the secondary network, we applied region-based training with the Dice and cross-entropy loss function, which increases the coherence of tumor internal structures (22).

The network can perform inference with a single input sequence, but it also supports multimodal inputs. For multimodal inputs, the corresponding segmentations are merged at various network levels by linearly combining the soft probability maps of each input by using level-specific weights. In the primary network, T2-weighted and fluid-attenuated inversion recovery sequences (if available) are prioritized, because their tissue contrasts enable clearer differentiation between potential tumors and surrounding areas. In the secondary network, greater weight is assigned to T1-weighted contrast-enhanced images (if available), because they are particularly effective for predicting internal tumor structures (23).

Validation

To evaluate the performance of TumorSynth, we conducted a comprehensive comparison in five steps. First, we assessed the performance of conventional brain parcellation tools and their limitations when applied to patients with tumors. The tools used in this evaluation included established methods such as FreeSurfer (17), GIF (13), and SynthSeg (9). We specifically examined how these tools perform parcellation of tumor-affected brains.

Then, we incorporated tumor-filling approaches to these tools to compare normal brain parcellations in patients with brain tumors against TumorSynth parcellations. Because conventional tools are not designed for parcellating brains with tumors, we used the lesion-filling methods of virtual brain grafting to reconstruct a tumor-free brain (5) before performing parcellation. We then masked out the tumor label from the obtained parcellation results to create a complete segmentation. In addition, we trained a supervised multimodal model with the identical real-image dataset that was used to train TumorSynth. The model, based on nnU-Net (22,24), was trained on 351 matched sets of T1-weighted, T2-weighted, T1-weighted contrast-enhanced, and fluid-attenuated inversion recovery sequences. The selected regions of interest cover a range of critical anatomic structures pivotal in neuroimaging analysis, including white matter, cortical gray matter, lateral ventricle, thalamus, putamen, brainstem, hippocampus (with paired structures averaged between the two sides), whole tumor, tumor core, and necrotic tumor.

Third, we performed experiments with missing sequences. We simulated scenarios where each of the four structural MRI sequences had a 25% chance of being absent (see Fig 3). We assessed the performance of TumorSynth in brain structure segmentation with incomplete imaging sequences and compared it with that of the previously mentioned models. For replacing missing sequences, FreeSurfer and GIF used T2-weighted sequences when T1-weighted sequences were unavailable and resorted to random sequences if both were missing. SynthSeg, which claims contrast agnosticism, applied majority voting across all available inputs to generate a single output. Whereas fully supervised models need all four sequences simultaneously, we padded the missing sequences for comparison. TumorSynth, however, is designed to operate with a single input and can flexibly expand to incorporate multiple modalities by merging their segmentations in a single output.

Figure 3:

Image shows performance of TumorSynth on images with example patients with tumors. Each row represents a different patient, with a varying number of input images or sequences. Postprocessing to refine tumor structure is applied when multimodal data are available. N/A indicates that a particular modality was not available during inference. Input sequences include T1-weighted (T1w), T1-weighted contrast-enhanced (T1c), fluid-attenuated inversion recovery (FLAIR), and T2-weighted (T2w) scans. The rightmost column shows the TumorSynth predicted mask, including normal brain tissue parcellation and the three tumor areas: the enhancing tumor (dark pink), tumor core (white), and the whole tumor (light pink).

In the next experiment, we assessed the quality of tumor segmentation. In this comparison, TumorSynth was compared with a sparsified-trained model (16), which was trained on real scans and has demonstrated strong performance in large-scale datasets. Finally, we aimed to mimic clinical scenarios and tested the performance of TumorSynth on clinical-related sequences and patients without tumors.

Statistical Analysis

We employed the Dice similarity coefficient (DSC), Hausdorff surface distance, and volume difference. The DSC quantifies the overlap between two segmentations, whereas the Hausdorff surface distance assesses the greatest distance between the boundaries of the predicted and reference standard segmentations. The volume difference is a calculation of the difference between the predicted and reference-standard volumes.

We compared each computed tumor and brain tissue mask with its corresponding reference-standard mask for each dataset. We then calculated the mean and SD for all evaluation scores. We used a two-tailed, unequal variance paired t test and McNemar test as appropriate for each metric to identify significant performance differences between TumorSynth and the other methods. Statistical analyses were performed by one author (W.J.) using R (version 4.4.2; R Foundation), with a significance level set at a P value less than .05 for all experiments, adjusting for multiple comparisons with the Bonferroni method.

Results

Patient Characteristics

In our study, we used four distinct datasets—the BraTS, University of California, San Francisco, Tiantan Hospital, and Tumor-free datasets—that encompassed a total of 1971 patients and 15 812 presurgical imaging sequences, including patients with glioma and patients without tumors (see Table 1). Of these, data from only 351 patients in the BraTS dataset were used as the training set, and the remaining data (n = 1620 patients) were reserved to validate the performance of TumorSynth across various scenarios. On the test set, the model achieved overall median DSCs of 0.89 (IQR = 0.06, P < .001) for segmentation of the unaffected brain volume and 0.89 (IQR = 0.05, P < .001) for segmentation of the tumor region. The patients’ clinical and demographic characteristics are shown in Table S1.

Segmentation Quality in Native Scans with Tumors

Results of the comparison of brain parcellation performance in patients with tumors are presented in Table 2. Because SynthSeg, FreeSurfer, and GIF were originally developed for tumor-free brains, the performance of these tools was suboptimal compared with that of TumorSynth. This limitation is demonstrated in Figure 4, which shows that the presence of a tumor considerably distorted the structural integrity of the brain, making these tools ineffective at accurately delineating the boundaries of adjacent tissues affected by the tumor.

Table 2:

Comparison of Brain Tissue Parcellation among Patients with Tumors across Four Tools—TumorSynth, SynthSeg, FreeSurfer, and GIF—and Key Brain Regions

Brain Region	TumorSynth		SynthSeg		FreeSurfer		GIF
	DSC	HSD	DSC	HSD	DSC	HSD	DSC	HSD
Brainstem	0.92 ± 0.01	7.32 ± 1.52	0.76 ± 0.11	13.26 ± 13.37	0.78 ± 0.14	9.12 ± 10.58	0.78 ± 0.12	14.12 ± 13.58
White matter	0.91 ± 0.02	9.16 ± 1.56	0.79 ± 0.12	14.97 ± 18.95	0.82 ± 0.13	10.84 ± 15.33	0.8 ± 0.13	15.84 ± 17.33
Hippocampus	0.87 ± 0.02	6.45 ± 2.83	0.42 ± 0.52	55.67 ± 30.77	0.48 ± 0.53	49.15 ± 29.75	0.43 ± 0.58	57.15 ± 31.47
Cerebral cortex	0.88 ± 0.06	8.26 ± 2.81	0.72 ± 0.12	17.33 ± 19.60	0.75 ± 0.12	16.84 ± 18.34	0.74 ± 0.13	17.25 ± 19.74
Putamen	0.87 ± 0.02	6.21 ± 2.15	0.55 ± 0.45	33.66 ± 30.62	0.53 ± 0.48	35.63 ± 30.91	0.55 ± 0.45	36.43 ± 28.92
Thalamus	0.85 ± 0.02	8.62 ± 2.13	0.44 ± 0.53	37.58 ± 42.20	0.42 ± 0.52	37.56 ± 2.49	0.48 ± 0.48	32.36 ± 32.49
Lateral ventricle	0.86 ± 0.05	7.76 ± 2.58	0.74 ± 0.13	17.05 ± 15.44	0.68 ± 0.14	18.5 ± 17.45	0.71 ± 0.12	14.86 ± 18.44

Note.—Data are means ± SDs. All results from four tools are significantly different from TumorSynth results (P < .0006). DSC = Dice similarity coefficient, HSD = Hausdorff surface distance.

Figure 4:

Images show comparison of brain tissue parcellation in patients with tumors across different tools: (A) original input image, (B) reference-standard segmentation, (C) TumorSynth results, (D) FreeSurfer outputs, (E) GIF outputs, and (F) SynthSeg outputs. In segmentations in panels B and C, the tumor region is highlighted in white, whereas in panels D, E, and F, the tumor region has been masked out. Tools shown in D, E, and F were not designed to be robust against tumors, resulting in false positives and the missing parts of normal brain tissue.

Brain Parcellation with Lesion Filling with Full Sequences

Results of the comparison of the performance of TumorSynth against that of the fully supervised model, FreeSurfer, GIF, and SynthSeg for patients with tumors are presented in Table 3. TumorSynth consistently outperformed the conventional tools (FreeSurfer, GIF, and SynthSeg) across the seven evaluated brain structures in terms of DSC, Hausdorff surface distance, and volume difference (P < .0006). Notably, TumorSynth achieved mean DSCs of 0.92, 0.92, and 0.87 for the cerebral cortex, white matter, and hippocampus, respectively. These results were not significantly different from those of the fully supervised model, for which the DSCs were 0.92, 0.93, and 0.87 for the same structures (P = .09) (Table S2).

Table 3:

Comparison of Brain Tissue Parcellation Performance between Popular Tools—SynthSeg, FreeSurfer, and GIF with Tumor-filling Techniques—and TumorSynth as Well as a Fully Supervised Model, in Key Brain Regions

Brain Region	TumorSynth			SynthSeg			FreeSurfer			GIF			Fully Supervised
	DSC	HSD	VDF	DSC	HSD	VDF	DSC	HSD	VDF	DSC	HSD	VDF	DSC	HSD	VDF
	Brain Structure Segmentation with Full Sequences on Scans with Lesions
Brainstem	0.92 ± 0.01	7.32 ± 1.52	0.071 ± 0.07	0.83* ± 0.11	10.22* ± 13.37	0.121* ± 0.12	0.79* ± 0.14	9.12* ± 10.58	0.13* ± 0.14	0.78* ± 0.12	14.15* ± 13.58	0.148* ± 0.13	0.92 ± 0.12	10.12* ± 3.58	0.111* ± 0.13
White matter	0.92 ± 0.02	9.16 ± 1.56	0.062 ± 0.06	0.83* ± 0.12	11.55* ± 12.95	0.106* ± 0.12	0.82* ± 0.13	10.84* ± 15.33	0.132* ± 0.12	0.8* ± 0.13	15.84* ± 16.28	0.136* ± 0.14	0.93 ± 0.13	11.83* ± 7.36	0.094 ± 0.14
Hippocampus	0.87 ± 0.02	6.45 ± 2.83	0.072 ± 0.05	0.73* ± 0.52	15.68* ± 10.77	0.105* ± 0.15	0.74* ± 0.53	19.15* ± 12.75	0.149* ± 0.16	0.75* ± 0.58	22.15* ± 21.87	0.138* ± 0.15	0.87 ± 0.58	10.15* ± 8.54	0.099 ± 0.05
Cerebral cortex	0.92 ± 0.06	8.26 ± 2.81	0.064 ± 0.04	0.82* ± 0.12	12.34* ± 9.60	0.108* ± 0.13	0.83* ± 0.12	16.84* ± 13.34	0.151* ± 0.15	0.82* ± 0.13	17.25* ± 19.74	0.138* ± 0.16	0.91 ± 0.13	13.25* ± 12.14	0.109* ± 0.16
Putamen	0.87 ± 0.02	6.21 ± 2.15	0.054 ± 0.04	0.75* ± 0.45	13.64* ± 10.62	0.124* ± 0.13	0.74* ± 0.48	10.63* ± 8.91	0.156* ± 0.15	0.75* ± 0.45	16.43* ± 9.92	0.143* ± 0.18	0.87 ± 0.25	12.43* ± 8.47	0.083 ± 0.08
Thalamus	0.86 ± 0.02	8.62 ± 2.13	0.072 ± 0.06	0.74* ± 0.53	17.56* ± 12.20	0.124* ± 0.12	0.72* ± 0.52	19.56* ± 15.49	0.151* ± 0.14	0.73* ± 0.48	22.36* ± 12.49	0.124* ± 0.15	0.87 ± 0.28	11.36* ± 7.96	0.084 ± 0.05
Lateral ventricle	0.87 ± 0.05	7.76 ± 2.58	0.064 ± 0.07	0.78* ± 0.13	17.03* ± 15.44	0.147* ± 0.22	0.78* ± 0.14	18.5* ± 17.45	0.176* ± 0.23	0.77* ± 0.12	14.86* ± 18.44	0.176* ± 0.25	0.88 ± 0.12	9.86* ± 5.41	0.076 ± 0.15
	Brain Structure Segmentation with Missing Sequences on Scans with Lesions
Brainstem	0.92 ± 0.01	8.52 ± 3.58	0.101 ± 0.07	0.82* ± 0.11	13.26* ± 13.35	0.141* ± 0.12	0.78* ± 0.14	13.02* ± 10.52	0.121* ± 0.14	0.75* ± 0.12	15.32* ± 13.78	0.128* ± 0.13	0.81* ± 0.12	15.12* ± 17.95	0.122 ± 0.13
White matter	0.91 ± 0.02	10.65 ± 4.56	0.075 ± 0.06	0.81* ± 0.12	14.97* ± 18.93	0.149* ± 0.12	0.82* ± 0.13	12.84* ± 16.93	0.112* ± 0.12	0.80* ± 0.13	15.74* ± 17.33	0.123* ± 0.14	0.80* ± 0.14	15.34* ± 16.73	0.129* ± 0.13
Hippocampus	0.87 ± 0.02	9.89 ± 6.38	0.067 ± 0.05	0.79* ± 0.52	25.67* ± 20.75	0.232* ± 0.25	0.76* ± 0.53	29.15* ± 19.78	0.239* ± 0.26	0.75* ± 0.58	27.35* ± 21.47	0.227* ± 0.25	0.73* ± 0.58	19.15* ± 11.47	0.163* ± 0.22
Cerebral cortex	0.9 ± 0.06	10.65 ± 8.19	0.064 ± 0.04	0.81* ± 0.12	17.33* ± 19.62	0.138* ± 0.13	0.79* ± 0.12	16.84* ± 18.64	0.132* ± 0.15	0.76* ± 0.13	17.69* ± 19.54	0.132* ± 0.16	0.75* ± 0.13	17.85* ± 19.94	0.155* ± 0.24
Putamen	0.87 ± 0.02	11.21 ± 9.58	0.074 ± 0.04	0.74* ± 0.45	23.66* ± 20.64	0.244* ± 0.23	0.76* ± 0.48	25.63* ± 20.36	0.253* ± 0.25	0.78* ± 0.45	26.46* ± 18.59	0.233* ± 0.28	0.72* ± 0.45	20.49* ± 18.32	0.189* ± 0.22
Thalamus	0.85 ± 0.02	11.69 ± 8.35	0.084 ± 0.06	0.74* ± 0.53	27.58* ± 22.26	0.229* ± 0.22	0.72* ± 0.52	27.56* ± 18.23	0.244* ± 0.24	0.70* ± 0.48	22.36* ± 22.12	0.314* ± 0.25	0.68* ± 0.48	28.38* ± 22.48	0.195* ± 0.23
Lateral ventricle	0.86 ± 0.05	9.78 ± 4.86	0.079 ± 0.07	0.77* ± 0.13	17.05* ± 15.43	0.167* ± 0.22	0.73* ± 0.14	18.98* ± 17.23	0.166* ± 0.23	0.72* ± 0.12	17.87* ± 19.49	0.156* ± 0.25	0.75* ± 0.32	20.53* ± 18.44	0.187* ± 0.27
	Brain Structure Segmentation on Tumor-Free Scans
Brainstem	0.92 ± 0.01	7.64 ± 2.59	0.07 ± 0.09	0.88* ± 0.11	7.46 ± 6.37	0.103* ± 0.12	0.87* ± 0.01	10.14* ± 3.20	0.082 ± 0.02	0.89* ± 0.03	10.52* ± 1.47	0.075 ± 0.07	0.78* ± 0.12	14.12* ± 13.58	0.101 ± 0.13
White matter	0.93 ± 0.02	9.64 ± 2.13	0.067 ± 0.06	0.9* ± 0.12	9.19 ± 5.95	0.088 ± 0.12	0.9 ± 0.02	12.97* ± 0.95	0.076 ± 0.02	0.89 ± 0.02	12.06* ± 1.61	0.078 ± 0.06	0.8* ± 0.13	15.84 ± 17.33	0.094 ± 0.14
Hippocampus	0.85 ± 0.02	6.89 ± 3.82	0.075 ± 0.07	0.84 ± 0.22	7.46* ± 6.77	0.097 ± 0.45	0.82* ± 0.02	7.33 ± 1.60	0.078 ± 0.03	0.83 ± 0.04	8.26* ± 1.81	0.067 ± 0.04	0.63* ± 0.38	37.15* ± 31.47	0.121* ± 0.45
Cerebral cortex	0.93 ± 0.06	8.68 ± 1.98	0.074 ± 0.09	0.93 ± 0.12	9.75 ± 9.60	0.102* ± 0.13	0.91 ± 0.02	8.73 ± 2.19	0.084 ± 0.02	0.92 ± 0.04	8.4 ± 2.07	0.063 ± 0.02	0.74* ± 0.13	17.25* ± 19.74	0.112* ± 0.16
Putamen	0.87 ± 0.02	8.21 ± 3.35	0.055 ± 0.03	0.84 ± 0.45	8.33* ± 5.62	0.098* ± 0.43	0.84 ± 0.02	7.58 ± 2.20	0.079 ± 0.02	0.81* ± 0.05	8.42* ± 2.19	0.071 ± 0.06	0.65* ± 0.35	36.43* ± 28.92	0.093* ± 0.48
Thalamus	0.87 ± 0.02	8.69 ± 0.43	0.077 ± 0.06	0.83* ± 0.53	8.26 ± 5.20	0.084 ± 0.52	0.84 ± 0.02	7.05 ± 1.44	0.083 ± 0.02	0.85 ± 0.05	7.75 ± 1.63	0.065 ± 0.07	0.68* ± 0.38	32.36* ± 32.49	0.114 ± 0.45
Lateral ventricle	0.89 ± 0.05	7.03 ± 2.53	0.065 ± 0.18	0.88 ± 0.13	11.37* ± 7.44	0.092 ± 0.22	0.9 ± 0.02	6.23 ± 0.12	0.093 ± 0.02	0.89 ± 0.05	6.51 ± 1.07	0.082 ± 0.05	0.71* ± 0.12	14.86* ± 18.14	0.106 ± 0.25

Note.—Data are means ± SDs. Three situations are simulated: full MRI sequences available in patients with tumors and missing MRI sequences in patients with tumors and then patients without tumors. DSC = Dice similarity coefficient, HSD = Hausdorff surface distance, VDF = volume difference.

^*Significant difference from TumorSynth results (P < .0006).

Brain Parcellation with Lesion Filling with Missing Sequences

In case of missing core sequences, TumorSynth outperformed other tools, particularly in the hippocampus, putamen, and thalamus regions. TumorSynth achieved mean DSCs of 0.92, 0.91, and 0.91 for the brainstem, white matter, and cerebral cortex, respectively, outperforming other methods, which showed more pronounced decreases in accuracy with missing sequences (P < .0006).

Tumor Structure Segmentation

We evaluated tumor segmentation using both full and missing sequences. The results are presented in Table 4.

Table 4:

Comparison of Inner Tumor Segmentation Tools, TumorSynth versus a Sparsified-trained Supervised Model, under Full and Missing Sequence Conditions (25% Probability of Being Absent)

Tumor Area	TumorSynth			Sparsified-trained
	DSC	HSD	VDF	DSC	HSD	VDF
	Tumor Structure Segmentation with Full Sequence
WT	0.9 ± 0.11	8.52 ± 10.82	0.08 ± 0.07	0.92 ± 0.11	9.11 ± 10.30	0.07 ± 0.07
TC	0.9 ± 0.06	9.06 ± 7.61	0.08 ± 0.06	0.92 ± 0.06	4.97* ± 6.95	0.08 ± 0.08
ET	0.81 ± 0.14	6.51 ± 5.83	0.08 ± 0.05	0.86* ± 0.10	6.67 ± 5.54	0.08 ± 0.04
	Tumor Structure Segmentation with Missing Sequences
WT	0.91 ± 0.12	8.53 ± 9.78	0.11 ± 0.08	0.90* ± 0.11	9.11 ± 10.30	0.09* ± 0.09
TC	0.89 ± 0.06	10.05 ± 7.89	0.09 ± 0.07	0.84* ± 0.16	14.37* ± 16.58	0.12* ± 0.13
ET	0.8 ± 0.14	8.56 ± 6.63	0.1 ± 0.08	0.76* ± 0.18	16.67* ± 15.24	0.15* ± 0.18

Note.—Values are presented as the mean ± SD. DSC = Dice score coefficient, ET = enhancing tumor, HSD = Hausdorff surface distance, TC = tumor core, VDF = volume difference, WT = whole tumor.

^*Significant difference from TumorSynth results (P < .003).

For full sequences, TumorSynth’s performance was comparable to that of the sparsified-trained model (16) on whole tumor and tumor core (P = .04 and P = .06 at the paired t test, respectively). However, in the enhancing tumor category, TumorSynth showed a lower DSC than the sparsified-trained model (P < .003 at the paired t test).

For missing core sequences, TumorSynth maintained consistent performance with a low SD, whereas the sparsified-trained model, heavily depending on T1-weighted contrast-enhanced images, showed a notable decrease in accuracy for tumor core and enhancing tumor (P < .003) (Table S3).

False-Positive Rate on Tumor-Free Scans

To validate the generalizability of TumorSynth, we evaluated its performance on images of patients without tumors. TumorSynth accurately identified the absence of tumor structures in these patients, with a low false-positive rate for detecting healthy brain tissue as tumor of 4% (seven of 160 cases). In comparison, the supervised method showed a higher false-positive rate of 16% (26 of 160 cases; P = .01 in the McNemar test).

Cross-modality Data Validation

Finally, we conducted cross-modality validation by also using noncore sequences such as apparent diffusion coefficient, diffusion-weighted imaging, susceptibility-weighted imaging, and fractional anisotropy (available for subsets of the University of California, San Francisco and Tiantan Hospital cohorts) for brain plus tumor. TumorSynth demonstrated superior adaptability, achieving average DSCs of 0.88, 0.85, 0.80, and 0.79 for these modalities, respectively, as shown in Figure 5 (Table S4).

Figure 5:

Graphs show the comparison of cross-modality performance average for brain tissue segmentation in patients with tumors across five tools—TumorSynth, SynthSeg, FreeSurfer, GIF, and a fully supervised model. The y-axis presents the Dice similarity coefficient (DSC) metrics. Only one input sequence was provided for each inference for this test, and zero padding was used when other input sequences were missing. TumorSynth was trained on T1-weighted (T1w), T1-weighted contrast-enhanced (T1ce), T2-weighted (T2w), fluid-attenuated inversion recovery (FLAIR), and domain randomized images and cross-validated on the other four modalities (apparent diffusion coefficient [ADC] maps, diffusion-weighted imaging [DWI], susceptibility-weighted imaging [SWI], and fractional anisotropy [FA] maps). SynthSeg was trained on T1-weighted and synthetic contrast images and then cross-validated on the other seven sequences. FreeSurfer and GIF were designed for T1-weighted images and validated on other sequences as well in this experiment. The fully supervised model was trained with multiple modalities (T1-weighted, T1-weighted contrast-enhanced, T2-weighted, and FLAIR scans) and used the same real scans as those used in TumorSynth training; however, it requires a full sequence to perform the inference, resulting in unsatisfactory performance in some situations.

Discussion

We introduce TumorSynth, a method for simultaneous brain tumor segmentation and healthy brain tissue parcellation. TumorSynth is designed to accurately delineate tumor boundaries while evaluating their spatial relationship with surrounding tissues. Our approach, TumorSynth, showed no significant performance difference compared with fully supervised methods with complete MRI sequences and is robust in handling missing sequences. Additionally, TumorSynth maintained a low false-positive rate when applied to patients without tumors. Finally, we evaluated the performance of TumorSynth in cross-domain settings. Despite not being trained on real images from the target domain, TumorSynth achieved DSCs on par with those of a fully supervised nnU-Net trained on that domain, while maintaining the flexibility to process images from other domains.

Traditional brain parcellation tools such as FreeSurfer, GIF, and SynthSeg typically require preparatory tumor-filling steps, including tumor masking and virtual brain grafting, to generate a lesion-free image and achieve acceptable brain parcellation results. However, the multiple registration steps involved in creating a lesion-free brain can lead to artifact accumulation and errors. Furthermore, the computational complexity of these processes—especially with nonlinear registration and separate parcellation and segmentation steps—makes them time-consuming.

A key strength of TumorSynth is its ability to adapt to missing or even unseen MRI sequences, a challenge for traditional models. TumorSynth achieves consistent performance by being trained dynamically on both real scans and domain-randomized synthetic images. This adaptability is especially crucial in clinical environments where comprehensive multimodal imaging may not always be available and where patients are scanned on different scanners with variable clinical protocols.

Our study has limitations. First, although TumorSynth performed well across multiple datasets, further validation on a broader spectrum of pathology cases, such as different tumor types and cases with multiple lesions or more severe anatomic distortion, is needed. Future work could also extend testing to other conditions with large lesions, such as infarcts, or postoperative scans. Although the self-adapting nature of TumorSynth improves its ability to learn from incomplete or limited data, its performance could be further improved by incorporating more real-world cases with reference-standard segmentations covering a wider range of pathology scenarios. Also, although the segmentation strategy on which TumorSynth is based has proven robust to varying image resolution and contrast (9), the lack of non–T1-weighted sequences for healthy controls prevented us from assessing its generalizability to additional MRI contrasts in this setting. Another area for future exploration is the impact of varying the ratio of real to synthetic scans during training, because different combinations may affect model performance. Finally, refining reference-standard labels through expert radiologist consensus could increase accuracy.

In conclusion, we introduced and validated TumorSynth, a model capable of simultaneously segmenting both healthy brain tissues and tumor regions. TumorSynth uses a Gaussian mixture model within a Bayesian segmentation framework that is trained on dynamically sampled synthetic scans with minimal real data exposure. Our method demonstrated strong performance in both healthy brain tissue and tumor segmentation, with a low false-positive rate across various imaging contrasts and modalities. The adaptability of TumorSynth to different imaging conditions and its robustness in challenging scenarios address key limitations of existing segmentation approaches, offering a promising solution for more reliable and efficient automated brain tumor analysis in clinical practice.

Disclosures of conflicts of interest

Please see ICMJE form(s) for author conflicts of interest. These have been provided as supplemental materials.