Skull-Stripping of Glioblastoma MRI Scans Using 3D Deep Learning

Abstract

Skull-stripping is an essential pre-processing step in computational neuro-imaging directly impacting subsequent analyses. Existing skull-stripping methods have primarily targeted non-pathologicallyaffected brains. Accordingly, they may perform suboptimally when applied on brain Magnetic Resonance Imaging (MRI) scans that have clearly discernible pathologies, such as brain tumors. Furthermore, existing methods focus on using only T1-weighted MRI scans, even though multi-parametric MRI (mpMRI) scans are routinely acquired for patients with suspected brain tumors. Here we present a performance evaluation of publicly available implementations of established 3D Deep Learning architectures for semantic segmentation (namely DeepMedic, 3D U-Net, FCN), with a particular focus on identifying a skull-stripping approach that performs well on brain tumor scans, and also has a low computational footprint. We have identified a retrospective dataset of 1,796 mpMRI brain tumor scans, with corresponding manually-inspected and verified gold-standard brain tissue segmentations, acquired during standard clinical practice under varying acquisition protocols at the Hospital of the University of Pennsylvania. Our quantitative evaluation identified DeepMedic as the best performing method (Dice = 97.9, Hausdorf f₉₅ = 2.68). We release this pre-trained model through the Cancer Imaging Phenomics Toolkit (CaPTk) platform.

1. Introduction

Glioblastoma (GBM) is the most aggressive type of brain tumors, with a grim prognosis in spite of current treatment protocols [1,2]. Recent clinical advancements in the treatment of GBMs have not increased the overall survival rate of patients with this disease by any substantial amount. The recurrence of GBM is virtually guaranteed and its management is often indefinite and highly case-dependent. Any assistance that can be gleaned from the computational imaging and machine learning communities could go a long way towards making better treatment plans for patients suffering from GBMs [3–11]. One of the first steps towards the goal of a good treatment plan is to ensure that the physician is observing only the areas that are of immediate interest, i.e., the brain and the tumor tissues, which would ensure better visualization and quantitative analyses.

Skull-stripping is the process of removing the skull and non-brain tissues from brain magnetic resonance imaging (MRI) scans. It is an indispensable pre-processing operation in neuro-imaging analyses that directly affects the efficacy of subsequent analyses. The effects of skull-stripping on subsequent analyses have been reported in the literature, including studies on brain tumor segmentation [12–14] and neuro-degeneration [15]. Manual removal of the non-brain tissues is a very involved and grueling process [16], which often results in inter- and intra-rater discrepancies affecting reproducibility in large scale studies.

In recent years, with theoretical advances in the field and with the proliferation of inexpensive computing power, including consumer-grade graphical processing units [17], there has been an explosion of deep learning (DL) algorithms that use heavily parallelized learning techniques for solving major semantic segmentation problems in computer vision. These methods have the added advantage of being easy to implement by virtue of the multitude of mature tools available, most notable of these being TensorFlow [18] and PyTorch [19]. Importantly, DL based segmentation techniques, which were initially adopted from generic applications in computer vision, have promoted the development of novel methods and architectures that were specifically designed for segmenting 3-dimensional (3D) MRI images [20–23]. DL, specifically convolutional neural networks, have been applied for segmentation problems in neuroimaging (including skull-stripping), obtaining promising results [16]. Unfortunately, most of these DL algorithms either require a long time to train or have unrealistic run-time inference requirements.

In this paper, we evaluate the performance of 3 established and validated DL architectures for semantic segmentation, which have out-of-the-box publicly-available implementations. Our evaluation is focusing on skull-stripping of scans that have clearly discernible pathologies, such as scans from subjects diagnosed with GBM. We also perform extensive comparisons using models trained on various combinations of different MRI modalities, to evaluate the benefit of utilizing multi-parametric MRI (mpMRI) data that are typically acquired in routine clinical practice for patients with suspected brain tumors on the final segmentation.

2. Materials and Methods

2.1. Data

We retrospectively collected 1,796 mpMRI brain tumor scans, from 449 glioblastoma patients, acquired during standard clinical practice under varying acquisition protocols at the Hospital of the University of Pennsylvania. Corresponding brain tissue annotations were manually-approved by an expert and used as the gold-standard labels to quantitatively evaluate the performance of the algorithms considered in this study.

In this study, we have chosen to take advantage of the richness of the mpMRI protocol that is routinely acquired in the cases of subjects with suspected tumors. Specifically, four structural modalities are included at baseline pre-operative time-point: native (T1) and post-contrast T1-weighted (T1Gd), native T2-weighted (T2), and T2-weighted Fluid Attenuated Inversion Recovery (FLAIR) MRI scans (Fig. 1). To conduct our quantitative performance evaluation we split the available data, based on an 80/20 ratio, in a training and testing subset of 1,432 and 364 mpMRI brain tumor scans, from 358 and 91 patients, respectively.

Fig. 1.

Example mpMRI brain tumor scans from a single subject. The original scans including the non-brain-tissues are illustrated in A, whereas the same scans after applying the manually-inspected and verified gold-standard brain tissue segmentations are illustrated in B.

2.2. Pre-processing

To guarantee the homogeneity of the dataset, we applied the same pre-processing pipeline across all the mpMRI scans. Specifically, all the raw DICOM scans obtained from the scanner were initially converted to the NIfTI [24] file format and then followed the protocol for pre-processing, as defined in the International Brain Tumor Segmentation (BraTS) challenge [12–14,25,26]. Specifically, each patient’s T1Gd scan was rigidly registered to a common anatomical atlas of 240 × 240 × 155 image size and resampled to its isotropic resolution of 1 mm³ [27]. The remaining scans of each patient (namely, T1, T2, FLAIR) were then rigidly co-registered to the same patient’s resampled T1Gd scan. All the registrations were done using “Greedy” (github.com/pyushkevich/greedy) [28], which is a CPU-based C++ implementation of the greedy diffeomorphic registration algorithm [29]. “Greedy” is integrated into the ITK-SNAP (itksnap.org) segmentation software [30,31], as well as the Cancer Imaging Phenomics Toolkit (CaPTk - www.cbica.upenn.edu/captk) [32,33]. After registration, all scans were down-sampled from a resolution of 240 × 240 × 155 to a resolution of 128 × 128 × 128, with anisotropic spacing of 1.875 × 1 875 × 1.25 mm³ with proper padding measures to ensure the anisotropic spacing is attained. Finally, the intensities found on each scan below the 2nd percentile and above the 95th percentile were capped, to ensure suppression of spurious intensity changes due to the scanner acquisition parameters.

2.3. Network Topologies

For our comparative performance evaluation, we focused on the most well-established DL network topologies for 3D semantic segmentation. The selection was done after taking into consideration their wide application in related literature, their state-of-the-art performance on other segmentation tasks, as established by various challenges [12,13,34,35], as well as their out-of-the-box applicability of their publicly available implementation, in low resource settings. The specific architectures included in this evaluation comprise the a) DeepMedic [20,21], b) 3D U-Net [22], and c) Fully Convolutional Neural network (FCN) [23].

DeepMedic [20,21] is a novel architecture, which came into the foreground after winning the 2015 ISchemic LEsion Segmentation (ISLES) challenge [34]. DeepMedic is essentially a 3D convolutional neural network with a depth of 11-layers, along with a double pathway to provide sufficient context and detail in resolution, simultaneously. In our study, we have applied DeepMedic using its default parameters, as provided in its GitHub repository github.com/deepmedic/deepmedic. As a post-processing step, we also include a hole filling algorithm.

As a second method, we have applied a 3D U-Net [22], an architecture that is widely used in neuroimaging. We used 3D U-Net with an input image size of 128 × 128 × 128 voxels. Taking into consideration our requirement for a low computational footprint, we reduced the initial number of “base” filters from 64 (as was originally proposed) to 16.

The third method selected for our comparisons was a 3D version of an FCN [23]. Similarly to the 3D U-Net, we used an input image size of 128 × 128 × 128 voxels. For both 3D U-Net and FCN, we used ‘Leaky ReLU’ instead of ‘ReLU’ for back-propagation with leakiness defined as α = 0.01. Furthermore, we used instance normalization instead of batch normalization due to batch size being equal to 1, due to the high memory consumption.

2.4. Experimental Design

Current state of the art methods typically use only the T1 modality for skull-stripping [36–41]. Here, we followed a different approach, by performing a set of experiments using various input image modality combinations for training DL models. Our main goal was to investigate potential contribution of different modalities, which are obtained as part of routine mpMRI acquisitions in patients with suspected brain tumors, beyond using T1 alone for skull-stripping. Accordingly, we first trained and inferred each topology on each individual modality separately to measure segmentation performance using different independent modalities, resulting in 4 models for each topology (“T1”, “T1Gd”, “T2”, “Flair”). Additionally, we trained and inferred models on a combination of modalities; namely, using a) both T1 and T2 modalities (“Multi-2”), and b) all 4 structural modalities together (“Multi-4”). The first combination was chosen as it has been shown that addition of the T2 modality improves the skull-stripping performance [14] and can also be used in cases where contrast medium is not used and hence the T1Gd modality is not available, i.e., in brain scans without any tumors. The second combination approach (i.e., “Multi-4”) was chosen to evaluate a model that uses all available scans. Finally, we utilized an ensembling approach (i.e., “Ens-4”) where the majority voting of the 4 models trained and inferred on individual modalities was used to produce the final label for skull-stripping.

We ensured that the learning parameters stayed consistent across each experiment. Each of the applied topologies needed different time for convergence, based on their individual parameters. For Deep-Medic, we trained with default parameters (as provided at the original github repository - 0.7.1 [commit dbdc1f1]) and it trained for 44 h. 3D U-Net and FCN were trained with Adam optimizer with a learning rate of 0.01 over 25 epochs. The number of epochs was determined according to the amount of improvement observed. Each of them trained for 6 h.

The average inference time for DeepMedic including the pre-processing and post-processing for a single brain tumor scan was 10.72 s, while for 3D U-Net and FCN was 1.06 s. These times were estimated based on the average time taken to infer on 300 patients. The hardware we used to train and infer were NVIDIA P100 GPUs with 12 GB VRAM utilizing only a single CPU core with 32 GB of RAM from nodes of the CBICA’s high performance computing (HPC) cluster.

2.5. Evaluation Metrics

Following the literature on semantic segmentation we use the following metrics to quantitatively evaluate the performance of the trained methods.

Dice Similarity Coefficient.

The Dice Similarity Coefficient (Dice) is typically used to evaluate and report on the performance of semantic segmentation. Dice measures the extent of spatial overlap between the predicted masks (PM) and the provided ground truth (GT ), and is mathematically defined as:

$$\mathrm{\text{Dice}}\text{=}\frac{2\left|\mathit{\text{GT}}\cap \mathit{\text{PM}}\right|}{\left|\mathit{\text{GT}}\right|+\left|\mathit{\text{PM}}\right|}*100$$ (1)

where it would range between 0–100, with 0 describing no overlap and 100 perfect agreement.

Hausdorf f₉₅.

Evaluating volumetric segmentations with spatial overlap agreement metrics alone can be insensitive to differences in the slotted edges. For our stated problem of brain extraction, changes in edges might lead to minuscule differences in spatial overlap, but major differences in areas close to the brain boundaries resulting in inclusion of skull or exclusion of a tumor region. To robustly evaluate such differences, we used the 95th percentile of the Hausdorff₉₅ distance to measure the maximal contour distance d, on a radial assessment, between the PM and GT masks.

$$H\mathit{\text{ausdor}}f{f}_{95}=\mathit{\text{percentile}}\left(d_{\mathit{\text{PM}},\mathit{\text{GT}}}\cup d_{\mathit{\text{GT}},\mathit{\text{PM}}},{95}^{\mathit{\text{th}}}\right)$$ (2)

3. Results

The median and inter-quartile range for Dice and Hausdorff₉₅ scores for each of the constructed models using 3 topologies and 7 input image combinations, are shown in Tables 1 and 2 and Fig. 2 and 3. DeepMedic showed a consistent superior performance to 3D U-Net and FCN, for all input combinations.

Table 1.

Median and inter-quartile range for Dice scores of all trained models and input image combinations. DM:DeepMedic, 3dU:3D U-Net.

	T1	T2	T1Gd	Flair	Multi-4	Multi-2	Ens-4
DM	98.09 ± 1.18	97.88 ± 1.07	97.86 ± 1.08	97.88 ± 1.03	98.19 ± 1.08	98.13 ± 1.08	97.94 ± 1.10
3dU	94.77 ± 2.30	96.01 ± 1.58	97.15 ± 1.66	96.08 ± 1.92	98.20 ± 1.19	98.13 ± 1.02	98.05 ± 0.76
FCN	97.65 ± 0.74	96.34 ± 1.32	97.16 ± 0.99	96.74 ± 1.11	93.34 ± 1.13	97.82 ± 1.18	97.46 ± 1.07

Table 2.

Median and inter-quartile range for Hausdorf f₉₅ scores for all trained models and input image combinations. DM:DeepMedic, 3dU:3D U-Net.

	T1	T2	T1Gd	Flair	Multi-4	Multi-2	Ens-4
DM	2.24 ± 1.41	2.24 ± 1.00	2.24 ± 1.27	2.24 ± 1.00	2.00 ± 1.41	2.24 ± 1.41	2.24 ± 1.00
3dU	11.45 ± 2.98	19.21 ± 2.85	3.00 ± 2.01	5.20 ± 5.85	1.73 ± 1.41	2.00 ± 1.50	2.00 ± 0.61
FCN	2.24 ± 1.00	3.61 ± 1.18	2.83 ± 0.93	3.00 ± 1.55	5.92 ± 0.93	2.24 ± 1.00	2.24 ± 0.76

Fig. 2.

Median and inter-quartile range for Dice scores of all trained models and input image combinations.

Fig. 3.

Median and inter-quartile range for Hausdorf f₉₅ scores of all trained models and input image combinations.

Overall, best performance was obtained for the DeepMedic-Multi-4 model (Dice = 97.9, Hausdorf f₉₅ = 2.68). However, for the model trained using DeepMedic and only the T1 modality obtained comparable (statistically insignificant, p > 0.05 - Wilcoxon signed-rank test) performance (Dice = 97.8, Hausdorf f₉₅ = 3.01). This result reaffirms the use of T1 in current state of the art methods for skull-stripping.

Performance of 3D U-Net was consistently lower when the network was trained on single modalities. However, the 3D U-Net-Multi-4 model obtained performance comparable to DeepMedic. Despite previous literature reporting a clear benefit of the ensemble approach [12,13], in our validations we found that the ensemble of models trained and inferred on individual modalities did not offer a noticeable improvement.

Illustrative examples of the segmentations for the best performing model (DeepMedic-Multi-4) are shown in Fig. 4. We showcase the best and the worst segmentation results, selected based on the Dice scores.

Fig. 4.

Visual example of the best (A) and the worst (B) output results.

4. Discussion

We compared three of the most widely used DL architectures for semantic segmentation in the specific problem of skull-stripping of images with brain tumors. Importantly, we trained models using different combinations of input image modalities that are typically acquired as part of routine clinical evaluations of patients with suspected brain tumors, to investigate contribution of these different modalities to overall segmentation performance.

DeepMedic consistently outperformed the other 2 methods with all input combinations, suggesting that it is more robust. In contrast, 3D U-Net and FCN had highly variable performance with different image combinations. With the addition of mpMRI input data, the 3D U-Net models (“Multi-4” and “Multi-2”) performed comparably with DeepMedic.

We have made the pre-trained DeepMedic model, including all pre-processing and post-processing steps, available for inference to others through our cancer image processing toolkit, namely the CaPTk [32,33], which provides readily deployable advanced computational algorithms to facilitate clinical research.

In future work, we intend to extend application of these topologies to multi-institutional data along with other topologies for comparison.