Analyzing magnetic resonance imaging data from glioma patients using deep learning

Abstract

The quantitative analysis of images acquired in the diagnosis and treatment of patients with brain tumors has seen a significant rise in the clinical use of computational tools. The underlying technology to the vast majority of these tools are machine learning methods and, in particular, deep learning algorithms. This review offers clinical background information of key diagnostic biomarkers in the diagnosis of glioma, the most common primary brain tumor. It offers an overview of publicly available resources and datasets for developing new computational tools and image biomarkers, with emphasis on those related to the Multimodal Brain Tumor Segmentation (BraTS) Challenge. We further offer an overview of the state-of-the-art methods in glioma image segmentation, again with an emphasis on publicly available tools and deep learning algorithms that emerged in the context of the BraTS challenge.

1. Diagnosing glioma patients from image information

Predicting clinical variables of interest, such as tumor molecular characteristics (Bakas et al., 2020a, 2017b; Fathi Kazerooni et al., 2020b; Binder et al., 2018), treatment response (Akbari et al., 2020), or prognosis (Fathi Kazerooni et al., 2020a; Bakas et al., 2020b) from imaging data has attracted great attention from the neuro-oncology community in recent years. These developments have in part been accelerated by the 2016 WHO classification protocol for grading glioma patients (Louis et al., 2016), that is based on a neuropathological evaluation of glioma tissue from a biopsy or resection, as it is biologically highly plausible that the driving genomic changes behind the molecular tissue changes can be non-invasively identified in the imaging phenotype.

Compared to the gold standard assessment from neuropathology, radiographic imaging-based evaluation of glioma biomarkers offers two key advantages: (i) Spatial heterogeneity within the tumor can easily be assessed and (ii) longitudinal assessment of changes becomes possible without a need for serial biopsies. To this end, a broad variety of computational tools has been developed, most of them relying on machine learning techniques, that support the increasingly complex visual analysis of the multivariate and longitudinal image data acquired in glioma patients.

1.1. Glioma imaging

Decisions about the diagnosis and therapy of brain tumor require optimal information, as treatment options are limited and those that might be considered, such as radiation therapy or tumor resection, may have life-changing side effects. To this end, a multitude of standard morphological, functional, and metabolic imaging modalities are used (Fig. 1), often in repeated exams with intervals as short as a few month time. Analysing these data adequately poses significant challenges both in clinical practice, when standard imaging protocols are used, as well as when testing advanced imaging sequences and searching for new imaging biomarkers.

Fig. 1.

Extended glioma protocol with advanced imaging (exemplarily for the UniBe/SCAN protocol). The basic protocol consists of DWI/ADC, 3-D T1w, T2w, FLAIR and 3-D T1w contrast-enhanced sequences. Protocol extensions encompass DTI, SWI, DSC-perfusion and MR-spectroscopy (single voxel MR spectroscopy and MR spectroscopic imaging).

Standard imaging protocols and biomarkers

During the last three decades, magnetic resonance imaging (MRI) remained the fundamental imaging technology for the diagnosis and localization of cerebral gliomas. The key objectives of MRI are the characterization of the tumor category and its differential diagnosis (e.g., brain abscess, lymphoma, or metastasis) with implications for clinical decision making about the path of care, planning of the most effective therapy regimen and disease monitoring under therapy. The quality of the MRI exams is dependent on many factors as, e.g., field strength, MRI sequence composition, scanner type, slice thickness and image contrast, with a considerable lack of standardization of scanner protocols. Further, no clear recommendations exist for the application of advanced neuroimaging techniques that encompass metabolic characterization (MR-Spectroscopy), vascularization and blood-brain-barrier deficiency (perfusion imaging), or tissue composition (diffusion-weighted imaging).

Therefore, joint consensus recommendations have been proposed by the United States National Brain Tumor Society (NBTS), the Society for Neuro-oncology (SNO), and the European Organisation for Research and Treatment of Cancer (EORTC). The proposed glioma imaging protocol (EORTC-NBTS) encompasses a standardized set of recommendations for anatomical MRI sequences to assess changes in tumor burden as an imaging endpoint in clinical trials. The minimum requirements for glioma imaging require consistent scanning with an 1.5 T or 3 T MRI scanner with the same imaging parameters are: (1) 3-dimensional T1 weighted imaging before and (2) after the administration of Gadolinium-based contrast enhancements, (3) a 2-D acquisition of fluid attenuated inversion recovery sequences (FLAIR), (4) an axial 2-D T2-weighted sequence, and (5) an axial 2-D diffusion weighted image (DWI). The acquisition of (6) additional T1-weighted spin echo sequences may further improve the detection of tumor recurrence associated with delayed enhancement (Kaufmann et al., 2020). All sequences display structural properties of the tumor, with T2 weighted scans highlighting the tissue water of the edema, and the T1 contrast-enhanced images showing areas of active tumor growth, where Gad-enriched blood is leaking into the tissue. DWI sequences, that include diffusion tensor imaging, offer insights into areas with modified tissue micro-structure, for example, when additional water or tumor cells increase tissue cellularity, or limit the natural anisotropy of tissue water diffusion. Most data sets discussed in the following (Section 2) encompass standard imaging parameters (1)–(4), but resources for (5)–(6) are also discussed (Section 2.3).

The integration of the 3D imaging sequences into the standard imaging protocol – e.g., (1) and (2), but also optional 3D T2 and diffusion scans – are essential for the volumetric assessment of tumor progression and the application for radiographic response assessment imaging criteria. Conventional 2-D measurements must be considered inadequate to longitudinally assess complex tumor geometry. Slice thickness and rotation induced directly affect the accuracy of bi-dimensional diameter measures errors of >30% (Schmitt et al., 2013; Reuter et al., 2014). In contrast, AI-supported volumetric measurements nowadays yield an excellent accuracy compared to conventional tumor response assessment (see Section 3). Limitations of the (qualitative) response assessment with conventional MRI encompass the lack of sensitivity to distinguish therapy related effects as pseudo-progression and radiation necrosis from true progression at the earliest time that is possible.

Advanced imaging sequences and biomarkers

To improve the diagnostic accuracy, conventional MRI can be complemented by advanced neuroimaging techniques that increase the diagnostic accuracy by evaluating changes in blood flow or in metabolic patterns.

Perfusion-weighted imaging is most frequently used (85%) as an advanced neuroimaging technique to distinguish between low- and high-grade gliomas and between true progression and pseudo-progression. Dynamic susceptibility contrast-enhanced (DSC) MRI is employed in the majority of centers, whereas T1-based dynamic contrast-enhanced (DCE) or a combination of both techniques is frequently restricted to comprehensive cancer centers. Both DSC and DCE-MRI show an excellent diagnostic accuracy to discriminate between high- and low grade gliomas and provide complementary information to discriminate tumor recurrence from therapy related effects. In a recent systematic review, encompassing 27 studies and 298 patients, the pooled sensitivity, specificity and sensitivity for the differentiation between high- and low grade tumors was 0.93, 0.90 and 0.96 and between tumor relapse and treatment-related changes was 0.88; 0.86 and 0.89 (Okuchi et al., 2019). A previous meta-analysis that investigated the discriminative power of DSC-perfusion and DCE perfusion MRI to separate viable tumor from treatment-related effects revealed similar results reported comparable results for both methods (Patel et al., 2017). However, they reported also considerable variability in threshold definitions and a lack of standardisation that hampers the implementation of quantitative perfusion MRI strategies across institutions. Intra-voxel incoherent motion (IVIM) and arterial spin labeling (ASL) perfusion complement the spectrum of perfusion imaging techniques (Le Bihan, 2019; Paschoal et al., 2018; Grade et al., 2015). The principal advantage of both techniques is their non-invasive character without the need to administer contrast. While IVIM estimates microcirculation in randomly oriented capillaries mimicking a pseudo-diffusion process, ASL uses radiofrequency pulses that saturate water protons to magnetically tag water molecules in the arterial blood. Both methods have shown some potential in tumor grading and outcome prediction, but remain still in an experimental stage.

Magnetic resonance spectroscopy (MRS) and magnetic resonance spectroscopic imaging (MRSI) are less frequently used in routine glioma imaging protocols, but also represent long-standing application domains for machine learning research, both for automating the signal processing (Menze et al., 2008; Das et al., 2017) and the diagnostic evaluation of the spectral data (Menze et al., 2006). Clinical indications encompass the characterization of a lesion, the differentiation between glial neoplasms and imaging phenotypes that mimic gliomas and the differentiation between true vs. pseudo-progression (Ryken et al., 2014; van Dijken et al., 2017; Hollingworth et al., 2006). More recently, 2-hydroxyglutarate MRS has been proposed as a promising non-invasive method to discriminate between isocitrate-dehydrogenase (IDH)-mutant and IDH- wild type gliomas (van den Bent et al., 2010; Verma et al., 2016). Mutations in IDH are highly prevalent among gliomas of lower grade (70–80%) and carry a better prognosis in grade III gliomas (Verma et al., 2016). A recent meta-analysis reported a higher sensitivity of 2-hydroxyglutarate MRS to differentiate between IDH-mutated and IDH-wild type gliomas than diffusion or perfusion imaging or localization-based features (Suh et al., 2018, 2019). The method has a great potential to overcome limitations of invasive biopsies related to intra-tumoral heterogeneity and subsequent biases of biopsy-based genomic analysis.

Using multi-parametric MRI

Overall, MRI remains a key methodology to diagnose and monitor patients with cerebral gliomas. Conventional structural imaging provides a basis for lesion stratification, treatment planning and monitoring and provides the basis for automated image analysis of lesion progression, predictive monitoring, and radiomic feature extraction (Zwanenburg et al., 2020).

Advanced imaging techniques complement the standard imaging workup and support the analysis and prediction of physiological and molecular characteristics reflected through the burden of disease and the potential of response to therapy. Moreover, with imaging revealing critical information about tumor and surrounding anatomy in the patient, glioma imaging is also always a crucial component in personalizing treatment decisions.

Recent work by Lipkova et al. (Lipkova et al., 2019), for example, used the patient’s glioma images together with a tumor growth model and a Bayesian machine learning framework to predict patient-specific tumor cell density with credible intervals from multimodal imaging data as a basis for personalized radiotherapy design.

1.2. Glioma imaging, radiomic profiling, and tumor biology

Classifying glioma by genotypes

Traditionally, gliomas were classified according to their histo-morphological relationship to the glial cell lineages as outlined in the 2007 WHO classification (Louis et al., 2007). However, this morphology-derived classification had several challenges for its scientific and clinical use. Most importantly, these were the rater-dependence of diagnoses (demonstrated for example by Kros et al. (Kros et al., 2007)) and – closely related – the ambiguity of morphologic appearance, leading to the definition of bucket diagnoses such as “oligo-astrocytoma”, which harbored tumors showing morphologic features reminiscent of both astrocytomas and oligodendrogliomas. With the advent of large-scale, high-throughput techniques for (epi)genome-wide analysis of tumors, several landmark discoveries have been made in many types of cancer, also in gliomas (Brennan et al., 2013; Brat et al., 2015; Sturm et al., 2012; Barthel et al., 2019). Chief among them was the discovery of point mutations in the isocitrate dehydrogenase 1 (IDH1) and to a lesser extent also in the isocitrate dehydrogenase 2 (IDH2) genes (Parsons et al., 2008; Bakas et al., 2018b). While they are rare (<10%) in WHO grade IV glioblastoma, the most malignant gliomas, they are a defining genomic event in WHO grade II and III gliomas (Yan et al., 2009), present in the majority of the tumors. Later analyses found that mutant IDH1 and IDH2 catalyze the production of the onco-metabolite 2-hydroxyglutarate (2HG), which is not present in healthy cells. Subsequently, it was discovered that 2HG is essential for the formation of IDH mutant gliomas (Lai et al., 2011) and leads to dramatic changes in the tumor epigenome (Sturm et al., 2012). Today, it is accepted that IDH-mutant and IDH-wildtype tumors are biologically different tumor entities, despite their similar (and sometimes indistinguishable) histo-morphologic appearance. Importantly, these tumors also significantly differ in their clinical course and treatment response, with IDH mutant tumors carrying a far better prognosis (Yan et al., 2009).

Later genome-wide studies used this knowledge about the impact of epigenetic changes for tumor development and progression and the relative stability of the epigenome (at least compared to mRNA expression analysis) for (un)supervised classification of gliomas (Wiestler et al., 2014; Sturm et al., 2012) and the identification of key molecular features in these subgroups. Collectively, these studies yielded a clear picture of a biology-driven classification of gliomas across WHO grades and entities. These findings have also resulted in a revised WHO classification released in 2016 (Louis et al., 2016), which now groups tumors according to a more integrative schema including their geno-typical characteristics, instead of based solely on their histological phenotype, as previously. This leads to situations where the genotype (e.g., presence of an IDH mutation as well as co-deletion of the short arm of chromosome 1 and the long arm of chromosome 19, 1p/19q codeletion) disagrees with the histological phenotype (in the above example an astrocytoma, which typically do not carry 1p/19q codeletion). In these cases, the genotype “beats” the phenotype and underlines the importance of tumor biology for classification.

Radio(geno)mic correlations

Identifying associations between the glioma genotype and their (MR) imaging phenotype has become an important field of research, often referred to as radiogenomics, and concepts from machine learning have contributed significantly to the analysis of the glioma imaging data. Initial studies focused on identifying key genomic alterations such as IDH mutation or 1p/19q codeletion from preoperative imaging data (see for example the studies by Kickingereder et al. (2015), Eichinger et al. (2017), or Chang et al. (2018)). Meanwhile, the focus of glioma image analysis shifts towards assessing heterogeneity or predicting clinical course using radiomic image features: In a 2017 paper published in the AJNR (Chang et al., 2017), the authors present a classifier trained to predict cellularity from voxel-wise regression analysis of FLAIR, ADC and T1c data on a set of 91 stereotactically localized biopsies. With this approach, they are able to calculate non-invasive, voxel-wise cellularity maps, which might for example aid the process of selection suitable biopsy locations, or enter a tumor modeling approach as in Menze et al. (2011), Lipkova et al. (2019). Along this line, Hu et al. (2017) extended this concept to predicting spatial heterogeneity of key molecular alterations in glioblastoma. From another data set of 48 targeted biopsies, they built multivariate decision-tree models to predict spatial presence of copy-number alterations of genes like the epidermal growth factor receptor (EGFR) in preoperative MR images.

The state-of-the-art analysis of glioma images

Overall, a growing body of research studies in the literature is not aiming only at quantifying tumor structures visible in the multi-parametric MR images, but at stepping beyond analyzing the apparent visual content to identify sub-visual cues using quantitative radiomic features (Suter et al., 2020). Uncovering correlations between image features with biological (molecular, genetic), as well as clinical variables is the ultimate goal of this work. In this, the segmentation and quantification of “semantic” tumor sub-compartments/structures, i.e., regions that can be named and associated with clear properties and even function within the tumor area, remains the first and most crucial step of any subsequent radiomic correlation study.

2. Benchmarking glioma image quantification in the BRATS challenge

Clinical imaging protocols routinely-acquired for the diagnosis of glioma cases reveal information about the anatomical structure, function, vascularization, among other tissue characteristics and standardization of the protocols across centers is only a recent development (Section 1). As a consequence, in the initial efforts for quantifying tumor image information, locally varying imaging datasets were used for developing machine learning algorithms from various researchers. This variation, and the absences of publically available reference data, prohibited a fair comparative algorithmic evaluation. This issue was overcome by the start of the annual “Multimodal Brain Tumor Segmentation Challenge” (BraTS, www.braintumorsegmentation.org) in 2012 (Menze et al., 2014; Bakas et al., 2018a, 2017a), which initiated the creation of a benchmarking environment and the collection, expert annotation, and distribution of a large clinically-acquired multi-institutional dataset of multi-parametric MRI scans from patients diagnosed with gliomas. To the best of our knowledge, the BraTS dataset is one of the largest publicly available curated datasets for glioma imaging and has been heavily used for computer vision and machine learning research. Although it has been initiated with image segmentation in mind, its establishment and recognition of its clinical relevance and potential has offered avenues to a multitude of the glioma image analysis tasks outlined above.

2.1. The public BraTS dataset

Image data

The clinically-acquired and -curated data made available during the BraTS challenges has grown significantly from including a few dozens of cases in 2012 to almost one thousand cases that are available in 2020. The complete BraTS dataset originated from a handful of institutions while its 2020 edition integrates data from 19 independent international institutions, as well as collections from The Cancer Imaging Archive (TCIA) (Clark et al., 2013). The majority of cases utilized in more recent instances of the BraTS challenge are acquired pre-operatively, but time-series with pre- and post-interventional follow-up scans have also been made available through BraTS in the past (BRATS 2016 data set). Cases originating from TCIA are accompanied by clinical, genetic, and pathological data, available via the Genomic Data Commons (GDC) Data Portal. Further scans are also available from TCIA and can be used in conjunction with the BraTS data.

All MRI data distributed through the BraTS challenges have been acquired by scanners with magnetic field strength of 1–3 T, and comprise for all patients and timepoints the basic structural imaging sequences: native T1-weighted (T1); T1 after administration of contrast agent (T1c); T2-weighted (T2); and T2 Fluid-Attenuated Inversion Recovery (FLAIR). To standardize the highly variable resolutions and orientations of the acquired scans, all data have been consistently rigidly aligned to the same brain anatomical template (Rohlfing et al., 2010) and interpolated to 1 mm³ isotropic resolution. Furthermore, in compliance with institutional requirements for anonymization, all imaging scans have undergone brain extraction prior to their public release.

Expert annotations of semantic structures

These multi-parametric MRI datasets are accompanied by tumor boundary annotations for each of the histologically distinct tumor sub-structures (Menze et al., 2014; Bakas et al., 2018a), generated and approved by clinical experts following a dedicated harmonized protocol. These labels comprise regions reflecting edematous/infiltrated tissue, as well as the potentially resectable tumor core with solid/enhancing, necrotic, and/or fluid filled compartments (Fig. 2). These labels have been considered as three independent sets of binary areas: (i) the whole tumor are (described by the union of all the classes), (ii) the tumor core area (with all tumor core labels, excluding the peritumoral edematous/infiltrated tissue), and (iii) the area of the active part of the tumor (being represented by the contrast-enhancing compartment). As these three areas have a direct translation into “semantically” meaningful tumor quantification tasks, and since assigning weights to inter-class classification errors is difficult in the current medical use case, they are evaluated during the BraTS challenge individually rather than a straightforward multi-class evaluation.

Fig. 2.

Semantic annotations available in the BRATS data set: Labels (shown in the left) summarize three semantic regions: whole tumor as visible from hyper-intense areas in T2w and FLAIR images (left column, yellow), the tumor core visible heterogenous signals in T2w MRI (central column, red), and the active tumor visible from intensity enhancements in post-Gd T1w scans (right column, blue).

In addition to the experts’ image annotations, a subset of the 2018–2020 datasets have been accompanied by clinical information on patient age and patient overall survival (Bakas et al., 2018a).

Availability

The BraTS data are available for general use under a creative commons license and can be downloaded from links available on the challenge website.¹ A clear distinction should be made to the subset used in different years, e.g., BraTS 2016 or BraTS 2020, as the focus of the evaluation changed in time as did the composition of the training, validation, and testing partitions of the data. For example, the 2020 data have only pre-operative scans, while the 2016 data have both pre- and post-interventional scans.

2.2. The BRATS benchmarking challenges

Evaluating image segmentation algorithms

The BraTS datasets and the annual BraTS benchmarking challenges, running in conjunction with the conference on Medical Image Computing and Computer-Assisted Interventions (MICCAI), have been instrumental in spear-heading the development of brain tumor segmentation algorithms. Specifically, numerous algorithms (between 10 in 2012 and more than 60 in 2019) have been comparatively evaluated every year. During the challenge workshop data are provided to the BraTS participants for training, validation, as well as for testing, and then the participants are required to submit their predicted segmentation labels for evaluation by the BraTS online evaluation portal. At the same time, brief descriptions of the algorithms appear in the challenge’s proceedings of each annual workshop, offering insights into how the algorithms differed in design, training parameters, or implementation configurations. The evaluation system calculates different scores, such as the Dice similarity coefficient, the Hausdorff distance, or volumetric mismatch for the longitudinal task, and the algorithms are scored according to these scores. As the question of how to find the one (or a few) top performing methods using multiple scores and across the three different segmentation tasks remains an open research question, various ranking heuristics have been used during the years.

Top performing algorithms

Among the algorithms that were considered to be top performing since BraTS 2012, different approaches had been employed, reflecting the technological advances in the domain of computer vision during the past 10 years. While the first algorithms from 2012 still employed generative probabilistic models (Menze et al., 2010, 2012a, 2015), or level-set based segmentation (Riklin-Raviv et al., 2010; Raviv et al., 2012), there has been a rapid adoption of machine learning approaches using random forest together with local image features. While already in 2012 a number of algorithms using random forest classifiers had been tested (Zikic et al., 2012; Geremia et al., 2012; Menze et al., 2012b; Stefan Bauer et al., 2012)), a random forest algorithm was not winning the challenge before 2013 (Tustison et al., 2013). The first deep learning methods were used in 2014 (Davy et al., 2014; Havaei et al., 2017; Zikic et al., 2014; Urban et al., 2014), although not yet winning the challenge even during the following year (BraTS 2015), where a hybrid generative-discriminative approach integrative of a biophysical tumor growth model to account for mass-effect was the winner, namely GLISTRboost (Bakas et al., 2015; Zeng et al., 2016). The first winning deep learning algorithm was during BraTS 2016 (Chang, 2016), which was based on a deep neural network with residual connections. From 2014 onwards, a steady improvement of the algorithmic performances could be observed (as measured in the BraTS testing datasets), and a continuously increasing number of the participating algorithms have become available as open source implementations. Algorithms that perform at the top for the BRATS 2020 data set reach Dice scores of up to 90 Beyond benchmarking The BraTS data have been commonly used for benchmarking algorithms beyond the focused settings of the official evaluation schema. Subsets of the BraTS data have been used in the Medical Segmentation Decathlon (Simpson et al., 2019), as well as in the Quantification of Uncertainties in Biomedical Image Quantification (QUBIQ) Challenge – with published baseline performances and an online evaluation portal – and they can further be used in private evaluation setups. This has fostered the development of brain tumor image quantification in a variety of medical image analysis and computer vision research projects, while glioma segmentation offers an example of early clinical translation of machine learning – and, more specifically deep learning – technology in a quantitative diagnostic evaluation of biomedical images (examples of which will be given in the following section).

2.3. Publicly available data beyond BraTS

Beyond the curated, annotated, publicly-available multi-institutional mpMRI scans of glioma patients released as part of the BraTS challenge, various additional data collections of glioma patients are becoming available, providing radiological, histopathological, molecular, and clinical information. Nevertheless a substantial joint computational-clinical effort would be required to curate and annotate these datasets given a harmonized protocol.

These collections, primarily hosted at TCIA (Clark et al., 2013), describe data of glioblastoma patients (TCGA-GBM² (Scarpace et al., 2016), QIN GBM Treatment Response³ (Prah et al., 2015; Jafari-Khouzani et al., 2015), CPTAC-GBM⁴ (CPTAC, 2018), IvyGAP⁵ (Puchalski et al., 2018; Nameeta et al., 2016), ACRIN-FMISO-Brain⁶ (Gerstner et al., 2016; Ratai et al., 2018; Kinahan et al., 2018), Brain--Tumor-Progression⁷ (Schmainda and Prah, 2018), RIDER NEURO MRI⁸ (Barboriak, 2015), UPENN-GBM-ICMR⁹ (Bakas and Davatzikos, 2020), and ReSPOND (Davatzikos et al., 2020)), as well as lower grade glioma patients (TCGA-LGG¹⁰ (Pedano et al., 2016) and LGG-1p19q-Deletion¹¹ (Akkus et al., 2017; Erickson et al., 2017)), and datasets combining all grades of gliomas (REMBRANDT¹² (Scarpace et al., 2015), QIN--BRAIN-DSC-MRI¹³ (Schmainda et al., 2016), and GLASS¹⁴ (GLASS Consortium, 2018; Bakas et al., 2020a), which is hosted in Synapse¹⁵).

3. State of the art deep learning segmentation

Offering a concurrent overview of deep learning-based brain segmentation methods related to glioma patients has become increasingly difficult with the advent of easy-to-train approaches based on deep learning architectures, such as the U-Net (Ronneberger et al., 2015) and its variation. To this end, we will focus on a set of methods that performed well in recent BraTS challenges, i.e., those that prove to perform well in a controlled setting. We attempt to identify – and report – common design choices, as well as differences between methods that may be linked to their strong performance on this type of segmentation task.

3.1. Image pre-processing

Brain tumor MRI typically consists of multiple MRI modalities. The BraTS dataset (Menze et al., 2014; Bakas et al., 2017a, 2018a) provides the most common ones: T1, T1c, T2, and FLAIR. Initial preprocessing steps are image registration followed by brain extraction (Smith et al., 2004; Jenkinson et al., 2012; Isensee et al., 2019; Kleesiek et al., 2014; Thakur et al., 2020).

Intensity harmonization

MRI intensities are notoriously non-standardized (Russell et al., 2014; Nyúl et al., 2000). In the pre-deep-learning era, methods were susceptible to potentially nonlinear intensity shifts resulting in a need for histogram matching or more sophisticated normalization methods (Kleesiek et al., 2014; Russell et al., 2014; Nyúl et al., 2000) to be applied. Bias field correction (Tustison et al., 2010) was regularly used to remove inhomogeneities in the images. Interestingly, this issue was alleviated with the emergence of deep learning techniques. While the previous random forest approaches used local image features, which limits the ‘receptive field’ that contributes to the inference at a voxel, an appropriately chosen network architecture ‘sees’ the full image and may be better suited to recognize – and ignore – large scale intensity changes that result from bias fields. To this end one might argue that deep learning techniques are more robust with respect to non-standardized intensity values and inhomogeneities, possibly due to their superior capacity, their end-to-end training forcing the extraction of robust feature representations and the application of data augmentation techniques. The now by far most prevalent intensity normalization technique is z-scoring, where the brain region of the images is normalized by subtracting its mean and dividing by its standard deviation (Kamnitsas et al., 2016, 2017b; Isensee et al., 2017, 2018; Myronenko, 2018; Jiang et al., 2019; McKinley et al., 2019; Jungo et al., 2018b).

Spatial harmonization

In particular when working with multi-institutional data and data originating from different MRI scanners, the voxel spacings can be heterogeneous and must be homogenized for processing with convolutional neural networks. Selecting a proper target spacing for resampling can be crucial for downstream performance. A large spacing will result in lower image resolution, making it easier to capture sufficient contextual information at the cost of reduced details in the resulting segmentation maps. A lower voxel spacing retains more details in images and segmentations but make the segmentation problem inherently more difficult due to the required larger input sizes. The training cases provided by the BraTS challenge were resampled to a common voxel spacing of 1 mm³ isotropic resolution, which strikes a good balance between image size and resolution.

3.2. Network architectures

According to our experience, the quality of a segmentation architecture is related to the expressiveness of the features it can learn, the amount of contextual information it can encode and how well it can upscale semantically rich low resolution representations to a full resolution segmentation.

Early architectures

DeepMedic (Kamnitsas et al., 2017a, 2016) was arguably one of the first tremendously successful brain tumor segmentation architectures and the first to be applied in 3D. As compared to newer architectures, it has a substantially smaller receptive field and processes patches of, for example, only 25 × 25 × 25 voxels. It makes up for the missing contextual information by using an additional feature extraction stream that processes downscaled image patches. Features of low and high resolution patches are recombined shortly before the segmentation decision is made. Other early architectures include Pereira’s (Pereira et al., 2016) and Kleesiek’s CNNs (Urban et al., 2014), and Dvorak’s patch-based prediction of structured labels (Dvorak and Menze, 2015).

Current U-Net architectures

Since the introduction of the U-Net in 2015 (Ronneberger et al., 2015), its derivatives define the state of the art in medical image segmentation, not just in brain tumor segmentation. U-Net consists of an encoder and a decoder network which are inter-connected by skip connections. The encoder follows a similar construction as classical image classification networks (Krizhevsky et al., 2012; Simonyan et al., 2014; He et al., 2016; Huang et al., 2017) by alternating feature extraction (convolutional layers) with down-sampling, thus successively aggregating the semantic information necessary for generating good segmentations. When passing through the encoder, feature representations need to undergo spatial pooling to enable subsequent convolutional kernels to effectively cover greater areas of the image. As a side effect, the feature maps suffer from an increasingly low spatial resolution. The purpose of the decoder is to upscale this semantic information under consideration of higher resolution feature maps forwarded to it from the encoder via skip connections. U-Net thus elegantly recombines spatial with semantic information at multiple locations throughout the decoder.

Even though standard U-Net-like architectures are competitive in brain tumor segmentation (Kamnitsas et al., 2017a; Isensee et al., 2018), most recent approaches make use of a variety of modifications. Common U-Net variations include the addition of residual blocks (He et al., 2016; Myronenko, 2018; Jiang et al., 2019; Zhou et al., 2018; Isensee et al., 2017; Wang et al., 2017) or densely connected convolutional layers (Huang et al., 2017; Zhao et al., 2019; McKinley et al., 2018, 2019; Jungo et al., 2018b). Attention mechanisms (Vaswani et al., 2017) can guide the network towards focusing of relevant spatial locations (McKinley et al., 2019), whereas squeeze and excitation (Hu et al., 2018; Zhou et al., 2018) can adaptively recalibrate feature responses. Various successful approaches (Wang et al., 2017; McKinley et al., 2018) replace some pooling operations in the encoder (and their mirrored up-sampling operations in the decoder) with dilated convolutions (Chen et al., 2017) to bypass down- and up-sampling and thus potential interpolation artifacts. The winning contributions to BraTS 2018 and 2019 used additional decoder branches with auxiliary tasks (Jiang et al., 2019; Myronenko, 2018) to regularize the networks.

3.3. Training scheme

Loss functions

The Dice loss (Milletari et al., 2016; Drozdzal et al., 2016) is the most popular loss for brain tumor segmentation. It directly optimizes the Dice score, the most widely used evaluation metric. Recently, the focal loss (Lin et al., 2017) has gained in popularity and was used by multiple highly successful algorithms (Zhao et al., 2019; McKinley et al., 2019, 2018). With the target regions of BraTS being the whole tumor, tumor core, and enhancing tumor rather than the three semantic classes, better results are typically achieved by optimizing these regions directly. Thus, many algorithms use the sigmoid function as the final nonlinearity in their architecture and compute the loss relative to the respective ground truth regions computed from the provided labels (Jiang et al., 2019; Zhao et al., 2019; McKinley et al., 2019, 2018; Isensee et al., 2018; Myronenko, 2018) or optimize the regions one after the other in a cascaded approach (Wang et al., 2017).

Regularization

Even though BraTS provides a large publicly available dataset for training, regularization plays a pivotal role in obtaining a good performing brain tumor segmentation algorithm. Data augmentation techniques are widespread, with mirroring and additive, as well as multiplicative intensity shifts being most prominent (Jiang et al., 2019; Zhao et al., 2019; McKinley et al., 2019; Isensee et al., 2018, 2017). Auxiliary tasks seem to be particularly effective on the BraTS dataset, with the winning contribution from 2018 using a separate VAE decoder branch to reconstruct the network input (Myronenko, 2018) and the winning entry in 2019 utilizing two decoders with different up-sampling strategies (Jiang et al., 2019). The authors of Zhao et al. (2019) report that concurrent learning of the target regions, as well as the semantic classes improve the segmentation performance. The loss function can also be used to improve the robustness of the network. For example, (Isensee et al., 2018) have demonstrated that the segmentation performance increases by using the combination of the cross-entropy and Dice loss as learning objective.

Considering uncertainty

The generation of manual reference annotations for glioma segmentation is tedious and can contain small errors that the networks should ideally ignore. To tackle this problem, inspired by developments in uncertainty estimation for segmentation in brain imaging (Jungo et al., 2018a; Jungo and Reyes, 2019), while (McKinley et al., 2018) developed a label-flip uncertainty loss, which enables the network to recognize potentially mislabeled voxels and reduce their influence during training.

3.4. Inference

Ensemble predictions

Ensembling plays a fundamental role in competitive segmentation performance, as powerfully demonstrated by Kamnitsas et al. (2017b) in their BraTS 2017 entry. Virtually all concurrent segmentation methods make use of some sort of ensembling, whether that is training multiple times with different random seeds (Myronenko, 2018; Jiang et al., 2019), using the models resulting from cross-validation (Isensee et al., 2017, 2018; Jiang et al., 2019), using weights from different epochs of the same network training (Jiang et al., 2019) or applying the network in different orientations (McKinley et al., 2019). Test time data augmentation in the form of mirroring and applying the model to different crops of the image and accumulating the predictions can also be used to boost the performance at the cost of an increased inference time.

Postprocessing rules

Postprocessing is often used in the BraTS dataset to boost a models performance with the challenges specific validation scheme in mind. For example, Isensee et al. (2018) observed that the accumulated Dice score for the enhancing tumor region improved when removing the enhancing tumor entirely from a predicted image if less than some volume threshold of enhancing tumor was predicted. This is due to BraTS awarding a Dice score of 1, if no false positive voxels are predicted in an image that does not contain the label in its reference segmentation. Even though this strategy also removed some true positive predictions, the net gain outweights the losses. This strategy was also applied by the BraTS 2019 winning contribution (Jiang et al., 2019). It should be noted that such a postprocessing, while useful for BraTS, may not be desired in a clinical setup when false positive predictions are less consequential than false negatives. As an alternative post-processing, McKinley et al. (2019) proposed further heuristic modifications of connected components, some of which leverage the uncertainty estimation of their label-flip loss.

3.5. Clinical translation beyond BraTS

Therapy response assessment is a critical aspect of monitoring treatment success in glioma patients. Due to inter-observer variability in the two-dimensional nature of the clinical state of the art (Wen et al., 2010), the variance of two independent measurements is high. Kickingereder et al. (2019) developed a segmentation method that, even though it was trained on 455 MRI scans from only a single institution, generalized well not only to other MRI scans from that same institution but more importantly across a large multi-institutional cohort (34 different institutions comprising a total of 2034 MRI scans). Quantifying tumor response based on their segmentation masks proved to be significantly more reliable and more reproducible than the clinical state of the art. Their pretrained segmentation model is publicly available¹⁶ and can be used as fully functioning segmentation tool (Fig. 3).

Fig. 3.

Segmentations of both the tumor and its sub-structures visible in the different image modalities (T1c and T2 images; left column) and the surrounding brain tissues (tumor and tissue segmentations, central column) are used as input to radiation treatment planning. Information of the patient specific brain anatomy are required to infer directions of most likely tumor cell infiltration using methods such as those by Lipkova et al. (2019) (infiltration maps with 5% and 20% infiltration isolines in red, right column).

4. Going beyond tumor boundaries

In an automated workflow of glioma patients, automated segmentation of tumor targets, as well as other structures such as organs at risk, and resection cavities holds the potential to reduce interobserver variability and accelerate the delineation process, leading to a more efficient and effective clinical workflow. Current state of the art approaches based on deep learning technologies have largely focused on brain tumor segmentation from multisequence MRI, tailored to neuroradiology tasks, and mainly on pre-operative scenarios (Menze et al., 2014; Bakas et al., 2018a; Kickingereder et al., 2019). While the automated segmentation offers great avenues for an objective neuro-radiogical evaluation of disease activity, beside glioblastoma (Kickingereder et al., 2019) is it has established itself in a similar evaluation of Multiple Sclerosis patients (Gabr et al., 2020), the image segmentation algorithms are also well suited to support radiation therapy planning.

Delineating gliomas for radiotherapy planning

The European Organization for Radiation Therapy in Cancer (EORTC) is a large and active network of researchers and clinicians working towards improvement of cancer patient treatment As part of a study performed by the EORTC RTQA group and the Emmanuel van der Schueren Fellowship Program for Quality Assurance in Clinical Trials, the delineation review process in an ongoing multicenter phase III trial was conducted with an accrual goal of 750 glioblastoma patients. Before participating centers could enter patients into the trial, each center had to complete a glioblastoma delineation benchmark case exercise, which was used to assess the interobserver variability of experts in a clinical trial. Results of this ongoing work indicate that despite the availability of delineation guidelines, glioblastoma delineation is subject to significant interobserver variability (Kaidar-Person et al., 2019). Studies have shown that non-adherence to protocol-specified radiotherapy requirements is frequent in prospective clinical trials, with major deviation rates ranging from 11.8% to 48.0%. Similarly, retrospective analyses of EORTC intergroup trials on low grade glioma and anaplastic glioma patients revealed that erroneous delineation of target volumes and organs at risk is a common cause of protocol deviations in clinical trials on brain tumor patients (Fairchild et al., 2012; André et al., 2018)).

Although radiotherapy for glioblastoma is considered a standard treatment, there is considerable delineation variability even among experts: Only moderate agreement, with a mean kappa of 0.58 among GTVs was observed in a delineation study, including fifteen panels of radiation oncologists from independent institutions (Wee et al., 2016). In radiotherapy planning, combining deep learning-based auto--segmentation and manual contour adjustment resulted in superior accuracy, consistency and efficiency for CTV delineation in a retrospective study of post-operative lung cancer patients (Bi et al., 2019). Similarly, deep learning-based auto-segmentation helped to improve CTV consistency for breast (Men et al., 2018) and rectal cancer (Men et al., 2017), and may thus streamline the radiotherapy workflow. However, no similar studies have been conducted for glioblastoma. Previous work has either focused on a mathematical description to take presumed microscopic tumor growth into account (Unkelbach et al., 2014a,b) or used non standard MRI, like DTI (Peeken et al., 2019) to derive novel representations of a CTV definition. Both approaches are still highly investigational and not applicable to current standard of care radiotherapy for glioblastoma in clinical trials.

Localizing organs at risk nearby the tumor

A first deep learning-based segmentation of Organs at Risk (OAR) from MRI has recently been proposed by Mlynarski et al., who presented preliminary results of a CNN-based deep learning approach to segment organs at risk from a single T1-weighted MR image. This indicates that a deep learning-based segmentation will also be capable of auto-segmenting organs at risk in multi-parametric MRI (Mlynarski et al., 2020). In line with the findings of Mlynarski et al. (2020), an effective OAR auto-segmentation approach needs to provide anatomically consistency results where shape prior of segmented of organs is incorporated. A different approach, presented in Orasanu et al. (2018), demonstrates the use of a classical atlas-based approach, with results refined via a deep learning-based contour detector working on triangular mesh representations. Similarly, the authors in Agn et al. (2019) proposed an approach that utilizes a deformable tetrahedral atlas of the brain and structures within a contrast-adaptive generative model for whole-brain segmentation and OAR segmentation. The approach also incorporates a tumor regularization using a conditional restricted Boltzmann machine. Differently from other approaches, the method in Agn et al. (2019) is based on a generative model and is designed to handle differences in imaging protocols. Interestingly, the evaluation of the approach not only relied on standard metrics such as Dice and Hausdorff distances, but also on metrics derived from dose volume histograms. Overall, the challenge when segmenting OAR is to attain a good balance between injected shape prior and image content driving the segmentation process. This is particularly important when segmenting cases where tumor mass effect can dramatically lower shape prior information, learned by a deep learning model.

In summary, tumor segmentation methods in radiation therapy have offer assistance in gross tumor delineation, as well as probabilistic margin definition and their conformance to natural anatomical barriers such as optic chiasm/nerve, brainstem interface, and falx cerebelli, cerebri, as well as the skull, as these preclude or limit the spread of glioblastoma. This is a future direction of research and a future application domain of brain tumor image segmentation.