Longitudinal Assessment of Posttreatment Diffuse Glioma Tissue Volumes with Three-dimensional Convolutional Neural Networks

Abstract

Neural networks were trained for segmentation and longitudinal assessment of posttreatment diffuse glioma. A retrospective cohort (from January 2018 to December 2019) of 298 patients with diffuse glioma (mean age, 52 years ± 14 [SD]; 177 men; 152 patients with glioblastoma, 72 patients with astrocytoma, and 74 patients with oligodendroglioma) who underwent two consecutive multimodal MRI examinations were randomly selected into training (n = 198) and testing (n = 100) samples. A posttreatment tumor segmentation three-dimensional nnU-Net convolutional neural network with multichannel inputs (T1, T2, and T1 postcontrast and fluid-attenuated inversion recovery [FLAIR]) was trained to segment three multiclass tissue types (peritumoral edematous, infiltrated, or treatment-changed tissue [ED]; active tumor or enhancing tissue [AT]; and necrotic core). Separate longitudinal change nnU-Nets were trained on registered and subtracted FLAIR and T1 postlongitudinal images to localize and better quantify and classify changes in ED and AT. Segmentation Dice scores, volume similarities, and 95th percentile Hausdorff distances ranged from 0.72 to 0.89, 0.90 to 0.96, and 2.5 to 3.6 mm, respectively. Accuracy rates of the posttreatment tumor segmentation and longitudinal change networks being able to classify longitudinal changes in ED and AT as increased, decreased, or unchanged were 76%–79% and 90%–91%, respectively. The accuracy levels of the longitudinal change networks were not significantly different from those of three neuroradiologists (accuracy, 90%–92%; κ, 0.58–0.63; P > .05). The results of this study support the potential clinical value of artificial intelligence–based automated longitudinal assessment of posttreatment diffuse glioma.

Keywords: MR Imaging, Neuro-Oncology, Neural Networks, CNS, Brain/Brain Stem, Segmentation, Quantification, Convolutional Neural Network (CNN)

Supplemental material is available for this article.

© RSNA, 2022

Summary

Three-dimensional U-Net convolutional neural networks segmented diffuse glioma tissue subregions on MR images with a high level of accuracy and assessed longitudinal changes at the level of neuroradiologists in patients undergoing routine posttreatment MRI.

Key Points

• ■ Convolutional neural networks trained on 396 multimodal MR images accurately segmented tissue subregions in posttreatment diffuse glioma, with median Dice scores ranging from 0.72 to 0.89 and volume similarities ranging from 0.90 to 0.96 in a held-out test set of 200 MR images.

• ■ A network trained to segment individual posttreatment diffuse gliomas at two consecutive time points demonstrated rates of accuracy between 76% and 88% for automatically classifying longitudinal changes in tumor tissue subregions relative to rates achieved by neuroradiologists.

• ■ Separate networks trained on registered and subtraction images from the two consecutive time points demonstrated rates of accuracy between 90% and 93% for classifying longitudinal changes, which was not significantly different from the accuracy rates achieved by neuroradiologists (90%–94%).

Introduction

MRI plays a central role in the evaluation of diffuse glioma disease progression and treatment response (1). Advanced artificial intelligence methods show promise for quantifying and assessing changes in tumor tissue volumes, which could improve diagnosis and treatment for patients with diffuse glioma (2).

The most common indication for glioma imaging is the longitudinal assessment of the disease burden after maximal safe resection, radiation, and chemotherapy (3). The U-Net convolutional neural network architecture (4,5) has excelled in tumor segmentation, with performance reaching human interrater reliability in the Multimodal Brain Tumor Segmentation (BraTS) challenges (5,6). However, the BraTS dataset is limited to a single time point: pretreatment brain MRI. The varied appearance of the brain after treatment, as well as the changes in subtle infiltrative tumor across imaging time points, make the accurate assessment of longitudinal changes in the tumor burden challenging. A method to quantitatively track the disease burden over time would be of great clinical value.

In this study, we trained neural networks to segment MR images of the brain after treatment in patients with all grades of diffuse glioma. In addition to training a network to segment posttreatment tumor tissues at individual time points, we trained separate longitudinal change networks to localize and quantify areas of changing tumor tissue types more precisely by including coregistered images from the two time points and subtraction images between the two time points. We then evaluated the performance of the networks and neuroradiologists to assess longitudinal changes in volumes of tumor tissue subregions.

Materials and Methods

Study Design and Patients

This study conducted at the University of California, San Francisco, Medical Center was in compliance with the Health Insurance Portability and Accountability Act and was approved by the institutional review board, with a waiver for written consent. A total of 298 patients (mean age, 52 years ± 14 [SD]; 177 men) with diffuse gliomas were retrospectively evaluated after excluding four initially selected patients with missing images (Table 1). Patients were identified from searches of institutional radiology archives (mPower Clinical Analytics; Nuance Communications) for MR images of the brain in consecutive discrete patients who underwent diffuse glioma posttreatment follow-up imaging between January 2018 and December 2019. There were 152 patients with high-grade astrocytoma (glioblastoma), 74 patients with grade 2 or 3 oligodendroglioma, and 72 patients with grade 2 or 3 astrocytoma. Patients typically received standard-of-care therapy, which included maximal surgical resection combined with either radiation or radiation and chemotherapy, depending on the tumor grade. Images from two posttreatment time points were obtained for each patient (596 total images). One hundred ninety-eight patients (396 images) were randomly selected to be included in the training set, with the remaining 100 patients (200 images) being included in the test set.

Table 1:

Patient Demographics, Diffuse Glioma Types, and Tumor Statistics

The publicly available 2020 BraTS training dataset (7,8) was used to train an initial network for preliminary segmentations of the posttreatment MR images. The 2020 BraTS dataset consists of 369 preoperative patients with diffuse glioma. Manual expert segmentation of the BraTS dataset delineated three tumor subregions: necrotic core (NCR); active tumor or enhancing tissue (AT); and peritumoral edematous, infiltrative, or treatment-changed tissue (ED). The whole-tumor extent is defined as the union of all three distinct subregions (ED, AT, and NCR), and the tumor core is defined as the union of the AT and NCR.

Imaging Data Acquisition

The details of the BraTS imaging acquisition parameters, representing a heterogeneous multisite preoperative dataset, are found elsewhere (7,8). Patients from the University of California, San Francisco, Medical Center underwent a standardized brain tumor MRI protocol that used one of four 3.0-T GE Discovery 750 (GE Healthcare) imagers. The imaging protocol included three-dimensional (3D) precontrast T1-weighted images; 3D postcontrast T1-weighted images; 3D T2-weighted images; and 3D T2 fluid-attenuated inversion recovery (FLAIR) images. Typical imaging parameters are detailed in Appendix E1 (supplement).

Reference Standard Voxelwise Annotations

Reference standard voxelwise segmentations of tumor tissue subregions were generated by a neuroradiology attending physician (J.D.R., with 1 year of experience as a neuroradiology attending physician and 5 years of segmentation experience) by using an iterative process initially refined from a model trained on the BraTS 2020 dataset, which is detailed in Appendix E2 (supplement). The tumor tissue subregions of NCR, AT, and ED followed the BraTS segmentation guidelines, with the following modifications made related to the posttreatment nature of the images: Resection cavities were not included in any of the tissue subregions; gliosis and postradiation changes manifesting as T2 or FLAIR hyperintensity were also included in the ED class; and smooth linear thin enhancement underlying the craniotomy or in the resection cavity were not included in the AT class, but any enhancing tissue that could possibly reflect tumor was included.

Reference Standard Longitudinal Change Categorical Annotations

Annotations for the longitudinal tumor volume change categories (increased, decreased, and unchanged) for the AT and ED were based on the neuroradiologist’s clinical assessment of significant changes in volume between time points, regardless of the magnitude of change, in contrast to Response Assessment for Neuro-Oncology (RANO) criteria (9), which require a 25% or more change in tumor volume to classify progression. For the training set, these change categories were created by a single neuroradiology attending physician with reference to the final radiology reports and unblinded review of the images (J.D.R.). For the test set, three board-certified academic neuroradiologists (L.P.S., J.E.V.M., and A.M.R., with 6 years, 5 years, and 1 year of post–neuroradiology fellowship experience, respectively) provided annotations for the longitudinal change categories and were blinded to the radiology reports or other clinical information. The final reference standard categorical change categories for the test set were determined by majority consensus from the three neuroradiologists’ annotations.

Image Preprocessing

We implemented an automated image preprocessing pipeline similar to that of BraTS, which included intermodality registration, 1 × 1 × 1 interpolation, skull stripping, and bias correction, as detailed in Appendix E3 (supplement). For the longitudinal change networks, additional preprocessing was performed, including registration and subtraction between time points, as detailed in Appendix E4 (supplement).

U-Net Convolutional Neural Network Architecture

For model training and inference, we used the default settings of nnU-Net (5), which is a self-configuring method that automatically performs preprocessing, network architecture, and hyperparameter tuning (5). Details of the nnU-Net implementation and training parameters are found in Appendix E5 (supplement). The architectures and example inputs and outputs of the posttreatment tumor segmentation network and the ED and AT longitudinal change networks are shown in Figure 1. The longitudinal change networks were developed to localize and better quantify and classify changes in ED and AT.

Figure 1:

Three-dimensional (3D) nnU-Net neural networks used for the longitudinal assessment of diffuse glioma. (A) Posttreatment tumor segmentation network with T1-weighted, postcontrast (T1_post) images; fluid-attenuated inversion recovery (FLAIR) images; and T2-weighted images used as four-channel inputs and four-class outputs, consisting of background, active tumor or enhancing tissue (AT), necrotic core (NCR), and peritumoral edematous, infiltrated, or treatment-changed tissue (ED). (B) ED longitudinal change network with time 1 FLAIR images, time 2 FLAIR images, and time 1 minus time 2 and time 2 minus time 1 subtraction images as four-channel inputs and two-class outputs (background, increase in ED). (C) AT longitudinal change network with time 1 T1_post minus T1-weighted precontrast (T1_pre) subtraction images; time 2 T1_post minus T1_pre subtraction images; time 1 minus time 2 subtraction images; and time 2 minus time 1 subtraction images as four-channel inputs and two-class outputs (background, increase in AT). Conv = convolution, Deconv = deconvolution, Norm = normalization, Seg = segmentation, Sub = subtraction.

Performance Metrics and Statistical Analysis

Tissue segmentation performance of the whole tumor, ED, tumor core, AT, and NCR subregions in the test set were evaluated by using the Dice metric (10), volume similarities, and the 95th percentile Hausdorff distance, as detailed in Appendix E6 (supplement).

To classify the longitudinal change categories for the AT and ED subregions from the tumor segmentation model, we applied a threshold for the percent change and minimum volume of AT, which was determined by optimizing classification accuracy in the training set. For the ED and AT longitudinal change networks, a total volumetric change threshold was applied, which was also determined by optimizing classification accuracy in the training set.

Average neuroradiologist performance was calculated relative to the final reference standard annotations. Among the three neuroradiologists’ annotations, Cohen κ statistics and interrater performance were also calculated. Accuracy rates of the tumor segmentation and longitudinal change networks for assessing longitudinal changes in ED and AT were compared with the accuracy rates demonstrated by neuroradiologists by using χ² tests. Statistical significance was defined as a P value less than .05.

Results

Tumor Tissue Volumes

Tumor volumetric information derived from manual segmentations of the training and test sets is shown in Table 1. A total of 63.1% (188 of 298) of patients had at least 0.02 cm³ of AT (84.9% [129 of 152] of patients with glioblastoma, 47.2% [34 of 72] of patients with astrocytoma, and 35.1% [26 of 74] of patients with oligodendroglioma), and 35.6% (106 of 298) of patients had at least 0.02 cm³ of NCR (56.6% [86 of 152] of patients with glioblastoma, 19.4% [14 of 72] of patients with astrocytoma, and 8.1% [six of 74] of patients with oligodendroglioma).

Segmentation Performance

The Dice scores, volume similarities, and 95th percentile Hausdorff distance metrics for the tissue subregions in the test set for the tumor segmentation and longitudinal change networks are shown in Table 2. Median Dice scores for the tumor segmentation network ranged from 0.72 to 0.89 compared with 0.43 to 0.70 for a model trained only on the BraTS 2020 data (Table E1 [supplement]). Example test cases from the tumor segmentation network are shown in Figure 2. Example test cases for the ED and AT longitudinal change networks are shown in Figure 3. Correlations and Bland-Altman plots between manual and predicted tumor subregion volumes and correlations between tumor subregion volumes and Dice scores are shown in Figure E1 (supplement) and are detailed in Appendix E7 (supplement).

Table 2:

Test Set Segmentation Performance Metrics

Figure 2:

Examples of tumor segmentation network–predicted segmentations. Four example cases segmented with two time points with example axial T2-weighted images, T1-weighted precontrast images (T1-pre), T1-weighted postcontrast images (T1-post), fluid-attenuated inversion recovery (FLAIR) images, and example tumor tissue class segmentations overlaid on the FLAIR image (green = peritumoral edematous, infiltrated, or treatment-changed tissue [ED]; yellow = active tumor or enhancing tissue [AT]; red = necrotic tumor core). The total percent change in the ED and AT subregions are shown between the two time points for each of the four sets of cases.

Figure 3:

Examples of longitudinal change network–predicted segmentations. (A, B) Examples of fluid-attenuated inversion recovery (FLAIR) longitudinal change network–predicted segmentations. Two example cases with example time 1 FLAIR images, time 2 FLAIR images, and time 2 minus time 1 FLAIR subtraction images and an example of predicted segmentations (red = increasing, green = decreasing) overlaid on the time 2 minus time 1 FLAIR images. The total predicted changes in the volume of peritumoral edematous, infiltrated, or treatment-changed tissue (ED) between time points are displayed for each case. (C, D) Example enhancement longitudinal change network–predicted segmentations. Two example cases with example time 1 and time 2 T1-weighted precontrast (T1-pre) images, T1-weighted postcontrast (T1-post) images, time 1 and time 2 T1-post minus T1-pre subtraction images, time 2 subtraction minus time 1 subtraction images, and predicted segmentations (red = increasing, green = decreasing) overlaid on the time 2 subtraction minus time 1 subtraction images. The total predicted changes in volume of active tumor or enhancing tissue (AT) between time points are displayed for each case.

Classification of Categorical Changes in Tumor Volumes

Test cases were automatically classified into increased, decreased, or unchanged longitudinal change categories by using thresholds established by optimizing training data classification performance accuracy. For the tumor segmentation network, thresholds were set at 15% and −15% for increased or decreased ED and at 15% and −20% for increased or decreased AT (with a minimum change of tissue of 0.5 cm³ required for AT). The tumor segmentation network achieved accuracies of 76% (ED) and 79% (AT) for predicting the three categorical change categories and achieved accuracies of 79% (ED) and 88% (AT) for predicting two categorical change categories (increased vs not increased). The full performance metrics, including the sensitivity, specificity, negative predictive value, positive predictive value, F1 score, and accuracy metrics, are shown in Table 3. For the longitudinal change networks, net change thresholds were set at 0.2 cm³ and −0.5 cm³ for increased or decreased ED volume and at 0.1 cm³ and −0.25 cm³ for increased or decreased AT volume. The longitudinal change networks were 91% and 90% accurate at predicting the three categorical change categories for ED and AT, respectively, and were 93% and 92% accurate at predicting the increased versus not increased categories, respectively. Higher longitudinal classification performance was associated with larger changes in tumor volumes and smaller tumors, as detailed in Appendix E8 (supplement).

Table 3:

Longitudinal Classification Task Performance Metrics

Neuroradiologist Interrater Reliability and Performance

The average neuroradiologist achieved accuracy rates of 90.3% and 91.7% for the three-class changes in ED and AT, respectively, and achieved accuracy rates of 93.7% and 94.0% for the two-class changes in ED and AT, respectively; the full performance statistics are shown in Table 3. Neuroradiologist performance was significantly higher than tumor segmentation network performance for all tasks (P < .05; Table 3), but there was no evidence of a difference compared with longitudinal change network performance (P > .05).

The average interrater reliability values of the three neuroradiologists for ED and AT were 81% and 84%, respectively, for the three-class longitudinal assessment task, with Cohen κ values of 0.58 and 0.63, respectively (Table 3). For the two-class longitudinal assessment task, the interrater reliability values were 87% and 88%, respectively (Cohen κ, 0.71 and 0.72), which are considered to indicate moderate agreement (11).

Discussion

Artificial intelligence methods designed for the longitudinal assessment of glioma can play a critical role in improving the accuracy and efficiency of assessments of the change in the tumor burden in both routine practice and clinical trials. In this study, we trained state-of-the-art, 3D U-Net neural networks for longitudinal assessment of posttreatment diffuse glioma MR images. The networks performed with high segmentation accuracy across glioma tissue subregions, and they classified categorical changes in volumes of tumor tissue types at the level of neuroradiologists.

Although most prior work has evaluated methods for segmentation of preoperative diffuse glioma tissue subregions, typically relying on the BraTS (7,8) dataset, two recent studies have performed automated segmentation of posttreatment glioma (12,13). These studies focused on posttreatment glioma segmentation in the context of the RANO criteria, achieving high Dice scores and good correlations with manual RANO measurements. In contrast, we explicitly designed a system for routine radiologic longitudinal assessment, which incorporated consecutive time points into a model that could classify and localize relatively small changes in tumor volumes. Multiple adaptations, including custom skull stripping, revised expert annotation protocols, and a longitudinal input, were required to develop a robust algorithm. Just as radiologists directly compare images from different time points, we created longitudinal change networks to precisely localize and quantify changes in ED and AT across time points. Ultimately, this approach achieved 90%–93% accuracy in the longitudinal classification of changes in tumor subregion volumes, which was not significantly different from the accuracy rates achieved by three neuroradiologists. Although the posttreatment tumor segmentation network had excellent segmentation metrics, similar to prior posttreatment glioma segmentation studies (12,13), and interrater reliability (14,15), the network’s accuracy for classifying changes in longitudinal tumor subregions was only 76%–88%. This was significantly lower than neuroradiologist accuracy, suggesting that an approach that does not directly incorporate longitudinal time points could have more limited clinical use. Cases misassigned by the tumor segmentation network often showed subtle changes in tumor tissue volumes within a large tumor or technical differences between images, which was less impactful on the longitudinal change networks.

Limitations of this study included the fact that all patients were imaged with a single type of imager at a single site and that the voxelwise annotations were generated by a single expert. The algorithm should ideally be trained on heterogeneous multisite data if the goal were to implement it broadly. Finally, this algorithm was designed to quantify changes in ED and AT volumes. Assessing whether these changes indicate true progression of disease or treatment-related changes, including pseudoprogression, remains a persistent challenge that might be addressed by incorporating relevant clinical information into future models.

Future directions also include testing whether such algorithms, when integrated into clinical systems, improve workflow efficiency and accuracy by automatically generating tumor volumes and regions of change that can be included in radiology reports. Future versions of this system could also be applied to different pathologic conditions and incorporate additional imaging modalities and clinical information. In conclusion, the results of the current study support the potential clinical value of artificial intelligence methods for longitudinal evaluation of posttreatment diffuse gliomas.