Joint Vestibular Schwannoma Enlargement Prediction and Segmentation Using a Deep Multi-task Model

Abstract

Objective:

To develop a deep-learning-based multi-task (DMT) model for joint tumor enlargement prediction (TEP) and automatic tumor segmentation (TS) for vestibular schwannoma (VS) patients using their initial diagnostic contrast-enhanced T1-weighted (ceT1) magnetic resonance images (MRIs).

Methods:

Initial ceT1 MRIs for VS patients meeting the inclusion/exclusion criteria of this study were retrospectively collected. VSs on the initial MRIs and their first follow-up scans were manually contoured. Tumor volume and enlargement ratio were measured based on expert contours. A DMT model was constructed for jointly TS and TEP. The manually segmented VS volume on the initial scan and the tumor enlargement label (≥20% volumetric growth) were used as the ground truth for training and evaluating the TS and TEP modules, respectively.

Results:

We performed 5-fold cross-validation with the eligible patients (n = 103). Median segmentation dice coefficient, prediction sensitivity, specificity, accuracy, and area under the receiver operating characteristic curve (AUC) were measured and achieved the following values: 84.20%, 0.68, 0.78, 0.72, and 0.77, respectively. The segmentation result is significantly better than the separate TS network (dice coefficient of 83.13%, p = 0.03) and marginally lower than the state-of-the-art segmentation model nnU-Net (dice coefficient of 86.45%, p = 0.16). The TEP performance is significantly better than the single-task prediction model (AUC = 0.60, p = 0.01) and marginally better than a radiomics-based prediction model (AUC = 0.70, p = 0.17).

Conclusion:

The proposed DMT model is of higher learning efficiency and achieves promising performance on TEP and TS. The proposed technology has the potential to improve VS patient management.

INTRODUCTION

Management of newly diagnosed, small, sporadic vestibular schwannoma (VS) generally follows the “wait-and-scan” approach, in which the knowledge of tumor growth behavior is unknown. Because not all tumors grow after diagnosis, imaging surveillance provides additional information, assisting in determining appropriate follow-up intervals and if therapeutic intervention should be involved.^1–3 In current clinical practice, VS enlargement is determined by measuring the greatest tumor linear dimension on serial magnetic resonance (MR) imaging with a fixed interval.^2,3 However, this linear measurement strategy is not the most sensitive method for characterizing VS growth, and the relatively uniform time intervals for imaging surveillance may lead to delayed treatment for fast-growing tumors or over-imaging for slow-or non-growing tumors.^1,4,5

Volumetric measurement of VS has demonstrated higher accuracy (ACC) and sensitivity (SEN) for tumor enlargement detection than the linear measurement in several studies.^6–8 And attempts have been made to alleviate the labor-intensive process of volumetric tumor delineation using semi-automatic or automatic tumor segmentation (TS) methods based on deep learning.^9–13 These approaches showed promising performance in both VS segmentation ACC on separate MR scans from different patients and measurement stability across longitudinal scans for the same patient, which are helpful for efficient VS enlargement detection.

Different from tumor enlargement detection which is based on tumor size measurement on follow-up images, tumor enlargement prediction (TEP) methods predict the probability of significant tumor volume growth in the early stage of VS diagnosis. Recent works have shown that VS MRI radiomic feature analysis with machine learning methods is predictive for TEP, which has the potential to be used as a basis for individualized treatment planning.^14–16 However, these studies focus on TEP for post-radiosurgery VS only, and 3D delineation of VS is needed as a prerequisite for radiomic feature extraction. Recently, the advent of multi-task learning methods in the area of artificial intelligence (AI) provides efficient ways to jointly image segment and labeling/prediction.¹⁷ Integration of two or multiple tasks (e.g., TS and response prediction) into one network has been reported to have better prediction performance and adequate segmentation ACC in several medical-related tasks.^17–19 In these approaches, a multi-task model learns from multiple training tasks simultaneously. It leverages the parameters of the shared model structures for different tasks, which help the model to exploit the inherent correlation between tasks and reduce the overfitting problem of deep learning.

In the present study, we sought to build a deep multi-task (DMT) model to automatically segment VS and identify VS patients at risk of fast tumor growth, based on MR images acquired at the initial diagnosis. We hypothesize that the DMT model can outperform the single-task tumor growth prediction model and TS model. To the best of our knowledge, this study is the first to develop a multi-task deep learning framework for VS to aid in personalized patient management.

MATERIALS AND METHODS

Data Collection and Pre-processing

This study was approved by our Institutional Review Board (STU 112016–004), and all methods were performed in accordance with the relevant guidelines and regulations. We retrospectively collected a random assortment of VS patients MRI data from our internal database, and a 1:1 fashion was taken for selecting patient with and without significant tumor growth (2 mm increase in tumor size as the threshold) according to the clinical reports. These imaging data include the first gadolinium-enhanced T1-weighted MR scans acquired with 1.5 T or 3 T magnet strength for VS diagnosis and their follow-up MRI scans obtained at least 5 months after the initial MRI. We excluded patients under 18 years old, who had any surgery or radiotherapy before or between these two scans, or diagnosed with neurofibromatosis type 2 for this study. Two observers (one neurotologist and one medical student) manually delineated the tumors using Velocity AI version 4.0 software (Varian Medical Systems, Palo Alto, CA). A total of 103 patients were included in this study. The median follow-up duration between the first and second MR scan was 7.2 months (interquartile range [IQR]: 6.3–12.1 months). The median manually contoured VS volume was 321.02 mm³ (IQR: 156.67–1018.79 mm³) and 450.47 mm³ (IQR: 183.43–1289.29 mm³) on the initial scans and follow-up scans, respectively. Based on the volumetric VS enlargement definition (≥20% volume growth), 62 tumors (60.19%) demonstrated tumor enlargement. The basic patient characteristics are summarized in Table I, of which the age, size, gender, laterality, race, and ethnicity distributions are representative of the general population of VS patients in the United States.^20–23 Detailed information about patient inclusion, imaging protocol, and tumor delineation can be found in Supplemental Digital Content 1–3, respectively.

TABLE I.

Patient characteristics.

Characteristic	Number
Patients (n)	103
Age (y)	62.54 (IQR: 54.95–69.14)
Sex
Male	51 (49.5%)
Female	52 (50.5%)
Race
White	87 (84.5%)
Black or African American	5 (4.9%)
Asian	3 (2.9%)
Some other race	4 (3.9%)
Unavailable/Unknown	4 (3.9%)
Ethnicity
Hispanic	8 (7.8%)
Non-Hispanic	90 (87.4%)
Unavailable/Unknown	5 (4.9%)
Side of tumor
Left	46 (44.7%)
Right	57 (55.3%)
Volume (mm3)	321.02 (IQR: 156.67–1018.79)

IQR = interquartile range.

Then, we loaded these scans and segmentation masks to Python version 3.8 (https://www.python.org) for data pre-processing, comprising image resampling, image cropping, image standardization, tumor volume measurement and enlargement labeling. All the images were resized to the same voxel spacing (0.45 × 0.45 × 3.0mm³), and we centered cropped these 3D images to size of 384 × 384 × 16, which covers the whole brain transversely and includes the tumor and its surrounding slices axially as the model input. We selected the target voxel size based on the median value of pixel spacing (0.45 mm) and slice thickness (3.0 mm) of MR images in our dataset. VS volumes on both the initial and follow-up MRIs were measured and recorded. Volumetric tumor enlargement was defined as volume change ≥20% between initial scan and follow-up scan following the established volumetric VS enlargement criteria.^1,6 Z-score image standardization method was applied on the initial scans to address the image intensity variability problem of MRI.²⁴

Deep Multi-task Model

The workflow of the DMT model is shown in Figure 1, and the model structure is shown in Figure 2. The model consists of two parts: a 3D-UNet structure TS module for automatic deep segmentation, and an AlexNet-style TEP module for deep learning-based prediction.²⁵ After image pre-processing, the segmentation module is used to detect tumor volume from the initial contrast-enhanced T1-weighted (ceT1) MRI scan and generate attention maps (deep features), which identify tumor and its surrounding region relevant for VS enlargement prediction. The attention maps here are from different layers of the ‘decoder’ part of 3D-UNet. With the input MR image and generated deep features maps from segmentation module, the TEP module is used to identify tumor in risk of fast growth (≥20% volumetric enlargement). The manually segmented VS volumes on initial scans were used as the ground truth for training and evaluation of the TS module, although the tumor enlargement labels were used as the ground truth for constructing and evaluation the TEP module. These two modules were trained simultaneously, and the loss function is defined as follows:

$$Loss=DiceLoss\left(\widehat{x},{\widehat{x}}^{\prime }\right)+\beta \cdot BCELoss\left(y,y^{\prime }\right)$$

where x is the 3D initial ceT1 MR image, $\widehat{x}$ and ${\widehat{x}}^{\prime }$ are the mask and predicted mask of tumor volume, y and y′ are the VS enlargement label and predicted enlargement probability. The DiceLoss was used to evaluate similarities in overlap between the ground truth and model-predicted masks. The BCELoss was used to measure the binary cross entropy between the tumor enlargement label and the TEP output. β is a weighting parameter to balance the importance of TE module and TEP module during model training, and it was set as 1.0 in our experiment.

Fig. 1.

Framework of the proposed deep multi-task (DMT) approach for joint vestibular schwannoma (VS) enlargement prediction and segmentation.

Fig. 2.

The architecture of the deep multi-task (DMT) network. The boxes represent feature maps, their surrounding numbers denote the number of channels in each layer.

The network implementation is based on the 3D U-Net proposed by Çiçek et al.²⁵ We implemented it with PyTorch version 1.8 (https://pytorch.org) and the model training processing was performed with a 24 GB NVIDIA GeForce RTX 3090 Graphics Card (NVIDIA, Santa Clara, CA). The details about model structure are shown in Figure 2. We randomly initialized weights as the starting point for model training and trained the DMT model using Adam optimizer with a batch size of 16 for 100 epochs. The initial learning rate was set as 10⁻³, and it would decrease by a factor of 10 if the validation dice coefficient stopped improving for 5 epochs. Three image augmentation operations were used in our experiment: (1) random rotation in the axial plane between [−15, 15] degrees; (2) random shifting of the image in the axial plane within [−32, 32] voxels; and (3) adding zeros-mean, 0.01 standard deviation Gaussian noise on the whole image. The dropout rate was set as 0.5 during model training.

Model Evaluation and Statistical Analysis

Five-fold cross-validation was performed to train and evaluate our method. All the patients were randomly split into 1 of 5 folds, and the training and validation cohorts followed a 4:1 fashion.

We used dice coefficient to evaluate the VS segmentation module in DMT model. A separate 3D-UNet with the same structure as our TS module, and a state-of-the-art nnU-Net model were trained specifically for VS segmentation using the same 5-fold cross-validation partition. The implementation of nnU-Net is based on the open-source code released by Isensee et al.²⁶ The same image pre-processing method, data augmentation method, implementation software, and hardware platform were used. Their segmentation results were recorded and compared with DMT model. Paired t-tests were used to evaluate the statistical significance of segmentation dice coefficients.

For TEP, we applied common binary classification evaluation metrics to evaluate the performance of the proposed DMT model, including prediction SEN, specificity (SPE), ACC, and area under the receiver operating characteristic (ROC) curve (AUC). A sample with a probability value larger than 0.5 was predicted to have tumor enlargement. Two separate tumor growth prediction models, a radiomics-based model and a 3D-AlexNet model, were trained and tested following the same data splitting as DMT model for comparison. The radiomics model used 3D radiomics features extracted from the delineated tumor area for model construction, radiomic feature extraction was performed using an open-source radiomics package that is compliant with the Image Biomarker Standardization Initiative (pyradiomics, version 3.0), and the model structure followed a multi-classifier binary prediction model.^27,28 The extracted radiomic features comprise 9 intensity features, 8 geometry features and 240 texture features. We used minimum redundancy maximum relevance (mRMR) selection method for radiomic feature pre-selection and immune algorithm for iterative feature selection and radiomic model construction.^28,29 The 3D-AlexNet used the same input and network structure of the TEP module in DMT model except for the deep feature-sharing mechanism. We used DeLong test to compare the ROC curves acquired from combining the validation cohorts in the 5-fold cross-validation using these prediction methods.³⁰ All evaluation and statistics were performed using Python version 3.8, and p < 0.05 was defined as statistically significant. Definitions or explanations about the related machine learning and statistics glossary are summarized in Supplemental Digital Content 4. Detailed information about the extracted radiomics features can be found in Supplemental Digital Content 5.

RESULTS

The manually delineated VS contour and predicted contours from the single-task 3D-UNet segmentation model and the proposed DMT model are shown in Figure 3. The distribution of dice coefficients between manual segmentation masks and deep learning model-predicted VS masks were shown in Figure 4. Power calculated for this study is summarized in Supplemental Digital Content 6. All the automatic VS segmentation models were in good agreement with the experts’ delineations. The median dice coefficient from 3D-UNet model was 83.13% (IQR: 74.38%–87.27%), the median dice coefficient from nnU-Net model was 86.45% (IQR: 81.89%–90.04%), and the median dice coefficient from DMT model was 84.20% (IQR: 77.47%–87.42%). Comparing with the segmentation dice coefficient distribution of DMT model using paired t-test, the separate 3D-UNet model has significantly lower segmentation performance (p = 0.03), and the nnU-Net model has marginally better performance (p = 0.16). The mean computational time of DMT model for each testing patient was 2.68 sec (standard deviation 0.05 s).

Fig. 3.

Visualization of segmentation results on samples of different volume levels. Input image with cropped regions of interest (in the red box), ground truth masks, predicted masks from the segmentation-specialized 3D-UNet, the segmentation-specialized nnU-Net, and the proposed deep multi-task (DMT) model were listed for comparison (A, B, C, D, E in the figure for different patients, respectively).

Fig. 4.

Boxplot demonstrating the range of the dice coefficients between expert contoured tumor masks and automatically predicted tumor masks using different deep segmentation models.

Table II and Figure 5 both demonstrate the merged 5-fold cross-validation performance of TEP using different models. The proposed DMT model has the best performance on SPE, ACC, and AUC, which are 0.78, 0.72, and 0.77, respectively. The radiomics model has the best SEN (0.75). We used 0.5 as the probability threshold to calculate SPE, SEN, and ACC. When comparing the ROC curves of different models, the proposed DMT model reached statistical significance when compared with single-task deep prediction model (AlexNet, AUC = 0.60, p = 0.01), but it is marginally better than the radiomics model (AUC = 0.70, p = 0.17).

TABLE II.

Tumor enlargement prediction performance of different models.

Model	Sensitivity	Specificity	Accuracy	AUC	p-value (To Multi-task)
Multi-task	0.68	0.78	0.72	0.77	----
AlexNet	0.63	0.58	0.61	0.60	0.01
Radiomics	0.75	0.58	0.68	0.70	0.17

The bold values indicate the best results of the related metrics.

Fig. 5.

The receiver operating characteristic (ROC) curves of different VS enlargement prediction models.

DISCUSSION

In this study, we first proposed a multi-task learning framework for VS enlargement prediction based on the initial diagnosis imaging. The presented method achieved accurate prediction of VS volumetric enlargement after diagnosis (AUC = 0.77), which significantly outperformed single task model with similar structure as initially hypothesized (AUC = 0.60, p = 0.01) but did not reach significance when compared to the radiomics model (AUC = 0.70, p = 0.17). This may help identify VS patients who have higher risk of fast tumor growth in early stage, encouraging treatment, and avoiding unnecessary imaging surveillance for patients having stable VS. Our model implementation based on PyTorch is available on https://github.com/wangkaiwan/VS-Growth-Prediction.

The proposed DMT model also allows automatic VS segmentation in addition to TEP. The on average 2.68 ± 0.05 sec automatic TS speed for each patient is promising to assist in clinic, versus minutes to manually delineate a tumor by human experts. Although the segmentation performance of the DMT model (median dice coefficient of 84.20%) was lower than the state-of-the-art model nnU-Net (p = 0.16) dedicated to image segmentation task only, it is significantly better than the single-task TS model which has very similar structure as the TS module in the DMT model (p = 0.03). From the segmentation result samples shown in Figure 3, the lower dice coefficients of the segmentation masks generated with the proposed DMT model may be caused by inaccurate prediction of tumor boundary, when compared with the predicted tumor masks from nnU-Net. As the primary goal of this study is to predict VS enlargement, it is important to clarify that the TS module in the DMT framework is mainly used to identify all relevant regions that contribute to the enlargement prediction performance. As such, the TS module may have included some nearby surrounding tissues of VS tumors, resulting in lower performance of segmentation measured by dice coefficient.

Of note, some researchers have demonstrated over 90% dice score performance on VS segmentation based on ceT1 scans using deep learning methods.^9,13 Apart from the reason we explained above, we believe the following two factors may contribute the gap between our results and previous publications: (1) number of patients and (2) difference in the patient population and tumor volume size. We used ceT1 MR scans of 103 patients in this study, although the number of included patients is 243 in the study of Shapey et al. (dice coefficient of 93.43%) and 214 in the study reported by Neve et al., (dice coefficient of 92%), respectively.^9,13 As the performance of deep learning based methods is related to the size of training dataset, the gap between the model performance is expected. Furthermore, the patient population and tumor size distribution could contribute more to the gap between our model and other studies. In this work, we used the initial diagnosis MR scans for TEP, and all of the patients followed the ‘wait-and-scan’ protocol. The median volume of VS tumors was 321.02 mm³ (IQR: 156.67–1018.79 mm³). However, in the work reported by Shapey et al., all the patients had a single sporadic VS treated with Gamma Knife stereotactic radiosurgery, and the median tumor volume is 1360 mm³ (IQR: 630–3150 mm³), which is much larger than the tumor volumes in our study. Neve et al., did not report the tumor volume, but they showed the tumor size instead. Among the 134 patients with VS, 87 patients (64.9%) have tumor size bigger than 11 mm, although the median tumor size in our patient cohort is 11.2 mm.^12,13 For smaller tumors/structures, it will be more challenging to segment and achieve higher Dice coefficient.

We compared the VS enlargement prediction performance with a TEP-specific network which has the same structure as the TEP module in the DMT model. The result showed that the TS module was actually critical for TEP, as the DMT model achieved significantly higher VS enlargement prediction AUC (AUC = 0.77) than the single-task model (AUC = 0.60). The most important reason for the performance difference between these two models could be the intrinsic high-data-efficiency and low-overfitting properties of multi-task learning method.

Recently, radiomics-based methods combined with traditional machine learning techniques have been applied to different prediction tasks for VS.^{14,15,31–33} Although promising prediction performance was achieved for post-radiosurgery response/enlargement prediction and active treatment requirement prediction, none of these methods provides end-to-end prediction, because tumor delineation is a prerequisite for MRI radiomics feature extraction. In contrast to a radiomics-based strategy, the proposed DMT framework combines TS and enlargement prediction in one model, generating a prediction result from the diagnostic images, which is more clinically efficient. For comparison, we also built a radiomics-based model using the radiomic features extracted from manually contoured tumors. The result showed that the TEP performance using radiomics (AUC = 0.70) is lower than the segmentation guided VS enlargement prediction (AUC = 0.77). One possible reason is that VS boundary and surrounding structures which can be detected by the TS module provided additional information for TEP. The similar findings have been verified in the tumor biology and treatment outcome prediction research using imaging features from peritumoral region.^34,35 From this angle, the TS module in the DMT model is necessary for detecting the region of interest for prediction and making the prediction process more reliable and efficient. Nevertheless, the DMT model did not achieve statistically significant better performance in terms of AUC as compared to radiomics model. This could be caused by the relatively small sample size of our study.

We reviewed our study according to the checklist for AI in Medical Imaging (CLAIM, please check the details in Supplemental Digital Content 7), which is a guidelines for reporting evaluation of machine learning models designed for medical applications.³⁶ There are also some limitations in this study. First, the small sample size of this study, single institution, and the retrospective nature of the dataset make the constructed DMT model subject to potential selection bias. Additionally, our study did not meet the CLAIM criteria 30–32 (robustness analysis and external validation) and 35–36 (performance evaluation on all data partitions), because the effect of different imaging protocol or patient management method across institution cannot be analyzed. The generalizability and clinical efficacy of the proposed model should be further investigated via larger multicenter cohort validation and prospective study. Second, the patient selection criteria were broad, and we did not perform subclass analysis with respect to tumor characteristics, such as volumetric size, tumor position, and type of growth, due to the limitation of sample size. Further research is warranted to determine if our strategy is more appropriate for certain VS subclasses. Finally, deep learning techniques are still under rapid growth, recently proposed transformer network structure and its variations have shown better segmentation and classification performance than CNN-based ones on lots of computer vision applications, including medical-related tasks, due to their intrinsic global attention mechanism.^37–39 Incorporating these novel techniques in our multi-task framework may further improve its performance for VS enlargement prediction and segmentation and further validation is warranted in a future work.

CONCLUSION

We presented a DMT framework to predict VS enlargement and segmentation mask simultaneously using the initial ceT1 MRI. The results show that the proposed DMT model is promising and worthy of prospective investigation on a larger cohort of patients.