Deep Learning of Imaging Phenotype and Genotype for Predicting Overall Survival Time of Glioblastoma Patients

Abstract

Glioblastoma (GBM) is the most common and deadly malignant brain tumor. For personalized treatment, an accurate pre-operative prognosis for GBM patients is highly desired. Recently, many machine learning-based methods have been adopted to predict overall survival (OS) time based on the pre-operative mono- or multi-modal imaging phenotype. The genotypic information of GBM has been proven to be strongly indicative of the prognosis; however, this has not been considered in the existing imaging-based OS prediction methods. The main reason is that the tumor genotype is unavailable pre-operatively unless deriving from craniotomy. In this paper, we propose a new deep learning-based OS prediction method for GBM patients, which can derive tumor genotype-related features from pre-operative multimodal magnetic resonance imaging (MRI) brain data and feed them to OS prediction. Specifically, we propose a multi-task convolutional neural network (CNN) to accomplish both tumor genotype and OS prediction tasks jointly. As the network can benefit from learning tumor genotype-related features for genotype prediction, the accuracy of predicting OS time can be prominently improved. In the experiments, multimodal MRI brain dataset of 120 GBM patients, with as many as four different genotypic/molecular biomarkers, are used to evaluate our method. Our method achieves the highest OS prediction accuracy compared to other state-of-the-art methods.

I. Introduction

Glioblastoma (GBM) is the most common grade IV malignant brain tumor [1]. About 12,760 cases of GBM in the USA have been confirmed in 2018 [2]. GBM is also the most deadly malignant tumors with a median survival time of only 18–24 months [3], and approximately 13,000 patients in the USA die of GBM each year [4]. Traditional treatment of GBM is surgical resection followed by radiation therapy and/or chemotherapy [5, 6]. However, the inherent heterogeneity of GBM usually causes variations in prognosis, making the overall survival (OS) time largely varied across individuals [7, 8]. Therefore, the accurate pre-operative prognosis of GBM patients is desired, which can provide essential information for planning the optimized and personalized treatment.

Recently, many machine learning-based studies have been conducted for the pre-operative prognosis of brain tumor patients. Particularly, they predict OS time in days by using regression or categorize it into long- or short-term based on classification using phenotypic features extracted from various types of pre-operative image data, i.e., mono- or multi-modal magnetic resonance imaging (MRI) brain data before craniotomy. According to the strategy of feature extraction, these studies can be roughly classified into two categories: (1) hand-crafted feature-based methods, and (2) deep learning-based methods. The basic idea of the hand-crafted feature-based methods is to calculate manually engineered features with either semi- or fully-automated methods (e.g., radiomics features from tumor segmentation results [9–12] or brain network properties [13]) and use traditional machine learning approaches (e.g., random forest [14, 15]) to do regression or classification based on the extracted features. For example, in [16], radiomics features of brain tumors are extracted from a public pre-operative multimodal MRI brain tumor dataset (BraTS) [17] and fed into the random forest to learn a regression model for OS prediction. In [18], some hand-crafted features, such as volume and surface irregularity of brain tumors, are extracted from images in BraTS. These features are used to train an artificial neural network [19] for OS prediction. Although hand-crafted feature-based methods have shown promising results, there is no systematic way to determine OS-related hand-crafted features but mostly depended on experience. Therefore, the deep learning-based methods have been proposed, which automatically learn OS-related, deeply embedded MRI features for better OS prediction without any prior knowledge [20]. For example, a deep learning-based OS prediction method is presented in [21, 22], where a multi-channel convolutional neural network (CNN) [23] is proposed to automatically learn discriminative features for OS prediction from multimodal MRI brain tumor data, including the contrast-enhanced T1 (T1c), diffusion-weighted imaging (DWI) and functional MRI. The learned features are used to train a support vector machine (SVM) [24], which predicts whether the patient’s OS time is long or short. In [25], multimodal MRI including T1-weighted, T1c, T2-weighted, and T2 fluid-attenuated inversion recovery (FLAIR) together with several clinical features (e.g., Karnofsky performance score) are used to train a CNN to achieve end-to-end prediction of short, medium, or long OS time for high-grade glioma patients.

Although all the aforementioned studies have indicated an essential value of brain imaging phenotype for OS prediction, most of them did not consider of using tumor genotype (i.e., genomic and/or molecular biomarkers), another type of prognostic biomarkers that has been proven to have a strong relationship with brain tumor patients’ OS time or response to certain treatments [26–33]. For example, the promoter methylation (met) of the O-6-methylguanine-DNA methyltransferase (MGMT) was found to be a positive genomic indicator on the good prognosis of GBM [26], as the MGMT of met often indicates better reactions to chemotherapy than that of unmethylation (unmet). In addition, long-term OS time was often observed when the glioma patient carries a genomic biomarker called isocitrate dehydrogenase 1/2 (IDH) of mutation (mut) [27], as tumors with mut IDH are more frequently unilobular lesions, while tumors with wild type (wt) IDH are prone to locate at high surgical risk positions such as the brainstem. In [28], the authors showed that the combined loss of chromosomes 1p/19q, i.e., co-deletion (cd) 1p/19q, may be associated with a good prognosis for GBM than that of intact (int), as cd 1p/19 can significantly enhance the response to the radiation therapy and temozolomide chemotherapy in gliomas [29]. Another genomic biomarker telomerase reverse transcriptase (TERT) promoter mutation was reported to be correlated with the poorer OS for anaplastic astrocytoma and GBM, because of the telomere elongation and high radiotherapy resistance [30, 31]. Furthermore, it is well observed that these genomic biomarkers are often combined or co-occurred [32, 33], indicating complex genome-genome and/or genome-prognosis associations. Such a complex scenario requires a comprehensive assessment of different types of genomic signatures for making a prognostic evaluation. For example, in [32], the authors found that glioma patients with both mut IDH and met MGMT have generally longer OS than any other combinations of IDH and MGMT statuses. In [33], the patients with both mut IDH and mut TERT usually have longer OS time than those with an only mut IDH. The main reason for ignoring these genomic biomarkers in most of the existing pre-operative prognosis studies is straightforward, that is, the tumor genotype information is unavailable pre-operatively unless through autopsy via invasive surgery or craniotomy (i.e., open the skull and take tumor tissue out to the lab). In our recent study [34], we proposed a machine learning-based genomic information (IDH and MGMT) prediction method based on multimodal MRI and demonstrated the feasibility of predicting genotype with pre-operative imaging phenotype. It further inspired us to make a better prognostic prediction based on imaging phenotypic features guided by genotypic information prediction according to their profoundly close relationship.

In this paper, we propose a new deep learning-based pre-operative OS prediction method for GBM patients. Our method does not only use conventional pre-operative multimodal MRI brain data of GBM that are widely adopted previously but also learns more effective prognostic features via simultaneously predicting tumor genotype (MGMT, IDH, 1p/19q, and TERT) to better predict OS time. Particularly, we implement a multi-task CNN to conduct the joint prediction of both tumor genotype and OS time, where the genotype-related features can be better learned from the pre-operative images and contribute to the OS prediction, leading to a much improved OS prediction accuracy. Experimental results with a single-center GBM bio-bank database including multimodal MRI, genomic signatures, and OS information demonstrates that our method outperforms both state-of-the-art radiomics-based and the deep learning-based OS prediction methods.

II. Material and Method

A. Dataset and preprocessing

The brain imaging data used in our method includes T1c MRI and DWI from 120 GBM patients collected based on a routine GBM diagnosis in the collaborated hospital. All images were captured by a Siemens Verio 3T scanner using the following protocols: 3D T1c MRI using TR (repetition time) = 1900 ms, TE (echo time) = 2.93 ms, TI (inversion time) = 900 ms, flip angle = 9 degree, slice number = 176, slice thickness = 1 mm, pixel spacing = 1 mm × 1 mm, scanning time = 7 min 47 s; and DWI using TR = 9700 ms, TE = 87 ms, slice number = 70, slice thickness = 3 mm, pixel spacing = 1.8 mm × 1.8 mm, direction = 20/30, b-value of 1000 s/mm2, scanning time = 15 min 30 s. The genomic biomarkers of GBM are obtained by genomic sequencing of GBM tissue sections after craniotomy in the same hospital, and each patient has all four (MGMT, IDH, 1p/19q, and TERT) or a part of the genomic biomarkers. Based on a regular follow-up, a total of 67 out of 120 patients have known OS time (dead during the follow up) while the remaining 53 patients were still alive by their last visit time (LV). Details of our samples are summarized in Table I.

TABLE I.

Summary of the GBM patients used

Age (years)	Gender	OS (days)	LV (days)
51.6±14.6	42 F/78 M	43.9±221.5 (67/120)	62.74±327.1 (53/120)
MGMT	IDH	1p/19q	TERT
63 unmet	86 wt	50 int	50 wt
42 met	7 mut	9 cd	56 mut
15 unknown	27 unknown	61 unknown	14 unknown

We derive three modalities, i.e., B0 (T2-weighted MRI), fractional anisotropy (FA), and mean diffusivity (MD), from the DWI data based on the tensor model using PANDA [35], a well-adopted pipeline tool for diffusion MRI analysis (https://www.nitrc.org/projects/panda/). FA and MD are considered to be informative in predicting OS time [21, 22]. T1 weighted MRI is not available due to its less value in the clinical setting (thus it is not included in the database). Therefore, in total, four imaging modalities (i.e., T1c, B0, FA and MD) are used in our study. Images of different modalities (skull stripped [36]) are aligned to its corresponding B0 image using rigid transformation (rotation and translation). Moreover, for each patient, tumor mask was generated by manually labeling on the T1c images with B0 images also considered by a senior neural radiologist (Y. Xu). The tumor mask includes the tumor entity (based on the enhancement but also include necrosis) and the edema area (delineated by B0 images). Examples of the four modalities and the corresponding tumor masks from two GBM patients are shown in Fig. 1.

Fig. 1.

Examples of the four MRI modalities (T1c, B0, fractional anisotropy (FA), and mean diffusivity (MD)) and the corresponding tumor masks from two randomly selected GBM patients. B0, FA, and MD are derived from diffusion-weighted imaging (DWI).

B. Multi-task CNN for joint prediction of tumor genotype and OS time

As aforementioned, there could be different combinations of genomic statuses for a GBM patient. Therefore, we design four tumor genotype prediction tasks (each of which is for predicting the genomic status of MGMT, IDH, 1p/19q, and TERT, respectively) and an OS prediction task. These five tasks are integrated into a multi-task CNN framework, where predictive features are jointly learned for accurate prediction of the tumor genotype as well as the OS time. In this way, genotype-related features learned from tumor genotype prediction tasks can be used to guide OS prediction, while the prognosis- or OS prediction-related features learned from the OS prediction can also help the tumor genotype prediction. Such a mutual benefit on the multiple tasks is expected to boost the performance of all the tasks as they are closely related to each other. An overview of the structure of the proposed multi-task CNN is shown in Fig. 2.

Fig. 2.

Overview of the network structure of the multi-task CNN for both genotype and OS prediction. There are five tasks: four for tumor genotype prediction, and one for OS prediction. Tumor genotype-related feature representations can be learned better with the help from the OS prediction, and vice versa.

The input of the multi-task CNN is a 3D multimodal (T1c, B0, FA, and MD) MRI patch covering the entire tumor. Generating such patches for training and testing will be described later in the Experiments. Briefly, after the first two common convolutional blocks, the network is split into five branches, each of which corresponds to one of the prediction tasks. High-level features learned from the four genotype prediction tasks are fed to the fully connected layer (FC) for the OS prediction task to provide it with tumor genomics features. Combining with the OS-related features, it leads to more accurate OS prediction. Moreover, other important clinical features (i.e., patient age, gender, tumor size, and tumor location) are fed to the FC layers of each prediction task. The tumor location is encoded by 27 binary digits as we evenly divided each brain into 3×3×3 non-overlapping blocks. If a block contains any part of the tumor, the corresponding digit of that part is denoted as 1, otherwise 0.

For each tumor genotype prediction task, it has two possible statuses: MGMT is either methylation (met) or unmethylation (unmet), IDH (or TERT) is either mutation (mut) or wild type (wt), and 1p/19q is either co-deletion (cd) or intact (int). For the OS prediction task, exact OS time in days is the target (i.e., it is a regression rather than a classification problem). Therefore, in the training phase, a softmax loss is applied in each tumor genotype prediction task for the binary prediction task, while the loss function in the OS prediction task is the Euclidean distance to the ground truth.

C. Creating training and testing sets

As aforementioned, the training and testing data of our network are 3D multimodal MRI patches, each of which contains an entire tumor. Since we have a tumor mask image for each patient, the center of the tumor, as well as the tumor bounding box, can be calculated. By calculating all the tumor bounding boxes for all the patients, the size of the 3D multimodal MRI patch can be determined (set to 64×64×32 voxels), large enough to cover entire brain tumors for all the patients. For the training data, the corresponding labels include four genomic biomarker statuses (“0” for unmet, wt and int, “1” for met, mut and cd, and “−1” for unknown) and OS in days (−1 for no OS information by LV). To augment the training data, we rotate tumors in the axial direction around the tumor center in the range of 0°~359° with a step size of 10° to generate “new” 3D multimodal MRI patches and we also generate their mirrored MRI patches. Moreover, to solve the imbalanced samples of different IDH statuses (86 wt vs. 7 mut) and 1p/19q (50 int vs. 9 cd), a step size of 2° is used during the rotation for IDH of mut and 1p/19q of co-deletion to generate more samples. In this way, 12,672 3D multimodal MRI patches with corresponding labels of four genomic statuses and OS time are available for training and testing. It is worth noting that, for incomplete samples with “−1” label (i.e., unknown genomic biomarker status or no OS time), no backpropagation is conducted at the corresponding multi-task CNN branches during the training. Therefore, “−1” is an indicator for no backpropagation, and the output of each branch in our network is either “0” or “1” (for each genotype prediction) or OS time (in days). No “−1” will be generated.

III. Experiments and Results

A. Cross-validation and competing methods

We adopt 10-fold cross-validation [37] to evaluate our method. At each time, 3D multimodal MRI patches with corresponding labels from 108 patients are used for training (90 patients) and validation (18 patients). The original (not augmented) 3D multimodal MRI patches of the rest 12 patients are used as the testing set. Balance of the training data (especially for IDH and 1p/19q) is considered. For comparisons, five conventional mono-task CNNs, each of which handles one of the tumor genotype predictions or the OS prediction task, are also constructed and evaluated. The architecture of each mono-task CNN is identical to that of any branch of the multi-task CNNs (Fig. 3). In addition, to demonstrate the effectiveness of adding genotype information to the OS prediction model, we build a new network based on the same architecture as the conventional mono-task CNN but with four genomic biomarkers added as additional clinical features besides age, gender, tumor size and position (they are fed to the second last layer, Fig. 3). Specifically, four digits are used to encode the statuses of the four genomic biomarkers in an order of MGMT, IDH, 1p/19q, and TERT, with “0” representing unmet/wt/int status and “1” for met/mut/cd status, while “−1” for unknown status, at each digit. We denote the mono-task CNN with additional genomic biomarkers as mono-task CNN+GB.

Fig. 3.

Mono-task CNN for tumor genotype prediction or OS prediction and mono-task CNN+GB for OS prediction (the competing models).

The batch sizes of our method and the six conventional mono-task CNNs are the same, which are set to 3. The training image data contain more than 10,000 3D multimodal MRI patches. The maximum iteration is set to 100,000 (i.e., around 30 epochs) for our method and the six mono-task CNNs. Considering that radiomics features [9, 10] combined with a random forest (denoted as RD-RF) [14, 15] are widely used in tumor prognosis prediction [38–40], RD-RF is also evaluated as one of the competing methods. Fig. 4 illustrates the flowchart of the RD-RF method. Specifically, radiologic features of input images are first extracted in the radiomics engineering stage. Then the resulted features are refined in a feature selection stage. The selected features are fed into a random forest for classification (genotype prediction) or regression (OS prediction), separately. In our experiment, five RD-RFs are trained, each of which is responsible for one tumor genotype prediction task or an OS prediction task. Each random forest includes 100 decision trees. The demographic and clinical features are also added to the selected radiomics features and fed into random forests. Of note, we did not train an all-in-one RD-RF model with all five tasks in one forest (or in five random forests separately but jointly making predictions) because it requires complete data, which results in inadequate samples and low performance (results not shown).

Fig. 4.

Radiomics and random forest (RD-RF) based genotype prediction or OS prediction

Same 10-fold cross-validation is adopted to evaluate the five RD-RFs for fair comparisons. The training data are the same as that used in our method and the mono-task CNNs. Since RD-RF requires complete data for each patient, we exclude the patients who have no corresponding tumor genotype or still alive (i.e., without OS time for training the OS prediction model), which results in 105 patients for MGMT status prediction, 93 for IDH status prediction, 59 for 1p/19q status prediction, 106 for TERT status prediction, and 67 for OS prediction. In the radiomics feature extraction stage, 93 widely used radiomics features, including both first-order and second-order features, are extracted from each MRI modality within the same 3D patches. Specifically, the first-order statistics include energy, entropy, mean, range, and so on, while the second-order descriptors consist of a gray-level co-occurrence matrix, a gray-level size zone matrix, and a gray-level run-length matrix. Furthermore, additional 16 features describing the shape information of the tumor are derived from the corresponding 3D tumor mask images, representing the overall tumor shape. These features have been widely used in the radiomics studies [16, 18]. The radiomics-based machine learning tool, Pyradiomics [41], is used to extract these features. Four additional clinical features (age, gender, tumor size, and location) are also used. In this way, the training data of each patient contains 392 features (i.e., 93 × 4+16+4). We applied ℓ₁ regularization-based feature selection [42]. A total of 47 features are selected for MGMT, 1p/19q, and TERT predictions, 59 features are selected for IDH prediction, and 62 features are selected for OS prediction.

B. Result evaluation

The OS time predicted by different methods for the 67 patients who have known OS time is compared to the real OS time by using the root mean squared error (RMSE). We also calculate the Pearson correlation coefficient (CC) between the real OS time and the predicted OS time. The predicted tumor genotype from each method is evaluated and compared with metrics including accuracy, sensitivity, and specificity. Table II shows the details of the evaluation. In addition, the Wilcoxon signed-rank test [43] with a significance level of 0.05 is performed over the predicted OS time among all the methods under evaluation. Specifically, compared to our method, the mono-task CNN for OS prediction reaches a worse performance (p = 0.0033), and the performance of the RD-RF is slightly better than the mono-task CNN but still poorer than that of our method (p = 0.0341). The mono-task CNN+GB (with a single OS prediction task but with full information of genomic biomarker statuses as additional clinical features) results in a good performance but with no statistically significant difference compared to our method (p = 0.0749). However, the mono-task CNN+GB significantly outperforms other mono -task CNNs (indicating the effectiveness of the genomic biomarker information in OS prediction). Of note, in the mono-task CNN+GB, genomic biomarker statuses are simply treated as ordinary clinical features (like age, tumor size and position); the comprehensive features related to the genotype-imaging phenotype associations cannot be jointly learned from the neuroimaging data. More importantly, the mono-task CNN+GB cannot be implemented in a real clinical scenario, where the statuses of genomic biomarkers usually remain unknown prior to the operation.

TABLE II.

Evaluation of the performance of mono-task CNN, mono-task CNN +GB, RD-RF, and our method on genotype and os prediction.

		MGMT	IDH	1p/9q	TERT	OS (RMSE / CC )
Mono-task CNN	Accuracy	0.724	0.925	0.814	0.632	26.1±175.0/0.587
	Sensitivity	0.730	0.965	0.880	0.640
	Specificity	0.714	0.429	0.444	0.625
Mono-task CNN+GB	-	-	-	-	-	197.0±134.0/0.4424
RD-RF	Accuracy	0.676	0.925	0.763	0.575	225.0±13.60/0.1151
	Sensitivity	0.683	0.988	0.860	0.560
	Specificity	0.667	0.143	0.222	0.589
Our method	Accuracy	0.790	0.946	0.881	0.660	177.0±130.0/0.4695
	Sensitivity	0.794	0.965	0.920	0.680
	Specificity	0.786	0.714	0.667	0.643

The left panel of Fig. 5 illustrates the predicted OS time using the four methods (our method and the three competing methods) vs. the ground truth of the 67 patients who have known OS time. It is clear that the predicted results from our method are more comparable to the ground truth than those from the mono-task CNN and the RD-RF. In addition, the OS time for the remaining 53 patients who were still alive by LV is also predicted by the three methods. As no ground truth has been established for them, we treat the prediction result as valid if the predicted OS time is longer (as it should be) than the period between the date of being diagnosed and the date of LV (i.e., the follow-up period). Again, our method leads the other methods for these 53 patients. Specifically, 32 out of 53 patients have longer predicted OS time than their follow-up period, while the number for the mono-task CNN, the mono-task CNN+GB and the RD-RF are 18, 29 and 20, respectively.

Fig. 5.

Left: scatter plots of the OS prediction results for the 67 patients against the ground truth using the mono-task CNN (a), mono-task CNN+GB (b), RD-RF (c) and our method (d); Right: survival curves plotted based on the predicted OS time for all the 120 patients (with or without OS ground truth) using three methods (blue: mono-task CNN, yellow: mono-task CNN+GB, green: RD-RF, purple: the proposed method). For reference, the real survival curve is plotted in red.

The survival curves (Kaplan-Meier plot) [44] of the 120 patients (including 67 with OS time and 53 without OS time) are plotted in the right panel of Fig. 5. This is another performance evaluation strategy for survival analysis. Survival curve, unlike the RMSE that can only be calculated using a subset of patients with known OS time, can be drawn using all the patients no matter whether they have or do not have known OS. Specifically, the survival curve based on the predicted results of our method (Fig. 5 (h)) is much closer to the real survival curve than those of the mono-task CNN (Fig. 5, (e)), the mono-task CNN+GB (Fig. 5, (f)) and the RD-RF (Fig. 5, (g)). Further log-rank tests [45] between the predicted curves and the true curve indicate that the mono-task CNN led to significantly discrepant survival curve compared to the true curve (p = 0.0425), and the mono-task CNN+GB and the RD-RF results show some discrepancy with p = 0.4224 and 0.1925, respectively. Our method leads to the largest p of 0.5152.

Furthermore, we divide the 67 GBM patients with known OS into two groups (i.e., long and short OS groups) based on their predicted OS time (threshold = 450 days, according to [46]). We then use a two-sample t-test [47] to compare their ground truth OS time between the two groups. If the OS prediction model well performed, the two groups divided based on the predicted OS should have significant differences in the real OS. Therefore, the smaller the p value is, the better the prediction performance could be. The resultant p values are 0.5182 (mono-task CNN), 0.0007 (mono-task CNN+GB), 0.0772 (RD-RF), and 0.0101 (our method). The result further validates that mono-task CNN+GB and our method outperform mono-task CNN and RD-RF. While the p value is smaller from the mono-task CNN+GB than that from our method, it could indicate larger separation between the short OS and long OS groups, rather than more consistency between the predicted and the real OS (as the latter was proved by a higher Pearson correlation coefficient (CC) and a better estimated survival curve by our method, see Fig. 5d, h and TABLE II).

C. Importance of different genomic biomarkers in OS prediction

It is also interesting to evaluate which genomic biomarker is more important in OS prediction and what is the order of importance for the four genomic biomarkers. To this end, we modify our multi-task CNN to several “dual-task CNNs”, each of which jointly predicts one of the genomic biomarker statuses and the OS time. The features learned from each single genotype prediction are fed to OS prediction, similar to the multi-task CNN. The same data and 10-fold cross-validation are adopted. Their respective OS prediction performances, measured by both RMSE and CC, are compared. MGMT leads to the best RMSE (182.0±131.0 days) and the second best CC (0.3520), while IDH has a comparable performance (RMSE = 192.0±157.0 days, CC= 0.3540), both of which are higher than those of the other two genomic biomarkers (1p/19q: RMSE = 212.0±188.0 days and CC= 0.2205, TERT: RMSE = 243.0±177.0 days and CC= 0.1738). Out of the 53 patients without OS time, 26 patients have predicted OS longer than their follow-up period using MGMT. The number is 25 for IDH, and 20 for 1p/19q and TERT.

We divide the GBM patients with known OS into two groups according to their predicted OS with each dual-task CNN and conduct statistical comparison of their ground truth OS between the two groups. The resulting p values are 0.0108 (MGMT), 0.0116 (IDH), 0.3082 (1p/19q), and 0.1577 (TERT), which further validate the result based on RMSE and CC (TABLE II).

According to these evaluation results, MGMT seems the most important genomic biomarker in OS prediction of all four genomic biomarkers, followed by IDH, 1p/19q, and TERT. Of note, even though only one genomic biomarker is involved in the OS prediction, the performance is still much better than that without using any genomic biomarker (mono-task CNN and RD-RF). The predicted OS time using the four dual-task CNNs vs. the ground truth of the 67 patients who have known OS are drawn using scatter plots as shown in Fig. 6.

Fig. 6.

Scatter plots of the OS prediction results for the 67 patients against the ground truth using four different dual-task CNNs. Each dual-task CNN deals with a joint prediction of one of the four genomic biomarkers and the OS time.

D. Importance of different imaging modalities in OS prediction

We also evaluate which MRI modality is more important for OS prediction. To do so, we use each MRI modality (i.e., monomodal MRI) as well as the genomic information as input to train our multi-task CNN. Specifically, T1c, B0, FA, and MD are separately fed into a multi-task CNN that jointly predicts all four genomic biomarkers status and the OS time. The same data and validation strategy (10-fold cross-validation) are used. The experimental results show that B0 leads to the smallest RMSE (218.0±179.0 days) with the highest CC (0.1947), similar to those for FA (RMSE = 222.0±193.0 days, CC = 0.0914) and MD (RMSE = 221.0±187.0 days, CC = 0.1872). T1c results in the largest RMSE (263.0±199.0 days) with CC = −0.0323. Out of the 53 patients without known OS time, 25 patients have predicted OS time longer than their follow-up period using only B0 or only FA. The number is 21 for MD and 20 for T1c. Detailed OS prediction results are shown in Fig. 7.

Fig. 7.

Scatter plots of the OS prediction results for 67 patients against the ground truth using multi-task (predicting both genomic biomarkers and OS time) CNNs with monomodal MRI.

Separated into long and short OS groups based on the predicted OS with each imaging modality, the patients’ real OS did not differ significantly between the two groups, where the p values based on two-sample t-tests are 0.6870 (T1c), 0.3533 (B0), 0.2757 (FA), and 0.2978 (MD).

From the above results, we can see that the B0 (T2-weighted MRI), FA, and MD have comparable performance, which is slightly better than T1c if only one imaging modality is allowed to use. In addition, B0 has the highest CC of all imaging modalities. However, the evaluation results also suggest that monomodal MRI is insufficient to achieve as good OS prediction as multimodal MRI does.

IV. Discussion

In this paper, we propose a new multi-task deep learning model to predict genomic biomarkers and OS time for GBM patients based on multimodal MRI. We found that the inclusion of the genomic biomarker prediction tasks significantly increases the accuracy of prognosis of GBM (reducing the mean RMSE by 84 days when comparing to the mono-task CNN, Table II), which confirms our hypothesis that the prognostic features could be better learned with the guidance of genotype prediction and further proves close associations among imaging phenotype, genotype, and clinical outcome (also see Figs. 5 and 6). Meanwhile, from Table II, the inclusion of OS prediction task also improves each genomic biomarker prediction in return. Compared to another recently trending technique, i.e., radiomics feature-based machine learning (RD-RF), our deep learning model (CNN) combines feature learning with classification/regression, thus significantly improves OS prediction accuracy (Table II). This further demonstrates the superiority of deep learning models compared to conventional machine learning (with separate feature engineering and classification/regression) in the medical imaging analysis. Moreover, the correlation coefficient of the predicted OS results using our model is 0.4695, which is the highest of all the methods under evaluation. To our best knowledge, it is the first paper simultaneously predicting genomic biomarkers and clinical outcome for GBM patients in a unified deep learning framework.

Furthermore, we carried out more experiments and found the importance of four major genomic biomarkers in the prognosis analysis follows the order of MGMT > IDH > 1p/19q > TERT (Fig. 7), which is consistent with the findings from the previous study [48]. Particularly, the MGMT methylation status usually indicates a sensitivity to chemotherapy in GBM patients [48], which can be directly related to the treatment outcomes. IDH and 1p/19q are, however, often used in grading for WHO grades II-III gliomas [45] and could be less useful for GBM (grade IV) prognosis. This finding may inspire the doctors to conduct an additional test to identify the MGMT status for the patients with potential GBM and make a corresponding treatment plan to improve their survival. Of note, such genomics preference may also be caused by the uneven distributions of different genomic statuses in GBM (see Table I, where the two statuses of IDH and 1p/19q are highly imbalanced [49]). However, TERT also shows an even distribution of the two statuses in the GBM samples we had, but it seems that TERT does not quite contributive to the OS prediction. A possible reason could be that the prognostic impact of TERT may further depend on both IDH and MGMT statuses [50].

Importance evaluation of different imaging modalities in the OS prediction for glioma patients has always been one of the hottest topics in the central nervous system (CNS) tumor studies [51–55]. Actually, it is one of the goals of a famous challenge with a large-sample publicly available tumor MRI dataset, multimodal brain tumor image segmentation benchmark (BraTS, 2012–2019) [17, 56, 57]. It includes four different MRI modalities (T1, T1c, T2, and FLAIR) with both tumor segmentation and outcome prediction as goals. The differences between our dataset and BraTS are that we only focus on the most aggressive glioma (i.e., GBM) while the BraTS data include both low- and high-grade gliomas (grades II-IV) and that the BraTS data include four high-resolution images while we only have two (T1c and DWI, where B0, FA, and MD are all the diffusion metric maps derived from the DWI). Therefore, our OS prediction could be more difficult than that of the BraTS challenge with less cost. In addition, our method does not require accurate tumor tissue segmentation and does not depend on tumor resection results, thus is more flexible and can be used as a purely presurgical prediction.

Compared to our previous study with functional MRI, DWI, and T1c for high-grade glioma OS prediction [21], our current results are similar, both emphasizing the importance of DWI and its derivatives (FA and MD) in OS prediction. Specifically, we found that the importance in OS prediction for different MRI modalities follows such an order: B0 ≈ MD ≈ FA > T1c, and the B0 is the most important image for OS prediction. Since B0 image is nearly a T2-weighted MRI, this result indicates that T2 MRI could be more important than T1c for the prognostic purpose for GBM. On the other hand, T1c is more important for grading brain tumors (enhancing areas mean high-grade gliomas and non-enhanced tumors are likely low-grade ones) and may be related to OS when all grades of the tumors are involved [58]. However, in our study, all the brain tumors are WHO grade IV GBM and all of them show enhancement in T1c, and T1c may only provide very limited prognostic information in this scenario. Previous studies reported that the peritumoral abnormalities (e.g., edema and infiltration) in gliomas could be closely related to OS compared to enhancement [59, 60]. Since B0 could show the edema region that can be used to evaluate tissue press or constriction due to tumors, tumor infiltration, tumor growth speed and aggressiveness [61], it can be more informative to the GBM prognosis. Similarly, MD and FA are also reported that both may contain infiltration information in the peripheral regions around gliomas [62].

Nevertheless, we emphasize the importance of multimodality-based OS prediction for GBM, as we found that under the same multi-task CNN (all with help from genotype prediction), predicting OS time with multimodal MRI is significantly more accurate (the error was reduced by 41 days as reflected by mean RMSE) than the best result of monomodal MRI (B0)-based prediction. The main reason is that different MRI modalities could provide complementary information about the tumor. For example, T1c can provide information of the enhancing tumor entity and can thus separate it from the necrotic region [63], while information of edema and texture information inside of the tumor can be better obtained by T2 (B0 from DWI in our study). Furthermore, FA and MD were reported to be capable of providing tumor infiltration information for the evaluation of its aggressiveness [62]. Therefore, we highly recommend that other MRI modalities (especially DWI) should be acquired if possible in addition to T1c for GBM prognostic evaluation in the future.

Our work is not without limitation. First, we only used DWI and T1c due to the limited modalities of the current data. We assume that including additional modalities (T1 and T2-FLAIR) could further improve genotype and OS prediction accuracy. Second, we used as many samples as possible by also including the patients without all four genomic markers. While our model can be more flexible and more suitable for future clinical applications, the missing data could still affect the performance. Third, our method only uses available presurgical data including pre-operative MRI and tumor location; in the future, intra-operative [64] and post-operative images, as well as the tumor resection information could be included to further update the OS prediction.

V. Conclusion

We proposed a new multi-task convolutional neural network (CNN) framework for pre-operative overall survival (OS) time prediction by simultaneously predicting important genomic biomarkers for glioblastoma (GBM) patients, which significantly outperform the conventional radiomics-based machine learning method and the mono-task deep learning model without prediction of genomic biomarkers. We innovatively integrated glioma genotype (MGMT, IDH, 1p/19q, and TERT) prediction to facilitate OS-related features learning, which significantly improves GBM prognosis accuracy. We found that MGMT status and diffusion-weighted imaging (DWI) are two important genotype and phenotype features in OS prediction. Our findings indicate the importance of genome-genome and/or genome-prognosis associations, highlight the close relationship among imaging phenotype, genotype, and clinical outcome, and also provide valuable clinical guidance for GBM treatment planning.