A deep learning fusion model for brain disorder classification: Application to distinguishing schizophrenia and autism spectrum disorder

Abstract

Deep learning has shown a great promise in classifying brain disorders due to its powerful ability in learning optimal features by nonlinear transformation. However, given the high-dimension property of neuroimaging data, how to jointly exploit complementary information from multimodal neuroimaging data in deep learning is difficult. In this paper, we propose a novel multilevel convolutional neural network (CNN) fusion method that can effectively combine different types of neuroimage-derived features. Importantly, we incorporate a sequential feature selection into the CNN model to increase the feature interpretability. To evaluate our method, we classified two symptom-related brain disorders using large–sample multi-site data from 335 schizophrenia (SZ) patients and 380 autism spectrum disorder (ASD) patients within a cross-validation procedure. Brain functional networks, functional network connectivity, and brain structural morphology were employed to provide possible features. As expected, our fusion method outperformed the CNN model using only single type of features, as our method yielded higher classification accuracy (with mean accuracy >85%) and was more reliable across multiple runs in differentiating the two groups. We found that the default mode, cognitive control, and subcortical regions contributed more in their distinction. Taken together, our method provides an effective means to fuse multimodal features for the diagnosis of different psychiatric and neurological disorders.

1. Introduction

Current diagnosis of mental illnesses heavily relies on clinical symptoms that do not reflect neurobiological substrate. Considerable effort has been made to identify mental disorders such as schizophrenia [1, 2] and autism spectrum disorder [3] by utilizing biologically meaningful features. Deep learning techniques have been applied to the neuroscience field [4] for differentiating brain disorders based on neuroimaging measures [5-9]. However, most previous work employs unimodal features, such as resting-state functional connectivity [10] or gray matter measures [11]. There has been evidence that fusion of different neuroimaging features can improve the ability in differentiating and predicting brain disorders [12, 13]. Unfortunately, jointly utilizing different types of neuroimaging measures in a deep learning model can be challenging.

Neuroimaging data has been widely used to investigate biomarkers of brain disorders and distinguish patients from healthy controls. The mostly used features include 1) brain functional networks and connectivity [14, 15] reflecting the interaction between separate brain regions and 2) brain structural morphology such as gray matter volume and density [16]. Among deep learning methods, convolutional neural network (CNN) model works well in automatically detecting important features through preserving spatial locality of neuroimaging measures. CNN has been applied to separate Alzheimer’s disease (AD) [17], mild cognitive impairment (MCI) [18], and schizophrenia (SZ) [19] from healthy group. However, most of the studies utilized a single type of features, likely due to the challenges of combining different high-dimensional measures.

Indeed, there has been some work which incorporates multimodal measures to differentiate mental disorders via deep learning. Salvador et al. [20] used gray matter volumes, functional activation maps, and resting-state low-frequency fluctuation and whole-brain connectivity maps, and then tested a set of strategies of fusing the classification results from unimodal classifiers. They suggested that a two-step sequential integration method performed better than other fusion methods but did not improve much beyond a unimodal classification approach. In the two-step method, an initial unimodal classification was used to select the most informative voxels from each 3D map, and then a second classification only considered those important voxels as inputs. Such an approach may not provide optimal use of joint information. Liu et al. [21] constructed cascaded CNNs to learn the multimodal features of magnetic resonance imaging (MRI) and positron emission tomography (PET) images for AD classification. In their method, multiple 3D CNNs were used on different local image patches, followed by a 2D CNN that helps generate multimodality correlated features. One of the challenges of this approach is the difficulty in explaining the contributing features. In this paper, we propose a new CNN fusion model to learn and combine different types of features for distinguishing patients with mental disorders, by taking advantage of a sequential forward feature selection and multi-level CNN fusion.

We evaluated our fusion method using neuroimaging data from schizophrenia (SZ) and autism spectrum disorder (ASD) patients, since the two disorders share similar clinical symptoms. Although SZ is defined as a psychiatric disorder and ASD comprises a range of neurodevelopmental disorders, SZ and ASD were previously considered to be the same disorder. Their differentiation has been a long-standing unsolved problem given the challenge of their heterogeneity and non-specificity [22-25]. Whole-brain functional connectivity [3, 26] has been used to compare and classify SZ (or ASD) from healthy population. Mastrovito et al. [27] classified SZ and ASD using brain effective connectivity estimated from resting functional magnetic resonance imaging (fMRI). In their study, a support vector machine (SVM) classifier was applied, resulting in a 75% SZ vs. ASD classification accuracy. No work has utilized both brain functional and structural information to classify the two disorders. In this paper, we evaluate our deep learning fusion model by applying it to the classification between SZ and ASD individuals based on multimodal neuroimaging measures.

2. Materials and Methods

2.1. Materials

We included fMRI and structural MRI (sMRI) data of 335 SZ and 380 ASD patients from the Bipolar-Schizophrenia Network for Intermediate Phenotypes-1 (BSNIP-1), Function Biomedical Informatics Research Network (FBIRN), Maryland Psychiatric Research Center (MPRC), and Autism Brain Imaging Data Exchange I (ABIDE I). For fMRI data, we removed the first six time points of volumes and then performed rigid body motion and slice-timing correction. The fMRI data were subsequently warped into standard Montreal Neurological Institute (MNI) space, resampled to 3 × 3 ×3 mm³ voxels, and smoothed using a Gaussian kernel with a full width at half maximum (FWHM) = 6 mm. For sMRI data, we segmented the structural images into gray matter, white matter and cerebral spinal fluid with modulated normalized parameters, and then smoothed the images using a Gaussian kernel with an FWHM = 6 mm. Finally, the preprocessed fMRI and sMRI data that passed the quality control were maintained.

2.2. Method

Our method can handle different neuroimaging measures represented as 3D/2D matrices and vectors for the classification between different groups. In this work, we employed multiple brain spatial functional networks, functional connectivity matrix, and brain structural morphology volume as the inputs. Specifically, for each subject, we estimated 53 brain functional networks and a functional network connectivity (FNC) matrix from resting-state fMRI data using our previously developed fully automated NeuroMark approach [28]. The data-driven independent component analysis (ICA) method is able to extract brain functional features with both individual-subject specificity and inter-subject correspondence, by taking advantaging of the network templates from a large sample of population as priors and a multiple objective optimization method [29]. The obtained 53 functional networks, each of which was represented by a 3D matrix, were assigned into seven functional domains including the sub-cortical (SC), auditory (AU), sensorimotor (SM), visual (VI), cognitive control (CC), default mode (DM), and cerebellar (CB) domains. Next, the FNC, represented by a 2D matrix, was obtained by computing the Pearson correlation coefficients between the time series of functional networks, thus reflecting the temporal interaction between different brain functional networks. In addition to the brain functional measures, we computed gray matter volume (GMV) from sMRI data, resulting in a 3D matrix for each subject. Finally, 53 brain functional networks (3D matrices), a 2D FNC matrix, and a 3D gray matter volume matrix were taken as inputs for each subject in the classification process.

Regarding both our proposed CNN fusion model using multiple brain measures and the CNN model only using single type of measures, we tested their performance via a 5-fold cross-validation procedure. The training and testing samples were consistent between the classifications using different methods. Figure 1 shows the CNN models in the classification using single type of neuroimaging measures. For the unimodal comparison, 3D CNNs were taken as the model for brain functional networks and gray matter volume, while a 2D CNN was used for functional network connectivity. For the 2D/3D CNN, the input was convolved by 2D/3D convolutional layers with rectified linear unit (ReLU) [30] activation, and each convolutional layer was followed by a max-pooling layer to down-sample the feature maps.

Figure 1:

The CNN models in the classification using single type of neuroimaging measures. For brain functional networks and gray matter volume, 3D CNNs were used. For functional network connectivity, a 2D CNN was used.

Figure 2 shows the pipeline of our method. We combine the characteristics from different types of measures to improve classification of brain disorders. We first pretrained a 3D CNN model for the brain functional networks of each functional domain, a 3D CNN model for the gray matter volume, and a 2D CNN for the FNC measures, separately. Next, the nine resulting CNN models were combined by adding additional fully-connected layers and then fine-tuning with the pre-trained CNN models fixed to build a unified model. The unified model was evaluated using the testing data. For the spatial functional networks in each domain, we performed a feature selection before pretraining, considering that the effects of different networks can be different.

Figure 2:

The pipeline of our proposed fusion model and the validation.

Feature selection plays a key role in classification, especially for the high-dimensional network measures. The features automatically transformed in deep learning are hard to decode. How to incorporate additional feature selection in deep learning is also difficult. In our method, a sequential forward network selection combined with the 3D CNN model was used to achieve the discriminative network set of each domain via an inner 5-fold cross-validation procedure within the training data. To do this, the spatial networks were added one by one as inputs for 3D CNN using the inner training data in each run to find a set of networks which maximized the classification accuracy on the inner testing data. Next, the networks that frequently presented in the optimal network sets across different runs were taken as discriminative networks. Finally, the selected networks were summed as inputs for pre-training 3D CNN.

To evaluate the classification performance for each method, we computed the accuracy, sensitivity, and specificity according to the predicted label of the testing data [31], and then recorded the mean and standard deviation of each metric across different runs for a comprehensive comparison between different methods. Furthermore, we also performed a two-sample t-test on the classification accuracy to compare the performance between our CNN fusion method and any single-modal method.

3. Results

3.1. The estimated neuroimaging measures

Figure 3 shows an example of the 53 brain spatial functional networks, including five SC, two AU, nine SM, nine VI, 17 CC, seven DM, and four CB networks. In our comparison experiments, each functional network was used as input for a 3D CNN model for classifying SZ and ASD. The FNC matrix reflects the temporal interactions between different functional networks, with a higher correlation strength representing stronger interaction between two networks. As demonstrated in Figure 4, the mean FNC matrixes showed a consistent pattern across datasets from different studies, indicating that the FNC strength was comparable between individuals and thus a good candidate for use in classification. It is also observed that functional connectivity between networks within the same functional domain (e.g. default mode domain) showed overall higher strength than those between networks from different domains. The blocky patterns motivate the use of a 2D CNN model for the single-type feature experiment. The 3D gray matter volume embedding voxel-based morphometry had larger dimensionality than each functional network. Similarly, the gray matter volume was used to train a 3D CNN model for distinguishing SZ and ASD patients, in order to compare with our fusion method.

Figure 3:

The 53 3D brain spatial functional networks that were assigned into seven functional domains. The networks in the same functional domain are displayed using different colors.

Figure 4:

The mean FNC matrix across subjects for BSNIP, FBRIN, MPRC, and ABIDE I, respectively. Each element in FNC matrix reflects the connectivity between two networks.

3.2. Our CNN fusion method outperformed the CNN models using single type of features

In the following, we report the results from both the CNN models using different single type of features and our fusion method. Regarding each evaluation measure (accuracy, specificity, and sensitivity), the mean and standard deviation across different runs are shown in Figure 5. Since different functional networks resulted in varied performances, we recorded the mean classification accuracy for networks in the same domain for a summary. Using single network, the cerebellar (mean accuracy =75%) and default mode (mean accuracy =70%) yielded a higher accuracy in differentiating the two disorders than other domains. FNC and GMV resulted in the same mean accuracy (77%) in the single-type feature methods; however GMV yielded higher sensitivity and lower specificity than FNC. As expected, the highest accuracy (mean accuracy = 87%) was achieved using our CNN fusion model by the combination use of multiple types neuroimaging features, compared to using single type of features alone. Furthermore, the standard deviation of each metric was smaller in our method relative to other methods, supporting that our fusion method was more robust.

Figure 5:

Classification evaluation using the testing data. In (A), we show the classification accuracy using different single type of measures and our fusion method. Since there were different networks, we averaged the classification accuracy across different networks in each domain for a summary. In (B), we show the classification accuracy obtained using the selected networks combination of each domain in our method for a comparison. Our fusion method is shown in parallel for comparison. In (A) and (B), for the classification accuracy, sensitivity, or specificity, the evaluation metrics in different runs were shown using an error bar.

As we mentioned, different networks in the same domain were utilized to search an optimal network combination through a sequential feature forward selection in our method. In Figure 5, we also show the classification accuracy using the network combination of each domain as the input of 3D CNN for an additional test. By using the optimal network set as input, the subcortical networks got the highest accuracy (mean accuracy = 83%), and default mode networks also performed well (the mean accuracy = 82%), both of which were higher than the accuracy of using single network.

Table 1 includes the statistical results from comparing our fusion method and the single-modal methods. The results show that our fusion method significantly outperformed any method using single-modal features, as all p-values from two-sample t-tests (fusion vs. single-modal methods) were smaller than 0.01. Also, we found that the improvement in our fusion method was greater relative to the functional network features than the GMV and FNC features.

Table 1:

Two-sample t-test results for comparing our fusion method and single-modal methods

Comparison between our fusion method and single-modal method	T-value	p-value
Fusion vs. Sub-cortical network features	8.10	1.26e-03
Fusion vs. Auditory network features	29.59	7.76e-06
Fusion vs. Sensorimotor network features	20.14	3.59e-05
Fusion vs. Visual network features	34.47	4.24e-06
Fusion vs. Cognitive control network features	31.80	5.83e-06
Fusion vs. Default mode network features	59.88	4.66e-07
Fusion vs. Cerebellar network features	10.26	5.08e-04
Fusion vs. FNC features	10.60	4.49e-04
Fusion vs. GMV features	5.91	4.11e-03

Taken together, in our method the network combination took effect by jointly taking advantage of different networks, and the fusion of different types of features further improved the classification performance.

3.3. The discriminating brain functional networks selected by our method

As described above, in each run, our method automatically selected one subset of functional networks in each domain by our proposed sequential forward selection algorithm. For each functional domain, we further summarized the identified network subsets across different runs to show which brain regions were mostly associated with the differences between the two disorders. Figure 6 shows the networks selected as features. Taking the default mode domain as an example, the anterior cingulate cortex, precuneus, and posterior cingulate cortex regions included the most discriminative features. For the cognitive control domain, the insula, middle frontal gyrus, supplementary motor areas were prominent. 20 of 53 networks presented in more than 80% frequency, and the default mode, cognitive control and sub-cortical networks occupied a large percentage among them.

Figure 6:

The summary of the discriminating network set in each functional domain selected by our method. The network denoted by an asterisk was selected in all runs.

4. Discussion and Conclusions

In this work, we proposed a novel deep learning method to jointly utilize different neuroimaging measures for disorder classification/diagnosis. Our method enables preserving the unique properties from different measures and the combination use of features with different dimensionalities. Importantly, our method provides a way to select the optimal features in CNN via a sequential forward selection, which improved the interpretability of features that is usually difficult in deep learning.

By taking the large-scale spatial functional networks, functional connectivity between networks, and gray matter volume as the joint input, our fusion method significantly outperformed the method using single-type of features in distinguishing SZ and ASD. Our method achieved the best classification accuracy and the strongest reliability compared to other methods. Furthermore, the combination of functional networks via feature selection in our method worked better than only using single network. Our results supported that the default mode, cognitive control and sub-cortical regions had more impacts in differentiating ASD and SZ. The discriminative regions primarily included anterior cingulate cortex, caudate, inferior frontal gyrus, paracentral lobule, and hippocampus. Previous work also supported their differences in default mode network [27, 32-33], which is consistent with our results. In summary, our proposed fusion CNN model shows promise for individualized classification of mental disorders and highlights the advantage of optimally combining multimodal features compared to only single-modal features. In future work, we will compare our fusion method with other fusion methods.

CCS CONCEPTS.

Artificial intelligence · Machine learning · Life and medical sciences