Modeling Task-based fMRI Data via Deep Belief Network with Neural Architecture Search

Abstract

It has been shown that deep neural networks are powerful and flexible models that can be applied on fMRI data with superb representation ability over traditional methods. However, a challenge of neural network architecture design has also attracted attention: due to the high dimension of fMRI volume images, the manual process of network model design is very time-consuming and not optimal. To tackle this problem, we proposed an unsupervised neural architecture search (NAS) framework on a deep belief network (DBN) that models volumetric fMRI data, named NAS-DBN. The NAS-DBN framework is based on Particle Swarm Optimization (PSO) where the swarms of neural architectures can evolve and converge to a feasible optimal solution. The experiments showed that the proposed NAS-DBN framework can quickly find a robust architecture of DBN, yielding a hierarchy organization of functional brain networks (FBNs) and temporal responses. Compared with 3 manually designed DBNs, the proposed NAS-DBN has the lowest testing loss of 0.0197, suggesting an overall performance improvement of up to 47.9%. For each task, the NAS-DBN identified 260 FBNs, including task-specific FBNs and resting state networks (RSN), which have high overlap rates to general linear model (GLM) derived templates and independent component analysis (ICA) derived RSN templates. The average overlap rate of NAS-DBN to GLM on 20 task-specific FBNs is as high as 0.536, indicating a performance improvement of up to 63.9% in respect of network modeling. Besides, we showed that the NAS-DBN can simultaneously generate temporal responses that resemble the task designs very well, and it was observed that widespread overlaps between FBNs from different layers of NAS-DBN model form a hierarchical organization of FBNs. Our NAS-DBN framework contributes an effective, unsupervised NAS method for modeling volumetric task fMRI data.

1. Introduction

Understanding the organizational architecture of functional brain networks (FBN) has attracted intense interest since the inception of neuroscience (Huettel, Song et al. 2004, Logothetis 2008). After decades of research using functional magnetic resonance imaging (fMRI), scientists have proposed a variety of methods for reconstruction of FBNs from fMRI data, such as general linear model (GLM) (Beckmann, Jenkinson et al. 2003, Barch, Burgess et al. 2013), independent component analysis (ICA) (McKeown 2000, Calhoun, Adali et al. 2001, Beckmann, DeLuca et al. 2005, Calhoun, Liu et al. 2009, Calhoun and Adali 2012), and sparse dictionary learning (SDL) (Hu, Lv et al. 2015, Lv, Jiang et al. 2015, Lv, Jiang et al. 2015, Lv, Jiang et al. 2015, Ge, Makkie et al. 2016, Zhang, Li et al. 2016, Lv, Lin et al. 2017, Zhang, Lv et al. 2018). Although these methods can achieve meaningful results, they are also limited by their shallow nature and useful information might be overlooked in the data modeling process. In recent years, deep learning models have attracted much attention in the field of machine learning and data mining. Stacked by multiple similar blocks, deep learning has the intrinsic ability of extracting hierarchical features from low-level raw data. Deep learning models that usually used in the field of image classification have also been introduced in the field of fMRI data modeling. In the past several years, there have been growing literature studies that applied deep learning on modeling of fMRI data and other associated applications (Han Wang, Cho, Raiko et al. 2013, Hjelm, Calhoun et al. 2014, Plis, Hjelm et al. 2014, Suk, Wee et al. 2016, Cui, Zhao et al. 2018, Hu, Huang et al. 2018, Huang, Hu et al. 2018, Li, Huang et al. 2018, Wang, Zhao et al. 2018, Zhao, Ge et al. 2018). These research studies demonstrated that deep learning models are more powerful on deriving meaningful latent sources (FBNs) from fMRI data, compared to traditional shallow methods. For instances, Zhao et al. used a spatiotemporal deep convolutional neural network (ST-CNN) to model 4D fMRI data (Zhao, Li et al. 2018). Huang et al. used a deep convolutional auto-encoder (DCAE) to model time series fMRI data (Huang, Hu et al. 2018). Wang et al. applied recurrent neural networks (RNN) on volumetric fMRI data (Wang, Zhao et al. 2018). A recent research (Hu, Huang et al. 2018) by our group demonstrated the high performances of RBM in deriving latent sources of fMRI data. However, the RBM models are still shallow, which cannot extract hierarchical features from fMRI data. In another recent research of our lab (Dong, Ge et al. 2019), Dong et al. proposed a volumetric sparse deep belief network (VS-DBN) to model task based fMRI data, and it focused on hierarchical FBN analysis. Briefly, a DBN is stacked by multiple restricted Boltzmann machine (RBM) (Fischer and Igel 2012), and hierarchical FBNs can be derived from each hidden layer of DBN. Although the VS-DBN showed interpretable hierarchical organization of FBNs [Dong, Ge et al. 2019], it did not focus on reconstructing meaningful task-specific FBNs and temporal features, and the manual process of model architecture design is very time-consuming and not optimal.

In recent years, aiming to automatically search for optimal network architecture of deep learning models, many novel Neural Architecture Search (NAS) methods, e.g., based on reinforcement learning (RL) (Zoph and Le 2016, Baker, Gupta et al. 2017, Pham, Guan et al. 2018, Zoph, Vasudevan et al. 2018) or evolutionary algorithms (EA) (Liu, Simonyan et al. 2017, Real, Moore et al. 2017, Xie and Yuille 2017, Miikkulainen, Liang et al. 2019, Real, Aggarwal et al. 2019), have been developed and applied in a variety of deep learning tasks. In RL based methods, each choice of the architecture component is considered as an action. A sequence of actions determines the architecture of neural network with a reward defined by training loss. In EA based method, architecture searching is based on swarm intelligence. Each individual in the swarm can learn from others with better performance, and the whole swarm can converge to a feasible optimal solution after iterations. Among other more recent NAS methods (Chen, Meng et al. 2018, Liu, Chen et al. 2019, Real, Aggarwal et al. 2019), AmoebaNet (Real, Aggarwal et al. 2019), an EA based method, showed superb performances on image classification over RL based methods, suggesting EA or RL enhanced EA might be good choices in many cases.

Although many existing NAS methods can learn network architectures that outperform manual designs, most of them were designed for image classification problems, which usually have high-quality labels. However, due to the lack of labels and high complexity of volumetric fMRI data, there is still few NAS applications in the field of brain imaging using fMRI. Different from applications in image classification, deep learning models of fMRI data have to be trained in an unsupervised fashion, aiming to extract latent sources of fMRI data and reconstruct spatial and temporal features. In general, the 4-dimensional fMRI data can be preprocessed into volumetric fMRI data (volumes) or temporal fMRI data (time courses). In this work, we used task fMRI data from human Connectome Project (HCP), in which each volume has 28546 voxels, while each time course has 176 to 405 time points. Recently, our group has proposed an aging evolution based NASNet (Zhang, Zhao et al. 2019) for modeling temporal fMRI data. However, since the volatility of time course usually cause more inter-subject variability than spatial volumes across different subjects, it is possibly more effective to model FBNs by using volumes as input (Schmithorst and Holland 2004). Therefore, we took volumetric fMRI data as input for modeling of FBNs and their associated temporal features.

In this work, we proposed a deep belief network with neural architecture search (NAS-DBN) framework to model volumetric task fMRI data. Firstly, we applied a multi-layer volumetric deep belief network (DBN) and designed a group-wise scheme that aggregated multiple subjects’ fMRI volume data for effective training of the DBN, with the purpose of discovering robust, meaningful functional brain networks (FBN) in task-based fMRI data. After training of the DBN model, the group-wise fMRI data was decomposed into hierarchical temporal features and spatial features. The temporal features were generated by each hidden layer of the DBN model, and the corresponding weights matrix of model can be mapped back to the 3D brain space and visualized as spatial features (FBNs). As a generative model, our NAS-DBN has the intrinsic ability of alleviating overfitting, since DBN maximizes the likelihood of estimation rather than minimizing the reconstruction error, which empowers DBN with the ability to extract more generative features than discriminate models (Hinton, Osindero et al. 2006, Bengio 2009). Besides, the training process of NAS-DBN does not depend on task designs, since DBNs are completely data-driven models. Thus, the proposed NAS-DBN can model both task-specific FBNs and resting state networks from task based fMRI data, and it simultaneously generates temporal responses that resemble the task designs. Secondly, aiming to find out the optimal network architecture of DBN in modeling fMRI volumes, we developed a NAS framework based on particle swarm optimization (PSO). The key idea is that the particle swarm in the NAS framework will temporally evolve and finally converge to a feasible optimal solution. We designed two steps of NAS that are detailed in the next section. After NAS, fine-tuning of parameters will be conducted to further improve the performance of the DBN model. To avoid unreliability of training loss caused by potential overfitting, testing loss was adopted as the fitness function of PSO, and the split ratio of testing and training set was set as 0.2.

To quantitatively evaluate the performance of the NAS-DBN framework, a series of experiments have been conducted and the results showed the effectiveness of our design. Furthermore, we used the DBN with optimal architecture to extract FBNs and temporal responses from task-based fMRI data of HCP 900 data and compared the results with GLM-derived brain networks and task designs. The results demonstrated that the NAS-DBN is a promising framework for simultaneously deriving meaningful FBNs and temporal responses from fMRI data. In addition, since our NAS-DBN is a pure data-drive model, we also found some resting state networks (RSN) from task-based fMRI data, which are quite similar to the RSN templates in (Smith, Fox et al. 2009). Finally, we derived multiple sets of FBNs from each hidden layer of the model, and we observed widespread overlaps between FBNs from different layers, forming a hierarchical organization of FBNs.

Figure 1 summarizes our PSO-based NAS framework (Fig.1(A)) and DBN structure (Fig.1(B)) for extracting FBNs from task-based fMRI data (Fig.1(C)). The particle swarm consists of 30 particles, each of which represents a subnet with different initial architectures (Fig.1(A)). The particle swarm can evolve and converge to an optimal solution, that is, a DBN model with optimal architecture. The NAS process has two steps. In the first step of NAS, we investigate two main hyper-parameters including the number of layer and the number of neurons in each layer that are equal in this step. These two parameters are used to construct a mapping between a particle position and a solution of network architecture design. After training, we use the trained model to predict testing dataset, and the testing loss of DBN is regarded as the fitness function of PSO, which will be minimized in the searching process. In the second step of NAS, based on the results of the first step, we construct a new mapping to further investigate the number of neurons in each layer that are different in this step. Particle coding of 2 steps are shown in Figure 2. After NAS, we applied this optimal architecture of DBN to model FBNs from task-based fMRI data. The k-dimensional temporal responses (k is the number of neurons in hidden layer) are generated by each hidden layer, and the weights (W1,W2, W3) of network are visualized and quantified as FBNs, which will be further compared with GLM-derived networks.

Figure 1.

Illustration of the proposed NAS-DBN framework for deriving functional brain networks and temporal responses from task-based fMRI data. (A) PSO based NAS framework. 30 particles are initialized with random position and velocity, each particle represents a sub-net of DBN with specific network architecture. Testing loss of each sub-net (fitness function of PSO algorithm) is calculated after training. Then, evaluation and updating operation is conducted according to standard PSO procedure. (B) The optimal architecture of DBN. After NAS, DBN with optimal architecture is trained on group-wise fMRI data, and hierarchical temporal responses can be generated from each hidden layer. (C) Functional brain networks (spatial map). The weight matrix of DBN model are interpreted and visualized as FBNs, which will be further compared with GLM derived templates by overlap rate.

Figure 2.

Particle coding in Step 1 and Step 2 of NAS framework

2. Materials and Preprocessing

In this paper, fMRI data from the HCP 900 subjects release was adopted as training dataset, which includes 865 subjects and all the data is available on https://db.humanconnectome.org. The stimuli were projected onto a computer screen behind the subject’s head within the imaging chamber. In all cases of the dataset, participants were scanned on the same equipment using the same protocol for each subject and the detailed acquisition parameters are shown in Table 1.

Table 1.

Imaging Protocol of HCP Q3 TFMRI Dataset

Parameter	Value	Parameter	Value
Sequence	Gradient-echo EPI	Matrix	104×90
TR	720 ms	Slice thickness	2.0 mm
TE	33.1 ms	Multiband factor	8
flip angle	52 deg	Echo spacing	0.58 ms
FOV	208×180 mm	BW	2290 Hz/Px

The fMRI preprocessing were implemented by FSL FEAT (FMRIB’s Expert Analysis Tool) and Nilearn (Abraham, Pedregosa et al. 2014), including spatial resampling to the MNI152 template, frequency filtering, detrending, normalization and masking. After masking, the 3D brain space was converted to a one-dimensional vector, then we concatenated all subjects’ time points to a group-wise matrix. After training, the weights of model can be mapped back to the brain 3D space (inverse operation of masking) and visualized as FBNs. The details of acquisition parameters and information of each task can be found in the literature (Lv, Jiang et al. 2015). All of 7 categories of behavioral tasks are used in this paper, including Emotion, Gambling, Language, Social, Relational, Motor, and Working Memory. The preprocessed dataset consists of 1,678,100 volumes in total and the details of each task are shown in TABLE II. In our experiments, for each task, we used 500 subjects for training, and 100 subjects for testing. Therefore, there are totally 970,000 volumes used as training samples, and 194,000 volumes used as testing samples.

Table2.

Size of HCP Q3 tfMRI Dataset

Task	Volumes	Duration (Min)	Samples
Emotion	176	2:16	152,240
Gambling	253	3:12	218,845
Motor	284	3:34	245,660
Language	316	3:57	273,340
Relational	232	2:56	200,680
Social	274	3:27	237,010
Working Memory	405	5:01	350,325

3. Method

3.1. PSO based NAS Framework

Particle Swarm Optimization (PSO) is a swarm intelligence based evolutionary computation algorithm that is originally proposed by Kennedy and Eberhart in 1995 (Poli, Kennedy et al. 2007). Due to its various advantages, such as less parameter requirements, simple formula, easy to implement, PSO has become a popular tool for solving various complex optimization problems. In this work, we adopted and designed a PSO based NAS framework to search for the optimal network architecture of DBN model. As mentioned in the overview, our NAS framework consists of two steps (see Figure 2). In the first step, we designed a two- dimensional encoding to map network architecture of DBN to a particle position. The dimensions of the particle represent the number of layer and the number of neurons in each layer with the range of (2, 10) and (20, 500), respectively. In this step, we assume the number of neurons in each layer is equal. In the second step, aiming to search for number of neurons for each layer independently, the dimensions of particle represent number of neurons of each layer, respectively, while the number of neurons determined in first step will give a suggestive range to each layer, and the number of layers is already determined in the first step. The number of neurons is with the range of (0.5N_neurons, 2.5N_neurons), and N_neurons denotes the number of neurons determined in step 1.

The evolutionary process of particle swarm mainly consists of two steps: evaluation and updating. First, after initialization, all particles are evaluated by a fitness function, which is defined by the testing loss of DBN. To avoid potential overfitting in NAS process, testing loss is adopted instead of training loss as an evaluation index of the model. After training, the trained model is applied to predict testing data, and the Mean Squared Error (MSE) between input and output is calculated as testing loss, also the fitness value of corresponding particle. Then the local best solution of each particle and the global best solution of whole swarm are recorded. Second, all particles’ velocities and positions are updated by the following equations:

$$v_{id}^{t+1}=w\cdot v_{id}^t+c_1\cdot r_1\left(p_{id}^t-x_{id}^t\right)+c_2\cdot r_2\left(p_{gd}^t-x_{id}^t\right)$$ (1)

$$x_{id}^{t+1}=x_{id}^t+v_{id}^{t+1}$$ (2)

Equations (1) and (2) are for velocity and position updating, respectively, where $x_{id}^t$ and $x_{id}^{t+1}$ are the current and next positions, respectively; $v_{id}^t$ and $v_{id}^{t+1}$ are the current and next velocities, respectively. The subscripts t, i, and d denote current iteration, subnet, and coding dimension, respectively; w is the inertia weight that reflects the inertia of particle motion; c1 and c2 are learning rate that affect the ratio of learning towards personal best and global best, making the searching process intelligent; r1 and r2 are two uniform random numbers selected from the interval [0,1], which give the searching process a certain randomness. The second and third parts of the right side of Equation (1) reflect that current particle’s next motion is affected by personal best position $\left(p_{id}^t\right)$ and global best position $\left(p_{gd}^t\right)$, as well as its previous motion. The initialization strategy of PSO is as follow: the particle positions are randomly generated in solution space, and the particle velocities are randomly generated in a range of [-vmax, vmax], where vmax is the upper bound of velocity. In this work, each dimension of vmax is set as 10% of solution space scale. After updating, the particle velocity will also be limited to [-vmax, vmax]. In addition, a uniform mutation strategy with variable mutation probability was introduced to increase the diversity of particle swarm. At the beginning of iteration, greater mutation probability makes the algorithm have better exploration ability, and smaller mutation probability makes the algorithm have better exploiting ability in the last stage of iteration. Therefore, the mutation probability was set to a linearly changing value from 0.1 to 0.01 with the increase of iteration number. After updating of particle position, the value of each particle dimension will be rounded to the nearest integer. Notably, to make a compromise between speed and accuracy in NAS, the DBN epoch varies from 5 to 20 along with the increasing of PSO iteration number. For iteration range of [1, 10], the DBN epoch is set as 5. In early iterations of PSO, lower number of DBN epoch helps the swarm quickly find potential optimal solution and reduce computation load at the same time. For iteration number range of [11, 20], the DBN epoch is set as 10. In the last 10 iterations, aiming to find optimal solution accurately, the DBN epoch is set as 20. After NAS, the epoch for optimal DBN is fixed to 20.

3.2. DBN Model of Volumetric fMRI Data

DBN (see Figure 3), constructed by blocks of Restricted Boltzmann Machines (RBM), is widely used for deep generative models (Hinton, Osindero et al. 2006). As a generative method, DBN has the intrinsic ability of dealing with overfitting compared to discriminate models, since it approximates a closed-form representation of the probability distribution of the input data. RBMs consist of an input layer with visible variables v_i and a hidden layer with latent variables h_i. Given the input data, RBMs can model the dependencies of a set of visible variables v_i and a set of hidden variables h_i. For each pair of a visible node v_i and a hidden node h_i, the connection models the joint probability distribution as follows:

$$P(v,h)=\frac{1}{Z}{e}^{-E(\mathit{v},\mathit{h})}$$ (3)

Figure 3.

The Structure of DBN model of group-wise fMRI data

where Z is a normalization term and E(v, h) is energy function defined as:

$$E(v,h)=-\sum _i\frac{{\left(a_i-v_i\right)}^2}{{\sigma }_i^2}-\sum _j{b}_j{h}_j-\sum _{ij}\frac{v_i}{{\sigma }_i}{W}_{ij}{h}_j$$ (4)

where W_ij represents network weights that reflect the interactions between v_i and h_i, a_i represents the visible bias, b_j represents the hidden bias, σ_i represents the standard deviations of visible nodes. To update the RBM models, an approximation to the gradient through Markov Chain Monte Carlo (MCMC) where contrastive divergence (CD) with truncated Gibbs sampling is applied to improve computational efficiency (Hu, Huang et al. 2018). In this work, our optimal architecture of DBN has 3 RBM building blocks. As similar to RBMs that can be trained by CD algorithm, our DBN model can be trained with the same method in a layer-wise manner.

The purpose of DBN is to train and learn weights/dictionary from each layer. For each atom in the dictionary, the so-called temporal response will be generated by each neuron of each hidden layer (illustrated in Figure 3), and the corresponding weight matrix can be mapped back to brain space and quantified as FBNs. Therefore, we can decompose the input fMRI data as a set of FBNs and temporal responses. Here, a group-wise volumetric scheme of DBN is proposed to model fMRI volumes. Considering the large inter-subject variability among human brains, arbitrary selection of a single individual may not effectively represent the population, thus a group-wise learning scheme is needed to reduce inter-subject variability by jointly registering the fMRI volumes to a common reference template corresponding to the group average. Since that the inter-subject variability is relatively more associated with the volatile time courses in different imaging sessions, it appears that taking volumes as input possibly works better than time series in terms of modeling the FBNs from fMRI data in this work [48]. Accordingly, a volume from the fMRI data was taken as a training sample, and a group-wise temporal concatenation was applied to all selected HCP subjects (see Figure 3). With respect to interpreting a trained DBN in the fMRI context, each row of weight vector was mapped back into the original 3D brain image space, which was the inverse operation of masking in preprocessing steps and was interpreted as an FBN. After the DBN was trained layer-wisely on a large-scale task fMRI dataset, each weight showed the extent of each voxel contributing to a latent variable. For deeper layers, the linear combination approach was used to interpret the connection. With this approach, W₃ × W₂ × W₁ was visualized for the third hidden layer as FBNs (Fig.1(C)), and W₂ × W₁ and W₁ for the second hidden layer and the first hidden layer respectively.

3.3. Implementations

The NAS-DBN for volumetric fMRI is inherently much more computationally expensive, compared to models for temporal fMRI time series. Considering 4D fMRI images and one single layer of RBM, there are around 20K trainable parameters for temporal fMRI time series DBN, but 20 million for volumetric fMRI DBN. Moreover, the population size and iteration size will add significant computational burden on the NAS process. To deal with this problem, in this paper, the TensorFlow (Abadi, Barham et al. 2016), which is a popular deep learning framework and provides great convenience of coding with GPUs, was adopted with high efficiency GPU computation to fill the gap. Based on TensorFlow, we designed and implemented a fast and flexible DBN model. Limited by computing resources, all subnets will be trained one by one and processed collectively. The main parameters of NAS-DBN implementation were set as follows: PSO iteration is 30, PSO population is 30. In NAS stage, epoch for DBN is changed from 5 to 20 along with the increasing of iteration number. After NAS, epoch for optimal DBN is set as 20. The code ran on a deep learning server with dual GeForce GTX 1080 TI of GPU cards.

4. Experimental Results

4.1. NAS performance of NAS-DBN

To quantitatively evaluate the effectiveness of our NAS-DBN framework, we ran 10 times of the NAS searching process independently and analyzed the statistical results. We used 500 subjects of HCP task dataset as the training set of NAS, and 100 subjects for the testing. After NAS, we used the same optimal architecture of DBN to model each task fMRI data independently.

Figure 4 shows statistical results in the first step of NAS. The optimal results in the first step show high consistence and robustness in the optimal number of layer and the optimal number of neurons, and the optimal architecture of model (highlight in red color) has 3 layers and 82 neurons in each layer. In most runs (8 out of 10), the result of the optimal number of layers is 3, except that only two results are 2 and 4 respectively. The best result in the optimal number of neurons is 82, and all results are in a range from 72 to 109. These statistical experiments demonstrated that our NAS framework can generate reliable and robust results of architecture design. Furthermore, as shown in Figure 5, we compared the testing loss of DBNs with optimal architecture and manually selected architectures with different layers or neurons. DBN (3,82) denotes that there are 3 hidden layers and 82 neurons in this DBN structure. It can be clearly seen that DBN with the optimal architecture from the first step of NAS has the lowest testing loss of 0.0207 compared to other manually designed DBNs, demonstrating the effectiveness of the NAS framework.

Figure. 4.

Statistical results of 10 independent runs in the first step of NAS

Figure. 5.

Testing loss of DBN with optimal architecture and DBNs with different layers or neurons.

In the second step, the optimal results of number of neurons for layer 1, layer 2 and layer 3 are 96, 84, and 80, respectively. The testing loss for this optimal architecture is 0.0193, which is very close to the result of first step, suggesting that the difference in number of neurons between different layers might have little effect on the results in a certain range. To evaluate the computational cost of our NAS framework, we compared the running time of NAS search method with the exhaustive search scheme. The running time of the optimal DBN is about 8 minutes, which is assumed as the average running time of models. As shown in Table 3, the running time of exhaustive search (product of average running time and number of all combinations) is really a very long time. By contrast, the running time of our NAS search method decreased significantly. To further validate the superiority of the proposed NAS-DBN over manually designed DBN, we compared testing loss of NAS-DBN with three manually designed DBN models, as shown in Table 4. For fair comparison, we set the same parameters in the training process for the four DBN models. The results show that the NAS-DBN has the lowest testing loss of 0.0193, which is 47.9%, 40.4%, and 33.2% less than the three manually designed DBNs, respectively. To further compare the performances of NAS-DBN and manually designed DBNs in modeling FBNs, the FBNs derived by these four models for 7 tasks are given in supplementary materials section B, and the results showed that the NAS-DBN derived FBNs have higher similarity to GLM templates than manually designed DBN derived FBNs. As shown in Table 4, the average overlap rates (defined by equation 5) of NAS-DBN to GLM on 20 task-specific FBNs is 0.536, which is 63.9%, 60%, and 56.7% more than three manually designed DBNs, respectively. From Table 4, we can see that the performances of NAS-DBN are better than manually designed DBNs on both testing loss and FBN modeling. Although it is still theoretically hard to find globally optimal solution, our NAS framework can find feasible suboptimal solution that outperform the manually designed models within acceptable time.

Table 3.

Running Time of Two Search Methods

Method	Time of Step 1	Time of Step 2	Total Time
NAS Search	4 h 57 m	10 h 29 m	15 h 26 m
Exhaustive Search	577 h	588126 h	588703 h

Table 4.

Comparison of NAS-DBN with 3 manually designed DBNs

	NAS-DBN	DBN-1	DBN-2	DBN-3
Structure	(96,84,80)	(150,80)	(200,200,200)	(512,256,128,64)
Testing loss	0.0193	0.0371	0.0324	0.0289
Ave of OR	0.536	0.327	0.335	0.342

4.2. Comparison of NAS-DBN with GLM

To examine the representation of task-based fMRI data, 7 task-specific DBNs were trained on fMRI data of 7 HCP tasks independently using the same sets of hyper-parameters. After training, the last hidden layer of NAS-DBN model yielded 80 temporal responses, and the corresponding weights matrix was mapped back to brain 3D space and visualized as 80 FBNs. Different from the typical model-driven method GLM, the NAS-DBN does not depend on task designs and has the ability of automatically deriving both task-specific FBNs and resting state networks. To quantitatively evaluate the performance of DBN in modeling task fMRI data, a comparison between NAS-DBN results and the widely known GLM activation results is investigated in this section. For fare comparison, all the functional networks derived by these two methods are thresholded at Z > 2.3 after transformation into “Z-scores” across spatial volumes. In this work, all the functional networks have the same color scales, so we only showed the color bar in Figure 6. The spatial overlap rate is defined to measure the similarity of two FBNs in accordance with previous literature studies. Here, the two functional networks N⁽¹⁾ and N⁽²⁾ are as follows, where n is the number of voxels in one volume. The value of $N_i^{(1)}$ can be 0 or 1. If the ith voxel of FBN (1) is activated, $N_i^{(1)}$ is 1. Otherwise, $N_i^{(1)}$ is 0.

Figure 6.

Comparison between GLM templates and similar FBNs derived by NAS-DBN in 7 tasks. Each network is visualized with a series of axial slices. (a) GLM templates and 10 FBNs of NAS-DBN from emotion task (E1, E2), gambling task (G1, G2), language task (L1, L2), social task (S1, S2), and relational task (R1, R2). (b) GLM templates and the other 10 FBNs of NAS-DBN from motor task (M1-M6), and working memory (W1-W4)

$$OR\left(N^{(1)},N^{(2)}\right)=\frac{\sum _{i=1}^n\left|N_i^{(1)}\cap N_i^{(2)}\right|}{\sum _{i=1}^n\left|N_i^{(1)}\cup N_i^{(2)}\right|}$$ (5)

With the similarity measure defined above, the similarities OR(N_DBN, N_GLM) between the NAS-DBN derived functional networks N_DBN and the GLM derived functional networks N_GLM were quantitatively measured. For each of GLM derived activation networks, we found the most similar FBN derived by NAS-DBN with the highest OR. Notably, we developed in-house GLM tools and obtained our own GLM templates by using group-wise fMRI data, which are quite similar to the widely known GLM templates (Barch, Burgess et al. 2013). Figure 6 shows the comparison between part of FBNs derived by NAS-DBN and GLM activation networks in 7 HCP tasks. We selected 20 task designs and the corresponding GLM activation networks are shown on the right column of Figure 6 (a) and (b), while the similar FBNs of NAS-DBN are shown on the left column of Figures 6 (a) and (b). Details of task designs are given in supplementary materials section A.

It can be seen clearly from Figure 6 that the FBNs of NAS-DBN are quite similar to the corresponding GLM derived activation networks, and each pair of networks have a high overlap rate. The ORs of 20 pairs of networks are listed in Table 5. For emotion task, all of 260 FBNs from layer 1, layer2, and layer 3 of NAS-DBN are given in supplementary materials section C. In addition, the corresponding temporal responses are also quite similar to the task designs. There are 80 neurons in the last hidden layer of NAS-DBN, and each neuron generated one temporal response. To validate the NAS-DBN’s ability of deriving meaningful temporal patterns, we selected 100 testing subjects for each task to observe the corresponding temporal responses learned from the NAS-DBN. We used the Pearson correlation coefficient (PCC) to measure the temporal similarity between learned temporal response and the hemodynamic response function (HRF) response of task design (convolution of task design and HRF Function). First, we averaged the 100 temporal responses and calculated the PCC between averaged signal and HRF response (PCC of Ave). As shown in Figure 7, it is easy to see that the averaged temporal responses learned from NAS-DBN have high correlations with the corresponding HRF responses. Second, we calculated PCC between each temporal response and HRF response, then obtained mean (Ave of PCC), standard deviation (Std), minimum (Min), and maximum (Max) of 100 PCCs. As shown in Table 6 and Table 7, the statistical results of NAS-DBN’s temporal responses are quite satisfactory, taking into consideration of inter-subject variability. In most task designs, the Ave of PCCs are higher than 0.5 except M4, and W1-W4, and the average value is as high as 0.537. We found that the Std values in W1-W4 are a little higher compared to other task designs. A possible reason is that the proportion of impulse during the whole stimulus series is quite low for W1-W4. Therefore, the influence of noise in the signal is relatively large. In summary, these experimental results demonstrated that the NAS-DBN model can automatically derive meaningful and interpretable spatial and temporal features, suggesting the effectiveness of our NAS-DBN framework.

Table 5.

Overlap Rates Between FBNs of NAS-DBN and GLM Activation Networks

E1	E2	G1	G2	L1	L2	S1	S2	R1	R2
0.512	0.534	0.479	0.503	0.622	0.441	0.687	0.580	0.485	0.477
M1	M2	M3	M4	M5	M6	W1	W2	W3	W4
0.513	0.529	0.614	0.612	0.548	0.563	0.630	0.587	0.590	0.494

Figure 7.

Comparisons between HRF responses and averaged NAS-DBN temporal responses (100 subjects) that correspond to the FBNs of NAS-DBN in Figure 6.

Table 6.

Statistical results of NAS-DBN temporal responses (Part I)

	E1	E2	G1	G2	L1	L2	S1	S2	R1	R2
PCC of Ave	0.821	0.834	0.826	0.817	0.845	0.867	0.812	0.754	0.781	0.753
Ave of PCC	0.652	0.539	0.547	0.624	0.564	0.541	0.635	0.614	0.533	0.520
Std	0.138	0.117	0.146	0.137	0.122	0.143	0.109	0.127	0.130	0.141
Min	0.318	0.322	0.287	0.385	0.425	0.347	0.478	0.359	0.314	0.297
Max	0.887	0.869	0.912	0.876	0.879	0.872	0.924	0.873	0.897	0.864

Table 7.

Statistical results of NAS-DBN temporal responses (Part II)

	M1	M2	M3	M4	M5	M6	W1	W2	W3	W4
PCC of Ave	0.877	0.758	0.769	0.737	0.821	0.827	0.749	0.763	0.755	0.735
Ave of PCC	0.624	0.566	0.575	0.487	0.514	0.523	0.425	0.432	0.413	0.414
Std	0.126	0.110	0.107	0.130	0.143	0.125	0.189	0.210	0.192	0.203
Min	0.382	0.347	0.296	0.284	0.312	0.348	0.215	0.179	0.184	0.177
Max	0.903	0.864	0.878	0.869	0.890	0.870	0.835	0.847	0.829	0.846

4.3. Resting State Networks Derived by NAS-DBN

As a pure data-driven model, the NAS-DBN can automatically derive not only task-specific networks but also resting state networks (RSN) from task based fMRI data. We selected 3 tasks including emotion, gambling, and motor to investigate similar FBNs compared to RSN templates that derived by ICA. We downloaded the visualization data of RSN templates from Nilearn website (Abraham, Pedregosa et al. 2014). More details of RSN templates can be found in (Smith, Fox et al. 2009).

As shown in Figure 8–10, we found 10 RSNs for the selected 3 tasks respectively, including “visual network” (RSN1–3), “default mode network” (RSN4), “cerebellum network” (RSN5), “sensorimotor network” (RSN6), “auditory network” (RSN7), “executive control network” (RSN8), “frontoparietal network” (RSN9,10). It is easy to see the remarkable performances of NAS-DBN in deriving similar RSNs compared with the widely known RSN templates, demonstrating that the NAS-DBN can simultaneously derive meaningful task-specific FBNs and RSNs. Interestingly, although most of these RSNs are found in layer 3, some RSNs are found in lower layers. For emotion task, RSN9 is found in layer 1, and RSN6 is found in layer 2. For gambling task, RSN3 and RSN8 are found in layer 2. For motor task, RSN1 is found in layer 1, RSN3 and RSN8 are found in layer 2. Except for the RSNs we found, there still exist some less similar RSNs in different layers, as well as some less similar task-specific FBNs and even some undefined FBNs. This result indicates that our NAS-DBN model can extract more useful information than traditional shallow models. Besides, we found that the RSNs learned from different tasks are slightly different from each other, suggesting that various task stimuli might affect the RSN patterns. Therefore, deeper exploration of RSNs under different conditions could be a promising way to identify brain states, which needs to be further studied in the future. The RSNs derived from other 4 tasks and ORs between NAS-DBN derived RSNs and RSN templates are given in supplementary materials section D.

Figure 8.

Comparison of NAS-DBN learned RSNs in emotion task and RSN templates. Each network is visualized with 3 most informative orthogonal slices.

Figure 10.

Comparison of NAS-DBN learned RSNs in motor task and RSN templates. Each network is visualized with 3 most informative orthogonal slices.

4.4. Hierarchical FBNs Derived by NAS-DBN

As a deep generative model, our NAS-DBN framework can extract hierarchical functional brain networks and temporal responses from each hidden layer. Here, we selected emotion task to investigate the NAS-DBN’s ability on deriving hierarchical FBNs, and analyses of other 6 tasks are given in supplementary materials section E. The weighs W₁ was visualized for the first hidden layer as FBNs, W₂ × W₁ for the second layer and W₃ × W₂ × W₁ for the third layer, respectively. To quantitatively measure the associations between different layers, the inheritance similarity rate (ISR) between a lower layer FBN N^(L)and a higher layer FBN N^(H) is defined as follows:

$$ISR\left(N^{(L)},N^{(H)}\right)=\frac{{\sum }_{i=1}^n\left|N_i^{(L)}\cap N_i^{(H)}\right|}{{\sum }_{i=1}^n(N_i^{(H)})}$$ (6)

Figure 11 shows the cluster heat maps of ISR matrix between layer 1 and layer 2 and the ISR matrix between layer 2 and layer 3 from emotion task. We can clearly see widespread ISRs between different layers, indicating that some lower layer FBNs are well contained in higher layer FBNs. That means the higher layer FBNs tend to be more complete and meaningful compared to lower layer FBNs. This phenomenon indicated the existence of hypothesized hierarchical organization of FBNs. To make it clearer, we found some similar task-specific FBNs from all of 3 hidden layers, as shown in figure 12. Interestingly, we can see clearly that the task-specific FBNs tend to be more similar to the GLM activation networks with the layer depth going higher.

Figure 11.

The hierarchy property measured with ISR across 3 hidden layers of NAS-DBN from emotion task. The left is the ISR between layer 1 and layer 2, and the right is the ISR between layer 2 and layer 3.

Figure 12.

Similar task-specific FBNs in different layers from emotion task. Each network is visualized with 7 axial slices.

In addition, RSNs from higher layer are also more meaningful than RSNs from lower layer, since most of 10 RSNs are found in layer 3. Therefore, we can conclude that FBNs from NAS-DBN layer 3 have higher similarities to GLM activated networks or RSN templates than FBNs from layer 2 and layer 1. Holistic brain atlases extracted from FBNs are given in supplementary materials, which can help understand the hierarchical organization of FBNs. We also showed all of 260 FBNs including 96 FBNs of layer 1, 84 FBNs of layer 2, and 80 FBNs of layer 3 from emotion task in the supplementary materials, which provides more direct evidence for the understanding of FBN hierarchy. To sum up, the proposed NAS-DBN model is intrinsically capable in deriving meaningful hierarchical FBNs of fMRI data.

5. Discussion and conclusion

We proposed a PSO based NAS-DBN framework for searching optimal architecture of DBN in modeling FBNs from volumetric fMRI data. 7 HCP Task fMRI datasets were used to evaluate and validate our NAS-DBN. Based on evolutionary computation, 30 subnets in our framework learn experience of each other, and the whole swarm can evolve and finally converge to a feasible optimal architecture of DBN. The statistical experiment of NAS showed high consistence and robustness of our architecture design. As mentioned before, most existing NAS methods are not suitable for fMRI studies, since they were designed mostly for classification problems of natural images. Due to the high dimension of volumetric fMRI data and unsupervised training of model, it is necessary to design and implement new NAS methods for modeling fMRI data. PSO is one of the most popular evolutionary optimization methods. Compared with other similar methods such as Genetic Algorithm (GA), Simulated Annealing (SA), PSO has more advantages on global optimization problems, especially fast convergence. That is why we selected PSO as our optimization method for model architecture design. NAS usually need plenty of computational resource. However, our experiments were implemented on two GPU cards within acceptable time (15 hours). The number of particles in the swarm is set as 30, and the iteration number is set as 30. The running time of a DBN is about 5 to 30 minutes, which affected by DBN structures and other training parameters. In order to further reduce calculation time, we changed the DBN epoch from 5 to 20 along with the increasing of PSO iteration. Thus, the proposed NAS-DBN can converge within acceptable time on single computer that is very meaningful under resource limited conditions. Although it is still hard to find the theoretically optimal solution, our NAS framework can quickly find feasible suboptimal solution that outperform manually designed models and continue to fine-tune the model during the training process, which is meaningful to complex model architecture designs. It is demonstrated that the proposed NAS-DBN model can automatically derive meaningful hierarchical functional brain networks including task-specific FBNs and resting state networks, and the corresponding temporal responses, which provides more useful information compared to traditional shallow models. Comparisons between GLM activation networks and FBNs by NAS-DBN validated that the functional brain networks learned by NAS-DBN are meaningful and can be well interpreted. Comparisons between HRF responses and temporal responses of NAS-DBN indicated the NAS-DBN can simultaneously generate meaningful temporal responses that resemble the task designs. The differences between RSNs learned from different tasks may help identify brain states, which need to be further studied in the future. Besides, we observed hierarchical organization of FBNs from three hidden layers of the NAS-DBN. It is shown that the FBNs tend to be more meaningful with the layer depth going higher, compared to GLM activation networks. Although there are many undefined FBNs derived by NAS-DBN, our model provided rich information from fMRI data that is important for future research. As a complete data-driven model, it is expected that the NAS-DBN can also work on resting state fMRI data, and provide help for clinical diagnosis by identifying the difference of FBNs derived from patients and healthy subjects.

In general, this work contributes an effective unsupervised NAS method on DBN for modeling of volumetric task fMRI data. The proposed NAS-DBN can find feasible optimal solution for DBN structure within acceptable time under limited computing resource, yielding a hierarchy organization of meaningful FBNs and temporal responses. In the future, the hierarchy organization of FBNs and FBN relevance across tasks will be further studied. Also, we will continue to optimize our model and investigate the possible clinical applications on brain disease identification.

Figure 9.

Comparison of NAS-DBN learned RSNs in gambling task and RSN templates. Each network is visualized with 3 most informative orthogonal slices

Highlights.

• A novel unsupervised PSO based deep belief network model with neural architecture search (NAS-DBN) in modeling functional brain networks (FBNs)

• The NAS-DBN acts as a hierarchical feature extractor that decompose the preprocessed fMRI data into spatial features (FBNs) and temporal features.

• The data-driven deep learning model provides much more latent information of fMRI than traditional models.

• We derived 260 of hierarchical FBNs and temporal features from each hidden layer of DBN, including meaningful task specific FBNs and 10 resting state networks (RSNs).