Multi-paradigm fMRI fusion via sparse tensor decomposition in brain functional connectivity study

Abstract

Functional magnetic resonance imaging (fMRI) is a powerful technique with the potential to estimate individual variations in behavioral and cognitive traits. Joint learning of multiple datasets can utilize their complementary information so as to improve learning performance, but it also gives rise to the challenge for data fusion to effectively integrate brain patterns elicited by multiple fMRI data. However, most of the current data fusion methods analyze each single dataset separately and further infer the relationship among them, which fail to utilize the multidimensional structure inherent across modalities and may ignore complex but important interactions. To address this issue, we propose a novel sparse tensor decomposition method to integrate multiple task-stimulus (paradigm) fMRI data. Seeing each paradigm fMRI as one modality, our proposed method considers the relationships across subjects and modalities simultaneously. In specific, a third-order tensor is first modeled by using the functional network connectivity (FNC) of subjects in multiple fMRI paradigms. A novel sparse tensor decomposition with the regularization terms is designed to factorize the tensor into a series of rank-one components, which can extract the shared components across modalities as the embedded features. The L_2,1-norm regularizer (i.e., group sparsity) is enforced to select a few common features among multiple subjects. Validation of the proposed method is performed on realistic three paradigm fMRI datasets from the Philadelphia Neurodevelopmental Cohort (PNC) study, for the study of the relationship between the FNC and human cognitive abilities. Experimental results show our method outperforms several other competing methods in the prediction of individuals with different cognitive behaviors via the wide range achievement test (WRAT). Furthermore, our method discovers the FNC related to the cognitive behaviors, such as the connectivity associated with the default mode network (DMN) for three paradigms, and the connectivity between DMN and visual (VIS) domains within the emotion task.

I. Introduction

Functional connectome analysis is crucial for uncovering individual-specific brain networks [1], [2]. Over the past few decades, a variety of functional connectome-based research studies have focused on identifying variations in behavioral and cognitive traits such as the intelligence and disease symptoms for individuals of different ages and genders [3], [4]. Functional connectivity has been proven to be a useful tool to explore function of human brain, which represents the functional nodes defined by regions-of-interest (ROIs) and edges defined by correlation between the time courses of ROIs.

Functional magnetic resonance imaging (fMRI) has been widely used to measure brain activity due to its noninvasiveness, and high spatial and temporal resolution [5]. Resting state (RS) fMRI is one of the tasks to investigate individual differences in the organizational patterns of functional networks, and measure intrinsic or spontaneous activity in the brain such as the default mode network (DMN) [6]. To improve the quantification of functional connectivity, a variety of analytical methods such as graph theory have been applied to fMRI data analysis [7], [8]. However, the resting state is an unconstrained state that may fail to capture the full range of individual differences in terms of functional connectivity [9]. Task-based fMRI is collected while participants are functionally involved with a specific task, which explores the individual differences due to brain activity changes stimulated by a paradigm [10], [11]. Finn et al. [2] estimated functional connectivity profiles using both task and resting state fMRI as a fingerprint that can accurately identify subjects from a large group. Greene et al. [12] predicted fluid intelligence by using a model built from task fMRI data (working memory (WM) or emotion (EM)), which outperformed a model built from resting state fMRI. These results also suggested that certain tasks may bring about meaningful finding across subjects with different traits, motivating a paradigm shift from resting to task-based functional connectivity analysis.

In recent study, Sripada et al. [13] found that resting-state connectivity can be leveraged to produce generalizable markers of neurocognitive functioning, and highlighted the importance of task control default mode network interconnections as a major locus of individual differences in cognitive functioning. In particular, when complementary information is obtained from multiple datasets generated by different task-trait experiments, data fusion has lead to novel findings in many recent neuroimaging studies. It has also been suggested in [3], [14] that there is a commonality between different modalities (different functional tasks or different imaging modalities) implicated by the same underlying pathology. Approaches such as machine learning and statistical analysis are widely applied in this research [15]–[18]. In the study of functional brain networks, Xiao et al. [19] proposed a framework with alternating diffusion map by combining two paradigms fMRI datasets to enhance the prediction of intelligence quotient (IQ). Kaufmann et al. [3] also reported commonality across different paradigms assessed via a set of task-reference comparison experiments. However, the underlying algorithms of these works were primarily based on the matrix decomposition framework, which reduced the dimension of each dataset separately and utilized similarity scores to identify the linked features between different datasets. As a result, they may not reflect complex interactions and specific structures of these multidimensional data [20].

To overcome this limitation, tensors, also called multidimensional arrays, are well recognized as a powerful tool to represent complex high dimensional data and have been applied to the fusion of structural and diffusion MRI data [21], a three-way array analysis for brain imaging [22], dynamic network states detection [23], etc. Specifically, Beckmann and Smith [20] presented tensor ICA, which applied independent component analysis (ICA) to tensor decomposition for multi-subject fMRI analysis. Tensor ICA decomposed a third-order tensor with dimensions of voxels × time × subjects into a set of independent spatial maps associated with time courses and subject modes. Based on tensor ICA, Hore et al. [24] presented a Bayesian method for the analysis of gene expression data from multiple tissue types, which decomposed the tensor into a series of latent components to uncover gene network linked to genetic variations. Similarly, Ma et al. [25] modeled multi-view graph data as tensors to learn graph embedding and captured the local structure. These studies demonstrate tensor decomposition to be a powerful approach for data representation and dimension reduction.

Despite recent success of employing tensor-based methods for fMRI signal analysis, most of these models ignore the subject-subject relation between modalities, which could otherwise improve the performance. To bridge the gap mentioned above, we propose a novel multi-paradigm sparse tensor decomposition method (MSTD) to integrate multiple fMRI datasets (here the modalities refer to fMRI data collected under multiple paradigms). We are motivated by the work in [26], [27], which leverage the information from multiple task-stimulus to uncover the individual differences in functional brain networks associated with cognitive abilities. The relationships of ROIs from both multiple subjects and multiple task-based fMRI datasets are aggregated to model a third-order tensor. Then the CANDECOMP/PARAFAC decomposition (CPD) based method [28] is utilized to factorize the tensor into a series of rank-one components. Note that to find the shared significant sparsity patterns across subjects, L_2,1-norm and L₁-norm regularizations are enforced in tensor decomposition. Each component in the decomposition represents a functional sub-network as a new embedding feature. Moreover, the features are extracted via hierarchical clustering for the further individual cognition identification. To validate the efficiency and effectiveness of the proposed model, i.e., MSTD, we applied it to classify and predict the WRAT score [29] using the Philadelphia Neurodevelopmental Cohort (PNC) dataset [30]. Results show that MSTD outperforms other existing methods, such as single modal LASSO model (SM) [31], multi-modal multi-task learning (MTL) [32], a manifold regularized multi-task learning (NM2TL) we proposed recently [33] and CPD.

The primary contributions of this paper can be summarized in three aspects. First, the proposed method integrates multiple brain functional information from three fMRI paradigms. That is, MSTD is applied to explore three functional brain networks simultaneously. Second, regularization terms are incorporated into tensor decomposition to obtain low-dimensional embedding of the brain network. Third, our method provides a practical framework to not only extract shared components via tensor decomposition but also associate the FNC with cognitive abilities.

The rest of this paper is organized as follows. Section II introduces the tensor decomposition method and our proposed model. Experimental results on the PNC dataset are presented in Section III, followed by conclusions in Section IV.

II. Method

A. Notation and preliminaries

In this paper, lowercase, boldface lowercase, boldface uppercase and calligraphic letters are used to denote scalars, vectors, matrices and tensors, respectively. For a matrix A, its transpose, Moore-Penrose pseudoinverse, (i, j)-th entry and (r)-th column are denoted as ${\mathbf{A}}^{\top }$, A^†, a_ij and a_r. Let $ℝ$ be the set of real numbers. For convenience, we present a list of notations and definitions used in this paper in TABLE I.

TABLE I:

Notations and definitions.

Notation/Abbr	Definition/Full name
ROI	Regions of interest.
FNC	Functional network connectivity.
RS	Resting state.
WM	Working memory task.
EM	Emotion task.
CPD	the CANDECOMP/PARAFAC decomposition.
WRAT	The wide range achievement test.
T(κ)	The mode-κ unfolding matrix of a tensor $\mathcal{T}$.
R	The rank of a tensor (also called the number of components).
δ	The number of ROIs.
I	The number of pairwise ROI correlations, and I = δ(δ − l)/2.
J	The number of subjects.
K	The number of fMRI paradigms.
F	The FNC representation matrix, and $\mathbf{F}\in {ℝ}^{\delta \times \delta }$.
○	The outer product, ${\mathbf{a}}_r\circ {\mathbf{b}}_r={\mathbf{a}}_r{\mathbf{b}}_r^{\top }$.
⊗	The Kronecker product, $\mathbf{A}\otimes \mathbf{B}=\left[\begin{array}{ccc}{a}_{11}\mathbf{B}& \cdots & a_{1J}\mathbf{B}\\ ⋮& \ddots & ⋮\\ a_{I1}\mathbf{B}& \cdots & a_{IJ}\mathbf{B}\end{array}\right].$.
⊙	The Khatri-Rao product, A ⊙ B = [a1 ⊗ b1 a2 ⊗ b2 ⋯ ar ⊗ br].
*	The Hadamard product, $\mathbf{A}\ast \mathbf{B}=\left[\begin{array}{ccc}{a}_{11}{b}_{11}& \cdots & a_{1J}{b}_{1J}\\ ⋮& \ddots & ⋮\\ a_{I1}{b}_{I1}& \cdots & a_{IJ}{b}_{IJ}\end{array}\right]$.
$\Vert \mathcal{T}{\Vert }_F$	The Frobenius norm of $\mathcal{T}$, $\Vert \mathcal{T}{\Vert }_F=\sqrt{{\sum }_{i=1}^I{\sum }_{j=1}^J{\sum }_{k=1}^K{\left(t_{ijk}\right)}^2}$.
||A||2,1	The L2,1-norm of A, $\Vert \mathbf{A}{\Vert }_{2,1}={\sum }_{i=1}^I\sqrt{{\sum }_{j=1}^J{\left(a_{ij}\right)}^2}$.

CPD is the most well-known tensor factorization method, which can represent a third-order tensor $\mathcal{T}\in {ℝ}^{I\times J\times K}$ as a linear combination of rank-one terms:

$$\mathcal{T}=\sum _{r=1}^R{\mathbf{a}}_r\circ {\mathbf{b}}_r\circ {\mathbf{c}}_r+\mathcal{E}$$ (1)

where R is the rank of a tensor and $\mathcal{E}\in {ℝ}^{I\times J\times K}$ is a three-way array denoting noise. In terms of the standard matrix representations of $\mathcal{T}$, the components are usually estimated by the minimization of the following equivalent objective function:

$$\begin{gathered} min\frac{1}{2}{\Vert {\mathbf{T}}^{(1)}-(\mathbf{B}\odot \mathbf{C})\cdot {\mathbf{A}}^{\top }\Vert }_F^2 \\ =min\frac{1}{2}{\Vert {\mathbf{T}}^{(2)}-(\mathbf{C}\odot \mathbf{A})\cdot {\mathbf{B}}^{\top }\Vert }_F^2 \\ =min\frac{1}{2}{\Vert {\mathbf{T}}^{(3)}-(\mathbf{A}\odot \mathbf{B})\cdot {\mathbf{C}}^{\top }\Vert }_F^2 \end{gathered}$$ (2)

where A, B and C are I × R, J × R and K × R matrices, respectively. For CPD, the Alternating Least Squares (ALS) algorithm [34], [35] is the most widely used due to its high speed and ease of implementation [36]. The ALS approach solves one matrix with the other two fixed alternatively in each iteration and repeats this procedure until the convergence criterion is satisfied. The update equations are given by Eq.(3).

$$\begin{gathered} \mathbf{A}\leftarrow {\mathbf{T}}^{(1)}{[{(\mathbf{C}\odot \mathbf{B})}^{\top }]}^{†} \\ \mathbf{B}\leftarrow {\mathbf{T}}^{(2)}{[{(\mathbf{C}\odot \mathbf{A})}^{\top }]}^{†} \\ \mathbf{C}\leftarrow {\mathbf{T}}^{(3)}{[{(\mathbf{B}\odot \mathbf{A})}^{\top }]}^{†} \end{gathered}$$ (3)

B. The proposed multi-paradigm sparse tensor decomposition

The Blood-Oxygen-Level-Dependent (BOLD) signal of fMRI measures the brain activity dynamically by detecting changes associated with blood flow at different spatial locations in the brain. In this way, the brain networks can be built based on the BOLD signals to represent functional connectivity across brain regions. For the FNC of individuals, the nodes can be defined by brain ROIs, and the weight of edge between two nodes is represented as the temporal correlation (covariance) of the fMRI time series. Then for the j-th individual, the network representation matrix F_j (j = 1, ..., J) can be obtained by calculating the Pearson’s correlation between each pair of ROIs. Since F_j is symmetric, it can also be converted into a vector via vectorization of upper triangular matrix or lower triangular matrix of F_j. So the FNC of all subjects is denoted as $\mathbf{G}\in {ℝ}^{J\times \frac{\delta (\delta -1)}{2}}$.

To take advantage of complementary information from multi-paradigm fMRI data, the data is tensorized as $\mathcal{T}\in {ℝ}^{I\times J\times K}$ by stacking the FNC of all paradigms. In our approach, we aim to learn a series of meaningful components for the representation of $\mathcal{T}$. For the multi-paradigm fMRI tensor $\mathcal{T}$, the objective function is formulated as the following optimization problem based on CPD.

$$min\frac{1}{2}\Vert \mathcal{T}-[[\mathbf{A},\mathbf{B},\mathbf{C}]]{\Vert }_F^2=min\frac{1}{2}{\Vert \mathcal{T}-\sum _{r=1}^R{\mathbf{a}}_r\circ {\mathbf{b}}_r\circ {\mathbf{c}}_r\Vert }_F^2$$ (4)

Here, each column of $\mathbf{A}\in {ℝ}^{I\times R}$ contains the ROIs’ correlation scores, which interpret each component as the functional sub-network. $\mathbf{B}\in {ℝ}^{J\times R}$is a matrix with b_jr(j = 1, 2, ..., J; r = 1, 2, ..., R) denoting the score of the r-th component for the j-th subject, and C is a K × R matrix with c_kr(k = 1, 2, ..., K; r = 1, 2, ..., R) denoting the weight of the r-th component for the k-th paradigm.

CPD offers a dimension reduction approach to map the original network data into a number of sub-network representations. Note that each component obtained by Eq.(4) consists of three vectors of scores indicating the relative contribution of each individual, FNC and paradigm. By enforcing sparsity constraints on these latent components, the most significant basis components of FNC will be selected and linearly combined to represent the original whole-brain functional network. More concretely, given an input tensor $\mathcal{T}\in {ℝ}^{I\times J\times K}$, we can learn R sparse functional sub-networks denoted as A, the new low-dimensional representation for each subject as B and the component weights in paradigm as C. To this end, the objective function is proposed based on CPD for multi-paradigm sparse tensor decomposition model as

$$\begin{array}{l}\underset{\mathbf{A},\mathbf{B},\mathbf{C}}{\min }\frac{1}{2}\Vert \mathcal{T}-[[\mathbf{A},\mathbf{B},\mathbf{C}]]{\Vert }_F^2+{\lambda }_1\Vert \mathbf{A}{\Vert }_1+{\lambda }_2{\Vert {\mathbf{B}}^{\top }\Vert }_{2,1}\hfill \\ s.t.\sum _{i=1}^I{a}_{ir}^2=1,\sum _{k=1}^K{c}_{kr}^2=1,\forall r\in \{1,2,\dots ,R\}.\hfill \end{array}$$ (5)

where λ₁ and λ₂ are two positive regularization parameters. In Eq.(5), the L₁-norm regularization is imposed on A, which can extract significant ROIs’ correlations by nonzero values and make each component to have a sparse sub-network representation. Considering both column sparsity and sparsity along the column direction to further identify the relevant significant components, the L_2,1-norm regularization is imposed on ${\mathbf{B}}^{\top }$ to extract the shared and significant components in all the subjects. The L_2,1-norm penalizes each column of B as a whole and enforces sparsity among the columns by removing unimportant components. λ₁ and λ₂ are tuning parameters, which not only control the feature extraction when reconstructing FNC but also determine the sparsity and scale of components. Since there is no golden criterion for selecting parameters, in our experiments, their values are experimentally determined to ensure that the reconstructed FNC exhibit a significant level of sparsity in terms of spatial distributions.

Besides, the constraints are also imposed on A and C, which normalize each column in each iteration to prevent these matrices from having arbitrarily large values, leading to a small value of B. As a result, a new low-dimensional representation B is obtained for each subject through multi-paradigm fMRI data. Fig.1(a) shows the procedure of multi-paradigm sparse tensor decomposition.

Fig. 1:

The main framework of the proposed method. (a). the procedure of multi-paradigm sparse tensor decomposition. (b). the procedure of clustering and extracting features for classification and prediction.

C. Optimization algorithm

To solve Eq.(5), an efficient algorithm is proposed by iteratively updating the matrices A, B and C. According to Eq.(2), the objective function can be rewritten as:

$$\mathcal{L}:=\frac{1}{2}{\Vert {\mathbf{T}}^{(1)}-(\mathbf{B}\odot \mathbf{C})\cdot {\mathbf{A}}^{\top }\Vert }_F^2+{\lambda }_1\Vert \mathbf{A}{\Vert }_1+{\lambda }_2{\Vert {\mathbf{B}}^{\top }\Vert }_{2,1}$$ (6)

or

$$\mathcal{L}:=\frac{1}{2}{\Vert {\mathbf{T}}^{(2)}-(\mathbf{C}\odot \mathbf{A})\cdot {\mathbf{B}}^{\top }\Vert }_F^2+{\lambda }_1\Vert \mathbf{A}{\Vert }_1+{\lambda }_2{\Vert {\mathbf{B}}^{\top }\Vert }_{2,1}$$ (7)

By taking the derivative of Eq.(6) with respect to A as zero as well as taking the derivative of Eq.(7) with respect to B as zero, we obtain

$$\frac{\partial \mathcal{L}}{\partial \mathbf{A}}=(\mathbf{A}{(\mathbf{C}\odot \mathbf{B})}^{\top }-{\mathbf{T}}^{(1)})(\mathbf{C}\odot \mathbf{B})+{\lambda }_{1}sgn(\mathbf{A})$$ (8)

$$\frac{\partial \mathcal{L}}{\partial \mathbf{B}}=\left((\mathbf{C}\odot \mathbf{A})\mathbf{B}-{\mathbf{T}}^{(2)}\right){(\mathbf{C}\odot \mathbf{A})}^{\top }+{\lambda }_2\mathbf{B}\mathrm{\Sigma }$$ (9)

where sgn(·) is the signum function, $\mathbf{B}\mathrm{\Sigma }=\partial ({\Vert {\mathbf{B}}^{\top }\Vert }_{2,1})/\partial \mathbf{B}$, and $\mathrm{\Sigma }\in {ℝ}^{R\times R}$ is a diagonal matrix with the r-th diagonal element as 1/∥b_r∥₂.

$$\mathrm{\Sigma }=\left[\begin{array}{l}1/{\Vert {\mathbf{b}}_1\Vert }_2\\ 1/{\Vert {\mathbf{b}}_2\Vert }_2\\ \ddots \\ 1/{\Vert {\mathbf{b}}_R\Vert }_2\end{array}\right]$$

Then we have the updated procedure.

$${\mathbf{A}}^{t+1}=\frac{{\mathbf{T}}^{(1)}\left({\mathbf{C}}^t\odot {\mathbf{B}}^t\right)-{\lambda }_{1}sgn(\mathbf{A})}{{\left({\mathbf{C}}^t\odot {\mathbf{B}}^t\right)}^{\top }\left({\mathbf{C}}^t\odot {\mathbf{B}}^t\right)}$$ (10)

$${\mathbf{B}}^{t+1}=\frac{{\mathbf{T}}^{(2)}{\left({\mathbf{C}}^t\odot {\mathbf{A}}^t\right)}^{\top }}{{\left({\mathbf{C}}^t\odot {\mathbf{A}}^t\right)}^{\top }\left({\mathbf{C}}^t\odot {\mathbf{A}}^t\right)+{\lambda }_2\mathrm{\Sigma }}$$ (11)

In addition, matrix C can be updated by

$${\mathbf{C}}^{t+1}={\mathbf{T}}^{(3)}({\mathbf{B}}^t\odot {\mathbf{A}}^t){({({\mathbf{C}}^t)}^{\top }{\mathbf{C}}^t\ast {({\mathbf{B}}^t)}^{\top }{\mathbf{B}}^t)}^{†}$$ (12)

where t is the t-th iteration. In addition, to ensure that the constraints in Eq.(5) are satisfied, we further normalize each component of A and C in each iteration. The iteration will terminate when the relative error of the objective function satisfies |e^t+1 − e^t| ≤ 10⁻⁶, where $e^t={\Vert {\mathcal{T}}^t-\mathcal{T}\Vert }_F/\Vert \mathcal{T}{\Vert }_F$. The pseudocode of the proposed optimization algorithm is summarized in Algorithm 1.

Algorithm 1.

multi-paradigm sparse tensor decomposition

Input: tensor $\mathcal{T}$, the number of components R, λ1 and λ2;
Output: A, B and C;
1:	Initialization: B, C
2:	for t = 1 to Max-Iteration do
3:	update A by Eq.(10)
4:	for r = 1 to R do
5:	${\mathbf{a}}_r^t={\mathbf{a}}_r^t/\Vert {\mathbf{a}}_r^t\Vert$
6:	end for
7:	update B by Eq.(11)
8:	update C by Eq.(12)
9:	for r = 1 to R do
10:	${\mathbf{c}}_r^t={\mathbf{c}}_r^t/\Vert {\mathbf{c}}_r^t\Vert$
11:	end for
12:	until converge
13:	end for
14:	Return matrices A, B and C.

D. the classification and prediction of cognitive abilities

Although a number of rank-one components are obtained by the proposed model, the unique solution is sensitive with the component number R. According to the previous studies [37], a uniqueness condition of CPD for a third-order tensor is provided that krank(A) + krank(B) + krank(C) ≥ 2R + 2, where the Kruskal rank of matrix A is the maximum value (krank(A)) ensuring that any krank(A) columns of matrix A are linearly independent.

As each component serves as the basis for sparsely representing the whole-brain functional network, it is essential to interpret the functional significance of those hundreds of network components and establish their correspondences across a group of subjects and multiple paradigms. In this paper, we mainly focus on several significant components that are highly correlated with cognitive behaviors assessed by WRAT. To identify and select such essential components that are more easily interpretable, a clustering procedure is employed, which is inspired by Block Term Decomposition (BTD) [38]. More specifically, the absolute value of all pairwise components’ correlations in A are calculated. Hierarchical clustering is then used to cluster the similar components, using the absolute value of correlation as a dissimilarity measure no matter whether the correlation are positive or negative. The clustering is terminated when no correlations between clusters are above 0.6 according to Ref. [24]. That is, for each a_v of A, if |corr(a_v, a_p)| > 0.6 (v, p = 1, 2, ..., R), we have L_v = L_v ∪ {a_p}. Then

$${\mathbf{a}}_v=\{\begin{array}{c}\frac{{\mathbf{a}}_v+{\sum }_{{\mathbf{a}}_q\in L_v}{\mathbf{a}}_q}{\left|L_v\right|+1},corr\left({\mathbf{a}}_v,{\mathbf{a}}_p\right)>0.6\\ \frac{{\mathbf{a}}_v+{\sum }_{{\mathbf{a}}_q\in L_v}\left(-{\mathbf{a}}_q\right)}{\left|L_v\right|+1},corr\left({\mathbf{a}}_v,{\mathbf{a}}_p\right)<-0.6\end{array}$$ (13)

where | · | is the cardinality of a set. The individual scores and the paradigm scores for the corresponding a_p can be regraded as the low-rank factorization for the corresponding matrix of the cluster. In other words,

$$\mathcal{T}\approx \sum _{v=1}^V{\mathbf{a}}_v\circ \left({\mathbf{B}}_v\cdot {\mathbf{C}}_v^{\top }\right)$$ (14)

where ${\mathbf{B}}_v\in {ℝ}^{J\times L_v}$ and ${\mathbf{C}}_v\in {ℝ}^{K\times L_v}$ are two rank-L_v matrices. Eq.(14) is a BTD of the tensor $\mathcal{T}$ in V rank-(L_v, L_v, 1) terms [39]. Based on the above procedure, the number of components is reduced while the component weights in both individual and paradigm are kept. That is, the same functional sub-networks are shared among all subjects under different weights in multiple paradigms, and the low-rank combination of individual and paradigm ensures that the components obtained by tensor decomposition can be used for further analysis. To validate our proposed method, we call the linear support vector machine (SVM) and support vector regression (SVR) functions in Matlab directly with default hyperparameters to test the subsequent classification and prediction power, respectively. SVM is well known as a state-of-the-art classifier and has been extensively used in fMRI data analysis [40] due to its high accuracy. Since the goal of this paper is to test the performance of the proposed framework for extracting embedded features of FNC, the low-dimensional representation B and the corresponding WRAT scores are used as the inputs to the SVM and the SVR models. The procedure of clustering and extracting features for WRAT classification and prediction is shown in Fig.1(b).

Computational analysis.

In Algorithm 1, the objective function $\mathcal{L}$ is solved by alternately updating matrices A, B and C. So here the time complexity is analyzed according to the update equations, i.e., Eq.(10), Eq.(11) and Eq.(12). In Eq.(10), T⁽¹⁾ is a I×JK matrix and (C^t ⊙ B^t) is a JK×R matrix, whose matrix-matrix multiplication has the time complexity of $\mathcal{O}(IJKR)(I\gg J,K,R)$. Similarly, the time complexity of Eq.(11) is $\mathcal{O}(IJKR)$. In Eq.(12), the time complexity of calculating the pseudoinverse of an R × R matrix is $\mathcal{O}(R^3)$, and the time complexity of computing matrix multiplication is also $\mathcal{O}(IJKR)$. In the stage of hierarchical clustering, the time complexity of computing correlation is $\mathcal{O}(I)$, and the time complexity of clustering all the components is $\mathcal{O}(IR)$. So in the whole procedure, the main computational cost is tensor decomposition. The orders of magnitude I, J, K and R are usually $\mathcal{O}({10}^4)$, $\mathcal{O}({10}^2)$, $\mathcal{O}({10}^1)$ and $\mathcal{O}({10}^2)$, respectively. MSTD iteratively optimizes matrices A, B and C to minimize the objective function and runs with the amount of computation in $\mathcal{O}({10}^9)$ at each iteration, which is acceptable for using multiple fMRI datasets.

III. Experiments and results

A. Data pre-processing

The data used in the experiments are acquired from the Philadelphia Neurodevelopmental Cohort (PNC) [30], which is a large-scale collaborative research project between the Brain Behavior Laboratory at the University of Pennsylvania and the Center for Applied Genomics at the Children’s Hospital of Philadelphia. The PNC project mainly focused on characterizing brain and behavior and their interactions with genetics by combining neuroimaging, clinical and cognitive phenotypes, and genomics techniques. Multi-paradigm neuroimaging data and multiple genetic factors for nearly 900 adolescents aged from 8 to 21 years are obtained from the PNC dataset, which are available in the dbGaP database [30].

In this paper, the relationship between individual differences in WRAT and brain activity is investigated during the engagement of cognitive abilities, measured with RS, EM and WM fMRI. The WRAT scores assessed the general cognition and learning ability (such as reading recognition, spelling and arithmetic computation) of subjects, which was a one hour computerized neurocognitive battery (CNB) administered in the PNC study. While some reservations remain, the WRAT is still an effective method to estimate intelligence quotient (IQ) values [41]. To mitigate the influence of age over the final results, we selected the 342 subjects whose ages were above 16 years [42]. The distribution of WRAT scores of 342 subjects is shown in Fig.2.

Fig. 2:

The WRAT score distribution among the 342 subjects.

All MRI scans were conducted on a single 3T Siemens TIM Trio whole-body scanner, with a single-shot, interleaved multi-slice, gradient-echo, echo planar imaging sequence. The repetition time and echo time were 3000ms and 32ms, and the total scan duration for RS, EM and WM were 6.2 minutes, 11.6 minutes and 10.5 minutes, respectively. Standard preprocessing steps were applied by using SPM12 (www.fil.ion.ucl.ac.uk/spm/), including motion correction, spatial normalization to standard MNI space, and spatial smoothing with a 3mm FWHM Gaussian kernel. The functional time series were band-pass filtered using the 0.01Hz to the 0.1Hz frequency range. 264 ROIs (containing 21,384 voxels) were extracted based on the Power coordinates [43] with a sphere radius parameter of 5mm, and calculated the Pearson correlation between the time courses of each pair of ROIs. A 264 × 264 correlation matrix (FC matrix) for each subject in each single fMRI paradigm was then obtained. To reduce redundant information, only the lower triangular portion of the symmetrical correlation matrix was properly reformed into a vector with 34,716 correlation values. In the subsequent analysis, these 34,716 values were the features extracted from all three fMRI paradigms for each subject.

B. Experimental setting

In our experiments, all regularization parameters in the model were tuned by a 4-fold cross-validation on the training set through a grid search within their respective ranges, including sparsity level λ₁ ∈ {10⁻³, 2 × 10⁻³, ..., 8 × 10⁻³} and λ₂ ∈ {8, 10, 12, 14, 16}. To assess the reliability of components, Ref. [21], [44] examined the components in different number R = 10, R = 50 and R = 100. They found additional components were showed when R = 100, and sometimes split the R = 50 components into finer subdivisions. According to this observation, we set the number of components R = 100 to guarantee the uniqueness of tensor decomposition with sufficient components for analysis. Furthermore, due to computational limitations, increasing the components number much beyond this would slow down the computation considerably and expand memory usage beyond an acceptable level. So a proper number finally depends on the quality of the data and the detail desired from the decomposition.

Note that a large number of features (34,716 ROI correlations) and a relatively small number of samples (342 subjects) may cause the problem of overfitting or high computational complexity. To address this, in each fMRI paradigm dataset, we excluded the features for which p-values of the correlation with WRAT scores are greater than or equal to 0.05, and took the union of three datasets to train the model. For further analysis, we investigate whether the proposed method can classify two discriminative subsets according to cognitive ability. Here, low and high cognitive subsets are defined by extracting the subjects which were in both top and bottom τ percent of the WRAT scores, τ ∈ {10, 20, 30} [45]. Through the subset extractions, three groups of subsets with different number of subjects can be obtained for classification experiments. A 4-fold cross-validation is used to evaluate the classification performance. That is, the complete data were randomly partitioned into 4 disjoint subsets of equal size; each subset was successively selected as the testing set, and the other 3 subsets were used to train the SVM classifier. Finally, the trained SVM model was applied to classify the WRAT of subjects in the testing set. We repeated this process 100 times to reduce the effect of sampling bias in the cross-validation.

With the selected components, the strength and significance of the correlation between ROIs is measured by the hypothetical test. The null hypothesis of no correlation between two ROIs is written as

$$H_0={\rho }_{11}={\rho }_{12}=\dots ={\rho }_{uw}={\rho }_{{\eta }_i{\eta }_j}=0$$

versus the alternative hypothesis

$$H_A:\exists u,w>0,{\rho }_{uw}\ne 0$$

where η_i and η_j are the numbers of voxels in the i-th ROI and the j-th ROI, and ρ_uw is the correlation between the u-th voxel and the w-th voxel. To test the hypothesis, we calculated the pair-wise correlation between the voxles from the i-th ROI and the j-th ROI, and averaged the correlation of each voxel-voxel pair to avoid the varying size of ROI as follows.

$${\rho }_{{\eta }_i{\eta }_j}=\frac{1}{{\eta }_i{\eta }_j}\sum _{u=1}^{{\eta }_i}\sum _{w=1}^{{\eta }_j}\left|{\rho }_{uw}\right|$$ (15)

Then the significance of the correlation is evaluated via comparing ρ_ij with the null statistics ${\rho }_{\mathit{ij}}^0$ with 10000 times permutations of the samples.

$$p_{value}=\frac{\left\{{\rho }_h^0\ge {\rho }_{ij};h=1,2,\dots ,H\right\}}{10000}$$

Here, random sampling from 342 subjects are performed repeatedly for H = 50 times, selected the same proportion of subjects for training and test sets.

C. Results

The results in Fig.3 show that sparsity parameters affect the classification accuracy in three different tests. Since there are more females than males as shown in Fig.2, we reported the p-values of gender by Chi-square test comparing the number of males and females in both high IQ and low IQ groups. The p-values in three tests are 0.1456, 0.1684 and 0.0665, respectively. The values are larger than 0.05 that indicate sex differences would not influence the WRAT classification performance in our experiments. For a relatively low number test, 68 subjects are retained from top and bottom 10% of 342 subjects with average WRAT scores 130.26 and 74.56, respectively. The highest accuracy of our model is 0.8591 when λ₁ = 5 × 10⁻³ and λ₂ = 8. For the 20% test, the average WRAT scores of subjects selected from two ends are 123.10 and 79.49, respectively. The highest accuracy is 0.7517 under the same parameters setting as 10% group. When τ = 30, the best classification performance is 0.6774 when λ₁ = 2 × 10⁻³ and λ₂ = 14. It can be seen from Fig.3 that our model is sensitive to the sparsity parameters only in a small range. To better understand the effects of these parameters, the performance of the tensor decomposition model with λ₁ = 0 and λ₂ = 0 is also tested. In this case, our proposed model is degraded to the CPD model. Table II shows the comparison of classification results between using our model and CPD. The accuracy of three tests increase approximately 15.75%, 8.17% and 2.43% relative to CPD, respectively. It demonstrates that the use of regularization terms in Eq.(5) improve the overall classification performance. Notably, as the number of subjects increases in both low and high cognitive groups, the difference in WRAT scores between the two groups decreases as the percent of individuals included increases. This is because the groups are expanded by choosing individuals whose WRAT scores are less distinct (top/bottom 30% WRAT scores vs. top/bottom 10% WRAT scores). It may lead to a moderate decrease in classification accuracy.

Fig. 3:

The classification performance on 4-fold cross-validation for various parameter values. (a). 10%. (b). 20%. (c). 30%.

TABLE II:

The comparison of classification performance between MSTD and CPD models.

Data selection	10%	20%	30%
Number of subjects	68	136	204
Gender (Male/Female)	34/34	62/74	89/105
P-value of gender	0.1456	0.1684	0.0665
Ave WRAT	102.41	101.29	100.91
(Top/Bottom)	130.26/74.56	123.10/79.49	118.91/82.90
Accuracy of CPD	0.7016	0.6700	0.6531
Accuracy of MSTD	0.8591	0.7517	0.6774
Accuracy improved	15.75%	8.17%	2.43%

To further validate the performance of our model, MSTD is also utilized to predict the WRAT score and compared the performance of MSTD with other competing algorithms: SM [31], MTL [32], NM2TL [33] and CPD, where SM is a single-task based model, MTL and NM2TL are the joint learning models for two tasks. A 4-fold cross-validation is applied to evaluate the WRAT prediction performance for all these methods and tested 100 times independently on the whole dataset of 342 subjects. The model performance was quantified with both the correlation coefficient (CC) and the root mean square error (RMSE) between predicted and actual WRAT score of subjects in the testing set. According to the classification accuracy based on the different groups, the parameters of MSTD were tuned in a smaller range (λ₁ ∈ {2 × 10⁻³, 3 × 10⁻³, ..., 8 × 10⁻³}, λ₂ ∈ {8, 10, 12, 14, 16}).

Fig.4 summarizes the performance of all methods for predicting cognitive abilities. It can be seen that MSTD model shows the best performance in both CC and RMSE. More specifically, MSTD obtains the best CC 0.4599 as well as the best RMSE 14.5682 by combining three fMRI paradigms. CPD achieves the second best CC 0.4090 and outperforms the other methods, i.e., 0.3796 by NM2TL using WM and EM datasets, 0.3218 and 0.3233 by MTL using WM and EM dataset, 0.3179 and 0.3225 by SM using WM and EM dataset. In RMSE, NM2TL achieves the second best result of 14.7559 which is a bit smaller than 14.9789 by CPD, 15.3024 for MTL using WM, 15.2762 by MTL using EM, 15.4841 by SM using WM and 15.3965 by SM using EM. From the prediction tasks, MSTD gets a better performance than CPD in terms of bot CC and RMSE than CPD. That is, the regularization terms indeed improve the model performance by extracting more discriminative features associated with cognitive behaviors. In addition, it can be observed that two tensor-based models using three paradigms provide better performance than the models using one or two fMRI paradigms. MSTD has the minimum error among the compared models. Except NM2TL, the RMSE of CPD is significantly smaller than that of the other four models. It reveals that using the information provided by multiple datasets can effectively improve prediction accuracy and reduce errors.

Fig. 4:

The cognitive abilities prediction performance of 5 methods. (a). Performance in CCs. (b). Performance in RMSEs.

D. Analysis and discussion

Although 100 components are obtained by the proposed method, only a fraction of them is related to cognitive behaviors. To facilitate the analysis of these components, we used the components obtained from the 10% group of WRAT classification and restricted the components to those that were highly related with cognitive abilities (output the top 5 ranking components) based on the component weights of SVM classifier. On the basis of the work from Power et al. [43], these five components in 12 functional networks are visualized via the BrainNet viewer [46] (https://www.nitrc.org/projects/bnv/). Those include Sensory/somatomotor Hand (SM/H), Sensory/somatomotor Mouth (SM/M), Cingulo-Opercular task control (CON), Auditory (AUD), Default Mode (DMN), Memory Retrieval (MRN), Visual (VIS), Fronto-Parietal task control (FPN), Salience (SN), Subcortical (SUB), Ventral Attention (VAN) and Dorsal Attention (DAN).

The 1st component reflects a complex sub-network composed of different regions. Fig.5(A-a) displays the number of connections in the 1st component. Here we added 1 to each element if the value of the correlation between two nodes is not zero. Fig.5(A-b) shows the paradigm weights of the 1st component occurring in three paradigms, which are 0.58, 0.59 and 0.56 for RS, EM and WM, respectively. It indicated that the sub-network of the 1st components appears in all three paradigms simultaneously. Fig.6(A) represents the functional connectivity sub-network in different regions from the axial view, including SM/H, DMN, FPN and DAN. In these figures, the nodes are distinguished with different colors according to the functional networks where they are located. The node size is used to indicate the node connectivity strength (NCS). For instance, the NCS of the i-th node is defined with the sum of the absolute value of the i-th row in matrix F. A greater NCS implies a stronger connection of one node to the regions. Also, the colors of the edge indicate the correlation or anti-correlation of two nodes in red or blue, and the width of the edge represents the magnitude of the connectivity strength between nodes.

Fig. 5:

The number of connections from the 1st component to the 5th component (from A-a to E-a) in 12 main functional brain networks, and the weights of five components occurring in three fMRI paradigms (from A-b to E-b).

Fig. 6:

The connectivity from the 1st component to the 5th component (from A to E).

The 2nd and the 3rd components are more concerned with DMN. As shown in Fig.5(B-a), the 2nd component exhibits significant contributions only in DMN, as well as the 3rd component shows the links between DMN and other regions in Fig.5(C-a). Based on these observations, the connectivity in DMN of the 2nd component is displayed from the sagittal left, the axial and the sagittal right view (from left to right) in Fig.6(B), and represented the main sub-networks between DMN and the other three regions SM/H, VIS and DAN in Fig.6(C). Since both of the components appear in all three fMRI paradigms, the weights of the 2nd component in RS, EM and WM are 0.57, 0.61 and 0.54 (Fig.5(B-b)), and weights of the 3rd component in RS, EM and WM are 0.53, 0.51 and 0.68 (Fig.5(C-b)), respectively.

It should be noted that connections in the first three components are mainly located at SM/H, DMN, FPN and DAN, which are in accordance with the previous studies in the literature. For instance, Ref. [2] reported that the medial frontal network and FPN emerged as the most successful in individual subject identification, and the combination of these two frontoparietal networks significantly outperformed either network alone. Ref. [47] also indicated the connections between pre-frontal and frontal cortices (FPN) comprising DAN were associated with the intelligence, and significant brain-behavior associations was also observed for DMN, FPN and VIS. The predictive power of sub-networks in SM/H and DMN were found by [48], which was in line with observations in the 1st component. Moreover, similar results for the 3rd component were also described in [47], [49]. The correlations between lower functional connectivity and higher intelligence scores involve the connections within cortical areas comprising DMN as well as the functional interactions between these areas and regions within SM/H and DAN. From the 2nd and 3rd components, the connectivity in DMN are stronger than other regions, which is consistent with the current knowledge that DMN exhibits relatively high connectivity within itself [50].

Through Fig.5(D) and (E), it can be found that the 4th and the 5th components reflect the contributions of two specific tasks (EM and WM) to cognitive behaviors. Specifically, Fig.5(D) shows that the connectivity of the 4th component is concentrated in SM/H-DMN and it emerges in EM and WM with the weights 0.59 and 0.79. Fig.6(D) displays the connections of the 4th component from the sagittal left, the axial and the sagittal right view. In Fig.5(E), the 5th component have a strong relationship between DMN and other regions, including VIS and DAN, and it occurs in EM with the highest weight of 0.86.

From the point of view of brain functional system, the common connectivity domains of the first three components include DMN-DMN, DMN-SM/H, DMN-FPN and DMN-DAN, which appear in all three paradigms. That is, the functional connectivity of these regions is highly related to the resting state which is activated when performing the working memory or emotion task. DMN activates in task-free states and is implicated in spontaneous thought, evaluation and memory; it works in both cooperative and antagonistic ways with task control networks [50], [51]. In addition, cognitive functions rely on a complex interaction between these networks, where task control networks provide top-down regulatory signals that modulate spontaneous processing unfolding in association cortices within DMN [13]. Moreover, the 4th component reveals a particular ROI node in the SM/H, which links with other brain regions and exists in both EM and WM but rarely in RS. It seems it is caused by the motor activity when the subjects press the button in a specific task. The 5th component indicates some special FNCs linking DMN with VIS and DAN. Compared with RS and WM, the 5th component appears more in EM. It is also described in [52] that the ROI is associated with complex cognitive functions and activated in the emotion task.

The selected FNCs are mainly within or across frontal, parietal, occipital and temporal lobes. Here we listed the top ranked ROI nodes of each component (the NCS is greater than 0.4) in detail (see Supplementary material). It can be observed that the majority of nodes in the 1st component are located in superior frontal gyrus, middle frontal gyrus and precuneus. The majority of nodes in the 2nd component is within middle temporal gyrus. The 3rd component contains three nodes within supramarginal gyrus, medial frontal gyrus and angular gyrus, respectively. The 5th component occurring in EM task involves more cerebral cortex areas, such as cingulate gyrus, precuneus, angular gyrus, middle temporal gyrus, supramarginal gyrus, posterior cingulate, sub-gyral and middle occipital gyrus. Besides, some nodes located in middle temporal gyrus, supramarginal gyrus and angular gyrus recurrently appear in the 2nd, 3rd and 5th components, which suggests that these overlapped brain domains are activated under different task-stimulus. These findings are consistent with previous studies that indicate that activation across several regions within frontal, parietal, occipital and temporal lobes significantly predict cognitive abilities [33], [53], [54].

In summary, our main contributions are in three main points. First, using a tensor model to combine the information from multi-paradigm datasets can effectively select features shared among multiple subjects. Second, utilizing regularization terms can further remove redundant information, leading to the improved accuracy of classification and prediction. Third, the experimental analysis on the PNC data can identify important variations within and between DMN and SM/H, FPN and DAN, as well as the connections between VIS, DAN and DMN of task control FNC, are particularly significant for highlighting individual differences in cognitive behaviors.

IV. Conclusion

In this paper, we proposed a sparse tensor-based method to identify the individual differences by brain functional connectivity, using multiple fMRI paradigms. Specifically, the FNC of subjects in multiple fMRI paradigms is modeled as a three-way tensor, which is then decomposed into a series of rank-one components. Furthermore, the sparsity regularization terms are incorporated into the tensor model so significant FNC subnetworks can be extracted. Utilizing the low-dimensional components of each subject as features extracted by our sparse tensor model, SVM and SVR were trained for classifying and predicting cognitive abilities. Our method is validated with the PNC dataset from a comprehensive study of brain development. The experimental results demonstrated that our model outperformed other competing models. Meanwhile, some cognition-relevant functional connectivity and brain regions are discovered, and part of them were consistent with the previous studies in the literature. In general, the proposed method in this paper shows a valid way to circumvent the overfitting problem in high dimension but small sample fMRI datasets. It considers the relationships across subjects and can select the shared features among multiple fMRI paradigms, which is effective in combining complementary information from multiple fMRI paradigms. As a result, it can help identify reliable biomarkers reflecting individual differences.