Explainable Multimodal Deep Dictionary Learning to Capture Developmental Differences from Three fMRI Paradigms

Abstract

Objective:

Multimodal-based methods show great potential for neuroscience studies by integrating complementary information. There has been less multimodal work focussed on brain developmental changes.

Methods:

We propose an explainable multimodal deep dictionary learning method to uncover both the commonality and specificity of different modalities, which learns the shared dictionary and the modality-specific sparse representations based on the multimodal data and their encodings of a sparse deep autoencoder.

Results:

By regarding three fMRI paradigms collected during two tasks and resting state as modalities, we apply the proposed method on multimodal data to identify the brain developmental differences. The results show that the proposed model can not only achieve better performance in reconstruction, but also yield age-related differences in reoccurring patterns. Specifically, both children and young adults prefer to switch among states during two tasks while staying within a particular state during rest, but the difference is that children possess more diffuse functional connectivity patterns while young adults have more focused functional connectivity patterns.

Conclusion and Significance:

To uncover the commonality and specificity of three fMRI paradigms to developmental differences, multimodal data and their encodings are used to train the shared dictionary and the modality-specific sparse representations. Identifying brain network differences helps to understand how the neural circuits and brain networks form and develop with age.

I. Introduction

NORMAL brain development is a complex process, from the establishment of basic cognitive functions in childhood to the gradual maturity of more complex self-regulatory functions throughout adolescence [1]–[3]. Functional magnetic resonance imaging (fMRI) can capture hemodynamic responses to neuronal activities by measuring the blood oxygenation level-dependent (BOLD) signal, based on which the changes in neural interaction and integration between functionally interconnected regions with development can be revealed [4]. Compared with BOLD signals, dynamic functional connectivity (dFC) measured by a sliding window approach can reflect time-varying dependencies between spatially separated brain regions. It helps to quantify the changes of correlation strength between functional activities of paired brain regions over time. Thus, there has been growing interest in identification of the recurring whole-brain functional connectivity patterns (i.e., states) based on dFC recently. These studies aim to divide the whole-brain dFC profiles into distinct states observed reliably across subjects throughout the fMRI scans [5]–[8]. It enables us to investigate the differences of states related to brain development, capture the transition mechanism among these states, and provide insights into neural brain dynamics from the perspective of functional connectivity [4], [5].

Compared with single modality methods for fMRI analysis, multimodal-based methods can take advantage of complementary information provided by different modalities. Studies have shown that integrating the multimodal prior or combining the complementary information from diverse modalities can promote model enhancement and diagnosis [9]–[11]. Many methods have been extended for multimodal data integration including multitask learning, linear regression, neural network, support vector machine, and dictionary learning [9]–[14]. Due to the ability to reduce dimensionality and identify the reoccurring patterns of dFC [4], multimodal dictionary learning methods have attracted considerable attention. For example, Li et al. [11] proposed a multimodal discriminative dictionary learning (mSCDDL) method based on a weighted combination strategy, and further applied it to fuse information from structural magnetic resonance imaging and fluorodeoxyglucose positron emission tomography for Alzheimer's disease classification. In [15], a ${\ell }_1$-norm regularized dictionary learning approach was proposed to identify the epilepsy-related dFC states, where the time courses representative of epileptic activity extracted by electroencephalogram are incorporated into the fMRI for dFC state analysis. In [16], multimodal dictionary learning was applied to the diagnosis of schizophrenia, which embeds the correlation information of multimodal data into the learning model. Additionally, to achieve the nonlinearity or higher-level features of data, Li et al. [10] improved the mSCDDL with the multi-feature kernel trick to obtain the nonlinear representations of data. D'Souza et al. [17] proposed a framework for Autism Spectrum Disorder's diagnosis, which couples a structurally-regularized dynamic dictionary learning model (sr-DDL) with a deep network to predict behavioral scores, where the dFCs of fMRI were decomposed by sr-DDL while constraining the decomposition by the FCs of diffusion tensor imaging.

Of particular note, the aforementioned methods either fail to uncover both commonality and specificity of different modalities, or overlook the nonlinear higher-level features of data, or have difficulty in explainability (i.e., it fails to identify the reoccurring patterns of dFC, or brain regions and FCs related to development or disease). To address these issues, we propose an explainable multimodal deep dictionary learning (EMDDL) method, which connects the multimodal dictionary learning in the original space and the encoding space through a sparse deep autoencoder (sDAE). Within this framework, all modalities share the same dictionary to reveal the inherent commonality. To achieve the specificity of each modality, Fisher cost is used to constrain the sparse representations due to its ability to learn the modality-specific features by avoiding the overlap of neighboring pairs between different modalities. Moreover, the shared dictionary and the modality-specific sparse representations are learned based on the multimodal data and their encodings of the sDAE. In this way, multimodal dictionary learning can attain the nonlinear higher-level features while reconstructing the original data for identifying the reoccurring patterns or functional connectivity related to development. To maintain the complex relationships among subjects, a hypergraph Laplacian regularization is used, which helps to enhance the learning ability through prior knowledge.

We apply EMDDL to the multimodal data from Philadelphia Neurodevelopmental Cohort (PNC) to recognize the developmental differences between children and young adults, where the three fMRI paradigms collected during two tasks and resting state are regarded as modalities. We found that both children and young adults tend to switch frequently among states during two tasks and stay within a particular state during rest. The main difference is that children have more diffuse functional connectivity patterns while young adults possess more focused functional connectivity patterns under three fMRI paradigms. Besides, the differences in functional connectivity between children and young adults are mainly related to information processing, cognition, emotion, and working memory under three fMRI paradigms.

II. Preliminary Work

In this section, some preliminary work is presented including hypergraph learning to preserve the higher-order relation- ships among subjects and Fisher cost to extract modalityspecific features.

A. Hypergraph Learning

Given that the traditional graph learning loses information inevitably by squeezing the complex relationships into pairwise ones, hypergraph has been widely applied to identify the high-order relationships among subjects [18], [19]. Generally, a hypergraph $\mathcal{G}=\left(\mathcal{V},\mathcal{E},\mathcal{W}\right)$ consists of three parts, namely, the vertex set $\mathcal{V}=\left\{{\mathcal{V}}_i\mid i=1,2,\cdots ,N_v\right\}$, the hyperedge set $\mathcal{E}=\left\{{\mathcal{E}}_i\mid i=1,2,\cdots ,N_e\right\}$ and the hyperedge weight $\mathcal{W}=\left\{{\mathcal{W}}_i\mid i=1,2,\cdots ,N_e\right\}$. To represent the relationships between hyperedges and vertices, the incidence matrix $\mathcal{H}\in {ℝ}^{N_v\times N_e}$ of hypergraph $\mathcal{G}$ is defined as

$$\mathcal{H}\left({\mathcal{V}}_i,{\mathcal{E}}_j\right)=\left\{\begin{array}{ll}1\hfill & {\mathcal{V}}_i\in {\mathcal{E}}_j\hfill \\ 0\hfill & otherwise\hfill \end{array}\right.$$

where the $\left(i,j\right)$-th entry of $\mathcal{H}$ denotes whether the $i$-th vertex belong to the $j$-th hyperedge. Based on the incidence matrix $\mathcal{H}$, the degree of the $i$-th vertex $d_{{\mathcal{V}}_i}={\sum }_{{\mathcal{E}}_j\in \mathcal{E}}{\mathcal{W}}_j\mathcal{H}\left({\mathcal{V}}_i,{\mathcal{E}}_j\right)$ and the degree of the $i$-th hyperedge $d_{{\mathcal{E}}_i}={\sum }_{{\mathcal{V}}_j\in \mathcal{V}}\mathcal{H}\left({\mathcal{V}}_j,{\mathcal{E}}_i\right)$ can be obtained. Then the diagonal matrices ${\mathcal{D}}_v\in {ℝ}^{N_v\times N_v}$ and ${\mathcal{D}}_e\in {ℝ}^{N_e\times N_e}$ are composed of the degree of all vertices and hyperedges respectively. Specifically, the $i$-th diagonal element of ${\mathcal{D}}_v$ and ${\mathcal{D}}_e$ are $d_{{\mathcal{V}}_i}$ and $d_{{\mathcal{E}}_i}$ respectively.

To construct a hypergraph, the $k$ nearest neighbor strategy is usually applied because the geometric structure relationship among data can be approximately represented by the nearest neighbor graph [18], [20]. Specifically, for a chosen vertex, the distances between the chosen vertex and other vertices are calculated, and then the $k$ nearest vertices are connected by a hyperedge. The weight of the $i$-th hyperedge is ${\mathcal{W}}_i=\frac{1}{k\left(k-1\right)}{\sum }_{\left\{{\mathcal{V}}_j,{\mathcal{V}}_l\right\}\in {\mathcal{E}}_i}\text{exp}\left(-\frac{{‖{\mathcal{V}}_j-{\mathcal{V}}_l‖}_2^2}{{\sigma }_i}\right)$, where ${\sigma }_i=\frac{{\sum }_{\left\{{\mathcal{V}}_j,{\mathcal{V}}_l\right\}\in {\mathcal{E}}_i}{‖{\mathcal{V}}_j-{\mathcal{V}}_l‖}_2^2}{k\left(k-1\right)}$. To obtain the diagonal matrix ${\mathcal{W}}_h\in {ℝ}^{N_e\times N_e}$, the hyperedge weight ${\mathcal{W}}_i$ is arrayed as the $i$-th diagonal element of ${\mathcal{W}}_h$. By analogizing the definition of a simple graph Laplacian matrix [21], hypergraph Laplacian matrix is defined as

$$L^h={\mathcal{D}}_v-\mathcal{S}$$ (1)

where $\mathcal{S}=\mathcal{H}{\mathcal{W}}_h{\mathcal{D}}_e^{-1}{\mathcal{H}}^{\text{T}}$ is the similarity matrix to define the similarity between each pair of vertices.

Compared with the traditional graph Laplacian regularization, hypergraph Laplacian regularization has the characteristics of preserving complex local geometric structure and incorporating the higher-order relationships among subjects, which are conducive to classification or clustering tasks in FC or dFC analysis [19].

B. Fisher cost

The Fisher discrimination criterion is to cluster the samples in the same modality and keep the samples in different modalities as far away from each other as possible, which helps to extract features corresponding to the specific modality [22]–[24]. Assume that the multimodal data $X=\left(x_1,x_2,\cdots ,x_N\right)\in {ℝ}^{p\times N}$ contains $M$ modalities with $N_m$ samples belonging to the $m$-th modality $N_m$ and ${\sum }_{m=1}^M{N}_m=N$, where $p$-dimensional vector $x_n$ is the $n$-th sample of $X$. The withinmodality scatter matrix $S_w$ and the between-modality scatter matrix $S_b$ of samples are defined as

$$S_w\left(X\right)={\displaystyle \sum }_{m=1}^M{\displaystyle \sum }_{n\in {\mathcal{N}}_m}\left(x_n-{\mu }_m\right){\left(x_n-{\mu }_m\right)}^{\text{T}}$$

$$S_b\left(X\right)={\displaystyle \sum }_{m=1}^M{N}_m\left({\mu }_m-\mu \right){\left({\mu }_m-\mu \right)}^{\text{T}}$$

where ${\mu }_m=\frac{1}{N_m}{\sum }_{n\in {\mathcal{N}}_m}{x}_n$ and $\mu =\frac{1}{N}{\sum }_{n=1}^N{x}_n$ are the modality mean and the overall mean respectively. Then, the Fisher cost is as follows

$$\mathcal{F}\left(X\right)=tr\left(S_w\left(X\right)\right)-tr\left(S_b\left(X\right)\right)+\parallel X{\parallel }_F^2$$

in which the Frobenius norm $\parallel \cdot {\parallel }_F$ is to ensure the convexity of the cost function [24].

To get a more concise expression and facilitate calculation [22], the Fisher cost $\mathcal{F}\left(X\right)$ can be rewritten as

$$\mathcal{F}\left(X\right)=tr\left(XF{X}^{\text{T}}\right)$$ (2)

where $F=2I-2F_1+F_2\in {ℝ}^{N\times N}$ with $I\in {ℝ}^{N\times N}$ being the identity matrix, $F_1\in {ℝ}^{N\times N}$ being defined as

$$F_1\left(i,j\right)=\left\{\begin{array}{ll}\frac{1}{N_m}\hfill & i,j\in {\mathcal{N}}_m\hfill \\ 0\hfill & otherwise\hfill \end{array}\right.$$

and $F_2\in {ℝ}^{N\times N}$ with each component of it being $1/N$.

III. Methodology

The details of EMDDL and the corresponding optimization algorithm are presented in this section, which can learn the shared dictionary and modality-specific sparse representations in both the original space and the encoding space.

A. Explainable Multimodal Deep Dictionary Learning

Multimodal dictionary learning methods can not only embed the high-dimensional features into low-dimensional space, but also boost learning performance with the combination of multiple modalities [12]. However, most of the existing methods either cannot simultaneously reveal the inherent commonality and specificity of different modalities, or overlook the nonlinear higher-level features of data, or have difficulty in explainability. To address these problems, we propose EMDDL which couples multimodal dictionary learning with sDAE. Specifically, by sharing the same dictionary through all modalities to capture the inherent commonality and constraining sparse representations with Fisher cost to obtain the specificity of each modality, the inherent commonality and the specificity of different modalities can be concurrently achieved in multimodal dictionary learning. Moreover, to achieve the nonlinear higher-level features of data and reconstruct the original data to identify the developmental differences in reoccurring patterns or FCs, both the shared dictionary and the modality-specific sparse representations are learned not only in the original space, but also in the encoding space of the sDAE at the same time. By alternating minimization algorithms, the sDAE, dictionary, and sparse representations can be sequentially obtained. The flowchart of EMDDL is shown in Fig. 1.

Fig. 1:

The flowchart of EMDDL.

Suppose that there are $M$ modalities with $N_m$ samples belonging to the $m$-th modality ${\mathcal{N}}_m$ and the training data $X=\left(X_{\left(1\right)},X_{\left(2\right)},\cdots ,X_{\left(M\right)}\right)\in {ℝ}^{p\times N}$ is composed of these $M$ modalities, where $N={\sum }_{m=1}^M{N}_m$ and the $m$-th modality is $X_{\left(m\right)}=\left(x_1^{\left(m\right)},x_2^{\left(m\right)},\cdots ,x_{N_m}^{\left(m\right)}\right)\in {ℝ}^{p\times N_m}$. Besides, the sDAE contains $2L+1$ layers with $r^{\left(l\right)}$ neurons in the $l$-th layer and $r^{\left(2L-l\right)}=r^{\left(l\right)}$ holds for $l=0,1,\cdots ,2L$.

EMDDL contains two parts including $J_{sDAE}$ and $J_{MDL}$, where $J_{sDAE}$ is to efficiently learn the nonlinear higher-level features of data and $J_{MDL}$ is to train multimodal dictionary learning in both the original space and the encoding space. The objective function of EMDDL is defined as

$$\begin{gathered} \underset{{\left\{{\tilde{W}}^{\left(l\right)}\right\}}_{l=1}^{2L},D,V}{\text{min}}\hspace{1em}{J}_{obj}=J_{sDAE}+J_{MDL} \\ \hspace{1em}\hspace{1em}s.t.\hspace{1em}{‖d_k‖}_2^2\le 1,\forall k=1,2,\cdots ,K \end{gathered}$$ (3)

where $J_{sDAE}$ and $J_{MDL}$ are defined as

$$\begin{gathered} J_{sDAE}=J_{recon}+{\lambda }_1{J}_{KL}+{\lambda }_2{J}_{W_F} \\ \hspace{1em}\hspace{1em}=\frac{1}{2N}\parallel X^{\left(2L\right)}-X{\parallel }_F^2+{\lambda }_1{\displaystyle \sum }_{l=1}^{2L-1}{\displaystyle \sum }_{j=1}^{r^{\left(l\right)}}KL\left(\rho \parallel {\rho }_j^{\left(l\right)}\right) \\ \hspace{1em}\hspace{1em}\hspace{0.33em}\hspace{0.33em}+\frac{{\lambda }_2}{2}{\displaystyle \sum }_{l=1}^{2L}{‖{\tilde{W}}^{\left(l\right)}‖}_F^2 \end{gathered}$$ (4)

$$\begin{gathered} J_{MDL}=J_{MD{L}_O}+J_{MD{L}_E}+{\lambda }_3{J}_{Fisher}+{\lambda }_4{J}_{hyper\hspace{0.17em}L} \\ \hspace{1em}\hspace{1em}=+{\lambda }_5{J}_{V_F}+{\lambda }_6{J}_{V_{{\ell }_1}} \\ \hspace{1em}\hspace{1em}\frac{1}{2N}\parallel X-DV{\parallel }_F^2+\frac{1}{2N}{‖X^{\left(L\right)}-D^{\left(L\right)}V‖}_F^2 \\ \hspace{1em}\hspace{1em}\hspace{0.33em}+\frac{{\lambda }_3}{2}tr\left(VH{V}^T\right)+\frac{{\lambda }_4}{2}tr\left(VL{V}^T\right)+\frac{{\lambda }_5}{2}\parallel V{\parallel }_F^2 \\ \hspace{1em}\hspace{1em}\hspace{0.33em}+{\lambda }_6\parallel V{\parallel }_1 \end{gathered}$$ (5)

where $X^{\left(2L\right)}\in {ℝ}^{r^{\left(2L\right)}\times N}$ is the reconstruction of the input data $X$ by sDAE, $D=\left(d_1,d_2,\cdots ,d_K\right)\in {ℝ}^{p\times K}$ is the dictionary with $K$ atoms in the original space, $X^{\left(L\right)}\in {ℝ}^{r^{\left(L\right)}\times N}$ and $D^{\left(L\right)}\in {ℝ}^{r^{\left(L\right)}\times K}$ are the encoding of $X$ and $D$ respectively (i.e., its outputs in the $L$-th layer), $V=\left(V_{\left(1\right)},V_{\left(2\right)},\cdots ,V_{\left(M\right)}\right)\in {ℝ}^{K\times N}$ consists of each $V_{\left(m\right)}=\left(v_1^{\left(m\right)},v_2^{\left(m\right)},\cdots ,v_{N_m}^{\left(m\right)}\right)\in {ℝ}^{K\times N_m}$ being the sparse representation corresponding to the $m$-th modality in both the original space and the encoding space. ${\tilde{W}}^{\left(l\right)}=\left(W^{\left(l\right)},b^{\left(l\right)}\right)\triangleq \left\{{\tilde{W}}_{ij}^{\left(l\right)}\right\}\in {ℝ}^{r^{\left(l\right)}\times r^{\left(l-1\right)}+1}$ for $l=1,2,\cdots ,2L$, in which $W^{\left(l\right)}\in {ℝ}^{r^{\left(l\right)}\times r^{\left(l-1\right)}}$ and $b^{\left(l\right)}\in {ℝ}^{r^{\left(l\right)}}$ are the connection weight matrix and bias of sDAE between $l$-th layer and $\left(l-1\right)$-th layer respectively. As defined in Appendix I-A, Kullback-Leibler divergence $KL\left(\rho \parallel {\rho }_j^{\left(l\right)}\right)$ measures the difference between two Bernoulli distributions with mean $\rho$ and ${\rho }_j^{\left(l\right)}$, where $\rho$ is a sparsity hyperparameter and ${\rho }_j^{\left(l\right)}$ is the average activation of neuron $j$ in the $l$-th layer of sDAE. Similar to the definition of $F$ in (2), $H\in {ℝ}^{N\times N}$ is given by $H=I-2H_1+H_2\hspace{0.17em}$, where $I\in {ℝ}^{N\times N}$ is the identity matrix, $H_1\in {ℝ}^{N\times N}$ is defined as

$$H_1\left(i,j\right)=\left\{\begin{array}{ll}\frac{1}{N_m}\hfill & i,j\in {\mathcal{N}}_m\hfill \\ 0\hfill & otherwise\hfill \end{array}\right.$$

and the each component of $H_2\in {ℝ}^{N\times N}$ is $1/N$. $L\in {ℝ}^{N\times N}$ consists of hypergraph Laplacian matrix of all modalities, which is defined as

$$L=\left(\begin{array}{cccc}{L}_{\left(1\right)}^h& 0& \cdots & 0\\ 0& L_{\left(2\right)}^h& \cdots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \cdots & L_{\left(M\right)}^h\end{array}\right)$$

where $L_{\left(m\right)}^h\in {ℝ}^{N_m\times N_m}$, the hypergraph Laplacian matrix of the $m$-th modality, is defined by (1). $\parallel V{\parallel }_1= {\sum }_{n=1}^N{\sum }_{k=1}^K\left|V_{nk}\right|$ with $V_{nk}$ being the $k$-th element of the $n$-th column of the matrix $V$.

In (4), $J_{recon}$ is to train the sDAE by minimizing the error between original data and its reconstruction. $J_{KL}$ is to prevent overfitting of the sDAE by controlling the activation of neurons. Compared with $L_1$-norm and $L_2$-norm, KullbackLeibler divergence has better sparsity ability, which helps to improve model performance, and the details can be seen in Appendix I-A. $J_{W_F}$ is to prevent overfitting of the sDAE by controlling the weights. In (5), $J_{MD{L}_O}$ is to learn the shared dictionary of all modalities and the modality-specific sparse representations based on the original data. Meanwhile, by encoding the original data and the shared dictionary through the sDAE, $J_{MD{L}_E}$ is to achieve the multimodal dictionary learning in the encoding space for capturing the nonlinear higher-level features of data. Inspired by [25], we use the same sparse representations to synchronously characterize the local geometric relationships between data and dictionary in the original space as to characterize those between encoded data and encoded dictionary in the encoding space. In other words, our objective is to use the sparse representations to capture the intrinsic local geometric relationships between data and dictionary. It helps to learn the locality-sensitive dictionary, resulting in improved generalization ability in reconstrcution or classification. By clustering the samples within modalities and separating the samples between the modalities, $J_{Fisher}$ helps to learn the modality-specific representations. $J_{hypherL}$ is designed to retain the complex neighborhood relationships of samples hidden in each modality. $J_{V_F}$ guarantees the convexity of Fisher cost and $J_{V_{{\ell }_1}}$ is to ensure the sparsity. The constraint on dictionary atoms is to prevent sparse representation from being too small due to the large dictionary. In addition, the positive parameters ${\lambda }_1$, ${\lambda }_2$, ${\lambda }_3$, ${\lambda }_4$, ${\lambda }_5$ and ${\lambda }_6$ are used to balance the network fitting, dictionary learning and the complexity of model.

B. Optimization

The alternating minimization algorithm is applied to solve the problem (3) to optimize the parameters ${\left\{{\tilde{W}}^{\left(l\right)}\right\}}_{l=1}^{2L}$, $D$ and $V$, which contains three parts, i.e., the training of sDAE, the learning of the dictionary, and sparse representations learning.

Denote

$$\left\{\begin{array}{l}{\tilde{h}}_n^{\left(l\right)}=\left(h_n^{\left(l\right)};1\right)\\ \text{}{z}_n^{\left(l+1\right)}={\tilde{W}}^{\left(l+1\right)}{\tilde{h}}_n^{\left(l\right)}l=0,1,\cdots ,2L-1\text{}\\ h_n^{\left(l+1\right)}=\phi \left(z_n^{\left(l+1\right)}\right)\end{array}\right.$$

$$\left\{\begin{array}{l}{\tilde{g}}_k^{\left(l\right)}=\left(g_k^{\left(l\right)};1\right)\text{}\\ q_k^{\left(l+1\right)}={\tilde{W}}^{\left(l+1\right)}{\tilde{g}}_k^{\left(l\right)}l=0,1,\cdots ,L-1\text{}\\ g_k^{\left(l+1\right)}=\phi \left(q_k^{\left(l+1\right)}\right)\end{array}\right.$$

where $\phi$ is a differentiable activation function which is the sigmoid function in this paper; $h_n^{\left(0\right)}=X_n$ and $g_k^{\left(0\right)}=d_k$, where $X_n$ is the $n$-th column of the multimodal data $X$ and $d_k$ is the $k$-th atom of the dictionary $D$. $X^{\left(l\right)}=\left(h_1^{\left(l\right)},h_2^{\left(l\right)},\cdots ,h_N^{\left(l\right)}\right)\in {ℝ}^{r^{\left(l\right)}\times N}$ $l=1,2,\cdots ,2L$ and $D^{\left(l\right)}=\left(g_1^{\left(l\right)},g_2^{\left(l\right)},\cdots ,g_K^{\left(l\right)}\right)\in {ℝ}^{r^{\left(l\right)}\times K}$, $l=1,2,\cdots ,L$ are the outputs in the $l$-th layer when the input data are $X$ and $D$ respectively.

1). The Training of SDAE:

To optimize the parameters of $\text{sDAE}\hspace{0.33em}{\left\{{\tilde{W}}^{\left(l\right)}\right\}}_{l=1}^{2L}$ with fixed $D$ and $V$, problem (3) can be rewritten as

$$\begin{gathered} \underset{{\left\{{\tilde{W}}^{\left(l\right)}\right\}}_{l=1}^{2L}}{\text{min}}\hspace{1em}\frac{1}{2N}\left({‖X^{\left(2L\right)}-X‖}_F^2+{‖X^{\left(L\right)}-D^{\left(L\right)}V‖}_F^2\right)+ \\ \hspace{1em}\hspace{1em}\hspace{1em}{\lambda }_1{\displaystyle \sum }_{l=1}^{2L-1}{\displaystyle \sum }_{j=1}^{r^{\left(l\right)}}KL\left(\rho \parallel {\rho }_j^{\left(l\right)}\right)+\frac{{\lambda }_2}{2}{\displaystyle \sum }_{l=1}^{2L}{‖{\tilde{W}}^{\left(l\right)}‖}_F^2 \end{gathered}$$ (6)

To update the parameters of $\text{sDAE}\hspace{0.33em}{\left\{{\tilde{W}}^{\left(l\right)}\right\}}_{l=1}^{2L}$, the backpropagation algorithm with gradient descent method is applied. Then, the gradient of ${\tilde{W}}^{\left(l\right)}$ is given by

$$\begin{gathered} \nabla {\tilde{W}}^{\left(l\right)}=\frac{1}{N}{\displaystyle \sum }_{n=1}^N\left(\text{Δ}{H}_n^{\left(l\right)}{\tilde{h}}_n^{{(l-1)}^{\text{T}}}+I\left(L-l\right)\text{Δ}{T}_n^{\left(l\right)}\right. \\ \hspace{1em}\hspace{1em}\hspace{1em}\left.+{\lambda }_{1}I\left(2L-1-l\right)\text{Δ}{S}_n^{\left(l\right)}{\tilde{h}}_n^{{(l-1)}^{\text{T}}}\right)+{\lambda }_2{\tilde{W}}^{\left(l\right)} \end{gathered}$$ (7)

in which $\text{Δ}{H}_n^{\left(l\right)}$ is defined as

$$\text{Δ}{H}_n^{\left(l\right)}=\left\{\begin{array}{ll}\left(h_n^{\left(l\right)}-x_n\right)\odot {\phi }^{\prime }\left(z_n^{\left(l\right)}\right)\hfill & l=2L\hfill \\ \left(W^{{(l+1)}^{\text{T}}}\text{Δ}{H}_n^{\left(l+1\right)}\right)\odot {\phi }^{\prime }\left(z_n^{\left(l\right)}\right)\hfill & l=2L-1,\cdots ,2,1\hfill \end{array}\right.$$

where the operation $\odot$ denotes the element-wise multiplication. $I\left(\cdot \right)$ is an indicator function defined by

$$I\left(s\right)=\left\{\begin{array}{ll}1\hfill & s\ge 0\hfill \\ 0\hfill & otherwise\hfill \end{array}\right.$$

$\text{Δ}{T}_n^{\left(l\right)}=\text{Δ}{T}_n^{\left(l\right)}\left(0\right){\tilde{h}}_n^{{(l-1)}^{\text{T}}}-{\sum }_{k=1}^K\text{Δ}{T}_n^{\left(l\right)}\left(k\right){\tilde{g}}_k^{{(l-1)}^{\text{T}}}$ with $\text{Δ}{T}_n^{\left(l\right)}\left(0\right)$ and $\text{Δ}{T}_n^{\left(l\right)}\left(k\right)$ being defined as

$$\text{Δ}{T}_n^{\left(l\right)}\left(0\right)=\left\{\begin{array}{ll}\left(h_n^{\left(l\right)}-D^{\left(l\right)}{V}_n\right)\odot {\phi }^{\prime }\left(z_n^{\left(l\right)}\right)\hfill & l=L\hfill \\ \left(W^{{(l+1)}^{\text{T}}}\text{Δ}{T}_n^{\left(l+1\right)}\left(0\right)\right)\odot {\phi }^{\prime }\left(z_n^{\left(l\right)}\right)\hfill & l=L-1,\cdots ,2,1\hfill \end{array}\right.$$

$$\text{Δ}{T}_n^{\left(l\right)}\left(k\right)=\left\{\begin{array}{r}\hfill V_{nk}\left(h_n^{\left(l\right)}-D^{\left(l\right)}{V}_n\right)\odot {\phi }^{\prime }\left(q_k^{\left(l\right)}\right)\hspace{1em}l=L\\ \hfill k=1,2,\cdots ,K\\ \hfill \left(W^{{(l+1)}^{\text{T}}}\text{Δ}{T}_n^{\left(l+1\right)}\left(k\right)\right)\odot {\phi }^{\prime }\left(q_k^{\left(l\right)}\right)\\ \hfill l=L-1,\cdots ,2,1\\ \hfill k=1,2,\cdots ,K\end{array}\right.$$

in which $V_n$ is the $n$-th column of $V$. $\text{Δ}{S}_n^{\left(l\right)}\left(t\right)$ is defined as

$$\text{Δ}{S}_n^{\left(l\right)}\left(t\right)=\left\{\begin{array}{ll}{R}^{\left(l\right)}\odot {\phi }^{\prime }\left(z_n^{\left(l\right)}\right)\hfill & \begin{array}{l}t=2L-l\\ l=2L-1,\cdots ,2,1\end{array}\hfill \\ \left(W^{{(l+1)}^{\text{T}}}\text{Δ}{S}_n^{\left(l+1\right)}\left(t\right)\right)\odot {\phi }^{\prime }\left(z_n^{\left(l\right)}\right)\hfill & \begin{array}{r}\hfill \begin{array}{l}t=1,2,\cdots ,2L-1-l\\ l=2L-2,\cdots ,2,1\end{array}\end{array}\hfill \end{array}\right.$$

where $R^{\left(l\right)}$ is a $r^{\left(l\right)}$-dimensional column vector with $i$-th element being $\left(\frac{-\rho }{{\rho }_i^{\left(l\right)}}+\frac{1-\rho }{1-{\rho }_i^{\left(l\right)}}\right)$, and $\text{Δ}{S}_n^{\left(l\right)}={\sum }_{t=1}^{2L-l}\text{Δ}{S}_n^{\left(l\right)}\left(t\right)$. The update formula for ${\tilde{W}}^{\left(l\right)}$ is

$${\tilde{W}}^{\left(l\right)}={\tilde{W}}^{\left(l\right)}-{\eta }_1\nabla {\tilde{W}}^{\left(l\right)}$$

where ${\eta }_1$ is the learning rate.

2). The Learning of Dictionary:

To update dictionary $D$ with fixed ${\left\{{\tilde{W}}^{\left(l\right)}\right\}}_{l=1}^{2L}$ and $V$, problem (3) can be rewritten as

$$\begin{array}{l}\underset{D}{\text{min}}\hspace{0.33em}\frac{1}{2N}\left(\parallel X-DV{\parallel }_F^2+{‖X^{\left(L\right)}-D^{\left(L\right)}V‖}_F^2\right)\\ s.t.\hspace{0.33em}{‖d_k‖}_2^2\le 1,\hspace{1em}\forall \hspace{1em}k=1,2,\cdots ,K\end{array}$$ (8)

The gradient descent method is used to optimize the above problem and the gradient of $D$ is given by

$$\nabla D=\frac{1}{N}\left(\left(DV-X\right)V^{\text{T}}+\text{Δ}R\right)$$ (9)

in which the $k$-th column of $\text{Δ}R$ is computed by ${\sum }_{n=1}^N\text{Δ}{R}_n^{\left(1\right)}\left(k\right)$ and $\text{Δ}{R}_n^{\left(l\right)}\left(k\right)$ is defined as

$$\text{Δ}{R}_n^{\left(l\right)}\left(k\right)=\left\{\begin{array}{ll}{W}^{{\left(l\right)}^T}\left(V_{nk}\left(D^{\left(l\right)}{V}_n-h_n^{\left(l\right)}\right)\odot {\phi }^{\prime }\left(q_k^{\left(l\right)}\right)\right)\hfill & \begin{array}{l}l=L\\ k=1,2,\cdots ,K\end{array}\hfill \\ W^{{(l)}^{\text{T}}}\left(\text{Δ}{R}_n^{\left(l+1\right)}\left(k\right)\odot {\phi }^{\prime }\left(q_k^{\left(l\right)}\right)\right)\hfill & \begin{array}{r}\hfill \begin{array}{l}l=L-1,\cdots 2,1\\ k=1,2,\cdots ,K\end{array}\end{array}\hfill \end{array}\right.$$

The update formula for $D$ is

$$D=D-{\eta }_2\nabla D$$

where ${\eta }_2$ is the learning rate. Considering the constraint on the dictionary, each column of the updated dictionary $D$ is normalized to unit length by

$$d_k=\frac{1}{{‖d_k‖}_2}{d}_k$$ (10)

3). Sparse Representations Learning:

With the fixed ${\left\{{\tilde{W}}^{\left(l\right)}\right\}}_{l=1}^{2L}$ and $D$, the sparse representations can be obtained by solving the following optimization problem

$$\underset{V}{\text{min}}f\left(V\right)+g\left(V\right)$$ (11)

where $f\left(V\right)$ and $g\left(V\right)$ are

$$\begin{gathered} f\left(V\right)=\frac{1}{2N}\left(\parallel X-DV{\parallel }_F^2+{‖X^{\left(L\right)}-D^{\left(L\right)}V‖}_F^2\right) \\ \hspace{1em}\hspace{1em}\hspace{1em}+\frac{{\lambda }_3}{2}tr\left(VH{V}^T\right)+\frac{{\lambda }_4}{2}tr\left(VL{V}^T\right)+\frac{{\lambda }_5}{2}\parallel V{\parallel }_F^2 \end{gathered}$$

$$g\left(V\right)={\lambda }_6\parallel V{\parallel }_1$$

To ensure the convexity of $f\left(V\right)$, ${\lambda }_5>{\lambda }_3\ge 0$ holds and the details can be seen in Appendix I-B. In problem (11), $f\left(V\right)$ is convex and differentiable, while $g\left(V\right)$ is convex but nondifferentiable. Thus, the Fast Iterative Shrinkage Thresholding Algorithm (FISTA) [26] is adopted to optimize $V$. The gradient of $f\left(V\right)$ with respect to $V$ is

$$\nabla V=\frac{1}{N}\left(D^{\text{T}}\left(DV-X\right)+D^{{(L)}^{\text{T}}}\left(D^{\left(L\right)}V-X^{\left(L\right)}\right)\right)+VS$$ (12)

in which $S={\lambda }_{3}H+{\lambda }_{4}L+{\lambda }_{5}I$. The Lipschitz constant of the gradient $\nabla V$ is given by (13) in Appendix I-C. Besides, the soft thresholding function in FISTA is defined as $S{T}_{\frac{{\lambda }_6}{L_f}}\left(\cdot \right)=sign\left(\cdot \right)max\left\{0,\left|\cdot \right|-\frac{{\lambda }_6}{L_f}\right\}$ with $\left|\cdot \right|$ representing absolute value function. The total optimization process is described in Algorithm 1.

IV. Results and Analysis

In this section, EMDDL is utilized to explore the dynamic functional connectivity changes of brain during two tasks and resting state.

A. Data Acquisition and Preprocessing

PNC is a large scale collaborative project between the Brain Behaviour Laboratory at the University of Pennsylvania and the Children's Hospital of Philadelphia, which contains data collected using three fMRI paradigms from nearly 900 youth aged from 8 to 22 , i.e., two tasks including emotion identification (Emoid fMRI) and working memory (Nback fMRI), and resting-state (Rest fMRI) [27]. All fMRI scans were collected on a single 3T Siemens TIM Trio whole-body scanner using a single-shot, interleaved multi-slice, gradientecho, echo planar imaging sequence. The Emoid fMRI, Nback fMRI and Rest fMRI scan durations were 10.5 minutes (210 TR), 11.6 minutes (231 TR) and 6.2 minutes (124 TR) respectively. During Emoid task, subjects were asked to identify 60 faces with neutral, happy, sad, angry, or fearful expressions. During Nback task to probe working memory, subjects were required to respond only when a presented fractal was the same as the one presented in the previous trial. During the resting-state scan, subjects were instructed to stay awake, keep eyes open, fixate on the displayed crosshair, and remain still. Of these, 123 children and 146 young adults completed all three paradigms. By using Statistical Parametric Mapping 12, motion correction, co-registration, spatial normalization to standard Montreal Neurological Institute space (spatial resolution of $3\times 3\times 3\text{ mm}$), and spatial smoothing with a 3 mm full width half maximum Gaussian kernel were implemented. Then, a regression procedure was used to remove the influence of motion and the functional time series were band-pass filtered using a $0.01\text{ Hz}$ to $0.1\text{ Hz}$ frequency range. According to the Power coordinates with a sphere radius parameter of $5\text{ mm}$ [28], 264 regions of interest (ROIs) containing 21384 voxels were extracted. The details of the 264 ROIs are shown in Table 2 of Supplementary material. Every subject file can be reduced to a $264\times T$ matrix by averaging the time series of all voxels in the same brain region, where the time point $T$ is 210, 231, and 124 for Emoid, Nback, and Rest fMRIs respectively.

We divided 264 ROIs into 13 functional networks to facilitate the understanding of functional connectivity relationships between the ROIs [28]. Among them, 12 functional networks including sensory/somatomotor network (SSN), cinguloopercular task control network (COTCN), auditory network (AN), default mode network (DMN), memory retrieval network (MRN), visual network (VN), frontoparietal task control network (FPTCN), salience network (SN), subcortical network (SCN), ventral attention network (VAN), dorsal attention network (DAN), and cerebellar network (CN), are mainly associated with the perception of movement, memory, language, vision, cognition and other functions of the brain, while there are 28 ROIs unrelated to any of the above functional networks which belong to the uncertain network (UN).

To capture the dynamic characteristics of the brain, dFC is obtained by calculating the Pearson correlation between the time-courses of the BOLD signals of pair regions within a window [29]–[31]. The details of obtaining the multimodal data can be seen in Appendix I-D. By grid search, we choose window length $w_l$ being 14, 17, and 33 for Emoid, Nback, and Rest fMRIs respectively, and scan length $s_l$ is 1 for all three modalities. Thus, each subject provides a dFC matrix $M_{dFC}\in {ℝ}^{C_{264}^2\times S_l}$ corresponding to Emoid, Nback, and Rest fMRIs, where $C_{264}^2=34716$. To reduce the complexity of computation, systematic sampling is used to select 20 sub-sequences from the dFC matrix corresponding to each modality of each subject [4]. Training data contains 80% of the subjects and the remaining subjects are test data.

B. Experimental Results

To evaluate the performance of the algorithm, the signal-to-noise ratio (SNR) [32] is used as evaluation index which is defined as

$$SNR=10lo{g}_{10}\left(\frac{\parallel X{\parallel }_F^2}{\parallel X-DV{\parallel }_F^2}\right)$$

Given that the grid search method can simply make a complete search over a given hyperparameters space, easily be parallelized to find more stable optimal hyperparameter [33], [34], it is used to select appropriate hyperparameters. Specifically, one of the hyperparameters is selected by the grid search method when other hyperparameters are fixed. By repeating the above process, all hyperparameters are optimized, and the results are shown in Figure 1 of the Supplementary material. There are 7 layers of sDAE with 34716, 10000, 6000, 1000, 6000,10000, 34716 units respectively. The number of atoms $K$ is 300 , the sparsity parameter $\rho$ is 0.1 , the regularization coefficients ${\lambda }_1$, ${\lambda }_2$, ${\lambda }_3$, ${\lambda }_4$, ${\lambda }_5$, and ${\lambda }_6$ are 0.0001, 0.0005, 0.0003, 0.0001, 0.0005, and 0.001 respectively, the $k$ nearest neighbor of hypergraph is 9. Because problem (6) and (8) are nonconvex, RMSProp algorithm is used to update ${\left\{{\tilde{W}}^{\left(l\right)}\right\}}_{l=1}^{2L}$ and $D$ due to the better generalization ability and it is less prone to overfitting [35]. For RMSProp algorithm, the learning rates ${\eta }_1$ and ${\eta }_2$ are 0.00005 and 0.00008 respectively, and the square gradient decay rates ${\xi }_1$ and ${\xi }_2$ are both 0.9.

Based on the optimal hyperparameters, we apply EMDDL to the training data to obtain the dictionary and sparse representations. The learning curves of loss functions and the SNR evaluation on both training data and testing data are shown in Fig. 2. The results testify that the sparse representations can characterize the local geometric relationships between data and dictionary in the two spaces. The SNR of EMDDL, multimodal dictionary learning (MDL) [36] and sparse deep dictionary learning (SDDL) [4] on testing data are shown in Table I. It shows that the multimodal-based methods have better reconstruction ability compared with the single modality methods, and the generalization ability in reconstruction of EMDDL are better than the other two methods. It testifies that integrating the multimodal prior or combining the complementary information from diverse modalities can promote model enhancement.

Fig. 2:

The learning curves of loss functions and the SNR evaluation on both training data and testing data of EMDDL. The curve is formed by the average of 10 repetitions, and the gray shadow is formed by the standard deviation of 10 repetitions.

TABLE I:

The SNR on testing data of various methods.

		Multimodal based methods		Single modality based methods
Paradigm	Method	EMDDL	MDL	SDDL
	SNR
Emoid		3.2577	2.6416	0.9338
Nback		3.8650	3.2029	1.2219
Rest		4.8046	4.1108	1.0622

C. States Analysis of Multimodal Data

To find the differences in reoccurring patterns of dFC (i.e., states) between children and young adults, $k$ means clustering method with the cityblock distance metric is used to obtain the reoccurring patterns of each group in each modality [37]. Specifically, sparse representations of each group in each modality are clustered, and then states can be obtained by multiplying the dictionary and the cluster centroid. We use the elbow criterion defined as within-cluster sums of distances to estimate the optimal number of dFC states, and the optimal number of dFC states for Emoid, Nback, and Rest fMRIs are 5, 5, and 4 respectively. To test whether the clustering results are consistent in multiple subgroups, we use the kappa coefficient as the indicator [38], and the details can be seen in Supplementary material. The results indicate that the clustering results obtained from two different subgroups are substantial agreement or perfect agreement in a large probability. For Emoid task, the proportions of each state for children are 14.07%, 22.28%, 17.52%, 25.53%, and 20.61% respectively, while the proportions of each state for young adults are 9.08%, 21.06%, 14.55%, 27.29%, and 28.01% respectively. For Nback task, the proportions in these states for children are 11.14%, 19.19%, 24.23%, 22.97%, and 22.48% respectively, while the proportions in these states for young adults are 7.5%, 14.28%, 22.53%, 25.58%, and 30.1% respectively. For Rest fMRI, the proportions of these states for children are 31.14%, 29.35%, 34.15%, and 5.37% respectively, while the proportions of these states for young adults are 21.64%, 13.87%, 27.43%, and 37.05% respectively.

To further investigate the time occupied divergence of each state, dwell time (DT) and fraction of time (FT) are estimated from the state transition vector [7]. DT represents how long an individual spends in a given state on average, and FT is to describe the total time spent in a given state. For a subject $i$, DT and FT of $k$-th state are defined by

$$D{T}^{\text{state }\left(k\right)}=mean\left(T{R}_{end}-T{R}_{start}\right)$$

$$F{T}^{state\left(k\right)}=\frac{sum\left(state\_vecto{r}_{\left(i\right)}==k\right)}{Total\hspace{0.17em}number\hspace{0.17em}of\hspace{0.17em}window}$$

where $T{R}_{start}$ and $T{R}_{end}$ are computed by

$$T{R}_{start}=count\left(difference\left(state\_vecto{r}_{\left(i\right)},k\right)==1\right)$$

$$T{R}_{end}=count\left(difference\left(state\_vecto{r}_{\left(i\right)},k\right)==-1\right)$$

in which "1" and "−1" mean that the specific window of $i$-th subject belongs to a certain state $k$ or not; $state\_vecto{r}_{(i)}$ is the states of the $i$-th subject in all window. The reoccurring patterns and time occupied divergence of Emoid, Nback, and Rest fMRIs for children and young adults are shown in Figures 3–5, where *, **, ***, **** and ***** denote the significant level 0.05, 0.01, 0.001, 0.0001, and 0.00001 respectively. In Figures 3–5 B, the dots represent DT values or FT values in each state of all subjects, and the blocks represent the box plot (i.e., the central mark indicates the median, and the bottom and top edges of the box indicate the 25 th and 75 th percentiles, respectively) of DT values or FT values in each state of all subjects. Moreover, we visualize the top 100 significant functional connectivity related to age in each state (i.e., the functional connectivity corresponding to the 100 smallest FDR-corrected $p$-values of two-sample $t$-test performed across subjecťs mean dFC by state) under Emoid, Nback, and Rest, which are shown in Figures 2–4 of the Supplementary material.

Fig. 3:

A. The states for the child groups (top) and young adult groups (bottom) during Emoid task. B. (a) and (b) are the values as well as box plot of DT and FT in each state for children and young adults during Emoid task respectively. (c) and (d) are the mean of DT and FT of all subjects for children and young adults in each state during Emoid task respectively.

Fig. 5:

A. The states for the child groups (top) and young adult groups (bottom) during rest. B. (a) and (b) are the values as well as box plot of DT and FT in each state for children and young adults during rest respectively. (c) and (d) are the mean of DT and FT of all subjects for children and young adults in each state during rest respectively.

To study the changes in reoccurring patterns over time under two tasks and resting state, we define the transition probabilities $P_{ij}$ from time $t$ to time $t+1$ as follows

$$P_{ij}=\frac{sum{\left\{I_{\left(S_t^n=i,S_{t+1}^n=j\right)}==1\right\}}_{n=1}^N}{sum{\left\{I_{\left(S_t^n=i\right)}==1\right\}}_{n=1}^N}i,j=1,2,\cdots ,s$$

where $S^n=\left(S_1^n,S_2^n,\cdots ,S_T^n\right)\in {ℝ}^{1\times T}$ is the state vector for $n$-th subject, and $S_t^n=i$ for $i=1,2,\cdots ,s$ ($s$ is 5,5 , and 4 for Emoid, Nback, and Rest respectively) represents that the $n$-th subject is in state $i$ at time $t$. $I_{\left(\cdot \right)}$ is an indicative function, which is 1 when the condition is true, otherwise it is 0 . The probability of each state at the initial time is defined as

$$P_i=\frac{sum{\left\{I_{\left(S_1^n=i\right)}==1\right\}}_{n=1}^N}{N}i=1,2,\cdots ,s$$

Specifically, for a given state $i_t$ at time $t$, we can calculate the transition probabilities $P_{i_{t}j}$ for $j=1,2,\cdots ,s$ from time $t$ to time $t+1$. Then we record the maximum transition probability and the corresponding state at time $t+1$, and denote them as $P_{i_t{i}_{t+1}}$ and $i_{t+1}$ respectively. By repeating the above steps, we can obtain the state transition curve with maximum state transition probability, which is shown in Figures 5–7 A of the Supplementary material. To further explore how the strength of functional connectivity changes over time, we count the proportion of enhancement and decrease of functional connections within or between functional networks during state transition, which is shown in Figures 5–7 B of the Supplementary material. To contrast the functional connectivity matrices between two adjacent states at the state transition point, we visualized the differences of the functional connectivity matrices between two adjacent states at the state transition point for each group under Emoid task and Nback task, which are shown in Figure 8 of the Supplementary material.

V. Discussion

A. Developmental Differences

1). The Common Developmental Differences of Three fMRI Paradigms:

Figures 3–5 A show the reoccurring patterns of three paradigms for both children and young adults. For the child group, we found that states 1, 2, and 3 in the resting state are similar to the Emoid states 2, 3, and 4 (Pearson correlation coefficient is 0.9687, 0.9631, and 0.9631 respectively) and the Nback states 5, 2, and 3 (Pearson correlation coefficient is 0.9744, 0.9856, and 0.9602 respectively). The analogous conclusions also can be found in the young adult group, where all reoccurring patterns in resting state are similar to Emoid states 2, 3, 4, and 5 (Pearson correlation coefficient is 0.9379, 0.9561, 0.9191 and 0.9588 respectively) and Nback states 3, 2, 5, and 4 (Pearson correlation coefficient is 0.9552, 0.9612, 0.9566, and 0.9696 respectively). It indicates that the reoccurring patterns of three paradigms are similar for a subject. The same conclusion also can be found in previous research [39], which reveals that no matter in resting state or task, the basic structure of the brain functional network remains relatively consistent. The finding testifies that the brain has a shared functional architecture during resting and many directed tasks, and the shared functional architecture of the brain can only modulate the connectivity pattern in response to task demands. In other words, the overlapping functional connectivity patterns between Rest fMRI and two task fMRIs suggest a shared functional architecture underlying and even shaping brain function, and a potential explanation of overlap is that the functional connectivity during resting constrains the activation of brain regions in response to task demands [40].

Although the brain shares the basic functional architecture during task and resting state, the basic functional organization between children and young adults are different. The number of within or between functional networks that children exist high-strength functional connections is 43 in state 1 and 2 in state 3 under Emoid task, 55 in state 1 and 10 in state 2 under Nback task, and 13 in state 2 under resting state. The number of within or between functional networks that young adults exist high-strength functional connections is 9 in state 1 under Emoid task, 20 in state 1 under Nback task, and 2 in state 2 under resting state. For all three fMRI paradigms, we found that children have many high-strength functional connections distributed widely among 13 functional networks, young adults have high-strength functional connections only within and between some functional networks. It is consistent with the previous studies that children show more diffuse functional connectivity patterns while young adults show more focused functional connectivity patterns, and the changes in functional connectivity patterns between children and young adults explain how brain function changes from an undifferentiated system to a specialized system as one grows up [3], [4]. The brain organization of distinct and stronger within-network communication can promote precise modulation efficiently because it can transfer more information in a short time [41]. Thus, compared with children with more diffuse functional connectivity patterns, the brain organization of young adults with more focused functional connectivity patterns can transmit information more efficiently and facilitate precise modulation during resting and two tasks.

Additionally, the functional connectivity among DMN, SC<, MRN, CN, AN, FPTCN, and SN is decreased in most reoccurring patterns for Emoid, Nback, and Rest fMRIs during development. DMN, so-called task-negative network, is broadly inactivated across tasks, which are closely related to numer- ous key brain functions such as integration of autobiographic information, self-monitoring, and social cognition [28]. It is reported that the functional activity in DMN never stops but regulates during the resting state [42]. SCN participates in memory, attention, perception, and consciousness, and dominates the motivation and emotion state independent control of cortical functions [43]. MRN is reported to be engaged during autobiographical memory retrieval that involves strategic search processes guided by self-knowledge and current goals, memory recovery associated with a rich sense of reexperience, monitoring, and other control processes [44]. CN is not just considered as the domain of motor control that receives information from widespread regions to affect the generation and control of movement, but also is thought to be involved in cognition and visuospatial reasoning [45]. AN innervated by autonomic nerves, involves activities related to sound information including collection, conduction, and processing [46], [47]. FPTCN involving working memory maintenance, predictive perceptual coding, and cognitive task, is thought to play an important part in mediating the allocation of attentional resources to compete for auditory information under varying degrees of perceptual demand [48]. SN is thought to regulate attention and behavior adaptively through the physical characteristics and the relevant information of the task, and also is considered to be a key interface for cognitive, homeostatic, motivational, and affective systems [49]. Both resting and task fMRIs suggest that the functions of the brain in processing information, working memory, and cognition are not mature in children compared with young adults [3].

The functional connectivity between SSN, COTCN, DAN, and some other functional networks is increased in some reoccurring patterns for resting and task fMRIs during development. SSN participates in the process of emotional feeling and cognitive activities [50]. COTCN is the key to coordinate information transmission and involves many complex cognitive tasks [51]. DAN controls external and attentiondemanding cognitive functions [52]. Three fMRI paradigms indicate that brain functions related to emotional feelings, cognition, and information transmission are still growing with age.

2). The Developmental Differences of Each fMRI Paradigm:

Figures 3–5 B show the time occupied divergence of children and young adults during task and resting state. Both children and young adults have lower DT and FT in each state for two tasks while having higher DT and FT in each state during rest. It indicates that subjects including children and young adults tend to switch frequently among states in tasks and prefer to stay in a particular state while resting. It reveals that the spontaneous functional activity is stable during resting state, and then the functional activity corresponding to task demands changes quickly when the participant is required to perform a task [53].

For Emoid fMRI, both children and young adults stay in states 2, 3, and 4 for about the same time, but children stay longer in state 1 while young adults stay longer in state 5 . Under the Emoid task, whether the initial state is 2, 3, 4, or 5, the children group will eventually switch to state 4 at time 9, and then they will switch back and forth between state 2 and state 4 . When the initial state is 1 , children group will stay in state 1 for the most time and then switch to state 3. No matter which the initial state is, the young adult group will eventually switch to state 5 at time 5 and stay at state 5 for a long while, and then they will switch to state 4 at time 18 . The result of the Emoid task indicates that children have more frequent state transitions between state 2 and state 4 , and the strength of functional connections within or between functional networks changes over time. Compared with children, the strength of functional connectivity within or between functional networks decreases at the early stage for young adults, and then they prefer to stay in state 5 .

For Nback fMRI, both children and young adults stay in states 2, 3, and 4 for about the same time, but children stay longer in state 1 while young adults stay longer in state 5 . Under the Nback task, no matter which the initial state is, the children group will eventually switch to state 4 at time 9, and then they will stay at state 4 until they switch to state 3 at time 18. The young adult group switch between state 4 and state 5 after time 7 in any initial state. The result of the Nback task indicates that the strength of functional connectivity for children changes over time during the frequent state transition at an early time, and then they will stay at state 4 for a while and finally switch to state 3. Unlike children, young adults prefer to stay for a while after switching to state 4 or state 5 , and the strength of functional connectivity within or between functional networks decreases first, then increases, and then decreases during state transition between state 4 and state 5.

For Rest fMRI, both children and young adults stay in state 3 for about the same time. Children stay longer in state 1 and state 2, whereas young adults prefer to stay in state 4 . Under the resting state, both children and young adults prefer to stay in a specified state with no change in the strength of functional connectivity within or between functional networks. We found that children prefer to switch among states with diffuse functional connectivity patterns during the two tasks and stay in states with diffuse functional connectivity patterns during rest. On the other hand, young adults switch among states with focused functional connectivity patterns in two task fMRIs and stay in states with focused functional connectivity patterns during rest.

For Emoid fMRI, along with the enhanced functional connectivity among SSN, COTCN, and DAN with age in states 4 and 5, the functional connectivity in the rest states declines to various degrees. For Nback fMRI, there exists enhanced functional connections within and between 13 functional networks in state 3 during development. Also, the functional connectivity decreases in the rest states with age. For Rest fMRI, in states 1,2, and 4, there are not only lessened functional connections which mainly exist among SCN, MRN, CN, DMN, AN, FPTCN, and SN, but also exist strengthen functional connections which are mainly among SSN, COTCN, and DAN. In state 3 of Rest fMRI, the functional connections within and between 13 functional networks enhance during development. We found that compared with children, the functional connectivity of young adults increases or reduces with time for resting fMRI while generally decreasing for the two tasks. It indicates that the changes of functional connectivity with age are more complex in resting, and the brain functions related to emotion and working memory are more mature and efficient during development [4], [41].

B. Future Work

In this paper, the functional connections of three fMRI paradigms were used for learning the modality-shared dictionary and modality-specific sparse representations. An interesting note is that the dynamic functional connections of multiple modalities from multiple subjects can be treated as a higher-order tensor by considering time dimension, subject dimension, or modality dimension. Higher-order tensor can maintain the structure relationship of different dimensions of data, which may be lost in a low dimensional form. Besides, by considering the brain regions as nodes of a graph, the functional connections can be viewed as the weight matrix of edges for a graph. Compared with regarding the functional connections as a feature vector, considering the functional connections as a graph feature can help to discover the structural relationships among brain regions. Hence, a meaningful future work is to incorporate the structural information of different dimensions of the data or the graph structural information of the brain into EMDDL, which may help to improve the model performance and capture the intrinsic topological structure of the brain.

VI. Conclusion

In this paper, we present an explainable multimodal deep dictionary learning method to capture the developmental differences between children and young adults from three fMRI paradigms. Specifically, the shared dictionary and the modality-specific sparse representations are learned based on the multimodal data and their encodings of the sDAE to simultaneously reveal the commonality and specificity of different paradigms. By applying the proposed method to the three fMRI paradigms from PNC, we found that children share a diffuse functional connectivity pattern while young adults share a focused functional connectivity pattern during both resting and two tasks. Three fMRI paradigms reveal that compared with children, young adults possess more mature and efficient functional networks for processing information. Children and young adults rarely transit from one state to other states during resting and prefer to switch among states over time during a task.

Fig. 4:

A. The states for the child groups (top) and young adult groups (bottom) during Nback task. B. (a) and (b) are the values as well as box plot of DT and FT in each state for children and young adults during Nback task respectively. (c) and (d) are the mean of DT and FT of all subjects for children and young adults in each state during Nback task respectively.

Appendix 1

A. The Comparison of Sparsity of Activations Among ${\text{L}}_1$-norm, ${\text{L}}_2$-norm, and Kullback-Leibler Divergence

Let $\rho$ be a small positive constant between 0 and 1 , and ${\rho }_j^{\left(l\right)}=\frac{1}{N}{\sum }_{n=1}^N{h}_{nj}^{\left(l\right)}$ with $h_{nj}^{\left(l\right)}$ being the $j$-th element of $h_n^{\left(l\right)}$ is the average activation of neural $j$ in the $l$-th layer, then the Kullback-Leibler divergence is defined as

$$KL\left(\rho \parallel {\rho }_j^{\left(l\right)}\right)=\rho \text{log}\frac{\rho }{{\rho }_j^{\left(l\right)}}+\left(1-\rho \right)\text{log}\frac{1-\rho }{1-{\rho }_j^{\left(l\right)}}$$

The penalty functions based on $L_1$-norm and $L_2$-norm are defined as

$$f_{L_1}={\displaystyle \sum }_{l=1}^{2L-1}{\displaystyle \sum }_{j=1}^{r^{\left(l\right)}}\left|{\rho }_j^{\left(l\right)}\right|$$

$$f_{L_2}={\displaystyle \sum }_{l=1}^{2L-1}{\displaystyle \sum }_{j=1}^{r^{\left(l\right)}}{\left({\rho }_j^{\left(l\right)}\right)}^2$$

The sparsity and SNR with $L_1$-norm, $L_2$-norm and KullbackLeibler divergence have been compared and the results are shown in Figure 9 of the Supplementary material. The results show that the sparsity of Kullback-Leibler divergence is better than that of $L_1$-norm in most hidden layers, and the sparsity of $L_2$-norm is the worst among the above three penalty functions. The SNR evaluation of EMDDL on the multimodal data in both the original space and the encoding space show that, EMDDL based on Kullback-Leibler divergence has better reconstruction ability compared with $L_1$-norm and $L_2$-norm.

B. The Proof of the Convexity of $\text{f}\left(\text{V}\right)$

The convexity of $f\left(V\right)$ depends on whether its Hessian matrix ${\nabla }^{2}f\left(V\right)$ is positive definite or not. Thus, as long as ${\nabla }^{2}f\left(V\right)$ is positive definite, the convexity of $f\left(V\right)$ can be guaranteed. The Hessian matrix ${\nabla }^{2}f\left(V\right)$ is

$${\nabla }^{2}f\left(V\right)=\frac{1}{N}\left(D^{\text{T}}D+D^{{(L)}^{\text{T}}}{D}^{\left(L\right)}\right)+S$$

According to the Weyl's inequality [54], we have

$$\begin{array}{l}{\lambda }_{min}\hspace{0.17em}\left({\nabla }^{2}f\left(V\right)\right)={\lambda }_{min}\left(\frac{1}{N}\left(D^{\text{T}}D+D^{{(L)}^{\text{T}}}{D}^{\left(L\right)}\right)+S\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\ge {\lambda }_{min}\left(\frac{1}{N}{D}^{\text{T}}D\right)+{\lambda }_{min}\left(\frac{1}{N}{D}^{{(L)}^{\text{T}}}{D}^{\left(L\right)}\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.33em}\hspace{0.33em}+{\lambda }_{min}\left({\lambda }_3{H}_2\right)+{\lambda }_{min}\left(-2{\lambda }_3{H}_1\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.33em}\hspace{0.33em}+{\lambda }_{min}\left({\lambda }_{4}L\right)+{\lambda }_{min}\left(({\lambda }_3+{\lambda }_5)I\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=\left.({\lambda }_3+{\lambda }_5\right)-2{\lambda }_3\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}={\lambda }_5-{\lambda }_3\end{array}$$

Where ${\lambda }_{min}\left(\cdot \right)$ denotes the smallest eigenvalue of a matrix. To ensure the positive definite of the Hessian matrix ${\nabla }^{2}f\left(V\right)$, ${\lambda }_{min}\left({\nabla }^{2}f\left(V\right)\right)$ should be greater than 0 . Thus, $f\left(V\right)$ is convex when ${\lambda }_3<{\lambda }_5$ holds.

C. The Lipschitz Constant of the Gradient $\nabla \text{V}$

For every $V^1,V^2\in {ℝ}^{K\times N}$, we have

$$\begin{gathered} {‖\nabla V^1-\nabla V^2‖}_2=\parallel \frac{1}{N}\left(D^{\text{T}}D+D^{{(L)}^{\text{T}}}{D}^{\left(L\right)}\right)\left(V^1-V^2\right) \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+\left(V^1-V^2\right)S{\parallel }_2 \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\le {‖\frac{1}{N}\left(D^{\text{T}}D+D^{{(L)}^{\text{T}}}{D}^{\left(L\right)}\right)‖}_2 \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{‖V^1-V^2‖}_2+{‖V^1-V^2‖}_2\parallel S{\parallel }_2 \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\le \left(\frac{1}{N}\left({‖D^{\text{T}}D‖}_2+{‖D^{{(L)}^{\text{T}}}{D}^{\left(L\right)}‖}_2\right)+\parallel S{\parallel }_2\right) \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{‖V^1-V^2‖}_2 \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=\left(\frac{{\lambda }_{max}\left(D^{\text{T}}D\right)+{\lambda }_{max}\left(D^{{(L)}^{\text{T}}}{D}^{\left(L\right)}\right)}{N}\right. \\ \left.\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+\sqrt{{\lambda }_{max}\left(S^{\text{T}}S\right)}\right){‖V^1-V^2‖}_2 \end{gathered}$$

Thus, the Lipschitz constant of the gradient $\nabla V$ is

$$\begin{gathered} L_f=\frac{1}{N}\left({\lambda }_{max}\left(D^{\text{T}}D\right)+{\lambda }_{max}\left(D^{{(L)}^{\text{T}}}{D}^{\left(L\right)}\right)\right) \\ \hspace{1em}\hspace{1em}+\sqrt{{\lambda }_{max}\left(S^{\text{T}}S\right)} \end{gathered}$$ (13)

where ${\lambda }_{max}\left(\cdot \right)$ denotes the largest eigenvalue of a matrix.

D. The Details of Obtaining Multimodal Data

There are 264 BOLD signals with $T_m$ time points for the $m$-th modality of the $n$-th subject. $f{c}_{nk}^{\left(m\right)}\left(i,j\right)$, the functional connectivity between the $i$-th ROI and the $j$-th ROI within the $k$-th window for the $m$-th modality of the $n$-th subject, is calculated based on the Pearson correlation coefficient, which is defined as follows

$$\frac{f{c}_{nk}^{\left(m\right)}\left(i,j\right)={\displaystyle \sum }_{t=1}^{w_l}\left(B_{nk}^{\left(m\right)}\left(i,t\right)-{\overline{B}}_{nk}^{\left(m\right)}\left(i\right)\right)\left(B_{nk}^{\left(m\right)}\left(j,t\right)-{\overline{B}}_{nk}^{\left(m\right)}\left(j\right)\right)}{\sqrt{{\displaystyle \sum }_{t=1}^{w_l}{\left(B_{nk}^{\left(m\right)}\left(i,t\right)-{\overline{B}}_{nk}^{\left(m\right)}\left(i\right)\right)}^2}\sqrt{{\displaystyle \sum }_{t=1}^{w_l}{\left(B_{nk}^{\left(m\right)}\left(j,t\right)-{\overline{B}}_{nk}^{\left(m\right)}\left(j\right)\right)}^2}}$$

where $B_{nk}^{\left(m\right)}\left(i,t\right)$ is the $t$-th BOLD signal value of the $i$-th ROI within the $k$-th window for the $m$-th modality of the $n$-th subject. ${\overline{B}}_{nk}^{\left(m\right)}\left(i\right)=\frac{1}{w_l}{\sum }_{t=1}^{w_l}{B}_{nk}^{\left(m\right)}\left(i,t\right)$ is the sample mean of the BOLD signals of the $i$-th ROI within the $k$ th window for the $m$-th modality of the $n$-th subject. By calculating the functional connectivity between any two ROIs within the $k$-th window for the $m$-th modality of the $n$-th subject, $C_{264}^2=34716$ functional connections can be obtained within the $k$-th window for the $m$-th modality of the $n$-th subject. For a BOLD signals with $T$ time points, we can obtain $S_l=\frac{T-w_l}{s_l}+1$ windows with window length $w_l$ and scan length $s_l$. Thus, a dynamic functional connection matrix $f{c}_n^{\left(m\right)}\in {ℝ}^{34716\times S_l}$ can be obtained for the $m$-th modality of the $n$-th subject. Let $X_{\left(m\right)}=\left(f{c}_1^{\left(m\right)},f{c}_2^{\left(m\right)},\cdots ,f{c}_{N_s}^{\left(m\right)}\right)\in {ℝ}^{p\times N_m}$ be the data of the $m$-th modality, and multimodal data $X=\left(X_{\left(1\right)},X_{\left(2\right)},\cdots ,X_{\left(M\right)}\right)\in {ℝ}^{p\times N}$ is composed of $M$ modalities. In which, $p=C_{264}^2=34716$, $N_m=S_l\times N_s$ with $N_s$ being the number of subjects, and $N={\sum }_{m=1}^M{N}_m$. The flowchart of calculating $f{c}_n^{\left(m\right)}$ is shown in Figure 10 of the Supplementary material.