Explainable Contrastive Multiview Graph Representation of Brain, Mind, and Behavior

Abstract

Understanding the intrinsic patterns of human brain is important to make inferences about the mind and brain-behavior association. Electrophysiological methods (i.e. MEG/EEG) provide direct measures of neural activity without the effect of vascular confounds. The blood oxygenated level-dependent (BOLD) signal of functional MRI (fMRI) reveals the spatial and temporal brain activity across different brain regions. However, it is unclear how to associate the high temporal resolution Electrophysiological measures with high spatial resolution fMRI signals. Here, we present a novel interpretable model for coupling the structure and function activity of brain based on heterogeneous contrastive graph representation. The proposed method is able to link manifest variables of the brain (i.e. MEG, MRI, fMRI and behavior performance) and quantify the intrinsic coupling strength of different modal signals. The proposed method learns the heterogeneous node and graph representations by contrasting the structural and temporal views through the mind to multimodal brain data. The first experiment with 1200 subjects from Human connectome Project (HCP) shows that the proposed method outperforms the existing approaches in predicting individual gender and enabling the location of the importance of brain regions with sex difference. The second experiment associates the structure and temporal views between the low-level sensory regions and high-level cognitive ones. The experimental results demonstrate that the dependence of structural and temporal views varied spatially through different modal variants. The proposed method enables the heterogeneous biomarkers explanation for different brain measurements.

1. Introduction

The brain activity remains latent construct that could not be directly measured with present technologies [18]. Non-invasive electrophysiology such as Magneto- and electo- phencephalography (M/EEG) shows insights into many healthy and diseased brain activity at millisecond but lacks spatial resolution. Additional modalities with higher spatial resolution at millimeter such as Magnetic resonance imaging (MRI), functional MRI(fMRI) and positron emission tomography (PET) have paved the way to human connectomics in clinical practice. However, this kind of technique is sluggish to reveal neuronal activity. The challenge of fusing non-invasive brain measurements is that different technique provides either high spatial or temporal resolution but not both [3,11,21]. The development of fMRI, MEG and MRI made it possible to obtain the system level of structural connectomics and functional connectomics. Many methods have been proposed to associate the connectomics from different modalities. Simple and direct correlational approaches have been commonly used to link SC and FC [9,10,22]. With the prior of SC, dynamic casual model could explain the functional signals in terms of excitatory and inhibitory interactions [5,17]. Graph models allow the extraction of system level connectivity properties associated with brain changes in the life cycle, such as attention and control networks related to late adolescence and aging process, the strength and organization of function connectivity related to neurological diseases and intrinsic brain activity of behavior performance during resting and task state [1,2,4]. Recently, graph harmonic analysis with Laplacian embedding and spectral clustering have been utilized for revealing brain organization [16]. Basically, the graph harmonic model use harmonic components to summarize the spatial patterns with the nodes of the graph. With the structurally informed components, the relationship among structural connectivity, functional connectivity and behavior performance could be decomposed.

Literature on previous graph based methods that utilizes graph theoretical metrics to summarize the function connectivity ignores the high-order interactions between ROIs [12,14,19]. The existing methods are not very suitable for the integration of structural connectivity, functional connectivity and behavior performance for the following reasons:

Lack of Individual and Group-Level Explanation:

existing methods especially for fMRI analysis assume that the nodes in the same brain graphs are translation invariant. Ignoring the correspondence of nodes of different brain ROIs limits the explanation in individual and group level.

Incomplete and Missing Data in Clinical Data Collection:

due to the scanner availability and patient demands, it is impossible to do multimodal assessment for all patients. Incomplete or missing data hinders the potential of multimodal usage. There are very few databases that provides public access to MEG, MRI and fMRI of the same subjects.

Violation of the Brain Dynamics Information:

the existing joint model uses the linearity assumption among latent variables from different modal measurements. The effective usage includes subject specific integration (structural connectivity), modal specific association (i.e. fMRI and MEG). However, the brain is highly dynamic and the linearity assumption is not applicable in many cases.

In the paper, we build a novel heterogeneous contrast subgraphs representation learning based method to exploit the coupling of structural and functional connectivity from different brain modality. The proposed method has the following advantages: 1) The proposed heterogeneous graph representation learning method utilizes the contrastive learning to explore the coupling information of different modality. The proposed method with the semantic attention model enables the complex and dynamics link between structural and functional connectivity within each modal measures. The proposed method is capable of modeling heterogeneous spatio-temporal dynamics and learn the contrast graph structures simultaneously.

2) The proposed method uses a causal explanation model to improve the individual and group-level interpretability. The explanation approach helps to locate the significant brain region with sex difference and neurodevelopment. The experimental results for gender classification with 1200 subjects from HCP have shown the performance of the proposed method with incomplete multimodal data (fMRI and MEG).

3) The proposed method utilizes graph convolution theory to link the brain structure and function. The experimental results with meta-analysis reveal the strength of structural-functional coupling patterns among functional connectivity, structural connectivity and behavior performance.

2. Problem Formulation

To associate heterogeneous multimodal brain measurements, a heterogeneous graph representation learning with semantic attention is introduced based on fMRI and filtered MEG data. Next we introduce dynamical neural graph encoder framework to associate the spatial and temporal patterns from structural and functional connectivity of multimodal brain measurements. Then, we give the details of the multi-view contrastive graph representation learning. The contrastive graph learning method makes sure the maximization of mutual information of the node representation from one view and the graph representation from another view. Finally, we discuss the interpretable causal explanations for the proposed method on graph. The overall framework is illustrated in Fig. 1.

Fig. 1.

Schematic illustration of the explainable contrastive graph representation with heterogeneous brain measurements. Top: training process, bottom: explanation process.

Heterogeneous Graph Representation Learning.

We use a graph ${𝒢}_i=\left(𝒱,{ℰ}_i\right)$ to represent the heterogeneous graph representation, with the node type $𝒱=\left[v_{t,i}\mid t=1,\dots ,T;i=1,\dots ,N\right]$ with $N$ brain ROIs and $T$ time points. The edge mapping $ℰ$ represents the connection of different brain ROIs in spatial and temporal domain. The aim of contrastive graph representation is to explore the spatial and temporal pattern of the fMRI and MEG data. Given two multimodal graph ${𝒢}_{𝒜}$ and ${𝒢}_{ℬ}$ with different time points $T_{𝒜}$ and $T_{ℬ}$ and their correspondence multivariate value $X_{𝒜}=\left[x_{t_1}^{𝒜},x_{t_2}^{𝒜},\dots ,x_{t_{𝒜}}^{𝒜}\right]$ and $X_{ℬ}=\left[x_{t_1}^{ℬ},x_{t_2}^{ℬ},\dots ,x_{t_{ℬ}}^{ℬ}\right]$. To integrate manifest variables of the brain with different spatial and temporal resolution, a dynamical neural graph encoder is proposed to explore the spatiotemporal dynamics. The data augmentation mechanism introduces a ${\mathrm{\Phi }}_{i_{i=1}}^P$ of $P$ set of multiview heterogeneous graph for each modality, where $P$ represents the total view of the brain measurement after data augmentation (i.e. the number of filter bands for MEG). We could define its corresponding adjacency matrices as ${\left[A_{{\Phi }_i}\right]}_{i=1}^P$. The node representation within each view is defined as $Z_{{\Phi }_i}^{𝒜}$ and $Z_{{\Phi }_i}^{ℬ}$. Then we use the heterogeneous latent attention module to aggregate the graph representation for each modal measurement $H^{𝒜}=f_{\psi }\left(L_{att}\left(Z_{{\Phi }_1}^{𝒜},Z_{{\Phi }_2}^{𝒜},\dots ,Z_{{\Phi }_P}^{𝒜}\right)\right)$.

2.1. Dynamical Neural Graph Encoder

We define the brain dynamical state with $N$ neurons as $\dot{z}(t)=f(z,l,t)$, where $z(t)={\left[z_1(t),z_2(t),\dots ,z_N(t)\right]}^T$ represents the internal states of $N$ neuron nodes at time $t$. $f(\cdot )$ denotes the nonlinear dynamical function of each node. And $l(t)={\left[l_1(t),l_2(t),\dots ,l_S(t)\right]}^T$ represents the external stimuli for $S$ neurons.

Within each single modality, we define a continuous neural-graph differential equation as follows,

$$\dot{Z}(t)=f_{G_{t_k}}\left(t,Z_t,{\theta }_t\right)\mathit{and}{Z}_t^{+}={ℒ}_{G_{t_k}}^j\left(Z_t,X_t\right)$$ (1)

where $f_G,{ℒ}_G^j$ are graph encoder networks. $Z_t^{+}$ is introduced to represent the value after discrete operation. $Z_t^{+}$ could represent the state ’jump’ for brain measurements such as task fMRI.

Heterogeneous Graph Output with Semantic Attention.

Within each brain modal measurement, we could obtain the a set of heterogeneous representation ${\left[Z_{{\Phi }_i}\right]}_{i=1}^P$. Then, we use a semantic level attention layer to associate the cross view heterogeneous graph representation $L_{\mathit{att}}$ with the learned weights ${\beta }_{{\Phi }_1},{\beta }_{{\Phi }_2},\dots ,{\beta }_{{\Phi }_p}=\mathit{Latt}\left(Z_{{\Phi }_1},Z_{{\Phi }_2},\dots ,Z_{{\Phi }_p}\right)$.

We define the importance of each view graph representation as,

$$e_{{\Phi }_i}=\frac{1}{N}\sum _{n=1}^N\mathit{tanh}({\overrightarrow{q}}^T\cdot \left[W_{\mathit{sem}}\cdot {\overrightarrow{Z}}_{n,{\mathit{\Phi }}_i}+\overrightarrow{b}\right])\mathit{and}{\beta }_{{\mathit{\Phi }}_i}=\mathit{softmax}(e_{{\mathit{\Phi }}_i}),$$ (2)

where $W_{\mathit{sem}}$ denotes the linear transformation and $\overrightarrow{q}$ is learnable. The heterogeneous node representation could be denoted by $H=f_{\psi }\left({\sum }_{i=1}^p{\beta }_{{\Phi }_i}{Z}_{{\Phi }_i}\right)$.

Next, we apply the readout function to the aggregated heterogeneous representation of each view with the shared projection head $f_{\varphi }(.)\in R^{d_h}$. In the experiment, we use an MLP with two hidden layers as the projection head. We get the projected representations ${\overrightarrow{h}}_g^{𝒜}$ and ${\overrightarrow{h}}_g^{ℬ}$. For each view, the node representation are concatenated as ${\overrightarrow{h}}_g=\sigma \left({\Vert }_{l=1}^L\left[{\sum }_{i=1}^n{\overrightarrow{h}}_i^l\right]W\right)$.

2.2. Mutual Information Based Training Process

In the training process, we maximize the mutual information between the node representation of one modal and the graph representation of another modal, i.e. the node representation of fMRI and the graph representaion of MEG and vice versa. The objective is defined with contrastive learning as follows,

$$\underset{\theta ,\varphi ,\psi }{\mathit{max}}\frac{1}{|G|}\sum _{g\in G}\frac{1}{|g|}\sum _{i=1}^{|g|}[MI({\overrightarrow{h}}_i^{𝒜},{\overrightarrow{h}}_g^{ℬ})+MI({\overrightarrow{h}}_g^{𝒜},{\overrightarrow{h}}_i^{ℬ})],$$ (3)

where $\theta ,\varphi ,\psi$ represent the parameters of heterogeneous dynamical neural graph encoder and projection head. $|G|$ denotes the total numbers of graph. $|g|$ is the number of nodes. MI is denoted as the dot production $MI\left({\overrightarrow{h}}_i^{𝒜},{\overrightarrow{h}}_g^{ℬ}\right)={\overrightarrow{h}}_i^{𝒜}\cdot {\left({\overrightarrow{h}}_g^{ℬ}\right)}^T$.

2.3. Explainable Causal Representation on Graphs

To highlight the importance of the brain ROIs, we introduce the explainable causal representation to encourage the reasonable node selection process. We train an explanation model to explain the multmodal graph representation approach based on granger causality. The explanation process are divided into two steps, the distillation process and explainer training process.

In distillation process, we use a subgraph $G_s$ to represent the main cause of the target prediction $y$. The explainable causal representation does not require the re-training of the dynamical graph encoder which could lower the computation complexity. We use ${\delta }_{G\setminus e_j}$ to represent the prediction error exclude the edge $e_j$. The model error is defined as ${\Delta }_{\delta ,e_j}={\delta }_{G\setminus e_j}-{\delta }_G$.

With the ground truth label $y$, we define the model error as the loss difference $ℒ\left(y,{\hat{y}}_G\right)$.

$${\delta }_G=ℒ\left(y,{\hat{y}}_G\right)\mathit{and}{\delta }_{G\setminus e_j}=ℒ\left(y,{\hat{y}}_{G\setminus e_j}\right),$$ (4)

Given the causal contributions of each edge, we could sort the top-$K$ most relevant edges for model explanation. After the model distillation process, we will train a new explainer model based on graph convolutional layer.

$$Z=GCN(A,Z)\mathit{and}\hat{A}=\sigma \left(Z{Z}^T\right)$$ (5)

where each value in $A$ represent the contribution of specific edge to prediction. The explanation model generate $\hat{A}$ as an explanation mask. We show more details of the explainable causal model in Supplementary material.

3. Experiments

In the experiments, we use the public available s1200 dataset from Human Connectome Project (HCP), which contains 1096 young adults. Additionally, about 95 subjects have resting-state and/or task MEG (tMEG) data. The resting-state fMRI is pre-processed following the minimal preprocessing pipeline [8]. Then the pre-processed data is registered into a standard cortical surface using MSMAll [8]. The cortical surface was parcellated into $\mathrm{N}=22$ major ROIs [7]. In addition, the averaged time course of each ROI is normalized using z-score. The restingstate MEG has been pre-processed using ICA to remove out artefacts related to head and eye movement. Sensor-space data were down-sampled 300 Hz using anti-aliasing filter. Next the MEG data were source-reconstructed with a scalar beamformer and registered into the standard space of the Montreal Neuroimaging Institute (MNI). Data were then filtered into 1–30 Hz and beamformed onto 6mm grid. The parcellation atlas and z-score normalization method of MEG are similar to resting-state fMRI.

3.1. Sex Classification

We first test the performance of the proposed method with sex classification task using HCP data. We adopt 5-fold cross validation on the 1091 subjects. We compare the proposed method with several state-of-the-art methods such as Long-Short-Term Memory (LSTM) [13], graph convolution LSTM (GC-LSTM) [20] and spatio-temporal graph convolution network (ST-GCN) [6]. The hidden state of LSTM was set to 256. A simple Multi-Layer Perception (MLP) with 2 hidden layers and ReLU activation is also included as the baseline method. For the proposed method, we report two kinds of sex classification accuracy. The first single model uses only the fMRI to explore the dynamics within the brain ROIs. The multimodal based method integrates both fMRI and MEG to exploit the spatio-temporal dynamics and achieves better sex classification performance.

The accuracy of sex classification is shown in Table 1. Comparing with the baseline method, the proposed method could learn the dynamic contrast graph representation between fMRI and MEG. The proposed method could take advantage of the high spatial resolution of fMRI and high temporal resolution of MEG to achieve the highest sex classification performance of 85.2%. The importance of the brain regions that contributes to the sex classification is show in Fig. 2. The causal explanation module provides us a new way to find individual-level and group-level biomarkers for sex difference.

Table 1.

Sex classification accuracy with different baseline models

Method	Accuracy
LSTM	0.808(0.033)
GC-LSTM	0.811(0.075)
MLP	0.770(0.051)
ST-GCN	0.839(0.044)
Proposed method with only fMRI	0.827(0.061)
Proposed method with multimodal	0.852(0.046)

Fig. 2.

Top $K$ important brain regions for sex classification

3.2. Brain Activity Decomposed with Functional and Structural Connectivity

In the second experiment, we use the structural-decoupling index which reveals the function and structure relationship to measure the energies of high pass decoupled activity versus low pass coupled activity per brain ROIs. The average structural-decoupling index for surrogate (with or without SC) and function signals is shown in Fig. 3. Without the SC prior knowledge, the surrogate shows significant decoupling patterns. While the knowledge of SC increases the coupling pattern in functional signals. Compared with the functional time courses, the high-level cognition network detaches from the SC. We also use the NeuroSynth meta-analysis on the same topic in [15] to assess the structural-decoupling index. As shown in Fig. 4, the structural-decoupling index associates the behaviorally relevant gradient based on FC data. We could find a macroscale gradient of regions related to low to high level cognition with the learned graph representation. Due to the fact that the functional connectivity comes from MEG data in the second experiment, we compare the gradient learned from the graph representation with the original MEG data. For example, the terms related to acting and perceiving such as “visual perception”, “multisensory processing”, “reading” and “motor / eye movement” are grouped into the top end. The terms related to complex cognition such as “autobiographical memory”, “emotion” and “reward-based decision making” are characterized into the other end. Similar organization phenomenon could be found in the previous research [15,16]. However, the gradient learned by original MEG data lacks the pattern of system organization.

Fig. 3.

Structural decoupling index shows brain activity between function and structure. Left: Surrogate brain activity without structural connectome. Middle: surrogate brain activity with structural connectome. Right: brain activity with decoupling difference to the surrogate

Fig. 4.

Behaviorally relevant gradient shows brain organization with Structural decoupling index. Left: the learned graph representation. Right: original MEG data

4. Conclusions

Brain activity is shaped by the anatomical structure. In the paper, we propose an explainable contrastive graph representation learning based model to associate heterogeneous brain measurements such as MRI, functional MRI, MEG and behavior performance. The framework allows key advantages to concentrate brain, mind and behavior in cognitive neuroscience. The proposed method outperforms the state-of-the-art methods in gender classification using fMRI and MEG data. Moreover, the framework could localize the important brain region with sex difference through a causal explanation model. The second experiment with meta analysis demonstrates that the structure-function coupling pattern with the learned contrast graph representation. Future work that links the function connectivity with other modal data (i.e. gene expression and microstructure properties) could be easily adapted to our framework.