Predicting Spatio-Temporal Human Brain Response Using fMRI

Abstract

The transformation and transmission of brain stimuli reflect the dynamical brain activity in space and time. Compared with functional magnetic resonance imaging (fMRI), magneto- or electroencephalography (M/EEG) fast couples to the neural activity through generated magnetic fields. However, the MEG signal is inhomogeneous throughout the whole brain, which is affected by the signal-to-noise ratio, the sensors’ location and distance. Current non-invasive neuroimaging modalities such as fMRI and M/EEG excel high resolution in space or time but not in both. To solve the main limitations of current technique for brain activity recording, we propose a novel recurrent memory optimization approach to predict the internal behavioral states in space and time. The proposed method uses Optimal Polynomial Projections to capture the long temporal history with robust online compression. The training process takes the pairs of fMRI and MEG data as inputs and predicts the recurrent brain states through the Siamese network. In the testing process, the framework only uses fMRI data to generate the corresponding neural response in space and time. The experimental results with Human connectome project (HCP) show that the predicted signal could reflect the neural activity with high spatial resolution as fMRI and high temporal resolution as MEG signal. The experimental results demonstrate for the first time that the proposed method is able to predict the brain response in both milliseconds and millimeters using only fMRI signal.

1. Introduction

In computational neuroscience, brain state often refers to wakefulness, sleep, and anesthesia. However, the precise and dynamical brain complexity is still missing. Dynamical neural representation is thought to arise from neural firing, which do not fire in isolation [11]. The development of modern brain measuring techniques makes it possible to infer the large-scale structural and functional connectivity and characterize the anatomical and functional patterns in human cortex. Electrophysiological methods provide a direct and non-invasive way to record brain with milliseconds temporal resolution which is not affected by the problems commonly caused by intermediate processes. fMRI uses brain activityrelated blood-oxygen-level-dependent (BOLD) to understand the neural representations in millimeter level, however it is too sluggish in the time domain. None of the current non-invasive brain recording techniques could measure the high resolution dynamics in space and time. This is a major challenge in neuroscience which draws attention of researchers to develop simultaneous simulation models of brain dynamics.

Since each brain region interacts with other regions, it is hard to measure the intraregional brain dynamics with single technique such as fMRI or MEG. With the increase of the strength of brain region interactions, the measured brain dynamical signals decrease. Recent study on complex biological system provides the powerful mathematical functions to model the coupling structural and functional brain activity using multimodal brain measurements [19]. Previous study use principle component analysis (PCA) [3, 12, 13] or linear models [9, 15] to quantify the inherent neural representations in either high dimensional or low dimensional. The high dimensionality is used for information encoding. The low dimensionality is used to encode information in complex cognitive or motor tasks [8]. Modern methods such as representational similarity analysis (RSA) [7] and multivoxel pattern analysis (MVPA) [10] is used to quantify the brain activity across task conditions. Further developments utilize the states defined as the transmission between different task activity to function higher cognitive dynamics [2, 3, 14, 16, 18].

The question remains: how to model the change and transmission between neurons with high spatial and temporal resolution? In this paper, we propose a novel method that use fMRI to predict the corresponding neural activity with high resolution in space and time. Our main contributions are summarized as follows:

1. The current non-invasive brain measuring technique could not identify the neural activity with high resolution in both spatial and temporal domain. MEG/EEG could resolve the brain activity in milliseconds but the spatial resolution is low. In contrast, fMRI could represent the neural dynamics in millimeters, but it is too slow for the rapid transmission between cortical sites. To link the mapping between brain regions and time points, we propose a general framework to project the time points onto polynomial basis. In the training step, the proposed method takes the ROI parcellated brain signals (fMRI and MEG) to learn the dynamical representation. In the testing step, the proposed method is the first to use original 4D fMRI signal to predict the internal brain response in space and time.

2. Most of the existing recurrent neural network (RNN) suffers from the vanishing gradient problem when dealing with long-term time series. It is challenging to represent the brain response at milliseconds with current RNN model. The proposed method could discretize and project the time points onto the polynomial basis, which is able to deal with any time length data using the memory compression.

3. The two experimental results demonstrate that the proposed method is able to resolve the neural activity in space and time. Its feasibility is validated when compared with fMRI in structural domain using brain networks of independent component analysis (ICA). The predicted results also show the similar pattern to MEG signal in temporal domain.

The proposed method is related to the fMRI and MEG data fusion research. The difference between the proposed method and existing data fusion method is summarized as follows. 1) The proposed method does not use the MEG data in the testing step. It is able to use 4D fMRI data to predict the brain internal states in space and time. The MEG data is used in the training step as a reference for the proposed method to learn the temporal pattern of brain sates. The research problem could be considered as the “super-resolution” in the temporal domain of fMRI data and keeps the structural details. However, as far as we know, most of the previous methods require both fMRI and MEG data to predict the internal states. It is hard to obtain simultaneous fMRI and MEG data at the same time. 2) Secondly, the proposed method could predict the high resolution temporal signals for each voxel of fMRI data. In the testing step, the proposed method does not require the preprocessing steps like ROI average or beamforming which could introduce noise in the preprocessed data. As far as we know, the proposed method is the first to predict the spatio-temporal brain internal states for each voxel using only fMRI data.

2. Spatio-Temporal Dynamical Modeling of Brain Response

Our approach benefits from the framework of Legendre Memory Unit (LMU) and online function approximation to learn the memory representation [6, 17]. The dynamical brain network with $N$ neurons could be modeled as

$$\begin{gathered} \dot{z}(t)=f(z,l,t)+e_1(t), \\ y(t)=O(z(t))+e_2(t), \end{gathered}$$ (1)

where $z(t)={\left[z_1\left(t\right),z_2\left(t\right),\dots ,z_N\left(t\right)\right]}^T$ represents the internal states of $N$ neuron nodes at time $t$. $f(\cdot )$ denotes the nonlinear dynamical function of each node. And $l(t)={\left[l_1\left(t\right),l_2\left(t\right),\dots ,l_S\left(t\right)\right]}^T$ represents the external stimuli for $S$ neurons. $y(t)={\left[y_1\left(t\right),y_2\left(t\right),\dots ,y_N\left(t\right)\right]}^T\in R^N$ is the observed time series of brain measurements (i.e. MEG or fMRI signals). $e_1\left(t\right)$ and $e_2\left(t\right)$ are time series noise of the internal states and measurements. $O(\cdot )$ is the output function which controls the output of the brain measurements at time $t$. The nonlinear brain function $f(\cdot )$ is modeled using recurrent neural network (RNN).

The whole framework of the proposed method is shown in Fig. 1. In the training stage, the proposed method consists of two parts, the RNN model with Polynomial Projection and the Siamese network to score the agreement between the network prediction and the original time series of brain activity. Both Region-of-interest (ROI) extracted fMRI and MEG signals are used to train the network. However, in the testing stage, only fMRI signal is included to predict the internal states in space and time. The experiments demonstrate that the proposed method could deal with both the ROI extract fMRI signal and the original 4D fMRI image. The predicted internal behavioral states show high resolution in spatial and temporal domain.

Fig. 1.

Illustration of the proposed spatio-temporal brain dynamical network (a) Training framework. Both ROI extracted fMRI and MEG signals are used in the training process. (b) Testing framework for generating the internal states which uses 4D fMRI as input.

We propose a Polynomial Projection Operators together with Recurrent memory to solve the dynamical internal state $z\left(t\right)$. Given the input $z\left(t\right)$ or $f(z,l,t)$, the proposed method aims to solve the future prediction $z(t+1)$ based on the cumulative history $f_{\le t}$. However, as temporal resolution for MEG signal is intractably high, there exists vanishing gradient problem when the model evolves over all time states. To solve the vanishing gradient problem, we project the input state onto the subspace and maintain the compressed historical representation. So there are two problems to be solved: how to quantify the approximation, and the way to learn the subspace.

Function Approximation.

We introduce the probability measure $\mu \in [0,\mathrm{\infty })$ to define the space of square function $<f,p{>}_{\mu }={\int }_0^{\mathrm{\infty }}f\left(x\right)p\left(x\right)d\mu \left(x\right)$. $p(\cdot )$ is defined on a $M$ subspace $𝒫$. The function $p_t\in 𝒫$ is utilized to minimize the approximation of $f_{\le t}$ with ${∥f_{\le t}-p_t∥}_{L_2\left({\mu }_t\right)}$, where ${\mu }_t$ is the measure ranges in $(-\mathrm{\infty },t)$.

Polynomial Basis Expansion for Subspace Learning.

To learn the suitable subspace, we define the polynomial basis $𝒫$ with parameter $M$ to represent the projected history, which means the historical dynamics could be represented using $M$ coefficients with the basis $𝒫$. $M$ represents the size of the compression. However, it is challenging to maintain the parameter ${\mu }_t$ when $t\to \mathrm{\infty }$. We show more details of the suitable polynomial basis in Supplementary material.

The first step is to choose the appropriate basis $p_m\in 𝒫$ in the projection operator. The projection operator takes the historical memory of $t$ and minimize the approximation of $f_{\le t}$ using ${∥f_{\le t}-p_t∥}_{L_2\left({\mu }_t\right)}$. According to approximation theory, we use the orthogonal polynomials of ${\mu }_t$ as the orthogonal basis and represent the coefficients $c_m(t):={⟨f_{\le t},p_m⟩}_{{\mu }_t}$.

The second key step is to differentiate the projection using the inner product ${<f,p>}_{\mu }={\int }_0^{\mathrm{\infty }}f\left(x\right)p\left(x\right)d\mu \left(x\right)$, which will lead to the similar result that $\frac{d}{dt}{c}_m\left(t\right)$ is expressed using ${\left(c_k\left(t\right)\right)}_{k\in M}$ and $f\left(t\right)$. Thus, $c(t)\in R^M$ satisfies the ODE form $\frac{d}{dt}c\left(t\right)=\alpha \left(t\right)c\left(t\right)+\beta \left(t\right)f\left(t\right)$, where $\alpha \left(t\right)\in R^{M\times M}$ and $\beta \left(t\right)\in R^{M\times 1}$. For each time $t$, We could change $c\left(t\right)$ to the ODE form,

$$\frac{d}{dt}c(t)=-\frac{1}{t}\alpha (t)c(t)+\frac{1}{t}\beta (t)f(t),$$ (2)

The whole derivation of $c\left(t\right)$ is shown in Supplementary material. Next we want to solve the ODE problem and obtain the coefficient $c\left(t\right)$ by discretizing the continuous function $c\left(t\right)$ and $f\left(t\right)$, which yields the recurrence $c_{k+1}=-{\alpha }_k{c}_k+{\beta }_k{f}_k$ using ${\left(f_k\right)}_{k\in M}$. The orthogonal basis is defined as follows,

$$c_{k+1}=(1-\frac{\alpha }{k})c_k+\frac{1}{k}\beta f_k,$$ (3)

2.1. The Siamese Network for Behavioral Prediction in Space and Time

To score the agreement between the time series prediction and original brain measurements, we introduce the Siamese network [1] which is the weight sharing network for comparing two views. It should be noted that the Siamese network is only used in the training process to help learn the structural and functional patterns of fMRI and MEG. We give detailed description of training and testing procedure in Supplementary material. The Siamese network is used to measure the similarity between fMRI and MEG signals. The Siamese network consists of two views, the predicted MEG signal using fMRI and the corresponding ground truth MEG signal. The two input views are preprocessed using the encoder network $\acute{O}(\cdot )$. The encoder network shares the same weights between two views. The encoder network $\acute{O}(\cdot )$ of the predicted MEG-like signal is followed by an MLP head $\Omega (\cdot )$ to match the output of the other view.

Encoder Network with Continuous Convolutions.

To predict the multimodal brain measurements using the orthogonal basis in Eq. 3, We could use the continuous convolutions to represent the output signals. Following the previous work of bilinear method, we converts the state $\alpha$ and $\beta$ in Eq. 3 into an approximation $\tilde{\alpha }=(I-\Delta /2\cdot \alpha {)}^{-1}(I+\Delta /2\cdot \alpha )$ and $\tilde{\beta }=(I-\Delta /2\cdot \alpha {)}^{-1}\Delta \beta$. Based on Eq. 3, the continuous convolutions is defined as,

$$\begin{gathered} y_k={\widetilde{O\alpha }}^k\tilde{\beta }{f}_0+{\widetilde{O\alpha }}^{k-1}\tilde{\beta }{f}_1+\cdots +\widetilde{O\alpha }\tilde{\beta }{f}_{k-1}+\widetilde{O\beta }{f}_k \\ y=\acute{O}\ast f\text{and}\acute{O}\in R^K=(\widetilde{O\beta },\widetilde{O\alpha \beta },\cdots ,{\widetilde{O\alpha }}^k\beta ), \end{gathered}$$ (4)

Similarity Measurement.

Given two kinds of inputs $y_1$ and $y_2$ at time $t$ with $y_1\left(t\right)=O\left(z_1\left(t\right)\right)$. $y_1\left(t\right)$ represents the predicted brain measurement of the output model $O(\cdot )$ in Eq. 1. $y_2\left(t\right)$ is the original time series of brain activity, such as fMRI or MEG signals. The similarity of two view is defined as

$$D({\rho }_1(t),{\eta }_2(t))=-\frac{{\rho }_1(t)}{{\Vert {\rho }_1(t)\Vert }_2}\ast \frac{{\eta }_2(t)}{{\Vert {\eta }_2(t)\Vert }_2},$$ (5)

where ${\rho }_1(t)=\Omega (\acute{O}(y_1(t)))$ and ${\eta }_2(t)=\acute{O}(y_2(t)).{\parallel \cdot \parallel }_2$ is $l_2$-norm.

In the experiment, we adopt the stop-gradient operation. The stop-gradient treat the second input ${\eta }_2\left(t\right)$ as constant. The symmetrized loss is denoted by

$${ℒ}_s=\frac{1}{2}D({\rho }_1(t),stopgrad({\eta }_2(t)))+\frac{1}{2}D({\rho }_2(t),stopgrad({\eta }_1(t))),$$ (6)

3. Experimental Results

We tested the fidelity of the proposed method based on the resting state fMRI and MEG from HCP dataset. The resting-state fMRI was pre-processed following the minimal preprocessing pipeline [5]. Then the pre-processed data was registered into a standard cortical surface using MSMAll [5]. The artefacts were removed using ICA-FIX. The cortical surface was parcellated into N=360 major ROI [4]. In addition, the averaged time course of each ROI was normalized using z-score. The resting-state MEG was pre-processed using ICA to remove out artefacts related to head and eye movement. Sensor-space data were down-sampled 300 Hz using anti-aliasing filter. Next the MEG data were source-reconstructed with a scalar beamformer and registered into the standard space of the Montreal Neuroimaging Institute (MNI). MEG signals were then filtered into 1-30Hz and beamformed onto 6 mm grid. We used theta (4–8 Hz), alpha (8–13 Hz) and beta (13–30 Hz) bands to filter the source-space data. The parcellation atlas and z-score normalization method of MEG were similar to the resting-state fMRI.

3.1. Spatio-Temporal Patterns of the Predicted Results Using 4D fMRI Image

We next applied the proposed method to acquire the dynamics of the behavioral representation for each voxel in fMRI signal. We used the whole 4D fMRI image to generate the corresponding behavioral states in the spatio-temproal domain. We show the spatial map of independent temporal signal using original MEG and the predicted results in Fig. 2. The spatial map was generated using the ICA. With the 25 generated ICA temporal components of MEG and predicted results, we paired the 8 resting state brain networks(RSNs) spatial map with that derived using resting state fMRI. The DMN pattern is shown in Fig. 2a. The nodes is highlighted in the medial frontal cortex and inferior parietal lobules. The patterns of left lateralized frontoparietal and sensorimotor network are shown in Fig. 2b and Fig. 2c. From Fig. 2, we could see that the spatial pattern of predicted results could match that of MEG in all the network. However, the spatial resolution of predicted result is much higher than MEG signal.

Fig. 2.

Brain networks acquired with ICA shown in the order predicted results (top) and MEG (bottom). (a) DMN, (b) left frontoparietal network, (c) sensorimotor network.

The results have shown that the temporal ICA components originate from the brain regions correlated with the RSNs spatial maps. We show more results in the Supplementary material. The proposed method could generate the high resolution behavioral states that are the perfect matches to the spatial pattern of the observed fMRI. Thus, the proposed method could provide fundamental role of brain dynamics related to behavioral measurements.

3.2. Temporal Pattern of Predicted Results

We finally evaluated temporal pattern of the predicted internal states compared with original 4D fMRI and MEG image. Figure 3 shows the averaged neural time series of the predicted results, fMRI and MEG signal in visual network. From Fig. 3, we could see that the proposed method inherits the high temporal resolution of MEG signal in a dynamical system. The proposed method predicts the hidden observation of the dynamical neural transmission locally and globally. In Table 1, we use the mean squared error (MSE) to measure the similarity between the ROI averaged predicted results and ROI averaged MEG signals. In addition, we also compare the proposed method with three baselines LSTM, GRU-D and proposed method without Siamese netwrok. Polynomial Projection Operators could be combined with The introduction of the spatio-temporal constraints into the dynamical system is consistent with the fundamental role in biological system. The proposed method provides chance to understand how behavioral representations evolve over time.

Fig. 3.

Averaged neural time series in visual network (a) fMRI, (b) predicted result, (c) MEG.

Table 1.

Temporal pattern prediction results with different baseline models

Methods	LSTM	GRU-D	Without Siamese network	Proposed method
MSE	0.3199	0.5805	0.1134	0.0577

4. Conclusions

To understand the complex brain dynamics, we need to record the activity in space and time, which could not be solved using current noninvasive techniques of brain measurements. We propose a novel computational model that could combine the information from several techniques and predict the internal brain activity with both high spatial and high temporal resolution. The proposed framework address the vanishing gradient problem by abstracting the long tern temporal relationship with functional approximation. In the training step, we use both fMRI and MEG data to learn the brain dynamical representation with Siamese network. While in the testing step, for the first time, the proposed method solves the problem of predicting the internal behavioral states with high resolution in spatial and temporal domain using only fMRI data. The potential of the proposed method to represent the spatio-temporal dynamics has been demonstrated using two experiments with HCP data.