Multimodal Functional Imaging Using fMRI-Informed Regional EEG/MEG Source Estimation

Abstract

We propose a novel method, fMRI-Informed Regional Estimation (FIRE), which utilizes information from fMRI in E/MEG source reconstruction. FIRE takes advantage of the spatial alignment between the neural and the vascular activities, while allowing for substantial differences in their dynamics. Furthermore, with the regional approach, FIRE can be efficiently applied to a dense grid of sources. Inspection of our optimization procedure reveals that FIRE is related to the re-weighted minimum-norm algorithms, the difference being that the weights in the proposed approach are computed from both the current estimates and fMRI data. Analysis of both simulated and human fMRI-MEG data shows that FIRE reduces the ambiguities in source localization present in the minimum-norm estimates. Comparisons with several joint fMRI-E/MEG algorithms demonstrate robustness of FIRE in the presence of sources silent to either fMRI or E/MEG measurements.

1 Introduction

The principal difficulty in interpreting Electroencephalography (EEG) and magnetoencephalography (MEG) data stems from the ill-posed electromagnetic inverse problem: infinitely many spatial current patterns give rise to identical measurements [10]. Additional assumptions on the spatial current patterns must be incorporated into the reconstruction process to obtain a unique estimate [3].

In addition to the general assumptions about the spatial current patterns such as minimum energy (or ℓ₂-norm), specific prior knowledge about activation locations can be obtained from other imaging modalities. Among them, functional Magnetic Resonance Imaging (fMRI) provides the most relevant information for the reconstruction due to its good spatial resolution. fMRI measures the hemodynamic activity, which indirectly reflects the neural activity measured by E/MEG. Extensive studies of neurovascular coupling have demonstrated similarity in spatial patterns of these two types of activations [13]. However, the timecourses of the neural and the vascular activities differ substantially, and their exact relationship is yet to be characterized in full. In addition to the differences in their physiological origins, E/MEG and fMRI have different sensitivity characteristics. For example, a brief transient neural activity may be difficult to detect in fMRI while sustained weak neural activity may lead to relatively strong fMRI signals but might have a poor signal-to-noise ratio in E/MEG.

The most straightforward way to incorporate fMRI information into E/MEG inverse estimation is the fMRI-weighted Minimum-Norm Estimation (fMNE) [1, 12]. This method uses a thresholded Statistical Parametric Map (SPM) from fMRI analysis to construct weights for the standard Minimum-Norm Estimation (MNE), leading to significant improvements when the SPM is accurate. However, the weights depend on arbitrary choices of the threshold and weighting parameters. Moreover, these weights are assumed to be time independent causing excessive bias in the estimated source timecourses. Sato et al. [15] combined the Automatic Relevance Determination (ARD) framework and fMNE to achieve more focal estimates. In this method, which we will refer to as fARD, the parameters of a hyper-prior are set based on the thresholded SPM. In addition to the arbitrary choice of the threshold similar to that in fMNE, the estimates computed via fARD are often unstable, especially in the regions where the vascular activity is weak.

Here, we propose a novel method, the fMRI-Informed Regional Estimation (FIRE), to improve the accuracy of the E/MEG source estimates. Since the relationship between the dynamics of the evoked neural and the evoked vascular signals is largely unknown, we only model the similarity of spatial patterns in the two processes, as opposed to the Kalman-filter approach in [6]. Furthermore, we expect that the shape of the activation timecourses varies across brain regions, especially for the neural activation timecourses. To account for this fact, FIRE treats the temporal dynamics in different brain regions independently. In other words, there is no constraint imposing similarity of the activation timecourses across regions. We assume the shape of the activation timecourses to be constant within a brain region, modulated by a set of location-specific latent variables. The regions are chosen based on subject-specific cortical parcellation [7]. Handling the temporal dynamics of the two types of activities separately while exploiting their common spatial pattern helps to preserve the temporal resolution of E/MEG and to achieve accurate source localization.

The prior on the latent variables encourages spatially smooth current estimates within a brain region. The prior also encourages the number of activated regions to be small, similar to the ARD approach [17], except that our prior is region-based rather than location-based. Both the activation timecourse model and the choice of brain regions in FIRE are similar to those employed in recent work by Daunizeau et al. [4]. However, Daunizeau et al. aim to symmetrically infer brain activities visible in either EEG or fMRI data, resulting in an extra random variable to model the vascular activity. The confidence of the estimated brain activities reduces when there are discrepancies between the EEG and the fMRI measurements. Furthermore, due to the complexity of this model, the estimation is limited to a coarse source space. Instead of aiming at a symmetrical inference, we focus on the estimation of current sources. We incorporate the fMRI information to reduce ambiguities in source localization usually present in E/MEG source estimation.

To fit the model to the data, we use the coordinate descent method, alternating between the estimation of current sources and of other model parameters. This iterative update scheme is similar to the re-weighted MNE methods such as FOCal Underdetermined System Solver (FOCUSS) [8]. In contrast to the re-weighted MNE, in our method the weights are jointly determined using both the estimated neural activity and the vascular activity measured by fMRI. Moreover, the estimates at different time points influence each other. The computation of the weights is related to problems arising in continuous Gaussian mixture modeling, which can be efficiently optimized using the Expectation-Maximization (EM) algorithm [5].

In the following, we first discuss the model underlying FIRE, the inference procedure, and the implementation details. We then present the experimental comparisons between FIRE and prior methods for joint E/MEG-fMRI analysis using both simulated and human data, followed by a discussion and conclusions.

2 Methods

2.1 Neurovascular coupling and data models

We assume that the source space comprises N discrete locations on the cortex parcelled into K brain regions. We denote the set indexing the discrete locations in region k by P_k and the cardinality of P_k by N_k.

Fig. 1 illustrates our model. The shape of the source timecourses is identical within a region but varies across regions. Specifically, we let u_k and v_k be the unknown waveforms in region k, associated with neural and vascular activities, respectively. We model the neural and the vascular activity strength through a hidden vector variable z = [z₁, z₂, · · · , z_N]^T. The continuous scalar z_n indicates the activation strength at location n on the cortical surface. Thus, the probabilistic model for the neural activation timecourse j_n and the vascular activation timecourse f_n at location n in region k can be expressed as

$$p({\mathbf{j}}_n,{\mathbf{f}}_n|z_n;{\mathbf{u}}_k,{\mathbf{v}}_k,{\eta }_k^2,{\xi }_k^2)=𝒩({\mathbf{j}}_n;z_n{\mathbf{u}}_k,{\eta }_k^2\mathbf{I})𝒩({\mathbf{f}}_n;z_n{\mathbf{v}}_k,{\xi }_k^2\mathbf{I}),$$ (1)

where ${\eta }_k^2\text{and}{\xi }_k^2$ are noise variances. We construct all matrices such that each row represents a location or a sensor and each column represents a particular time point. Thus, we let N × T_J matrix J = [j₁, j₂, · · ·, j_N]^T be the neural current on the cortex for all T_J time points. We assume that the vascular signal f_n at location n is directly observable through fMRI. We let N × T_F matrix F = [f₁, f₂, · · · , f_N]^T be the fMRI measurements on the cortex over T_F time points. Note that our neurovascular coupling model captures only the spatial alignment between the two types of activities; it does not impose temporal similarity between the signals.

Fig. 1.

Graphical interpretation of FIRE. The hidden activity z models the neurovascular coupling relationship. The hidden current source distribution J is measured by E/MEG, producing observation Y. F denotes fMRI measurements. Vectors u and v are the unknown neural and vascular waveforms in a certain brain region, respectively. The inner plate represents N_k vertices in region k; the outer plate represents K regions. The bottom left and right plates represent T_J and T_F time points in the neural and the vascular measurements, respectively.

The neural currents j_n are detected with E/MEG described by the standard observation model. We let M × T_J matrix Y = [y(1), y(2), · · ·, y(T_J)] be the E/MEG measurements at all T_J time points. Column t of matrix J, j(t), denotes the neural current distribution at time t. The quasi-static Maxwell’s equations imply that E/MEG signals at time t are instantaneous linear combinations of the currents at different locations:

$$\mathbf{y}(t)=\mathbf{\text{Aj}}(t)+\mathbf{e}(t)\forall t=1,2,\cdots ,T_{\mathbf{J}},$$ (2)

where e(t) is the measurement noise. The M × N forward matrix A is determined by the electromagnetic properties of the head, the geometry of the sensors, and the locations of the sources. With spatial whitening in the sensor space, e(t) ~ 𝒩 (0, I). The number of sources N (~ 10³ −10⁴) is much larger than the number of measurements M (~ 10²), leading to an infinite number of solutions satisfying Eq. (2) even for e(t) = 0. In general, j_n should be modeled as three timecourses corresponding to the three Cartesian components of the current. However, due to the columnar organization of the cortex, we can further constrain the current orientation to be perpendicular to the cortical surface and consider a scalar timecourse at each location.

2.2 Priors and Parameter Settings

To encourage the activation patterns to be smooth within a region, we impose a prior on the modulating variables. Specifically, we define z_k = {z_n}_n∈Pk and assume z_k ~ 𝒩 (0, ${\gamma }_k^2$ Φ_k), where the variance ${\gamma }_k^2$ indicates the activation strength in region k, and Φ_k is a fixed matrix that acts as a regularizer by penalizing the sum of squared differences between neighboring locations. This spatial prior is particularly important for the brain regions where vascular activity is too weak to measure, but the neural activity can be detected by E/MEG.

Our Φ_k is similar to the regularizer used in the Low Resolution Brain Electromagnetic Tomography (LORETA) [14], except that we apply Φ_k to individual brain regions while LORETA’s spatial regularizer is applied to the whole brain. We assume separate variance ${\gamma }_k^2$ for different brain regions since the strength of current is expected to vary significantly between regions with and without active sources. This choice is similar to the recent work in the application of ARD to E/MEG reconstruction [15,17], except that their work assumes independent γ² for each location in the brain.

Since the forward model A is underdetermined, the current distribution J, produced by our neurovascular coupling model, can fully explain the E/MEG data. In other words, without the noise term ${\eta }_k^2$ (i.e., j_n = z_nu_k), the fMRI data can exert too much influence on the reconstruction results. Although we can estimate the noise variance of the current source timecourses ${\eta }_k^2$ by extending the inference procedure, we find the corresponding estimate unstable without a prior. Based on preliminary empirical testing, we fix ${\eta }_k^2=1$. With proper temporal whitening of the fMRI data, we can also assume that ${\xi }_k^2={\eta }_k^2.\text{Fixing}{\eta }_k^2={\xi }_k^2$ helps to significantly reduce the computational burden of the estimation.

To summarize, our model can be mathematically expressed as

$$p(\mathbf{Y},\mathbf{J},\mathbf{F},\mathbf{z};\mathbf{\Theta })=p(\mathbf{Y}|\mathbf{J})p(\mathbf{J},\mathbf{F}|\mathbf{z};\mathbf{\Theta })p(\mathbf{z};\mathbf{\Theta }),$$ (3)

where Θ = [θ₁, θ₂, · · · , θ_K] is the combined set of parameters, and θ_k = {u_k, v_k, ${\gamma }_k^2$}. p(Y|J) is the E/MEG data model in Eq. (2). p(J, F|z; Θ) is our neurovascular coupling model in Eq. (1), and p(z; Θ) is the prior on z. Therefore,

$$\begin{array}{l}\text{log}p(\mathbf{Y},\mathbf{J},\mathbf{F},\mathbf{z};\mathbf{\Theta })={\displaystyle {\sum }_{t=1}^{T_{\mathbf{J}}}}\text{log}𝒩(\mathbf{y}(t);\mathbf{\text{Aj}}(t),\mathbf{I})+\hfill \\ {\displaystyle {\sum }_{k=1}^K}{\displaystyle {\sum }_{n=1}^{N_k}}\text{log}\left[𝒩\right({\mathbf{j}}_n;z_n{\mathbf{u}}_k,{\eta }_k^2\mathbf{I}\left)𝒩\right({\mathbf{f}}_n;z_n{\mathbf{v}}_k,{\xi }_k^2\mathbf{I}\left)\right]+{\displaystyle {\sum }_{k=1}^K}\text{log}𝒩({\mathbf{z}}_k;\mathbf{0},{\gamma }_k^2{\mathbf{\Phi }}_k).\hfill \end{array}$$

2.3 Inference

Our goal is to estimate the current source J and the timecourses u and v. We treat the activation strength z as an auxiliary variable, and marginalize it out in the analysis. We formulate the inference as

$$\begin{array}{cc}\{{\mathbf{J}}^{*},{\mathbf{\Theta }}^{*}\}\hfill & =\underset{\mathbf{J},\mathbf{\Theta }}{\text{arg max}}\text{log}p(\mathbf{Y},\mathbf{J},\mathbf{F};\mathbf{\Theta })\hfill \\ \hfill & =\underset{\mathbf{J},\mathbf{\Theta }}{\text{arg max}}\text{log}{\displaystyle {\int }_{\mathbf{z}}}p(\mathbf{Y},\mathbf{J},\mathbf{F},\mathbf{z};\mathbf{\Theta })d\mathbf{z}=\underset{\mathbf{J},\mathbf{\Theta }}{\text{arg max}}\text{log}\left(p(\mathbf{Y}|\mathbf{J})p(\mathbf{J},\mathbf{F};\mathbf{\Theta })\right).\hfill \end{array}$$ (4)

With marginalization of z, p(J, F; Θ) acts as the prior for J. Since both J and F are linear functions of z, p(J, F; Θ) is a continuous Gaussian mixture model.

The difficulty in inference with the proposed model is caused by the intertwining between space and time, reflected by the intersection of the temporal plates and the spatial plates in Fig. 1. That is because the output of a given E/MEG sensor is a mixture of signals from the entire source space. Hence, the inference must be performed for all time points and all locations simultaneously. FIRE is thus substantially more computationally demanding compared to standard temporally independent E/MEG estimation and voxel-wise fMRI analysis.

Due to the special structure in our model when one set of variables is fixed, we can derive an efficient gradient descent method with two alternating steps. In the first step, we fix Θ and derive a closed-form solution for J. In the second step, we fix J and show that Θ can be efficiently estimated through the EM algorithm. Section 5 discusses an alternative approach to this inference problem.

For a fixed Θ = Θ̂, p(Y, J, F; Θ̂) is a jointly-Gaussian distribution. Thus, the estimate of J is the conditional mean:

$$\widehat{\mathbf{J}}=\underset{\mathbf{J}}{\text{arg max}}\text{log}p(\mathbf{Y},\mathbf{J},\mathbf{F};\widehat{\mathbf{\Theta }})=E\left[\mathbf{J}|\mathbf{Y},\mathbf{F};\widehat{\mathbf{\Theta }}\right]={\mathbf{\Gamma }}_{\mathbf{w},\mathbf{J}}^{\mathrm{T}}{\mathbf{\Gamma }}_{\mathbf{w}}^{-1}\mathbf{w},$$ (5)

where w^T = [(vec (Y))^T (vec (F))^T] includes both E/MEG and fMRI measurements. Operator vec (·) concatenates adjacent columns of a matrix. Γ_w is the covariance matrix of w, and Γ_w,J is the cross-covariance matrix between w and vec (J). Thus, E/MEG and fMRI measurements jointly determine the estimate of the neural activity. Eq. (5) is similar to the standard MNE solution [10], but it also includes the correlation between Y and F and the correlation among different time points of J.

For a fixed J = Ĵ, we optimize the parameters Θ:

$$\widehat{\mathbf{\Theta }}=\underset{\mathbf{\Theta }}{\text{arg max}}\text{log}p(\widehat{\mathbf{J}},\mathbf{F};\mathbf{\Theta }).$$ (6)

As shown in Fig. 1, when the current distribution J is fixed, the E/MEG measurement Y does not provide additional information for the parameter estimation. Since the parameters for different regions are independent for a fixed Ĵ, the estimates for different regions can be obtained independently. Furthermore, parameter Θ can be efficiently estimated using the EM algorithm by re-introducing the latent variable z, which is the auxiliary variable describing activation strength. This method can be thought of as an extension of the EM algorithm for probabilistic PCA [16] to two sets of data [2]. For region k, the parameter estimates θ̂_k can be obtained by optimizing the lower bound of the log-probability:

$$\text{log}p\left({\{{\widehat{\mathbf{j}}}_n,{\mathbf{f}}_n\}}_{n\in P_k};{\theta }_k\right)\ge {\displaystyle {\int }_{{\mathbf{z}}_k}}q({\mathbf{z}}_k)\text{log}p\left({\{{\widehat{\mathbf{j}}}_n,{\mathbf{f}}_n\}}_{n\in P_k},{\mathbf{z}}_k;{\theta }_k\right)d{\mathbf{z}}_k,$$ (7)

where q(z_k) = p (z_k|{ĵ_n, f_n}_n∈Pk; θ̂_k) is the posterior probability computed in the E-step. Since {ĵ_n, f_n}_n∈Pk and z_k are jointly-Gaussian distributed for a fixed θ̂_k, q(z_k) is also a Gaussian distribution. We use 〈·〉_q to denote the expectation with respect to the posterior distribution q(z_k), i.e., 〈·〉_q ≜ E [·|{ĵ_n, f_n}_n∈Pk; θ̂_k]. Since the M-step depends only on quantities related to the first- and the second-order statistics of z_k, we only need to update those quantities in the E-step:

$$\begin{array}{cc}\hfill {\langle {\mathbf{z}}_k{\mathbf{z}}_k^{\mathrm{T}}\rangle }_q& \leftarrow {\left[\frac{1}{{\gamma }_k^2}{\mathbf{\Phi }}_k^{-1}+\left(\frac{{\mathbf{u}}_k^{\mathrm{T}}{\mathbf{u}}_k+{\mathbf{v}}_k^T{\mathbf{v}}_k}{{\eta }_k^2}\right)\mathbf{I}\right]}^{-1}\hfill \\ \hfill {\langle {\mathbf{z}}_k\rangle }_q& \leftarrow \frac{{\langle {\mathbf{z}}_k{\mathbf{z}}_k^{\mathrm{T}}\rangle }_q}{{\eta }_k^2}{\left[\left({\mathbf{u}}_k^{\mathrm{T}}{\widehat{\mathbf{j}}}_1+{\mathbf{v}}_k^{\mathrm{T}}{\mathbf{f}}_1\right),\cdots ,\left({\mathbf{u}}_k^{\mathrm{T}}{\widehat{\mathbf{j}}}_{N_k}+{\mathbf{v}}_k^{\mathrm{T}}{\mathbf{f}}_{N_k}\right)\right]}^{\mathrm{T}}\hfill \\ \hfill {\langle {\mathbf{z}}_k^{\mathrm{T}}{\mathbf{\Phi }}_k^{-1}{\mathbf{z}}_k\rangle }_q& \leftarrow {\langle {\mathbf{z}}_k\rangle }_q^{\mathrm{T}}{\mathbf{\Phi }}_k^{-1}{\langle {\mathbf{z}}_k\rangle }_q+\text{tr}({\mathbf{\Phi }}_k^{-1}{\langle {\mathbf{z}}_k{\mathbf{z}}_k^{\mathrm{T}}\rangle }_q).\hfill \end{array}$$

In the M-step, we fix q(z_k) and optimize Eq. (7). With some algebra, we arrive at the update equations for the model parameters:

$${\widehat{\mathrm{u}}}_k\leftarrow \frac{{\mathrm{\sum }}_{n\in P_k}{\langle z_n\rangle }_q{\widehat{\mathbf{j}}}_n}{\text{tr}({\langle {\mathbf{z}}_k{\mathbf{z}}_k^{\mathrm{T}}\rangle }_q)},{\widehat{\mathbf{v}}}_k\leftarrow \frac{{\mathrm{\sum }}_{n\in P_k}{\langle z_n\rangle }_q{\mathbf{f}}_n}{\text{tr}({\langle {\mathbf{z}}_k{\mathbf{z}}_k^{\mathrm{T}}\rangle }_q)},\text{\hspace{1em} and \hspace{1em}}\widehat{{\gamma }_k^2}\leftarrow \frac{{\langle {\mathbf{z}}_k^{\mathrm{T}}{\mathbf{\Phi }}_k^{-1}{\mathbf{z}}_k\rangle }_q}{N_k}.$$ (8)

We iterate the EM algorithm until convergence which usually takes less than ten iterations. We then re-estimate J according to Eq. (5).

To summarize, the algorithm proceeds as follows:

1. Initialize Ĵ as the MNE estimate.

2. Until convergence:

   1. Compute Θ̂ using the EM algorithm: E-step for the hidden variable z followed by M-step for the model parameters Θ.

   2. Update Ĵ according to Eq. (5) for Θ = Θ̂.

3 Implementation

For the computation of the forward matrix A, we need to specify the E/MEG forward model and the source space. We employ the single-compartment boundary-element model for the MEG forward computations [9]. The source space is confined to a mesh on the cortical surface with approximately 5-mm resolution, corresponding to about 5000 vertices per hemisphere.

The functional regions are defined by parceling the cortical folding pattern using the FreeSurfer software, resulting in 35 parcels per hemisphere [7]. The boundaries of adjacent parcels are defined along sulci. We merge adjacent parcels that contain fewer than 30 vertices. Our neurovascular coupling model requires an orientation reference for each brain region. Here, we set the orientation reference to be the largest left singular vector of the matrix formed by the outward cortical normals within a region.

We apply the standard preprocessing to fMRI data, then estimate the hemodynamic response function (HRF) at each voxel with a 15-bin finite impulse response regressor covering a 20-s time window using the FS-FAST software (MGH, Boston, MA). The estimated HRF is used as f_n in our model. For a source space of N ~ 10⁴ vertices and timecourses of T_J ~ 10² and T_F ~ 10¹ samples, FIRE takes less than 20 iterations until the energy function reduces less than 0.1% from the energy of the previous iteration. In each iteration of the coordinate descent algorithm, the estimate of Θ takes 30 seconds, while the estimate of J takes 4 minutes on a standard PC (2.8 GHz CPU and 8 GB RAM), leading to the total run time of approximately 1.5 hours. Estimating J involves an inversion of an (MT_J + NT_F) × (MT_J + NT_F) dense symmetric matrix Γ_w, which is too large to store in memory. Instead, we employ the conjugate gradient descent method to solve the corresponding system of linear equations. It usually takes 100 iterations until convergence.

4 Results

We first compare FIRE to MNE, fMNE, and fARD using simulated data. We then extend the comparison to human MEG and fMRI data from a somatosensory study.

4.1 Simulation studies

To simulate MEG measurements, we created two patches on the cortical sheet, with current source orientation along the outward normal to the cortical surface. Shown in the lateral-occipital view of the right hemisphere (Fig. 2), Patch A contains 20 vertices and is located in the inferior parietal region. Patch B contains 32 vertices and is located in the superior parietal region. We simulated neural and vascular timecourses in these two patches for three different scenarios: no silent activity, silent vascular activity, and silent neural activity. In the two cases with silent activities, we kept the activity of patch B unchanged while silencing neural or vascular activity in patch A. The simulated neural signals are shown as solid black lines in the rightmost column of Fig. 2. The activation maps corresponding to the peaks of the two simulated neural signals are shown in the first column.

Fig. 2.

Current source estimates in three scenarios. Lateral-occipital view of the right hemisphere is shown. Patch A and patch B are highlighted in the top left panel; the rest of the figures follow the same convention. (a) Neither neural nor vascular activity is silent. (b) Vascular activity in patch A is silent. (c) Neural activity in patch A is silent. The first column illustrates the simulated current distributions with a selected threshold at the peak activations. The next four columns show the estimates from MNE, fMNE, fARD, and FIRE. Hot/cold colors correspond to outward/inward current flow. The rightmost column shows the simulated (black solid) and the estimated (dashed) timecourses from the most active vertices in patch A (top) and B (bottom) for the corresponding methods.

For the forward calculations, we employed the sensor configuration of the 306-channel Neuromag VectorView MEG system used in our human studies and added Gaussian noise to the signals. The resulting signals have a SNR of 3 dB, within the typical SNR range of real MEG data. Since the two patches are close in the highly folded cortex and they exhibit neural activity during overlapping time intervals, it is particularly difficult to obtain accurate current source estimates.

Columns two to five in Fig. 2 depict the current estimates using different methods. Following [12], the fMNE weighting parameters are set to 1 and 0.1 for active and inactive fMRI locations, respectively. Since the estimates from different methods are not directly comparable in amplitude, the threshold for each method is chosen to be 1/6 of the maximum absolute value of the corresponding current estimates J*. The rightmost column in Fig. 2 presents the estimated timecourses (dashed) of the most active vertex, in terms of energy, in both patches.

No silent activity

As shown in Fig. 2(a), the MNEs extend across adjacent gyri. fMNE, fARD, and FIRE correctly localize the two patches at the peak activation, but FIRE provides a better estimate of the spatial extent of the activations. The fARD estimate is unstable, as reflected by the large fluctuations in the estimated timecourses in patch B (green).

Silent vascular activity

When the vascular activity in patch A is silent, fMNE shows excessive bias towards patch B. Without a large weight, the amplitude of the estimated timecourses (blue) in patch A is significantly lower than the corresponding estimates in patch B. It would be therefore easy to miss neural activation in patch A when interpreting the results (column three in Fig. 2(b)). In contrast, by combining neural and vascular information in the re-weighted scheme, FIRE avoids such a bias. Its estimate in patch A (column five) is similar to that obtained from MNE (column one). As the weights for patch B increase and the weights for patch A decrease in the fARD update, the estimate in patch B explains the activation in patch A. As shown in the timecourse panel, the estimated timecourse in patch B (green) is similar in shape to the simulated timecourse in patch A (black solid). The change of sign is due to the fact that the outward normals for patch A and patch B are in approximately opposite directions.

Silent neural activity

As shown in Fig. 2(c), all methods can correctly localize the neural activity in patch B, except for the small false positive in patch A for fARD. By assigning identical weights to patches A and B, fMNE estimates a timecourse for patch A (blue) that is noisier than the corresponding one produced by FIRE (red). FIRE suppresses the weights for patch A since the current estimates in that patch are close to zero; its results are closer to the simulations.

4.2 Median-nerve experiments

We also tested the method using human experimental data. The median nerve at the right wrist was stimulated according to an event-related protocol, with a random inter-stimulus-interval ranging from 3 to 14 s. This stimulus activates a complex cortical network [11], including the contralateral primary somatosensory cortex (cSI) and bilateral secondary somatosensory cortices (cSII and iSII).

MEG and fMRI data were acquired in separate sessions. The MEG measurements were acquired using a 306-channel Neuromag VectorView MEG system. A 200-ms baseline before the stimulus was used to estimate the noise covariance matrix of the MEG sensors. An average signal, computed from approximately 100 trials, was used as the input to each method. The fMRI images were acquired using a Siemens 3T machine (TR=1.5 s, 64×64×24, 3×3×6mm³, single channel head coil). Anatomical images, from a 3T scanner, were used to construct the source space and the forward model.

In the leftmost column in Fig. 3, approximate locations for cSI (solid), cSII (dashed), and iSII (dashed) are highlighted on the fMRI activation maps (p ≤ 0.005). Given the expected activations, we partitioned the post-central region into two regions, separately covering cSI and cSII. Note that in the noisy SPM, the sites of fMRI activations do not exactly agree with the locations of the expected current sources.

Fig. 3.

Human median-nerve experiments. In the first column, approximate locations for cSI (solid), cSII (dashed), and iSII (dashed) are highlighted on the fMRI activation maps. Columns two to five show the current estimates obtained via MNE, fMNE, fARD, and FIRE at 75 ms after stimulus onset. Hot/cold colors indicate outward/inward current flow.

Columns two to five in Fig. 3 present the estimates at 75 ms after stimulus onset, during which cSI, cSII, and iSII should be activated. The threshold was set separately for each hemisphere since the activation in iSII is much weaker than that in cSI and cSII. For each method, the threshold is set to be 1/6 of the maximum absolute value of the corresponding current estimates. MNE produces a more diffuse estimate, including physiologically unlikely activations at the gyrus anterior to the cSI area. In contrast, FIRE pinpoints cSI on the postcentral gyrus. With the prior knowledge from fMRI, the detected cSII and iSII activations using fMNE, fARD, and FIRE are within the expected areas. The fMNE and fARD show stronger weighting towards the fMRI, reflected by the activations in the temporal lobes. Due to the highly folded cortex and uncertainties in MRI-fMRI registration, fMRI cannot distinguish between the walls of the central sulcus and the post-central sulcus, causing both walls to show strong vascular activity after mapping of the fMRI volume onto the cortex. Hence, fMNE, fARD, and FIRE estimates extend to both sulcal walls mentioned above.

5 Discussion

The coupling of spatial and temporal domains in the joint fMRI-E/MEG analysis constrained many previous models to operate on a coarse source space. The use of region-based neurovascular coupling model proposed in this paper reduces the computational burden, leading to a tractable reconstruction in a densely sampled source space similar to that typically used in MNE. Since the objective function is not convex, FIRE depends on the initialization. We believe MNE estimate is a reasonable choice for initialization since it is unbiased.

As an alternative to the coordinate descent inference procedure proposed in this work, one could treat both J and z as latent variables in the EM framework. Since J, z, and the measurements are jointly Gaussian, the posterior distribution of the latent variables is also Gaussian, leading to a closed-form update. Similar to the derivations in Eq. (8), the M-step updates depend on the second order statistics of the latent variables. Since J is not fixed in this EM procedure, the estimate at each location depends on the estimate at all other locations in the source space, as opposed to the region-independent estimation when J is fixed. Therefore, the computation of the second-order statistics is infeasible except for an extremely coarse discretization of the source space.

The estimation of u_k and v_k is closely related to the canonical correlation analysis (CCA). CCA seeks vectors to project two high dimensional data sets ({ĵ_n}_n∈Pk and {f_n}_n∈Pk in our case) to a low dimensional space so as to maximize the correlation coefficient. The probabilistic interpretation of CCA has been established in [2].

Our neurovascular coupling model is designed for fixed-orientation current estimates, since the latent-variable model assumes that the spatial concordance of neural and vascular activities is characterized by a scalar. For free-orientation current estimates, the neurovascular coupling model would have to be adjusted to handle the correspondence between the current flow in three directions and a single vascular activation timecourse at a certain location. Moreover, FIRE assumes a single activation waveform pair, u and v, in a region. The validity of this assumption depends on the size of the region and the distance between two activation sources. We cannot directly extend FIRE to multiple activation waveform pairs per region, since such an extension does not capture the fact that the shape of the vascular activation timecourses from two distinct sources is often highly similar but the neural processes are different. In the situation where there are two distinct current sources in one region, our preliminary results demonstrate that FIRE can localize the two current sources, but the estimated timecourses are combinations of the true timecourses. We defer the extension for free-orientation estimate and the extension for multiple activation sources per region to future work.

6 Conclusions

In contrast to most joint fMRI-E/MEG models, FIRE explicitly takes into account the inherent differences in the data measured by E/MEG and fMRI. The corresponding estimates can be efficiently computed with an iterative procedure which bears similarity with re-weighted MNE methods, except that the weights are based on both the current estimates in the last iteration and the fMRI data via the proposed neurovascular coupling model. This construction of the weights reduces the excessive sensitivity to fMRI present in many joint fMRI-E/MEG analysis methods, leading to more accurate current estimates as demonstrated by analysis of both simulated and human data.