FMRI-EEG Integrated Cortical Source Imaging by use of Time-Variant Spatial Constraints

Abstract

In response to the need of establishing a high-resolution spatiotemporal neuroimaging technique, tremendous efforts have been focused on developing multimodal strategies that combine the complementary advantages of high-spatial-resolution functional magnetic resonance imaging (fMRI) and high-temporal-resolution electroencephalography (EEG) or magnetoencephalography (MEG). A critical challenge to the fMRI-EEG/MEG integration lies in the spatial mismatches between fMRI activations and instantaneous electrical source activities. Such mismatches are fundamentally due to the fact that fMRI and EEG/MEG signals are generated and collected in highly different time scales. In this paper, we propose a new theoretical framework to solve the problem of fMRI-EEG integrated cortical source imaging. The new framework has two principal technical advancements. First, by assuming a linear neurovascular coupling, a method is derived to quantify the fMRI signal in each voxel as proportional to the time integral of the power of local electrical current during the period of event related potentials (ERP). Second, the EEG inverse problem is solved for every time instant using an adaptive Wiener filter, in which the prior time-variant source covariance matrix is estimated based on combining the quantified fMRI responses and the segmented EEG signals before response averaging. A series of computer simulations were conducted to evaluate the proposed methods in terms of imaging the instantaneous cortical current density (CCD) distribution and estimating the source time courses with a millisecond temporal resolution. As shown in the simulation results, the instantaneous CCD reconstruction by using the proposed fMRI-EEG integration method was robust against both fMRI false positives and false negatives while retaining a spatial resolution nearly as high as that of fMRI. The proposed method could also reliably estimate the source waveforms when multiple sources were temporally correlated or uncorrelated, or were sustained or transient, or had some features in frequency or phase, or had even more complicated temporal dynamics. Moreover, applying the proposed method to real fMRI and EEG data acquired in a visual experiment yielded a time series of reconstructed CCD images, in agreement with the traditional view of hierarchical visual processing. In conclusion, the proposed method provides a reliable technique for the fMRI-EEG integration and represents a significant advancement over the conventional fMRI-weighted EEG (or MEG) source imaging techniques, and is also applicable to the fMRI-MEG integrated source imaging.

INTRODUCTION

Neural activities elevate electromagnetic signal changes accompanied by hemodynamic and metabolic changes. These changes are the basic sources for most noninvasive neuroimaging techniques. For example, electroencephalography (EEG) and magnetoencephalography (MEG) monitor the electrophysiological activity inside the brain by measuring the induced electromagnetic field using electric or magnetic sensors over the scalp surface (Nunez and Srinivasan, 2005; He, 2004; Hämäläinen et al., 1993). Both EEG and MEG have an intrinsic high temporal resolution that allows tracking rapid neurophysiologic processes at the neuronal time scale of milliseconds. However, the EEG (or MEG) source imaging often suffers from ambiguities in defining the anatomical locations of underlying electrical sources due to the need of solving highly ill-posed inverse problems (for reviews, see He, 1999; Pascual-Marqui, 1999; Darvas et al., 2004). On the other hand, functional magnetic resonance imaging (fMRI) (Bandettini et al., 1992; Kwong et al., 1992; Ogawa et al., 1992) is capable of mapping neural activations across the entire brain by utilizing a so-called blood oxygen level dependent (BOLD) contrast derived from a combination of hemodynamic and metabolic responses (Ogawa et al., 1990). Functional MRI has rapidly gained a prominent position in neuroscience research owing to its excellent spatial resolution and specificity. However, it is also generally accepted that fMRI is not suitable for studying the temporal aspect of rapid neuronal events since the hemodynamic response evolves in seconds rather than milliseconds (Boynton et al., 1996). In the past decade, the complementary advantages of EEG/MEG and fMRI have attracted great interests to integrate these modalities in an attempt to provide a multimodal neuroimaging technique with millimeter spatial and millisecond temporal resolutions (Liu et al., 1998; Dale and Halgren, 2001; Liu et al., 2006a). Furthermore, the feasibility of simultaneously recording EEG and fMRI makes EEG more preferable than MEG in terms of being integrated with fMRI (Allen et al., 1998, 2000; Goldman et al., 2000; Bonmassar et al., 2001; Gotman et al., 2002; Im et al., 2006).

The principle of fMRI-EEG integration lies in the linkage between hemodynamic response and neural activity. Early studies (Rees et al., 2000; Heeger et al., 2000) have found a linear relationship between human fMRI response and primate neuronal spike activity in the visual areas (V5 and V1). Nevertheless, important findings obtained from simultaneously recorded BOLD and intracranial electrical signals on primates suggest that the BOLD response is linearly correlated with the power of local field potential (LFP), which represents the synchronized synaptic inputs of a given neural population (Logothetis et al, 2001; Logothetis, 2002). Since then, evidence has increasingly suggested that the BOLD fMRI signal primarily reflects synaptic activity rather than neuronal spike activity (Arthurs and Boniface, 2002; Lauritzen and Gold, 2003; Martindale et al., 2003). This is also in light of the fact that the hemodynamic response is driven by the metabolic energy demand, nearly all of which is imposed by synaptic activity instead of action potential firing (Mathiesen et al., 1998; Arthurs and Boniface, 2002). The observed correlation between the BOLD signal and the neuronal spiking rate (Rees et al., 2000; Heeger et al., 2000) might be explained by the post-synaptic current flow in correlation with the spike activity of the pre-synaptic neurons (Heeger and Ress, 2002; Arthurs and Boniface, 2002). In addition, the energy contained in the synaptic current flow (proportional to the square of current density or LFP) can be further thought of as the physical energy correlate of metabolic energetics. In line with the pioneering work done by (Logothetis et al. 2001), another recent study using the visual stimulation with different contrasts and frequencies suggests that the BOLD fMRI response varies linearly with the power (instead of the magnitude) of the current source activity summed over the entire stimulus duration (Wan et al., 2006). In consideration of the above evidences together, it is reasonable to assume that regions in the brain that show increased BOLD responses are also on average more electrically active over time (Dale and Sereno, 1993).

Under this assumption, fMRI activation maps resulting from statistical analyses of fMRI time series are typically used as a priori information regarding where the EEG sources are likely located. Depending on different EEG source models, the fMRI map can be used to constrain the locations of multiple current dipoles, namely the fMRI-constrained dipole fitting (Ahlfors et al., 1999; Korvenoja et al., 1999; Fujimaki et al., 2002; Vanni et al., 2004), or to constrain the distributed source distribution over the folded cortical surface or in the 3-D brain volume, namely the fMRI-constrained current density imaging (George et al., 1995; Liu et al., 1998; Dale et al., 2000; Wagner et al., 2000; Babiloni et al., 2005; Ahlfors and Simpson, 2004; Sato et al., 2004; Phillips et al., 2005; Liu et al., 2006b; Mattout et al., 2006).

While applications of dipole fitting techniques are often questionable when brain activities are spatially distributed and not confined to focal regions, the distributed source imaging has relative wider applicability. Existing methods for the fMRI-constrained current density imaging have been implemented under different frameworks such as Wiener estimation (Dale and Sereno, 1993; Liu et al., 1998; Dale et al., 2000), weighted minimum norm (Wagner et al., 2000; Babiloni et al., 2005; Ahlfors and Simpson et al., 2004), Bayesian estimation (Sato et al., 2004; Phillips et al., 2005; Mattout et al., 2006) and Twomey regularization (Liu et al., 2006b).

All these approaches have limitations in two important aspects. First, there is no generalized method to quantify the fMRI signals such that the resulting quantification has an explicit physical interpretation in the context of EEG source imaging (Liu et al., 2006b). It is perhaps such limitation that makes it difficult to develop a principled way of using the fMRI data in solving the EEG inverse problem. Moreover, when the prior spatial constraint is derived from the fMRI activation map after applying a statistical threshold as in most existing methods, the fMRI-constrained source reconstruction is also subject to the choice of threshold as well as various methods of fMRI analysis (Ahlfors and Simpson, 2004). Secondly but more importantly, an fMRI-derived “time-invariant” spatial constraint (Lin et al., 2006) is applied when imaging the temporally variable current source distribution that accounts for the event-related potentials (ERP) during the entire period of interest. However, such a time-invariant spatial constraint may entail both fMRI false positives and false negatives, as a result of possible mismatches between locations of fMRI activations and instantaneous source activities, namely the fMRI-EEG mismatches.

Most of the fMRI-EEG mismatches are fundamentally caused by the highly different temporal scales in which the fMRI and EEG data are generated and collected. Neural activities evolve so fast that the brain function is always carried out readily. In response to a single “event”, the evoked neural activity is most substantial within a very short period of time ranging from tens to hundreds milliseconds. Since neuronal events are accompanied with instantaneous electrical responses, the scalp EEG signals collected with a sufficiently fast sampling rate carry the information about the “current” status of underlying neural activities (Nunez and Srinivasan, 2005). On the other hand, the neuronal activity also induces delayed and sluggish hemodynamic responses measured by fMRI. It has been shown that the change of BOLD fMRI signals only happens about 3 seconds after the event onset (Boynton et al., 1996). Since such a delay is significant relative to the short duration of neural activity, one can only infer from the fMRI data the “past” status of neural activity. The sluggish hemodynamics may also be appreciated by considering the neurovascular coupling system as a low-pass filter (or a temporal point spread function) (Logothetis et al., 2001; Friston et al., 1994). The rapid neural dynamics is effectively smoothed out, and the fMRI response accordingly reflects the energetic effect of neural activity averaged (or accumulated) over time (Boynton et al., 1996; Mathiesen et al., 1998; Martindale et al., 2003; Wan et al., 2006). Lastly, the fMRI data acquisition is also limited by the scanning speed (typically up to 50 ms per slice), which is often too slow to probe the temporal aspect of neural activity. In short, the spatial locations of fMRI activations cannot be simply equated with those of electrical activities at every millisecond considering that the fMRI has much lower temporal resolution and specificity than the EEG.

The fMRI-EEG mismatches can be further categorized into two types, namely fMRI extra sources and fMRI invisible sources (Liu et al., 1998; Wagner et al., 2000; Liu et al., 2006b). The fMRI extra sources represent the source regions deemed as active in fMRI but are not the EEG sources at certain time instants. During a short period of interest following the event onset, the fMRI activations have to be thought of as “static” (or time-invariant) while the EEG signals are variable and the source imaging is carried out instant by instant. Electrical source activities at certain time instants may only involve a subset of the activated fMRI areas, whereas other areas may appear as false positives if including them all as time-invariant spatial constraints (Liu et al, 2006b). The fMRI invisible sources are the real EEG sources but not deemed as active by fMRI. A transient current source may generate observable EEG signals whereas it may last too briefly to induce a sustained BOLD response. Under this condition, the fMRI-derived time-invariant spatial constraint includes false negatives, which often result in the underestimation of fMRI invisible sources as reported in several independent studies (Liu et al., 1998; Liu et al., 2006b). Therefore, dealing with such unavoidable spatiotemporal mismatches is essential to establishing a reliable fMRI-EEG integrated neuroimaging.

In this paper, we introduce a new theoretical framework on the integration of fMRI and EEG. This new framework distinguishes itself from other existing approaches by 1) using a unified quantification of BOLD-fMRI responses that can be interpreted as proportional to the time integral of EEG source power (or source variance with respect to a zero mean), and 2) deriving a set of “time-variant” spatial constraints in the form of source covariance matrices by fusing data from both fMRI and EEG. The derived time-variant source covariance matrices are incorporated into solving the instantaneous EEG inverse problem by means of the Wiener filter. In contrast to the conventional Wiener filter method with a time-invariant source covariance matrix (Dale and Sereno, 1993; Liu et al., 1998; Dale et al., 2000; Lin et al., 2006), we shall refer to the proposed approach as the “adaptive Wiener filter”, considering the time-variant source covariance matrix is derived in a data-driven manner.

In both computer simulation and experimental settings, we evaluated the performance of the proposed method, as opposed to the inverse solution based on the EEG alone (Hämäläinen and Ilmoniemi, 1984) and the fMRI-constrained inverse solution based on the Wiener estimation with a constant spatial prior (Liu et al., 1998; Dale et al., 2000). In our simulations, effects of both fMRI false positives and false negatives were investigated. The estimation accuracy under typical or complicated source temporal dynamics was also assessed. In our experimental investigation, the spatiotemporal cortical activity responding to a unilateral visual stimulus was imaged by using three algorithms in comparison, i.e. the minimum norm estimation (Hämäläinen and Ilmoniemi, 1984) based on the visual evoked potential (VEP) alone, or using the proposed adaptive Wiener filter and the conventional 90% fMRI weighted algorithm (Liu et al, 1998) based on both fMRI and EEG.

METHODS

Quantification of BOLD-fMRI signals

Since an fMRI experiment is often conducted in a block-design manner, we first derive a quantification method based on the BOLD-fMRI response evoked by a block of repeated stimuli, and then generalize the method to other experimental designs such as the event-related design.

Assume a single stimulus at time 0 evokes synaptic current sources s⃑(r,t) (where r indicates location in the brain and t indicates time) that last for a very short period T_s typically ranging from several tens to several hundreds milliseconds. Then the source activity evoked by a block of N stimuli, denoted as g⃑(r,t), can be written as Eq. (1).

$$\stackrel{\rightharpoonup }{\mathbf{g}}(\mathbf{r},t)=\sum _{n=1}^N\delta (t-n{T}_{\mathit{ISI}})\ast \stackrel{\rightharpoonup }{\mathbf{s}}(\mathbf{r},t)$$ (1)

where δ(t) is a delta function, T_ISI is the inter-stimulus-interval (ISI) and * denotes convolution. Eq. (1) is valid when the ISI is longer than the neuronal refractory period. It is particularly so in most ERP studies, in which the ISI is designed to be even longer than T_s in order to ensure that the electrical activity responding to preceding stimuli recovers to the resting state (or the baseline) before the response to next stimulus is elicited.

Assume the induced BOLD-fMRI response f(r,t) relates to the power of local synaptic current by a linear system that is characterized by the hemodynamic impulse response function (HRF) h(t), plus noise. Such a linear system is illustrated in Fig. 1 and mathematically expressed as Eq. (2) to Eq. (4):

$$f_s(\mathbf{r},t)=g^2(\mathbf{r},t)\ast h(t)$$ (2)

$$f_s(\mathbf{r},t)=\sum _{n=1}^N\delta (t-n{T}_{\mathit{ISI}})\ast s^2(\mathbf{r},t)\ast h(t)$$ (3)

$$f(\mathbf{r},t)=f_s(\mathbf{r},t)+f_n(\mathbf{r},t)$$ (4)

where g(r,t) and s(r,t) are the magnitudes of g⃑(r,t) and s⃑(r,t) respectively, f_s(r,t) is the “signal” part of the fMRI response and f_n(r,t) is the “noise” part.

Figure 1.

Illustration of the linear system that describes the relationship between electrophysiological signals and BOLD-fMRI responses under a train of repeated stimuli. See the detailed description in the text.

In terms of describing the relationship between the neural activity and the BOLD fMRI response, the model being assumed as Eq. (2) is similar to a linear transform model that serves as the theoretical core of the widely used fMRI statistical parametric mapping (Friston et al, 1994; Boynton et al, 1996; Dale and Buckner, 1997). It should be noted that here we input the neural activity assessed by the power of synaptic current flow to the linear system (i.e. HRF) (Logothetis et al., 2001; Wan et al., 2006; also see Introduction), instead of using the stimulus function.

Note that Eq. (3) can be re-organized as Eq. (5):

$$f_s(\mathbf{r},t)=s^2(\mathbf{r},t)\ast \sum _{n=1}^N\delta (t-n{T}_{\mathit{ISI}})\ast h(t)$$ (5)

We define a predictor signal p(t) as Eq. (6).

$$p(t)=\sum _{n=1}^N\delta (t-n{T}_{\mathit{ISI}})\ast h(t)$$ (6)

Then, Eq. (5) can be further re-written as Eq. (7) and Eq. (8).

$$f_s(\mathbf{r},t)=s^2(\mathbf{r},t)\ast p(t)$$ (7)

$$f_s(\mathbf{r},t)=\underset{T_s}{\int }{s}^2(\mathbf{r},\tau )p(t-\tau )d\tau$$ (8)

Since the hemodynamic impulse response h(t) evolves much slower than the electrophysiological impulse response, it keeps approximately constant over the duration of the electrical source signal evoked by a single stimulus. Mathematically, we assume p(t) ≈ p(t − τ) for 0 ≤ τ ≤ T_s. Then Eq. (8) can be simplified as Eq. (9)

$$f_s(\mathbf{r},t)=p(t)\underset{T_s}{\int }{s}^2(\mathbf{r},\tau )d\tau$$ (9)

We define β(r) as Eq. (10) and re-write Eq. (9) as Eq. (11)

$$\beta (\mathbf{r})=\underset{T_s}{\int }{s}^2(\mathbf{r},t)dt$$ (10)

$$f_s(\mathbf{r},t)=\beta (\mathbf{r})p(t)$$ (11)

For a time series of fMRI response with N_f discrete samples over time, it immediately follows from Eq. (4) and Eq. (11) that an over-determined linear regression model can be described using a vector notation as Eq. (12).

$$\mathbf{F}(\mathbf{r})=\beta (\mathbf{r})·\mathbf{P}+{\mathbf{F}}_n(\mathbf{r})$$ (12)

where F(r) = [f(r,1) f(r,2)… f(r, N_f]^T and F_n(r) = f_n(r,1) f_n(r, 2)… f_n(r, N_f)]^T are a vector of the fMRI time series and its “noise” part respectively, and P = [p(1) p(2)… p(N_f)]^T is a vector of the predictor signal.

Based on Eq. (12), the minimum least-squares estimate of β(r), denoted as β̂(r), can be computed from Eq. (13)

$$\widehat{\beta }(\mathbf{r})={({\mathbf{P}}^{\text{T}}\mathbf{P})}^{-1}{\mathbf{P}}^{\text{T}}\mathbf{F}(\mathbf{r})$$ (13)

As expressed in Eq. (13), β̂(r) is a quantity that exclusively depends upon the BOLD response, the experimental protocol and the hemodynamic impulse response function. More importantly, β̂(r) can be interpreted, by the definition of β(r) in Eq. (10), as an estimate of the time integral of current source power over the period of T_s. This interpretation is meaningful in the context of electrical source imaging and is explicitly used later when incorporating the fMRI into solving the EEG inverse problem.

The above method can also be applied to the quantification of the event-related fMRI response, simply by changing the predictor function in Eq. (6) to Eq. (14).

$$p(t)=\delta (t)\ast h(t)=h(t)$$ (14)

The above method can be further generalized to more complicated experimental designs with multiple or mixed tasks. Correspondingly, multiple predictor functions need to be defined by convolving the mathematically expressed experimental protocol with the HRF. If the quantification of BOLD responses has a multivariate nature, the regression parameters should be defined for each stimulus or task independently. And we can set up a general linear model (GLM) (Friston et al., 1995) similarly as Eq. (12) and compute the regression parameters using a multivariate linear algorithm.

EEG inverse problem of cortical source imaging

As originally proposed in (Dale and Sereno, 1993), the EEG source space is constrained to a realistically-shaped cortical surface extracted from high-resolution T1-weighted magnetic resonance (MR) images. For the source modeling, the folded cortical surface can be extracted from the segmented boundary between the gray matter and the white matter. The cortical current density distribution is modeled by thousands of current dipoles evenly placed on the folded cortical surface, and the orientation of each dipole is constrained to be perpendicular to the local cortical patch (Dale and Sereno, 1993; Liu et al., 2006b). For the volume conductor modeling, the head model contains 3 compartments (scalp, skull and brain) and the electrical conductivity is assumed to be piece-wise homogeneous. After tessellating 3 surfaces that separate these compartments, the boundary element method (BEM) (Hämäläinen and Sarvas, 1989) can be employed to solve the EEG forward problem.

As a result, a transfer matrix A can be numerically computed to link the underlying current source distribution, s(t), to the recorded scalp potentials, x(t), under the existence of measurement noise, b(t), for every time instant t.

$$\mathbf{x}(t)=\mathbf{A}\hspace{0.17em}\mathbf{s}(t)+\mathbf{b}(t)$$ (15)

where A is a N_x-by-N_s matrix (N_x is the number of EEG sensors, N_s is the number of current sources), s(t) is a N_s-by-1 vector, x(t) and b(t) are N_x-by-1 vectors.

At any time instant, the spatial vectors s(t), x(t) and b(t) can be viewed as stochastic processes with their index sets over source or sensor locations. Assuming both s(t) and b(t) have zero means, we define the source auto-covariance matrix and the noise auto-covariance matrix as Eq. (16) and Eq. (17), respectively.

$${\mathbf{C}}_{\mathbf{s}}(t)=E\left[\mathbf{s}(t){\mathbf{s}}^{\text{T}}(t)\right]$$ (16)

$${\mathbf{C}}_{\mathbf{b}}(t)=E\lfloor \mathbf{b}(t){\mathbf{b}}^{\text{T}}(t)\rfloor$$ (17)

If a priori information is given to both C_s(t) and C_b(t), a linear inverse operator G(t) as Eq. (18) can be used to estimate s(t) from x(t), denoted as ŝ(t).

$$\mathbf{G}(t)={\mathbf{C}}_{\mathbf{s}}(t){\mathbf{A}}^{\text{T}}{(\mathbf{A}{\mathbf{C}}_{\mathbf{s}}(t){\mathbf{A}}^{\text{T}}+{\mathbf{C}}_{\mathbf{b}}(t))}^{-1}$$ (18)

$$\widehat{\mathbf{s}}(t)=\mathbf{G}(t)\mathbf{x}(t)$$ (19)

In practice, the noise covariance matrix, C_b(t), can be estimated directly from the EEG data (Fuchs et al., 1998), whereas a priori knowledge of C_s(t) is usually unavailable and hence it is typically assumed to be proportional to an identity matrix (Hämäläinen and Ilmoniemi, 1984) or a spatial Laplacian operator (Pascual-Marqui et al., 1994). It has also been demonstrated that the Wiener filter in Eq. (18) has an equivalent formulation of weighted minimum norm (Hauk, 2004; Liu et al., 2006a).

It is worthwhile to emphasize that in the Wiener filter formulation as Eq. (18), the stochastic process is referred to the spatial domain instead of the time domain and the source covariance matrix is accordingly assumed to be time-variant.

fMRI-EEG integrated adaptive Wiener filter

Assuming there is no a priori cross-correlation between different source locations, the time-variant source covariance matrix can be expressed as a diagonal matrix as Eq. (20).

$${\mathbf{C}}_{\mathbf{s}}(t)=\mathit{diag}({\sigma }_1(t),{\sigma }_2(t),\dots ,{\sigma }_{N_s}(t))$$ (20)

Recall that, at any source location, the quantification of BOLD-fMRI response as previously described represents an estimate of the time integral of source variance (see definition in Eq. (16)) over the ERP time period. Therefore, we introduce the fMRI-derived spatial constraints to the sum of source variances, as expressed by Eq. (21).

$$\sum _{t\in T_s}{\sigma }_i(t)={\widehat{\beta }}_i\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }(1\le i\le N_s)$$ (21)

where β̂_i is the quantified fMRI response at the i-th source location.

However, Eq. (21) alone is insufficient to obtain the time-variant source covariance matrix at every time instant. The reason is obvious because the temporal variation of source variance cannot be resolved by fMRI due to its lack of temporal resolution.

We further use a data-driven approach described as below to retrieve the temporal variation of the source variances based on the segmented EEG data before response averaging. This approach is based on a common practice in ERP experiments that the electrical response is repeatedly induced by a train of stimuli (or tasks). The EEG response in each repetition is referred to as an epoch that is usually time-locked to the onset of the stimulus.

Denote x(t,k), s(t,k) and b(t,k) as the random column vectors of EEG recordings (N_x-by-1), source signals (N_s-by-1) and measurement noise (N_x-by-1) at the time t and in the k-th epoch. A valid assumption is that source signals in different epochs represent independent observations of the identical stochastic process, s(t), for any specific time t, as illustrated in Fig. 2. Accordingly, we can compute an inverse estimate of the instantaneous source auto-covariance if the number of epochs is sufficiently large.

Figure 2.

The source signal at any i-th location s_i(t) is not a stationary stochastic process over time, but its values at any give time t_j sampled in different epochs are i.i.d. Therefore s(t,k), for 1 ≤ k ≤ N_e, represents an independent observation of a stochastic process s(t) for each individual time t(note that s(t) is a stochastic process with its index sets over source locations).

By collecting signals at the time t from all N_e epochs, we define Eq. (22) to (24):

$$\mathbf{X}(t)=[\mathbf{x}(t,1),\mathbf{x}(t,2),\dots ,\mathbf{x}(t,N_e)]$$ (22)

$$\mathbf{S}(t)=[\mathbf{s}(t,1),\mathbf{s}(t,2),\dots ,\mathbf{s}(t,N_e)]$$ (23)

$$\mathbf{B}(t)=[\mathbf{b}(t,1),\mathbf{b}(t,2),\dots ,\mathbf{b}(t,N_e)]$$ (24)

Then, we re-write the forward model in Eq. (15) in a matrix notation as Eq. (25).

$$\mathbf{X}(t)=\mathbf{A}\hspace{0.17em}\mathbf{S}(t)+\mathbf{B}(t)$$ (25)

The singular value decomposition (SVD) of X(t) is written as Eq. (26)

$$\begin{array}{l}\mathbf{X}(t)=\mathbf{U}(t)\mathbf{\Lambda }(t){\mathbf{V}}^{\text{T}}(t)\\ =\sum _{q=1}^{N_q}{\lambda }_q(t){\mathbf{U}}_q(t){\mathbf{V}}_q^{\text{T}}(t)\end{array}$$ (26)

where N_q = min(N_x,N_e ), the singular vector U_q(t) is the q-th column vector in the matrix U(t) that represents a spatial component of scalp potentials at the time t, the singular vector V_q(t) is the q-th column vector in the matrix V(t) that represents the variation (over epochs) of the corresponding spatial component U_q(t).

We truncate the spatial components that do not satisfy the discrete Picard condition (Hansen, 1990), since the truncated spatial components are not sufficiently smooth to be associated with any source activity and hence are dominated by noise perturbation.

After the truncation, we compute the maximum likelihood estimates (MLE) of distributed sources that account for each of the Q remaining spatial components, written as Eq. (27).

$${\stackrel{\sim }{\mathbf{s}}}_q(t)={\mathbf{A}}^{\text{T}}{(\mathbf{A}{\mathbf{A}}^{\text{T}}+r_q\hspace{0.17em}\mathbf{I})}^{-1}{\mathbf{U}}_q(t)\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }(1\le q\le Q)$$ (27)

where s̃_q(t) is an N_s-by-1 vector that represents the estimated source distribution underlying the q-th spatial component U_q(t) at the time t, and the regularization parameter r_q can be obtained by using the “L-curve” method (Hansen, 1992).

Combining the MLE for all Q components, we can obtain a set of source estimates for the time t in all N_e epochs, collectively denoted as S̃(t).

$$\stackrel{\sim }{\mathbf{S}}(t)=\sum _{q=1}^Q{\lambda }_q(t){\stackrel{\sim }{\mathbf{s}}}_q(t){\mathbf{V}}_q^{\text{T}}(t)$$ (28)

where S̃(t) is an N_s-by-N_e matrix with each row representing the variation of estimated source signals over epochs.

Following the definition in Eq. (16), the variance of source estimates at the i-th location for the time t, denoted as σ̃_i(t), can be written as Eq. (29).

$${\stackrel{\sim }{\sigma }}_i(t)=\frac{1}{N_e}{\stackrel{\sim }{\mathbf{S}}}_i(t){\stackrel{\sim }{\mathbf{S}}}_i^{\text{T}}(t)$$ (29)

where S̃_i(t) is the i-th row vector in the matrix S̃(t).

Substitute S̃(t) by Eq. (28) and re-write Eq. (29) as Eq. (30).

$$\begin{array}{l}{\stackrel{\sim }{\sigma }}_i(t)=\frac{1}{N_e}\left(\sum _{q=1}^Q{\lambda }_q(t){\stackrel{\sim }{\mathbf{s}}}_{iq}(t){\mathbf{V}}_q^{\text{T}}(t)\right){\left(\sum _{q=1}^Q{\lambda }_q(t){\stackrel{\sim }{\mathbf{s}}}_{iq}(t){\mathbf{V}}_q^{\text{T}}(t)\right)}^{\text{T}}\\ =\frac{1}{N_e}\left(\sum _{q=1}^Q\sum _{p=1}^Q{\lambda }_p(t){\lambda }_q(t){\stackrel{\sim }{\mathbf{s}}}_{ip}(t){\stackrel{\sim }{\mathbf{s}}}_{iq}(t){\mathbf{V}}_p^{\text{T}}(t){\mathbf{V}}_q(t)\right)\end{array}$$ (30)

Since ${\mathbf{V}}_q^{\text{T}}(t){\mathbf{V}}_q(t)=1$ and ${\mathbf{V}}_p^{\text{T}}(t){\mathbf{V}}_q(t)=0$ for any p ≠ q, Eq. (30) can be simplified as Eq. (31).

$${\stackrel{\sim }{\sigma }}_i(t)=\frac{1}{N_e}\sum _{q=1}^Q{({\lambda }_q(t){\stackrel{\sim }{\mathbf{s}}}_{iq}(t))}^2$$ (31)

Repeat the above procedures for all the time points during the period T_s, a time course of source variance estimates can be computed for every source location i, collectively denoted as a vector σ̃_i as Eq. (32).

$${\stackrel{\sim }{\mathbf{\sigma }}}_i=[{\stackrel{\sim }{\sigma }}_i(1),{\stackrel{\sim }{\sigma }}_i(2),\dots ,{\stackrel{\sim }{\sigma }}_i(T_s)]$$ (32)

Clearly, Eq. (32) contains the temporal information regarding the variation (over time) of the source variance inversely estimated from the EEG epochs, and Eq. (21) represents the spatial information regarding the time integral of source variance as quantified from independent fMRI measurements. Combining Eq. (32) and Eq. (21), we derive the prior estimates of source variance from both fMRI and EEG data, as expressed by Eq. (33).

$${\sigma }_i(t)=\frac{{\stackrel{\sim }{\sigma }}_i(t){\widehat{\beta }}_i}{{\displaystyle \sum _{t\in T_s}}{\stackrel{\sim }{\sigma }}_i(t)}$$ (33)

Once σ_i(t) is calculated for every source location i and every time point t, it becomes immediately straightforward to compute the time-variant source covariance matrices based on Eq. (20) and estimate the cortical source distribution based on Eq. (18) and Eq. (19) for every time point individually. Qualitatively speaking, a larger value of source variance at a specific location means that the inverse solution is more preferentially located at that location.

In contrast to the conventional Wiener filter approach (Liu et al., 1998; Dale et al., 2000) based on time-invariant spatial constraints, the method described as above is called the “adaptive Wiener filter” in the sense that the time-variant source covariance matrix is derived from the combination of fMRI and EEG in a data driven manner.

Computer simulations

The performance of the proposed method was evaluated in a series of computer simulations. The simulation setting was based on a realistically-shaped head volume conductor and cortical surface models that were constructed from high resolution T1-weighted MR images of a human subject (256 slices, matrix size: 256×256, voxel size: 1×1×1 mm³). The electrical conductivities of the scalp, skull and brain were set to be 0.33, 0.0165, and 0.33 S/m, respectively (Lai et al, 2005; Oostendorp et al, 2000; Zhang et al, 2006). The cortical current density source model consisted of around 7,000 current dipoles evenly placed on the cortical surface. 128 electrodes with a standard montage were co-registered to the boundary element head model. Fig. 3 shows the head model, cortical surface model and the electrodes used in the simulation setting.

Figure 3.

Realistic cortical source model and head volume conductor model. The head model consists of 3 triangulated boundary meshes (shown in dark gray). 128 electrodes (shown in purple) are coregistered to be over the scalp surface. The cortical source model is on the folded cortical surfaces (the left hemisphere shown in blue and the right hemisphere shown in brown).

Based on the above settings, we simulated current source signals, BOLD fMRI signals and EEG recordings. We computed the inverse solutions from the simulated data using three algorithms. The detailed procedures are described step by step as follows.

• 1) Current source activities. 3 current dipoles were placed on 3 selected locations on the folded cortical surface. To simulate the source activity, representative waveforms were assigned to these dipole sources such that the source signals were simulated to be temporally uncorrelated or correlated, or to be transient or sustained, or to have the same frequency but different phases or the same phase but different frequencies, or to have time courses obtained in a real experiment.

• 2) BOLD-fMRI responses. The BOLD-fMRI signals were generated from the simulated current sources. The source waveforms were repeated every 500 ms for a total of 30 s. For each source location, the BOLD signal was simulated by convolving the time course of source power with a gamma-function HRF suggested by (Boynton et al., 1996), written as Eq. (34).

  $$h(t-\epsilon )=\frac{{(t/\tau )}^{n-1}{e}^{-(t/\tau )}}{\tau (n-1)!}$$ (34)

  where n = 3, τ = 1.25 sec and ε = 2.5 sec as in the default settings of a widely used fMRI analysis software - BrainVoyager QX (BrainInnovation, Netherlands).

The spatial distribution of the simulated BOLD signals was then convoluted with a gaussian spatial kernel with a given full-width-half-maximum (FWHM). Here the FWHM was chosen to be 4 mm, which represented a reasonable value consistent with the known spatial resolution of fMRI according to previous experimental studies (Engel et al., 1997). Gaussian white noise (GWN) was added into the simulated BOLD time courses so that the BOLD response had a signal-to-noise ratio (SNR) equal to 10.

• 3) EEG recordings. From the simulated spatiotemporal current source distributions, we further simulated scalp EEG signals at 128 electrodes using the BEM-based forward calculation (Hämäläinen and Sarvas, 1989; He et al., 1987). 250 trials of EEG data were simulated as the summation of the forward solution and GWN. The SNR in each trial was around 0.6 and the ERP signal averaged from all the trials had a SNR equal to 10.

• 4) Source imaging. Finally, we reconstructed the spatiotemporal source distribution based on the simulated EEG alone using the minimum-norm algorithm (Hämäläinen and Ilmoniemi, 1984), or based on both the simulated fMRI and EEG using time-invariant (Liu et al., 1998; Dale et al., 2000) or time-variant spatial constraints (as proposed in this paper).

Visual Stimulation Experiment

To assess the applicability of the proposed adaptive Wiener filter algorithm, we also employed real fMRI and EEG data collected during a preliminary experiment with a unilateral checkerboard visual stimulus.

The experiment was conducted with a healthy subject (male, age 22) under the approval of the institutional review board (IRB) at the University of Minnesota. Informed consent was obtained from the subject before the experiment. The experiment included 2 separate sessions with the identical visual stimuli for the EEG and fMRI data collection respectively. The visual stimulation was a rectangular checkerboard within the lower left quadrant of the visual field; the checkerboard pattern was reversed at 2 Hz. In the EEG experiment, 6-second breaks without stimulation were randomly inserted into the otherwise 4-minute continuous visual presentation, such that the subject had a break about every 20 seconds on average to avoid the neural adaptation. In the fMRI experiment, the visual stimuli were presented in six 30-second blocks separated by seven 30-second resting blocks without stimulation. The stimuli delivered through a DLP projector were back-mirrored to the subject inside the MRI scanner. For both EEG and fMRI experiments, the subject was instructed to always gaze at a central fixation point.

The EEG signals collected from a 64-channel system (BrainAmp MR 64 plus, BrainProducts, Germany) with a 1000-Hz sampling rate were sequentially subject to the ocular artifact rejection, band-pass filtering (0.3 – 40 Hz), segmentation with respect to the stimulus onsets, pre-stimulus baseline correction, linear trend removal. After these preprocessing steps, the data in 380 segmented epochs was averaged to yield the visual evoked potentials (VEP). The anatomical MRI and fMRI data were collected in a 3-T MRI system (Siemens Trio, Siemens, Germany). The whole-head T1-weightd MR images (matrix size 256×256, 1mm slice thickness) were acquired using the Turboflash sequence (TR/TE = 20 ms/5 ms). The T2*-weighted fMRI data was acquired from 16 axial slices (matrix size 64×64, 5mm thickness) covering the visual cortex using the echo planar imaging (EPI) sequence (TR/TE = 1000 ms/35 ms). The MRI and fMRI data were analyzed using BrainVoyager QX (Brain Innovation, Netherlands). The EPI volumes underwent several preprocessing steps including three-dimensional motion correction, slice scan time correction and linear trend removal. Then, the fMRI data was aligned with the anatomical MR images. The fMRI activation map was obtained by statistical analysis using a general linear model (Friston et al., 1994).

Three cortical current density imaging algorithms were applied to image the cortical responses on the hemisphere (right) contralateral to the stimulation (left). The 90% fMRI-weighted Wiener estimation (Liu et al., 1998; Dale et al., 2000) and the proposed algorithm used both EEG and fMRI data, whereas the minimum-norm algorithm used the VEP data lone.

RESULTS

BOLD-fMRI simulation and quantification

Fig. 4 shows the simulated BOLD fMRI signals and the quantified fMRI maps. The simulated fMRI responses were induced by a 30-sec block of stimuli with an ISI of 500 ms (Fig. 4.A). The electrical current sources were located at the lateral occipital sulcus (blue), the medial occipital dorsal area (red) and the intraparietal sulcus (green) on the right hemisphere (Fig. 4.B, right). Representative source locations were selected to be within the sulcal fundus, the sulcal wall and the gyral crown, respectively. The source waveforms shown in corresponding colors were simulated as 3 temporally uncorrelated gauss functions (Fig. 4.B left). After applying a 2-D spatial gauss smoother with a given FWHM (Fig. 4.C), the simulated BOLD responses emerged within 3 extended regions. The noise-contaminated fMRI signals surrounding the “red” source are plotted in Fig. 4.D). These time courses had a similar shape as the predictor signal (Fig. 4.E) such that they could be fitted with the predictor signal simply after scaling. The quantified fMRI response at each cortical location was precisely the scaling factor that allowed for the best fit between the scaled predictor signal and the measured BOLD response. This scaling factor was also found approximately proportional to the averaged BOLD signal change at the steady state with stimuli relative to the resting state without stimuli.

Figure 4.

A) Stimulation protocol, B) current source waveforms evoked by a single stimulus and the source locations on the cortex; the 3 sources are indicated by different colors, C) a gauss function describing the spatial resolution of fMRI, D) simulated BOLD-fMRI time series at SNR=10, E) the predictor signal (the unfilled gray box indicates the duration of the stimulation block), F) the map of quantified fMRI responses, G) the quantified fMRI map with different fMRI spatial resolutions as described by the FWHM (2, 4, 6 mm).

The map of quantified fMRI responses is shown in Fig. 4.F). Three activated regions were revealed. The extent of these regions depended upon the fMRI spatial resolution (or specificity), which was inversely proportional to FWHM. With an increasingly larger FWHM, the fMRI response extended from a point source to a larger region, in which the quantified values of fMRI responses became gradually smaller than the time integral of corresponding source powers (Fig. 4.G). According to Eq. (33), the time-variant spatial constraint at any specific cortical location is proportional to the quantified fMRI response. Therefore, the observed effect of FWHM also implies that the lower specificity of fMRI give rises to weaker fMRI constraints to the EEG inverse solution.

Time-variant spatial constraints

Fig. 5 illustrates an example of probing the temporal change of source variance based on the segmented EEG epochs. To simplify the illustration, let us look closely at a period of time while a single source is activated.

Figure 5.

An example of the SVD analysis on the segmented epochs. A) The EEG data was simulated from a single source activity, with the source location on the cortex shown on the left and the source waveform shown on the right; B) the largest 3 SVD components of the data at the peak latency (40 ms) over all the epochs. The singular values are shown in the left column, the decomposed spatial components shown in the middle and the corresponding variations over epochs shown in the right; C) the plot of the largest singular value as a function of time; the spatial components associated with the largest singular value at 7 representative time points are displayed along the plotted time course.

The source location and waveform is displayed in Fig. 5.A). Since the ERP signal after response averaging had a SNR of 10, the EEG data in every epoch had an average SNR of 0.6. Fig. 5.B) shows the SVD components obtained from the data at the 40-ms latency in all the epochs. Only 3 components with largest singular values are shown. The singular value for the first decomposed component was found to be much larger than those for the second and third components. Clearly, the first spatial component¹ more likely represented the scalp potential field generated by internal brain sources, whereas the second and third spatial components attributed to external recording noises. The first temporal component², which reflected the variation of the global field strength over epochs, had a DC offset with small variation. However, dramatic oscillations around 0 were observed in the second and third temporal components. Based on these observations, we inferred that the first SVD component contained the scalp potential field arising from the source activity at 40 ms, as well as its variation over epochs. Applying the discrete Picard condition also ended up with the same conclusion that only the first component at this time instant satisfied the Picard condition while all other components should be truncated.

For a better understanding of this conclusion and its implication, it is worthwhile to mention an important feature of SVD. With the singular value gradually decaying from its maximum value toward zero, both the left and right singular vectors necessarily tend to have more and more oscillations. Therefore, if the signal is less variable over epochs than the noise, which are the case in practical ERP studies, then the SVD component(s) with the largest singular value(s) mostly reflect the variance of source activities instead of that of the noise. On the other hand, these components also tend to be spatially smooth and more likely to satisfy the Picard condition. Accordingly, the DC line shown in Fig. 5.B) indicates that the source signal at 40 ms is repeated over epochs with little variation.

After repeating the SVD analysis to every time instant, we plotted the largest singular value as a function of time shown in Fig. 5.C). For several representative time points (from 10 to 70 ms stepped by 10 ms), the corresponding first spatial component was also displayed along the plotted time course. The source amplitudes at 10 and 70 ms were very small as shown in Fig. 5.A). The decomposed spatial components at these 2 latencies also appeared to be too noisy to satisfy the Picard condition. At other selected time points, the spatial patterns appeared to be similar and the resultant inverse solutions of these spatial patterns should also be spatially consistent to each other. These results demonstrate that the spatial component(s) with largest singular value(s), if satisfying the Picard condition, reflect the potential fields projected from the source space to the sensor space and contain the information on where the source is most likely located at the latency of interest.

By comparing Fig. 5.C) with Fig. 5.A), we also noticed that the temporal change of largest singular values was in general agreement with the time course of the real source amplitude. This finding further supported our previous argument that the principal SVD components nicely isolated the signal from the noise.

While the spatial component(s) with largest singular value(s) indicate the source locations at different latencies, the time course of largest singular values represents the temporal variation of source amplitude. In addition, the decomposed spatial components had the field strength equal to 1 because the singular vectors were always unitary vectors. According to Eq. (27) and Eq. (31), the absolute amplitudes of source estimates for the principal SVD components should be proportional to the largest singular values. Therefore, the similarity between Fig. 5.A) and 5.C) further justify the plausibility of estimating the time-variant source variance from the segmented epochs.

To assess the accuracies of the estimated time-variant source variances in both time and space, Fig. 6 summarizes the results of source variance estimation based on the EEG or the combination of fMRI and EEG. The source configuration and stimulation protocol used here were identical as in Fig. 4. 3 temporally uncorrelated (Fig. 6.B) sources were selected from the cortex (Fig. 6.A) to generate the BOLD fMRI and EEG signals. The quantified fMRI map with a 4-mm FWHM is shown in Fig. 6.D).

Figure 6.

A) Source locations, B) source waveforms, C) the spatial distribution on the cortex of the average MLE of source variance averaged over the entire time period, D) the map of quantified BOLD fMRI responses, E) the time courses of the MLE of source variances at three source locations respectively, F) the instantaneous cortical distributions of the MLE of source variances at 3 peak latencies (40, 100 and 160 ms) identified from E), G) the instantaneous cortical distributions of time-variant source variances derived from combination of fMRI and EEG at the 3 peak latencies.

The MLE of source variances were computed from the segmented epochs. Fig. 6.E) plots the time courses of the source variance MLE at each of 3 source locations respectively. The shape of these time courses agreed with the corresponding source waveforms with mild distinction. The maximal source variance MLE arrived at exactly the same latencies (40, 100 and 160 ms) as the real source activities. 2 “bumps” were observed in the estimated time course of the “green” source, which was caused by the cross-talks in the inverse solution from the “blue” and “red” sources respectively. However, the absolute values of the estimated source variances tended to be much smaller than those of the real source signals, except for the “red” source which was located at the gyral crown.

Fig. 6.F) shows the spatial distribution of the source variance MLE at 3 peak latencies (40, 100 and 160 ms) identified from Fig. 6.E). From the segmented EEG, the estimated map of source variances clearly spread out and the spatial maximum was not correctly localized to the original location of the source that was activated at the corresponding time instant. Fig. 6.C) shows the spatial distribution of the mean MLE of source variance averaged over the entire time period. This spatial distribution is qualitatively consistent with the quantified fMRI map (Fig. 6.D), although Fig. 6.C) exhibits spatial ambiguities and biases.

All these results based on the EEG alone demonstrate that the temporal change of source variances could be well retrieved from the segmented EEG epochs, but the absolute values of source variance were usually underestimated or significantly biased due to the ill-posed nature of the EEG inverse problem.

We further derived a set of time-variant spatial constraints by combining the “static” constraint from the fMRI and the “dynamic” constraint from the EEG based on Eq. (33). Fig. 6.G) shows the spatial distribution of the fMRI-EEG derived source variances at 3 peak latencies. In contrast to Fig. 6.D), Fig. 6.G) provided more accurate spatial prior constraints for each of 3 latencies respectively than the time-invariant spatial constraints based on the fMRI alone. The fMRI false-positives that occurred in the time-invariant fMRI constraints were successfully excluded from affecting the subsequent inverse solution. As opposed to Fig. 6.F), Fig. 6.G) also provided more localized and accurate prior constraints than those based on the EEG alone, by taking advantage of the high-spatial resolution of fMRI.

fMRI-EEG integrated source imaging with fMRI false positives

Following the example used in Fig. 6, we further imaged the cortical current density distribution at 3 peak latencies (40, 100 and 160 ms) using 3 algorithms: the proposed adaptive Wiener filter, the Wiener estimation using a time-invariant fMRI constraint and the minimum-norm estimation based on the EEG alone. Fig. 7.A) shows the reconstructed current density visualized on the cortical surface. Obviously, the use of time-variant spatial constraints in the adaptive Wiener filter algorithm resulted in the most accurate imaging results among all 3 algorithms. As shown in the top row in Fig. 7.A, 3 focal sources were revealed at each of 3 latencies respectively without showing any interference among them, in agreement with the fact that all 3 sources were temporally uncorrelated. The conventional Wiener filter also identified 3 sources with high spatial resolutions, benefiting from the use of fMRI spatial constraints. However, spurious sources were clearly observed at 40 ms and 100 ms. On the other hand, the minimum-norm solution without using any fMRI constraint had a low spatial resolution as often experienced in the EEG (or MEG) source imaging.

Figure 7.

Source distribution reconstruction and source waveform estimation under fMRI false positives. The source locations and simulated source waveforms are the same as used in Fig. 6. A) Reconstructed cortical current density distribution at 3 peak latencies (40, 100 and 160 ms) shown in 3 columns, by using 3 different algorithms: the adaptive Wiener filter (1^st row), the conventional Wiener filter using time-invariant spatial constraints (2^nd row) and the weighted minimum norm based on the EEG alone (3^rd row); B) Estimated source waveforms at each of 3 source locations color-coded as in Fig. 6. The estimated waveforms are shown in solid lines and the simulated original waveforms are shown in dashed lines. The 3 rows also correspond to 3 algorithms as in Fig. 7.A).

The time courses of source estimates at these 3 source locations using different algorithms are plotted (in solid lines) in Fig. 7.B) in comparison with the real source waveforms (in dashed lines). Note that the real and estimated waveforms are plotted in different scales as indicated at the left and right axes respectively. Among all 3 algorithms, the source waveforms reconstructed by using the adaptive Wiener filter were the closest in both shape and absolute amplitude to the real source waveforms. The estimated time courses using the conventional Wiener filter were clearly more accurate than the minimum-norm solution; however compared to the adaptive Wiener filter, obvious positive and negative “bumps” were observed obviously at 40 ms and slightly at 160 ms, which were essentially because of the fMRI false positive priors that amplified the cross-talks among sources in the inverse solution.

fMRI-EEG integrated source imaging with fMRI false negatives

We further investigated the performance of the proposed adaptive Wiener filter approach in dealing with the fMRI missing sources (i.e. fMRI false negatives). Again, 3 sources were selected from the cortical surface (see Fig. 8.A). The source waveforms consisted of 2 gauss functions (for the “red” and “green” sources) that were temporally uncorrelated and a short-duration pulse (for the “blue” source) with smaller peak amplitude (see Fig. 8.C).

Figure 8.

Source distribution reconstruction and source waveform estimation under fMRI false negatives. A) 3 source locations, B) quantified BOLD fMRI map, C) the simulated source waveforms, D) Reconstructed cortical current density distribution at 3 peak latencies (40, 100 and 160 ms) shown in 3 columns, by using 3 different algorithms: the adaptive Wiener filter (1^st row), the conventional Wiener filter using time-invariant spatial constraints (2^nd row) and the weighted minimum norm based on the EEG alone (3^rd row); E) Estimated source waveforms at each of 3 source locations color-coded as in Fig. 8.C). The estimated waveforms are shown in solid lines and the simulated original waveforms are shown in dashed lines. The 3 rows also correspond to 3 algorithms as in Fig. 8.D).

Since the “blue” source had an almost transient and smaller activity that was insufficient to induce a large and sustained BOLD response, we could not identify this source from the map of quantified BOLD responses (Fig. 8.B) and hence the “blue” source behaved as an fMRI invisible source. This fMRI invisible source became problematic in the conventional Wiener estimation of current density. At 40 ms when the “blue” source was activated and reached its peak amplitude, the source activity at (or around) the “blue” source location could not be imaged, as shown in the middle row of Fig. 8.D). Instead, the spurious activity at an instantaneous fMRI false positive region (around the “green” source) was observed. At the other 2 latencies, spurious source activities with small amplitudes still emerged in the estimated current density distribution when the Wiener filter algorithm with a time-invariant spatial constraint was used.

Although the “blue” source was transient, the EEG-based source imaging could still detect this source activity due to the high temporal resolution of EEG. But again, the imaged source distribution had a low spatial specificity. By taking full advantage of the temporal information from the EEG, the adaptive Wiener filter could clearly image and localize the “blue” source activity at 40 ms, as well as the other 2 source activities at 100 ms and 160 ms respectively.

The superiority of the adaptive Wiener filter was further confirmed by its ability to estimate the waveforms of both fMRI visible and invisible sources. As shown in Fig. 8.E), only the adaptive Wiener filter approach could recover the transient source activity in the estimated source time courses, without sacrificing the estimation accuracies to other sustained source activities.

Reconstructing source activities with complicated temporal dynamics

To assess the applicability of the proposed adaptive Wiener filter approach to reconstructing the source activities with complicated temporal dynamics, we simulated 3 source activities with a variety of temporal features. These 3 sources were located at the same locations as in the previous stimulations. Then we compared the source waveforms estimated by using the adaptive Wiener filter approach (in solid lines) with the originally simulated source waveforms (in dashed lines). The results are summarized in Fig. 9.

Figure 9.

Estimated source waveforms compared to the real source waveforms with different temporal features. 3 sources locations are the same as in Fig. 8. The originally simulated waveforms are shown in dashed lines and the estimated source waveforms are shown in solid lines.

In Fig. 9.A), all 3 sources had gauss-function time courses that were overlapped over time. The shapes of the reconstructed source waveforms were close to gauss functions and the onset and peak latencies were also consistent with the simulated real source activities. However, slightly more fluctuations were observed in the reconstructed waveforms than in the real source waveforms.

In Fig. 9.B), a transient source activity (“red”) was included in addition to two overlapping gauss-function waveforms (“blue” and “green”). The “red” source was also an fMRI invisible source which occurred at the time point when both the other two sources were also active. In this situation, the time courses of both the fMRI visible and invisible sources were also well reconstructed.

In Fig. 9.C), the “green” source was a transient source temporally overlapping with a sustained “red” source. The “red” source also had a short-duration peak around the same time when the sustained “blue” source reached its peak amplitude. Clearly in this situation, both the fMRI false positives and false negatives existed with and without overlaps over time. Nevertheless, we could still estimate the source waveforms that were consistently accurate for both the sustained and transient source activities.

In addition, we simulated 3 rhythmic activities that had the same frequency but different phases (Fig. 9.D) or the same initial phase but different frequencies (Fig. 9.E). In both situations, the frequency and phase information were well reflected in the estimated source waveforms without significant distortion, whereas we also observed distinctions in absolute amplitudes between the estimated and real source waveforms

In Fig. 9.F), we simulated the source activities by using the “realistic” dipole source waveforms estimated from a real ERP data set acquired in a visual experiment. In this case, the source activities had fairly complicated temporal features. The estimated source waveforms were highly correlated with the simulated real source waveforms, without loosing even detailed temporal dynamics.

The results in Fig. 9 all demonstrate good performances of the proposed adaptive Wiener filter algorithm to accurately estimate the source activities with a high temporal resolution, based on simulated fMRI and EEG data.

Experimental results

In response to the unilateral visual stimulation (Fig. 10.A), the activated cortical areas at the contralateral hemisphere were revealed in the fMRI activation map (Fig. 10.B). The fMRI activation map indicated a dorsal visual pathway covering V1, V2, dorsomedial areas (such as V3 and V7), intraparietal sulcus (IPS) as well as medial temporal (MT) area (also known as V5). The top row of Fig. 10.C shows the time course of global field power of VEP, which indicates three VEP peak latencies (76, 112 and 212 ms). The 2^nd through 4^th rows of Fig. 10.C show the reconstructed contralateral CCD distribution using three imaging algorithms, respectively. From the CCD images reconstructed by only using the VEP data, the dorsal pathway was seen gradually extending from lower-tier visual areas to high-tier visual areas. By using our proposed adaptive Wiener filter to integrate the fMRI and EEG data, a consistent sequence of activities was observed with a much enhanced spatial resolution, showing the pathway starting from V1/V2 and then V3/V3a and finally V5/V7 and IPS. The observed cortical visual pathway was generally in agreement with the well-known hierarchical organization of the visual system (Felleman and Van Essen, 1991). In contrast, the imaging results obtained by using the conventional 90% fMRI-weighted approach also had an improved spatial resolution compared to the EEG-alone source imaging. However, it imposed a false positive source region in and around V1/V2 at the latency of 212 ms, whereas a more likely high-tier EEG source around V5 observable from the EEG data was missed.

Figure 10.

A) The pattern-reversal checkerboard visual stimulation, B) fMRI activation map with a corrected threshold p<0.01, and C) the global field power of VEP and the dynamic cortical source distribution at three VEP latencies (76, 112, 212 ms after the visual onset) imaged from EEG alone (1st row), or fMRI-EEG integration using our proposed adaptive wiener filter (2nd row) and the conventional 90% fMRI weighted algorithm (3rd row). Both the source images and the fMRI activation map are visualized on an inflated representation of cortical surface.

DISCUSSION

The proposed adaptive Wiener filter method on the fMRI-EEG integration significantly differs from the existing methods in two important aspects: 1) quantifying the fMRI response as proportional to the time integral of EEG source power, 2) estimating the time-variant source covariance matrices from both fMRI and EEG data. Our pilot simulation results have demonstrated that the proposed method is capable of handling the mismatches between locations of fMRI activations and EEG source activities at any time instant (including fMRI false positives and false negatives), as well as probing the millisecond temporal dynamics of source activities. Our preliminary experimental application also suggests a robust performance of improved spatial resolution (relative to the EEG-alone inverse estimates) and temporal resolution (relative to the fMRI).

In previously reported fMRI-EEG multimodal neuroimaging studies, the fMRI data analysis yields an fMRI activation map, which is then used to constrain the EEG inverse solutions in a subsequent step; however, the pixel values in the fMRI activation map have not been rigorously given any quantitative interpretation that is also physically relevant to the following step of the fMRI-constrained EEG source imaging or localization. In contrast, the fMRI data in the present study are quantified as well-defined physical constraints to the EEG inverse solutions. The time integral (over the ERP period) of source power is constrained to be proportional to the quantified fMRI response for each source location. In this sense, the proposed fMRI quantification method also represents a more specific inference regarding underlying neural activities. Existing fMRI analysis methods are often used to quantify the level by which the BOLD signal is statistically correlated to the designed protocol of tasks of interests. Based upon such a statistical analysis, inferences are often made voxel by voxel regarding how likely a region in the brain is involved in the neural computation underlying the tasks. Clearly, such an inference is conceptual rather than physical. In the present study, we quantitatively relate the fMRI signal over a time scale of seconds to the averaged electrical signal over a much shorter time scale of tens of milliseconds. In short, we suggest using the proposed fMRI quantification method in all fMRI-EEG multimodal neuroimaging studies, or even in fMRI studies.

The proposed fMRI quantification method relies on an assumption that the relationship between neural activity and fMRI response is linear. Although such an assumption is commonly made in the fMRI analysis and interpretation, the nonlinear aspect of fMRI response has also been observed, such as under repetitive stimulation (Janz et al, 2001). The nonlinearity may arise from the neural and/or hemodynamic adaptation or habituation (Janz et al, 2001), or the vascular refractory effect (Cannestra et al, 1998), or the interplay of multiple factors. However, our recent investigation of neurovascular coupling based on the VEP and CBF data under graded brain suppression shows that the linear model is still a good approximation of neurovascular coupling relationship, although it contains a subtle nonlinear component (Zhang et al, 2007). Furthermore, the nonlinearity of fMRI response is probably less of concern in most experiments combining ERP and fMRI (as discussed in this paper), where the adjacent stimuli (in ERP paradigms) are often required to be sufficiently separated in time to avoid the interference among stimuli.

In most existing fMRI-EEG (or -MEG) integrated source imaging algorithms, the fMRI-derived spatial constraints have to be assumed to be time-invariant within a short time period during which the source imaging is carried out instant by instant. This is essentially because the source temporal information cannot be resolved by fMRI. However, when the “static” fMRI spatial information is taken as a time-invariant weighting matrix (in the fMRI-weighted minimum norm formulation) or source covariance matrix (in the Wiener filter formulation) for every time point, it is almost unavoidable that the imaging results would be affected by the fMRI-EEG mismatches, which may happen at any time and also vary over time. Previous efforts have been mainly focused upon choosing an empirical weighting factor for locations inside the fMRI activation relative to those outside (Liu et al, 1998; Babiloni et al, 2003; Ahlfors and Simpson, 2004). In such a way, fMRI constraints are weakened such that distortions due to fMRI false positives and false negatives may be alleviated but perhaps never removed. However, this type of methods, no matter what the fMRI weighting factor is chosen to be, has a tradeoff between the robustness against invalid fMRI prior constraints and the spatial resolution of the instantaneous inverse solution. If the fMRI weighting factor is chosen to be 1, the inverse solution is effectively the minimum norm solution; if the fMRI weighting factor is chosen to be infinitely large, the inverse solution can be highly biased by the fMRI-EEG mismatches. On the other hand, the proposed adaptive Wiener filter approach is aimed at correcting fMRI-EEG mismatches in a much more fundamental way, in consideration of the fact that the mismatches are essentially caused by the different time scales at which the fMRI and EEG signals are generated and collected respectively. The proposed adaptive Wiener filter method holds the promise to remove the mismatches instead of attempting to alleviate the resulting distortions with a cost of lower spatial resolution.

In addition, the assumption of a time-invariant source covariance matrix is also not technically valid. A time-invariant source covariance matrix implies that the source signal at any specific location is a stationary stochastic process over time and its sampled values at all time points are drawn independently from an identical distribution. Such an assumption is not valid in practice because neural networks in the brain are always carried out by source signals with rapidly and coordinated temporal evolvement that are certainly not exchangeable over time. In contrast, the estimation of time-variant source covariance matrices in the proposed adaptive Wiener filter method relies on the assumption that at any time instant, the scalp potential maps recorded in different epochs are independent observations of a stochastic process with the index set over channels, which is a reasonable assumption when the electrical responses are repeatedly induced over epochs by the same stimulus or task.

There are more factors other than those addressed in this paper that may also give rise to the spatial mismatches between the fMRI activations and instantaneous source activities. The fMRI spatial specificity and the cross-modal co-registration error may lead to a difference between locations of neural activities and those of the detected fMRI activations (Wagner and Fuchs, 2001). However, these factors are probably more technical rather than fundamental, in a sense that they may be resolved (or bypassed) before applying the integrated imaging algorithm. For example, one may use a spin-echo sequence and increase the field strength to eliminate the contribution from large draining veins, such that the collected fMRI signals originate exclusively from the microvasculature (Yacoub et al, 2003), resulting in an improved fMRI specificity. One may also use advanced geometric manipulations with optimization procedures that combine the landmark and surface point fitting procedures to ensure a good enough co-registration of difference coordinate systems respectively for the fMRI and EEG data.

In the present study, we assume a linear transform model to describe the relationship in time between BOLD fMRI signals and the power of electrical responses. However, recent studies also investigated the correlation between BOLD signals and the EEG (and LFP) power spectrum (Mukamel et al, 2005; Niessing et al, 2005; Kilner et al, 2005). It was suggested that the BOLD responses were positively correlated with the high-frequency components of EEG (or LFP) signals and negatively correlated with the low-frequency components. Such an obvious difference between the models described in time and in frequency does not necessarily mean they are contradictory to each other. Instead, we believe that their difference only reflects the different aspects of the hemodynamic-to-electrophysiological coupling addressed in these two types of models. The model in time as used here describes how the BOLD signals relate to the event-related electrical response, whereas a model in frequency describes how the BOLD signals reflect the change of continuous electrical signals. It remains unclear how the fMRI-EEG/MEG multimodal neuroimaging should benefit from these recently reported findings (Mukamel et al, 2005; Niessing et al, 2005) on the correlation of BOLD signals with frequency components of EEG power spectrum, although it is indeed desirable.

Although the singular value decomposition of the recorded EEG over epochs also allows computing the noise covariance matrix in a time-variant manner, it may not be necessary under most circumstances since the noise is not time-locked to the “events” and can be assumed to be time-invariant. Therefore, one can estimate the noise covariance matrix simply from the pre-onset part of EEG epochs. For every pre-onset time point, a covariance matrix can be computed from the multi-channel EEG data over epochs. Then, the noise covariance matrix can be obtained by averaging the computed covariance matrices at all pre-onset time points.

It may also be worthwhile to mention that the information from fMRI and EEG have also been proposed to be fused in a symmetric fashion (Trujillo-Barreto et al, 2001; Daunizeau et al, 2007), in which a common hierarchical Bayesian model are formulated for both fMRI and EEG. These symmetric Bayesian approaches may represent another feasible way that potentially leads to a reliable fMRI-EEG integrated neuroimaging.