Integration of EEG/MEG with MRI and fMRI in Functional Neuroimaging

Abstract

EEG and MEG are important functional neuroimaging modalities for studying the temporal dynamics of neural activities and interactions, but the accurate localization of neural activities still remains a challenging problem. Combining EEG/MEG with MRI or/and functional MRI (fMRI) holds promise to significantly increase the spatial resolution of electromagnetic source imaging, and at the same time, allows tracing the rapid neural processes and information pathways within the brain, which cannot be achieved using these modalities in isolation. In this paper, we review some recent progresses in multimodal neuroimaging, with special emphasis on the integration of EEG, MEG with MRI and fMRI. Some examples are shown to illustrate the importance of the combined source analysis in clinical and experimental studies.

1. Introduction

The past decade witnessed an explosive development of functional neuroimaging techniques to observe and measure brain activity in human. Neuronal activity can elevate electromagnetic signal changes (the primary effects) as well as hemodynamic and metabolic changes (the secondary effects). These changes are the basic sources for all the modern functional neuroimaging techniques. Electroencephalography (EEG) or magnetoencephalography (MEG) has enabled the estimation of the neural sources from the scalp potentials or magnetic fields recorded through high-density sensors by solving a so-called “inverse problem”. Since the propagation of bioelectromagnetic fields is effectively instantaneous [1–3], EEG and MEG have a remarkable merit of high temporal resolution at sub-millisecond scale, capable of detecting the rapid changes of neurophysiologic processes. However, these techniques suffer from the ambiguities in defining the precise locations of brain activity due to the ill-posedness of the inverse problem [4, 5].

On the other hand, functional magnetic resonance imaging (fMRI), using endogenous blood oxygenation level dependent (BOLD) contrast, is another well established technique for mapping human brain function. The benefit of fMRI is, conversely, its high spatial resolution; activated brain areas can be localized with millimeter spatial resolution. But the slow response time of the BOLD response limits its capability to characterize how, rather than where, the brain performs its tasks [6–8].

Tremendous efforts have been made to integrate information across multiple modalities during the same task in an attempt to establish an alternative high-resolution spatiotemporal imaging technique [9–12]. In this paper, we will discuss different approaches that have been proposed for multimodal neuroimaging with special emphasis on the integration of EEG, MEG and MRI, fMRI. Some applications will be shown to illustrate the efficacy and importance of these techniques in clinical and neuroscience studies. Finally, we will discuss some remaining challenges and problems in the multimodal integration.

2. Integration of EEG/MEG and MRI

Both EEG and MEG signals arise from the postsynaptic currents of cortical neurons, reflecting the electrical behavior of neuronal interactions [1, 2]. The distributed synaptic currents generate electromagnetic field, propagating within a passive volume conductor that comprises brain structure, cerebrospinal fluid, skull and scalp [1, 4]. The derivation of the scalp potentials or magnetic fields from brain current sources is known as “forward problem”, which requires the modeling of the head volume conductor. The accuracy of inverse solution depends crucially on the degree of agreement of the forward model with the actual conducting properties and shape of the head. Although the spherical model has an analytical forward solution and is still widely used in inverse source imaging, some comparative studies have shown that realistic geometric head model can provide substantial improvement in predicting EEG/MEG signals outside the head [13, 14].

Typically, T1-weighted MR images are used to build the realistic geometric model for individual patients or subjects since they present high contrast and allow for the segmentation of different head compartments and other brain structures of interest. Subsequently, boundary element method (BEM) [13, 15–16] or finite element method (FEM) [17–20] is commonly employed to implement the numerical forward calculation. Notably, the MR-based diffusion tensor imaging (DTI) has recently been suggested to map the conductivity tensor of human brain considering the high correlation between electrical connectivity tensor and water self-diffusion tensor, which holds the potential to further refine the head modeling by taking the anisotropy of white matter into account [65]. Another benefit from the integration of MRI and EEG or MEG is the visualization of source localization or distribution relative to the brain anatomy, which facilitates the comparison with other imaging methods or intracranial recordings and is therefore of clinical significance [21, 22].

2.1 Realistic Geometry Cortical Potential Imaging

One application of MRI-EEG combined source analysis is known as cortical potential imaging (CPI) technique in realistic-geometry head model using BEM [16, 21, 23] or FEM [18, 68]. Specifically in the BEM-based CPI, three shells (the scalp, skull and brain) are segmented from MR images and triangulated into fine elements; each shell is assumed to be homogeneous but has different electrical conductivity. Since brain electrical sources exist only inside the brain, Green’s 2^nd identity is applicable within the volume between epicortical and scalp surfaces [24]. A linear relation can thus be derived to directly link the scalp potentials with the cortical potentials by treating the layers of low-conductivity skull and scalp [16]. Applying the inverse technique, one can estimate the cortical potential distribution from the measured scalp potentials [5, 16, 25]. Note that cortical potential imaging does not require any a priori assumption on the source space of brain electrical events. Rather, it provides a noninvasive estimate of epicortical potentials, which are routinely measured during surgical evaluation using subdural electrode arrays [21].

It has been shown that the cortical surface potentials imaged from either simulation or real data can provide much higher resolution in observing the brain activity than scalp potentials [21,22, 26–28]. Particularly, in our previous experimental validation study [21], the cortical potential distribution was estimated from the patient’s somatosensory evoked potentials (SEPs), and the result was well consistent with directly recorded cortical potentials (electrocorticograms - ECoGs) in the same patient, as shown in Figure 1. This study suggested the possibility of using cortical potential imaging as a noninvasive alternative of invasive recordings for aiding presurgical planning in epilepsy patients.

Figure 1.

An example of cortical potential imaging in a human subject. (a) Scalp potential map at 30 ms after the onset of right median nerve stimuli; (b) the estimated cortical potential map at latency 30 ms; (c) the direct recorded subdural grid potentials at latency of 30 ms following open-skull surgery. All the maps are normalized and the colorbars are shown on the right. The correlation coefficient between the estimated and recorded subdural grid potentials is 84%. (Revised from [21] with permission)

2.2 Anatomically-constrained cortical current density imaging

Another cortical surface imaging technique is so-called cortical current density (CCD) imaging, which is based on the physiological observation that EEG or MEG signals are primarily generated by currents flowing through the apical dendrites of cortical pyramidal cells. Also, because of the columnar organization of the cortex, all possible current source locations and orientations can be constrained anatomically given the shape of cortical surface [11]. In practice, the cortical envelop can be extracted from T1-weighted MR images with high contrast between the gray matter and the white matter [29]. By dividing the whole cortical sheet into a large number of patches, the source model is usually represented by thousands of elementary current dipoles disposed over the folded cortical surface with outward-pointing direction perpendicular to the local cortical patch. The inverse problem then turns to retrieving the scalar distribution of dipole strength over the oriented cortical patches [11, 30–34].

Anatomical constraints from MRI can improve the cortical source reconstruction by excluding the solutions from physiologically unrealistic locations or directions, which in theory gives rise to reduced dimensionality of solution space and a less underdetermined inverse problem. Clearly, the efficacy of anatomically constrained CCD imaging essentially relies on the precision of the convoluted cortical surface reconstruction from MR images [11].

2.3 Realistic-geometry 3-D distributed source imaging

Instead of imaging brain electrical sources over two-dimensional surface like cortical potential imaging or cortical current density imaging techniques, the three-dimensional (3-D) source distribution within the brain volume can also be reconstructed from the external electromagnetic measurements. 3-D distributed source imaging is motivated by at least three reasons. 1) Capability of imaging 3-D distributed brain source distribution makes it more straightforward to perform the cross-validation studies and integrated analysis of EEG/MEG with other functional brain tomography modalities, such as fMRI, positron emission tomography (PET) or single photon emission computed tomography (SPECT). 2) EEG and MEG may be generated by activity in subcortical structures that cannot be revealed in cortical surface imaging techniques. 3) Source modeling in 3-D brain volume is a more generalized representation of distributed sources, which avoids the uncertainties of the location and directional constraints in surface-based model. Similar to the aforementioned, here the main purposes of combining with MRI are to compartmentalize the brain volume in which 3-D grid points are extracted and to facilitate the realistic modeling of the head volume conductor with minimal error of forward calculation [35].

The 3-D distributed source imaging is a highly ill-posed inverse problem because the solution is desired on the orthogonal triples of “regional dipoles” located at 3-D grid points throughout the brain volume. The representative inverse algorithms include minimum norm (MN) [36], lead-field normalized minimum norm (WMN) [37] and the Laplacian weighted minimum norm (LWMN) [33], also known as LORETA [38]. The MN estimates the source distribution with the smallest L2-norm solution vector that would match the measured data, but undesirably favors superficial sources. The WMN compensates the lower gains of deeper sources by using lead field normalization, while the LORETA combines the lead field normalization with the spatial Laplacian operator, thus gives the depth-compensated inverse solution under the constraint of smoothly distributed sources. One application of the realistic-geometry LORETA-based 3D source imaging technique for studying fast repetitive unilateral finger movement evoked brain activities is shown in Figure 2 [39].

Figure 2.

The three dimensional current source distribution estimated by LORETA in fast repetitive unilateral finger movement tasks for the left hand is shown in the figure. The analysis was performed at the first peak after the execution of the finger movement, i.e. 50 ms after EMG peak (Fig. 2.a). The current sources reconstructed by LORETA appear in the motor cortex (Fig. 2.b), which is related to the execution of the movement, and premotor area (Fig. 2.c), which is related to the coordinating of the movement. (Revised from [39] with permission)

3. Integration of EEG/MEG and fMRI

The integration of EEG/MEG and fMRI works under the hypothesis that the regions with the greater fMRI BOLD responses have larger possibility of being electrically active over the time period of interest [11, 31, 40–41]. This hypothesis is reasonable in that neural activity, modulating neuronal firing and generating EEG signals, increases the demands for oxygen and induces larger cerebral blood flow and consequently produces larger fMRI responses [42]. Although at this time the physical and physiological basis that accounts for the correlation between fMRI signal and neural electrical activity is not yet well understood, a positive coupling between local hemodynamic response and electrical activity has been observed in both animal and human experimental studies [43, 44].

3.1 fMRI-guided equivalent current dipole fitting

The coupling between brain electrical activity and fMRI measurement suggests that one can use EEG or MEG to trace the time course of local neural activation revealed by fMRI. One common approach is to assume that external electromagnetic signals are generated by a relatively small number of focal sources, usually referred as equivalent current dipole (ECD) [15, 45–48]. The activation foci derived from fMRI images can be used to guide the placement of dipole locations, and the dipole orientation and strength are then fitted so as to best explain the observed bioelectromagnetic measurements over the time period of the relative neural process. Modified from this procedure is a technique of “fMRI-seeded dipoles”, which only uses the fMRI foci as initial guesses (or seeds) for dipole locations and adjusts the dipole parameters using nonlinear optimization algorithms [49, 50].

In some cases, these techniques lead to significant conclusion in resolving the timing details of localized activation and the interactions across different neuron assemblies that cannot be obtained when using fMRI alone. On the other hand, the fMRI-guided initial guess of dipole locations has largely alleviated the local minima problem inherent in nonlinear optimization procedures as one may have experienced in traditional multiple dipole fitting problems solely based on EEG or MEG.

As the conventional fMRI techniques with the aid of these ECD-based approaches can provide additional temporal information beyond identifying the locations of neural activation, they have the potential of revealing the dynamic neuronal events, which can hardly be achieved by fMRI alone. However, these techniques are always limited by their inability to resolve complex spatiotemporally distributed activities that are usually induced by a high-level cognitive task. When a large extent of neuronal network is active in parallel, the applicability of simple dipole model would be questionable; or when a large number of different brain regions are involved, the nonlinear fitting procedure may fail to reach an optimal solution.

3.2 fMRI-constrained cortical current imaging

In order to account for the distributed nature of the neural sources inside the brain, fMRI has also been incorporated in estimation of the continuous distribution of cortical current density. In this approach, fMRI data are used to bias the EEG/MEG inverse solution towards those locations deemed as statistically significant in the view of fMRI [40, 41].

This can be implemented under different mathematical frameworks, such as 1) in the context of the generalized Wiener filtering, which states that an optimal linear estimator can be derived given the covariance matrices of both source and sensor noise are known, fMRI spatial information is encoded into the diagonal elements of source covariance matrix [11, 40]; 2) based on the regularization strategy of generalized weighted minimum norm, one can set the weight matrix according to the fMRI significance or p-value so as to allow for the fMRI preference [31]. It is worth pointing out that these two frameworks are fundamentally equivalent, as we can see that the Wiener estimator as (1)

$$G=R{A}^T{\left(AR{A}^T+C\right)}^{-1}$$ (1)

where A is the forward transfer matrix, C is the sensor noise covariance matrix, and R is a source covariance matrix derived from fMRI activation maps. Since a voxel in fMRI must be either active or non-active, the fMRI-derived source covariance can be written as (2)

$$R=R_f\cdot r$$ (2)

where the diagonal elements of R_f are set to a positive value f only for those dipoles whose locations are deemed active in fMRI while other diagonal elements are set to 1, and r is a regularization parameter. The inverse operator as (2) is also the minimizer of a weighted minimum norm function

$$\mathrm{\Delta }=\parallel C^{-1/2}(b-Ax){\parallel }_2^2+\frac{1}{r}\parallel R_f^{-1/2}x{\parallel }_2^2$$ (3)

The parameter f controls the amount of bias towards fMRI active locations, which depends on the confidence in our hypothesis that neuronal and hemodynamic activities are co-located. The regularization parameter r can either be determined by standard approaches like “L-curve”, or be calculated if the noise covariance C is accurately estimated [66].

A further refinement of this technique is to quantitatively consider the reliability of source estimation with respect to the effect of sensor noise. Due to the linearity of both forward and inverse operators, the fMRI-constrained current density estimation can be normalized by the noise sensitivity, yielding unit-free dynamic statistical parametric maps (dSPM) of brain activity [41]. This technique was used to study the cortical activity during a task involving semantic processing of visually presented words. The subjects were required to decide whether each word referred to an object or animal that is usually more than one foot in size (in any dimension). The words were either novel (i.e., presented only once in the task) or were repeated multiple times. Cortical activity in terms of dSPM was estimated from either MEG alone or MEG in conjunction with fMRI at four representative latencies in a single subject, and visualized in the subject-specific inflated cortical surface [64]. The results are shown in Figure 3.

Fig. 3.

The top row shows the snapshots of estimated cortical activity at four representative latencies in a single subject. Statistical parametric maps calculated from MEG during a novel-verbal size judgment task are shown of the noise-normalized anatomically constrained dipole amplitudes. Activation was seen to spread rapidly from visual cortex around 80 ms to occipitotemporal, anterior temporal and prefrontal areas. Sulcal and gyral cortex are shown in dark and light gray, respectively. The significance threshold for the dynamic statistical maps is p < 0.001. The top row shows the snapshots of fMRI-biased cortical activity estimated from MEG at the same four latencies as in the top row in a single subject. The sequence of activity in the fMRI-biased estimates was generally similar to those constrained by anatomy alone. However, the fMRI-biased estimates were considerably more focal. All activations are significant for both fMRI (p < 0.1) and MEG (p < 0.001) (Modified from [35] with permission).

Furthermore, some latest progresses have been made to estimate the connectivity patterns among different cortical regions of interests (ROIs) using the estimated cortical current density via directed transfer function [12] or structural equation modeling [54]. Cortical connectivity estimation provides an important perspective regarding the causal relation and information flow pathways, and thus specifically addresses how, not just where and when, neuronal assemblies are activated and coordinated. Notably, a recent study on cortical connectivity estimation based on the combination of EEG and fMRI has been reported [12]. The high-density EEG as well as MRI/fMRI data were collected in separately examinations when the same subject was performing the same visually trigger finger-tapping task. By use of the linear inverse operator constrained by fMRI, the cortical current density was noninvasively estimated from EEG and then averaged to develop representative waveforms for pre-selected ROIs. DTF technique was applied to estimate the directional causality in various frequency bands among different ROIs, and the connectivity pattern was visualized in a subject-specific cortex reconstructed from MRIs (See Figure 4).

Figure 4.

a) Estimates of the statistical significance of the cortical activity in selected ROIs depicted on the realistic representation of the cortex in a human subject. Such estimation was performed first by estimating the cortical activity with the use of the a priori information from fMRI during the solution of the linear inverse problem. Then, in order to obtain the statistical significance of cortical estimated waveforms, the z-score transformation was first applied and then represented. Note the z-score scale of the waveforms, ranging from 0 (the baseline) to 10; this latter value corresponding to a level of statistical significance equal to p < 10⁻⁷. The figure at the center shows the different ROIs drawn on different colors on the cortical surface reconstruction of the subject. b, c) Connectivity patterns estimated in the alpha frequency band by application of DTF on the estimated cortical waveforms in one subject. The connectivity estimates are relative to the period before the EMG onset b) and from the EMG to the end of recordings c). Each pattern is represented with arrows moving from one cortical area toward another. The color and size of the arrows code the level of strength of the functional connectivity observed between ROIs. The labels indicate the names of the ROIs employed. (from ref [12] with permission)

4. Discussion

Multimodal functional neuroimaging has received significant attention in the past decade and has already advanced our understanding of the spatiotemporal pattern of brain activation and connectivity underlying perception, motion and cognition. However, challenges still remain in several aspects.

Although the integration of fMRI and EEG/MEG has a sound physiological basis, it is still important to allow for the possibilities that electrical sources may be missing in fMRI, or that fMRI active regions produce no observable electromagnetic signals, or slight displacement between fMRI and EEG/MEG source locations. In general, these mismatches arise from the differences in physiological and physical rationales of these imaging modalities, as well as the spatiotemporal scales at which the brain processes are monitored [40, 41, 51–53].

Functional MRI reflects oxygen consumption and usually can observe the sustained neural activities that last at least tens of milliseconds to elicit detectable change of MR signal. But human brain can undergo rapidly shifting processes characterized by a temporal scale of a millisecond, which in principle can be “visible” in EEG or MEG. Also, a cortical patch can generate observable electromagnetic signals simply by increasing the firing synchronicity of a small percentage of neurons with little modification of its metabolic consumption, and therefore it may be missed in fMRI [51]. Conversely, EEG/MEG is more sensitive to sources at superficial cortical regions than deep sources that are relatively far away from the recording sensors. Such depth dependency often limits their capability to be visible to activities at deep sulci or subcortical regions, while the sensitivity of fMRI is relatively uniform in the whole brain volume. Even if we assume fMRI can reveal all the neural activations and all the activated neuron assemblies can generate observable EEG signals, there are still possible mismatches because of the difference in time scale to which these two imaging modalities are applied. EEG inverse is carried out at almost every single instant, but fMRI, in simplification, can be interpreted as the average or integral of the power of neural activity within a period of time. One can expect that at a region deemed active in fMRI can be “silent” at some time slices, as long as the activities at multiple areas are not completely in synchrony. The possible difference in the locations of neurons and the involved vessels can cause the slight displacement of fMRI “hotspots” from the real sources. Besides, the uncertainties of current dipole modeling on the curved cortical surface may not represent all the source location.

The potential distortion due to these mismatches or uncertainties should be considered in fMRI-EEG/MEG combined source analysis. In fMRI-guided equivalent dipole fitting, it may result in a significant estimation error if not all dipole locations are specified at fMRI foci. To address this problem, an extra independent current dipole, without any spatial constraint, is usually considered into the dipole fitting procedures to improve the reliability of source imaging against the possible fMRI-EEG disagreement [47]. In cortical current imaging, the Twomey regularization [55] has been proposed to reduce the distortion effect due to fMRI missing sources. Specifically, in which the fMRI-constrained solution is used as an initial estimate of cortical current density, and is subsequently adjusted by fitting it to the EEG data again without fMRI constraint but using Twomey regularization in an attempt to produce a more reliable solution [34]. Figure 5 shows the computer simulation results of this method in comparison with conventional fMRI-EEG integrated Wiener estimation, which suggest applying Twomey regularization can alleviate the distortion effect due to fMRI invisible sources.

Fig. 5.

Typical examples of effects of fMRI invisible and fMRI extra sources on three cortical source imaging methods: EEG-alone weighted minimum norm, 90% fMRI-constrained Wiener filter and Twomey algorithm. The upper row shows the results of simulation on fMRI invisible sources; the lower for fMRI extra sources.”

Although both EEG and MEG can be combined with fMRI in theory, one important merit of EEG over MEG is the possibility of simultaneous acquisition of EEG and fMRI signals. This is particularly necessary for clinical application, such as in epilepsy studies, when the activity of interest cannot be reliably reproduced. Simultaneous data acquisition would in theory ensure EEG and fMRI reflect the exact same brain status. However, concurrent acquisition of EEG and fMRI has been proven to be very challenging since it is prone to large artifacts induced by the static and the time-variant fields of the MR scanner, as well as the accompanying acoustic noise [56, 57]. Specially designed mapping systems, and real-time and post-processing algorithms have been investigated and developed to avoid and/or remove the artifacts in order to obtain reasonable quality EEG [56, 58–61, 68]. Nevertheless, it still remains to be demonstrated that the simultaneous EEG-fMRI recording techniques can capture EEG activity as accurately as EEG-alone acquisition in clinical settings.

The further methodological development for EEG-fMRI integration requires a quantitative understanding of the coupling between hemodynamic response and electrophysiological activity is highly desired. This would probably require the extensive experimental studies by use of fMRI or optical imaging methods and invasive cortical and/or subcortical electrical recordings in animal models [62]. In the microscopic level, it may also be essential to interpret the functional data (such as BOLD-fMRI) by mathematically linking the membrane ionic currents, energy metabolism and blood oxygen transport [63].