How can EEG/MEG and fMRI/PET data be combined?

In the last few years, the functional brain imaging community has witnessed numerous efforts (and perhaps even more discussion) directed at multimodality data fusion: combining high‐quality localization information provided by the hemodynamic‐based brain imaging methods such as PET and fMRI with high‐quality temporal data generated by the electromagnetic‐based techniques such as EEG and MEG [Dale and Halgren, 2001]. The article in this issue by Vitacco et al. [2002] provides an interesting research effort aimed at this problem, one that forces us to confront a number of critical questions about the entire data fusion enterprise.

Almost every neuroscientist, and certainly every functional neuroimager, tries in one way or another to combine data from multiple methods. Three distinct approaches are used.

Converging evidence

Converging evidence is the most common method, although it is, typically, not an explicit attempt at combining data from different techniques. Results from other analyses that support one's findings are brought forth in the discussion section of an article. A standard sentence might look like: “The activation we found in region X is consistent with studies in patients with focal lesions in region X and in ERP studies where a latency in the Y component on task Z has been observed during ….” Examples of this sort can be found by looking at the articles in just about any issue of this and similar journals, as well as at just about all of our own publications, which humility prevents us from listing in the bibliography. This kind of qualitative approach has a number of well‐known limitations. Although the one most cited is selective attribution (i.e., not mentioning studies that don't support one's findings), the main limitation is actually that the complexity of the brain with its many interacting elements makes it extremely difficult to say whether or not two findings obtained using methods with different spatial and temporal features do or do not agree. For example, because we do not have a solid understanding of the neural substrates of fMRI activations, nor of specific EEG/MEG components, nor of the effects of a focal lesion on how a neural network behaves, it is far from trivial to suppose, for instance, that a statistically significant Z‐score in the left inferior frontal gyrus and a large left anterior negativity at 200 msec after stimulus presentation correspond to the same thing.

Converging evidence can, of course, be more formally assessed by performing meta‐analyses of data that have been transformed into some canonical coordinate system and evaluating to what extent there is agreement across studies and recording modalities [Fox et al., 1998]. One can implement the converging evidence approach in one study by doing the same experiment multiple times using different recording techniques: the same subjects perform the same task while undergoing, for example, fMRI in one session and MEG in a separate session. Disbrow et al. [2001], for instance, took this approach in a study on the somatosensory system testing whether bimanual (vs. unimanual) stimulation has distinct spatial activation patterns (fMRI) and is also associated with neurophysiological responses that differ in their timing, amplitude, and spatial origin (MEG). Finally, in the clinical domain, converging evidence across recording techniques is of critical importance. For example, the presurgical mapping carried out with fMRI or MEG that might be provided to a neurosurgeon to aid in surgical planning must agree closely with intracranial recordings.

Direct data fusion

In the direct data fusion approach, two data sets are directly combined using some mathematical/statistical algorithm [George et al., 1995]. The Vitacco et al. [2002] study presented in this issue illustrates this method. The main assumption, as emphasized by Dale and Halgren [2001], is that the critical signals generated by each method correspond to the same set of underlying neural generators. The most common method that has been employed to combine hemodynamic and electromagnetic data assumes that there are a few underlying equivalent current dipoles that generate the EEG/MEG data, and uses the local maxima obtained by PET/fMRI as constraints on localizing these EEG/MEG dipole sources [Ahlfors et al., 1999]. Other source estimation methods for EEG/MEG are possible, however, such as assuming that the sources of the EEG/MEG data are spatially distributed [Ioannides et al., 1993; Sekihara et al., 2002], and these lead to temporally continuous EEG/MEG values throughout the brain (or, depending on the method, at the cortical surface). The LORETA method [Pascual‐Marqui et al., 1994] used by Vitacco et al. [2002] is one such distributed source modeling method (LORETA stands for low‐resolution, electric tomography algorithm; it provides the smoothest possible 3D current distribution in the brain that can generate the observed scalp field). One problem with all these source estimation procedures is well‐known [Dale and Halgren, 2001; Hamalainen et al., 1993]: the inverse problem of determining a unique set of sources that yield the surface‐recorded distribution of electromagnetic activity is ill‐posed; in the absence of constraints, there is no unique solution. This lack of uniqueness obviously affects any data fusion effort.

Vitacco and colleagues [2002] asked the following question: for a semantic monitoring task (silent word reading coupled with indicating by a button press whether each presented word belonged to the category of food; the control task was looking at a fixation cross), do the brain regions that show large activations by fMRI also show large LORETA values? To deal with the different spatiotemporal resolutions of the two methods, they averaged their LORETA values across all temporal epochs, and attempted to identify each LORETA local maximum with its nearest fMRI local maximum in a statistically meaningful manner. They found that this could be done for group mean data, but on an individual basis, only half of the subjects showed significant correspondence between the fMRI and LORETA patterns. That the authors were able to obtain good results for the group data is encouraging for the advocates of direct data fusion. With a more constrained experimental design, or improvement in data acquisition and analysis, it is possible that agreement in more individual subjects can be obtained. It is worth noting, as the authors do, that the idea behind his approach is that once the correspondence between important local maxima for each data type has been made, one can then perform multimodal integration at the full spatial and temporal resolutions of each method.

Nonetheless, several critical issues remain unresolved by this approach, and these will need to be addressed by future research. First, in collapsing the electromagnetic data over time, are the local maxima so produced somewhat artificial creations? Second, the local maxima obtained from fMRI/PET data generally correspond to the case where two or more conditions are contrasted against one another (the subtraction paradigm; see Horwitz et al. [1999] for a discussion of both the subtraction and covariance paradigms). This can result in important nodes in the neural network under study being missed, because such a node may be as active during one condition as during another; what changes between conditions is the interregional functional connectivity. The net effect is that attempting a correspondence between local maxima would miss such “unactivated” nodes. Once again we are confronted with the serious issue of how to relate the sources of the signals between the two data types [Nunez and Silberstein, 2000].

The extent to which a multimodal imaging approach can incorporate temporal information is itself highly problematic. There are neurophysiological phenomena that simply do not (cannot) show up in a meaningful way in hemodynamic signals. These phenomena include 1) the fact that timing (latency) variation of particular response peaks is behaviorally relevant (e.g., 30 msec latency variation of the major auditory response, N1, is associated with different perceptual attributes); 2) that some of the peaks are small and transient and will not show up as hemodynamic activation peaks (as Vitacco et al. [2002] also point out); and 3) that oscillatory or spectral changes are implicated as relevant for neural representation but are not reflected in hemodynamic signals in any obvious way.

Computational neural modeling

The third way by which diverse data can be “compared” is through the use of computational neural models that can simulate the different data types [Horwitz et al., 1999, 2000]. The idea here is to construct a large‐scale biologically realistic neural network model that can perform the cognitive tasks under investigation. The model would be constructed so as to be able to generate simulated fMRI/PET data and simulated EEG/MEG data that can be compared to experimentally observed values. The critical notion is that data types with different spatiotemporal properties are not compared directly to one another, but are compared inside a neural model that incorporates specific hypotheses about how particular cognitive operations are mediated neurally. That is, the assumptions one makes are about how macroscopically‐measured data are related to neuronal physiology, not about how these data are related to each other. The major disadvantage of this approach is that modeling is meant to simplify what actually is going on, and thus it is hard to know if lack of agreement between computational and experimental results means the model and its corresponding hypotheses are too simple, or just wrong. The other major limitation of this approach is that no such model has been constructed; there are dynamic recurrent network models that relate neuronal electrophysiological data to fMRI/PET [Arbib et al., 1995; Corchs and Deco, 2002; Tagamets and Horwitz, 1998], as well as models that relate neuronal data to EEG/MEG signals [Nunez and Silberstein, 2000; Vaughan and Arezzo, 1988; Wood and Allison, 1981], but to our knowledge, no model yet exists that can simulate both types of data, although the construction of such models is underway.

Efforts at direct data fusion will continue, as will efforts at constructing large‐scale neural models that can simulate both hemodynamic and electromagnetic data. It will likely be the case that the difficulties and limitations that each approach encounters will actually strengthen our knowledge through mutual feedback as to how to proceed to multimodality integration.