Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2020 Dec 1.
Published in final edited form as: J Neurosci Methods. 2019 Aug 21;328:108401. doi: 10.1016/j.jneumeth.2019.108401

Method for spatial overlap estimation of electroencephalography and functional magnetic resonance imaging responses

N Heugel 1, E Liebenthal 1,2, S A Beardsley 1,3
PMCID: PMC6810902  NIHMSID: NIHMS1538914  PMID: 31445115

Abstract

Background:

Simultaneous functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) measurements may represent activity from partially divergent neural sources, but this factor is seldom modeled in fMRI-EEG data integration.

New Method:

This paper proposes an approach to estimate the spatial overlap between sources of activity measured simultaneously with fMRI and EEG. Following the extraction of task-related activity, the key steps include, 1) distributed source reconstruction of the task-related ERP activity (ERP source model), 2) transformation of fMRI activity to the ERP spatial scale by forward modelling of the scalp potential field distribution and backward source reconstruction (fMRI source simulation), and 3) optimization of fMRI and ERP thresholds to maximize spatial overlap without a priori constraints of coupling (overlap calculation).

Results:

FMRI and ERP responses were recorded simultaneously in 15 subjects performing an auditory oddball task. A high degree of spatial overlap between sources of fMRI and ERP responses (in 9 or more of 15 subjects) was found specifically within temporoparietal areas associated with the task. Areas of non-overlap in fMRI and ERP sources were relatively small and inconsistent across subjects.

Comparison with existing method:

The ERP and fMRI sources estimated with solely jICA overlapped in just 4 of 15 subjects, and strictly in the parietal cortex.

Conclusion:

The study demonstrates that the new fMRI-ERP spatial overlap estimation method provides greater spatiotemporal detail of the cortical dynamics than solely jICA. As such, we propose that it is a superior method for the integration of fMRI and EEG to study brain function.

Keywords: fMRI, EEG, ERP, Joint Independent Component Analysis (jICA), Auditory, Oddball, P300

1. Introduction

The neuroimaging of brain activity with simultaneous functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) capitalizes on the complementary strengths of the methods, but poses challenges with respect to cross-method data integration and interpretation (Ahlfors and Hamalainen, 2012; Dale et al., 2000; Kujala et al., 1995; Liu and He, 2008; Mangalathu-Arumana et al., 2012). FMRI measures the hemodynamic response related to neural activity at high spatial (millimeter) but low temporal (on the order of seconds) resolution, whereas EEG provides a high temporal (on the order of milliseconds) but low spatial resolution (centimeter) measurement of large-scale neural activity. Due to the differences in spatiotemporal resolution and type of measurement, fMRI and EEG provide complementary information but may reflect the activity of partially divergent neural sources in the brain.

Both the blood-oxygen-level-dependent (BOLD) fMRI and scalp event related potential (ERP) responses are thought to be near-linearly correlated with local field potentials (LFPs), but only scalp ERPs are also correlated with fast spiking activity (Logothetis, 2008; Logothetis et al., 2001; Mathiesen et al., 1998; Mazzoni et al., 2010; Viswanathan and Freeman, 2007). Second, depending on the location and orientation, synchronous electric sources may summate or attenuate one another, resulting in a potential mismatch with the BOLD response. Third, because the primary generators of ERP activity are presumed to be large pyramidal cells in the neocortex that have a highly regimented structure and orientation relative to the cortical surface, the source activity is typically modeled by electrical dipoles limited to the neocortical layer and with an orientation perpendicular to the cortical surface (Ahlfors and Hamalainen, 2012). Thus, there may be a mismatch between the imaging modalities, for example if subcortical sources are measured only with fMRI.

We propose a data-driven approach to estimate the spatial overlap between maps of brain activity generated from fMRI and ERP measurements that assumes that overlapping, but not identical, sources of neural activity contribute to each measurement. That is, different from fMRI-informed integration approaches that limit or bias the ERP source reconstruction to regions detected as active with fMRI (Aftanas et al., 1998; Bobes et al., 2018; Dale et al., 2000; Huster et al., 2012; Ou et al., 2009; Xu et al., 2018), or ERP-informed integration approaches that use predefined ERP features to analyze the fMRI data (Liebenthal et al., 2003; Bénar et al., 2007; Debener, 2005; Jann et al., 2009; Mizuhara et al., 2005; Murta et al., 2015; Portnova et al., 2018), the approach presented here does not impose coupling between the neuroimaging measurements. Neurogenerative approaches that model the generation of EEG and fMRI signals to estimate the sources that best explain experimental data allow uncoupling between the measurements (Huster et al., 2012; M J Rosa et al., 2010; M. J. Rosa et al., 2010; Sotero and Trujillo-Barreto, 2008, 2007). However, neurogenerative modeling does not explicitly inform on the extent of coupling and uncoupling between the modalities..

The proposed fMRI and ERP spatial overlap estimation (fMRI-ERP SOE) method is applied to task-related activity extracted from individual data and consists of the following key steps: 1) distributed source reconstruction of the task-related ERP activity (ERP source model), 2) transformation of the volumetric fMRI activity to the ERP spatial scale by forward modelling of the scalp potential field distribution and backward source reconstruction (fMRI source simulation), and 3) optimization of fMRI and ERP thresholds to maximize spatial overlap without a priori constraints of coupling (overlap calculation). The representation of both fMRI and ERP signals in a common ‘nonnative’ source imaging space enables the fMRI-ERP SOE approach to maximize the ability to spatially correlate fMRI and ERP sources of activity while minimizing assumptions regarding neuroimage coupling.

In this study, the extent to which simultaneous BOLD fMRI and ERP measurements reflect common versus distinct sources of neural activity was estimated in an auditory oddball paradigm with parametric variation of the deviant size. The results indicate that approximately 73% of the activity measured with ERPs overlapped spatially with that recorded with fMRI, and vice versa. Most, but not all, of the regions in which activity was recorded with only fMRI or only ERPs were adjacent to areas of joint activity, suggesting a relatively tight but imperfect coupling between the neuroimaging measures in this paradigm.

2. Methods

EEG and fMRI were acquired simultaneously from 24 subjects as they performed an auditory oddball discrimination task with five levels of tone frequency deviants. Details of the experiment design, image acquisition, pre-processing and jICA were reported previously (Mangalathu-Arumana et al., 2012) and are briefly summarized here.

2.1. Subjects

Twenty fours subjects, ages 18-40, participated in the original study (Mangalathu-Arumana et al., 2012). Of these, seven subjects were excluded from the analysis because their anatomical MR-images did not cover the entire skull (as needed to construct a head model). Two additional subjects were excluded from the group analysis during preprocessing because jICA returned components with non-physiological data structures (z-scores >30 and no spatiotemporal variation across electrodes), indicating a failure of jICA to parse the signals into independent components. Data from the remaining fifteen subjects (8 females and 7 males) were used in the current analyses. All subjects provided written informed consent according to the Institutional Review Boards of the Medical College of Wisconsin and Marquette University, and were compensated for their participation in the study.

2.2. Experimental design

The experiment consisted of an auditory oddball paradigm with four-tone sequences, each composed of three standard 1000 Hz tones and one deviant tone (in 3rd or 4th position), presented binaurally. The task consisted of pressing one of two buttons to indicate whether the deviant tone sounded higher or lower in frequency than the standard tones (Mangalathu-Arumana et al., 2012). The tones were 100 ms duration with rise/fall times of 5 ms and were presented at 800 ms stimulus onset asynchrony. The deviant tone frequencies (five lower, and five higher, than the standard tone frequency) were selected individually to correspond to 50, 65, 75, 85, and 95% task performance accuracy (as determined in a prescan test). 144 trials were presented per task level, for a total of 720 trials, broken into 12 runs, acquired in 2 sessions on separate days. The onset of each tone sequence was jittered relative to the time of image acquisition, such that the deviant tone was always presented 4 s before the middle of the next image acquisition block, and the image acquisition coincided with the estimated peak of the BOLD response (Hall et al., 1999). Auditory stimuli were delivered using a pneumatic, MRI-compatible headphone system (Avotec, Inc., Stuart, FL), and the sequence of stimulus presentation was controlled with the Presentation software (Neurobehavioral Systems Inc., San Pablo, CA).

2.3. Data acquisition and preprocessing

The study was conducted on a GE 3T Signa Excite scanner (GE Health Care, Milwaukee, WI). High-resolution whole brain anatomical images were acquired first in each session, using a 3D spoiled gradient-echo (SPGR) sequence (0.9×0.9×1mm voxels). Functional MR images consisted of axially-oriented T2*-weighted, gradient-echo, echo planar images acquired using a clustered volume acquisition and covering the whole brain (TE=25ms; flip angle=77°; TR=2s; stimulus blocks = 7s; 3×3×3.5mm voxels), such that a single functional volume was acquired during each trial 4 seconds after stimulus onset. EEG was recorded continuously during fMRI, at 500Hz sampling rate, using an MRI-compatible MagLink system consisting of a 64-channel MagLink cap (62 monopolar electrodes, and 2 bipolar leads for ECG and VEOG), SynAmps amplifier, and a Scan 4.4 Workstation (Compumedics Neuroscan, Inc., TX). Sintered Ag/Ag-Cl electrodes were positioned per the extended International 10–20 system, with a hard-wired reference at CPz.

MR image preprocessing was performed in AFNI (Cox, 1996). The raw fMRI 2D image slices at each time point were transformed to 3D and spatially registered to the third functional image in the first run. The functional image series was then registered to the anatomical image (consisting of an average of the anatomical images from sessions 1 and 2, to obtain higher anatomical accuracy) using the align_epi_anat.py program in AFNI. Multiple regression was performed to estimate the BOLD activity associated with the response to the deviant in each task level, using level 1 (corresponding to 50% performance accuracy) as a baseline, and translation and rotation motion parameters estimated during registration were used as noise covariates.

The raw EEG was preprocessed with the Scan 4.4 Edit module (Compumedics Neuroscan, Inc. TX). Channels with a variance > 20μV in the baseline period (−200 to −50 ms) were excluded from further analysis. An average of 7 (range 0-9) channels per subject were excluded. The EEG was filtered using a 0.1–30 Hz zero-phase bandpass FIR filter with a 48 dB/octave roll-off. The ballistocardiogram artifact introduced by the MR environment was corrected (Ellingson et al., 2004). Removal of MR gradient artifacts was unnecessary because the clustered functional image acquisition design for simultaneous fMRI/EEG (Liebenthal et al., 2003) prevented the epochs of interest from being contaminated.

ERPs were computed for each task level using an epoch time from −200 ms to 800 ms relative to deviant onset. The epochs were demeaned to compensate for slow drifts occurring during EEG acquisition. Epochs in which the signal exceeded ±200 μV were deemed to contain artifacts and were discarded. The remaining epochs were sorted and averaged by task level. The average number of accepted epochs per subject and level was 86%.

As with the fMRI preprocessing, the ERP response to task level 1, the hardest task level, was subtracted from the ERP responses to the other four task levels. The resulting fMRI and ERP responses associated with task levels 2-5 were used as input for jICA.

2.4. Joint-ICA

The fMRI and ERP datasets were submitted to within-subject jICA, as described previously (Mangalathu-Arumana et al., 2012). The datasets were vectorized and concatenated by level. The input to jICA consisted of four features, each containing the fMRI and ERP responses for one of the four task levels, such that the resulting components represented within-subject fMRI and ERP responses that co-varied across task levels (known as Multi-run jICA).

Components containing task-relevant activity were identified in each imaging modality as those exceeding an amplitude threshold of p<0.05 relative to the distribution of activity across all components in that modality. The joint (multimodal) component containing the most active samples (either ERP time points or fMRI voxels) was used to define a component activity threshold. The threshold (in samples) for including components in the subsequent analyses was set to #samplesinmostactivecomponenttotal#ofcomponents, based on a monte carlo simulation of the random distribution of active samples across components when task-related activity is constrained to a single component. This approach set a permissive (low) threshold for the inclusion of task-related activity from other components by underestimating the number of active samples per component when no task-related activity is present. In 12 subjects, only one joint component passed the threshold. In the remaining 3 subjects, there were 2 suprathreshold joint components, which were summed for subsequent analysis to avoid losing activity of interest.

2.5. FMRI and ERP spatial overlap estimation (fMRI-ERP SOE) method

A schematic overview of the fMRI and ERP spatial overlap estimation (fMRI-ERP SOE) method is show in Figure 1. The approach includes three main components, 1) distributed source reconstruction of the task-related ERP activity (ERP source model), 2) transformation of the volumetric fMRI activity to the ERP spatial scale by forward modelling of the scalp potential field distribution and backward source reconstruction (fMRI source simulation), and 3) optimization of fMRI and ERP thresholds to maximize spatial overlap without a priori constraints of coupling (overlap calculation). It is important to note that the proposed analysis pipeline can be applied to data that is not processed using jICA. It can also be applied separately to the individual jICA components, which would be recommended in studies where the task-related activity is consistently parsed to several components (e.g., studies with more experimental levels).

Figure 1.

Figure 1.

Open in a new tab

Workflow for fMRI and ERP spatial overlap estimation (fMRI-ERP SOE) method. The functional imaging measurements used as input to jICA, and the anatomical images used to create the head and cortical surface models, are shown in orange. The structural/functional preprocessing steps used to create the anatomical model and extract task-related activity are shown in green. Processing steps for the task-related ERP activity are shown in red and those for the fMRI activity, including projection to the common source space, are shown in blue. The steps for ERP and fMRI threshold optimization, and characterization of source map spatial overlap, are shown in purple.

2.5.1. ERP source model

Distributed source reconstruction was used to localize the cortical areas contributing to the scalp ERPs (see steps in red, Figure 1). For each subject, models of the head and pial surface were created in Freesurfer (https://surfer.nmr.mgh.harvard.edu/) using the recon-all script on the subject’s averaged MR anatomy. The head model was then imported into Brainstorm (http://neuroimage.usc.edu/brainstorm/), the pial surface was down sampled to 15,000 vertices, and a boundary element model (BEM) of the cortical surface was created. Whitened and depth-weighted linear L2-minimum norm estimates were used to estimate the amplitude of source activity at each vertex oriented perpendicular to the cortical surface. Finally, the absolute value of source activity at each vertex and time point was converted to z-scores computed relative to the baseline period (−200ms to −50ms).

2.5.2. FMRI source simulation

To facilitate comparisons in the spatial domain, the fMRI volumetric activity was simulated as ERP activity by projection to the cortical surface, forward construction of the scalp potential field distribution, and backward source reconstruction (see steps in blue, Figure 1). For each subject, the volumetric fMRI component was projected orthogonally onto the inflated pial surface in Freesurfer using mri_surf2vol and smoothed with a 6mm full width half maximum Gaussian kernel to fill in small spatial discontinuities resulting from the projection. The map of surface activity was imported into Brainstorm and down sampled to 15000 vertices. The amplitude of activity at each vertex was multiplied by a 400 ms unit amplitude square-wave to simulate a time course and the resulting spatiotemporal activity was forward projected to the scalp electrodes using the lead field matrix. Gaussian noise was added to each electrode to match the noise covariance of the experimentally-measured EEG. The fMRI activity was then submitted to the same source localization procedures used for the ERP analysis.

2.5.3. Overlap calculation

The estimate of overlap between fMRI and ERP source maps may vary as a function of individual differences in activation, and the statistical threshold defining significant activity. In the present work, we opted to identify on an individual basis, the amplitude threshold for the fMRI source map and the temporal and amplitude thresholds for the ERP source map that maximized their spatial overlap on the cortical surface (see steps in purple, Figure 1). This individualized optimization procedure assumes that simultaneous fMRI and ERP measurements largely reflect the local field potential activity of the same neural sources, with partial divergence driven primarily by differences in the spatiotemporal resolution of the measurements.

The thresholds were optimized on a vertex-wise basis. The amplitude thresholds spanned the range [0, 95] % of the maximum amplitude (steps of 0.01), and the ERP temporal thresholds spanned the range [20, 800] ms (20 ms steps). Thresholds were optimized using a class membership function (O) based on the vertex-wise correspondence between the ERP and fMRI source maps,

Ο=(Intersection∗Union2∗Excusion3)∕(total−vertices∕2)∧6

where the intersection corresponds to the number of vertices with significant activity in both imaging modalities, the union corresponds to the number of vertices with significant activity in at least one imaging modality, and the exclusion is defined by the difference between the union and the total number of vertices. The denominator scales the membership function to one when the ERP and fMRI maps overlap completely and the union is balanced by the exclusion. Optimization of the membership function within the threshold space was used to maximize the spatial overlap between fMRI and ERP sources while promoting sparsity in the activation map.

The optimized thresholds were applied to create binary ERP and fMRI source maps, with edge correction to account for stochastic variations in the boundaries of active regions. Vertices that were inactive but surrounded by active vertices in one imaging modality, and were active in the other modality, were labeled as active in both modalities.

To determine the statistical significance of the optimized ERP temporal threshold, a Monte Carlo simulation with 1000 iterations was used to determine the likelihood that a phase-randomized signal with the same power spectrum would exceed the temporal threshold. The distribution of consecutive data points in the phase-randomized signal that met or exceeded the ERP amplitude threshold at each vertex was calculated to determine the 0.05 confidence interval. A Šidák correction was applied to adjust for multiple comparisons across vertices (Sidak, 1967). Vertices in which the activity did not exceed the temporal threshold were marked as inactive in the binary maps. In a second Monte Carlo simulation with 1000 iterations, the spatial distribution of active vertices in the fMRI and ERP source maps was randomized to estimate the probability of obtaining the observed spatial overlap between source maps.

2.5.4. Overlap estimation

Regions of overlap (and nonoverlap) between ERP and fMRI source maps were estimated within, and across, subjects. For the analysis across subjects, the individual cortical surface models were aligned to Freesurfer's FSaverage anatomy using spherical transformations, and the same alignment transform was applied to the individual signed Z-score and binary overlap maps. The individual binary maps were summed at each vertex to determine the number of subjects with supra-threshold activity across imaging modalities.

Regions of interest (ROIs) were created that corresponded to the areas of overlap in ERP and fMRI activity across the group. First, an ERP-fMRI overlap mask was created by thresholding the group overlap count map at 9 subjects, and applying a cluster threshold of 3 vertices, resulting in a corrected p< 0.05 (computed using a 1000 iteration Monte Carlo simulation of expected cluster sizes based on a randomized distribution of the vertices with overlapping fMRI/ERP activity in the source projection space). The overlap mask was then grown by one vertex in every direction to smooth the boundaries and account for 99% of the fluctuations in the spatial extent of backward projected ERP sources simulated using the experimentally measured electrode noise covariance.

The final ERP-fMRI overlap mask consisted of six distinct ROIs, in the right and left superior temporal planes, right lateral superior temporal sulcus, right and left inferior parietal lobules, and right ventral central sulcus. Within in each ROI, the number of vertices per time point that exceeded the global (p<0.05) amplitude threshold were counted in each subject and averaged across subjects to create mean time courses of activation.

The degree of fMRI-ERP overlap obtained with the SOE approach was compared to the degree of fMRI-ERP overlap obtained with strictly jICA, that is, without simulating the fMRI as ERP activity, and with independent thresholding of the components in each modality. When the SOE approach was not used, the ERP amplitude threshold in each subject was set leniently (p<0.1) using the amplitude distribution across all components to maximize spatial overlap and monte carlo simulations were used to set the temporal threshold (=50ms) resulting in a corrected map-wise threshold of p<0.05. Volumetric fMRI in the each jICA component was projected onto the cortical surface. The fMRI amplitude threshold was set to p<0.05 using the amplitude distribution across all components and monte carlo simulations were used to set the vertex cluster threshold resulting in a corrected map-wise threshold of p<0.05. Counts of the number of subjects with activity at each vertex were computed as detailed above, for ERP, fMRI, and ERP - fMRI overlap.

3. Results

Table 1 summarizes the results of the optimal (i.e., maximizing the overlap) threshold calculation and the ERP and fMRI spatial overlap estimation in each subject. Across the group, the mean amplitude threshold of the ERP maps was a z-score of 2.68 (standard deviation – SD=2.14). The mean temporal threshold of the ERP maps was 442 ms (SD=258.2 ms). The mean amplitude threshold of the fMRI maps was 9 (SD=2.1). The optimized ERP amplitude and temporal thresholds were negatively correlated across subjects (R= −0.85). FMRI amplitude thresholds were not correlated with ERP amplitude or temporal thresholds (R=−0.01 and R= 0.23, respectively).

Table 1.

Individual ERP (amplitude and temporal) and fMRI (amplitude) threshold values that maximize the overlap between ERP and fMRI source maps.

Subject
Number		 ERP
amplitude
threshold
(Z-score)		 ERP
temporal
threshold
(ms)		 fMRI
amplitude
threshold
(Z-score)		 Extent of
fMRI activity
overlapped
by ERP
activity (%)		 Extent of
ERP activity
overlapped
by fMRI
activity (%)
4009 0.37 740 5.71 69.1 73.03
4018 4.53 20 3.92 70 73.5
4137 1.44 640 18.58 76.3 74.3
4138 0.25 740 5.24 79.2 66.8
4170 2.39 660 7.29 80 84
4177 0.66 620 12.4 65.9 67.3
4192 1.22 720 9.78 74.5 76.8
4237 1.57 420 6.75 81.4 76.1
4282 1.05 580 9.7 69.8 71.3
4283 3.76 440 21.63 82.2 85.5
4284 6.14 260 7.79 58.15 65
4287 3.12 300 5.35 69.9 78.01
4296 4.39 60 5.71 64.7 62.6
4410 2.06 400 5.34 79.5 78.3
4498 7.3 40 10 66.7 68.3
Open in a new tab

Figure 2 shows an example of the class membership function following an exhaustive search of the ERP amplitude and temporal threshold space for a representative subject. The figure illustrates the trade-off between the ERP amplitude and temporal thresholds that resulted in similar levels of overlap between ERP and fMRI source maps.

Figure 2.

Figure 2.

Open in a new tab

Example of the ERP threshold optimization map for a representative subject (#4287) at an fMRI amplitude threshold of 5.35. The colormap depicts the output from the overlap class membership function as a function of ERP amplitude and temporal threshold for the subject. The extended region of of dark red indicates a trade-off between amplitude and temporal thresholds to mazimize the spatial overlap between the ERP and fMRI source maps.

The extent of spatial overlap between ERP and fMRI source maps across subjects is shown in Figure 3. For the fMRI source maps, the extent of overlap with the ERP source maps ranged from 58.15% to 82.2 %, with a mean overlap of 72.5% (SD=7.2%). For the ERP source maps, the extent of overlap with the fMRI source maps ranged from 62.6% to 85.5%, with a mean overlap of 73.4% (SD=6.7%). Across subjects, the extent of ERP activation overlapped by fMRI activation and extent of fMRI activation overlapped by ERP activation were positively correlated (R= 0.73). For the sample population, the slope of the best fit line (y = 0.68x + 23.85) was not significantly different from a unit line (ANCOVA, F (1)=1.37, p=0.187).

Figure 3.

Figure 3.

Open in a new tab

Spatial overlap between ERP and fMRI source maps computed with the fMRI-ERP SOE method. The percentage of ERP activation overlapping with fMRI activation is plotted against the percentage of fMRI activation overlapping with ERP activation (blue squares). The two measurements were positively correlated (R=0.73), and their relationship was well described by the line y=0.68x+23.85 (in black). The unit line is shown in green. The percentage overlap obtained with a randomized spatial distribution of ERP and fMRI activity is represented by the red square dot (red circle denotes +5 standard deviations).

Figure 4 shows the ERP and fMRI source maps, and areas of activity overlap, in a representative subject. For this subject, the suprathreshold fMRI activity overlapped with 84% of the suprathreshold ERP activity, and the suprathreshold ERP activity overlapped with 80% of the fMRI activity.

Figure 4.

Figure 4.

Open in a new tab

Example of the spatial overlap estimation between ERP and fMRI source maps in a representative subject (#4237). Activity exceeding the significance threshold only in the ERP source map is shown in red, only in the fMRI source map is shown in blue, and in both the ERP and fMRI source maps is shown in green.

Figure 5A shows the count of subjects with suprathreshold activity in both imaging modalities at each vertex on the cortical surface mesh. Regions of consistent ERP and fMRI overlap in activity (in 9 or more subjects) were observed in the bilateral insula and inferior parietal lobule, and in the right superior temporal gyrus and ventral central sulcus. Additional regions with less consistent ERP and fMRI overlap (in 6-8 subjects) were observed in the middle temporal, frontal, and superior parietal cortices (primarily in the right hemisphere). Figure 5B shows six ROIs in which there was consistent ERP and fMRI overlap, and the time course of activation within the ROIs measured as the mean (across subjects) number of suprathreshold vertices. In the early (<300 ms) period of the ERP, there was only weak activity in the superior temporal cortex, peaking in the bilateral insula at 140 ms, and in the right superior temporal sulcus and insula at 242 ms. The later (>300 ms) ERP activity, which was stronger and sustained, originated from the perisylvian ROIs.

Figure 5.

Figure 5.

Open in a new tab

Group fMRI and ERP spatial overlap estimation (fMRI-ERP SOE) map. (A) Colormap representing the number of subjects with overlapping fMRI and ERP activity at each vertex. For visualization, the maps were thresholded at 6 subjects. B) ROIs with consistent (in 9 or more subjects) ERP and fMRI activity overlap, and the time course of activity in each ROI. The time course was measured as the mean (across subjects) number of suprathreshold vertices at each time point.

Figure 6 shows regions in which there was significant activity in one imaging modality but not the other, in 6 or more subjects. Small regions of non-overlap between fMRI and ERP activity were observed in the Sylvian fissure, and in the posterior superior temporal and orbitofrontal cortex. The maximal number of subjects with activity in only one modality was 8.

Figure 6.

Figure 6.

Open in a new tab

Non-overlap between fMRI and ERP source maps, computed with the fMRI-ERP SOE method. Non-overlap is expressed as the number of subjects with activity in one modality (ERP, top; fMRI, bottom) but not the other. For visualization, the maps were thresholded at 6 subjects. The color bar indicates the number of subjects with non-overlap at that location.

For comparison with a generic method (using only jICA), the spatial overlap between fMRI and ERP joint components that were thresholded independently is shown in Figure 7. A high degree of spatial overlap was observed across subjects within each imaging modality (up to 8 subjects), in temporoparietal regions implicated in the auditory oddball response. However, between ERP and fMRI maps, the regions of maximal overlap were limited to 4 subjects (Figure 7c).

Figure 7.

Figure 7.

Open in a new tab

Group spatial overlap in (A) ERP, (B) fMRI, and (C) both ERP and fMRI, estimated using strictly jICA, and independent thresholding of source maps in each neuroimaging modality. Overlap is expressed as the number of subjects with activity at each vertex. For visualization, the maps were thresholded at an overlap of 2 subjects (note the different scale of 2-8 subjects in this Figure).

The benefits of jICA with fMRI-ERP SOE versus just jICA can be seen from a comparison of Figures 5a and 7c. Figure 5a shows the spatial overlap across subjects between ERP and fMRI source maps calculated with the fMRI-ERP SOE method. Figure 7c shows the spatial overlap across subjects between fMRI and ERP source maps without SOE correction of the spatial bias related to the ERP source localization. The spatial overlap peaks at 13 subjects for the fMRI-ERP SOE method and at only four subjects for just jICA.

4. Discussion

In this paper, we present a data-driven method to estimate the spatial overlap between maps of brain activation originating from simultaneous fMRI and ERP measurements, termed fMRI-ERP SOE. The fMRI-ERP SOE method is based on the parsimonious assumption that simultaneous fMRI and ERP responses largely reflect local field potentials generated by the same neural sources. However, a distinctive feature of the fMRI-ERP SOE method is that it allows divergence in the sources of activity contributing to the measurements in each modality.

FMRI-ERP SOE of sources of the auditory oddball response revealed regions of consistent (in 9 or more of 15 subjects) activity overlap between imaging modalities in the insula, superior temporal, and inferior parietal cortices, in-line with previously reported sources of this response (Arumana, 2012; Justen and Herbert, 2018; Liebenthal et al., 2003; Linden et al., 1999; Polich, 2007). Strong activity in these areas was observed predominantly after 300 ms, in the time window of the P3 ERP response associated with cognitive processing during attentive oddball detection (Picton, 1992; Polich, 2007). The earlier (<300 ms) and weaker (due to subtraction of the level 1 response) activity coincided with the latencies of the N1 and MMN responses associated with automatic auditory processing and oddball detection (Campbell et al., 2007; Näätänen et al., 2007; Naatanen and Picton, 1987). The magnitude of the P3 is influenced by the cognitive context and demands of the task, whereas that of the N1 and MMN is influenced primarily by the physical properties of the stimuli. In the present experimental design in which the baseline (level 1) corresponded to chance deviant detection, the variation in deviant responses across the experimental levels (2-5, corresponding roughly to 65-95% deviant detection) reflected primarily a change in attentive perceptual and cognitive processing of the deviants, as indexed by P3, and minimally a change in subliminal auditory processing of sound frequency, as indexed by N1 and MMN.

The fMRI-ERP SOE method displayed important benefits relative to strictly jICA of fMRI and ERP. JICA with fMRI-ERP SOE revealed a greater extent of reliable activation in the right superior temporal cortex, considered to contribute to the generation of the auditory oddball response, than just jICA of the same data. Consistent activity was detected in the right superior temporal sulcus in both modalities when estimated with fMRI-ERP SOE (Figure 5A), but not when estimated strictly with jICA. Activity in the right superior temporal sulcus was observed in the jICA-fMRI map (Figure 7B) but not the jICA-ERP map (Figure 7A), resulting in no overlap in this area with jICA. The weak EEG sensitivity to sources of activity in the superior temporal cortex may be due to incomplete (and inconsistent across subjects) coverage of this area by EEG electrodes especially in the ventral portion. Indeed, a study comparing fMRI and ERP activation maps in visual and auditory paradigms reported fMRI-ERP uncoupling specifically in auditory superior temporal areas, and attributed it to sparse EEG coverage (Minati et al., 2008).

The advantages of using fMRI-ERP SOE to characterize the spatial relationships between fMRI and ERP sources center on the 1) projection of both datasets into a common (non-native) source space that accounted for the effects of spatial bias during source localization; and 2) the optimization of the fMRI and ERP thresholds to maximize the overlap between the sources measured with each method. In this sense, the considerably lower overlap of using jICA alone (i.e., when assumptions of neuroimage coupling are minimized), could be related to the spatial error associated with ERP source localization, as well as the arbitrary nature of neuroimaging data thresholding.

The fMRI-ERP SOE also revealed brain regions in which activity was less consistent across subjects, notably in the right inferior frontal cortex (Figure 5A). The right inferior frontal cortex is considered to be part of a ventral attention network involved in stimulus-driven orientation and deviance detection (Justen and Herbert, 2018; Knight, 1984; Pardo et al., 1991; Posner, 1992, 1990; Soltani and Knight, 2000). The variability in activation of right frontal areas could reflect attentional fluctuations and individual differences in cognitive control during task performance. However, differences in measurement quality could also contribute to inconsistent signal amplitudes in frontal cortex. Specifically, frontal activity measured with EEG could in some instances be generated by tangentially oriented sources in bilateral temporal cortex as opposed to radially oriented frontal sources (Ahlfors and Hamalainen, 2012). In line with the possibility of ‘ghost’ frontal sources in the present ERP data, the most consistent activation across subjects in the jICA-ERP map was in the right inferior frontal cortex (8 subjects, Figure 7A), but this area was inconsistently activated in the jICA-fMRI map (2 subjects, Figure 7B).

Across the group, approximately 73% of the activity at each brain location was measured with both neuroimaging modalities, and the remainder 27% was measured in one modality but not the other (Table 1, Figure 3). The activity measured in only one of the modalities largely fell adjacent to areas of ERP-fMRI spatial overlap (see Figure 4, in a representative subject) and was in the same vicinity in both modalities (e.g., sylvian fissure), suggesting that it reflected primarily differences in the extent of overlapping activation rather than complete uncoupling between the modalities. However, there were also regions of single modality activity that were not adjacent to areas of overlap; for example, activity measured only with EEG in the right inferior frontal cortex and bilateral parietotemporal cortex in the representative subject. The nonoverlap group map (Figure 6), however, showed that areas of non-overlap were largely inconsistent across subjects. Taken together, the present results suggest the existence of comparatively small areas of modality uncoupling, in variable brain locations. The results demonstrate the potential utility of the fMRI-ERP SOE method to investigate the factors contributing to uncoupling between the fMRI and EEG measurements. The strong positive correlation, with a near unit-slope, between the extent of fMRI and ERP spatial overlap (Figure 3) is also indicative of limited uncoupling between the measurements in each modality. In this analysis, uncoupling between the modalities, i.e., activity measured with one neuroimaging modality but not the other, would be observed as a positive correlation with a steeper slope (reflecting more extensive ERP activity), or a shallower slope (reflecting more extensive fMRI activity).

The optimal amplitude and temporal thresholds for maximizing the spatial overlap between imaging modalities spanned a wide range, with ERP z-scores between 0.25 and 7.3, ERP temporal thresholds between 20 and 740 ms, and fMRI z-scores between 3.9 and 21.6. The amplitude and temporal ERP thresholds were found to be negatively correlated, suggesting a trade-off between them. The amplitude thresholds were generally higher for fMRI than ERP, likely because of the additional temporal threshold applied strictly to ERP signals. The threshold values identified as optimal for maximizing overlap were in some subjects well outside the range typically used for neuroimaging data analysis, yet their application resulted in statistically significant and biologically plausible activation maps. Thus, the proposed threshold optimization and fMRI-ERP SOE analysis may have value for exploring data more comprehensively than possible with more typical analysis approaches.

The fMRI-ERP SOE approach shares some similarities with neurogenerative modeling approaches. Both use statistical methods to investigate the relationship between fMRI and EEG without imposing a constraint of coupling between the signals. However, neurogenerative approaches attempt to reduce the errors associated with EEG source reconstruction by informing the forward generative model with physiological parameters. Our approach on the other hand, attempts to reduce the spatial discrepancies between EEG and fMRI by forward modeling the fMRI as an EEG scalp distribution and applying the same source reconstruction procedure to both modalities (i.e., biasing the fMRI to the EEG spatial scale). While the accuracy of neurogenerative approaches depends on the accuracy of the EEG forward model, our approach accepts that EEG source reconstruction is imprecise and applies the same bias to the fMRI. Thus, the methods can perhaps best be seen as complementary. For example, the fMRI-ERP SOE can be used to identify areas of coupling and uncoupling, and this information can be used to constrain and improve the precision of a neurogenerative source model.

The fMRI-ERP SOE is a data-driven method to quantitatively estimate the spatial relationship between sources of brain activity measured with fMRI and EEG. The present study demonstrated that the addition of this method provides greater spatiotemporal detail of the cortical dynamics than solely jICA, a common method for multimodal integration (Arumana, 2012; Ma et al., 2013; Mangalathu-Arumana et al., 2012; Moosmann et al., 2008). Furthermore, the fMRI-ERP SOE method provides a means to estimate the degree of non-overlap between the sources measured with each neuroimaging modality, and this aspect could be useful to study uncoupling. As such, we propose that the addition of fMRI-ERP SOE provides a more comprehensive method for the integration of data from the two neuroimaging modalities. The fMRI-ERP SOE method could be used to study the conditions under which uncoupling can occur. For example, at high stimulation rates in healthy individuals (Goense and Logothetis, 2008; Lachaux et al., 2007; Muthukumaraswamy and Singh, 2008; Nagarajan et al., 1999) and in individuals with compromised neurovascular coupling (Faraci et al., 1990; Kazama et al., 2003; Mackert et al., 2008).

One limitation of the approach is the long computation time. The exhaustive search of the amplitude and temporal threshold space performed to maximize spatial overlap is computationally demanding, requiring several hours of processing per subject. In experimenting with optimization approaches, we consistently found that the fMRI-ERP overlap metric plateaued for different ERP amplitude and temporal threshold combinations (see Figure 2 for an example in one subject). Due to this, gradient based approaches that would have sped-up the optimization process were inadequate because they would not consistently converge to the global maximum. To ensure detection of the threshold combination that maximized the spatial overlap between fMRI and ERP measures, it was necessary to employ an exhaustive search of the threshold space in the present study.

In the future, the fMRI-ERP SOE method could be refined and expanded in several ways. First the exhaustive search to optimize thresholds could be made more efficient by refining the class membership function to emphasize sensitivity to a global solution in the amplitude/temporal threshold space. This would in turn facilitate more efficient searches for the optimal thresholds using, for example, gradient-based approaches. The fMRI-ERP SOE approach could be applied to continuous EEG/fMRI data by convolving the fMRI timeseries with the simulated evoked time course. Finally, the fMRI-ERP SOE could also be examined when subcortical sources are also modeled.

In summary, the new fMRI-ERP SOE analysis pipeline for estimating the overlap of sources of activity measured simultaneously with fMRI and ERP revealed the dynamics of perisylvian regions associated with auditory oddball detection at a spatiotemporal detail not available with measurements from just one of the imaging modalities, or strictly jICA of measurements from both modalities (Mangalathu-Arumana et al., 2012). The fMRI-ERP SOE suggested that areas of non-overlap in sources of activity between the modalities were relatively small and inconsistent across subjects, at least in this paradigm. Future research should examine the factors contributing to uncoupling between fMRI and ERP measurements, whether physiological (e.g., due to individual differences in neuroanatomy or function), and/or methodological (e.g., due to modality differences in imaging sensitivity in specific brain regions).

Highlights.

  • A new method is proposed to estimate spatial overlap between fMRI and ERP source maps

  • A key feature of the approach is that the fMRI activity is simulated as ERP activity

  • fMRI and ERP thresholds are then optimized to maximize overlap

  • In task-related areas, fMRI and ERP activity overlapped in 9 or more of 15 subjects

  • Areas of fMRI-ERP non-overlap were small and inconsistent across subjects

5. Acknowledgements:

This research was supported in part by the National Center for Advancing Translational Sciences, National Institutes of Health, through Grant Number UL1TR001436, Advancing a Healthier Wisconsin Research and Education Program, National Institute on Deafness and Communication Disorders R01 DC006287, and National Science Foundation awards OCI-0923037 and CBET-0521602.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

6. References:

  1. Aftanas LI, Lotova NV, Koshkarov VI, Makhnev VP, Mordvintsev YN, Popov SA, 1998. Non-linear dynamic complexity of the human {EEG} during evoked emotions. Int. J. Psychophysiol 28, 63–76. [DOI] [PubMed] [Google Scholar]
  2. Ahlfors SP, Hamalainen M, 2012. MEG and EEG: source estimation, in: Handbook of Neural Activity Measurement. Cambridge University Press, Cambridge, UK, pp. 257–285. [Google Scholar]
  3. Arumana JM, 2012. Integration of EEG-fMRI in an Auditory Oddball Paradigm Using Joint Independent Component Analysis. Marquette University. [Google Scholar]
  4. Bénar C, Schön D, Grimault S, Nazarian B, Burle B, Badier M, Marquis P, Liegeois-Chauvel C, Anton J, 2007. Single-trial analysis of oddball event-related potentials in simultaneous EEG-fMRI. Hum. Brain Mapp 28, 602–613. [DOI] [PubMed] [Google Scholar]
  5. Bobes M, Lage-Castellanos A, Olivares E, Hidalgo-Gato J, Iglesias J, Castro-Laguardia AM, Valdes-Sosa P, 2018. ERP Source Analysis Guided by fMRI During Familiar Face Processing. Brain Topogr. 1–21. [DOI] [PubMed] [Google Scholar]
  6. Campbell T, Winkler I, Kujala T, 2007. N1 and the mismatch negativity are spatiotemporally distinct ERP components: Disruption of immediate memory by auditory distraction can be related to N1. Psychophysiology 44, 530–540. 10.1111/j.1469-8986.2007.00529.x [DOI] [PubMed] [Google Scholar]
  7. Cox R, 1996. AFNI: Software for Analysis and Visualization of Functional Magnetic Resonance Neuroimages. Comput. Biomed. Res 173, 162–173. [DOI] [PubMed] [Google Scholar]
  8. Dale AM, Liu AK, Fischl BR, Buckner RL, Belliveau JW, Lewine JD, Halgren E, 2000. Dynamic Statistical Parametric Mapping: Combining fMRI and MEG for High-Resolution Imaging of Cortical Activity. Neuron 26, 55–67. [DOI] [PubMed] [Google Scholar]
  9. Debener S, 2005. Trial-by-Trial Coupling of Concurrent Electroencephalogram and Functional Magnetic Resonance Imaging Identifies the Dynamics of Performance Monitoring. J. Neurosci. 25, 11730–11737. 10.1523/JNEUROSCI.3286-05.2005 [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Ellingson ML, Liebenthal E, Spanaki MV, Prieto TE, Binder JR, Ropella KM, 2004. Ballistocardiogram artifact reduction in the simultaneous acquisition of auditory ERPS and fMRI. Neuroimage 22, 1534–1542. 10.1016/j.neuroimage.2004.03.033 [DOI] [PubMed] [Google Scholar]
  11. Faraci FM, Baumbach GL, Heistad DD, Faraci FM, Baumbach GL, Heistad DD, 1990. Cerebral Circulation: Humoral Regulation and Effects of Chronic Hypertension. J. Am. Soc. Nephrol 1(1), 53–57. [DOI] [PubMed] [Google Scholar]
  12. Goense JBM, Logothetis NK, 2008. Neurophysiology of the BOLD fMRI Signal in Awake Monkeys. Curr. Biol 18, 631–640. 10.1016/J.CUB.2008.03.054 [DOI] [PubMed] [Google Scholar]
  13. Hall DA, Haggard MP, Akeroyd MA, Palmer AR, Summerfield AQ, Elliot MR, G EM, Bowtell RW, 1999. “sparse” temporal sampling in auditory fMRI. Hum. Brain Mapp 7, 213–223. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Huster RJ, Debener S, Eichele T, Herrmann CS, 2012. Methods for Simultaneous EEG-fMRI: An Introductory Review. J. Neurosci 32, 6053–6060. 10.1523/JNEUROSCI.0447-12.2012 [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Jann K, Dierks T, Boesch C, Kottlow M, Strik W, Koenig T, 2009. BOLD correlates of EEG alpha phase-locking and the fMRI default mode network. Neuroimage. 10.1016/j.neuroimage.2009.01.001 [DOI] [PubMed] [Google Scholar]
  16. Justen C, Herbert C, 2018. The spatio-temporal dynamics of deviance and target detection in the passive and active auditory oddball paradigm: A sLORETA study. BMC Neurosci. 19, 1–18. 10.1186/s12868-018-0422-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Kazama K, Wang G, Frys K, Anrather J, Iadecola C, 2003. Angiotensin II attenuates functional hyperemia in the mouse somatosensory cortex. Am. J. Physiol. Circ. Physiol 285, H1890–H1899. 10.1152/ajpheart.00464.2003 [DOI] [PubMed] [Google Scholar]
  18. Knight RT, 1984. Decreased response to novel stimuli after prefrontal lesions in man. Electroencephalogr. Clin. Neurophysiol 59, 9–20. [DOI] [PubMed] [Google Scholar]
  19. Kujala T, Alho K, Kekoni J, Hämäläinen H, Reinikainen K, Salonen O, Standertskjöld-Nordenstam C-G, Näätänen R, 1995. Auditory and somatosensory event-related brain potentials in early blind humans. Exp. Brain Res 104, 519–526. 10.1007/BF00231986 [DOI] [PubMed] [Google Scholar]
  20. Lachaux J-P, Fonlupt P, Kahane P, Minotti L, Hoffmann D, Bertrand O, Baciu M, 2007. Relationship between task-related gamma oscillations and BOLD signal: New insights from combined fMRI and intracranial EEG. Hum. Brain Mapp 28, 1368–1375. 10.1002/hbm.20352 [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Liebenthal E, Ellingson ML, Spanaki MV, Prieto TE, Ropella KM, Binder JR, 2003. Simultaneous ERP and fMRI of the auditory cortex in a passive oddball paradigm. Neuroimage 19, 1395–1404. [DOI] [PubMed] [Google Scholar]
  22. Linden DEJ, Prvulovic D, Formisano E, Völlinger M, Zanella FE, Goebel R, Dierks T, Goethe-universität JW, 1999. The Functional Neuroanatomy of Target Detection: An fMRI Study of Visual and Auditory Oddball Tasks. Cereb. Cortex 9, 815–823. [DOI] [PubMed] [Google Scholar]
  23. Liu Z, He B, 2008. FMRI-EEG Integrated Cortical Source Imaging by use of Time-Variant Spatial Constrains. Neuroimage 39, 1198–1214. 10.1146/annurev.neuro.31.060407.125627.Brain [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Logothetis N, 2008. What we can do and what we cannot do with fMRI. Nature 453, 869–78. 10.1038/nature06976 [DOI] [PubMed] [Google Scholar]
  25. Logothetis NK, Pauls J, Augath M, Trinath T, Oeltermann A, 2001. Neurophysiological investigation of the basis of the fMRI signal. Nature 412, 150–157. [DOI] [PubMed] [Google Scholar]
  26. Ma S, Phlypo R, Calhoun VD, 2013. Capturing Group Variability using IVA: A Simulation Study and Graph-theoretical Analysis. University of Maryland, Baltimore County, Dept. of CSEE, Baltimore, MD 21250: The Mind Research Network and University of New Mexico, Dept. of ECE, Albuquerqu 3128–3132. [Google Scholar]
  27. Mackert B-M, Leistner S, Sander T, Liebert A, Wabnitz H, Burghoff M, Trahms L, Macdonald R, Curio G, 2008. Dynamics of cortical neurovascular coupling analyzed by simultaneous DC-magnetoencephalography and time-resolved near-infrared spectroscopy. Neuroimage 39, 979–986. 10.1016/J.NEUROIMAGE.2007.09.037 [DOI] [PubMed] [Google Scholar]
  28. Mangalathu-Arumana J, Beardsley SA, Liebenthal E, 2012. Within-subject joint independent component analysis of simultaneous fMRI/ERP in an auditory oddball paradigm. Neuroimage. 10.1016/j.neuroimage.2012.02.030 [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Mathiesen C, Caesar K, Akgören N, Lauritzen M, 1998. Modification of activity-dependent increases of cerebral blood flow by excitatory synaptic activity and spikes in rat cerebellar cortex. J. Physiol. 512, 555–566. 10.1111/j.1469-7793.1998.555be.x [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Mazzoni A, Whittingstall K, Brunel N, Logothetis NK, Panzeri S, 2010. Understanding the relationships between spike rate and delta/gamma frequency bands of LFPs and EEGs using a local cortical network model. Neuroimage 52, 956–972. [DOI] [PubMed] [Google Scholar]
  31. Minati L, Rosazza C, Zucca I, D’Incerti L, Scaioli V, Bruzzone MG, 2008. Spatial correspondence between functional MRI (fMRI) activations and cortical current density maps of event-related potentials (ERP): A study with four tasks. Brain Topogr. 21, 112–127. 10.1007/s10548-008-0064-3 [DOI] [PubMed] [Google Scholar]
  32. Mizuhara H, Wang LQ, Kobayashi K, Yamaguchi Y, 2005. Long-range EEG phase synchronization during an arithmetic task indexes a coherent cortical network simultaneously measured by fMRI. Neuroimage. 10.1016/j.neuroimage.2005.04.030 [DOI] [PubMed] [Google Scholar]
  33. Moosmann M, Eichele T, Nordby H, Hugdahl K, Calhoun VD, 2008. Joint independent component analysis for simultaneous EEG – fMRI: Principle and simulation 67, 212–221. 10.1016/j.ijpsycho.2007.05.016 [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Murta T, Leite M, Carmichael DW, Figueiredo P, Lemieux L, 2015. Electrophysiological correlates of the BOLD signal for EEG-informed fMRI. Hum. Brain Mapp 36, 391–414. 10.1002/hbm.22623 [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Muthukumaraswamy SD, Singh KD, 2008. Spatiotemporal frequency tuning of BOLD and gamma band MEG responses compared in primary visual cortex. Neuroimage 40, 1552–1560. 10.1016/J.NEUROIMAGE.2008.01.052 [DOI] [PubMed] [Google Scholar]
  36. Näätänen R, Paavilainen P, Rinne T, Alho K, 2007. The mismatch negativity (MMN) in basic research of central auditory processing: A review. Clin. Neurophysiol 118, 2544–2590. 10.1016/j.clinph.2007.04.026 [DOI] [PubMed] [Google Scholar]
  37. Naatanen R, Picton T, 1987. The N1 Wave of the Human Electric and Magnetic Response to Sound: A Review and an Analysis of the Component Structure. Psychophysiology 24, 375–425. [DOI] [PubMed] [Google Scholar]
  38. Nagarajan S, Mahncke H, Salz T, Tallal P, Roberts T, Merzenich MM, 1999. Cortical auditory signal processing in poor readers. Proc. Natl. Acad. Sci. U. S. A. 96, 6483–8. 10.1073/pnas.96.11.6483 [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Ou W, Nummenmaa A, Golland P, Hämäläinen MS, 2009. Multimodal Functional Imaging Using fMRI-Informed Regional EEG/MEG Source Estimation. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Pardo JV, Fox PT, Raichle ME, 1991. Localization of a human system for sustained attention by positron emission tomography. Nature 349, 61–4. 10.1038/349061a0 [DOI] [PubMed] [Google Scholar]
  41. Picton TW, 1992. The P300 wave of the human event-related potential. J. Clin. Neurophysiol 9, 456–479. 10.1097/00004691-199210000-00002 [DOI] [PubMed] [Google Scholar]
  42. Polich J, 2007. Updating P300: An Integrative Theory of P3a and P3b. Clin. Neurophysiol 118, 2128–2148. 10.1016/j.clinph.2007.04.019.Updating [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Portnova GV, Tetereva A, Balaev V, Atanov M, Skiteva L, Ushakov V, Ivanitsky A, Martynova O, 2018. Correlation of BOLD Signal with Linear and Nonlinear Patterns of EEG in Resting State EEG-Informed fMRI. Front. Hum. Neurosci 11, 654 10.3389/fnhum.2017.00654 [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Posner M, 1992. Attention as a cognitive neural system. Curr. Dir. Psychol. Sci 1, 11:14. [Google Scholar]
  45. Posner M, 1990. The Attention System Of The Human Brain. Annu. Rev. Neurosci 13, 25–42. 10.1146/annurev.neuro.13.1.25 [DOI] [PubMed] [Google Scholar]
  46. Rosa MJ, Daunizeau J, Friston KJ, 2010. EEG-fMRI Integration: A Critical Review of Biophysical Modeling and Data Analysis Approaches. J. Integr. Neurosci 9, 453–476. 10.1142/S0219635210002512 [DOI] [PubMed] [Google Scholar]
  47. Rosa MJ, Kilner J, Blankenburg F, Josephs O, Penny W, 2010. Estimating the transfer function from neuronal activity to BOLD using simultaneous EEG-fMRI. Neuroimage 49, 1496–1509. 10.1016/j.neuroimage.2009.09.011 [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Sidak ZK, 1967. Rectangular Confidence Regions for the Means of Multivariate Normal Distributions. J. Am. Stat. Assoc 10.1080/01621459.1967.10482935 [DOI] [Google Scholar]
  49. Soltani M, Knight RT, 2000. Neural Origins of the P300. Crit. Rev. Neurobiol 14, 26 10.1615/CritRevNeurobiol.v14.i3-4.20 [DOI] [PubMed] [Google Scholar]
  50. Sotero RC, Trujillo-Barreto NJ, 2008. Biophysical model for integrating neuronal activity, EEG, fMRI and metabolism. Neuroimage 39, 290–309. 10.1016/j.neuroimage.2007.08.001 [DOI] [PubMed] [Google Scholar]
  51. Sotero RC, Trujillo-Barreto NJ, 2007. Modelling the role of excitatory and inhibitory neuronal activity in the generation of the BOLD signal. Neuroimage 35, 149–165. 10.1016/j.neuroimage.2006.10.027 [DOI] [PubMed] [Google Scholar]
  52. Viswanathan A, Freeman RD, 2007. Neurometabolic coupling in cerebral cortex reflects synaptic more than spiking activity. Nat. Neurosci 10, 1308–1312. 10.1038/nn1977 [DOI] [PubMed] [Google Scholar]
  53. Xu J, Sheng J, Qian T, Luo Y, Gao J, 2018. EEG/MEG source imaging using fMRI informed time-variant constraints. Hum. Brain Mapp 39, 1700–1711. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES