SSVEP Signatures of Binocular Rivalry During Simultaneous EEG and fMRI

Abstract

Background

Binocular rivalry is a perceptual phenomenon that arises when two incompatible images are presented separately, one to each eye, and the observer experiences involuntary perceptual alternations between the two images. If the two images are flickering at two distinct frequencies, electroencephalography (EEG) can be used to track the frequency-tagged steady-state visually evoked potential (SSVEP) driven by each image as they compete for awareness, providing an objective measure of the subjective perceptual state. This spontaneous alternation in perceptual dominance is believed to be driven by neural processes across widespread regions in the brain, but the real-time mechanisms of these processes remain unclear.

New Method

The goal of this study was to determine the feasibility of investigating binocular rivalry using a simultaneous EEG-fMRI approach in order to leverage the high temporal resolution of EEG with the high spatial resolution of fMRI.

Results

We have developed novel techniques for artifact removal and signal optimization for the rivalry-related SSVEP data collected simultaneously during fMRI.

Comparison with Existing Methods

Our methods address several significant technical challenges of recording SSVEP data in the noisy fMRI environment, and enabled us to successfully reconstruct SSVEP signatures of rivalry in a group of healthy human subjects.

Conclusion

Further development and application of these techniques will enable more comprehensive integration of EEG and fMRI data collected simultaneously and could have significant implications for EEG-fMRI studies of brain activity in general.

Introduction

Binocular rivalry is a perceptual phenomenon that arises when two incompatible images are presented, one to each eye, and the subject experiences involuntary perceptual alternations between the two images. The stimuli from corresponding points in each retina converge in early visual cortex, from where the two inputs compete for dominance to produce a unified perceptual outcome [1, 2]. Because awareness changes over time while the stimulus is held constant, binocular rivalry has become a widely-used paradigm for studying the neural correlates of conscious perception. Details of the specific neural computations involved in resolving rivalry remain obscure, however.

For example, the resolution of rivalry is thought to involve not only mutual inhibition between the two inputs, but also the engagement of neural populations comprising a distributed network of feed-forward and feedback connections [3-6]. Our group previously showed that attention is required for sustained rivalry between dichoptic images [6]. Several other studies have also suggested the importance of attentional modulation of binocular rivalry and the idea that this modulation is driven by top-down control factors [7–10]. There is also evidence suggesting that frontal-parietal regions are involved in top-down control of perceptual switches, but it is unknown how these regions interact with bottom-up processes of rivalry in the early visual cortex [11–16].

Binocular rivalry has been studied using behavioral experiments, invasive single-unit or local field potential (LFP) recordings, as well as non-invasive neuroimaging techniques such as electroencephalography (EEG) and functional MRI (fMRI). One important feature of some of these studies is the use of stimuli that allow signals from each eye to be separated and tracked One way to achieve this is to use “frequency tagging”, in which the two stimuli flicker at different temporal frequencies, generating steady-state visually evoked potentials (SSVEPs) at those two frequencies which can then be detected using electrophysiological measures [17, 18]. Brown and Norcia (1997) showed that when a subject reports perceiving one eye's stimulus, the EEG shows a much larger oscillation at that eye's stimulation frequency. Thus, by measuring the amplitudes of the EEG signals at these two frequencies in real-time, one can obtain a direct measure of the status of the underlying neural network, which is highly predictive of the subjective perceptual report

While these SSVEP signatures provide reliable information about the temporal dynamics of binocular rivalry, scalp EEG is inherently limited in spatial resolution, making it difficult to assess the interplay of bottom-up and top-down mechanisms of rivalry. Functional MRI (fMRI), on the other hand, can provide precise localization of the neural activity, but with relatively low temporal resolution. Thus, fMRI can help pinpoint brain regions active during a period of ongoing rivalry but cannot address the exact temporal sequence of activation between these regions. Previous imaging studies of binocular rivalry have been limited to the use of a single imaging modality (often either EEG or fMRI).

Simultaneous EEG-fMRI leverages the high temporal resolution and low spatial resolution of EEG (a direct measure of electrical neural activity) with the low temporal resolution and high spatial resolution of fMRI (an indirect measure of neural activity). In this manner we can directly compare real-time electrophysiological data from the EEG with hemodynamic BOLD data from the fMRI. Analyses that combine these signals, such as EEG-informed fMRI and fMRI-constrained EEG source localization provide important information not available using either modality on its own [19–21]. Further integration of the neuroimaging data with behavioral data can provide even more powerful analyses, where ongoing brain activity during binocular rivalry is compared to the spontaneous perceptual switches reported by subjects.

However, recording the fMRI and EEG signals simultaneously and obtaining useful data remain a significant technical challenge. In particular, the electrical noise induced in the EEG recording apparatus by the large magnetic and RF electrical fields is difficult to remove. Efforts have been made to develop methods for integrated multimodal neuroimaging, in particular combining fMRI with EEG (or MEG) [19–23].

Little work has been reported, to our knowledge, on the integration of frequency-tagging SSVEP methods with functional MRI, though initial efforts to more generally study SSVEPs using EEG-fMRI have been promising [24]. Further development of these techniques as applied to binocular rivalry could significantly improve our understanding of its neural mechanisms. Given the very limited SNR of traditional SSVEP signals, improved EEG processing procedures are critical in order to extract high-quality continuous SSVEPs from EEG data collected in the MRI scanner.

The goal of the research reported here was to develop methods to improve the quality of SSVEP signals gathered during fMRI scanning. We collected simultaneous EEG-fMRI data during blocks of continuous binocular rivalry using frequency-tagged stimuli. A “replay” condition, where subjects were presented with a single alternating image to both eyes, was used as a control. We report improved methods for several stages of data processing that enhanced the quality of the extracted SSVEP signals of interest, including: 1) identification and removal of the cardioballistic artifact; 2) spatial filtering of the SSVEP signals to isolate the frequencies of interest; 3) estimation of the time-varying amplitude of the SSVEP components at the frequencies of interest.

Methods

Subjects

Twenty-three healthy human subjects (17 male, 6 female, median age 23 years) participated in the study after providing informed written consent in accordance with the Institutional Review Board at the University of Minnesota. An additional 9 subjects (3 male, 6 female) were recruited, but were excluded after screening tests revealed difficulty perceiving the binocular rivalry stimulus.

Stimulus Design

Subjects viewed a dichoptic stimulus consisting of a red and black circular dartboard pattern presented to one eye, and a similar green and black pattern to the other, flickering at different spatial frequencies (See Fig. 1a). Each pattern was presented within an annular window extending from 0.5 to 3.75 deg visual angle. Stimuli were presented on a neutral gray background with a central fixation cross, and the stimuli rotated at 11.25°/sec. The red and green stimuli contrast reversed at 6.67 and 8.54 Hz, respectively. All visual stimuli were projected onto a screen inside the MR scanner with a 60 Hz refresh rate. Stimulus frequencies were optimized and selected prior to the EEG-fMRI sessions based on our group's EEG-alone and behavioral studies of various SSVEP rivalry stimuli. The flicker frequencies 6.67 Hz and 8.54 Hz are easily implemented with a 60 Hz display, and they are perceptually similar, reducing potential stimulus bias that might arise using disparate flicker rates. In previous EEG-alone studies, we used 6.67 Hz and 7.5 Hz, but there was strong residual MR artifact at 7.5 Hz, so 8.54 Hz was selected instead. Importantly, the chosen frequencies of 6.67 and 8.54 Hz induced robust perceptual shifts during rivalry across subjects without inducing excessive mixing/fusion periods. This was essential for obtaining dominance durations long enough (2-5 seconds) for adequately reconstructing the target SSVEP signals using our chosen algorithms. We found that subjects had difficulty in resolving rivalry for sufficient durations using lower frequency stimuli, possibly due to the visual complexities of the rivaling images. Additionally, it is possible that higher frequency (>10 Hz) stimuli could have been used to induce even stronger SSVEP responses in the visual cortex and improve overall SNR in the scanner. However, higher frequencies resulted in similar problems with excessive perceptual mixing of the rivaling images. At the beginning of each EEG-fMRI session, SSVEP stimuli were first balanced for mean luminance, after which small adjustments were made for each subject to equalize perceived brightness.

Figure 1. Binocular rivalry stimulus design.

Left) Rivalry condition and Right) Smooth replay condition, showing the red wedge smoothly expanding over a 1 sec interval.

The stimulus for the control condition (“replay”) was designed to approximate the experience of binocular rivalry, but with each eye receiving identical images that physically alternated between the two patterns. The physical alternations between patterns were gradual rather than abrupt, with a small wedge of one pattern smoothly expanding to obscure the other over the course of 1 second (See Fig. 1b). The duration of each red or green epoch was randomly drawn from durations reported by that subject during the rivalry condition.

In both rivalry and replay conditions, subjects were instructed to maintain fixation on the center of the screen, and report whether they were seeing the red or green stimulus by holding down the 1 or 2 button, or to hold down both if the stimulus was a roughly equal mix of the two.

Stimulus Presentation

Subjects viewed the binocular rivalry stimulus inside the MRI scanner through prism lenses that redirected two images on the left and right halves of the screen to both appear at the same location in the center of the subject's visual field [25]. A standard headcoil-mounted mirror allowed subjects to view the stimuli on a screen at the top of the bore while in a supine position. The prism lenses were 10PD 37mm acrylic squares (Bernell Corporation), mounted 6mm apart on a transparent acrylic plate. We constructed a vertical divider that extended from the headcoil mirror to the center of the screen at the back of the scanner bore to prevent cross-talk between the two halves of the stimulus. This divider, as well as the inside surface of the scanner bore, were covered with black felt to reduce reflections in the peripheral visual field.

Simultaneous EEG and fMRI Acquisition

EEG and ECG were recorded using a 64-channel MRI-compatible amplifier (BrainAmp MRplus, Brain Products). All signals were referenced to an electrode at the FCz position, and sampled at 5000 Hz. A single ECG electrode was placed on the subject's upper back. After preparing the cap with conductive gel, electrode impedances were all below 20 kOhm. The cables were routed from the electrode cap, parallel to the axis of the scanner bore, to the amplifiers and battery which were placed as far from the scanner isocenter as possible. In order to ensure the gradient-induced artifacts were as consistent as possible, EEG sampling was synchronized to the timing of the MRI gradients using the SyncBox provided with the EEG system, which connected directly to the 10MHz clock on the gradient computer. After completing the EEG-fMRI session, we recorded the position of each electrode on the scalp using a magnetic 3D tracking system (FasTrak, Polhemus).

Structural and functional MRI were acquired using a Siemens Skyra 3T scanner with a custom, high slew-rate gradient insert developed for use in the Human Connectome Project [26]. We used a 16-channel receive-only head coil, which included gaps near the vertex of the head through which our EEG cables could pass. We acquired whole-brain BOLD functional data using a typical GE-EPI pulse sequence (FA=90°; TR=2200ms; TE=30ms; 3mm isotropic voxels; 36 axial slices; with fat saturation pulse).

EEG Signal Preprocessing

The MR gradient artifact was removed using a PCA-based optimal basis set (OBS) algorithm [27]. When detecting and removing the cardioballistic artifact (CBA), there were many subjects whose ECG recordings were of poor quality, most commonly due to subject motion. Accordingly, we selected the channel with the most pronounced and consistent artifact, by examining the autocorrelogram for the EEG and ECG electrodes. We then determined the timing of each heartbeat artifact in this channel using an R-peak detection algorithm adapted from Liu et al [28]. The ECG R-peak is a large, fast deflection occurring once per beat. The EEG artifact, however, may contain multiple peaks of similar size and shape, and the detection algorithm was not always consistent as to which peak it identified. An additional realignment step was implemented that shifted each heartbeat by the peak lag in the cross-correlogram between the artifact and the template.

The final CBA correction procedure was based on a combination of ICA, OBS, and an information-theoretic rejection criterion [28]. Briefly, the signal was decomposed into independent components, which were rejected if the mutual information between the component's time course and the CBA artifact was sufficiently high. The remaining components were then divided into epochs around each heartbeat and an optimal basis set was obtained across all epochs to fit and remove the artifacts (Fig. 3).

Figure 3. EEG preprocessing pipeline.

Precise determination of heartbeat timing was crucial for removing the artifact. When ECG recordings were corrupted, we identified the EEG electrode with the strongest autocorrelation peak (Channel selection inset). EEG channels lack the distinct peak present in ECG, and detected artifacts are frequently misaligned (Artifact realignment insert, red arrows). We realigned artifacts after initial detection by their peak cross-correlation lag.

Detection of noisy electrodes and data epochs was performed before CBA detection, and again after CBA correction. Each time, noise-contaminated electrodes were identified visually from an average power histogram. Epochs were excluded if the mean global field power (mGFP) exceeded roughly 5 standard deviations above the mean. Electrodes were first re-referenced to a common average of electrodes connected to the same amplifier, and then to the combined average. Each amplifier had 32 channels, and these were distributed in an interlaced grid such that each set of 32 electrodes covers the entire scalp. The routing of each cable from the electrode cap to its amplifier was different, often creating two distinct patterns of baseline shifts that need to be corrected individually. This process of identifying bad electrodes and epochs and re-referencing was repeated to avoid excluding electrodes whose noise was restricted to globally noisy time windows and vice versa.

Optimized Spatial Filter

When recording SSVEPs outside the MRI scanner, a surface Laplacian [29] centered on an occipital EEG electrode can usually be used to reject background noise and produce a time course with adequate SSVEP signal strength for further analysis [6, 17]. In the MR environment, however, residual gradient and cardioballistic noise are often so strong that use of the aforementioned filters including PCA, ICA and Surface Laplacian is not sufficient for weak brain signals.

In this study, we developed a non-linear optimization procedure to identify a set of channel weights for each subject that, when multiplied by the EEG signal, produced a single time course with increased SSVEP signal-to-noise ratio in the frequency domain. Given a multichannel EEG time series S, an N × T channel-by-time matrix, and N × 1 channel weight vector w, we derived a 1 × T spatially-filtered time course s_w = w^T S. The spectral SNR for this time course was defined as the magnitude of the Fourier transform at a given target frequency divided by the average magnitude of a narrow band (±0.5Hz) of neighboring frequencies (Eq (1)), adapted from [6].

$$SpecSNR(s,f)=\frac{\mid {\mathcal{F}}_f\left(s\right)\mid }{\frac{1}{2\eta }\sum _{m=-\eta }^{\eta }\mid {\mathcal{F}}_{f+m}\left(s\right)\mid },\eta =0.5\mathrm{Hz}$$ (1)

Thus, for each frequency-tagged SSVEP signal of interest, we obtain a quantitative measure of its SNR that can be maximized in an iterative fashion based on the weight vector w. The spectral SNR can be similarly calculated for the harmonics of the frequency-tagged SSVEPs. The overall SSVEP quality of the signal filtered using a given set of weights w, was defined as the sum of the spectral SNR for both SSVEP frequencies as well as their second- and third-order harmonics, weighted by a scalar α < 1 to reduce the contribution of the higher order harmonics, which are expected to have lower SNR (Eq (2)).

$$Q_{SSVEP}\left(w\right)=\sum _{k=1}^3\sum _{f_1,f_2}{\alpha }^{k-1}SpecSNR(w^{T}S,kf),$$ (2)

Intuitively, this quality metric represents the total SNR of all neural responses in the early visual cortex directly related to the rivaling frequency-tagged stimuli. For each subject, we performed a constrained maximization of this quantity, and we obtain a set of weights representing an optimized spatial filter for that subject. Each electrode weight can be represented by a unitless value constrained to the range [-1,1] to limit the search space. Electrode weights can then be iteratively changed in order to maximize the SNR of the 6.67 Hz and 8.54 Hz signals, utilizing the cost function provided by Eq. (2). These weights can be represented as a scalp topography. The resulting topographies are generally weighted toward occipital electrodes, but better account for the noise by down-weighting electrodes with high residual noise, which varied from subject to subject (See Fig. 5 for examples).

Figure 5. Optimized spatial filter for each subject.

(a) Electrode weights were unitless and constrained between [-1, 1] for the optimization procedure. For each subject, these weights were normalized with respect to the maximum weight value for visualization. The optimized filter topographies appear quite variable, but are largely concentrated on the occipital area, with additional variability to help reduce residual noise. (b) The spectral SNR for each subject using the Hjörth transform (red), spherical Laplacian (green), and the optimized filter (blue). The optimized filter consistently outperforms both Laplacian approximations. (c) The mean across subjects of each spatial filter type illustrates the spatial features that are preserved across subjects.

SSVEP Amplitude Estimation

To estimate the time-varying amplitude envelope of the two SSVEP signals, we employed a phase-specific amplitude demodulation approach. Previous work has used a more general adaptive recursive least-squares (RLS) filter [6, 17, 18]. The adaptive RLS filter uses a sliding window and a pair of sine and cosine matched filters to estimate the magnitude and phase of the SSVEP over time, equivalent to the short-time Fourier transform (STFT) evaluated at a single frequency.

$$h_f\left(n\right)=\frac{2}{L}\sum _{k=n-L}^{n}y\left(k\right)(\cos (2\pi fk∕f_s)+i\sin (2\pi fk∕f_s))$$ (3)

The magnitude and phase of the SSVEP at frequency f during a window of length L are computed as A = |h_f| and ϕ = tan⁻¹(h_f). The resulting complex signal h_f can represent an SSVEP with any phase, or even an SSVEP with time-varying phase.

In many applications, however, the phase of the SSVEP is constant and is either known from theoretical considerations or can be determined empirically. In this case, we can project the complex RLS envelope onto that phase and reject noise that is present at the SSVEP frequency but not phase-locked to the stimulus.

$$h_{f\psi }\left(n\right)=A\left(n\right)\cos (\varphi \left(n\right)-\psi )$$ (4)

We were able to utilize this projection method given that the phase of the SSVEP could be determined empirically based on the data. Projecting RLS magnitudes onto a known phase allows us to estimate time-varying SSVEP activity of any polarity. This is an important consideration, given that EEG activity is generally considered to be generated by dipolar sources, which can create both positive and negative potentials on the scalp. In contrast, using the magnitude of the RLS envelope restricts estimates to positive values. With a strictly positive envelope, the presence of additive noise means that only allowing positive values will bias the estimate when the envelope is near zero.

Results

Cardioballistic Artifact Removal

ECG recordings were unusable in 8 out of our 23 subjects. Typically, these problems arose when subjects repositioned themselves between scans and displaced the ECG electrode placed on their back. Even in subjects with clean ECG recordings, the delay between the QRS complex and the peak of the EEG artifact varied by ±20-30ms from beat to beat. Fig. 4 illustrates this problem for a representative subject.

Figure 4. Using EEG electrode for CBA detection.

(a) The electrode with the sharpest autocorrelation peak was selected for CBA detection (red box). (b) The FC4 artifact had several similar peaks, which can lead to misalignment of some of the artifacts. (c) Using electrode FC4 for CBA detection improves artifact removal compared to ECG timing (bottom middle vs. bottom left). After an additional cross-correlogram realignment step, the artifact is more consistent and the residual artifact after removal is reduced significantly (bottom right). [Only a subset of electrode autocorrelograms is shown for illustration. CBA artifacts are from electrode POz in a representative subject. Solid gray filled traces at the bottom of each graph represent the artifact variance across beats.]

The left column of Fig. 4c depicts the CBA aligned to R-peaks from clean ECG recording. The artifact is fairly consistent, but there is a large residual artifact after correction (bottom row).

To remove this residual artifact, we first identified the EEG channel with the sharpest autocorrelation peak between 500 and 750ms, which is FC4, for the example shown in Fig. 4a. Unlike the QRS complex of the ECG, the artifact in FC4 contains several peaks of similar size (Fig. 4b). The initial artifact detection was therefore imprecise, as seen in the middle column of Fig. 4c, but even with this poor alignment, the artifact was removed more completely, with a residual variance 2-3 times smaller than when using ECG.

To remove artifacts more thoroughly, they were realigned by finding the peak lag of the cross-correlogram between each artifact and a template. The template for this realignment step was the heartbeat from the previous step with the highest median correlation coefficient with all other heartbeats. This template performed better than either a) the first clean heartbeat detected or b) the average artifact after initial detection. Using these realigned events for the artifact rejection step, we produced a signal with residual variance around 5-10 times smaller than using only ECG (Fig. 4c, right column).

Optimized Spatial Filter

As described in the methods section, challenges exist to extract clean EEG signals from simultaneous recordings of EEG in a MRI scanner. The conventional use of PCA, ICA and surface Laplacian was insufficient to handle the excessive noise as observed from the EEG recordings made in the scanner.

To mitigate this problem, we interpolated the EEG recordings at electrodes onto a higher resolution surface distribution, and derived the surface Laplacian on that surface. We identified the location on that surface with the largest SSVEP signal, and then inverted the interpolation to create a set of electrode weights that approximate the spherical Laplacian at that location. Fig. 5c shows the mean of these electrode weightings across subjects, and demonstrates the increased spectral SNR compared to the Hjörth transform (green vs. red).

To further improve SNR, we used an iterative gradient-descent optimization to search through the space of possible electrode weights. We used a five-fold cross validation to determine the number of gradient-descent iterations that maximized SNR when applied to trials excluded from the training data. After approximately 10-15 iterations, performance on test data leveled off or decreased as overfitting occurred. We then re-ran the optimization on the subject's entire data set using the appropriate number of iterations.

Fig. 5a shows the optimized topography for each of the subjects in the experiment, and Fig. 5b shows the optimized spectral SNR for each subject (in blue), compared to the spherical Laplacian (green) and Hjörth (red) approximations.

As a final optimization step, we estimated SSVEP amplitudes using a phase-specific approach (see Methods). This allowed us to reject noise at the stimulus frequency and to properly estimate the sign of the signal. Fig. 6 displays representative examples of extracted SSVEP signals for a single subject following phase-specific demodulation. In general, the optimized spatial filter resulted in improved contrast between the two frequency-tagged signals of interest. Importantly, the optimized filter correctly predicted the subject's percept in block (b) whereas the Laplacian filter did not. Block (c) also showed increased signal separation using the optimized filter versus the Laplacian. Both plots also show relatively little contrast between SSVEP signals during the period where the subject reported a mixed percept (c). Fig. 7 shows the overall improvement in SSVEP SNR across all processing steps. In addition, Fig. 8 shows representative SSVEP source localization and fMRI activation results for one subject. We found consistent co-localization of activity in primary visual areas and extrastriate areas during both rivalry and replay blocks across subjects.

Figure 6. SSVEP Reconstruction using Phase-Specific Demodulation.

SSVEP envelopes were extracted following the use of both an optimized spatial filter (top) and a standard Laplacian spatial filter (bottom). Representative results for a single subject are displayed for a 42 second period of continuous rivalry. See text for details.

Figure 7. SSVEP signal improvement at each processing stage.

Red bars indicate power at the stimulus frequency. The timecourses show the amplitude of the SSVEP signal around locations where one frequency peaks. The light gray trace is the average of these peaks. The dark gray trace is the average of the other frequency. During binocular rivalry, the other frequency is generally suppressed at peaks of the first frequency. The greater depth of suppression seen at right shows that the optimized filtering greatly improved our ability to see this neural correlate of rivalry. Topographies are displayed using normalized weight units.

Figure 8. Individual subject co-localization of SSVEP sources and fMRI activation during binocular rivalry.

Both SSVEP source maps (top row) and fMRI activation maps (bottom row) show significant changes in primary visual cortex and extrastriate cortex during rivalry. Similar activation patterns were observed for the replay conditions across subjects.

Discussion

We have developed several signal processing techniques to improve measurement of continuous, time-varying SSVEPs during simultaneous EEG-fMRI, and validated these methods on our sample of 23 subjects in a binocular rivalry paradigm. To our knowledge, this is the first study to establish the feasibility of reconstructing rivalry related SSVEP signals from EEG data collected during concurrent fMRI.

We made three specific improvements to standard processing procedures. First, we were able to significantly improve the SNR of the rivalry-related SSVEP signals by optimizing the removal of the CBA in the EEG induced during fMRI acquisition. Given the subject-specific variations in the CBA and non-homogenous propagation of such artifacts across the EEG amplifiers, this was a crucial step for later extraction of the SSVEP signals. Second, we developed an optimized spatial filter for maximizing the subject-specific SNR of the target SSVEP frequencies which showed enhanced performance as compared with both the Hjörth and spherical Laplacian spatial filters. Finally, we developed a phase-specific demodulation approach to recover negative envelopes in the SSVEP signals. Furthermore, general linear model (GLM) analysis revealed strong BOLD activation in the primary visual cortex during ongoing blocks of binocular rivalry across subjects. This fMRI activation was generally co-localized with SSVEP sources in visual cortex, as determined using source analysis techniques of the optimized SSVEP topographies. This finding is consistent with other fMRI studies of the visual system and can be further supported by our group's previous EEG-fMRI research of visually evoked potentials (VEPs), further validating the efficacy of our proposed multimodal EEG-fMRI methods.

Using these novel methods we successfully reconstructed continuous SSVEP amplitudes which demonstrated interocular suppression, under much more challenging conditions but with results similar to previous studies of binocular rivalry using EEG only [6]. The large variations in subject experiences of binocular rivalry point to the importance of developing subject-specific methods for SSVEP signal optimization when evaluating EEG correlates of rivalry, both in and out of the scanner. Our work addresses a number of technical challenges associated with collecting SSVEP data in the scanner and establishes the feasibility of using such techniques for studying binocular rivalry.

We aimed to develop novel methods for improving the data quality of EEG collected in the MR environment. There are two major types of artifact that contaminate EEG recorded simultaneously with fMRI. The first is the gradient artifact (GA) induced by the MR spatial encoding gradients. These are several orders of magnitude larger than the EEG signal, but they are very consistent and somewhat confined to the high frequency range [30, 31]. The second is known as the cardioballistic artifact (CBA) or ballistocardiogram (BCG), and it is generated as a consequence of blood pulsing through the head and scalp after each heartbeat [32, 33]. Ballistic head rotation due to cerebral blood volume changes and scalp expansion due to superficial blood flow create small movements in the wires and electrodes of the EEG system. When these displacements occur within the large static magnetic field, current is induced. In addition, the time-varying flow of conductive blood through the magnetic field creates Hall Effect voltages. Modeling and experimental studies suggest that reducing head rotation using head restraints can reduce the size of the artifact by nearly 60%, but that this only reduces the absolute beat-to-beat variability of the artifact by around 20% [34].

The variability of the CBA (correlation between 0.8 and 0.9), along with its spectral proximity to EEG signals of interest, makes it more difficult to remove than the large but consistent gradient artifact (correlation > 0.999). The gradient artifact can be largely mitigated using average artifact subtraction (AAS), where the artifact template is derived from a sliding window around 60 seconds. The most successful CBA rejection algorithms take a statistical approach, using principal components analysis (PCA) to identify a basis set whose weighted sum can then be fitted to each artifact individually and then subtracted [27, 28]. Independent component analysis has also shown promise for addressing CBA removal [35, 36].

Our combinatorial procedure for removing the CBA from the occipital EEG data was highly effective as indicated by the low residual variance we observed following CBA removal using our methods versus established methods. Importantly, we utilized a multi-step CBA removal procedure which first optimized the alignment of the occipital CBA with the EEG channel showing the peak autocorrelation in the 500-750 ms range in order to account for subject-specific variations in the latency between heartbeats and the CBA. This step also effectively dealt with cases where the ECG signal was excessively noisy and its alignment with the occipital CBA imprecise. Next, a cross-correlogram for re-alignment of the occipital CBAs with a template heartbeat was performed prior to ICA for identification of signal components most strongly correlated with the occipital CBA. Since the final artifact removal step is partially based on the exclusion of independent components with mutual information in relation to the specified electrode, we excluded occipital electrodes when selecting the CBA channel to avoid rejecting components potentially related to the SSVEP signals of interest. These procedures substantially improved removal of the CBA and its residual noise compared to methods which utilized only the ECG for detecting/aligning the CBA prior to ICA.

In addition to improving removal of the CBA, we utilized a novel optimization procedure for spatial filtering of the EEG data in order to maximize the occipital SNR of the target SSVEP frequencies for each individual subject. The standard Hjörth transform is a simple Laplacian filter which utilizes a center-surround approach to enhance SNR at a particular location by rejecting far field contributions, and has been utilized for extracting SSVEP envelopes in previous EEG-only studies [29]. However, it is limited because it must be centered on an actual EEG electrode. Thus, such a spatial filter results in low SNR if the peak signal in the topography lies somewhere between two electrodes. We found that this was a significant limitation of the Hjörth transform for extracting SSVEP data collected in the scanner due to the presence of noise throughout the frequency spectrum and generally low SNR of the target frequencies.

To address these limitations, we interpolated the EEG data to a higher resolution spatial domain and used a spherical Laplacian to determine the location of the maximal SSVEP signal for each subject. These results were then inverted to create a set of weights for the original EEG electrodes to produce a pre-optimized, subject-specific SSVEP topography. The use of the subject-specific spherical Laplacian did improve SSVEP SNR compared to the Hjörth transform, but did not result in sufficiently high SNR for a number of subjects. Thus, we further optimized each subject's spherical Laplacian topography by utilizing a gradient-descent optimization process for maximizing the SNR of the target SSVEP frequencies. This final step substantially improved the SNR of the target SSVEP frequencies and allowed for robust SSVEP extraction across all subjects. It is important to note that during all experiments, both hemifields contained both frequencies (images) throughout the course of each rivalry or replay block. Thus, we did not expect any difference between the topographies of the two SSVEP frequencies. This was verified based on our observation of the SSVEP topographies for the two target frequencies, which were essentially identical in all cases. For this reason, we ultimately decided to report topographies representing the combined activation of both SSVEP signals, as displayed in Fig. 5.

Finally, we utilized a phase-specific demodulation approach to recover the negative envelopes in the extracted SSVEP timecourses, further enhancing their signal quality. This phase-specific fitting accounted for noise in the target SSVEP frequencies that was not phase-locked to the stimulus. Our optimized spatial filtering procedure combined with the phase-specific demodulation significantly increased not only the strength of the SSVEP envelopes, but also increased the negative correlation between the signals driven by the two eyes, suggesting a true improvement in signal quality beyond simply enhancing the quantity we explicitly optimized. Furthermore, the quality of the rivalry-related countermodulating SSVEPs was similar to that of EEG-only data our group has previously collected and was well correlated with our behavioral data.

There are several limitations of this work that must be addressed. Importantly, the data analysis procedure we describe here is specifically designed to isolate the single largest generator of the SSVEP signal. The SSVEP, while not linear, can be thought of like a convolution of a VEP and a spike train. Other sources of the SSVEP with different latencies will then have different phases which our methods currently ignore. However, this approach in combination with the Laplacian spatial filter was well suited for our purpose of isolating specific SSVEP sources directly beneath the target scalp location (visual cortex). Future work should investigate the potential of additional computational methods, such as canonical correlation analysis (CCA), for reconstructing SSVEPs collected during simultaneous EEG-fMRI. Several studies have successfully utilized CCA and related methods for SSVEP classification in SSVEP-based brain computer interfaces (BCIs) [37–39]. Although CCA appears promising for real time differentiation of SSVEP signals using EEG alone, it has yet to be evaluated in the high-noise setting of the MR scanner. Regardless, a multifaceted approach combining CCA with our novel methods for SSVEP SNR optimization could prove fruitful.

Additionally, future work should investigate both EEG-informed fMRI and fMRI-constrained EEG analysis methods in order to truly integrate the two data sets and improve understanding of relationships between the electrophysiological and hemodynamic mechanisms of binocular rivalry. Last, additional sources of EEG noise, such as inhomogeneities in the scanner's magnetic field, may have affected overall SNR of the extracted SSVEP signals. Future work should address these additional sources of noise to further improve SSVEP extraction in the scanner.

Conclusion

The MR environment presents many challenges for accurate EEG recording. To our knowledge, this study represents the first successful reconstruction of continuous time-varying SSVEP amplitude during simultaneous EEG-fMRI. Our EEG-based heartbeat detection and realignment approach improved cardioballistic artifact rejection, and may prove beneficial for the broader EEG-fMRI community. Our optimized spatial filter successfully rejected residual MR artifacts and produced an SSVEP amplitude estimation with higher SNR than using a single occipital electrode or spatial Laplacian transform. Combining these techniques, we successfully reconstructed continuous SSVEP amplitude and demonstrated the presence of interocular suppression during binocular rivalry from EEG data collected in the scanner. This work paves the way for more comprehensive multimodal imaging studies of binocular rivalry in the future and will improve the quality of EEG data for simultaneous EEG-fMRI studies in general.

• - EEG-fMRI of binocular rivalry

• - Optimized extraction of frequency-tagged SSVEPs

Figure 2. Binocular rivalry stimulus presentation inside MRI scanner.

Prism lenses were used to bring stimuli from two sides of the screen to appear in the center of the subject's visual field. A divider prevented crosstalk, and the inner wall of the bore was covered with a black felt panel to reduce peripheral reflection of the stimuli.