Multimodal EEG-fMRI: advancing insight into large-scale human brain dynamics

Abstract

Advances in the acquisition and analysis of functional magnetic resonance imaging (fMRI) data are revealing increasingly rich spatiotemporal structure across the human brain. Nonetheless, uncertainty surrounding the origins of fMRI hemodynamic signals, and in the link between large-scale fMRI patterns and ongoing functional states, presently limits the neurobiological conclusions one can draw from fMRI alone. Electroencephalography (EEG) provides complementary information about neural electrical activity and state change, and simultaneously acquiring EEG together with fMRI presents unique opportunities for studying large-scale brain activity and gaining more information from fMRI itself. Here, we discuss recent progress in the use of concurrent EEG-fMRI to enrich the investigation of neural and physiological states and clarify the origins of fMRI hemodynamic signals. Throughout, we outline perspectives on future directions and open challenges.

Introduction

Functional magnetic resonance imaging (fMRI) has provided an important, non-invasive window into the dynamics of human brain activity and its alteration with neurological and neuropsychiatric disorders. Recent advances in imaging technology have enabled measurement of blood oxygen level dependent (BOLD) signal patterns at sub-second temporal resolution and spatial scales down to cortical layers and columns[1]. Such signals are used to probe regional and network-level function associated with specific cognitive processes during tasks, to infer intrinsic functional architecture, and to examine how brain activity is modulated across a range of conscious and cognitive states[2].

Although the neuroimaging field is assembling a richer and more detailed picture of fMRI spatiotemporal organization, there are presently a number of challenges in interpreting such patterns and the hemodynamic signal fluctuations on which they are based. The BOLD fMRI signal is an indirect and sluggish measure of neural activity, and arises from the latter through complex neurovascular coupling mechanisms[3]. The interpretation of correlated fMRI signals is complicated by systemic physiological responses (such as respiration and blood-pressure changes) that exert spatially structured, dynamic influences on blood oxygenation across the brain, as well as by artifacts such as those induced by head motion[4]. Moreover, the time course measured in a given fMRI voxel can reflect the overlapping activity of its constituent neural populations in multiple brain circuits, including those involved in regulating brain states such as arousal[5], whose respective contributions have been challenging to parse with fMRI alone.

Integrating fMRI with electrophysiological signals offers one promising avenue for bridging the gap between hemodynamic and neural fluctuations, and for clarifying the origins of fMRI signals and their large-scale patterns (Fig. 1). Scalp EEG provides a non-invasive view into brain electrical activity, and is complementary to fMRI: EEG captures fast electrical activity, with the ability to resolve temporal markers of natural or pathological state changes (e.g., sleep and seizure activity), but with coarser spatial resolution and different physiological origins compared to fMRI[6]. Together with continued improvements in EEG-fMRI hardware and denoising algorithms, efforts toward leveraging the complementary strengths of these modalities have presented new opportunities for tracking brain-wide hemodynamic correlates of EEG-defined neural state changes, for fusing EEG and fMRI information through joint analysis, and for pinpointing the neural basis of fMRI hemodynamic response properties. Indeed, while much literature has focused on the use of fMRI to aid localization of EEG signals, the converse (EEG-informed fMRI analysis[7]) also offers a valuable avenue for boosting the total information that one can obtain about human brain function.

Figure 1.

Linking measures of dynamic brain function across modalities. Prior work has sought to establish relationships between various aspects of EEG (left) and fMRI (right) measurements. Unifying models of such features within and across modalities, and gaining a deeper understanding of the neuro-electric underpinnings of BOLD correlations, are important areas of ongoing development.

The following sections present several areas of recent development in the use of simultaneous EEG-fMRI to increase understanding of macroscale human brain dynamics. We first describe its use in examining dynamic sleep/wake states. We then discuss ways in which concurrent EEG-fMRI can enrich the study of large-scale brain network connectivity and intrinsic activity beyond either modality alone. Finally, we discuss how integrating these modalities enables probing mechanisms underlying BOLD hemodynamics, which can strengthen the utility of fMRI itself.

Simultaneous EEG-fMRI of dynamic state change

The brain constantly moves through internal states that shape behavioral responses and the processing of incoming information[8]. Neuromodulatory state changes are therefore important to incorporate into models of brain function, yet are difficult to study using fMRI alone. Fluctuation in natural states can be temporally decoupled from experimentally controlled conditions, and many established markers of state-change have electrical signatures that are faster than BOLD hemodynamics but are readily measured using scalp EEG. Therefore, one major framework in fusing EEG-fMRI has involved the use of EEG – with its high temporal resolution - to track the occurrence or continuous modulation of electrical activity over time, and fMRI - with its superior spatial resolution - to investigate the corresponding whole-brain correlates, particularly in deep subcortical structures that regulate brain states but which are less accessible to scalp EEG[9].

For example, a number of studies have mapped the voxelwise fMRI correlates of specific EEG features that index variations in arousal and cortical excitability, such as the amplitude[10] and phase synchrony[11] of the alpha rhythm during quiet wakefulness. Alternatively, one may examine whole-brain fMRI patterns surrounding temporally localized electrical events such as k-complexes and sleep spindles (reviewed in[12]). EEG may also be invoked to detect temporal epochs of different sleep stages, and one can then use concurrently acquired fMRI to query how hemodynamic signals and network connectivity patterns are modulated during different sleep stages and in the transitions between sleep and wakefulness[13-15].

In this framework, an exciting recent direction is to harness high-resolution fMRI to map the covariation of cortical layers[16] and small subcortical nuclei [17] with electrophysiological markers of arousal, either in spontaneous activity or with experimental tasks that allow for quantifying interactions between state changes, information flow, and cortical-subcortical interactions (Fig. 2a; see also section Laminar EEG-fMRI to probe effective connectivity below). Further, non-BOLD aspects of brain function that can be detected using fMRI, such as CSF flow[18], can also be studied in conjunction with EEG-determined arousal states, representing another promising area of future work. Innovations in fMRI data analysis techniques, such transient co-activation patterns[17] and methods based on hidden Markov models[13], may also prove fruitful for uncovering novel whole-brain features linked with arousal states.

Figure 2.

Examples of emerging methods for integrating EEG oscillatory information with fMRI to investigate state-dependent brain activity. (a) Laminar fMRI mapping of trial-by-trial variation EEG power in a visual attention paradigm, across frequencies and cortical depth. Adapted from [16]. (b) Illustration of a pattern-recognition approach for inferring a continuous measure of alertness from fMRI data, based on method described in [23].

Tracking brain states in fMRI signals

As fMRI signals are indeed modulated with arousal states, the interpretation of fMRI signals and network connectivity can be impacted by unmodeled changes in arousal. For example, fluctuations in the whole-brain average ("global") fMRI signal are known to increase from alert wakefulness to drowsiness and light sleep, leading to more widespread positive correlations between fMRI regions of interest[5]. Dynamic variations in connectivity between major intrinsic networks were observed to track EEG spectral measures of alertness[19,20], and wakefulness fluctuations are regarded as one of the main sources of time-varying BOLD dynamic connectivity changes[21]. These findings speak to the importance of recording EEG or other indicators of arousal during fMRI, particularly when systematic differences in vigilance are expected between study populations. They also highlight the importance of recognizing and modeling signatures of state change in fMRI, which may be learned with the help of concurrent EEG.

Despite the utility of EEG for monitoring brain states, practical difficulties involving the cost, setup-time, and subject tolerance of MR-compatible EEG can preclude its use in a number of studies. However, growing evidence suggests that it is possible to identify robust signatures of state change in fMRI signals across the brain, leading to the possibility of decoding arousal states from fMRI alone. By combining EEG-fMRI with machine learning methods, recent studies have demonstrated the ability to classify between EEG-defined sleep stages[15] or between high and low arousal states[22] based on fMRI functional connectivity, and to produce a continuous temporal index of alertness based on frame-by-frame fMRI activity patterns ([23,24]; Fig. 2b). In addition to providing practical tools for tracking brain states and guiding the interpretation of fMRI data, this research direction can also advance understanding of whole-brain signatures of brain states that accompany naturally shifting arousal states.

Linking electrophysiological and fMRI activity patterns

Even in the absence of overt behavior or sensory stimulation, the BOLD fMRI signal exhibits a remarkable degree of spatiotemporal structure, aligning closely with structural networks and cortical gradients. The field has drawn heavily upon these patterns to infer principles of functional organization and their disruption across disorders. However, open questions have remained about the relation of these hemodynamic phenomena to neural activity. For example, since hemodynamic changes driving fMRI signal variations can arise from both neuronal and systemic physiological changes regulated by the autonomic nervous system[4], disentangling these respective contributions has been a long-standing challenge for fMRI. Concurrent EEG may be of significant value in clarifying the potential sources of fMRI signal variations and interpreting fMRI connectivity patterns, as it captures a measure of neural activity without hemodynamic confounds, and reveals neural oscillations that form a key mechanism for distributed brain communication.

Much prior work has sought to relate BOLD signal to the time-varying power in specific electrophysiological frequency bands (see[25,26] for recent reviews). Invasive electrode recordings in animals and humans have shown that the time-varying power of gamma-band local field potential relates to the local BOLD signal[27] and its distributed connectivity patterns[28], supporting the neural origins of large-scale fMRI 'networks'. The power of lower-frequency EEG bands (such as theta, alpha, and beta) has, however, exhibited more variable coupling with fMRI networks[29,30], and may be state-dependent (e.g.,[31]). In addition, a given resting-state network or voxel does not appear to have a one-to-one relationship with a canonical frequency band, but rather, a signature that spans multiple frequency bands[32,33]. Recent work also indicates that correlates of fMRI networks may be obtained using direct recording of infra-slow EEG time courses with fMRI[34].

Another popular depiction of the repertoire of neuronal states derives from the topology of scalp potential fields instead of specific brain rhythms. Building off observations that EEG scalp potential topologies are commonly dominated by a few prototypic patterns (“microstates”) that each remain stable for a brief period of time (60-120 ms) before transitioning to another, it has been postulated that each global brain state associates with one specific microstate, and that the microstate sequence reflects ongoing changes in consciousness and cognition. Microstates have also been viewed as important neurophysiological signatures of fMRI resting-state networks, since microstate dynamics are scale-free, comprising ultra-slow fluctuations[35]; and regressing the microstate sequence filtered at the hemodynamic range against resting-state fMRI signals yields spatial patterns akin to typical functional networks (reviewed in[36]).

Moving forward, the unification of multiple derived features within and across modalities (Fig. 1), and a deeper understanding of the neuro-electric underpinnings of structured BOLD correlations - and the sources governing variability in EEG-fMRI coupling - are important areas of ongoing development. Computational modeling and simulation may provide one potential avenue toward resolving links and discrepancies between electrophysiological and BOLD measurements[37]. Alternative data-driven approaches, such as characterization of temporal sequences of EEG broadband power[38], may also reveal new aspects of how network dynamics at fast EEG timescales are linked with slower scales of fMRI hemodynamics. In addition, beyond the aforementioned EEG features, there are additional EEG phenomena whose link with fMRI has been less heavily studied to date (e.g., phase-resetting responses and phase-amplitude coupling[39], and 1/f parameters[40]). The interconnections between these parameters and other EEG features, and with the fMRI signal, are worth further investigation.

A further consideration involves the maximum degree of alignment between EEG and fMRI signals that can be attained. For instance, EEG reflects only a subset of all neural activity, as the observed signals are dominated by synchronized post-synaptic potentials from assemblies of pyramidal neurons arranged in open field configurations in gyral crowns. This factor contributes to neurophysiological limits on the temporal variance that an EEG signal could account for in an fMRI signal – and further, may vary based on cortical folding and depth. An understanding of the theoretical upper and lower bounds for the EEG-fMRI relationship will be another important line of future work.

Finally, while relationships between fMRI and EEG are most often determined using EEG measures derived in sensor space, studies have more recently begun to examine and discover correspondences between simultaneously measured fMRI and source-localized whole-brain EEG connectivity[41-43]. Given the mounting interest in whole-brain connectivity approaches for the study of individual differences and disease prediction[44], future work may examine whether source-localized EEG connectivity can add new dimensions of neurophysiological information to such questions[45].

Laminar EEG-fMRI to probe effective connectivity

Another young, yet promising area is to link scalp EEG rhythms with laminar- (or cortical-depth) specific fMRI signals, which holds intriguing potential for unveiling effective connectivity and probing the roles of different neuronal rhythms in mediating directed information flow. Evidence from invasive animal literature has suggested that feedforward and feedback communications in the primary cortex arise from distinct layers of cortical activity and are subserved by different neuronal rhythms (reviewed in[46]). Although technical challenges pertaining to data acquisition and analytical strategies remain to be addressed in order to fully differentiate functional information across layers, advances in high-resolution fMRI have made it viable to begin investigating similar questions noninvasively in humans. In one encouraging example, a recent study was able to replicate findings from previous animal experiments that different brain rhythms were marked by differential profiles of laminar-specific fMRI signals, likely reflecting their distinctive roles in mediating feedforward and feedback information processes[16]. An exciting future avenue will be to continue validating the efficacy of existing EEG-laminar fMRI frameworks using “ground-truth” insights gained from animal model, with the ultimate goal of leveraging this multi-modal framework to advance our understanding of various brain rhythms and effective connectivity contextualized under consciousness and higher-level cognition[46-48].

Neurophysiological underpinnings of BOLD hemodynamic response

Stimulus-driven hemodynamic responses can exhibit several interesting transients, such as a negative initial dip preceding the primary positive signal change and a prolonged negative undershoot before recovery to baseline. Despite over two decades of investigations, no consensus has been reached regarding the precise vascular or neuronal mechanisms giving rise to each of these transient features. Scalp EEG measures can complement existing efforts of biophysical modeling in understanding the neuronal modulation of specific phases of hemodynamic changes. For instance, the potential neural relevance of the post-stimulus undershoot (PSU) has recently been evidenced by a sequence of studies that demonstrated the link between the magnitudes of PSUs and the power of specific EEG rhythms, starting by showing that stronger post-stimulus mu-band activity predicted more pronounced PSUs in the somatosensory system[49] then generalizing to other EEG oscillations and to the visual system[50,51].

Such a multi-modal framework may be readily extended to elucidate neuronal contributions to other key aspects of hemodynamics, including newly uncovered features associated with data collected at high spatiotemporal resolution. In this direction, one potential application involves determining the neuronal correlates of emerging fast fMRI phenomena; recent work has shown that fMRI can track functionally-relevant oscillations up to the delta wave frequency[52], challenging the conventional notion that hemodynamic changes are inherently slow. Importantly, this line of work points to the possibility of imaging delta oscillations in deep brain structures that are not accessible to scalp EEG. Open questions remain as to whether these fMRI-derived signals arise from identical neuronal sources as their EEG counterparts at the same frequencies, or are still driven by slower-scale neuronal dynamics but manifest at higher frequencies due to nonlinearities of the hemodynamic system. Direct comparisons with EEG recordings may hence provide a means toward clarifying these issues.

Challenges and open questions

This work has highlighted ways in which concurrent EEG-fMRI may provide valuable information for identifying state-related fMRI fluctuations and shed new light on associations between electrophysiological and hemodynamic processes. Nonetheless, several challenges remain. One stems from the known, close interactions between neural activity and systemic physiology, both of which impact BOLD signal. As such, covariation between EEG and systemic physiological changes (respiration, heart rate) complicates the interpretation of associated fMRI responses[55]. A recent illustration comes from a study of the k-complex, an electrical event in NREM sleep that is associated with both electrophysiological and autonomic nervous system activity, including changes in respiration and vascular tone[56]. Separating the associated fMRI response into neurogenic versus purely vascular-mediated components is not straightforward, and is further complicated by observations that low-frequency physiological effects can manifest in similar spatial patterns as fMRI networks[57,58]. To address these challenges, one exciting direction, enabled by recent technological advances[59], involves integrating concurrent PET with EEG-fMRI. PET, with its sensitivity to specific neurotransmitters and metabolic processes, can provide a new dimension on neural activity and help to dissociate neural from systemic effects[60].

As discussed above, high-resolution fMRI has enabled a fine-grained characterization of the brain’s functional architecture, which – when integrated with EEG – provides an exciting avenue to probe the relationship between various brain rhythms and functional activity in deep brain structures and across cortical layers and columns (Fig. 3). However, to compensate for reduced functional sensitivity at smaller voxel sizes, these high-resolution fMRI investigations are often carried out at magnetic fields higher than 3T, in which case the safety considerations of tissue heating, along with EEG/MRI signal degradation, are exacerbated[61]. To overcome these challenges, careful simulations and heating tests, optimized set-ups through adaptive cabling[62] or invoking caps of high-resistance materials[63] are warranted.

Figure 3.

Spatial and temporal scales of EEG and fMRI enabled by recent technological and analytic advances. Electromagnetic source imaging, using high-density EEG recordings and validated with intracranial EEG, pushes the spatial limits of EEG to enable mapping brain networks at high spatiotemporal resolution (upper left; adapted from[53]). Technological advances in fMRI pulse sequences and high-field imaging enable long-range connectome mapping at the resolution of cortical layers (upper right; adapted from[47]) and single vessels (middle right; adapted from[54]). (Lower right) fMRI and high temporal resolution may resolve neural oscillations up to delta-wave frequency (adapted from[52]).

Box: Examples of open questions in multimodal EEG-fMRI:

• What aspects of electrophysiological signals underpin both local temporal dynamics and long-range functional network patterns exhibited by hemodynamic signals?

• How well can time-varying changes in brain states (determined using EEG recordings) be inferred from fMRI signals?

• How does the coupling between different EEG rhythms and fMRI signals depend on cortical layer / depth?

• What are the EEG signatures of inter-trial variability of stimulus-driven hemodynamic responses (magnitude and timing)?