Multi-scale neural decoding and analysis

Abstract

Objective.

Complex spatiotemporal neural activity encodes rich information related to behavior and cognition. Conventional research has focused on neural activity acquired using one of many different measurement modalities, each of which provides useful but incomplete assessment of the neural code. Multi-modal techniques can overcome tradeoffs in the spatial and temporal resolution of a single modality to reveal deeper and more comprehensive understanding of system-level neural mechanisms. Uncovering multi-scale dynamics is essential for a mechanistic understanding of brain function and for harnessing neuroscientific insights to develop more effective clinical treatment.

Approach.

We discuss conventional methodologies used for characterizing neural activity at different scales and review contemporary examples of how these approaches have been combined. Then we present our case for integrating activity across multiple scales to benefit from the combined strengths of each approach and elucidate a more holistic understanding of neural processes.

Main results.

We examine various combinations of neural activity at different scales and analytical techniques that can be used to integrate or illuminate information across scales, as well the technologies that enable such exciting studies. We conclude with challenges facing future multi-scale studies, and a discussion of the power and potential of these approaches.

Significance.

This roadmap will lead the readers toward a broad range of multi-scale neural decoding techniques and their benefits over single-modality analyses. This Review article highlights the importance of multi-scale analyses for systematically interrogating complex spatiotemporal mechanisms underlying cognition and behavior.

1. Introduction

Neural recording methodologies enable us to probe neural activity and investigate how the brain implements cognitive processes and generates behavior [1, 2]. There are many technologies—electrical, optical, and chemical—that allow us to observe and perturb neural activity at different temporal and spatial scales. At any given scale, neural activity encodes rich information related to behavior and cognition [3–18]. Work to date has conventionally focused on neural activity acquired using a single modality and, thus, at a single scale. Decoding and modeling neural dynamics using measurements from one modality limits investigations to dynamics within the same level. However, complex spatiotemporal activity supports behavior and cognition, and cross-scale dynamics can reveal a deeper and more comprehensive understanding of system-level neural mechanisms [19, 20]. Since the brain exhibits functional structure across a range of disparate scales, from single neurons (micro-scale) to networks of brain areas (macro-scale), multi-scale and multi-modal techniques can overcome tradeoffs in the spatial and temporal resolution of a single modality and scale. Uncovering multi-scale dynamics is essential for illuminating the mechanistic understanding of brain function and to harness scientific insight for neuroscientifically grounded clinical treatment. A paradigm shift, enabled by state-of-the-art neurotechnologies, is currently underway to analyze neural activity across multiple scales.

Often, we are interested in activity from individual neurons and use electrodes positioned within neural tissue to record single-unit (SUA) or multiunit activity (MUA). This activity reflects action potentials (‘spikes’) from one or a small number of neurons in the brain and can be used, for example, to determine how neuronal firing rates co-vary with behavioral variables. Additionally, we commonly examine oscillatory dynamics of neural activity, which can be obtained from local field potential (LFP), electrocorticography (ECoG), and electroencephalography (EEG) signals. While each of these signals reflect summed electrical activity which manifest as neural oscillations in the brain [21], they are recorded at different scales using different recording modalities. In addition to electrophysiological recording techniques, functional brain imaging techniques such as functional magnetic resonance imaging (fMRI) and functional near-infrared spectroscopy (fNIRS) offer a different perspective on brain function [22]. Instead of investigating electrical signals, they capture the hemodynamic consequences of the underlying neural activity. Positron emission tomography (PET) is another functional imaging technique that uses radiotracers to measure metabolic changes, as well as blood flow and neurochemical activity. Although these markers are indirect measurements of brain activity, fMRI, fNIRS, and PET are valuable techniques due to their spatial coverage of the whole brain, non-invasiveness and clinical availability [23]. These techniques all have their own strengths and challenges, and the combination of these modalities has the potential to further clarify the neural mechanisms underlying cognitive processes and behavior.

Many cognitive processes have been investigated across multiple modalities independently, including movements [24–29] and decision-making [7, 9, 10, 12, 18], reward [14, 16, 17], memory [6, 8], and arousal [3, 4], and each modality provides a different perspective on the underlying neural activity. One significant outcome from this large body of impressive scientific work is that the same cognitive process can be observed at different spatial scales. This supports the notion that the heterogeneity of different techniques can provide researchers with different perspectives that, when combined to capture the interplay between scales, can provide a more comprehensive picture of neural function. Thus, it is increasingly imperative to conduct multi-scale analyses.

In addition to creating a richer understanding of neural processes, applying multiple modalities in the same experimental setting may overcome disadvantages from each individual modality, such as the compromise between temporal and spatial resolutions [30]. For example, electrophysiological techniques have high temporal resolution and spatial specificity increases with invasiveness, but typically comes at the cost of decreasing spatial coverage. On the other hand, functional brain imaging techniques have more complete spatial coverage, but suffer from temporal delay and lower temporal resolution due to the data acquisition process and sluggishness of the hemodynamic signal being measured [31]. The combination of these electrophysiological and functional imaging techniques therefore offers both high temporal and high spatial resolutions. In addition to acquiring high-quality complementary data, multi-scale analyses help us quantify the coupling between scales and across data types (see section 3.3.2). These merits may be the key to unlocking transformative progress in behavioral and cognitive neuroscience [32].

Combining multiple modalities in a single experimental inquiry has allowed researchers to achieve better performance in brain–machine interfaces (BMIs) and to gain new mechanistic insights. For example, mathematical models combining spiking activity with LFPs improved neural prosthetic prediction accuracy [25, 33–36]. Synergistic signals from multiple neural scales also minimized electrical and computational power requirements and increased the longevity of a BMI devices [34]. Furthermore, more recent studies demonstrated that measuring multiscale activity enhanced decoding performance and provided information about connectivity or causality between different neural modalities in brain networks [35–38]. Finally, other studies incorporated information from multiple non-invasive methods to form a space- and time-resolved views of object recognition in human’s vision [39–42] and to predict fMRI blood oxygen level-dependent (BOLD) activity in the amygdala from scalp-level EEG signals [43]. As highlighted by these select examples, multi-scale analyses have already significantly improved neuroscientific understanding and neural engineering applications, and merit more in-depth studies.

In this paper, we describe typical methodologies used for characterizing neural activity at different temporal and spatial scales. For each method, we describe the state-of-the-art analysis that can be applied to this data type. We then present our case for integrating activity across multiple scales to benefit from the combined strengths of each approach and elucidate a more holistic understanding of neural processes as well as more robust strategies for decoding information from the brain. We examine various combinations of neural activity at different scales and analytical techniques that can be used to integrate or illuminate information across scales, as well the technologies that enable such studies. We support our case with examples of exciting works that have leveraged multi-scale analysis to push forward both scientific and clinical frontiers. Finally, we conclude with our perspective on the challenges and potential for future multi-scale studies, and a discussion of the power of these approaches.

2. Scales of neural interrogation and decoding

2.1. Spikes

Action potentials of individual neurons induce stereotypical extracellular ionic fluctuations, which can be measured as consistent waveforms (‘spikes’), detectable with microelectrodes or microelectrode arrays, for high temporal resolution decoding on a millisecond timescale (figure 1). Due to the sharp decay of such signals with distance, an electrode tip must be at most 20 µm to 100 µm from a neuron to reliably resolve its activity [44]. Therefore, a large number (10 s to 100 s or more) of electrodes need to be implanted in parallel or integrated into a single probe in order to extract enough information for meaningful spike-based decoding. This idea defines the next frontier of high-throughput recordings of spiking activity and spike-based neural decoding of behavioral variables or stimuli [45]. Decoders using a Kalman filter (KF) and related variations, such as the recalibrated feedback intention–trained Kalman filter (also known as ReFIT-KF), are most commonly used for spike-based decoding [24, 46]. Other novel closed-loop filters and techniques are also being developed, such as the closed-loop decoder adaptation (CLDA) [47, 48] and the closed-loop point process filter [49].

Figure 1.

Description and comparison of different modalities. The diagram on the top depicts the general spatial scales of the signal acquisition techniques, ranging from cm to μm levels. The boxes in the middle demonstrate the raw data formats of each class of modality. The table on the bottom shows the strengths, weaknesses, and the primary targets. Spatial and temporal resolutions of each modality are also included.

Recent advances in high density, high channel count electrophysiology technology have the potential to greatly accelerate such processes [50]. However, large-scale spike-based decoding is still in its infancy, likely owing to the only recent availability of relevant recording technology, such as Neuropixels probes, and associated publicly available datasets [51, 52]. Recent work using this technology found that a deep neural network can decode natural image stimuli from hundreds of neurons recorded in the mice visual cortex [53, 54]. In another recent study [55], behavioral video data of a head fixed mouse were decoded from a Neuropixels dataset with 2688 neurons. The researchers effectively reduced the high dimensional input video with convolutional autoencoders and demonstrated that Bayesian models and dense neural networks both outperformed their linear counterparts for neural decoding. Large-scale recording with Neuropixels probes also facilitates the discovery of distributed cognitive and behavioral processes including choice and movements from signals across the mouse brain [56, 57]. Neuropixels recordings also revolutionized traditionally used spike sorting algorithms [58]. Thus, these modern techniques have already advanced the field.

In primate research, there have also been efforts to apply high channel count electrodes or arrays to decode behaviors from spiking activity [24, 26, 58, 59]. Berger and colleagues [60] reported the first use of 192 channel wireless recording for freely moving non-human primates. Their support vector machine (SVM) model was adequate for decoding multiple walk-and-reach targets. Clinically, there are also promising outcomes that successfully implemented high channel count BMIs [61–66]. Specifically, Hochberg and colleagues [61] demonstrated the first clinical BMI application where a 96-channel microelectrode array was used to detect neural ensemble activity in primary motor cortex. Most recently, using 192 channels and threshold crossing spike rates as input to a recurrent neural network decoder, Willett and colleagues [67] decoded imagined handwritten characters in humans with over 99% accuracy at a decoding speed comparable to the typing speed of age-matched controls, significantly improving the efficiency of brain–machine interface based communication.

Despite the promising outlook of decoding with high channel count spiking activities, the simultaneously increased power consumption, communication bandwidth, and computational delay for collecting, transmitting, and analyzing large volume spike data, have greatly prevented the field from advancing into real-time investigations [68] and the ultimate clinically translatable BMI applications [69] where low power, wireless, and low latency decoding systems are desired. In addition, the signal quality of spikes is prone to deterioration over time [70–73]. Spike sorting, the process of using spike waveform features to cluster detected spikes into putative individual neuron signals, has so far been identified as one of the most time- and power-consuming decoding steps. Innovations on this traditional feature extraction process, such as fully automated approaches [74] or the use of spike-related metrics such as spiking-band power (SBP) [69], could greatly advance real-time large-scale spiking-based decoding applications. This highlights the potential benefit of cross-scale data acquisition and analysis, whereby data that does not have these typically computationally expensive limitations (e.g. LFP) can be used to complement spike-based approaches. For example, Mehring et al showed that LFPs can be regarded as an additional source for decoding brain activity [25]; LFP activity, along with spikes, can be leveraged in a multi-scale model to decode a 3D reach task [35]. In addition, some material-based strategies are being developed to mitigate the problem of signal degradation over time [75]. For example, the surface coating of the electrodes [76] can be integrated with conductive polymers [77], shape-memory polymers [78], carbon nanotubes [79], flexible polymer [80], flexible mesh electronics [81], or nanoelectronic threads [82, 83] to improve the performance and reduce the interfacial mechanical mismatch. In summary, the next generation of spike-based neural decoding will likely be fueled by the synergy of high-channel count recording, advanced electrode materials, and multi-scale methods.

2.2. LFP

LFPs reflect electrical neural activity that is typically recorded extracellularly in neural tissue using invasive microelectrodes (figure 1). LFPs are thought to be generated by synchronized synaptic currents arising from hundreds of neurons and to capture key integrative synaptic processes that cannot be captured in spiking activity [21]. This signal is generally dominated by low frequency oscillations, with typical analysis focusing on activity below 200–300 Hz. Both time domain and frequency domain analyses are common with LFP. In the time domain, features of the neural signal may be used to determine location in the brain, for example changes in activity may indicate in which cortical layer [84] or hippocampal region the probe is located [85]. This is important for linking layer-specific or regional activity within a brain area to a particular behavior or process. It is also common to use a stimulus-triggered average of the LFP signal to determine the strength or properties of LFP caused by the stimulus [86, 87]. In the frequency domain, conventional analyses include computing power spectra or spectrograms, coherence, and phase synchrony. Power in different biologically-relevant frequency bands can be used to predict or decode a variety of behavioral variables [25, 35, 36, 88–90], from the goals of saccadic eye movements [91] to the value of stimuli [92] using classification techniques such as logistic regression, linear discriminant analysis (LDA), SVM, and other linear decoders. In addition, LFP has been shown to encode movement-related variables, such as the position and velocity of the subject’s hand [25, 27]. In addition to being inferred solely from LFPs, decoding of movement kinematics can be complemented by information from other modalities [25, 33–36, 93].

2.3. ECoG

ECoG measurements are extracellular field-potentials measured from electrodes placed on the surface of the brain (figure 1). ECoG electrodes can be placed directly on the pial surface (sub-dural) or on top of the dura but beneath the skull (epidural). Similarly, ECoG measurements are often divided into categories based on the size of the electrodes: µECoG refers to measurements made with electrodes on the micron scale (20–200 µm), while measurements with larger electrodes are macro-scale ECoG, typically referred to simply as ECoG. All ECoG measurements capture meso-scale field-potential signals reflecting aggregate dynamics of nearby circuit activity [21, 94] and therefore are typically analyzed in similar ways to other field potentials (e.g. LFP, EEG). The spatial resolution of ECoG signals, however, increases with decreasing electrode size and proximity to the nervous tissue. Similarly, the frequency ranges considered for ECoG analyses will vary depending on the electrode properties, with smaller electrodes able to capture higher frequency oscillations. At an extreme, sub-dural µECoG with 20 µm contacts can resolve MUA from superficial layers of the brain [95]. High-frequency ECoG activity (high-gamma, 80–200 Hz) has also been shown to be closely related to MUA [96–99].

As a meso-scale invasive measurement, ECoG has been most widely used for studying and decoding neural activity spanning multiple brain areas at the millisecond timescales relevant for behavior in clinical settings and in animals [21, 94, 100, 101]. As recording channel counts increase, ECoG recordings have proven useful for both high-density, high resolution spatial mapping of centimeter-sized brain areas [102–104] and to monitor highly distributed processing spanning nearly a full brain hemisphere [105]. Recent decoding and analysis techniques correspondingly aim to leverage the population-level dynamics and spatiotemporal characteristics of ECoG. For instance, ECoG can capture travelling waves of cortical activity that may influence ongoing neuronal processing [106–109]. One challenge in decoding and analyzing ECoG signals is their high dimensionality due to broad spectral feature spaces (multiple frequency ranges of interest) and large channel counts. Data-driven machine learning algorithms like recurrent neural networks [110] and unsupervised hierarchical clustering of neural dynamics [111] have proven valuable for decoding high-dimensional data. Another strategy is to leverage low-dimensional projections to capture spatiotemporal dynamics. Low-dimensional embedding has enabled development of state-space based decoding algorithms for ECoG [112, 113] and may be particularly useful for decoding ECoG population dynamics on a single-trial basis [114].

One reason ECoG is thought to hold promise for neural interface applications is its strong decoding performance combined with increased stability compared to other recording modalities (e.g. spiking activity) [115]. Several recent papers have demonstrated reliable real-time motor BMI control with ECoG [28, 29]. Understanding how to model potential longer-term dynamics and non-stationarities within ECoG signals is a recent area of work [116] and may be critical for enabling longitudinal studies and clinical neural interfaces.

2.4. EEG

EEG is a non-invasive technique for linking human neural activity to behavior which was first described almost 100 years ago [117]. This technique was first used to record large-scale patterns of ‘alpha’ oscillations, rhythmic ~10 Hz oscillations dominant over parietal and occipital electrodes, with subsequent studies identifying activity in adjacent frequency bands (e.g. beta, delta, etc [118]). However, discrepancies in the precise frequency boundaries between studies [119], along with considerable inter- and intra-subject variability in oscillation frequencies [120, 121] poses challenges to linking oscillations to behavior. Sidestepping such frequency differences by analyzing time-domain signals, early work focused on analyzing EEGs locked to particular stimuli or cognitive events by averaging data over many trials, so called event-related potentials [122]. Different ERP components have been identified for numerous cognitive operations spanning the breadth of study in cognitive neuroscience and have been useful in decoding cognitive operations in health and disease [123].

Subsequent research has emphasized time-frequency analysis of EEG signals to investigate spectral signatures of cognitive functions [124], with influential work showing that alpha laterality indexes spatial attention [125] and working memory capacity [126]. Building upon these findings, recent work has used inverted encoding models (IEMs) which flip the direction of inference. As the name implies, rather than asking how experimental manipulations modulate neural responses, IEMs attempt to predict neural responses using explicit models of neural activity and aim to determine how neural responses underlie stimulus representations [127]. This approach has been used to infer or ‘reconstruct’ the contents of memory or the focus of attention based on EEG patterns on single trials [128–131]. Because IEMs focus on inferring mental representations from neural activity, they are well suited for multiple measurement modalities with varying temporal or spatial scales. Indeed, since IEMs have been successfully used with fMRI [127, 132] and with EEG [133] we anticipate that IEMs will continue to grow with the increase in multi-modal imaging (see section 3.3.2).

2.5. fMRI

Traditionally, fMRI studies interrogate the blood oxygenation-level dependent (BOLD) contrast in a univariate (voxel-by-voxel) fashion to characterize task-induced neural activity in particular regions or throughout the brain. This imaging approach is non-invasive and allows whole-brain BOLD activity to be recorded simultaneously, providing the most comprehensive spatial coverage of all methods discussed so far (figure 1). This technique has enabled insights in deep brain function. For example, univariate fMRI studies showed that the striatum and the medial orbitofrontal cortex are consistently involved in processing subjective value during both decisions concerning value and receipt of valuable outcomes, irrespective of whether these outcomes are money or primary reinforcers [134], but with stronger involvement of the striatum in actual behavior [17]. Moreover, the striatum consistently processes reward prediction errors [135]. These findings may also have clinical relevance; as compared to healthy controls, patients with disorders such as depression [136] or psychosis [137] show reduced striatal activity during the receipt of rewards.

Univariate analyses depend on a uniform relation between neuronal activity and the investigated function, in both individual voxels and across participants. Multivariate methods can overcome this limitation of univariate analyses and decode heterogeneous relations between function and neuronal activity. They typically converge with univariate methods for homogeneous function-activity relations (for value functions, see [138]). Multivariate methods train classifiers on unsmoothed, voxel-wise, patterns of activity in different conditions and apply the trained classifier to independent test data. These classifiers exploit biased sampling at the level of fMRI voxels: the signal in a given voxel is thought to reflect the specific proportion of intermixed neuronal populations [139]. For example, one voxel and its underlying vasculature may sample more strongly from negative coding neurons whereas another voxel may sample more strongly from positive coding neurons and together the pattern of activity would differ between high and low value conditions.

Relatively recently developed multivariate methods provide insights not only on patterns of local activation but also on patterns of connectivity between regions and networks [140]. One variant of this approach uses condition-specific connectivity estimates from psychophysiological interaction models to train cross-validated SVMs. This approach revealed reward identity specific connectivity patterns of the central orbitofrontal cortex [141]. Specifically, the predictions of rewarding sweet and savory odors were differentially related to olfactory (i.e. piriform) cortex. Thus, the central OFC may extract sensory features of a reward from primary sensory cortices and endow them with identity-specific value. Moreover, integration of multivariate information appears to take place also in the visual domain [142], with anterior temporal lobe integrating shape information from lateral occipital cortex and color information from V4. Importantly, multivariate approaches to connectivity go beyond parallel analyses of activity patterns in different regions and may eventually capture not only interactions and information synchronicity but also transformations of information from region to region [140].

Combining fMRI with pharmacological interventions can help characterize the role of functional systems or the nature of represented information more precisely. For example, the dopamine D2 receptor antagonist amisulpride changes the whole-brain connectivity of medial and central but not of posterior-lateral orbitofrontal regions [143]. In particular, during rest it enhances multivariate connectivity of medial orbitofrontal cortex with more dorsal regions of frontal cortex and reduces connectivity of central orbitofrontal cortex with the midbrain and higher-order sensory cortices, suggesting that dopamine contributes to reconfiguring functional circuits. Moreover, dopamine D2-receptor blockade increases decoding of multivariate reward signals in medial orbitofrontal cortex, multivariate effector signals in motor and premotor cortex, and univariate reward signals in ventral striatum [144]. These effects could explain how antipsychotic drugs achieve their therapeutic benefits and suggest that network representations of specific types of information are under dopaminergic control.

Combining fMRI with transcranially applied electrical or magnetic brain stimulation can clarify whether the effects of stimulation on behavior arise through local effects at the stimulation site, through effects elsewhere, or through connectivity. For example, striatal connectivity and central orbitofrontal activity reflecting value are respectively reduced by downregulation of the temporoparietal junction [145] or the lateral prefrontal cortex [146] with transcranial magnetic stimulation (TMS). Moreover, determining stimulation sites based on resting state connectivity can allow stimulation to affect activity in regions that are not directly accessible to TMS, by virtue of their connections with the stimulated regions [147, 148].

2.6. PET

While fMRI has good spatial resolution, it lacks neurochemical specificity. PET overcomes this limitation and, at the expense of temporal resolution, can target a variety of neurotransmitter systems in healthy humans, both statically (e.g. receptor density) and dynamically (neurotransmitter release; for review, e.g. see [149]). Molecular PET can quantify the targets (receptors and transporters) of specific neurotransmitters by imaging radioligands in vivo. Other methods such as voltammetry and microdialysis are limited to intra-operative use in patients while PET can be used in healthy humans. Of particular functional relevance is the displacement of radioligands (which results in a change in binding potential) by endogenous transmitter release. This method has first been established for dopamine but more recently extended to serotonin, noradrenaline, acetylcholine and other neurotransmitters [150]. PET studies showed that neurochemical makeup varies between individuals of different ages [86, 151], and individual differences in this makeup are behaviorally and clinically relevant. For example, humans with higher capacity to synthesize dopamine in the dorsal striatum are more willing to exert cognitive effort than humans with lower synthesis capacity [152]. Moreover, patients with schizophrenia who respond to treatment show a negative correlation between prefrontal cortex grey matter volume and the striatally measured capacity to synthesize dopamine [153]. More generally, PET may contribute to tailoring therapeutic interventions to subtypes of patients and improving clinical outcomes.

Radioligands differ in their affinity to the natural receptors. For example, [¹¹C]raclopride acts as relatively low-affinity antagonist on dopamine D2 receptors and has been used to show dopamine release in the receptor-rich striatum induced by engaging activities, such as playing a video game [154], by unexpected receipt of monetary reward but not unexpected monetary loss [155] and by drug challenges such as amphetamine (e.g. [156]) or alcohol [157]. D2 receptor antagonists with higher affinity than raclopride (e.g. [¹¹C]/[¹⁸F]fallypride) and agonists (e.g. [¹¹C]PHNO) can reveal dopamine release in regions with fewer dopamine receptors and have been combined with dynamic models of endogenous dopamine release. One such study showed that a reward learning task induces stronger striatal dopamine release than a non-rewarded control task and that individuals with better performance in the learning task also show stronger dopamine release as well as reward-related increase in engaging in everyday activities [158]. Thus, the role of dopamine in reward learning appears to extend to real life.

Traditionally, PET assesses net regional radioligand binding, which requires that endogenous changes of neurotransmitter release are relatively long-lasting. Recent research combining PET with voltammetry indicates that PET may also be sensitive to minute-by-minute fluctuations in radioligand binding, making it possible to detect phasic dopamine release with raclopride [159]. This research suggests that phasic and tonic dopamine levels may be more strongly related than sometimes assumed. Because this method measures change rather than absolute levels of dopamine, it is also applicable to regions outside the striatum. One application of this method compared the ingestion of milkshake against tasteless solution [160]. Immediately after ingestion, regions such as the medial and lateral orbitofrontal cortex, anterior insula, and dopaminergic midbrain showed an enhancing effect of milk-shake on dopamine release, in line with value coding. In contrast, twenty minutes later, dopamine release occurred in regions such as the pallidum and frontal pole. Resolving these separate peaks would not have been possible with conventional PET. Future efforts to validate this method for measuring the release of dopamine and other neurotransmitters will benefit from a multi-methodology and multi-scale approach.

2.7. Beyond single-scale analyses

While tremendous neuroscientific progress has been made using the above methods in isolation, each method has tradeoffs which must be considered. These tradeoffs limit our ability to understand behaviors which likely arise from activity across spatial and temporal scales, such as human vision and memory which are thought to occur via local and interareal neural activity patterns. For example, prominent theories of human episodic memory argue that it occurs through local changes in synaptic weight between neurons combined with distributed interactions between the hippocampus and neocortex [161]. It follows that with the limitations of any particular method, no single technique holds a privileged position in fully understanding memory systems, and that combinatorial methodological approaches should provide a more complete picture as compared to any method in isolation. Below, we discuss progress towards combining recording modalities to gain a richer and more complete view of how neural activity patterns across spatial and temporal scales give rise to behavior.

3. Multi-scale analyses

3.1. Advantages of multi-scale approaches

One of the primary advantages of multi-scale and multi-modal analyses is the formation of a more complete picture of the neural processes giving rise to behavior. Unimodal methods each have a unique set of strengths and weaknesses (figure 1). For example, noninvasive electrophysiological methods like EEG and MEG have high temporal resolution but poor spatial resolution, and the inverse is true for fMRI, which has relatively poor temporal resolution but relatively high spatial resolution. Even an invasive method like ECoG, which has high temporal resolution and good spatial specificity, suffers from limited spatial coverage. However, high resolution in both the spatial and temporal domains is essential for building a more complete understanding of the neural processes underlying cognition. Therefore, one way around these limitations is to combine data from multiple recording modalities and scales of measurement, either collected simultaneously (e.g. EEG-fMRI) or collected separately but combined analytically, to effectively exploit the strengths of each method.

For example, recent work has advanced a new method for analytically combining (or ‘fusing’) the high temporal resolution of magnetoencephalography (MEG)/EEG data with the high spatial resolution of fMRI data [39–42]. This approach does not require the data from different modalities to be collected simultaneously, allowing the combination of techniques like MEG and fMRI, which cannot be collected at the same time. Briefly, this approach uses a technique related to representational similarity analysis [162] to link (or ‘fuse’) multivariate patterns of brain activity from single modalities to each other [42]. This method can be restricted to a priori regions of interest or it can be conducted in an exploratory fashion throughout the brain [40]. For example, this ‘fusion’ approach has been applied to asynchronously collected MEG and fMRI data to identify temporally and spatially precise signatures of human object recognition [39]. The improved spatiotemporal resolution achieved by fusing MEG and fMRI data allowed a more complete picture of the neural processes underlying human visual object processing, which evolves rapidly over time and involves distinct contributions from a hierarchy of brain regions.

A second crucial benefit of combining data across scales and modalities is the opportunity to understand one data type with the addition of information from another. For instance, EEG has some key advantages over fMRI in cost and portability, making it an attractive option for clinical and translational settings. However, its limited spatial resolution and poor source localization, particularly for subcortical regions like the amygdala, can limit its usefulness. Recent work creatively circumvented this limitation by using simultaneous EEG-fMRI to build a multivariate model linking data-driven features of the EEG signal (termed a ‘fingerprint’) to fMRI BOLD activity in the amygdala [43, 163]. This EEG ‘finger-print’ enabled the prediction of amygdala fMRI activity in held-out data, demonstrating improved spatial specificity over conventional EEG-only approaches [163]. This technique was exploited for EEG-based neurofeedback training in soldiers facing stressful military combat training [43]. Soldiers were successfully trained using an EEG fingerprint to down-regulate their amygdala activity and reduce their emotional reactivity. This provides promising evidence for the utility of such a multi-modal approach in creating low cost, portable, and practical clinical applications (see section 5).

3.2. Necessary tools for multi-scale analysis

To successfully implement multi-scale analyses, essential experimental tools (i.e. neurotechnologies) and computational tools (i.e. analytical methodologies) are required based on the intended modalities. From the technology perspective, the main challenges for simultaneous acquisition of multi-scale recordings lie in making a single device capable of recording or manipulating neural activity at different scales, or in developing devices for different recording modalities that are compatible with each other. For example, in order to work simultaneously with MRI, the electrical and magnetic properties of recording electrode materials have to be considered. In this case, copper, gold, silver, and carbon are preferred considering their electromagnetic properties as well as biocompatibility [164]. Furthermore, researchers have designed cutting-edge materials [165, 166], such as Opto-E-Dura, a stretchable ECoG array that is compatible with additional wide-field calcium imaging, two-photon calcium imaging, and acutely inserted microelectrode arrays [165].

In terms of analytical methods, computational and statistical tools, like the fusion approach described above, are needed to relate different types of data that inherently have disparate spatiotemporal dynamics and underlying properties. Correlational or coherence analyses are usually performed to study the interaction between spikes and LFP. Forward and inverse models have been applied across different types of field potential signals to identify large-scale brain dynamics [94, 167–173]. Mean-while, machine learning approaches are frequently used to decode experimental tasks and to predict future movements or cognitive processes based on the recorded signals [174, 175]. The development of these tools is critical and will provide us with new insights into the interplay between different modalities. In the following sections, we present discussions of powerful new multi-scale approaches that combine neural data from two or more modalities/scales.

3.3. Multi-scale examples and case studies

To have a clear visualization of current progress and efforts on single or multi-modal analysis, we customized a MATLAB script to extract the numbers of research articles corresponding to each single modality or each pair of multiple modalities from PubMed. The counts of the articles satisfying each query are returned from PubMed and illustrated as a heat-map (figure 2). The keywords we used include fMRI, PET, EEG, ECoG, LFP and spikes. We also include the expanded terms such as ‘functional magnetic resonance imaging’ for fMRI. The detailed protocol is provided in the caption of figure 2.

Figure 2.

The heat map on the left shows the numbers of PubMed search results (see Supplementary Materials available online at stacks.iop.org/JNE/18/045013/mmedia) for each recording modality (shown along the diagonal) or combination of recording techniques (upper off diagonal) from 1938 (the first article on EEG retrieved from PubMed) to 2021. That is, the diagonal elements indicate the numbers of unimodal publications, while the off-diagonal elements show the numbers of publications that investigated two modalities in one study. The keywords we used for the six modalities include the acronyms and the expanded forms (e.g. (‘fMRI’ OR ‘functional magnetic resonance imaging’) for fMRI). However, we only used the keyword ‘unit activity’ for spikes, single- and multi-unit activity because ‘spikes’ itself would include irrelevant results such as viral spike proteins. For entries that involve two modalities, we used the AND operator in PubMed, such as (‘PET’ OR ‘positron emission tomography’) AND (‘EEG’ OR ‘electroencephalography’) for PET-EEG studies. Data is shown using a logarithmic color axis. Three representations of typical analyses in the combined modalities are shown in the boxes on the right, including fMRI-EEG, fMRI-PET, and spike-ECoG (e.g. see analyses in [42, 218, 277, 278]). While we have limited this heatmap to pairwise combinations, examples of work simultaneously spanning three or more modalities and/or scales are briefly discussed in sections 3.3.1 and 3.3.5.

3.3.1. Multi-scale intracortical measurements

Intracortical measurements include spikes, LFP, and ECoG activity, which span a range of temporal and spatial scales. The combinations of these types of signals can thus provide rich information about neural function. For example, spikes represent activity from individual neurons at a fast temporal scale while LFPs are believed to represented local synaptic activity generated from a population of hundreds of neurons [21] and contain oscillatory activity that may play a key role in coordinating activity across brain regions [176–178]. Analyzing the relationship between single neuron and population-level activity can reveal the integrative properties of neurons, as well as the organization of cortical circuits [179]. One notable difference with the combination of spikes and LFP compared to other signals is that they can be recorded using the same modality—implanted extracellular microelectrodes. For every electrode channel on which spikes are recorded, LFP signals can also be simultaneously acquired. This means that the joint analysis of simultaneously recorded spikes and LFPs provides an opportunity to assess the relationship between spikes and the strength of synaptic activity measured co-locally or at different locations.

A standard analysis used to characterize interactions between spikes and LFPs is the spike-triggered average of the LFP (figure 2). This time-domain metric may be used to quantify postsynaptic responses to spiking at a particular location (where LFP is recorded) [180] and determine latency between spikes and events in the LFP [94]. Alternatively, spikefield coherence is a spectral-domain metric that can be used to characterize synchronization of spike timing at particular phases of LFP activity [181] and is indicative that a local neuron contributes to the field potential signal [88, 182]. Similarly, a spike-triggered spectrum of LFP activity can be used to determine how power in different frequency bands relates to local spiking activity. Other metrics used to characterize interactions include spike-field Granger causality [183], Volterra models of LFP-spike timing relationships [179], and multiscale causality estimation [184].

Aside from analyses that characterize relationships between these two scales of activity, there are also efforts to leverage information from both scales in unison to learn something about behavior or cognition. Two distinct modeling approaches have been developed to interpret neural activity principally from spikes and LFPs, including statistical modeling and biophysical modeling. With regard to statistical modeling, earlier methods leveraged spiking and LFP activities from similar timescales in the same models [25, 33, 34]. For instance, Khorasani et al [185] proposed an adaptive filtering method using weighted common average reference filter, and decoded kinematic force information based on both LFP and MUA with a partial least square model. The decoding accuracy using the LFP and MUA outperformed models that relied solely on either LFP or MUA. The authors suggested that LFP and MUA may have information that is different or supplementary in terms of the covariates they investigated. As another example, Stavisky et al [34] recorded signals from the motor cortex when macaques performed a reaching task. They concluded that the performance of hybrid decoders from LFPs and spikes outperformed unimodal models, especially when the signal quality of spiking measurements was mediocre to poor. Sub-sequent works modeled combining activity from different spatial and temporal aspects [35, 36, 38]. Specifically, Hsieh et al developed a model that regarded spikes as a point process at fast millisecond timescale but treated LFP at slower timescale, which collaboratively improved the decoding of movement when using activity from motor cortex [35]. Their model can adaptively and separately update parameters at different rates for LFPs and spikes in closed-loop simulations. On the other hand, some studies modeled spikes and LFPs in a biophysical manner, where they aimed to identify the neural sources that contribute to the recording patterns in spikes or LFPs [21, 94, 180]. For example, the integrate-and-fire neuron model [186, 187] and its derivative, the leaky integrate- and-fire (LIF) model [188], are commonly used to describe spiking neurons and study brain functions. Moreover, Mazzoni et al successfully predicted LFPs from the LIF model and provided a simple formula that could quantitatively link neural models and LFP measurements [189].

In another recent work, spikes and LFP were used by an LDA classifier to improve accuracy of decoding value representations in a choice task with stimuli of differing values [92]. Other work has used an SVM with both spike and LFP data from lateral prefrontal cortex to decode information related to cue position, visually attended location, and saccade direction [190]. Using this approach, the authors were able to quantify how much additional information spikes carried about these variables by examining the decoding performance improvement from combined spike and LFP data, compared to using LFP signals alone. A similar approach has been used to demonstrate how anterior cingulate cortex spike and LFP activity can be used to decode intensity of pain responses to different noxious stimuli, indicating that single units play a role in the representation of pain and that LFP signal can predict nociceptive intensity [191].

Further examples come from the motor control literature, where researchers found that between spiking and LFP activity, spiking encodes direction more strongly than LFP, while high frequency (100–300 Hz) LFP best encodes speed and low frequency (10–40 Hz) LFP best encodes movement initiation [192]. These findings indicate that distinct information is represented by each signal type, and BMI performance may therefore be optimized by strategically using direction, speed, and go/no-go information from spikes and LFP. Furthermore, even in cases in which spikes and LFP encode similar information [25, 34, 193], combining these two modalities can still improve the decoding performance. For example, Abbaspourazad et al [36] used a multi-scale low-dimensional model to describe the predicted behavior and spike-LFP activity characterized by some principal modes. They found that combined spike-LFP activity can learn multi-scale predictive modes more effectively and more accurately predict naturalistic movements. In addition to representing distinct information in both scales, this study demonstrated that adding these two signals with similar contents also improved decoding performance by acting as a form of denoising. Additionally, the use of high-frequency LFP (on the order of~10²–10³ Hz) as a proxy for threshold crossing-detected action potentials is gaining attention. This LFP band was hypothesized [194] to contain information similar to low-pass-filtered spikes and had more robust chronic performance than threshold crossing spike rates. Nason et al [69] defined SBP as the power in the 300–1000 Hz band. They found a robust correlation between SBP and single-unit firing rate, across changes in firing rate or recording noise level. SBP produced more accurate predictions of macaque finger movements than the threshold crossing rate feature when both inputs entered SVM decoders. These works demonstrate the power of leveraging both spikes and field potentials to yield neuroscientific insights and develop robust BMI paradigms that can translate to the clinical domain.

Another form of multi-scale electrophysiology is integrating intracortical spike and/or LFP recordings with ECoG (figure 2). Integrating these modalities synchronously requires addressing physical compatibility of the two recording modalities, which has seen considerable progress over the past decade (e.g. [195]). Existing strategies for integration include using ECoG arrays fabricated with holes through which intracortical probes can be inserted [195–197], performing sparse ECoG measurements without a monolithic array that provide space to insert probes [198], placing multi-unit arrays underneath ECoG arrays [199], and custom devices with mixed electrode formats [200]. Spatial registration of the two measurements may be critical for analysis and interpretation of signals and is therefore an important consideration in methodological design.

Multi-scale analysis methods for ECoG and intra-cortical recordings are similar to those of spike-LFP. This includes both examining temporal relationships between the signals (trigger-based analyses, coherence) and using mixed neural features as inputs for decoders. Comparison of signals suggests that ECoG may capture aspects of neural processing that are distinct from spikes and LFPs [196, 200, 201]. For example, classification of spoken vowels from combined intracortical and ECoG recordings was highest when decoders used a mixture of spikes, LFP, and ECoG features, with spiking combined with ECoG providing the largest benefits of any two pairs [200]. Interestingly, simultaneous macro ECoG and multi-electrode intracortical array recordings suggest that for a matched number of electrodes, ECoG recordings in visual cortex provided higher decoding accuracy of visual stimuli than LFP or spiking activity [201]. The difference in brain area coverage may partly contribute to these differences and highlights the potential utility of multi-scale measurements. Fully leveraging multi-scale measurements for decoding algorithms will require new mathematical approaches that can handle the statistical and temporal differences between spiking activity and field potentials, which is an emerging area of work [35].

3.3.2. EEG and fMRI

Beginning in the mid-1990s (e.g. [202]), simultaneous collection of EEG and fMRI data has enabled the direct linking of electrical brain activity and hemodynamic responses. For example, during resting periods, the BOLD signal in the occipital cortex has been found to correlate negatively with alpha activity [203], that is, cortical inactivity measured with fMRI is associated with large-scale synchronous activity measured with EEG. This demonstrates that simultaneous EEG-fMRI does not merely localize the neural sources of EEG signals, but can identify distinct neural signatures across the two modalities that correspond to the same underlying mental state. Furthermore, EEG and fMRI have complementary strengths in temporal resolution (milliseconds vs seconds) and spatial resolution (centimeters vs millimeters). Exploiting their combined spatiotemporal resolution overcomes the limitations of each and allows for the characterization of the temporal dynamics of cognition in precise neural circuits [204].

The key advantage of simultaneous collection of EEG and fMRI data, as opposed to post-hoc combination of data collected separately (e.g. [42, 205]), is to guarantee identical sensory stimulation, perception, and behavior which makes it possible to investigate how intrinsic brain states interact with extrinsic processing (figure 2). Acquiring these data involves the use of specialized EEG hardware that is safe and compatible with the MR environment, although the data are highly sensitive to artifacts and methods like independent component analysis (ICA) must be used to identify and remove artifactual signals [206]. A bulk of the simultaneous EEG-fMRI studies conducted thus far have focused on healthy subjects: based on the scientific articles published in PubMed since 2014 using the key word ‘simultaneous EEG-fMRI’, nearly 60% of all studies have focused on healthy control subjects rather than on patients with neurological impairments [207].

The simultaneous acquisition of EEG and fMRI data has been used to study the brain at rest and during the execution of specific tasks, yielding insights that would not have been possible within a single modality [208]. Linking electrical activity to specific neural circuits provides more mechanistic explanations of cognition. For example, low-frequency theta-alpha oscillations (4–13 Hz) during a recognition memory test have been shown to emanate from a canonical memory circuit consisting of the hippocampus, the striatum, and the prefrontal cortex [209]. High-frequency gamma-band oscillations (30–100 Hz), on the other hand, have been linked to occipito-parietal network activation during tests of visual perception [210]. Furthermore, studies of decision making have shown that modulations of power in beta (12.5–30 Hz) and theta bands, associated with positive and negative feedback, are linked to activation in the subcortical reward network and in fronto-parietal areas, respectively [211].

Finally, EEG and fMRI have been combined in neurofeedback studies which allow for targeted manipulation of brain activity. In healthy adults, modulation of the thalamic nuclei during the retrieval of happy autobiographical memories produced changes in posterior alpha power that correlated with changes in the thalamic BOLD signal [212]. In conclusion, the simultaneous combination of EEG and fMRI offers the opportunity to characterize the relationship between temporally precise electrical brain activity and spatially specific regional hemodynamic activity to better understand the neural dynamics of behavior and cognition.

3.3.3. Multiple imaging modalities: fMRI and PET

The combination of fMRI with PET (e.g. [149, 213]; figure 2), which has been developed over the past 15 years [214] enhances and informs insights gained with single-modality methods. While both imaging modalities provide functional measures of neural activity, they offer distinct advantages. fMRI measures BOLD signals which are thought to reflect regional time-varying changes in neural activity and is able to achieve a higher temporal resolution than PET [215], whereas PET is able to visualize metabolic processes and provide information on neurotransmitter-specific activity [216]. Combining these two modalities enables the study of how BOLD activation relates to neurotransmitter release, which is particularly powerful when linking these measures to a particular behavior or investigating an intervention [14, 149, 217–221].

One line of research in this domain showed that striatal BOLD responses correlate with relative dopamine D2/D3 receptor occupancy in the striatum of non-human primates [218]. In humans, BOLD activity elicited by reward anticipation in the dopamine producing substantia nigra and ventral tegmental area correlates with reward-related striatal dopamine release as measured with PET [14]. Relatedly, striatal BOLD activity induced by reward anticipation correlates with dopamine transporter [219] and dopamine D2 receptor [220, 221] availability in the dopamine producing regions and with dopamine release in the striatum [222]. Thus, combining PET with fMRI can elucidate neurochemical systems which synergistically implement cognitive and motivational functions. Using combined fMRI-PET machines together with time-resolved PET [159] will allow future investigations of behavior and cognition to control for sensory stimulation and more tightly integrate neurochemical information with BOLD responses.

3.3.4. Functional imaging (fMRI or PET) with intracortical measurements

Combining a non-invasive imaging modality with invasive intracortical methods can provide complementary information about network-level and cellular-level neural activity. While functional imaging methods such as fMRI and PET enjoy large spatial coverage of the brain, they provide only indirect measurements of neural activity. The combination of one of these modalities with extracellular recording techniques using penetrating electrodes, can both provide multi-scale spatial information regarding neural activity as well as help inform the relationship of the imaging metrics (i.e. BOLD or PET signal) to cellular function (similar to what was described in section 3.3.2 for EEG-fMRI). For example, it has been demonstrated in parietal cortex of macaques that small clusters of task-selective neurons can facilitate large fMRI activation [223]. Additionally, in Parkinsonian patients it was shown that spiking in the globus pallidus internus correlated with thalamic glucose metabolism, as measured through PET, and that variations in spiking rates are associated with individual differences in glucose metabolic activity [224].

Although there is power in combining these multi-scale techniques, scientific and technical challenges have limited their combined popularity in the research and clinical communities. The first simultaneous recordings of LFP and spiking activity with fMRI reported that LFP signals had a stronger contribution to and higher correlation with BOLD responses compared to spiking activity [225]. Evidence supports that this is likely due to the spatial specificity of spiking activity, in contrast to LFP signals which integrate activity over a large spatial region [225]. This weak relationship between BOLD fMRI and spikes may explain the low number of fMRI-spike studies (n = 48, see figure 2).

Other challenges include (a) the need for MR-compatible electrodes which do not cause significant artifacts in MR images and (b) the appropriateness for both modalities in a given subject. To the first point, there is active work to develop extracellular recording methods to acquire LFP, spikes, or both, that minimize susceptibility artifacts during fMRI. These methods generally leverage new signal acquisition and processing paradigms [225, 226], and/or novel electrode designs and materials [227–229]. To the second point, invasive intracortical methods are not typically used in human subjects unless clinically motivated. For this reason, non-invasive methods (EEG, fMRI, PET) are typically leveraged in studies with healthy human subjects. On the other hand, intracortical methods are more easily implemented in animal models. However, fMRI and PET are both sensitive to movement, which introduces additional challenges with these functional imaging methods in awake animals. For this reason, many fMRI-spike and PET-spike studies are performed on anesthetized animals (e.g. [225, 227–230]), although some studies utilize specialized MR-compatible restraint systems to limit movement during imaging (e.g. [223, 231]).

3.3.5. Combining three or more modalities/scales

While more limited, there are notable examples that have simultaneously acquired data across three or more scales and/or recording modalities. This is most feasible for multi-scale electrophysiological approaches. For example, as discussed in section 3.3.1, decoding of speech production from ventral sensorimotor cortex benefits from simultaneous information from spike, LFP, and ECoG recordings [200]. In addition, Moosmann et al investigated the correlation between fNIRS, fMRI, and EEG [232], and Golkowski et al simultaneously measured EEG, PET, and fMRI in patients suffering from disorders of consciousness [233]. These combinations have also been explored by other research groups, highlighting the potential for incorporating three or more modalities in one experimental setting [234, 235]. Combining across three or more scales in this way can be particularly beneficial, both for characterizing the types of information carried at different scales [201], and for capitalizing on complementary sources of information to build more robust decoders for applications like BMI [236]. Finally, simultaneous recordings across more than two scales/modalities have proven indispensable for understanding the underlying neural source(s) of the signals measured with noninvasive modalities like EEG [237] and fMRI [225].

4. Challenges for future multi-scale studies

While we have highlighted many exciting and impressive works (figure 2) that have combined multi-scale neural activity for a variety of neuroscientific, clinical, and engineering goals, there are still challenges ahead to promote the advancement of multi-scale efforts. One major challenge is the need for multi-scale modeling and analytical methods to provide fundamental infrastructure for understanding neurological processes, disease, cognition, and behavior. New, versatile frameworks to elucidate multi-scale activity at a mechanistic level will enable this challenge to be addressed. Another major challenge is the need for harmonious technologies to enable cross-scale interrogation of neural activity. Although significant insights can be made with data that is not acquired synchronously, neurotechnologies that allow activity recorded using different modalities simultaneously will serve to broaden our toolkit for making consequential advancements.

The two core challenges presented above are affected by several related factors. Firstly, there is a lack of software options that can operate across multiple modalities. While leading electro-physiology companies, such as Plexon Inc. and Black-rock Microsystems, already provide solutions for recording activity such as spikes, LFP, ECoG, and EEG simultaneously, there are limited commercial options for simultaneous acquisition of other combinations of data types, such as electrophysiological with functional imaging data. Simultaneous acquisition of EEG and fNIRS can be completed with the same head montage, but the data itself must be acquired with different systems. This need for cohesive software options goes hand-in-hand with hardware consolidation to support recording neural activity synchronously at different scales from different modalities. Secondly, funding mechanisms that encourage collaborations between researchers with specializations in different modalities would fuel progress across all fronts: analysis, neurotechnology advancement, and software and hardware development. These efforts cannot thrive without the investment of resources from funding agencies and labs. Thirdly, scientific meetings that promote cross-dialogue and discussion about insights and challenges across modalities can play a role in bringing together teams of researchers with a range of expertise. Existing conferences in the neural engineering and neuroscience fields can promote this objective with special sessions, forums, or satellite meetings focused on multi-scale and multi-modal methods. Fourthly, to facilitate this paradigm of integration of different types of neural activity, there is also a need for a common lexicon to communicate across modalities. Altogether, addressing these challenges and considerations will enable an exciting future in multi-scale studies.

5. Outlook and translational applications

Understanding multi-scale interactions can facilitate not only our knowledge of neural function, but also the development of rehabilitative therapies that may leverage information across scales to create more effective and potentially restorative treatments. Brain imaging in the context of rehabilitative interventions is often utilized through neurofeedback or BMI platforms. Multimodal neurofeedback approaches take advantage of the different measurement scales provided by each imaging modality. For example, the high spatial resolution of fMRI can be combined with the high temporal resolution of EEG. This can lead to stronger therapeutic effects by providing patients with more detailed information to develop better neural strategies compared to a single modality alone [238]. Young et al [239] demonstrated the first implementation of simultaneous neurofeedback using fMRI and EEG data to train healthy participants to self-regulate neural activity in an emotion regulation task. The addition of EEG recordings allowed examination of the electrophysiological correlates of fMRI neurofeedback training, including frontal EEG asymmetry and EEG coherence, that could be translated to patients with major depressive disorder [240] and post-traumatic stress disorder [241].

BMIs exploit an individual’s ability to modulate a target brain region and translate this to control of an external device. These platforms have been widely investigated as a method to assist impaired individuals’ ability to interact with the world around them. However, certain challenges exist that limit the usability of BMIs, such as the requirement to optimize control features and identify successful mental strategies to properly control a device [242]. Additionally, a phenomenon known as ‘BMI illiteracy’ exists, in which around 15%–30% of individuals are unable to learn to control a BMI [243]. Multimodal approaches have been suggested to address these challenges by providing the BMI with more detailed information from the temporal dynamics of electrical activity and the hemodynamic fluctuations in the cortex [242]. This can improve a BMI’s ability to decode the user’s intention which leads to better usability and control [244, 245].

Noninvasive brain stimulation, such as TMS and transcranial direct current stimulation (tDCS), has also been investigated as a treatment for major depressive disorder, schizophrenia, epilepsy, phobia, and stroke rehabilitation [246]. However, many behavioral manifestations of neurological and psychiatric disease are the result of alterations in distributed brain networks, and neuroimaging technology can be utilized to determine the optimal electrode placements for multichannel tDCS [247]. Computational models have also been built for tDCS [248] and TMS [249] to better estimate current flow patterns in the brain, to design new electrode montages, or to improve dosimetry. The simultaneous acquisition of neuronal and hemodynamic responses along with stimulation can provide an objective form of feedback to guide the stimulation procedure, quantify the stimulation effects, and investigate the underlying neural dynamics [246, 250–252]. Furthermore, the combination of fMRI and EEG with concurrent stimulation can further elucidate the effects of stimulation on connected brain regions [145, 253]. However, a major challenge with simultaneous stimulation and measurement is the introduction of strong artifacts in the obtained signal. In the case of concurrent TMS and EEG, the electromagnetic field generated by the stimulation pulse can induce a large measurement artifact that is much stronger than the typical electro-physiological brain activity measured by EEG [254]. Additional care must be given to compensate for these artifacts by using recently developed EEG amplifiers to characterize the stimulus pulse and filter it from the measured signal [255] or to measure activity only after the stimulation artifact has subsided [256]. Despite these challenges, many studies have already leveraged the power of combining TMS/tDCS with electrophysiological recordings or functional imaging techniques for various applications. For example, the combination of offline repetitive TMS (rTMS) and fMRI shows causal relationship between neural computations and strategic social behavior [257]; simultaneous MEG/EEG and TMS can identify the causal influence of oscillatory entrainment on working memory performance [258]; and offline rTMS can provide lasting disruption of the hemodynamic activity in the prefrontal cortex measured with fMRI [259]. In addition to overcoming technological barriers to bridge different techniques, computational models also play a critical role in integrating information across modalities. For example, a model that characterizes both fMRI activation and electrical field mapping using tDCS can address inter-subject and within-subject variabilities [260]; a biophysical model has also been demonstrated to assist with personalizing TMS protocols in conjunction with resting-state fMRI [261]. These promising results highlight the synergy of noninvasive modulatory approaches such as TMS/tDCS and noninvasive neural recording techniques such as fMRI/EEG.

Capturing multi-scale interactions can also help elucidate the neural dynamics underlying healthy neurophysiological processes. A combination of electrophysiological and hemodynamic imaging can be beneficial to understand how neuronal changes relate to neurovascular coupling [262]. For example, a multimodal approach of EEG and fNIRS revealed an inverse relationship between oxyhemoglobin concentration and alpha and beta power during a motor task [263]. This can be beneficial to clarifying the neural dynamics during motor imagery, a popular target for BMIs and neurofeedback for its potential applications for stroke rehabilitation [264].

There is considerable evidence across neuroscientific disciplines that the brain integrates and represents information across spatial and temporal scales. In perception, information is integrated within 300–600 µm cortical columns and more than 30 distinct brain regions have been identified which together support our understanding of the visual world [265, 266]. Similarly, cascading molecular changes can modulate synaptic strength to encode memories [267]. Regarding memory, it has been recognized since the stimulation studies of Wilder Penfield that memories are widely distributed throughout the brain [268]. Contemporary theories of episodic memory argue for the integration and segregation of information distributed between the hippocampus and neocortex as central to memory organization [161, 269]. Neural oscillations exert an influence on both local and distributed neural populations and may subserve integrative functions (reviewed in [21, 270, 271]). It follows that interrogation of oscillations should be informative for understanding behaviors which require integration over scales. Indeed, mounting evidence has shown that oscillations provide similar or better readout of behavior than local neuronal spiking [272–276]. These findings indicate that approaches which interrogate population-level activity measured using multiple methods will provide more insight into behavior by leveraging combinatory power when analyzing data [51] across different spatial or temporal scales of analysis.

Data availability statement

No new data were created or analyzed in this study.