Naturalistic Stimuli: A Paradigm for Multi-Scale Functional Characterization of the Human Brain

Abstract

Movies, audio stories, and virtual reality are increasingly used as stimuli for functional brain imaging. Such naturalistic paradigms are in sharp contrast to the tradition of experimental reductionism in neuroscience research. Being complex, dynamic, and diverse, naturalistic stimuli set up a more ecologically relevant condition and induce highly reproducible brain responses across a wide range of spatiotemporal scales. Here, we review recent technical advances and scientific findings on imaging the brain under naturalistic stimuli. Then we elaborate on the premise of using naturalistic paradigms for multi-scale, multi-modal, and high-throughput functional characterization of the human brain. We further highlight the growing potential of using deep learning models to infer neural information processing from brain responses to naturalistic stimuli. Lastly, we advocate large-scale collaborations to combine brain imaging and recording data across experiments, subjects, and labs that use the same set of naturalistic stimuli.

Introduction

Progress in functional brain imaging has been remarkable. While technological advancements push the limits of imaging resolution and speed, experimental paradigms also evolve in parallel. Traditional paradigms use simple stimuli or tasks designed to control for extraneous variables and thus isolating a variable of interest and specifically testing how that variable changes brain activity. Such paradigms set up specific hypotheses for mapping specific functions onto specific brain regions in line with the long-standing focus on functional specialization, e.g., face perception in the right fusiform gyrus [1]. Findings resulting from these paradigms, however, tend to be piecemeal and sometimes incohesive, and largely ignore the distributed and flexible nature of many perceptual and cognitive processes [2]. Since the early 2000s, many functional brain imaging studies have shifted their focus to the task-free resting state [3]. This paradigm shift redirects the focus from mapping functions to mapping the brain connectome as a collection of resting state networks. The resting state is not so simple as is nominally implied but varies across visceral, arousal, and mental states [4–6]. Nevertheless, the resting state is not explicitly biased by any overt task and thus useful for mapping the brain’s intrinsic functional architecture while being convenient to leverage big data [7]. Although resting state networks are relatively straightforward to map, their functional interpretation is often speculative due to the lack of functional context in a task-free condition.

In this paper, we advocate naturalistic stimuli as a paradigm complementary to traditional task paradigms, which are overly specific, and resting-state paradigms, which are entirely non-specific. Naturalistic stimuli, e.g., movies, speech, virtual reality, set up a more ecologically relevant condition for brain research and involve rich spatial and temporal contexts to engage a wide range of neural processes and mechanisms [8]. Computational models are increasingly available to decompose seemingly complex stimuli into explainable features and to relate the decomposed features to specific neurons, voxels, regions, and networks observable with brain imaging and recording techniques. Naturalistic paradigms present an opportunity to combine data across different spatiotemporal scales, functional systems, imaging modalities, and individuals or laboratories. In the following, we review the recent technical innovations and neuroscientific findings that support this perspective. Then we outline future directions for utilizing naturalistic stimuli to curate and leverage “big data” for high-throughput and multi-scale functional characterization of the human brain.

Reproducible brain responses to naturalistic stimuli

The brain enables humans to explore and interact with a complex and dynamic world. Stimuli are considered to be naturalistic if they mimic the real-world while being diverse and dynamic. Typical examples include movie, speech, or music, that involve perceptual, cognitive, and emotional experiences common in daily life. A defining feature of naturalistic stimuli is their ability to evoke brain responses that are highly reproducible within and across subjects as first demonstrated by Hasson and colleagues [9]. Specifically, different subjects viewing the same movie exhibit consistent responses in the stimulus-evoked cortical locations [10] and even in the white matter [11]. Inter-subject functional correlations further separate stimulus-related network interactions from stimulus-unrelated ongoing activity [12, 13]. Being naturalistic is necessary for stimuli to induce highly reliable cortical responses. Natural stimuli tend to generate more reliable responses than artificial stimuli [14]. For example, an artificial video that preserves the spectral magnitude but randomizes the spectral phase of a natural movie does not elicit reproducible fMRI responses even in early visual areas [15].

Different movies may further engage different regions across attention, limbic, and cognitive-control cortical networks [14]. Other than videos, speech and music are increasingly used to investigate neural processes for speech recognition [16], language comprehension [17, 18], and music perception [19]. Each type of naturalistic stimuli engages a broader set of brain regions and more diverse modes of network interactions than its artificial counterparts. Arguably, a relatively smaller battery of different types of naturalistic stimuli may offer more comprehensive and ecologically relevant perspectives to investigate the entire brain than is possible with many more traditional paradigms with strict experimental reductionism. The high throughput is a compelling advantage of using naturalistic stimuli in brain research. We also refer readers to a recent review by Leopold and colleagues [20].

Naturalistic stimuli can induce reliable brain responses observable with different techniques, e.g., functional magnetic resonance imaging (fMRI) [14], magnetoencephalography (MEG) [21] and electroencephalography (EEG) [22], electrocorticography (ECoG) [23], local field potential (LFP) or even single-unit activity [24]. As such, naturalistic stimuli engage reliable neural processes in a wide range of spatiotemporal scales to give rise to rich experiences common across individuals. While most studies focus on one type of neural signal, relating different types of signals under the same natural stimuli is a compelling approach to compare and integrate findings obtained with different techniques [25–27].

In daily-life experiences, sensory information is embedded in rich spatial and temporal contexts. For biological vision, humans have different perceptual understandings of the same input placed in different spatial contexts (e.g., through contextual modulation [28]) or take different actions upon the same object depending on its temporal context (e.g., through process memory [29] and recurrent computation). It is impractical to design or synthesize artificial stimuli to adequately exemplify the diverse contexts that occur in a dynamic and realistic environment. As a nonlinear and dynamical system, the brain does not follow the superposition principle. Brain responses to stereotypical stimuli, while useful for descriptive understanding, do not amount to a computational and generalizable understanding of neural information processing under naturalistic conditions. From the perspective of nonlinear system identification, the process of modeling and understanding the brain should benefit from choosing the system’s input signal that follows the statistics of natural stimuli, which drive continuous learning and inference in the brain.

Integrating brain data across different scales and modalities

Modern imaging and recording techniques are complementary to one another. No single modality by itself offers sufficiently high spatial and temporal resolution or specificity to reveal a complete picture about brain function [30]. The use of the same naturalistic stimuli across different experiments, subjects, and labs set up a common context to bridge reproducible brain responses observed with various modalities across different spatiotemporal scales (Figure 1).

Figure 1.

Integrating brain and behavioral data across different scales and modalities given the use of a common set of naturalistic stimuli. a) Different subjects undergoing the same naturalistic stimuli exhibit consistent brain responses such that data collected from different subjects or even different labs can be integrated. b) Different types of naturalistic stimuli can be used to probe the interactions within and across neural and cognitive systems responsible for perception, action, memory, emotion, and cognition. c) Given the same naturalistic stimuli, brain responses observed with various imaging or recording techniques can be linked to one another, d) in order to bridge neural processes across different scales relevant to neurons, cortical layers or columns, circuits, regions, networks, and systems. Color-coded arrows in the illustration of cortical layers indicate directed functional connectivity in the feedforward (red) and feedback (blue) directions.

One strategy is to combine low-resolution and high-resolution fMRI signals to bridge cortical regions to layers. Typical fMRI data with blood oxygen level dependent (BOLD) contrast are collected at 3 Tesla with >2mm isotropic resolution and ~1s intervals. Sub-millimeter fMRI with contrast sensitized to small vessels is attainable at ≥7 Tesla to resolve activity by layers within a 2-4mm cortical depth [31, 32]. Since its resolution and contrast are hard to attain at a sufficiently fast speed for the whole brain, acquisition of layer-specific fMRI signals is often restricted to a limited brain slab. Given the same naturalistic stimuli, the fMRI signals measured with different resolutions and fields of view are presumably reproducible across subjects and experiments. It is thus reasonable to correlate whole-brain fMRI signals with layer-specific fMRI signals, to resolve feedforward and feedback input that terminates at different cortical layers (Figure 2c) [33]. The correlations are expected to bridge layer-specific circuitry to large-scale networks and add directionality to otherwise undirected functional connectivity [34]. To measure correlations or other types of statistical dependence alike, brain signals are viewed as random variables.

Figure 2.

Spectral and laminar features of feedforward and feedback cortical processes. a) In a log-log scale, the power spectrum of a neural signal exhibits a broadband 1/f-like pattern plus narrowband components at different frequency bands (delta, theta, alpha, beta, and gamma). The narrowband and broadband components can be separated by using algorithms such as the Irregular Resampling Auto-Spectral Analysis (IRASA) [36]. Plausible interpretation of the separated spectral components is to take the broadband power as a surrogate of population spiking output from the recorded neurons, the narrowband power in a lower frequency band (e.g. alpha) as a surrogate of the feedback synaptic input to the neuronal population, the narrowband power in a high frequency band (e.g. gamma) as a surrogate of the feedforward input. b) The feedforward and feedback connections between two neuronal populations at different hierarchical levels synchronize their output-input signals. The spiking output at the higher level is synchronized with the feedback input at the lower level. The spiking output at the lower level is synchronized with the feedforward input at the higher level. c) The feedforward and feedback connections originate from and terminate at different cortical layers. The feedforward input terminates at the granular layer and the feedback input terminates at the supragranular and infragranular layers. Such distinctive laminar profiles are likely resolvable with high-resolution fMRI that report synaptic input into different cortical laminae within each cortical column.

This practice is more reasonable under naturalistic stimuli (rather than traditional tasks), which are inherently irregular and complex. Correlational patterns under naturalistic stimuli also appear to be more similar to the patterns spontaneously emerging in the resting state [35]. It is plausible that the intrinsic functional architecture is more functionally relevant to neural processes given naturalistic stimuli rather than non-naturalistic stimuli.

Such a strategy can be taken further to bridge multimodal signals observed with fMRI and electrophysiology [26, 27]. Unlike fMRI, electrophysiology (e.g., EEG, MEG, ECoG, LFP) captures a broader spectrum of neural dynamics and separates neural components by frequency: broadband scale-free activity and narrowband oscillations [36–38]. Different frequencies are associated with distinctive cortical mechanisms [39]. Increasing evidence suggests that fast oscillations in the gamma band (30-70Hz) reflect feedforward input, slow oscillations in the alpha (8-13Hz) or beta (13-30Hz) band reflect feedback input [40–42], and broadband activity reflects spiking activity [43] (Figure 2). The spectral signatures of directed network communications presumably correspond to distinct laminar profiles observed with layer fMRI [44, 45] (Figure 2c). Therefore, naturalistic stimuli set up a convenient condition for integrating distinct neural and anatomical features into a converging understanding of directed functional connectivity at the circuit level.

Relating electrophysiology to fMRI during observations of the same naturalistic stimuli can also help elucidate the mechanisms of neurovascular coupling and lay a stronger foundation for the interpretation of fMRI-observed activity and connectivity [26, 46]. Spectrally or temporally separable neural components all contribute to the fMRI signal [47, 48]. Understanding their differential contributions to the fMRI signal is critical to the interpretation of fMRI in terms of neuronal input vs. output, excitation vs. inhibition [49]. Whereas simultaneous acquisition of neural and fMRI signals is technically challenging and ethically constrained, the use of the same naturalistic stimuli makes it more feasible to compare neural and fMRI signals collected from different sessions and subjects, especially when neural signals are recorded invasively from human brains [46]. In this regard, Park and colleagues [27] addressed the relationship between single-unit activity within a voxel and the fMRI activity throughout the brain, attempting to bridge the perhaps largest gap in both spatial and temporal scales between electrophysiology and fMRI. Comparing or integrating data across different scales is not trivial. When brain signals are down to a finer spatial scale, it is arguably less likely to expect a one-to-one correspondence between units or voxels across different datasets. It is perhaps more reasonable to expect the correspondence in representation, a virtual readout of multi-variate pattern through either a linear or shallow nonlinear function that analytically resembles how a neuron reads out the pattern of its input neurons through synaptic connectivity. To define such a readout function, it is also necessary to separate data into independent sets for training, cross-validation, and testing to avoid circular analysis.

Naturalistic stimuli usually involve neural processing within and across multiple functional systems in the human brain (Figure 1b). For example, reading vs. listening to a story activates different unimodal sensory areas but engages the same transmodal areas selective to the linguistic content regardless of the sensory pathway used to convey it [50, 51]. Likewise, music imagery and perception activate a common set of cortical regions underlying the same mental processes even in the absence of bottom-up sensory input [19, 52]. With naturalistic paradigms, it is plausible to evaluate the time-locked synchrony of brain responses between different cortical locations or systems across different perceptual or cognitive conditions.

Bridging brain data to human behavior through computational models

Brain responses to naturalistic stimuli are often analyzed with non-parametric and model-free methods [9, 53]. Despite their simplicity, such analyses can only lead to observational findings. One can answer where in the brain are activated by the stimuli but cannot address how the stimuli are processed through the activated regions in order to arrive at the resulting perceptual or behavioral outcomes. An emerging approach that addresses these issues is to use deep neural networks to model neural information processing that bridges stimuli to behaviors and explains brain responses given naturalistic stimuli.

The past decade has witnessed great strides in deep neural networks for artificial intelligence [54], allowing machines to perform perceptual and cognitive tasks sometimes even better than humans do. Such networks are inspired by biological systems and are increasingly used as brain models [55, 56]. These models stack many layers of neurons into a hierarchical system are usually trained with a large number of natural images, speeches, videos, texts, etc. The trained models can unpack naturalistic stimuli down to hierarchical features and predict human-level behaviors.

For vision, the most commonly used neural networks are feed-forward only, passing information bottom-up and layer by layer until the representation at the top layer is readily usable to define a perceptually driven concept. For example, convolutional neural networks (CNN) have been used to model brain responses to naturalistic videos [10, 57] (Figure 3). From videos, CNNs extract features and organize them by layers at an increasing level of abstraction. The model-extracted hierarchical feature representations can explain the brain’s hierarchical cortical representations of the same videos [10, 58, 59] (Figure 3b). As a “digital mirror” of the brain, such models can be applied to decode fMRI scans and reconstruct the video input and/or describe its semantic content [10, 60, 61] (Figure 3d). These models are however not perfect and notably different from the brain, especially for their lack of a mechanism for feedback, lateral, or recurrent computation. As such, the off-the-shelf CNNs are unable to fully encode or decode brain activity. More biologically plausible models are emerging [62–65] to incorporate top-down prediction and recurrent computation. Promising directions are, for example, to build network models that include the ventral and dorsal visual streams, account for the selectivity and bias in retinal sampling, separate inference for foveal vs. peripheral input, predict eye movement, and yield time-varying percepts in degraded, multi-stable, or illusory conditions.

Figure 3.

Using deep neural networks to encode and decode fMRI signals from human subjects watching natural videos, a) A deep convolutional neural network (CNN) serves as a model of feedforward processing in the human visual system [10]. For brain encoding (blue arrows), the CNN extracts multiple layers of visual features from every video frame and a linear encoding model uses the feature representations to explain or predict the fMRI response at every cortical location. For brain decoding (purple arrows), the decoding model combines the fMRI responses across locations to reconstruct the low-level visual feature and categorize the semantic content in the video. b) The encoding models reveal the hierarchical organization of the visual cortex. Different layers of visual features differentially explain the fMRI responses at individual locations. The lower to higher layers in the CNN progressively map onto the striate to extrastriate cortex, c) Both the CNN and the visual cortex encode selective representations specific to human faces. The FFA in the brain (purple curve) and the ‘Face’ unit in the CNN (blue curve) exhibit the same response dynamics during the video, showing peak responses occurring when the video shows human faces, d) This figure shows the visual reconstruction from cortical responses through the decoder of a variational autoencoder model [60]. The top row shows the original video frames in the natural movie stimuli. The second row shows the visual reconstruction based on the decoding model trained and tested with the same subject’s brain. The third row shows the reconstruction performance, where the decoding model is trained on one subject but tested on a different subject. Images in this figure are modified from prior publications with permission.

Beyond vision, neural network models are also beginning to demonstrate their ability to explain cortical responses during natural language processing. For example, shallow or deep neural networks trained with a large corpus of texts are able to convert thousands of words into vectors embedded in a continuous feature space [66, 67]. This echoes the notion of a continuous semantic space that the brain encodes in order to organize or create concepts [18, 68]. Using word embedding models to explain fMRI responses during narrative story comprehension, findings from a recent study shed light on distributed cortical networks that not only store individual concepts but also infer relations between concepts [18]. Language models are also demonstrated to be able to decode sentence meanings from brain scans [69] or decode text from ECoG recordings [70].

Whereas models of neural computation mostly separate sensory perception and semantic cognition, the brain uses continuous neural processing to ground cognition in perception and action [71, 72]. This gap awaits to be filled with multimodal neural network models that first use lower layers to perform neural information processing for sensory and/or motor modalities and then fuse information at higher layers for conceptual abstraction. Such models are expected to be closer to how embodied cognition emerges in the brain.

Brain-inspired models alongside functional brain imaging or recording with naturalistic video and speech stimuli will advance artificial intelligence and neuroscience in synergy. On one hand, computational models are used to substantiate and test neuroscientific hypotheses against brain responses and human behaviors, and thus support progress in neuroscience. On the other hand, brain and behavioral data during naturalistic stimuli can test models for biological plausibility and thus advance artificial intelligence towards human intelligence. With the increasing popularity of naturalistic paradigms, the synergy between artificial intelligence and neuroscience is booming at an unprecedented speed.

Challenges and opportunities

A unique opportunity is to use a common set of naturalistic stimuli for collaborative efforts in functional imaging and recording across different modalities, subjects, experiments, and labs. The notion of big data has been increasingly embraced and leveraged in large-scale projects such as the Human Connectome Project [73]. The HCP has collected human fMRI data with natural audiovisual stimuli in addition to a much larger set of resting state and task fMRI data. There is a lack of large-scale efforts to extend the naturalistic paradigm to other spatial or temporal scales inaccessible with standardized fMRI protocols.

Of particular interest are intracranial neural recordings attainable from patients with electrodes implanted for clinical diagnosis or surgical planning. The electrode implantation requires invasive surgical procedures and is only justifiable by clinical needs. In each patient, the implanted electrodes often sample limited regions and are unable to cover the whole brain. To optimally utilize intracranial neural recordings, it is desirable to use naturalistic stimuli for high-throughput characterization of many functional domains. The use of the same naturalistic stimuli induces reproducible neural responses and allows for multiplicative combinations of the signals recorded at different brain locations from different patients (Figure 1).

Paradigms with naturalistic stimuli are well suited to assemble “big data” resources [74, 75] and promote interdisciplinary collaborations. Psychologists and neuroscientists design the stimuli to diversely and inclusively sample functional domains. Imaging scientists push the resolution to image the whole brain at a fast speed or selected regions for fine spatial details. Neurophysiologists record neural activity with implanted electrodes or non-invasive sensors.

Cognitive scientists evaluate perceptual, behavioral, emotional, and cognitive outcomes. Computational scientists design models of neural information processing to link stimuli to behaviors. Putting them together into a collaborative ecosystem, multidisciplinary efforts can be synergized to bridge brain activity across mesoscopic and macroscopic scales, bridge the gap between neural and vascular measurements, link neural signals to neural computation, and connect the brain to behavior. Such combinatorial power will be the unique premise for using naturalistic paradigms for experimental and computational neuroscience.

Challenges also come along with this opportunity. The neuroinformatic infrastructure and open science culture are rapidly growing [76], but remain inadequate to fully support multi-modal and multi-site data sharing and harmonization, as well as advanced analytical methods such as deep learning. Software for data standardization, analysis and integration awaits community-wide coordination and adoption to assure reproducibility and scalability. Although highly reliable and reproducible, brain responses to the same stimuli are not strictly identical across subjects. Individual variations are inevitable and of interest for basic science and clinical research. Initial progress has been made to characterize individual variation during naturalistic stimuli [77], while new techniques show the initial promise to align data, representations or models across individuals or datasets [78–80]. Despite these challenges, it is foreseeable that experimental paradigms with naturalistic stimuli are at the next frontier of imaging, recording, and modeling the brain in a realistic, complex, dynamic environment.