Reliability of cortical activity during natural stimulation

Abstract

Response reliability is complementary to more conventional measurements of response amplitudes, and can reveal phenomena that response amplitudes do not. Here we review studies that measured reliability of cortical activity within or between human subjects in response to naturalistic stimulation (e.g., free viewing of movies). Despite the seemingly uncontrolled nature of the task, some of these complex stimuli evoke highly reliable, selective, and time-locked activity in many brain areas, including some brain regions that often do not show much response modulation with conventional experimental protocols. This activity provides an opportunity to address novel questions concerning natural vision, temporal scale of processing, memory, and the neural basis of inter-group differences.

Reliability of neuronal responses

Neuronal responses are more reliable (reproducible) under naturalistic stimulus conditions than under conventional laboratory conditions using artificial stimuli. Mechler et al. [1] reported that the responses of neurons in the visual cortex were more reliable for moving edges (abundant in natural vision) than for drifting sinusoidal gratings (common only in psychophysical experiments). Yao et al. [2] measured neural responses to movies of natural scenes; response reliability increased with repeated presentation of the same movie (over ten trials, each trial lasting 31 seconds), but not with repeated stimulation by white noise or flashed bars. Belitski et al. [3] reported that both firing rates and gamma band (60–100 Hz) local field potentials (LFPs) were highly reliable in response to repeated presentations of movie clips.

Analogous with the results of these electrophysiological studies, although on a far coarser time scale, are the results of a number of studies that show that human brain activity can be highly reliable under naturalistic stimulus conditions [4–9]. These studies used blood oxygen level dependent (BOLD) functional magnetic resonance imaging (fMRI) to measure brain activity while subjects were engaged with realistic, complex stimuli (free viewing of movies or listening to audiobooks or music). The data were analyzed by comparing the evoked fMRI response time courses across different subjects (inter-subject correlation [4]; henceforth inter-SC), or by comparing response time courses evoked within the same subject by repeated presentations of the same stimulus (intra-subject correlation [5, 10]; henceforth intra-SC). See Supplementary Figure 1 for details of the data analysis. Despite the seemingly uncontrolled task (free viewing and/or listening) and the complex nature of the stimuli, some stimuli evoked highly reliable and time-locked activity in many brain areas [4, 10] (see Figures 1–2 for some illustrative examples).

Figure 1.

Reliability within and between subjects

Upper panels: Responses evoked by the Sergio Leone movie, from early auditory cortex (A1+, see Glossary) in each of four subjects. Each subject watched the movie twice. The two curves in each panel correspond to responses evoked within each subject (S1, S2, S3, and S4) by the repeated presentations. Black numbers: Correlation coefficient between the two response time courses within each subject (intra-SC). Red numbers: Correlation coefficients across subjects (inter-SC). Lower panel: Response time courses for each subject, averaged across the two repetitions, and grand mean averaged across the four subjects (black curve). The results shown here for the Sergio Leone movie (and in subsequent figures) are from a replication of the originally published data set [4]; we repeated the experiment in order to have a direct comparison between this and the other films (same fMRI scanner and data acquisition protocol) for Figure 2.

Figure 2.

Reliability of responses to different movies

Medial and lateral views of “inflated” right hemisphere depicting the inter-SC maps for four different movies. Posterior areas (P) are toward the middle of each panel, while anterior areas (A) are facing the sides. The fMRI measurements for all four movies were acquired with the same equipment and procedures. In addition, to have comparable statistical power, we matched the number of subjects (n=8), and the length of the time courses of all stimuli by extracting seven minutes of fMRI measurements from each of them. Inter-SC was computed by first splitting the eight subjects into two groups, averaging the response time courses separately for each group, and then computing the correlation coefficients between the two resulting response time courses at each cortical location. Colored regions represent locations for which the correlations exceeded a threshold value (0.25, p<0.002; chosen because it was above the highest inter-SC value exhibited by any voxel for two different movies). The Hitchcock episode and the Leone and Chaplin movies evoked far greater and more extensive inter-SC than the real-life, unedited video. The Hitchcock episode evoked more reliable responses in prefrontal cortex than the other three stimuli.

In what follows, we first review the evidence that some naturalistic stimuli (e.g., movies, TV shows, and audiobooks) evoke highly reliable, time-locked, and functionally selective response time courses throughout the brain, even in brain regions that often do not show much response modulation with conventional experimental protocols (see Box 1 for a brief overview of a broader set of measurements and hypotheses related to the use of naturalistic stimuli). Next, we review evidence for dissociation between response amplitudes and response reliability. Finally, we summarize some examples of how measurements of response reliability during naturalistic stimulus conditions have been (and might be) applied to characterize brain responses during natural vision (see [11] for a discussion of the implications of these results for filmmakers).

Response reliability and response selectivity

An initial demonstration of response reliability to natural stimuli measured human brain activity during free viewing of a segment from The Good, the Bad and the Ugly, a well-known film by Sergio Leone (1966) [4]. Activity in the early auditory cortex serves as an illustrative example of the results (Figure 1). The responses in the auditory cortex (A1+, see Glossary for abbreviations of anatomically and functionally defined brain areas) of each subject were similar during repeated presentations of the same movie excerpt (Figure 1, high intra-SC, black numbers), indicating that the stimulus induced reliable responses in this brain region within each individual. The responses were similar not only within each individual subject, but also across subjects (Figure 1, high inter-SC, red numbers). In other words, the movie exerted considerable “control” over the response time courses in this brain area, evoking similar responses within and between subjects. In addition to A1+, about 45% of the cortex showed high (and statistically significant) intra-SC and inter-SC [4, 10] during movie watching, including auditory areas in the temporal lobe, visual areas in the occipital lobes, and multisensory and language areas in the temporal and parietal lobes (Figure 2B).

Not all stimuli are equally effective in evoking reliable responses in different areas of the brain (Figure 2). First, there was no evidence for inter-SC or intra-SC in response to different movie segments [5], verifying that the activation is time locked to the content of the movie. Second, unlike commercially produced videos, a real-life, unedited video of a concert, taken from a fixed single viewpoint, induced high inter-SC in only a small fraction of the cortex (less than 5%), mainly in early visual and auditory cortical areas (Figure 2D). The low inter-SC for this real-life movie clip indicates that not all natural stimuli have the capacity to induce reliable responses throughout the brain. Likewise, even within the primary visual cortex, neurons appear to respond more reliably to continuous movie clips [2, 3] than to flashed natural images [12], although a direct comparison between the two types of stimuli has yet to be published.

The level of reliability in any particular brain area varies from low to high depending on the content of the stimuli. For example, a segment from Charlie Chaplin’s City Lights (1931) did not evoke reliable responses in early auditory cortex because there was no sound track (Figure 2C, see also [5]). The use of the TV episode “Bang! You’re Dead,” directed by Alfred Hitchcock (1961) [11], evoked the most widespread inter-SC (see Figure 2A), including large regions of the lateral and medial prefrontal cortex and some of the so-called “default mode” or “intrinsic” brain areas, not evident during viewing of the other movies (compare Figure 2A and Figure 2B–C). Similar results were obtained for the Academy Award–winning (best picture) movie, Crash, directed by Paul Haggis (2004) [7].

These reliable responses, although widespread, are nonetheless selective in that they differ from one brain area to another. Figure 3A plots the response time courses evoked by the Hitchcock episode in four brain regions. Figure 3B (see also Supplementary Figure 3) presents a matrix of inter-region correlations for 15 arbitrarily selected brain regions, including visual cortical areas, an auditory area, and areas of temporal, parietal, and prefrontal cortex. Rresponses differed across brain regions during movie watching. For example, A1+ was positively correlated with some temporal and frontal areas, but uncorrelated or negatively correlated with occipital and parietal areas. pDLPFC exhibited a completely different profile of correlations, showing some positive correlations with occipital and parietal areas and negative correlations with temporal regions.

Figure 3.

Selectivity of responses in different brain regions

(A) Responses evoked by the Hitchcock episode, from each of four distinct brain regions (see Glossary for abbreviations). Upper panels: Response time courses averaged separately across each brain region and separately for each of two groups of subjects (blue, group 1, n=12; red, group 2, n=11). Black numbers: Correlation coefficients between the two response time courses within each brain region, across the two groups of subjects. Red numbers: Correlation coefficients across brain regions. Lower panel: Mean response time courses for each brain region, averaged across all 23 subjects. (B) Inter-region correlations and within-region inter-subject correlations for 15 example brain areas. Upper panel: Matrix of inter-region correlations computed by first averaging across all 23 subjects. Lower panel: Within-region inter-subject correlations computed by first averaging the response time courses separately for the two groups of subjects, and then computing the correlation coefficients between the two resulting response time courses within each brain region. Color indicates the strength of the correlation coefficients. Although the response time courses within a specific brain region are highly reliable across viewers, they are unique and distinctive across brain areas.

Response correlations (also termed functional connectivity) have been used to infer functional interactions between brain regions [13–18]. Inter-region correlations, however, are used here for a different purpose: to demonstrate that the response time course within a specific brain region is unique and distinctive, yet highly reliable across viewers. This selectivity of response demonstrates that the bulk of inter- and intra-SC cannot simply be attributed to non-specific, spatially global effects like arousal (see also Supplementary Figure 3). Similar selectivity to movie stimuli has been demonstrated with intra-cranial EEG recordings [19].

Response reliability and selectivity in these experiments no doubt depended on the unique functional specialization of each brain area. Indeed, response amplitudes evoked by naturalistic stimulation have been used to characterize the functional specialization of some brain areas [4, 7, 8, 20–24].

Response reliability versus response amplitude

The reliability of cortical activity can reveal phenomena that response amplitudes do not. To demonstrate this, eye movements and brain activity were measured simultaneously while subjects viewed movies (without sound tracks) played forward and backward in time [5]. The eye movements were highly reliable across viewers and very similar across repeated presentations of the same movie for both the forward and backward presentations, verifying that the level of engagement was comparable across the two conditions. The brain activity in the visual cortex was, likewise, highly reliable for both the forward and backward films. In some other cortical areas (e.g., precuneus, LS, TPJ, and FEF), however, brain activity was much less reliable during the backward presentations than during the forward. In contrast, disrupting temporal order had no effect on response amplitudes, even in brain areas in which response reliability was markedly different, establishing a clear dissociation between these two measurements. The response amplitudes for the forward and backward movies were indistinguishable from each other in all of the brain areas, even those in which response reliability exhibited a dependence on temporal order. Similarly, the power spectra for both the forward and backward movies were indistinguishable, demonstrating that observed differences in reliability across regions were not the result of a decrease in the response amplitudes in any frequency band.

The low response reliability to backward movies in some brain areas indicates a failure to “lock in” to a consistent sequence of neural states (and corresponding cognitive states) while viewing the temporally disrupted stimuli. Indeed, the low response reliability correlated with high variability in viewers’ comprehension of the backward movies, as measured by a questionnaire administered to viewers in a control experiment [5]. At the same time, the strong response amplitudes in these brain regions were hypothesized to reflect incessant processing, aimed at extracting meaningful information from the movie stimuli. Neurons apparently exhibited large amplitude response modulations while processing the backward movies, but the responses were unreliable because the “interpretation” of the stimulus was different across subjects and from one viewing to the next.

The dissociation between response reliability and response amplitudes can be used to reveal phenomena that response amplitudes alone do not reveal, a number of examples of which are summarized in the following sections.

Hierarchy of temporal receptive windows in the human cortex

It is well established that neurons along the visual cortical pathways have increasingly larger spatial receptive fields [25]. This is a basic organizing principle of the visual system; neurons in high-level visual areas receive input from many other neurons, in early visual areas, that have smaller receptive fields, thereby accumulating information over space. Real-world events occur not only over extended regions of space, but also over extended periods of time. It was hypothesized that a hierarchy analogous to that found in spatial receptive field sizes might also exist for the temporal response characteristics of different brain regions [5]. Specifically, if the temporal receptive window (TRW) of a neuron is defined as the length of time prior to a response during which sensory information may affect that response, it was predicted that there would be a hierarchy of increasing TRWs as one moved from low-level (sensory) to higher-level (perceptual and cognitive) brain areas.

This prediction was tested by fMRI measurements of cortical activity in response to manipulation of the temporal structure of silent films by Charlie Chaplin and Buster Keaton [5]. The temporal order of each movie sequence was randomly shuffled at each of three different time scales: short (4±1 sec), intermediate (12±3 sec), and long (36±4 sec). Each of the shuffled films was presented twice, and the reproducibility of the responses was measured across repeated presentations, separately for each of the three time scales. A complementary experiment used time-reversal (i.e., showing the movie backward) to disrupt temporal order and assess TRWs (see above, Response reliability versus response amplitude).

The results revealed that the reliability of responses varied systematically across different brain regions as a function of the temporal structure (Figure 4). Response reliability in early visual areas was not affected by manipulating the temporal structure of the movie, indicating that neurons in those areas have short TRWs. On the other hand, response reliability in several higher brain areas depended on sensory information accumulated over longer time scales, revealing a hierarchy of TRWs spanning from short (~4 s) to intermediate (~12 s) and long (~36 s). These results supported the hypothesis that there is a hierarchy of progressively longer TRWs in the human brain. The order of the exact same set of events within a temporal sequence can have a strong effect on the responses in brain areas that accumulate information over time, but have little effect on areas with short TRWs. The short TRWs observed in the early visual cortex support the notion that these brain areas are optimized for rapidly processing the instantaneous visuospatial properties of a stimulus. Many cognitive processes (e.g., verbal communication, prediction, event segmentation, theory of mind, etc.), however, require accumulation of information over time. Delineating brain areas with long TRWs is a necessary first step toward localizing such processes. Moreover, the inter-SC analysis takes into account the entire time-dependent response profile, not just the magnitude of response modulation, and thus can be used to better characterize the modulations of the response time courses over long time scales.

Figure 4.

Response reliability and temporal receptive windows (adapted from ref. [5])

(A) Response correlation across repeated presentations, as a function of temporal disruption of the same movie, in each of several brain regions. Black bars: Response correlations for the unshuffled original movies that had the most coherent temporal structures. Red, green and blue: Response correlations for movies that were shuffled at long (36±4 seconds), intermediate (12±3 seconds), and short (4±1 seconds) time scales, respectively. Asterisks indicate that the response correlations were significantly smaller (p<0.05, one-tailed t-test after applying Fisher transformation to normalize the distribution of the r values) than those evoked by the unshuffled original version. Early visual areas (V1, MT+, see Glossary) exhibited no difference across conditions. LO, FFA, PPA, STS, and precuneus exhibited smaller correlation values when the films were shuffled at a short time scale. LS, TPJ, and FEF responses were reproducible only for the longest time scales. (B) Map of brain regions with different temporal receptive windows. Blue: Brain regions in which the response correlations were high for all shuffled movies (at long, intermediate, and short time scales). Green: Regions in which the response correlations were high only for the long and intermediate time scales, but not when the shots were shuffled at a short time scale (e.g., LO, PPA, FFA, STS). Red: Regions in which the response correlations were high only for the longest time scales (e.g., LS, TPJ, and FEF).

Response reliability, neuronal spiking, and fMRI in the human auditory cortex

Naturalistic stimulation, because it evokes highly reliable responses, can be used to relate different research techniques. For example, inter- and intra-SC were used to compare electrophysiological responses recorded from two epilepsy patients with the fMRI responses obtained from eleven healthy subjects who watched the same segment of the Leone movie [26]. The firing rates of 53 single neurons in Heschl’s gyrus (auditory cortex) were recorded with intracranial depth electrodes (implanted for the purpose of presurgical planning). Twenty out of 30 neurons in Patient 1, and 17 out of 23 in Patient 2, showed reproducible responses across repeated presentations of the movie (intra-SC). The spiking activity of these neurons in each patient was summed and converted into a predicted fMRI response time course by adopting a linear systems (convolution) model of the hemodynamics. The predicted fMRI responses derived from single units were strongly correlated (r=0.75, p≈0) with fMRI measurements from the healthy subjects. The broad methodological implication of these findings lies in the demonstration that, at least under natural stimulus conditions, human fMRI responses can be trusted as a faithful measure of the average firing rates of the underlying neuronal population. By capitalizing on the reliability of responses to the movie soundtrack in auditory cortex (high inter-SC, see Figure 1), it was not necessary to acquire both types of measurements from the same human subject to make this comparison.

Encoding of real-world events into memory

Movies are encoded into memory even when subjects are not explicitly instructed to do so, and can be retained in memory for many months [27]. This tendency was used to study the neural bases of episodic encoding of realistic events. Memory for the narrative content of a TV episode was assessed three weeks after subjects watched it in the scanner [6]. To increase the ecological validity of the study, participants were not informed of a pending memory questionnaire and were not asked to explicitly memorize the episode’s content. The subsequent memory results of each subject were used in combination with the inter-SC analysis to reveal brain regions in which the inter-SC is greater during successful, as compared to unsuccessful, memory formation. The analysis revealed a set of brain areas whose response time courses were significantly more correlated across subjects during the portions of the movie that were successfully encoded into memory. These regions included the parahippocampal gyrus; STS, anterior temporal poles, and the TPJ.

Inter-group differences in brain activity across clinical groups

Calculating inter-SC with typical healthy subjects might be used as a benchmark for detecting abnormalities in brain function in various clinical groups. The idea is simple: brain activity in any given clinical group (e.g., schizophrenia, autism, depression, anxiety, etc.) is hypothesized to be manifest in dysfunctional responses that deviate from the normal range, in a manner that is unique to each disorder. By comparing the responses in each brain area within a given clinical group to a normal response profile measured from matched healthy subjects, it may be possible to detect brain responses that deviate from the norm. A number outside the normal range for any particular brain area, somewhat akin to thresholds set for a blood test, may provide a unique functional marker, relevant for diagnosis and for evaluating the efficacy of intervention.

As an initial step in testing the capacity of inter-SC and intra-SC to detect inter-group differences, activity in autistic adults and healthy neurotypical adults (matched for age and gender) was compared during free viewing of the Leone move [28]. Cortical activity was much less reliable in individuals with autism than in typical individuals (low inter-SC). When the responses within an autistic individual were measured across repeated presentations of the movie, idiosyncratic responses that were reliably replicated within each individual (high intra-SC) were found. After attenuating the idiosyncratic responses from each individual time course, by averaging the time courses across all autistic individuals, a more typical response profile was uncovered which resembled that seen in typical subjects. These findings indicate that the neural activity of individuals with autism is characterized by idiosyncratic responses that, although reliable within an autistic individual, are both highly variable across autistic individuals and different from the responses observed within typical subjects. These findings may pave the way to future research aimed at characterizing the idiosyncratic response profiles, which, in turn, might contribute to a better understanding of the autism spectrum and its diagnosis. This was the first study to adopt such an approach for investigating cortical differences in a clinical population, but this approach presents a potentially valuable non-invasive protocol for characterizing altered brain responses associated with a given neurological disorder, mental illness, or developmental disability.

Response reliability in the “default mode”/“intrinsic system” of brain areas

The “default mode” [29–32] brain areas, also termed the “intrinsic system” [10], exhibit decreases in activity during external stimulation relative to rest. There has been much speculation about the possible functions of the “default mode” [30, 33–35]. Previous studies reported that these brain areas responded not only with low response amplitudes (below baseline) but also unreliably during free viewing of movies, as compared to sensory-motor areas [4, 10]. This response profile has led to the suggestion that the “default mode” brain areas specialize in intrinsically oriented functions [10]. Despite the lower reliability observed in the “default mode”/“intrinsic system,” three studies have nonetheless succeeded in revealing some reliable responses to naturalistic stimulation in several “default mode” brain areas. Wilson et al. [8] found high inter-SC during narrative speech comprehension in aCing, pCing, mPFC, and precuneus. Similarly, the Hitchcock TV episode [11], as well as Haggis’s movie [7], evoked reliable responses in the TPJ, aCing, OFC, and mPFC (Figure 2A and Supplementary Figure 2), albeit of less reliability of the evoked responses when compared to sensory-motor “extrinsic areas” [10]. Measurements of response reliability with naturalistic stimuli may, therefore, provide a tool, complementary to the resting state protocol, for characterizing functional properties of the “default mode”/“intrinsic system”.

Conclusion and limitations

This paper reviews a series of studies that exploited inter- and intra-subject correlation to characterize the reliability of cortical response time courses across individuals. These studies found that under natural viewing conditions a large portion of the cortex evinces reliable, and selective, responses that are shared across all viewers [4–10]. In contrast to the shared responses, some brain responses were reliable across repeated presentations only within a particular individual (Supplementary Figure 4) or within a well-defined group of subjects [28]. Thus, the inter-SC and intra-SC analysis methods can be used as a “social-neuroscience” tool to dissociate between neuronal processes that are shared by all people, those that are unique to a given sub-group, and those that are idiosyncratic to an individual.

The use of natural stimuli exemplifies a growing trend in neuroscience (see Box 1) toward studying the human brain in more realistic and natural settings [8, 21, 23, 24, 36–48]. As with any approach, however, inter- and intra-SC analysis of fMRI responses has limitations. There are some technical limitations, such as the limited precision with which different brains can be functionally aligned for computing inter-SC. However, the techniques continue to improve. For example, inter-SC was substantially increased by aligning the functional neuroanatomy of individual brains based on shared patterns of neural activity elicited during movie watching [49]. So any conceptual limitations to the approach are more important than the technical limitations. Conventional (simplified and highly controlled) protocols may suffer from lack of ecological validity. Measuring response reliability to naturalistic stimuli does not suffer from a lack of experimental control; indeed, movies exert considerable control over the responses in many brain areas. This approach does suffer, nonetheless, from a different conceptual shortcoming. Due to the complexity and multidimensionality of the stimuli, it is difficult to disentangle the intervening variables that drive the reliable responses in a particular brain area. Manipulating the stimuli [5], modeling the stimuli [20], using reverse correlation [4], or combining inter-SC with behavioral protocols [6] are all steps toward overcoming this limitation. Regardless, this approach offers an opportunity to reveal aspects of brain function and dysfunction that, once discovered, can be characterized parametrically using more conventional protocols.

Box 1: Naturalistic stimulation and free viewing

Measuring response reliability during movie watching is part of a growing trend in neuroscience toward looking at brain responses to natural stimuli [24, 37, 39, 42, 43]. There are a few interconnected theoretical and methodological issues driving this trend.

Can neuronal responses recorded during natural vision be explained as a combination of responses to simplified stimuli? Some aspects of the response properties recorded under rudimentary conditions can be generalized to real-world situations [4, 20, 40, 50, 51]. For example, the responses of visual neurons in the lateral geniculate nucleus were measured with both simple, artificial stimuli and complex, naturalistic stimuli; a functional model that fit the firing rate responses to simple stimuli, with the same parameters, predicted the bulk of the firing rate responses to complex stimuli [51]. Similarly, in the olfactory cortex responses to a complex natural odor were explained as the sum of the responses to its individual, well-characterized odor components [40].

Some studies, however, report the opposite [52–55]. For example, several studies reported that the tuning curves of neurons in the primary visual cortex, defined using single-orientation stimuli, were different when additional non-preferred orientations were presented in the visual field in conjunction with the preferred stimuli [54, 55]. However, it was later shown that models developed from conventional measurements with sinusoidal grating stimuli could be extended by adding spatially- and temporally-tuned inhibition, to predict neuronal responses during natural vision [56, 57]. Consequently, it has been argued that the best use of naturalistic stimuli is to test the predictions of functional models developed on the basis of artificial stimuli [58].

Is spike-timing more precise during natural stimulation?One specific hypothesis about how responses to natural and artificial stimuli might differ is that the precision of spike-timing evoked with natural stimuli might be higher than that evoked with artificial stimuli [1, 2, 42, 59, 60]. For example, injection of pseudo-random current, which resembled the synaptic activity that would be expected in vivo, produced spike trains with precise timing, reproducible to less than 1 ms. Constant current injections, on the other hand, evoked spike trains that were imprecise and highly variable across repeated trials [60]. In another experiment, a short movie was shown repeatedly while recording from visual neurons in the lateral geniculate nucleus of anesthetized cats. The measured firing rates were exactly zero much of the time and, when the neurons did fire, the probability of firing was often very high [59]. These and other studies indicate that the timing of neural responses might be more precise with “natural” stimulation, even when matched for overall firing rate. The question of whether the brain uses such temporal precision as a means to convey information is still under debate.

Are natural stimuli encoded more efficiently than artificial stimuli?Another specific hypothesis about the difference between natural and artificial stimuli is that neurons might encode natural stimuli more efficiently by adapting, on evolutionary, developmental, and behavioral timescales, to the statistical properties of the environment inhabited by the organism [43, 44, 61–64]. Specifically, there is evidence that neural responses evoked with natural stimuli are more sparse and/or statistically independent than those evoked with artificial stimuli [37, 47, 59, 65, 66].

Do the functional properties of neurons in the brain match the statistical properties of natural scenes?To address this question it is first necessary to characterize the spatial and temporal statistical regularities of the natural environment, including the statistical properties of basic elements such as luminance, contrast, color, and contour elements. These statistical properties are then used to model various neuronal processes associated with, for example, orientation and spatial frequency content of natural images [67], fixation selection [68], contour grouping [69–71], motion estimation [64], distance estimation [72], and the acoustic structure of natural sounds and speech [45]. These links between perception and natural stimulus statistics suggest that the characteristics of neural responses and coding can best be understood by probing with naturalistic stimuli.

Some complex behaviors can be expressed only within real-life natural contexts.In this review we emphasize another, complementary reason for using naturalistic stimuli. Empirical research in psychology and neuroscience has largely attempted to achieve, via abstraction and simplification, maximal control over as many variables as possible, while isolating or randomizing any other intervening or potentially confounding factors. For example, many experiments reduce spatial and temporal complexity, presenting brief (e.g., less than 500 ms), rudimentary visual (e.g., Gabor patches) or auditory (e.g., pure tones) stimuli. This approach has obvious advantages and has served us well. These conventional experimental protocols, however, lack the complexity that makes up real life. For example, many of our daily cognitive processes (e.g., reading manuscripts, engaging in dialogues, interacting socially, or seeing movies) unfold only over relatively long time scales [5]. Thus, restricting the experiments to brief time intervals undoubtedly restricts and narrows the scope of phenomena to be studied. We argue in this review that some types of naturalistic stimulation can result in highly reliable, selective, and time-locked activity in many brain areas, thereby providing a high level of experimental control while embracing the complexity of naturalistic stimuli.

Glossary

Abbreviations for each of a number of brain areas, defined either functionally or anatomically. Talairach coordinates, where reported, correspond to the center of an 8 mm box used to define a region of interest for the analyses reported in Figure 3 and Supplementary Figure 3. References, where listed, provide a detailed description of how a region of interest was defined for Figure 3 and Supplementary Figure 3.

A1+

approximate location of primary auditory cortex, a region in Heschl’s gyrus where the responses to a forward soundtrack were highly correlated with the time-reversed responses to the backward soundtrack

aCing

anterior cingulate. Talairach coordinates: x=−1, y=22, z=31

aDLPFC

anterior dorsolateral prefrontal cortex. Talairach coordinates: x=40, y=34, z=18

CS

central sulcus

FEF

frontal eye field, an area near the junction of the precentral sulcus and the superior frontal sulcus that responds more during saccades than during fixation [73]

FFA

fusiform face area, an area in the vicinity of the fusiform gyrus that responds more to faces than buildings or objects [74]

IPS

intraparietal sulcus

LO

subregion of lateral occipital cortex that responds more to objects than buildings or faces [74]

LOFA

lateral occipital face area, an area in the vicinity of the inferior temporal sulcus that responds more to faces than buildings or objects [74]

LS

lateral sulcus

mPFC

medial prefrontal cortex. Talairach coordinates: x=−2, y=45, z=30

MT+

MT complex, an area in the vicinity of the dorsal/posterior limb of the inferior temporal sulcus that responds more to visual motion than stationary stimuli [74]

OFC

orbito-frontal cortex. Talairach coordinates: x=−1, y=51, z=4

pDLPFC

posterior dorsolateral prefrontal cortex. Talairach coordinates: x=44, y=127, z=96

PPA

parahippocampal place area, an area in the vicinity of the collateral sulcus that responds more to buildings than to objects or faces [74]

PCS

subregion of post-central sulcus that responds when observing manual manipulation of objects

pSTS

posterior superior temporal sulcus

STS

superior temporal sulcus

STS-face

subregion of superior temporal sulcus that responds more to faces than buildings or objects [74]

TOS

subregion of transverse occipital sulcus that responds more to buildings than objects or faces [74]

TP

temporal pole. Talairach coordinates: x=47, y=8, z=−12

TPJ

temporal-parietal junction

V1+

approximate location of primary visual cortex, defined as a region in the Calcarine sulcus that responds to visual stimulation