Human Neurophysiology: Sampling the Perceptual World

Abstract

As we move our gaze through a complex scene, the retinal image is constantly shifted and overwritten. A new study using human intracranial recordings offers a fresh perspective on how the brain creates a sense of perceptual continuity through natural visual behavior.

In Brief:

As we move our gaze through a complex scene, the retinal image is constantly shifted and overwritten. A new study using human intracranial recordings offers a fresh perspective on how the brain creates a sense of perceptual continuity through natural visual behavior.

As we explore the world around us, our brain enlists its subordinate sensory organs to gather information about the scene. In humans, directing eye gaze is particularly important, because our impressions of objects, spatial relations and social information are largely determined by vision. Sensory input is by nature imperfect and fragmented. Thus, a major task of the brain is to build a plausible perceptual scenario from incomplete sensory details, a process Helmholtz called ‘unconscious inference’ [1]. In the case of vision, our restless gaze adds another twist: each fraction of a second, a saccadic eye movement abruptly shifts our focus, dragging the image across the retina and overwriting the previous pattern of photoreceptor stimulation (Figure 1A). Nonetheless, our brain protects us from these raw sensory events, and we perceive a seamless world across space and time, with only a vague awareness that our eyes are always moving. Since the time of Helmholtz, researchers have puzzled over how the brain is able to ignore, or perhaps compensate for, the image disruptions caused by eye movements (for a recent review, see [2]). In this issue of Current Biology, Podvalny et al. [3] approach natural gaze behavior from a new angle, recording its impact on neural activity in the visual object pathway during a social exchange.

Figure 1. Human visual experience is composed sequences of brief fixations directed to points of interest in a scene.

(A) The brain directs the eyes across a scene in order to acquire detailed information in important locations. With each new fixation, the retinal image is overwritten, yet the brain integrates the snapshots into a stable perceptual framework. (B) In the Podvalny et al. [3] study, neural responses during natural viewing differed fundamentally between retinotopic and face-selective visual cortex. In retinotopic visual cortex (left), responses during each fixation showed an initial suppression and then a transient excitatory response that was highly dependent on the gaze sampling dynamics. By contrast, sites in face-selective visual cortex (right) were marked by unique and characteristic temporal response profiles that were invariant to details of each fixation.

How should one think about perceptual continuity in vision? Historically, researchers have focused on the spatial problem of saccadic displacement and the brain’s capacity to maintain perceptual stability despite large image shifts on the retina. These studies have typically employed tightly controlled paradigms, for example in which a trained monkey or human is cued to make saccades between bright dots on a dark background. Psychophysical and electrophysiological findings suggest that the brain counteracts such retinal image displacement through several tricks. For one, it keeps track of its own oculomotor actions and sends this information to perceptual centers via a ‘corollary discharge’ signal [2]. Based in part on such signals, neurons in certain cortical areas then appear to anticipate the sensory consequences of each saccade through an active remapping of visual space [4]. Further, the visual brain is able to use landmark stimuli in order to establish spatial correspondences before and after each eye movement [5]. As these tricks pertain to space and action, they fit with the idea that perceptual stability is a problem of the dorsal visual stream. Related questions, however, such as how the brain maintains a persistent representation of objects across fixations, may not reduce to a spatial problem. In their study, Podvalny et al. [3] focus on object-selective responses in the ventral visual pathway and the extent to which they are shaped by spontaneous eye movements.

While a handful of previous studies [6–10] in monkeys and humans have investigated the interaction between gaze behavior and neural responses to complex objects, most work in this area has aimed to minimize the contribution of eye movements by flashing images briefly during periods of steady gaze fixation. The natural viewing approach applied by Podvalny et al. [3] goes beyond previous free-viewing studies by allowing subjects to turn their head, move their eyes, and engage in natural social exchanges. The subjects were patients whose brains were covered with dozens of subdural electrocorticogram (ECoG) electrodes that had been temporarily implanted to monitor epileptic activity. The patients additionally wore two types of head-mounted camera — one to record the contents of the scene in front of them, and the other to track their eye position within the scene. As is common in ECoG studies, the authors took electrophysiological power in the high frequency gamma range as an instantaneous measure of local neural activity.

This novel combination of methods allowed Podvalny et al. [3] to track neural activity at sites across the brain during a natural conversation, and at the same time determine precisely the evolving image content on the retina. With this, they investigated the extent to which self-generated gaze behavior shaped visual responses in two broadly defined cortical areas: early retinotopic cortex (Broadmann’s areas 17 and 18), and later object-selective cortex, including regions selective for faces. In analysing the data, they concentrated on periods of stable fixation, rather than the saccades themselves, framing their question as one of visual sampling rather than responses to eye movements per se.

The most salient finding was that gaze behavior had very different effects on neural responses in the early and late cortical areas. In the early visual areas, details of the gaze and low-level stimulus properties strongly influenced responses. Once the eye had landed, responses were strongly shaped by the gaze dwell time, or fixation duration (Figure 1B, left). The main excitatory response depended strongly on the luminance contrast between the pre- and post-saccadic image content. In addition to this main excitatory response, there was an initial dip in activity that appeared to begin even while the eye was in motion — a neural response feature that may relate to the eye movement itself. This dip was absent in control experiments in which the subject was shown flashed stimuli in a more conventional manner.

Further along in the ventral visual pathway, responses were very different. Podvalny et al. [3] focused mainly on ventral cortical areas known to be specialized in the processing of socially relevant stimuli such as faces [11,12]. Unlike in the early visual cortex, the fixation parameters in object-selective cortex did not affect the structure of neural responses. Individual sites exhibited stereotypical patterns of neural responses that, while selective for complex object content, were uncoupled from the details of each fixation: brief and long fixations led to the same response profiles (Figure 1B, right). Responses here did not depend strongly on low-level image contrast, and did not display an initial dip in their fixation response. Instead, the responses appeared to be governed by an internal dynamic that was invariant to the fixation sampling process itself.

These findings highlight a characteristic of high-level visual responses in the human brain that was previously unknown, and were only revealed during natural visual behavior. Evidently, along the ventral visual pathway there is a transition in which object selective neural responses gain invariance to the details of oculomotor sampling. One interpretation of this finding, supported by the authors, is that this transition may be an important step for achieving perceptual stability and continuity of complex visual information. This is a fascinating prospect, and one that calls for more investigation through future experiments. The findings also raise several new questions. For example, what internal principles govern the stereotypical temporal dynamics observed in object selective cortex, and why would different sites exhibit such distinct temporal profiles? It would be of great interest to learn whether such invariant dynamics are present among individual neurons, or perhaps in local populations of interacting neurons. As the present study [3] did not break down the fixation content in great detail, in part because of the challenges and limitations of recordings in human patients, more work is needed to understand whether these pulses are associated explicitly with foveation, whether their strength is affected by the recent fixation history, and how broadly the principles extend to stimuli beyond faces. These issues are important for understanding the extent to which the invariant response profiles observed in the present study might serve as a basis to support the continued perception of visual objects more generally.

In many ways, this study [3] is on the leading edge of a recent trend in systems neuroscience to complement traditional stimulus presentation with more naturalistic paradigms [10,13,14]. The obvious risk in taking such a step is the loss of control over quantifiable stimulus parameters. But temporary relief from tight testing paradigms can also be liberating. Experimentally, the brain is comfortable, if not eager, to step into its more natural set of conditions, and corresponding neurophysiological data are often straightforward to acquire. Importantly, recent work has shown that under the right conditions neural signals can show a high degree of reliability when faced with complex stimuli during free viewing [10,15], with naturalistic paradigms recently inviting a range of new neuroscientific questions [16–18]. The new study [3] is in that category, and the results may be an important step for understanding how the brain processes complex sensory information in the context of natural actions, all the while maintaining the continuous sense of a stable and meaningful world.