Saccades and shifting receptive fields: anticipating consequences or selecting targets?

Abstract

Saccadic eye movements cause frequent and substantial displacements of the retinal image, but those displacements go unnoticed. It has been widely assumed that this perceived stability emerges from the shifting of visual receptive fields from their current, presaccadic locations to their future, postsaccadic locations in anticipation of the retinal consequences of saccades. Although evidence consistent with this anticipatory remapping has accumulated over the years, more recent work suggests an alternative view. In this opinion article, we examine the evidence of presaccadic receptive field shifts and their relationship to the perceptual changes that accompany saccades. We argue that both reflect the selection of targets for saccades rather than the anticipation of a displaced retinal image.

Predictive remapping in nonhuman primates

Humans and other primates constantly redirect their gaze in order to scan their environment. This behavior is necessary to overcome the lack of high acuity vision in the visual periphery, and is largely achieved via saccades (see Glossary). Saccades ultimately lead to the foveation of important visual stimuli, and thus allow the brain to process fine spatial details contained within those targets. However, saccades not only lead to fast sweeps of the retinal image (motion), but also introduce substantial differences betweenthe presaccadic and postsaccadic retinal images (displacement)[1] (Figure 1). However, both disruptions go unnoticed, and instead we perceive the world as stable. This perceptual stability is entirely an illusion, and it is one that has puzzled scientists at least since the time of Helmholtz in the 19th century [2].

Figure 1.

Saccades, visualstability, and predictive remapping. (A) Illustration of a sequence of three saccades within a visual scene. (For simplicity, only one eye is depicted.) (B) The retinal images of the three fixation periods shown together with the swift sweep of the retinal projection during saccades. Note, retinal images have been reflected horizontally and vertically. (C) Schematic display of predictive remapping. Three receptive fields (RFs) are depicted during two periods of fixation (left and middle panel) and shortly before a saccade (right panel). Following asaccade from fixation point 1 (FP1) (left panel) to FP2 (middle panel), RFs are displaced by the eye movement, and remain fixed in retinocentric coordinates. With predictive remapping (right panel), RFs are thought to shift to their future, postsaccadic, locations before the saccade begins.

Although the lack of perceived motion during saccades is generally thought to result from a loss of visual sensitivity around the time of eye movements [3,4], such a mechanism is unlikely to account for the lack of perceived retinal image displacement. Over the past 20 years, seemingly convergent evidence has led to a widespread notion that an anticipatory updating of visual receptive fields (RFs), or ‘predictive remapping’, mitigates the perception of retinal image displacement [1,5]. In particular, RFs have been reported to shift from their current, presaccadic locations to their future, postsaccadic locations in anticipation of an upcoming saccade. Thus, in principle, these anticipatory shifts could contribute to the integration of visual information across eye movements. The first evidence of predictive remapping was observed within the lateral intraparietal cortex (area LIP) of the macaque monkey [5]. Many LIP neurons become responsive to stimuli (probes) presented at the future postsaccadic RF location just prior to the onset of the eye movement. Subsequently, similar findings were reported for neurons within the superior colliculus (SC) [6], frontal eye field (FEF) [7,8] (Figure 2), and several areas within extrastriate visual cortex (including V2, V3, and V3a) [9].

Figure 2.

Receptive field (RF) shifts in the frontal eye field (FEF). (A) Electrophysiological recordings in the macaque FEF. (B) Stimulus layout and timeline as used in [8] together with the responses of a FEF neuron. Responses to probes presented long before a saccade, and during fixation at fixation point 1 (FP1) are plotted in grey. Responses to probes shown shortly before a saccade are plotted in gold. FF stands for ‘future field’. (C) Stimulus layout used in [13]. (D) Two example RFs from [13]. RF1s (top) denote the RF centers measured during fixation at FP1, and long before a saccade. RF2s (middle) denote the RF centers measured during fixation at FP2, long after a saccade. The presaccadic RFs (PRFs) (bottom) denote the RF centers measured immediately before a saccade from FP1 to FP2. The gold vectors indicate the predicted remapping vector based on the RF1 centers (black crosses). The white squares represent the predicted remapping RF center based on the RF2s. (E) Population RF centers of all measured FEF RFs plotted as a function of visual space from [13]. Gray vectors are based on the differences between the RF1s and the RF2s and provide an empirical estimate for remapping. Gold vectors are based on the RF1s and PRFs.

While more recent studies of predictive remapping aimed to address more detailed questions of a seemingly well-established mechanism [10–12], the nature of the presaccadic RF shifts in previous remapping studies has been inferred entirely from a few probe locations. Therefore, the lack of detailed RF measurements in both the earlier and more recent studies has left the validity of the predictive remapping hypothesis in question. For example, in a study Sommer and Wurtz [8], monkeys initially fixated upon a fixation point, which was displaced after some time, and the animals were required to make a saccade to its new location. Before the onset of the eye movement, the responses of FEF neurons were probed (Figure 2A,B). The probe in each trial could be presented at one of two locations; that is, the probe could either be presented inside a neuron's RF, as established long before an eye movement during fixation, or the probe could be presented inside the expected remapped RF, referred to as ‘future field’ (FF) in [8]. Figure 2B shows the responses of an example neuron from Sommer and Wurtz [8]. Long before a saccade, and during fixation, the neuron is responsive to a probe presented inside its RF, but it does not respond to a stimulus presented inside the FF. However, immediately before a saccade, the neuron responds to a probe presented inside the FF and stops responding to a probe presented at the RF location. From this basic result, it was concluded that FEF RFs shift presaccadically to their postsaccadic locations.

Following the design used in [8], FEF RFs were measured in greater spatial detail both during fixation and during the presaccadic period by Zirnsak et al. [13] (Figure 2C). Overall, it was found that FEF RFs do not remap before saccades; that is, they do not shift to their future (postsaccadic) locations. Instead, they converge massively toward the saccade target (Figure 2D,E) resulting in a distortion (‘compression’) of visual space (Box 1). It appears that FEF neurons collectively select the space occupied by targets of eye movements (Box 2), rather than predict the retinal consequences of those movements. Note that similar evidence of converging RFs was previously reported in extrastriate area V4 [14]. In addition to more detailed RF measurements, it was of equal importance to sample multiple RF locations in space to discriminate between predictive remapping and convergent RF shifts. As is depicted in Figure 2D and E, RFs that are close to the fovea during fixation will presaccadically converge toward the target, but those shifts will also be in the direction of the saccade, as predicted by the remapping hypothesis. Figure 2D shows two example RFs from Zirnsak et al. [13]. Each panel plots the average neuronal activity as a function of probe location for three different periods. The top panels depict RFs measured during fixation at fixation point 1, long before a saccade. The middle panels depict RFs measured long after a saccade during fixation at fixation point 2. The bottom panels depict the presaccadic RFs measured immediately before a saccade. It is apparent that both examples deviate from the remapping hypothesis. However, it is also apparent that the presaccadic RF of example 1 overlaps to a large degree with the hypothetically remapped RF. Reducing the set of possible probe locations down to two, as in the example from Sommer and Wurtz [8] shown in Figure 2B, would lead to very similar results; that is, presaccadically, a probe presented inside the RF would fail to cause strong responses, whereas a probe presented inside the FF would yield strong responses. Therefore, one would likely conclude that the RF of example 1 has remapped. In contrast, applying the same logic to example 2 would lead to a negative result. While the responses to probes presented inside the RF would also be reduced, the probe presented inside the FF would fail to cause responses and, hence, one would fail to conclude any presaccadic RF shift at all.

Predictive remapping in humans

The first evidence of predictive remapping in human observers utilized an orientation adaptation technique and demonstrated a saccade dependent change in the resulting tilt after-effect (TAE) (Figure 3) [15]. The TAE is usually strongest if probed at the same retinocentric location where it was previously induced by an adapter stimulus. However, it was demonstrated that the TAE decreased gradually at the present, presaccadic location of the adaptor and simultaneously increased at the future, postsaccadic location of the adaptor when observers were about to make a saccade. Similar findings were reported for the motion after-effect [16], and this apparent presaccadic transfer of visual after-effects has been interpreted within the predictive remapping framework. In other words, immediately before saccade onset, RFs of the adapted neuronal population are thought to shift along a vector equal to the impending saccade, and thus a decreased after-effect is observed at the presaccadic adaptor location. In addition, at the newly shifted RF location, the after-effect increases. In both studies, however, testing of presaccadic transfer of the after-effects was limited to locations at which it is difficult to distinguish between predictive remapping and convergent RF shifts, similar to the nonhuman primate studies discussed above. Specifically, the adaptor was presented close to the fovea and a presaccadic transfer of the after-effect is predicted towards the saccade target for both kinds of RF shifts.

Figure 3.

Psychophysical evidence of receptive field (RF) shifts in human observers. (A) Stimulus sequence as used in [15]. Observers made saccades from fixation point 1 (FP1) to FP2 and judged the orientation of the probe stimulus (left versus right). (B) Spatial layout of stimuli used in [15] (top) and [17] (bottom). In [15], the adaptor was presented at the fovea, and probes were presented either at the adaptor position (AP) or the expected remapped position (RP). In [17] the adaptor was presented beyond and above the saccade target. The test between remapping and convergent RF shifts was done by comparing the tilt after-effect (TAE) measured at the RP with the TAE measured at the saccade target position (SP). (C) Results from [15]. The magnitude of the TAE is plotted as the proportion of the maximum TAE measured at the adapted location during fixation (gray). Presaccadically (gold), the TAE increases at the RP and decreases at the AP. (D) Results from [17]. The magnitude of the TAE at the RP and SP is given by the separation between the psychometric functions. Solid curves for leftward adaptors (inset) and dashed curves for rightward adaptors. Gray curves plot data measured long before a saccade. Shortly before saccade onset [blue shaded area in (C)], the TAE (gold curves) increases at the SP and decreases at the RP.

In a more recent study [17], the authors used the paradigm pioneeredin [15], but instead of presenting the adaptor close to the fovea, they presented it above and slightly beyond the saccade target, with respect to the direction of the saccade (Figure 3B). This design separates the locations where one would expect to see a transfer of the TAE, depending on whether presaccadic RF shifts follow the remapping prediction or the convergent RF prediction. Therefore, if RFs are predictively remapped, one would expect a presaccadic increase in the TAE if probed at the postsaccadic adaptor location, which, in this design, was actually located further away from the saccade target compared to the presaccadic adaptor location. By contrast, if RFs converge toward the target presaccadically, one would expect an increase in the TAE probed at a location closer to the saccade target, compared to the postsaccadic adaptor location. The latter effect was observed. Thus, insofar as the presaccadic transfer of the TAE reflects RF shifts, the combined evidence argues that RFs converge toward the saccade target, and this is consistent with the detailed measurements of presaccadic RFs within V4[14] and the FEF [13]. Nonetheless, additional measurements in both space and time are needed to fully characterize the presaccadic changes in after-effects.

Shifts of spatial attention prior to saccades

As previously mentioned (Box 2), a good deal of evidence indicates that spatial attention shifts to the saccade target prior to a saccade [18]. In one of the earlier psychophysical studies [19], human observers had to discriminate visual stimuli while simultaneously preparing and executing saccades to one of several locations. The authors found that discrimination performance was superior if the visual stimulus was presented at the location of the impending saccade, compared to performance at other locations (Figure 4A). Evidence of presaccadic shifts of attention to the targets of saccades[20,21] is further corroborated by neurophysiological studies in nonhuman primates. These studies show that neuronal responses are modulated presaccadically in a manner similar to that observed during covert attention [22], when fixation is maintained. For example, the visual responses of area V4 neurons increase when an animal either covertly attends to [23,24], or prepares a saccade to a RF stimulus [25–28] (Figure 4B). Furthermore, consistent with the performance of human observers, both the attention and presaccadic modulation are accompanied by an increased ability of V4 neurons to discriminate RF stimuli [23,29]. Finally, V4 RFs have been reported to converge towards non-RF targets, both during covert attention [30–32] and during saccade preparation [14], presumably increasing the number of neurons that are effectively processing this region.

Figure 4.

Saccades and shifts of attention. (A) Left, stimulus sequence as used in [18]. Human observers fixated a central fixation point. A string of 5 letters was presented to the left and to the right of the fixation point. The central three letters of each string were surrounded by a colored ellipse and served as potential saccade targets cued subsequently by a colored triangle. Shortly before saccade onset the letter strings were briefly changed. Among distracters (‘5's and ‘2's) a single discrimination target (‘E’ or ‘9’) was shown observers had to report its identity. Right, presaccadic discrimination performance. Discrimination was greatest when the location of the discrimination target and the saccade target (ST) were matched. (B) Presaccadic activity in macaque area V4. The sustained visual response over time (gray) to a receptive field (RF) probe is increased presaccadically (gold) when the monkey plans a saccade to the RF probe [27]. (C) Left, stimulus layout as used in [35]. Right, discrimination performance shown as a function of time relative to saccade onset. The solid line shows performance measured at the first saccade target, the dashed line shows performance measured at the second saccade target, and the dotted line shows performance at the remapped location of the second saccade target. All three measurements increase with time whereas performance measured at the control location (not shown) did not increase with time and was close to chance level (50%).

Anticipatory remapping of attentional pointers

Evidence for predictive remapping has primarily been found in structures involved in the control of saccades, specifically, area LIP, the SC, and the FEF. Furthermore, LIP, the SC, and, particularly, the FEF (see Box 3) have been implicated in the control of visuospatial attention, and all three areas exhibit properties suggesting they each contain a ‘priority map’ [33], in which peaks of activity point to physically salient and/or behaviorally relevant stimuli. Recently [34], it was postulated that the evidence of predictive remapping might be interpreted in terms of an updating of ‘attentional pointers’ within the respective priority maps in anticipation of an upcoming eye movement.

Psychophysical evidence for this proposal has been provided in a series of studies with human observers [35–37]. In one particular experiment [35], a double-step saccade paradigm, in which observers have to plan a sequence of two saccades before executing the first movement of the sequence, was combined with a concurrent discrimination task (Figure 4C). Observers initially had to fixate a central fixation point. Two lines appearing at the fixation point signaled the saccade sequence observers were to execute. In the depicted case, the first saccade was made to the right green rectangle, and the second saccade was made to the rectangle above and to the left of the first saccade target. Within each rectangle, a constant stream of upright gratings was shown, each one followed by noise. At some point in time within a given trial, one of the gratings, either at the location of the first saccade target, the second saccade target, the remapped location of the second saccade target, or at a control location, could undergo a leftwards or rightward change in orientation. Observers had to report the direction of the orientation change at the end of a trial. It was demonstrated, that in addition to the above-mentioned increase in discrimination performance at the first saccade target, performance also increases at the second target, even before the start of the first saccade [38]. Furthermore, discrimination performance also increases at the remapped, future retinocentric location of the second saccade target [35]; specifically, the location of the second saccade target after completion of the first saccade.

Given that RFs appear to converge rather than remap, as described above, how might the evidence of remapping of attention be explained? First, it is important to consider the number of foci of convergence in a double-step saccade task. In the single-step saccade paradigm [13], the focus of convergence was the (single) saccade target location, and likely a center of attentional deployment [19]. Although it awaits experimental evidence, a simple hypothesis would be that in a two-step saccadic sequence, two retinocentric foci of convergence would exist presaccadically, corresponding to the two saccade target locations and two centers of attentional deployment [38,39]. Evidence from [35] suggests that there may be an additional, third, focus of convergence that emerges at the predicted, postsaccadic location of the second target. Two neurophysiological observations appear consistent with this interpretation. First, an earlier study indeed seems to show that area LIP neurons signal the remapped second target [40]. Second, FEF neurons appear to signal three locations before the onset of a two-saccade sequence, specifically, the first target location, the second target location, and the future, postsaccadic location of the second target [41]. However, neither the psychophysical nor the neurophysiological study seems to have had sufficient spatial resolution to clearly distinguish between two and three foci. For example, in [35] it remains to be determined if the increased performance at the second target location and at the remapped second target location clearly results from two distinct foci or merely two points on a single focus centered on the second target (see also [42] and [43] for a related issue). This distinction is important, as the latter case would not require any additional mechanism other than retinocentric target selection [44,45].

Concluding remarks and future directions

We have discussed the evidence of RF shifts during the preparation of saccadic eye movements. Thus far, this evidence suggests that previous observations of predictive remapping may be explained instead by a convergence of RFs toward the targets of saccades. The apparent convergence of RFs seems to be in accordance with other neuro-physiological and psychophysical evidence of a predominant role of saccadic targets in vision [46–52]. However, future experiments will be necessary to adequately understand the nature of presaccadic RF shifts. These experiments might, for example, address RF shifts in terms of their (i) temporal dynamics, (ii) generalizability to different types of RF stimuli (e.g., flashed versus stable), (iii) similarities across different saccade paradigms (e.g., memory-guided versus visually guided saccades), (iv) similarities across the visual system, and (v) similarities during single saccades, sequences of saccades, as well as during naturalistic visual scanning. Although one presumes that addressing the above questions will elucidate the function of presaccadic RF shifts, it is likely that their function will only be determined by causal tests of a link between RF shifts and visual perception. In particular, it will be crucial to establish whether RF shifts are necessary for perceived visual stability across saccades.

Box 1. Saccades and compression of visual space.

While vision is stable across eye movements under natural viewing conditions, strong perceptual distortions can be induced under controlled laboratory settings (Figure IA). In a prototypical experiment, human observers are asked to perform an eye movement from an initial fixation point to another stimulus, the saccade target. In addition, a third visual stimulus, the probe, is briefly flashed around the time of the saccade and observers have to indicate the probe's location after completing the eye movement. For probe presentations long before and after a saccade, observers indicate the probe location to be close to veridical. For probe presentations close to saccade execution, increasing errors in probe localization occur peaking at approximately the onset of the eye movement. Under certain viewing conditions, the pattern of mislocalization resembles a ‘compression of visual space ’ [53,54], meaning that probes are indicated much closer to the saccade target than they have actually been presented. For a quantitative account of compression [55], it has been demonstrated that simple gain modulations centered at the cortical representation of the saccade target can explain key features of the psychophysical (Figure IB) and neural (Figure 2E) data. It is assumed that neurons with RFs relatively close to the saccade target are activated more strongly by a given probe around the time of a saccade, as are neurons with RFs at locations further away from the target. This pattern of activity results in a distortion of the neuronal population response toward the saccade target (Figure IC), similar to the observed mislocalization in humans. The pattern of activity further leads to convergent RF shifts within downstream areas, which are driven by those distorted responses [56].

Figure I Compression of visual space. (A) Mislocalization of stimuli presented close to saccade onset by human observers [54]. Vector origins indicate the veridal positions of stimuli and vector endpoints indicate the mean indicated positions. (B) Mislocalization of stimuli presented close to saccade onset by a model of perisaccadic processing [55]. (C) Illustration of how simple gain modulations can distort the neuronal population response. The gray curve represents the purely stimulus-driven (white solid rectangle) response of simple model neurons assuming Gaussian RFs. The solid, gold curve represents the response of the same set of model neurons now modulated by a multiplicative signal (dashed gold). The response is distorted toward the center of the signal and reading out the maximum of the response would lead to a mislocalization of the stimulus towards the center of the signal as well (white rectangle). Note that all responses are shown normalized.

Box 2. Saccades and visuospatial attention.

The relationship between eye movements and selective spatial attention has long been of interest, not only because it is inherent that gaze shifts reflect the overt deployment of attention, but also because of evidence that saccades might be sufficient to cause the perceptual improvements that define the orienting of covert attention [57]. Early hints of an interdependence between saccades and attention came from psychophysical studies in humans, such as one reporting a positive correlation between visual discrimination of briefly presented target stimuli and the direction of ensuing saccades [58]. Later studies reported that visual discrimination is facilitated at the endpoints of saccades, even when observers are instructed to attend elsewhere [19,20]. Psychophysical results such as these [18] lend credence to the proposal that saccades and spatial attention reflect a common neural mechanism [59]. Specifically, the proposal holds that shifts of covert attention are driven by neurons engaged in the planning of saccades, yet those plans are not executed. Studies investigating the neural mechanisms of visual spatial attention and saccades indeed found evidence that the two are largely, though not completely, overlapping (see Box 3). Although some psychophysical evidence suggests that spatial attention and saccade planning can be dissociated in terms of perceptual benefits [60,61], neurophysiological evidence suggests that saccade planning is sufficient to bring about the prototypical neural correlates of spatial attention within visual cortex [62].

Box 3. The FEF and visuospatial attention.

The FEF serves not only as part of a network of retinotopic visual areas [63] and a network that controls saccades [64], but it is also part of a network of prefrontal areas involved in executive functions. Each cortical column of FEF neurons comprises a set of distinct functional classes of neurons that either signals the presence of visual stimuli at a retinocentric location, the preparation of a saccade to that location, or both [65]. In addition, many FEF neurons encode the location of remembered stimuli (or the location of prepared saccades) over several seconds, consistent with a role in working memory [66]. More recent evidence from human and nonhuman primate studies has also established a role of FEF neurons in the control of visuospatial attention. Specifically, this evidence suggests that at least a subset of FEF neurons directly modulates the gain of signals within posterior visual cortex in a location-specific manner, and thus provide a source of attentional filtering within the spatial domain [67].

Like neurons within visual cortex, FEF activity is enhanced when covert attention is directed to a visual stimulus within a neuron's RF [68]. However, unlike posterior visual areas, but similar to posterior parietal cortex [69,70], FEF activity is also enhanced when attention is directed to a RF without visual stimulation [66,71]. As with posterior parietal cortex [72,73], permanent or temporary inactivation of the FEF leads to deficits consistent with visuospatial neglect [74,75] suggesting that both structures are necessary for normal spatial attention. Perturbations of FEF activity can initiate both the behavioral deployment of spatial attention [76] and the neural correlates of that deployment within visual cortex [77,78] indicating that changes in FEF neuronal activity are sufficient for attentional control. Although it remains to be determined to what extent the FEF's control in visuospatial attention operates principally through other structures (e.g., parietal cortex, SC) in modulating posterior visual representations, the existence of projections of layer 2/3 FEF neurons to extrastriate visual areas [79] suggests that at least some of that control is direct. Taken together, evidence to date indicates that local, columnar networks of FEF neurons modulate visual cortical signals in conjunction with the preparation of saccades to specific spatial locations, and this provides a basis, at least in part, for visuospatial attention [67,80].

Glossary

Compression of visual space

a visual illusion occurring around the time of saccades. Briefly presented stimuli are perceptually mislocalized closer to saccade targets. This phenomenon might be related to gain modulation induced distortions of neuronal population responses.

Focus of convergence

point in retinocentric coordinates to which receptive fields shift. Here, each point of convergence may simultaneously reflect a peak in gain modulation of neuronal responses, in the probability of a saccadic eye movement and in a perceptual benefit. Furthermore, the focus of convergence may also reflect a center of perceptual compression.

Fovea

central region of the retina where visual acuity is greatest.

Gain modulation

change in the input/output ratio of a neuronal response.

Lateral intraparietal area (LIP)

brain region within macaque parietal cortex. Together with the frontal eye field (FEF), located in prefrontal cortex, and the superior colliculus (SC), located in the midbrain, LIP appears to play a key role in the control of visually guided saccades and the deployment of visuospatial attention.

Presaccadic receptive field shift

general term used here to describe an effective change in the receptive field before the onset of a saccade.

Predictive remapping

anticipatory shift of receptive fields from their current, presaccadic locations to their future, postsaccadic locations before saccade onset.

Receptive field (RF)

region of sensory periphery (e.g., retina) where stimulation (e.g., with light) changes the spiking response of a given neuron.

Retinocentric coordinates

eye centered spatial reference frame relative to the fovea.

Saccadic eye movements (saccades)

quick, jerk-like, movements of the eyes used to position targets at the center of gaze (onto the fovea).

Tilt after-effect (TAE)

a visual illusion observed after a prolonged exposure to a certain orientation (adaptor stimulus). A subsequently shown, probe stimulus, of a different orientation is perceived as tilted from its veridical orientation, and away from the adaptor.

Visual area V4

located in extrastriate visual cortex, V4 is considered part of the ventral stream. V4 neurons are selective for shape and color. The activity of V4 neurons can be modulated by attention and saccade preparation.