Receptive-field remapping and transsaccadic space updating

Abstract

Since its discovery, perisaccadic receptive-field (RF) remapping—the shift of RFs around the time of saccades—has inspired many studies exploring its origins, characteristics, and functional implications. We review recent developments leading to a circuit model that links spatiotemporal properties of remapping to functional requirements of transsaccadic visual stability. The model uses center/surround connections to store a stimulus’ retinal position as a population activity. This activity profile is shifted backward (against the saccade) as a wave by directional connections gated by the corollary discharge of the saccade command. When the accumulative backward shift of the population activity equals the saccade magnitude (equivalent to forward RF remapping of the same magnitude), the presaccadic retinal position is updated to the correct postsaccadic position. The model explains translational mislocalization for stimuli flashed around saccades as imperfect updating. The model is equivalent to the classic head-direction model, suggesting a general strategy of neural computation.

Introduction

We see the world via retinal inputs. However, the retina moves with the eye. This raises the age-old question of how we perceive a continuous, stable world despite changing retinal images accompanying eye movements [1]. The answer is conceptually simple: the visual system must use extra-retinal signals to “compensate” for the eye-movement effects but the underlying physiological computation may not be straightforward [2,3]. In this review, we focus on perisaccadic RF remapping as one way to realize such computation. Although remapping was discovered decades ago [4], there have been controversies regarding both its physiology and function [5]. Recent work helps address some of the controversies by measuring spatiotemporal RFs during remapping (from about 100 ms before to about 100 ms after saccades) and proposing a mechanistic model linking perisaccadic remapping to transsaccadic visual stability [3,6,7].

Forward and convergent RF remapping

Remapping refers to the observation that around the time of saccades some cells’ RFs (perisaccadic RFs or pRFs) change their locations. Duhamel et al. first reported that a fraction of cells in lateral intraparietal area (LIP) show visual responses at their future (postsaccadic) RF (fRF) locations, accompanied by reduced responses at their current (presaccadic) RF (cRF) locations, even before the saccade onset [4] (cf. Fig. 1a). Later, Wang et al. found that LIP cells’ pRFs also cover the midpoints between their cRFs and fRFs [8]. In these cases, the pRFs shift in the pending saccade direction and are referred to as predictive or forward remapping (Fig. 1a). Similar remapping cells of varying frequencies of occurrence have been reported in superior colliculus (SC), frontal eye Fig. 1. RF remapping. cRF and fRF (black disks) indicate a cell’s classic RF when the eye looks at initial fixation (cross) and the target (square), respectively. (a) A controversy concerns whether the cell’s perisaccdic RF (pRF) shifts forward (red arrow) or convergently (green arrow). (b) Wang et al. (2024)’s remapping results in LIP and FEF. During the delay period, dRFs first shift slightly toward the initial fixation (blue arrow) but then turn toward the target (green arrow). During the perisaccadic period, pRFs first shift toward the target (green arrow) but when the saccade is about to start and thereafter, the pRF remapping becomes predominantly forward (red arrow). (c) Spatiotemporal profile of pRF forward remapping in LIP/FEF. Red disks represent snapshots of a cell’s pRF at four times relative to the saccade onset. Dashed circles indicate eye positions. The cRF and fRF (black disks) are also shown for reference but they are sometimes covered by the overlapping pRF. (d, e) Forward remapping of RFs is equivalent to backward updating of population responses. The red disks in panel d are identical to those in panel c but are plotted relative to the initial fixation (cross) regardless of the eye position because the stimuli for measuring the pRF is flashed before the saccade onset; they represent the pRF progression of a single cell whose cRF is centered at the blue-dot position. By contrast, the red disks in panel e represent the progression of the population response of all cells (with different cRF positions) evoked by a single stimulus flashed at the green-dot position. fields (FEF), V2, V3, V3A, and V4 [9–11,6,12]. Since the remapping happens perisaccadically in the saccade direction, it likely depends on corollary discharges (CDs) of saccade motor commands, an extra-retinal signal. Sommer and Wurtz demonstrated this dependence by showing a CD pathway from SC to FEF via the thalamus [13,14]. Forward remapping cells appear to “anticipate” the retinal-image patches that will fall on their RFs during and after saccades and thus could contribute to transsaccadic visual stability[4,15,16]. However, the underlying mechanism was unclear.

Fig. 1.

RF remapping. cRF and fRF (black disks) indicate a cell’s classic RF when the eye looks at initial fixation (cross) and the target (square), respectively. (a) A controversy concerns whether the cell’s perisaccdic RF (pRF) shifts forward (red arrow) or convergently (green arrow). (b) Wang et al. (2024)’s remapping results in LIP and FEF. During the delay period, dRFs first shift slightly toward the initial fixation (blue arrow) but then turn toward the target (green arrow). During the perisaccadic period, pRFs first shift toward the target (green arrow) but when the saccade is about to start and thereafter, the pRF remapping becomes predominantly forward (red arrow). (c) Spatiotemporal profile of pRF forward remapping in LIP/FEF. Red disks represent snapshots of a cell’s pRF at four times relative to the saccade onset. Dashed circles indicate eye positions. The cRF and fRF (black disks) are also shown for reference but they are sometimes covered by the overlapping pRF. (d, e) Forward remapping of RFs is equivalent to backward updating of population responses. The red disks in panel d are identical to those in panel c but are plotted relative to the initial fixation (cross) regardless of the eye position because the stimuli for measuring the pRF is flashed before the saccade onset; they represent the pRF progression of a single cell whose cRF is centered at the blue-dot position. By contrast, the red disks in panel e represent the progression of the population response of all cells (with different cRF positions) evoked by a single stimulus flashed at the green-dot position.

Early remapping studies probed a cell’s responses at only a few positions so the full extent of pRF changes was not measured. To rectify this shortcoming, Zirnsak et al. sampled FEF cells’ responses from an array of positions [5]. They found that pRFs converge toward the saccade target (Fig. 1a), instead of translating forward toward fRFs. They concluded that such remapping cannot contribute to visual stability.

Recent studies help resolve the controversy by measuring detailed spatiotemporal properties of remapping [6,7], similar to measurements of spatiotemporal RFs in visual areas [17]. Neupane et al. did so first in V4 [6]. Like Zirnsack et al., Neupane et al. flashed dots before saccade onset on a grid of spatial locations. However, unlike Zirnsack et al. who pooled neuronal responses over one, large 300-ms window, Neupane et al. divided cells’ responses into 50 ms bins and examined how pRFs evolve in time. They found that V4 pRFs shift gradually over time and that this remapping is stronger in the forward than in the convergent direction. Additionally, the V4 remapping starts after the saccade offset, later than the LIP/FEF remapping.

To address the aforementioned controversy regarding the remapping direction in FEF and LIP, Wang et al. measured the spatiotemporal properties of pRFs in these areas with a delayed saccade paradigm [7]. In most remapping studies, the offset of the initial fixation and the onset of the saccade target occur simultaneously to initiate the saccade. It is known that the target onset [18] and the pending saccade [19,20] both draw attention to the target, and RFs shift toward attentional loci [21,22]. To separate the effects of the target onset and the saccade, and to examine the remapping time course in detail, Wang et al. introduced a delay between the target onset and the fixation offset (the saccade go signal) [7]. They found that RFs during the delay period (dRFs) first shift slightly toward the initial fixation point but then turn toward the target (Fig. 1b), likely reflecting attentional switch from the initial fixation to the target as the saccade task approaches. Early in the perisaccadic period, pRFs still shift toward the target but when time gets near the saccade onset and thereafter the pRFs shift predominantly forward (Fig. 1b). Like V4, forward remapping is significantly stronger than convergent remapping in LIP and FEF.

Forward RF remapping and transsaccadic space updating

Given the dominant, forward remapping across saccades in LIP/FEF, how could it contribute to transsaccadic perceptual stability? Many possibilities have been proposed but most are based on the assumption that cells’ pRFs remap completely to their fRFs before the saccade onset [4,23,15,24,25]. We now know that on average LIP/FEF cells’ pRFs shift gradually from their cRF locations to near their fRF locations, from about 100 ms before the saccade onset to about 100 ms after the saccade offset (i.e., about 150 ms after the saccade onset) [7] (Fig. 1c). The pRFs thus move through intermediate locations instead of jumping from the cRFs to fRFs directly. As explained elsewhere [3], such results pose difficulties for previous proposals.

Recent theoretical considerations and simulations of a circuit model showed the equivalence between the spatiotemporal properties of pRF remapping in LIP/FEF and transsaccadic updating of population responses required for achieving perceptual stability [3,7,26]. First note that the pRF remapping was measured with dots flashed on a large grid of positions before the saccade onset [3,7]. This implies that pRF snapshots at different times (red disks in Fig. 1c) should all be plotted relative to the initial fixation (cross) regardless of the eye position. The resulting retinotopic representation of the pRF (Fig. 1d) shows a spatiotemporal orientation reminiscent of a motion detector but as argued elsewhere [3], pRF remapping is unlikely for detecting saccade-induced motion but for transsaccadic updating. Second, the location of a given stimulus must be decoded from the responses it evokes from cells with different RF locations (population response), instead of from any individual cell’s responses to different stimulus locations (spatial tuning or RF). The question, then, is how to infer the population responses corresponding to the measured pRFs. Formally, it can be shown that forward remapping of pRFs of a certain magnitude is exactly equivalent to a backward (against the saccade) shift of the population responses of the same magnitude (when the population response is a function of the cells’ classic RF locations) [26]. Intuitively, consider a cell whose retinotopic RF center is indicated by the blue dot in Fig. 1d. When a stimulus is flashed at the fRF location well before the saccade onset (green dot in Fig. 1e), it initially evokes a population response from cells tuned around the green dot (red disk in Fig. 1e at −100 ms). The blue cell does not respond at that time because the green dot is well outside its RF. Across the saccade, however, the population response propagates backward as a wave to reach the blue cell (red disk at 150 ms in Fig. 1e). Therefore, when the blue cell’s pRF is measured with many dots flashed before the saccade and covering a large area, its pRF appears to shift gradually forward to the fRF across the saccade (Fig. 1d) as it receives backward-propagating activities originated from different cells between its cRF and fRF at different times.

The backward shift of a stimulus’ population response by the saccade magnitude (Fig. 4b) updates the stimulus presaccadic retinotopic position to its correct postsaccadic retinotopic position by effectively subtracting the saccade vector [3]. For a transient stimulus flashed well before a saccade, the updating means that the brain represents its correct retinotopic position after the saccade. In a double-step memory saccade task, a standard demonstration of transsaccadic visual stability, such updating of the second target after the first saccade is needed for performing the second saccade correctly. For a persistent (stationary) stimulus in daily life, the updating means that the internally updated position agrees with the reafferent input from the same stimulus after the saccade. This agreement is proposed to be the basis for transsaccadic visual stability [3]. When the agreement fails, the errors could drive learning to match the updating magnitude to the saccade magnitude again.

A circuit model for forward remapping and backward updating

The entire pRF remapping time course is likely driven by CDs of saccade commands because stimuli for measuring pRFs were flashed before the saccade onset and there was no additional reafferent visual contribution to the pRFs. For the same reason, what is remapped/updated must be the memory representations of the flashed stimuli. Then, to implement the remapping/updating process in a circuit model, we need a set of connections to maintain in memory the population response representing the retinotopic position of a flashed stimulus, and another set of connections, gated by the CD of a saccade, to shift the population response backward across the saccade, to the updated position. These two sets of connections are shown in Fig. 2a and illustrated in Fig. 2b. The first set is symmetric, center-excitation/surround-inhibition connections among cells tuned to different retinotopic locations (red). This so-called Mexican-hat connectivity is consistent with physiological interactions among cells in LIP [27] and FEF [28], and provides attractor dynamics for maintaining memory responses [29,30]. The second set is CD-gated directional connections (blue) with inhibition and excitation in the forward and backward directions, respectively, to propagate responses backward as a wave (or equivalently, remap RFs forward) [8,7,3,31]. Both sets of connections emerge automatically in neural networks trained to correctly update stimuli’s retinotopic positions across saccades [7]. In simulations, when one collects a single cell’s responses to dots flashed before the saccade onset over many positions, the resulting pRF shows gradual forward remapping (Fig. 2c). Equivalently, when one collects the responses of all cells activated by a single dot flashed before the saccade onset at one position, this population response profile shifts backward which updates the dot’s retinotopic position across the saccade (Fig.2d). The forward remapping and backward updating have exactly the same magnitude, which is proportional to the total CD signal, and can be set to the saccade magnitude by adjusting the CD signal via learning. The same circuit also correctly updates retinotopic positions of persistent stimuli across saccades [3,7].

Fig. 2.

A circuit model for RF remapping and population-response updating across saccades. (a) Recurrent connection strengths among model LIP/FEF cells as a function of the difference between the cells’ preferred retinotopic positions (cRF centers). Symmetric, center-excitation/surround-inhibition connections (red) form memory for a flashed stimulus, and directional connections (blue, for rightward saccades) are gated by CDs to update the memory. (b) Schematic of the network structure of the circuit. Circles indicate a few cells’ RF center locations in retinotopic coordinates. The retinal input of a stimulus evokes a population response among cells tune at and near the stimulus position (black curve) which is sustained by the symmetric connections (red) as a memory. This population memory response is shifted backward (dashed black arrow) as a traveling wave when the CD activates directional connections (blue). (c) Simulation of a given cell’s pRF activated by dots flashed 200 ms before the saccade onset over many positions. The cell’s ordinary RF (cRF) is centered at 0 deg. The shaded rectangle indicates saccade duration. (d) Simulation of all cell’s population response to a given dot flashed 200 ms before the saccade onset at the 20-deg position.

Despite different motivations and considerations, the remapping/updating model is formally equivalent to Zhang’s classic head-direction model [29], with the CD signal replacing the vestibular signal, and the non-periodic retinotopic space replacing the periodic head direction.

Perisaccadic perceptual mislocalization

Although persistent objects in daily life appear stable across saccades, we mislocalize brief stimuli flashed around saccades, a phenomenon known as perisaccadic perceptual mislocalization [32–34]. Typically, subjects make a saccade while a stimulus is flashed at various times. Subjects report the stimulus location after the saccade. Perisaccadic mislocalization has two components, a translational (or shift) component along the saccade axis and a convergent (or compressive) component toward the saccade target [33,35]. The convergent component depends on a postsaccadic visual reference [36], suggesting that it might involve other factors in addition to the saccade. The translational component dominates in the dark; the mislocalization is in the forward direction for stimuli flashed around the saccade onset, and disappears or reverses to the backward direction for stimuli flashed around the saccade offset [33,34,36].

The circuit model [7] naturally explains the translational component of perisaccadic mislocalization [3,37]. Since forward remapping in LIP/FEF starts about 100 ms before the saccade onset and finishes about 100 after the saccade offset, the CD signal driving the process likely have a similar time course (Fig. 3a) [3,7]. For a stimulus flashed well before the saccade onset, it will evoke a population response before the CD signal starts (Fig. 3a, top) and the population response will be shifted backward by the entire CD time course with a total, cumulative magnitude equal to the saccade magnitude (Fig. 3b, top). (The same circuit also shifts responses to persistent stimuli by the saccade magnitude to ensure visual stability in daily life [3].)

Fig. 3.

A schematic illustration of how the circuit model explains the translational component of perisaccadic perceptual mislocalization. The three rows represent the cases when a stimulus is flashed well before the saccade, at the saccade onset, and at the saccade offset, respectively. (a) Temporal relationships. The vertical dashed lines indicate the saccade onset and offset times. The green and red bars indicate retinal flash time and the response onset time in LIP/FEF and the gray shading indicates the CD time course. (b) Retinotopic spatial relationships. The dashed lines indicate the presaccadic and postsaccadic retinal positions of a stimulus with a fixed position in the world. The solid and dashed curves represent the stimulus-evoked population responses initially and after the updating, respectively. Note that when the stimulus is flashed at the saccade offset, the initial population response centers at the postsaccadic stimulus position.

Now consider a stimulus flashed at the saccade onset (Fig. 3a, middle). This retinal input reaches LIP/FEF after a 40 ms delay and generates a population response in LIP/FEF after another 10 ms. The total response latency of about 50 ms combined with the early start of the CD signal implies that the population response misses large part of the CD time course [3,7]. Consequently, the backward shift of the response falls far short of the required saccade magnitude, and this insufficient backward updating after the saccade results in a forward translational mislocalization (Fig. 3b, middle). Next consider a stimulus flashed at the saccade offset. Because the saccade has ended, the stimulus’ retinotopic position should not be updated. Yet, the population response evoked by the stimulus still catches the tail part of the sluggish CD time course (Fig. 3a, bottom), producing a small backward shift [3,7]. This unnecessary backward updating of the population response after the saccade generates a small backward translational mislocalization (Fig. 3b, bottom). By simulating the circuit model for different stimulus onset times, one can explain the observed mislocalization curve [3].

Attentional modulation of forward remapping/backward updating

Wang et al.’s results [7] are consistent with the interpretation that attention causes convergent RF shifts toward attentional loci (independent of saccades per se) and the CDs of saccade commands, which become available around the saccades, cause forward shifts toward fRFs. Interestingly, although forward remapping is not produced by attention, it is modulated by attention to stimuli used to measure pRFs [38–42]. For example, Joiner et al. used some distractor dots to reduce attention to the single dot for measuring pRFs in each trial, and found that remapped responses in FEF are reduced [39]. Rolfs et al. asked subjects to make successive saccades to two (targets) of six Gabor stimuli [40]. Their results suggest that forward remapping occurs only near the two targets, the attentional loci, but not the other stimuli. Thus, it is best to measure pRF remapping by flashing a single dot at a time [6,7] instead of full-screen white noise. Also note that unlike the initial fixation and saccade target, the flashed dot for measuring RFs changes positions from trial to trial and covers a large area across trials. For this and other reasons [7] the flashes are unlikely to produce RF convergence toward any specific position.

The attentional modulation of forward remapping could be explained by a reduced symmetric connectivity in the circuit model such that it could not sustain memory activities alone, but requires, in addition, attentional enhancement of the activities. Then, without attention, memory activities would decay away before being fully updated/remapped across saccades (unpublished results). The decay would also effectively reset memories when attention is withdrawn from them.

Predictions of the circuit model

The predictions are summarized in Box 1. (1) For stimuli flashed before the saccade onset, the closer they are to the saccade onset, the more CD time course their population response in LIP/FEF will miss, and the smaller the updating will be. This is equivalent to the prediction that for single cells’ pRFs measured with flashes before the saccade, forward remapping magnitudes are smaller for later flashes. This prediction has been tested and confirmed [3]. (2) If pRFs were measured with stimuli flashed at least 150 ms before the saccade onset, then their final forward remapping magnitude, achieved around 100 ms after the saccade offset, should equal to the saccade magnitude. In Wang et al. [7], the medium time of the stimuli was about 100 ms before the saccade onset. In this case, the prediction is weaker: the final forward remapping magnitude should be proportional to the saccade magnitude. This prediction has been tested and confirmed [7]. (3) When stimuli’s response latency from the retina to LIP/FEF is increased, the forward mislocalization (for flashes around the saccade onset) should increase because the responses miss more of the CD time course, and the backward mislocalization (for flashes around the saccade offset) should decrease because the responses avoid more of the unneeded CD time course (cf. Fig. 3). This latency could be manipulated by varying stimuli’s contrast and size. (4) The physiological counterpart of prediction 3 is that for stimuli flashed at about the same time relative to the saccade, those evoking shorter/longer latency responses should miss less/more CD time courses in LIP/FEF and produce larger/smaller final forward remapping magnitudes. (5) The symmetric and CD-gated connectivity patterns are also predictions.

Box 1. Predictions of the circuit model for transsaccadic remapping/updating:

1. For single LIP/FEF cells whose pRFs are measured with flashes before the saccade onset, forward remapping magnitudes are smaller for later flashes.

2. The final remapping magnitude, achieved around 100 ms after the saccade offset in LIP/FEF, is proportional to the saccade magnitude.

3. Psychophysically, when stimulus attributes (such as contrast and size) are manipulated to increase/decrease response latency, the forward mislocalization (for flashes around the saccade onset) should increase/decrease, and the backward mislocalization (for flashes around the saccade offset) should decrease/increase.

4. For stimuli flashed about the same time relative to the saccade, those evoking shorter/longer-latency responses in LIP/FEF should have larger/smaller final forward remapping magnitudes.

5. The symmetric and CD-gated connectivity patterns are also predictions.

Predictions 1,2, and 4 concern pRF remapping of single cells. If population responses evoked by single stimuli in many cells tuned to different retinotopic positions can be measured, then there are corresponding predictions which are identical to the pRF predictions except that population responses shift in the backward direction across saccades.

Open questions

(1) Whereas Neupane et al. and Wang et al. found that pRFs in V4 and LIP/FEF shift from the cRFs to fRFs gradually through intermediate positions [6,7], Denagamage et al. found that V2 pRFs jump from the cRFs to fRFs [12]. (2) V4 remapping starts after the saccade offset [6], much later than that of LIP/FEF [7]. Surprisingly, Denagamage et al. found that V2 responses jump from the cRFs to fRFs before the saccade onset, implying earlier and faster V2 remapping than even FEF/LIP remapping.

The above two issues involve studies that differ in many ways in addition to the different brain areas. For example, Neupane et al. and Wang et al. measured pRFs with small dots flashed before the saccade onset whereas Denagamage et al. flashed Gabor patches continuously across the saccade. Continuous stimulation requires precise temporal alignment of eye position and stimuli and separation of cells’ responses into retinal/afferent and laterally remapped components. To resolve the issues, future studies should focus on relevant technical details.

(3) The time-dependent CD strength is proportional to the speed of remapping/updating [29]. Although many forms of the CD work in the model [3,7], it is unknown how saccade motor command is converted into a signal of remapping/updating speed.

(4) Relatively little is known about remapping of stimulus features [12,43,44] and its relationship to content integration/comparison and inattentinal/change blindness across saccades [45,46].

(5) The circuit model is based on the detailed spatiotemporal properties of remapping in LIP/FEF [7]. Future studies should measure such properties in other areas (e.g., SC) to provide a more complete picture of transsaccadic processing.

(6) Gain fields, the eye-position modulation of neuronal responses, can provide a craniotopic representation of stimuli and thus stable perception of space across eye movements including saccades [2]. Whether and how gain fields and remapping/updating are integrated at the circuit level are important topics for future research.

Conclusions

Measurements of spatiotemporal properties of perisaccadic receptive fields using stimuli flashed before the saccade onset have helped resolve a controversy regarding the direction of receptive-field remapping. Theoretical considerations show that forward remapping of receptive fields is equivalent to backward updating of retinotopic positions of stimuli required for transsaccadic perceptual stability. A simple circuit model implements the remapping/updating process, with testable predictions. The model is equivalent to the head-direction model suggesting a general strategy of neural computation. Future studies are needed to address remaining open questions on remapping and its relationship to gain fields.