Perisaccadic and attentional remapping of receptive fields in lateral intraparietal area and frontal eye fields

SUMMARY

The nature and function of perisaccadic receptive field (RF) remapping have been controversial. We use a delayed saccade task to reduce previous confounds and examine the remapping time course in the lateral intraparietal area and frontal eye fields. In the delay period, the RF shift direction turns from the initial fixation to the saccade target. In the perisaccadic period, RFs first shift toward the target (convergent remapping), but around the time of saccade onset/offset, the shifts become predominantly toward the post-saccadic RF locations (forward remapping). Thus, unlike forward remapping that depends on the corollary discharge (CD) of the saccade command, convergent remapping appears to follow attention from the initial fixation to the target. We model the data with attention-modulated and CD-gated connections and show that both sets of connections emerge automatically in neural networks trained to update stimulus retinal locations across saccades. Our work thus unifies previous findings into a mechanism for transsaccadic visual stability.

In brief

The nature of perisaccadic receptive field remapping has been controversial. Wang et al. show that remapping directions in the LIP and FEF change with time from attentionally driven convergent remapping to corollary-discharge-driven forward remapping. Simulations suggest that connectivity patterns for both types of remapping interact to enable transsaccadic visual stability.

Graphical abstract

INTRODUCTION

Visual information enters the brain via the retina. However, a retinotopic representation is inadequate for spatially accurate perception because eye movements change the retinal location of a given object in the world. Nonetheless our spatial perception is largely independent of gaze. A classic demonstration is the double-step task.^1,2 Subjects must make successive saccades to two flashed targets, both of which disappear before the first saccade, in the absence of other visual reference. The retinal and oculomotor vectors of the first saccade are identical. However, the first saccade creates a dissonance between the spatial and retinotopic locations of the second saccade target, and the brain must compensate for this dissonance. Helmholtz³ theorized that the brain used the motor signal for the first saccade to update the visual representation and create a spatially accurate visual signal for perception and action. Duhamel et al.⁴ showed that Helmholtz’s theory has a physiological correlate. When monkeys fixate, a given object in space occupies the receptive field (RF) of a neuron, the current RF (cRF). When monkeys make a saccade, the saccade will move the RF to a new spatial location, the future RF (fRF), even though the object’s spatial location has not changed. Neurons in the lateral intraparietal area (LIP) respond to a stimulus in the fRF around the saccade onset, effecting a forward shift of the RF in the direction of the saccade. Because the shift of some neurons starts even before the eye moves, the signal causing the shift must arise from a motor signal feeding back to the sensory system, a phenomenon now known as corollary discharge (CD). This forward shift is found in many brain areas, including the superior colliculus (SC),⁵ the frontal eye fields (FEF),⁶ V3,⁷ and the parietal reach area.⁸ In the LIP, the perisaccadic visual responses are not limited to the cRF and fRF. Instead, stimuli positioned in any of the spatial locations across which the saccade will sweep the retinal RF will evoke a response (Figure 1A). Stimuli closer to the fRF evoke responses with longer latencies than stimuli closer to the cRF.⁹ Forward remapping has been postulated to provide a key mechanism for the transsaccadic maintenance of a spatially accurate signal.^10–12 One may wonder whether the remapping itself produces perceptual mislocalization, and we have considered this issue elsewhere¹³ and in the discussion.

Figure 1. RF remapping and the recording paradigm.

The cross, square, and arrow represent the initial fixatio n point, saccade target, and saccade vector, respectively.

(A and B) Two types of perisaccadic RF remapping have been reported: (A) forward shift in the saccade direction and (B) convergent shift toward the target. The labels are explained below.

(C and D) We used a delayed saccade task for single-unit recordings: (C) the trial sequence, with probes omitted and the dashed circles representing eye positions, and (D) the detailed time course. Four probes were flashed, one for each of the four epochs: pre-target (current), delay, perisaccadic, and post-saccadic (future). A cell’s RF mapped from these periods will be denoted cRF, dRF, pRF, and fRF, respectively. (No dRF is included in A and B, as previous studies did not measure it.) For each epoch, the probe stimulus (filled white squares in the bottom row) appears randomly at one of the spatial array positions (open white squares, not shown in the experiment). The array size and location were tailored for each cell according to pilot mapping.

The original remapping studies only probed a cell’s responses at a few positions. To measure perisaccadic RFs more completely, an influential study of Zirnsak et al.¹⁴ sampled FEF cells’ responses from an array of spatial positions. The study found that around saccade onset, RFs shift toward the saccade target (Figure 1B) instead of toward the fRFs. The conclusion was that this convergent shift is a substrate of attention to the saccade target, unrelated to the maintenance of a spatially accurate signal across saccades, despite neuropsychological evidence (discussion). The study argued that the forward shift may be an artifact of under-sampling convergent RFs of cells with small eccentricities. However, the methods used caused a couple of confounds: (1) previous studies show that both the abrupt onset of visual stimuli¹⁵ and saccade targets¹⁶ evoke attention, and RFs shift toward the locus of attention.¹⁷ Most remapping studies, including that of Zirnsak et al., use the onset of the saccade target as the saccade go signal and thus could not separate the effect of the attention evoked by the target onset from that of the saccade CD on the RF shifts. (2) The study integrated neuronal activities from 50 to 350 ms after the probe onset. This large time window must average attentional and CD effects together, and the strong attention from the target onset could mask the CD effect. Additionally, around the saccade onset, neurons in LIP (one synapse away from the FEF) exhibit progressive RF shifts from cRF to fRF,⁹ encompassing the entire portion of the space between them (Figure 1A). The large time window could average response from the enlarged RF and underestimate the maximum forward-shift amplitude. Neupane et al.¹⁸ reported both forward and convergent RF remapping in V4. Although the V4 remapping time course is too slow to contribute to transsaccadic visual stability, the study raises the possibility for the existence of both types of remapping in the LIP and FEF. Additionally, it would be interesting to compare the LIP and FEF.

Here, we used a delayed saccade task to separate the attentional effect of the target onset and the effect of the saccade CD and recorded LIP and FEF single-unit activities evoked by probes at different locations and times. The task allowed us to investigate the time course of the RF remapping in detail. We found that in the delay period, RFs shifted slightly toward the initial fixation 50–100 ms after the probe onset but shifted toward the target 250–300 ms after the probe onset. In the perisaccadic period, RFs shifted toward the target 50–100 ms after the probe onset, but 25–75 ms after the saccade onset, the shifts became predominantly forward, toward fRFs. When we integrated neuronal activities from 50 to 350 ms after the probe onset, perisaccadic RFs were still closer to the fRFs than to the target, indicating stronger forward than convergent remapping. Since it first appeared in the delay period when the saccade was suppressed, convergent remapping is not really perisaccadic but attentional.

To explain our data, we constructed a circuit model by integrating CD-gated directional connections for forward remapping⁹ and attention-modulated center/surround connections^19,20 for convergent remapping. We further demonstrated that both sets of connections emerged automatically in artificial neural networks trained to update retinal positions of stimuli across saccades. Mechanistically, the center/surround connections provide attractor dynamics to represent the retinal position of a stimulus as a population activity bump, and the CD-gated connections move the activity bump for transsaccadic updating. The result has the surprising functional implication that the center/surround connections might not only be modulated by attention to produce convergent RF remapping but might also work synergistically with the CD-gated connections to enable accurate spatial perception across saccades.

RESULTS

We used a delayed saccade task (Figure 1C) to sample a cell’s responses from a grid of spatial positions (tailored for each cell according to pilot RF mapping) and four time epochs (after the monkeys achieved initial fixation, after the appearance of the target, after the disappearance of the fixation [the go signal for the saccade], and well after the saccade, respectively). In a given trial, we flashed one probe stimulus in each epoch at a random grid position, and across trials, we sampled all grid positions and epochs. We name the four epochs pre-target (current), delay, perisaccadic, and post-saccadic (future) periods (Figure 1D) and denote a cell’s RFs mapped in these periods as its cRF, dRF, pRF, and fRF, respectively. We recorded a total of 391 and 427 single units from the LIP and FEF, respectively, in 3 macaques. We then screened the data to select cells with significant visual responses, well-sampled RFs, and significant RF shifts in the delay or perisaccadic epoch (STAR Methods). Table S1 shows the numbers of remaining cells after each screening step.

The direction and amplitude of RF remapping changed with time

We measured a cell’s remapping in the delay and perisaccadic periods as the shifts of its dRF and pRF centers relative to its cRF center and defined the forward direction as the direction from the cRF center to the fRF center. We present the RF heatmaps of example cells in Figure 2A and the population shift directions in Figure 2B. Only cells with significant RF shifts in the delay or perisaccadic periods (STAR Methods) were included in the population analysis. For the delay period, the dRFs obtained 50 to 100 ms after the delay probe onset shifted slightly in directions between the initial fixation and the target (Figures 2A, second column, and 2B, first column), but 250 to 300 ms after the delay probe onset, the shifts turned more toward the target (Figures 2A, third column, and 2B, second column). For the perisaccadic period, the pRFs obtained 50 to 100 ms after the perisaccadic probe onset shifted toward the target (Figures 2A, fourth column, and 2B, third column). However, 25 to 75 ms after the saccade onset, the pRFs shifted mostly forward, toward the fRFs (Figures 2A, fifth column, and 2B, fourth column). The mean shift directions were all significant except the early delay period of the LIP (see the p values in Figure 2B, plots). The mean shift directions changed significantly with time in both the LIP and FEF (Figure 2B, caption). In Figures S1–S8, we show that these results were robust against variations in analysis parameters and methods. Figure S6 shows the distributions of the shift vectors (with both directions and amplitudes).

Figure 2. RF remapping in the LIP and FEF.

(A) RF heatmaps of an example LIP cell (top row) and FEF cell (bottom row) from different epochs (columns). In each map, the cyan cross, square, and dashed circle indicate the fixation point, saccade target, and eye position, respectively. The small black cross marks the RF center. The cyan lines in dRF and pRF maps indicate the center shift relative to the cRFs. The scale of normalized responses is shown on the right. The fifth column is based on the saccade onset alignment of the trials, whereas the other columns are based on the probe onset alignment.

(B) The dRF and pRF shift directions of the LIP (top row) and FEF (bottom row) populations from different epochs (columns). In each polar plot, we align the cells’ cRFs at the center and saccade directions along the positive horizontal. The cells’ mean fRF (forward), target, and initial-fixation directions are indicated by the blue, red, and green squares, respectively. Each open dot represents a cell’s RF shift direction, and the thick black line represents the circular mean whose significance is indicated by the p values from Rayleigh test. The dashed orange lines mark the 95% confidence interval of the mean direction. The circular histogram shows the distribution of the cells’ shift directions. The mean shift directions changed significantly across time in both the LIP (p = 1.0 × 10⁻⁵, F_3,283 = 9.0) and FEF (p = 2.1 × 10⁻⁹, F_3,316 = 15.5), with Watson-Williams multisample test. The fourth column is based on the saccade onset alignment of the trials, whereas the other columns are based on the probe onset alignment. The cell numbers (N) of the panels are different because the screening method was applied to each area, epoch, and alignment separately.

To understand these results, note that in the delay period, the monkeys had to maintain the initial fixation and suppress any saccades. This must increase the amount of attention to the fixation in the delay period compared to the pre-target period, producing RF shifts toward the fixation. As time progressed, the monkeys must allocate more attention to the target in anticipation of the pending saccade, and RFs shifted more toward the target. Also note that the perisaccadic probe was flashed before the saccade onset (STAR Methods). Therefore, the forward shifts of pRFs around and after the saccade must reflect the CD-based predictive remapping instead of reafferent visual inputs at changed eye positions (as there was no new perisaccadic stimulus).

To examine the time course of the delay and perisaccadic RF remapping in detail, we used a moving window of 50 ms to analyze the dRF and pRF shifts as a functionoftime.¹⁸ For the delay period, the shift direction changed gradually from a direction between the initial fixation and target to the target (Figure 3A, green). For the perisaccadic period, the shift direction changed gradually from the target to the fRF (Figure 3B, green). When the shift directions pointed toward the initial fixation and, subsequently, toward the target, the shift magnitudes were well short of the distance between the cRF and the fixation or target (Figure S6). When the shift directions pointed to the fRF, the shift magnitudes approached the distance between the cRF and fRF (Figures 3, purple, and S6). This forward shift would provide an adequate transsaccadic update of the retinal location of the probe stimulus, which was flashed before the saccade (see models below).

Figure 3. Remapping time courses.

(A) Delay period (dRF).

(B) Perisaccadic period (pRF). LIP and FEF results are in the top and bottom rows, respectively. Each image shows the mean-normalized shift magnitude (purple curve, left y axis) and the mean shift direction (green curve, right y axis) as a function of time. The amplitude of one (purple horizontal line) means a shift magnitude equals the saccade magnitude. The average fRF, target, and initial-fixation directions are indicated by the green dotted, dashed, and dot-dashed lines, respectively. Each data point is calculated from the responses of the 50 ms window centered around that point. The light purple and green regions indicate 1 SEM (number of cells indicated in each plot). The third column is based on the saccade onset alignment of the trials, whereas the other columns are based on the probe onset alignment.

When we used the larger time interval used by Zirnsak et al. (50–350 ms after the onset of perisaccadic probe), we did find the partial shifts toward the target that they described (Figure 4A). This is not surprising given that the time interval they used included both the time in which we found convergent remapping and the time in which we found forward remapping. Note that the cells’ pRF centers were significantly closer to their fRF centers than to the targets (see tests in Figure 4B), indicating that, on average, the forward remapping was stronger than the convergent remapping during the perisaccadic period in both the LIP and FEF. This is consistent with the distributions of shift directions in Figure 2B: the period with a mean forward-shift direction (forth column) has a more focused directional distribution than does the period with a mean convergent-shift direction (third column).

Figure 4. Analysis of our perisaccadic data using the method of Zirnsak et al.

The LIP and FEF results are shown in the top and bottom rows, respectively. The polar plot of remapping directions (A) has the same format as that of Figure 2B. In (B), each cell’s pRF-to-target distance is plotted against its pRF-to-fRF distance. The p values in (B) indicate that for both the LIP and FEF, the pRFs were significantly closer to the fRFs than to the targets on average (two-sided Wilcoxon signed-rank test).

When a cell’s cRF center is close to the fixation (small eccentricity), the forward and convergent directions become similar (cf. Figures 1A and 1B). Zirnsak et al.¹⁴ used this fact to argue against previous findings of forward remapping. Figure S9 shows the eccentricity distributions of our LIP and FEF cells. They are comparable to the distributions of Zirnsak et al.’s cells (their extended data Figure 4). Therefore, the two studies should have similar discriminability of the remapping directions.

A circuit model for forward and convergent remapping

To understand our data, we first constructed a circuit model to generate the two types of observed RF remapping. We then trained artificial neural networks to explore functional implications of the required connectivity patterns in the circuit model. For the circuit model, we considered a 2D array of cells with their RF centers topographically arranged (Figure 5A, black circles). A rightward saccade is to be made from the cross to the square, and we record from the filled black cell. The above physiological results suggest that two mechanisms may be responsible for the convergent and forward RF shifts, respectively. The first mechanism is an attention-modulated circuit for convergent shift. Inspired by the physiological evidence of center-excitatory/surround-inhibitory modulation of visual responses around the saccade target in both the LIP¹⁹ and FEF,²⁰ we hypothesized center-excitatory/surround-inhibitory connections among the cells according to the distance between their RF centers (Figure 5A, red lines; only the connections from the cell at the target location are shown). When there is attention at a location, we assume that connections from the cells tuned to that location are enhanced. This center/surround connectivity in the spatial domain is similar to that in the orientation domain.^21–24 Such connectivity simulated convergent shifts of orientation tuning curves after perceptual learning or adaptation^23–25; here, we used it to account for convergent shifts of spatial tuning (i.e., RFs) induced by attention.

Figure 5. The circuit model for explaining both forward and convergent RF shifts.

(A) Schematic model structure. Black circles represent RF centers of a 2D array of topographically arranged LIP or FEF cells. The small cross and square indicate the initial fixation and target positions, respectively. The filled black circle is the cRF of the cell under recording, and the gray circle is its fRF center. Only a small fraction of the connections are shown for clarity.

(B) The attention-modulated symmetric connectivity (red) and CD-gated anti-symmetric connectivity (blue, for rightward saccades) as a function of the difference between two units’ preferred locations (RF centers).

(C) Simulations of RF shifts in the delayed saccade task. The first and last images represent a model cell’s cRF and fRF, respectively. The second and third panels represent the cell’s early (dRF1) and late (dRF2) RFs in the delay period. The fourth and fifth images represent the cell’s early (pRF1) and late (pRF2) RFs in the perisaccadic period. In each image, the cyan cross and square are the initial fixation point and target, respectively, and the dashed cyan circle approximates the eye position. The black cross marks the RF center. The thin cyan line in each dRF or pRF image indicates the shift from the cRF center to the dRF or pRF center.

The second mechanism is the CD-gated directional connections that can explain perisaccadic forward expansion of RFs, as shown by Wang et al.⁹ The CD signal comes from the SC, which has a map of saccade directions. We assume that SC neurons for a given saccade direction gate the connections between the LIP/FEF neurons whose RFs are aligned with the saccade axis. In Figure 5A, only a small subset of the connections for the second-row cells gated by the CD for a rightward saccade are shown (blue lines). These connections are normally off. However, around the onset of the saccade, they are turned on by the CD signals, allowing stimulation at the cell’s fRF (gray circle), and at the region between the fRF and cRF, to propagate backward to the recorded cell, generating perisaccadic forward RF shifts.

We considered a 2D array of 50 × 50 LIP/FEF units covering a space of 50° × 50°, each receiving feedforward visual inputs and recurrent inputs from other units via the attention-modulated center/surround connections and CD-gated directional connections. For the delay period, we divided the attention between the fixation point and the target, and in the perisaccadic period, we introduced the CD signal (STAR Methods). We probed the model with flashes in the four epochs as in the experiment to measure cRFs, dRFs, pRFs, and fRFs. By weighting the two mechanisms differently, it is straightforward to generate cells with various degrees of forward and convergent shifts as we found in the LIP and FEF (an example shown in Figure 5C).

The model makes two predictions (Figure 6, top row). (1) Since the CD-gated connections propagate neuronal responses from a cell’s fRF to its cRF, a distance equal to saccade amplitude, the pRF forward-shift amplitude should increase with the saccade amplitude. (2) In contrast, the strengths of the attention-modulated center/surround connections depend on the distance from the attentional locus, so the pRF convergent shift amplitude should vary with the distance between a cell’s cRF center and the target. cRFs near the target have little room to shift toward the target, and those far away are barely affected by attention at the target. There is thus an intermediate, optimal distance for maximal convergent shift. To test these predictions, we pooled the LIP and FEF cells whose pRFs from 0 to 100 ms after the saccade onset shifted between the fRF and target directions and did a parallelogram decomposition of each shift vector into its forward and convergent components. The results (Figure 6, second row) are consistent with the predictions. To test whether the convergent shift peaked at an intermediate Crf-to-target distance, we divided the distance range of the data points (Figure 6, middle right) into three equal intervals and found via ANOVA that the mean shift changed significantly with the distance interval (p = 0.0094, F_2,90 = 4.9). We then did post hoc two-tailed t tests between adjacent distance intervals and found that the mean shift of the middle interval is significantly greater than that of the first interval (p = 0.0025, t₈₀ = 3.1) but not significantly greater than that of the third (p = 0.42, t₅₅ = 0.80). This is because the third interval had insufficient data points due to a technical limitation: when the cRF-to-target distance is large, it is difficult to keep all RFs of a cell within the display area. This aspect of the model prediction needs further verification.

Figure 6. Test the model predictions.

First row: the predicted forward-shift amplitude as a function of saccade amplitude and the convergent-shift amplitude as a function of the cRF-to-target distance. Second row: the pooled LIP/FEF pRF data for testing the predictions. Third row: fitting the model to the data. For the second and third rows, the blue lines in the left column are linear fits, and the red curves in the right column are the moving average with a window size of 10°. For the forward component, the slopes of the data and the model are not significantly different from each other (p = 0.79, t₁₈₉ = 0.26, t test). For the convergent component, the distributions of the data and the model are not significantly different from each other (p = 0.12, Peacock’s 2D Kolmogorov-Smirnov test).

To explain our data more quantitatively, we obtained distributions of the model parameters by fitting the model to the perisaccadic LIP and FEF data and interpolated each parameter distribution as a mixture of Gaussians (STAR Methods). We then randomly resampled parameters from these distributions and ran the model to obtain the pRF shifts. The results (Figure 6, bottom row) matched the data well (see Figure 6’s caption).

The emergence of the required connectivity patterns in neural networks trained to update retinal positions across saccades

Both the attention-modulated center/surround connections and the CD-gated directional connections in our circuit model are motivated by physiological evidence in the LIP and FEF. Nevertheless, one may argue that the model is ad hoc, designed specifically to explain the forward and convergent remapping. Is there a simple, functional consideration that leads to both connectivity patterns automatically? As we noted above, pRF remapping appears to update the retinotopic location of remembered (and disappeared) stimuli across saccades, a requirement for performing the double-step memory saccade task (introduction). We therefore hypothesized that the two sets of connections in the circuit model are for such transsaccadic updating, with the center/surround connectivity for storing the retinal position of a stimulus of interest^26–28 and the CD-gated connectivity for updating the memory across saccades.^4,9,29

We tested this hypothesis by training neural networks on the predictive updating task and examining whether randomly initialized connections converge to the required patterns after training. We considered horizontal saccades of opposite directions only; saccades along other axes can be similarly treated. The neural networks consisted of two layers of units. The first layer provided visual inputs to the second layer, which simulated LIP/FEF cells. The second-layer units were trained to produce activity patterns representing the correct retinal locations of input stimuli across saccades. The connections from the first- to second-layer units were translationally invariant³⁰ (convolutional) so that retinotopic inputs specified the feedforward component of LIP/FEF RFs. The second-layer units were fully and recurrently connected to each other with three sets of weights. The first two sets were gated by CD signals for saccades of opposite directions, whereas the third set was not gated by the CD signals but could be optionally modulated by attention (STAR Methods). All connection weights in the network were randomly initialized. The visual input in the first layer was a Gaussian bump of population response centered at the initial retinal location of a stimulus (Figure 7A, left, for a 50 ms stimulus). Two additional input units provided CD signals for opposite saccade directions. The desired output in the second layer was the same Gaussian bump centered at the correct retinal position of the (disappeared) stimulus across saccades (Figure 7A, middle, for a rightward saccade started at 150 ms and thus a leftward displacement of the representation for the stimulus’ retinal position). The weights were trained by minimizing the quadratic difference between the actual and desired outputs. Many variations of the simulation produced similar results (see the supplemental information).

Figure 7. Automatic generation of the required connectivity patterns in the circuit model by training neural networks.

We trained the networks to predictively update retinal positions of stimuli across saccades. (A) and (B) are for the case of brief input stimuli, and (C) and (D) are for the case of persistent input stimuli. (A) and (C) show the test input (not included in the training), the desired output, and the actual output; all plots represent the population responses of units tuned to different retinal positions (x axis) as a function of time (y axis). (B) and (D) show the average connection weights as a function of the difference between two units’ preferred positions (RF centers). The red and blue curves are for the attention-modulated and CD-gated connections, respectively. Only the CD-gated connections for rightward saccades are shown. The shaded areas indicate 1 SEM (N = 50).

After the training converged and the actual output resembled the desired output for test inputs not used during the training (Figure 7A, right), we determined the units’ mean connection weights to other units as a function of the distance between their preferred positions. Figure 7B shows the results, with the attention-modulated and CD-gated connections in red and blue, respectively. The CD-gated connections shown are for rightward saccades, and the mirror pattern for leftward saccades was also learned (data not shown). Remarkably, these connectivity patterns closely resemble those we chose for the circuit model in Figure 5B. Note that the connections in Figures 7B and 5B are comparable only in their shapes, not in their scales. This is because the circuit model and the artificial neural networks used different scales to represent the units’ activities.

We then repeated the above neural network training but with persistent visual stimuli (Figure 7C). Interestingly, we obtained very similar results, with not only the CD-gated directional connections but also the center/surround connections (Figure 7D). The reason is that during saccades, the desired output position is different from the input position (Figure 7C, middle and left). When the eye is moving, a static stimulus sweeps new retinal positions, but it takes time for this new input to reach the LIP/FEF (reafference delay). In contrast, the desired output should reflect the correct retinal position of the stimulus without the reafference delay. The symmetric center/surround connectivity is needed to create an attractor activity pattern, which was updated by the asymmetric CD-gated directional connectivity, independent of the input activity pattern.³¹ We also considered the case where both brief and persistent input stimuli were trained together and where the attentional modulation was turned off and again obtained similar results (see Figures S10 and S11).

We conclude that the connectivity patterns required by the circuit model emerge automatically and robustly in neural networks trained to update the representation of stimuli’s retinal positions across saccades. As in the circuit model, the center/surround connections and the CD-gated directional connections in the trained networks can generate convergent and forward RF remapping, respectively. Indeed, the backward shift of the population activity in the output of Figure 7 is equivalent to the forward shift of the RFs.¹³ Although the center/surround connections and the CD-gated connections in the circuit model are for explaining convergent and forward RF shifts, respectively, our neural network models suggest that they work synergistically to enable transsaccadic visual stability.

DISCUSSION

In the 19^th century, Hermann von Helmholtz examined a patient who was blind in one eye from diabetes and sustained a paralysis of the lateral rectus muscle in the normally seeing eye.³ When the patient tried to look in the direction of the paralyzed muscle, he perceived that the visual world moved in the opposite direction and then drifted back. Helmholtz postulated that under normal conditions, the brain uses the oculomotor signal to feedback to the visual system and adjust for the saccade. The discovery of perisaccadic forward remapping^4–8 provided a physiological mechanism for Helmholtz’s theory. Neuropsychological evidence also supports this theory: patients with parietal lesions^32,33 cannot perform the double-step task when the subject makes the first saccade in the direction contralateral to the lesioned cortex. Furthermore, monkeys with inactivation of the medial dorsal nucleus of the thalamus, which relays CDs of saccadic commands from the SC to the FEF, cannot perform the double-step task, and their FEF neurons do not exhibit perisaccadic forward remapping.^29,34 However, Zirnsak et al. questioned the existence of forward remapping¹⁴ and instead showed that when they analyzed the activity of FEF neurons in the interval from 50 to 350 ms after the appearance of probe, which appeared around the time of the saccade target, many neurons seemed to remap their RFs toward the target.

In humans³⁵ and monkeys,¹⁶ the abrupt onset of a visual stimulus evokes attention, as measured by a change in perceptual threshold, as does the planning of a saccade.^16,36 Because of the large time interval (50–350 ms after probe onset) used by Zirnsak et al., we wondered if their results might be confounded by the presence both of an attentional event (the appearance of the saccade target) and the generation of a saccade, which would result in the combination of convergent remapping evoked by the saccade target and forward remapping evoked by the CD of the saccade command. Here, we addressed this controversy by using a delayed saccade task to separate the appearance of the target and the generation of the saccade. We recorded from the LIP and FEF with matched procedures and found that the LIP and FEF showed similar patterns of remapping, which varied with time: during the perisaccadic period, RFs converged toward the target shortly after the probe onset, but around the time of saccade and onward, the shift directions became predominantly forward, toward the fRF. When we used a large time window to integrate perisaccadic activities,¹⁴ the pRFs were closer to the fRFs than to the targets, indicating stronger forward remapping than convergent remapping. We further found that the convergent shift started in the delay period when the saccade command, and its CD, must be suppressed, and the shift direction turned from between the initial fixation and the target to the target. Thus, unlike the forward shift that depends on the saccade CD,²⁹ the convergent shift appeared to follow attention from the initial fixation to the target. We conclude that both types of remapping are present in the LIP and FEF and that forward remapping is not an artifact of under-sampling convergent remapping. The convergent and forward RF shifts may be viewed as attentional remapping and perisaccadic remapping, respectively. These two types of remapping have also been found in V4.¹⁸ However, because the time course of V4 remapping is much slower than that of the LIP/FEF, V4 cannot contribute to transsaccadic visual stability but might inherit its remapping from the LIP/FEF.

Because our delayed-saccade paradigm helps distinguish between the forward- and convergent-remapping mechanisms, we were able to construct a circuit model for both types of remapping. Specifically, we integrated attention-modulated center/surround connections and CD-gated directional connections to explain the convergent and forward RF shifts, respectively. The model’s predictions on the forward-shift amplitude as a function of the saccade amplitude and the convergent shift amplitude as a function of the cRF-to-target distance are consistent with the data. We then showed that both sets of connections emerged automatically and robustly in neural networks trained to update representations of retinal positions across saccades. Since this updating is needed for the double-step memory saccade task, it can be viewed as an operational definition of transsaccadic stability. We suggest that the CD-gated connections and the center/surround connections together specify a mechanism for transsaccadic stability. The mechanism follows a classic prescription³¹: symmetric center/surround connections produce attractor dynamics to represent a stimulus as a population activity bump, whereas the asymmetric CD-gated connections move the activity bump for updating across saccades. Although we initially used the center/surround connections to explain convergent remapping, they might be an integral part of the transsaccadic stability mechanism.

In addition to the fixation and target, the flash of the probes must also attract attention. However, since the probes were presented at many spatial locations covering a large area, the attentional effect of the probes, unlike that of the fixation or target, must be spread out spatially across trials and thus is relatively weak. More importantly, the probes were flashed over the same area to measure all RFs of a given cell. Since remapping is the difference between a cell’s dRF/pRF and cRF, the attentional effects of the probes must be canceled by the subtraction.

Zirnsak et al.¹⁴ suggested that convergent RF remapping explains compressive perceptual mislocalization: stimuli flashed briefly around the saccade onset are perceived as occurring at the saccade target when post-saccadic visual references are present.^37,38 However, whether convergent remapping produces compressive mislocalization is unclear and depends on, among other things, whether the positional decoder is aware of the remapping.¹³ Moreover, when saccadic adaptation is used to dissociate the post-saccadic eye position and the target position, the perceived compression is toward the eye position, not the target position,³⁹ whereas visual and movement responses in the LIP and SC represent the target position, not the eye position.^40,41 Therefore, one would not expect convergent remapping (in the LIP and SC at least) to explain compressive mislocalization. Additionally, a key timing difference between the existing perisaccadic remapping and mislocalization studies makes it difficult to compare them (see below).

In addition to perisaccadic RF remapping, a prominent physiological finding relevant to transsaccadic perceptual stability is the gain fields, the modulation of visual response by eye position.⁴² Whereas perisaccadic remapping may realize the stability by predictively updating retinal representations across saccades, gain fields may do so by combining eye position and retinal representations to form head-centered representations.⁴³ Recent studies suggest that gain fields and perisaccadic remapping may be responsible for transsaccadic stability at long and short timescales, respectively,^44,45 consistent with their respective dependence on slow proprioceptive eye-position signals and fast saccade CD signals.^29,46 On the other hand, it is theoretically possible to integrate fast CD signals to provide fast, predictive eye-position signals. The existence of such CD integrators is an open question for future research.

Our study may also have functional implications on potential relationships between working memory and attention. According to our models, although the center/surround connections may store working memories of visual stimuli including saccade targets, the same connections can be modulated by attention to generate convergent RF shifts. The relationship between working memory and attention has been discussed in the literature.⁴⁷ Our work, however, suggests a specific mechanism: attention to a stimulus modulates the connections that are responsible for storing the stimulus in working memory. Therefore, LIP and FEF circuits might integrate mechanisms for working memory, attention, saccade planning, and transsaccadic visual stability all together.

Limitations of the study

At noted in the results, because of the technical difficulty of measuring all RFs of a cell completely at large cRF-to-target distances, we do not have sufficient data with large cRF-to-target distances to test the prediction that convergent RF shifts have a peak at an intermediate distance (Figure 6, middle right). Convergent shifts should decrease at large cRF-to-target distances because of the limited range of influence of attention at the target. Therefore, if this prediction fails, it would cast doubts on the assumption that convergent shifts are produced by attention.

Our models do not take cortical magnification into account. In our simulations, each pixel represented a fixed visual angle of space. One way to imagine how the simulation results may change with the inclusion of cortical magnification is to let each pixel represent an increasingly larger extent of space as the eccentricity increases. This would scale up the RF size, and scale down the cell density and connection density, with the eccentricity but would be unlikely to qualitatively change the effects of CD-gated and attention-modulated connections on RFs.

Although the connectivity patterns in our circuit and neural network models are plausible based on relevant physiological data, we do not have any direct evidence for them. They should be viewed as predictions for future tests.

Ideally the time course of forward RF remapping should match that of the saccadic eye movement, with the shift magnitude equal to zero and saccade amplitude at the saccade onset and offset, respectively. Then, the corresponding population activity representing a stimulus would move backward by the saccade amplitude, perfectly updating the stimulus’ retinal position across the saccade. Indeed, this is how our trained neural networks solve the updating problem, using the center/surround connections to maintain the population activity bump and the CD-gated directional connections to move the bump. However, the actual time course of forward remapping starts a little before the saccade onset (at least in the LIP) and approaches the saccade amplitude a little after the saccade offset (Figure 3B, right column). This is good enough for performing the double-step saccade task because the updating of the second target’s retinal position only needs to be completed within the reaction time of the second saccade. However, the perceptual consequence of the difference between the ideal and the observed forward remapping is not understood. As noted above, our previous analysis indicates that one critical factor is whether or not perceptual decoders are aware of the remapping.¹³ Additionally, it is unclear whether saccadic control and visual perception use the same or different positional decoders. Finally, perisaccadic RF remapping studies and perisaccadic perceptual mislocalization studies differ in timing and thus may not be comparable: the former present perisaccadic stimuli before the saccade onset and measure remapping at different times across the saccade, whereas the latter present stimuli at different times across the saccade and measure perception (or perceptual memory) well after the saccade. Further studies are needed to resolve the issue.

STAR★METHODS

Detailed methods are provided in the online version of this paper and include the following:

RESOURCE AVAILABILITY

Lead contact

Further information and requests for resources should be directed to and will be fulfilled by the lead contact, Ning Qian (nq6@columbia.edu).

Materials availability

This study did not generate new unique reagents.

Data and code availability

• The data for this paper are available at: Mendeley Data: https://doi.org/10.17632/w6y53574zp.1.

• The data analysis and simulation codes are available at: https://github.com/XiaoW2633/CellReport, https://doi.org/10.5281/zenodo.10672764.

• Any additional information required to reanalyze the data reported in this work paper is available from the lead contact upon request.

EXPERIMENTAL MODEL AND STUDY PARTICIPANT DETAILS

Animals

Three male adult rhesus monkeys (Macaca mulatta) weighing from 9 to 11 kg participated in this study. All procedures were approved by the ethics committee at Beijing Normal University, and by the Columbia University IACUC as being compliant with the NIH Guide for the Care and Use of Laboratory Animals. We have complied with all relevant ethical regulations. There is no evidence for sex-related differences in the basic visual function (transsaccadic space perception) we studied here.

METHOD DETAILS

Animal preparation

We surgically implanted two subconjunctival search coils⁴⁸ (one for each eye; Crist Instrument Sclera, sample rate at 2.7KHz), a head restraint post, and two recording chambers (for LIP and FEF, respectively; PEEK), for each monkey. We positioned the recording chambers according to our experience and/or the MRI scans. We centered the LIP chambers for the three monkeys, respectively, at 3, 10, and 3.2 mm posterior to the interaural plane, and 13, 15, and 15 mm lateral from the middle line. We centered the FEF chambers for the three monkeys at 28, 18, and 23.5 mm anterior to the interaural plane, and 13, 15, and 18 mm lateral from the middle line. The two recording areas were verified later (see below).

Recording procedures

Monkeys sat 57 cm away from a 55-inch Samsung Monitor (60 Hz, 1920×1080 pixels). We used insulated tungsten microelectrodes (0.3–1.0 MU, FHC) to record single-unit activity. We inserted the electrodes though dura via stainless steel guide tubes, and controlled their advancement in the cortices with micromanipulators (NAN Instruments). Neuronal activities collected by the electrodes were amplified (Alpha Omega) and filtered (268–8036 Hz) before online sorting with AlphaLab SnR (Alpha Omega). We identified LIP based on persistent activities in the delay period of a memory saccade task,⁴⁹ and FEF according to micro-stimulation (100 ms, 0.05 mA, biphasic pulses) evoked saccades of fixed vectors.^50,51 After several recording sessions, we did MRI scan of the first two monkeys’ LIP chambers and the third monkey’s LIP and FEF chambers, confirming that the LIP recording sites were within the lateral bank of the intraparietal sulcus, and the FEF recording sites were in the anterior bank of arcuate sulcus. The same recording and analysis procedures were applied to LIP and FEF.

After isolating a single unit with a template-matching method, we first did a pilot mapping of its visual RF: while the monkey maintained central fixation in each trial, we flashed a sequence of 6 probe stimuli (1° × 1°) at random locations sampled from an 8×8 array with adjacent locations separated by 6° in both horizontal and vertical directions. A probe lasted 33 ms and successive probes were separated by 400 ms. Each location had about five responses. If visual inspection determined that the unit showed clear responses for at least one probe location, we moved on to the main, delayed saccade task (Figure 1). We tailored the array of probe positions to cover the unit’s cRF-fRF region and the target region, according to the pilot RF mapping and the planned saccade target for the unit. Across cells, the array varied from 4×5 to 10×12 positions, with 5×8, 5×9, and 6×8 the most common. The spacing between adjacent positions (along both horizontal and vertical axes) varied from 2° to 6°, with 6 the most common. The saccade amplitude varied from 5° to 30°, with 15° and 20° the most common. Despite our effort, the RFs of some cells were not measured sufficiently complete because of the limited display area and large RFs and/or large saccades; these cells were excluded (see below). As shown in Figure 1 and described in the text, the delayed saccade task allowed us to measure a cell’s cRF, dRF, pRF, and fRF from the pre-target (current), delay, perisaccadic, and postsaccadic (future) epochs of a trial.

In the actual experiments, the initial fixation point and the target were both red squares of 0.3° width, but for the ease of illustration, we represented them as cyan crosses and squares, respectively, in the figures.

Data analysis

We screened and processed the data as follows. (1) We selected the cells with significant visual responses. For each epoch, we aligned repeated trials to the probe onset. For the perisaccadic epoch, we additionally aligned the repeated trials to the saccade onset. For each epoch and probe position of a cell, we calculated the response as the mean firing rate 50–150 ms after the probe onset or 0–100 ms after the saccade onset (these windows were chosen because they contained most of the activities), and the baseline as the mean firing rate 0 to 50 ms before the probe onset. We found the probe position that had maximal response, and then performed a single Wilcoxon rank-sum test (two-sided) against the corresponding baseline at the 0.05 significance level. This procedure avoided multiple comparisons over all the probe positions. Cells were selected separately for each epoch and alignment. Every selected cell in an epoch/alignment has to pass the tests in that epoch/alignment, regardless of its tests in other epochs/alignment. Therefore, there is no accumulation of type 1 error and no need to correct for multiple comparisons across the epochs/alignments. For a selected cell in an epoch, we followed Zirnsak et al.¹⁴ to normalize its spatial responses according to (r_k−r_min)/(r_max−r_min) for all k, where r_k is the response at probe position k, and r_max and r_min are the maximum and minimum responses across all probe positions. An advantage of this normalization is that because of the subtraction of r_min, any non-visual (such as saccade related) responses are discounted. Also note that because we always place the target outside a cell’s RF, saccade contribution to the measured responses must be minimal. We obtained similar results without the normalization procedure (see below). (2) We selected cells with well-measured RFs. For each epoch and alignment of a cell, we linearly interpolated the normalized responses across positions to obtain the RF heatmap. We traced the response contour at 85% of the maximum (contour criterion) and required that each probe position within the contour had at least 5 trials and that 80% of contour were covered by the sampled positions (completeness criterion). We then estimated the RF center as the center-of-mass of the responses within the region set by the contour criterion. We used the 85% contour criterion instead of Zirnsak et al.’s¹⁴ 75% because the higher value determined the RF center more reliably. We obtained similar results with different values for the two criteria (see below). Note that the contour criterion was for selecting a region around the main response peak for calculating the center of an RF, but not for measuring the total size of an RF that includes all statistically significant responses. (3) We selected cells with significant RF shifts. For each cell, we calculated the shifts of its dRF and pRF centers relative to its cRF center in visual angles, and determined the significance of a shift via the following bootstrapping.⁵² For each epoch and probe position of a cell, we assumed that the spike count of a trial followed a Poisson distribution with the mean equal to the measured mean spike count. We then simulated the recording and analysis of the cell, with 1000 repeats, by sampling spike counts from the distributions with the trial numbers equal to those of the actual experiment in each repeat. To determine, for example, whether the 1000 dRF centers shifted significantly from the 1000 cRF centers in the simulation, we calculated their overlaps along the axis linking the mean dRF center and cRF center, and required the overlap to be less than 5%. The total number of the recorded LIP and FEF cells and the numbers of the cells that survived each screening step are shown in Table S1. After the screening, we investigated how remapping changed with time by choosing various 50 ms windows to determine dRFs and pRFs as detailed in the main text. The Poisson distribution used above may underestimate the variance of neural activities. We obtained similar results when we replaced it by a non-parametric procedure (see below).

We considered each recorded cell as a distinct sample. After the screening described above, the numbers of cells for different time periods were different, and for this reason, results from different time periods could not be treated as repeated measures. We thus used Watson-Williams test to determine whether remapping directions changed significantly over time in Figs. 2, S1–S5, and S7–S8.

We also applied Zirnsak et al.’s method¹⁴ to analyze perisaccadic remapping in our LIP and FEF data (Figure 4). In addition to screening for cells with sufficient trials, well-sampled RFs, and significant RF shifts in perisaccadic epoch, we selected the trials in which perisaccadic probes occurred within 150 ms before saccade onset, used the responses from 50 to 350 ms after the probe onset for all epochs, and set the contour criterion to 75%.¹⁴

We used MATLAB to perform the data screening and analysis. For statistical tests, we used MATLAB with one exception: we used R to test the slope difference between the data and the model fit in Figure 6 (left column) as the interaction between the category (data or model) and the saccade amplitude.

The contour and completeness criteria were the two key parameters described above, set to 85% and 80%, respectively, in the main text. We did additional data analysis to demonstrate that our physiological conclusions are robust against variations in these criteria. We focused on Figure 2B of the main text as it contained the main results on the RF shift directions in the delay and perisaccadic periods for both LIP and FEF. In Figs. S1 and S2, we kept the contour criterion at 85%, but set the completeness criterion to 90% and 70%, respectively. In Figure S3 to S5, we changed the contour criterion to 75%, a value used by Zirnsak et al.’s,¹⁴ and set the completeness criterion to 90%, 80%, and 70%, respectively. These figures all show results similar to those in Figure 2B of the main text.

In the main text, we showed the distributions of the RF shift directions at four time points (Figure 2B). For completeness, we show in Fig. S6 the distributions of the shift vectors (both the directions and amplitudes).¹⁸

For the main text, we normalized a cell’s responses from different probe positions of an epoch according to (r_k−r_min)/(r_max−r_min) for all positions k¹⁴ where r_k is the response at position k, and r_max and r_min are the maximum and minimum responses across all positions. To show that the normalization does not affect our conclusions, we reanalyzed our data without the normalization. Specifically, from each mean visual response to a probe position we subtracted the corresponding mean baseline response (from the 50 ms period before the probe onset) and all the other procedures remained the same. The results, shown in the new Fig. S7, are very similar to Figure 2 of the main text which used the normalization. The reason is that the normalization changes the response scale but not the relative responses among probe locations for a given time period. Consequently, the normalization has no impact on the RF location or shape or shift. Fig. S7b is not identical to Figure 2B because for the latter, the minimum response among the probe positions, instead of the baseline response, was subtracted from the response of each probe position. As we noted above, subtracting the minimum response has the advantage of removing any saccade-related responses from the visual responses.

Cells’ responses usually peaked shortly after the probe onset and then started to decline. In addition, saccade suppression may reduce perisaccadic responses. When the same scale is used to plot the RF heat maps from different time periods of a trial, as in Fig. S7a, it may be difficult to see some of the RFs. This is why we normalized RF heat maps from different time periods separately in Figure 2A. To reduce the effect of noise, we used the contour criterion to select the most responsive regions of RFs to estimate the RF centers and their shift vectors, and tested whether a mean shift direction is significant or not.

The Poisson distributions we used in the bootstrapping step for selecting cells with significant RF shifts may underestimate the variance of neural activities. Since there are only several trials for each probe location and time period (typical of single unit recordings), it is difficult to justify any specific forms of probability distribution. Additionally, other probability distributions may have their own problems. For example, although Gaussian distribution allows different mean and variance to achieve any value of the Fano factor, it cannot describe skewed distributions. And skewed distributions are the rule rather than exception because neural activities have the zero cutoff (no negative firing). The only choice seems to be the non-parametric procedure of resampling from the recorded trials with replacement, which ensures that over a large number of repeats, the sampled responses and the actual responses have the same statistics including the variance. However, resampling with replacement also requires large numbers of recorded trials to work well. With our small trial numbers, the procedure might exaggerate fluctuations of activities across probe locations. For example, for a given repeat of bootstrapping, at one location the samples may come from the least responsive trial whereas at another location the samples may come from the most responsive trial, and these extremes will occur at different locations for different repeats. Nevertheless, we used resampling with replacement to select cells with significant RF shifts and re-analyzed our data as a benchmark. The results, shown in Fig. S8, are similar to those of Figure 2B of the main text. Therefore, our conclusions are robust against the variance of neural activities.

During a trial, the saccade could bring edges of the display area into a cell’s pRF2, contributing to the recorded responses. However, among the 104 LIP cells and 113 FEF cells in Figure 2B (last column), there were only 7 cells in each area whose pRF2s included the edges. More importantly, the normalization procedure we used must discount this edge effect: for a given cell with a fixed saccade vector, the edge responses must be the same for different probe locations. When we subtracted the minimum probe response from every probe response in the normalization step, the edge response must be removed.

Circuit model

We simulated a 2D array of 50×50 LIP/FEF units covering a space of 50° × 50°, each unit governed by the equations:

$$\begin{gathered} \frac{\tau du(x,y)}{dt}=-u(x,y)+W(x,y)\mathrm{*}r(x,y)+I \\ r\left(x,y\right)=\alpha \mathrm{m}\mathrm{a}\mathrm{x}[u(x,y),0] \\ \mathrm{I}(\mathrm{x},\mathrm{y},\mathrm{t})=\frac{1}{{\mathrm{b}}^{\mathrm{a}}\mathrm{\Gamma }(\mathrm{a})}{\mathrm{t}}^{\mathrm{a}-1}{\mathrm{e}}^{-\frac{1}{\mathrm{b}}}\mathrm{e}\mathrm{x}\mathrm{p}\left[-\frac{{\mathrm{x}}^2+{\mathrm{y}}^2}{2{\sigma }_{\mathrm{s}}^2}\right] \end{gathered}$$

where $u(x,y)$ and $r(x,y)$ are, respectively, the membrane potential (relative to spike threshold) and firing rate (relative to background rate) of the unit at location $(x,y),\tau =20\mathrm{m}\mathrm{s}$ is the membrane time constant, $\alpha$ is a constant relating $u(x,y)$ to $r(x,y)$ (which affects the model only through its product with $W$, specified below), * denotes spatial convolution, $W$ specifies connections between the units, and $I$ represents the feedforward visual inputs with $a=5$, $b=10\mathrm{m}\mathrm{s}$. For each unit, its connection matrix $W(x,y)$ to other units at relative coordinates $(x,y)$ is a sum of two parts: (1) center-surround connections modeled as a difference between two circularly symmetric 2D Gaussians:

$$W_1=w_{\mathit{exc}}\mathrm{e}\mathrm{x}\mathrm{p}\left[-\frac{x^2+y^2}{2{{\sigma }_{\mathit{exc}}}^2}\right]-w_{\mathit{inh}}\mathrm{e}\mathrm{x}\mathrm{p}\left[-\frac{x^2+y^2}{2{{\sigma }_{\mathit{inh}}}^2}\right]$$

(with $\alpha w_{\mathit{exc}}=4$, $\alpha w_{\mathit{inh}}=2$, ${\sigma }_{\mathit{exc}}={12}^{\circ }$, ${\sigma }_{\mathit{inh}}={18}^{\circ }$), and (2) asymmetric connections with excitation in the opposite direction of the pending saccade⁹; for horizontal saccades, we used the antisymmetric form: $W_2=\beta \partial W_1/\partial x$ along the horizontal axis, where $\beta =12$ determines the relative strength between $W_1$ and $W_2$. (For another saccade axis, $W_2$ should be rotated to the saccade axis.) Many expressions for $W_2$ would work; we choose the derivative of $W_1$ for its known property in shifting activity profile in the direction where $W_2$ is excitatory³¹ and for reducing the number of free parameters. Note that Wang et al.⁹ used the CD-gated excitatory connections against the saccade direction and a global inhibition. We used an equivalent connectivity with inhibitory and excitatory connections along and against the saccade direction, respectively. Figure 5B shows the shapes of $W_1$ and $W_2$ along $x$ for rightward saccade. For the delayed saccade task, $W_1$ is multiplied by an attentional modulation factor: $1+w_{\mathit{att}}\mathrm{e}\mathrm{x}\mathrm{p}\left[-\left(x^2+y^2\right)/\left(2{{\sigma }_{\mathit{att}}}^2\right)\right]$ centered at the attentional locus (initial fixation or target position), where ${\sigma }_{\mathit{att}}={15}^{\circ }$; this is equivalent to multiplying the activities of the relevant units by the same factor. Similarly, $W_2$ is multiplied by a $\mathrm{C}\mathrm{D}$ gating factor: $w_{CD}\mathrm{e}\mathrm{x}\mathrm{p}\left[-{\left(t/{\sigma }_{CD}\right)}^6/2\right]$ where $w_{CD}$ is not independent but determines the $\mathrm{C}\mathrm{D}$ strength through its product with $\beta$, $t$ is measured relative to the saccade onset time, and ${\sigma }_{CD}=65\mathrm{m}\mathrm{s}$. For the six time periods in Figure 5C (cRF, dRF1, dRF2, pRF1, pRF2, and fRF), we set $w_{\mathit{att}}$ at the initial fixation to 0.4, 0.8, 0.6, 0, 0, and 0, $w_{\mathit{att}}$ at the target to 0, 0.2, 0.3, 0.45, 0.2, and 0, and $w_{CD}$ to 0, 0, 0, 0.1, 0.9, and 0, respectively. $w_{\mathit{att}}$ at the initial fixation was larger for dRF1 and dRF2 than for CRF because after the target onset, the monkeys had to suppress any saccades to the target and maintain the initial fixation. Visual inputs from a stimulus to the LIP/FEF units were modeled as a circular Gaussian centered at the stimulus with ${\sigma }_{\mathrm{s}}=7^{\circ }$. We probed the model with flashes in the four epochs as in the experiment to measure cRFs, dRFs, pRFs, and fRFs. Many variations of the model and/or the parameters produced similar results. For example, the attentional modulation and CD-gating functions can be replaced by simple step functions, and the parameters can be optimized to fit individual cell’s RF shifts (see below). We implemented the circuit model with COSIVINA,⁵³ an open source toolbox for MATLAB.

The model predicts that the forward-shift amplitude grows with the saccade amplitude and that the convergent-shift amplitude depends on the CRF-to-target distance, with a maximal shift at an intermediate distance (see text). To make these predictions more quantitative, we obtained distributions of the model parameter set $\mathrm{\Theta }=\left(w_{\mathit{exc}},{\sigma }_{\mathit{exc}},w_{\mathit{inh}},{\sigma }_{\mathit{inh}},w_{\mathit{att}},{\sigma }_{\mathit{att}},w_{CD}\right)$ by fitting the model to the perisaccadic LIP and FEF data. We focused on these parameters as they are most relevant to the RF shifts. We first initialized the parameters to random values, each drawn from the uniform distribution $U(0,20)$. For each recorded cell with a shift vector $\overrightarrow{s}$, we used PyTorch’s differentiation engine and Adam optimizer (both learning rate and weight decay set to 0.01) to perform gradient descent on the parameters by minimizing the cost function:

$$L(\mathrm{\Theta })=\frac{1}{2}{\parallel {\overrightarrow{s}}^{\prime }-\overrightarrow{s}\parallel }^2-\lambda {\overrightarrow{s}}^{\prime }\cdot \overrightarrow{s}$$

where ${\overrightarrow{s}}^{\prime }$ is the shift vector produced by the model. The first term minimizes the difference between $\overrightarrow{s}$ and ${\overrightarrow{s}}^{\prime }$. We included the second, dot-product term because when $\overrightarrow{s}$ and ${\overrightarrow{s}}^{\prime }$ have small amplitudes, the first term can be small even when $\overrightarrow{s}$ and $\overrightarrow{s}$ point at different directions. The second term ensures that the two vectors point in similar directions. We let $\lambda =0.01$ as it produced good fits for all cells. Because of the limited number of recorded cells, we pooled together the optimized parameters obtained from fitting both the LIP and FEF cells. For a given saccade amplitude, we used Scikit-learn’s Gaussian kernel density estimation to fit each parameter distribution, with the bandwidth optimized by the GridSearchCV function. We treated the parameters independently as we did not have nearly enough data to determine their joint distribution. We then sampled parameters from these distributions and run the model to predict the forward-shift amplitude as the function of the saccade amplitude, and the convergent-shift amplitude as a function of the cRF-to-target distance (Figure 6). The cRF centers were resampled from the distribution of the measured cRF centers by fitting a mixture of Gaussians.

Artificial neural networks

We trained artificial neural networks to predictively update retinal locations of stimuli across saccades and demonstrated automatic and robust emergence of both the center/surround connections and the CD-gated directional connections needed in the circuit model for explaining the convergent and forward RF shifts, respectively. A network consisted of two layers of units: the first layer provided visual inputs, originated from retina, to the second, LIP/FEF layer (output). For simplicity, we only considered the horizontal dimension and horizontal saccades. Each layer had 100 units representing 100° of space. The connections from the first to second layer was translationally invariant³⁰ (convolution kernel size of 5) so that the second layer preserved the retinotopic representation of the first layer. There were two additional input units with a one-hot representation of the CD signals for the two opposite directions of saccades. For a given saccade, the relevant CD unit was turned on for the duration of the saccade. The second-layer units are fully and recurrently connected with three sets of weights. The first two sets were multiplicatively gated by the two CD input units for opposite saccade directions, respectively, whereas the third set was optionally modulated by attention. The simulations in Figs. 7 and S10 included attentional modulation at the target over the entire time course (which covered a small time window containing the saccade). The modulation had the same form as that for the circuit model, with ${\sigma }_{\mathit{att}}={15}^{\circ }$ and $w_{\mathit{att}}=0.4$, centered at the target whose initial retinal position was assumed to be at 25° and 75°, respectively, for rightward and leftward saccades of ${25}^{\circ }$ from initial fixation at $0^{\circ }$ and 100°, respectively. In Figs. S11 and S12, we showed an example where we obtained similar connectivity patterns in the absence of attentional modulation. The dynamics of the units was governed by equations identical to those for the circuit model above, and we used the ReLU activation function.

The networks were trained on the task of predictively updating the retinal position of visual stimuli across saccades. Specifically, the output units should have the population activity pattern representing the correct retinal position of an input stimulus across saccades without reafference delay. As in most models, we assume that positional coding of the output units was fixed without changing with transient RF shifts.¹³ Both the input and desired output are Gaussian activity patterns with $\sigma =6^{\circ }$. We considered both brief and persistent input stimuli, with one stimulus per training trial. The brief stimuli appeared for 50 ms before saccades and then disappeared whereas the persistent stimuli stayed for the duration of the simulations. Importantly, the input units provided inputs to LIP/FEF units and we assumed a 50 ms reafference delay from retina to LIP/FEF for persistent stimuli (Figure 7C, left). The output activity pattern was trained to compensate for this delay by using the CD signals (Figure 7C, middle). Therefore, regardless of whether the input stimuli were brief or persistent, the desired output was the same: an activity pattern representing the correct retinal position of the stimuli without delay. This is what we mean by “predictive updating.”

The model was trained to minimize the mean squared error between desired output and actual output as follows:

$$L\left(\theta \right)=\frac{1}{2}\sum _i\sum _t{∥h_{tj}-y_{ti}∥}^2,$$

where $h_{t,i}$ and $y_{t,i}$ are the desired and actual outputs for unit $i$ at time step $t$. All weights were randomly initialized with a uniform distribution $u\left[-\frac{1}{\sqrt{n}},\frac{1}{\sqrt{n}}\right]$, where $n$ is the total number of weights of a given type (feedforward or recurrent) a unit receives. For the feedforward weights, $n=5$ (the convolutional kernel size), and for the recurrent weights $n=100$ (the number of recurrent units). We updated the weights with the Adam optimization algorithm (learning rate was 0.001, weight decay was 0.01). The model was implemented in PyTorch.

In addition to Figure 7 of the main text, we ran simulations to show the automatic emergence of both the attention-modulated center/surround connections and the CD-gated directional connections under many other conditions, with some examples included in SI. In Fig. S10, we trained a neural network on both brief input stimuli and persistent input stimuli together. In Fig. S11, we repeated the simulation in Figure 7A of the main text but without the attentional modulation. In both cases, we found similar connectivity patterns to those in Figure 7. It is not surprising that attention is not important for learning the connectivity patterns. To perform the task of updating the stimulus retinal positions, a network had to develop the center/surround connectivity to maintain the attractor activity pattern and the CD-gated directional connectivity to move the attractor pattern appropriately.³¹ These requirements do not depend on attentional modulation. Once the connections are learned, attention can modulate the center/surround connectivity to enhance processing at the attended location and cause convergent RF shifts.

We also consider brief input stimuli with a smooth temporal response profile given by a gamma function with a scale parameter of 5 and a shape parameter of 10 ms, and a smooth version of the CD signal using the same CD-gating function as in the circuit model. The results, shown in Fig. S12, are again similar to those of Figure 7 of the main text.

KEY RESOURCES TABLE.

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Deposited data
Single unit recordings from LIP and FEF	This paper	Mendeley Data: https://doi.org/10.17632/w6y53574zp.1
Experimental models: Organisms/strains
Rhesus monkeys (Macaca mulatta)	One from Center for Laboratory Macaques, Huangshan, China; Two from Xishan Zhongke Laboratory Animals, Suzhou, China	N/A
Software and algorithms
Code for data analysis and simulations	This paper	https://github.com/XiaoW2633/CellReport, https://doi.org/10.5281/zenodo.10672764

Highlights.

• Remapping directions in LIP/FEF change from convergent to forward in a delayed saccade task

• Unlike convergent remapping, forward remapping can adequately guide perception and saccades

• Corollary-discharge-gated and attention-modulated connections explain the remapping

• These connections emerge in neural networks trained to perform transsaccadic updating