Visual Remapping

Abstract

Our visual system is fundamentally retinotopic. When viewing a stable scene, each eye movement shifts object features and locations on the retina. Thus, sensory representations must be updated, or remapped, across saccades to align pre- and post-saccadic inputs. The earliest remapping studies focused on anticipatory, pre-saccadic shifts of neuronal spatial receptive fields. Over time it has become clear that remapping can come in various flavors and may be mediated by multiple neural mechanisms. This review attempts to organize the various forms of remapping into a functional taxonomy based on experimental data and ongoing debates about forward vs. convergent remapping, pre- vs. post-saccadic remapping, and spatial vs. attentional remapping. We integrate findings from primate neurophysiological, human neuroimaging and behavioral, and computational modelling studies. We conclude by discussing persistent open questions related to remapping, with specific attention to binding of spatial and featural information during remapping and speculations about remapping’s functional significance.

1. INTRODUCTION

Humans rely heavily on vision to navigate and interpret our surroundings. The intrinsic organization of our visual system poses an odd paradox: visual information about scene features are initially encoded according to the position of features on the retina (i.e., in eye-centered, or retinotopic coordinates), but for most animals, humans included, the eyes are in almost constant motion. We make multiple large, saccadic eye movements each second (O’Regan & Lévy-Schoen 1983), and each saccade changes the retinotopic locations of screen features (see Figure 1). Despite these changes, we perceive the world as stable, and our conscious perception is of objects in world-centered, spatiotopic coordinates. Thus, our visual system must update objects’ retinotopic locations to align visual input from before and after the saccade, in a process commonly known as remapping.

Figure 1: Saccades can disrupt both visual and attentional representations in retinotopic brain areas.

Pre-saccadic (A) and post-saccadic (B) representations of a complex visual scene by a population of retinotopic visual neurons. Black plus indicates the current fixation position, arrow indicates the upcoming saccade, and the colored dashed circles indicate the neurons’ spatial receptive fields. Before the saccade, the purple neuron represents the small distant child, but after the saccade the same object is now represented by the red neuron, while the purple neuron represents a different object. In the context of attention, if spatial attention is directed to the small child before the saccade, facilitating activity in the purple neuron, then in the absence of some form of remapping, attention will be mislocalized to the wrong target (trees) after the saccade.

In visual neuroscience the term remapping refers to several distinct, but possibly related, visual and neurophysiological phenomena. Duhamel and colleagues (Duhamel et al. 1992) originally used the term to refer to pre-saccadic changes in the spatial position of neuronal receptive fields (RFs) in macaque area LIP. They reported that a subset of LIP neurons responded to stimuli in their future fields, instead of their current RFs, starting 100-200ms before saccade onset. Since then there have been numerous reports of remapping activity, in monkey visual areas V2, V3, V3A (Nakamura & Colby 2002), V4 (Hartmann et al. 2017; Neupane et al. 2016a), cortical association areas like the FEF (Sommer & Wurtz 2006; Umeno & Goldberg 1997, 2001), midbrain structures like the superior colliculus (Churan et al. 2012; Walker et al. 1995); in humans using fMRI (Fairhall et al. 2017; Lescroart et al. 2016; Medendorp et al. 2003; Merriam et al. 2003, 2007), EEG (Parks & Corballis 2008), and MEG (Fabius et al. 2020); as well as behavioral evidence of remapping (Golomb et al. 2008, 2014b; Hunt & Cavanagh 2011; Jonikaitis et al. 2013; Mathôt & Theeuwes 2010a; Melcher 2007; Rolfs et al. 2011; Szinte et al. 2018).

Over time it has become clear there are several types of remapping. Some of this diversity may reflect different neuronal substrates and functional roles. For example, there is controversy over whether remapping occurs in the classic, forward sense (as described by Duhamel et al. 1992), or occurs in a convergent form (Section 2.1), where receptive fields remap towards the saccade target (Neupane et al. 2016a; Tolias et al. 2001; Zirnsak et al. 2014). Other studies disagree on whether remapping is primarily a predictive, pre-saccadic process, or includes critical post-saccadic components as well (Section 2.2). And while early studies focused on spatial RF remapping, it has become clear that attentional facilitation also remaps (Cavanagh et al. 2010; Golomb et al. 2008, 2010a; Marino & Mazer 2018; Mirpour & Bisley 2012; Rolfs et al. 2011; Yao et al. 2016b), perhaps via a different mechanism (Section 2.3). One issue in the field is that small differences in experimental approach can alter and confound interpretation of results. A related issue is that early studies of remapping were generally designed to test forward, predictive, spatial RF remapping, and often didn’t incorporate elements necessary to discriminate between the other forms of remapping subsequently identified.

A major goal of this review is to place findings from neurophysiological studies in nonhuman primates, neuroimaging and behavioral studies in humans, and computational and conceptual models of visual remapping into a coherent framework that can facilitate future research. For each of the debates previewed above, we attempt to highlight and integrate findings from all of these methodologies. In Section 2, we start with identifying and describing the different types of experimentally observed remapping reported in the literature in order to organize them into a taxonomy that accurately reflects what is currently known about both physiological mechanisms and perceptual or behavioral function. We also review evidence that the type of remapping observed might be partially determined by task demands and/or context (Section 2.4). Section 3 reviews the state of computational models of remapping, with the goal of identifying the essential components and computations of the remapping circuit. Finally, we conclude by discussing some critical open questions and future directions for remapping studies, including whether object features and identity information are remapped across saccades (Section 4.1), and larger theoretical questions related to the functional implications of remapping for visual stability (Section 4.2).

2. TAXONOMY AND REVIEW OF EXPERIMENTALLY OBSERVED REMAPPING EFFECTS

2.1. Convergent vs Forward Remapping

Physiological remapping was first described in LIP of the awake, behaving monkey by Duhamel and colleagues (1992), who observed that in a subset of LIP neurons, RFs translated in the direction of an upcoming saccade just before saccade initiation. This translation “remapped” the pre-saccadic RF to its expected post-saccadic location (termed the “future field”), enabling signaling of visual stimuli at locations not normally visible to these retinotopic neurons (see Figure 2A). Because this type of remapping is usually the same direction and amplitude as the upcoming saccade vector it is commonly referred to as forward remapping (Marino & Mazer 2016).

Figure 2: Forward vs convergent remapping.

(A-B) Single neuron schematic for forward spatial remapping (A) and convergent remapping (B). Black dot is initial fixation target (current gaze position), and green dot is saccade target. Black dashed circle depicts the neuron’s current receptive field (RF), red dashed circle indicates the forward remapping field location (future field), and purple indicates the convergent remapping field location (toward saccade target). (C) Time course of forward spatial remapping from a retinal perspective, with schematic centered on current gaze position at each time point. (Note Panel A depicts remapping instead from a world-centered perspective.) Timeframes illustrate how the retinal RF position changes under the influence of the saccade plan, shifting to a new retinal location during pre-saccadic remapping, and returning to the classical RF position once fixation is acquired at the saccade target (green). Yellow shading indicates the fovea and parafoveal area of the retina. (D) Convergent spatial remapping can enhance processing at saccade endpoints. All RFs shift towards the saccade target during remapping, resulting in an increase in the density of the neural representation around the saccade target that can mimic attentional gain effects.

Later studies revealed a second form of remapping, known as convergent remapping (Tolias et al. 2001; Zirnsak et al. 2014; Neupane et al. 2016), where pre-saccadic shifts translate the RF towards the saccade endpoint (see Figure 2B). Convergent mapping was first reported in area V4 of the awake monkey by Tolias and colleagues (2001). Careful probing of pre-saccadic sensitivity to visual probes at multiple visual field locations allowed them to precisely measure the position of the remapped RF and demonstrated shifts towards the saccade endpoint, and not parallel to the saccade vector, as initially observed in LIP.

Like Tolias et al (2001), Zirnsak and colleagues (2014) used probe stimuli to estimate the pre-saccadic RF location in FEF and found systematic shifts towards the saccade target, indicating convergent, and not forward remapping. They went on to use a decoding approach to show that convergent remapping in FEF could lead to perceptual compression of space and behavioral mislocalization of stimuli in the vicinity of the target. Zirnsak and colleagues have also presented evidence of convergent remapping in humans (Zirnsak et al. 2011) and argued that presaccadic convergent remapping effects could account for the well-known attentional facilitation of visual processing at saccade endpoints (for review see Zirnsak & Moore 2014).

Not all experimental paradigms, particularly those used in studies prior to the realization remapping might not always be parallel to the saccade vector, are capable of distinguishing between forward and convergent remapping (Marino & Mazer 2016). Distinguishing between forward and convergent remapping requires multiple saccade targets and/or extensive presaccadic probing of visual field locations to precisely determine the position of the remapped RF.

Neupane et al. (2016a) recently re-examined remapping in V4 and reported both forward and convergent remapping in the same neurons. RFs were determined from responses to sparse noise stimuli, which were used to assess time-varying changes in RF position. This analysis revealed two temporally distinct remapping phases: forward remapping in the early phase, and convergent remapping in the late phase. Neupane and colleagues speculated that the two phases could reflect different functional roles of forward and convergent remapping – with forward remapping contributing to perceptual stability and convergent remapping involved in attention (although see Hartmann et al. 2017 for an alternative interpretations).

A recent human behavioral study also attempted to differentiate between forward and convergent remapping by probing a high spatial resolution grid of stimulus locations at different times prior to the saccade (Szinte et al. 2018). They reported both a shift of attention towards the saccade target and clear evidence of an attentional focus at the forward remapping location, consistent with forward remapping of attention, as originally reported by Rolfs et al (2011; but see Arkesteijn et al. 2019). It is worth noting that unlike the neurophysiological studies discussed earlier, the Szinte behavioral study was explicitly designed to measure remapping of attention. The relationship between spatial remapping and attention is addressed below (Section 2.3).

The different functional roles of convergent and forward remapping largely remain a matter of speculation (see Open Questions Section 4.2). Recent studies suggest that timing may be a critical factor distinguishing forward from convergent remapping (Hartmann et al. 2017; Neupane et al. 2016a; Szinte et al. 2018), and further experiments are needed along these lines. As noted above, the interpretation of some physiological and behavioral studies performed before the realization that some remapping might be convergent remains ambiguous since convergent remapping can be mistaken for forward remapping in simpler experimental designs. In the behavioral literature, this issue is further complicated by discrepancies over which stimulus locations reflect the correct behavioral analog of the neurophysiological forward remapping location (see Rolfs et al. 2011). Moving forward, it is important that studies routinely include sufficient stimulus locations and saccade vector variation – as well as timing and attentional manipulations – to reliably distinguish between forward and convergent forms of remapping.

2.2. Pre-saccadic vs. post-saccadic remapping

Thus far we have focused on predictive remapping, specifically how receptive fields, attentional states, neural activity, and behavior update in anticipation of saccades. While the functions and mechanisms of predictive remapping are certainly important, focusing solely on predictive remapping obscures other critical remapping processes. There are at least two post-saccadic remapping effects that have been reported in the literature. First is “memory trace” remapping. A number of early single-neuron studies demonstrated that the remapped visual signal does not necessarily occur before the saccade; in many cases, the neuronal response to a stimulus presented before the saccade – at the remapped location – doesn’t start until well after the saccade is over (Umeno & Goldberg 2001). This type of memory trace remapping has been commonly exploited in human fMRI experiments (Lescroart et al. 2016; Merriam et al. 2003, 2007). Importantly, although the memory trace response is post-saccadic, the remapping mechanism is predictive, since it reflects a response to stimuli presented before saccade initiation, at the remapped RF location.

There is, however, a second post-saccadic aspect of remapping. In addition to understanding how activity is remapped to locations that will become behaviorally relevant after the saccade, it is also important to consider how remapping is turned off (Figure 3). Remapped spatial RFs must be restored to normal retinotopic locations, and – especially in the case of attention – facilitation of ongoing activity in neurons reflecting the previously attended retinotopic location must cease. In the absence of saccades, it’s been shown that attention “turns on” in V1 neurons representing the new location of the attentional focus before it “turns off” in those representing the old location (Khayat et al. 2006). Thus, the onset and offset of remapping can be asynchronous, which could carry important consequences for perceptual stability.

Figure 3. Predictive Remapping vs Retinotopic Attentional Trace.

(A) Hypothetical responses of two visual neurons with different spatial receptive fields. The beige interval indicates the period prior to saccade initiation, the blue interval after. Yellow circle represents to-be-attended spatiotopic location. Before the saccade, the attended location falls within Neuron A’s receptive field; after the saccade, it falls in Neuron B’s. ‘Predictive remapping’ is when Neuron B begins to respond in anticipation of the saccade. ‘Retinotopic attentional trace’ is when Neuron A continues to respond for a period of time after the eye movement. Thus, there is a period of time where both spatiotopic and retinotopic locations are facilitated. (B) Corresponding locations for a behavioral study. Figure adapted from Golomb 2019. (Note to Annual Reviews: I am an author of this article, and the publisher has confirmed that I retain the rights to re-publish it.)

Golomb et al. (2008) had human subjects plan and execute a saccade to one location, while sustaining covert attention at a (different) cued spatiotopic location, and then presented visual probes at different delays after the saccade. For probes flashed immediately after the saccade (within 150ms), attentional facilitation was strongest when probes appeared at the previously attended retinotopic location; it was only after longer delays that attention was fully disengaged from the previously attended location. This lingering attention at the (wrong) retinotopic location has been termed the “retinotopic attentional trace”, and has since been demonstrated across a variety of behavioral tasks (Golomb et al. 2010b, 2011; Jonikaitis et al. 2013; Mathôt & Theeuwes 2010b), observed in both fMRI and ERP experiments (Golomb et al. 2010a; Talsma et al. 2013) and replicated using computational modeling (Bergelt & Hamker 2019; Casarotti et al. 2012).

Importantly, the existence of the retinotopic attentional trace does not in any way preclude a predictive remapping component, particularly if remapping onset/offset timing is asynchronous. In other words, remapping of attention becomes apparent when the temporal dynamics of attentional modulation are not perfectly matched and synchronized with saccade dynamics, and this asynchrony can occur at two separate points: in predictive remapping, the attentional focus begins shifting (remapping onset) before saccade onset, while in the case of the retinotopic attentional trace, the dynamics of attention disengagement (remapping offset) outlast the saccade offset. This idea is formalized in the “dual-spotlight” theory (Golomb 2019). The implications of the dual-spotlight theory for both neural activation and behavior are illustrated in Figure 3. When attention is directed to a spatiotopic location, neurons with RFs covering the post-saccadic attended location begin to fire in anticipation of the saccade (predictive remapping), while neurons with RFs covering the pre-saccadic attended location may persist in firing for a brief period of time even after the saccade has been executed (retinotopic attentional trace). Behaviorally, this would result in simultaneous attentional facilitation at two different visual field locations during the perisaccadic period, as observed by Golomb et al (2008, 2011, 2014b).

Mechanistically, visual and attentional stability across saccades can be supported by multiple mechanisms, as reviewed in Section 3. Oculomotor feedback, which is critical for remapping, operates at multiple time scales: corollary discharge signals from the superior colliculus (SC) are rapid (Sommer & Wurtz 2006, reviewed in 2008), while proprioceptive oculomotor signals are relatively slow (Sun & Goldberg 2016). Bergelt & Hamker’s (2019) neuro-computational model of remapping (Section 3) formally accounts for this dual-spotlight pattern based on convergence of the fast corollary discharge and slow proprioceptive signals, simulating both predictive remapping and retinotopic attentional trace effects.

Marino & Mazer (2018) found an analogous pattern of attentional remapping effects in area V4 neurons, where attention is predictively engaged in neurons with RFs that will occupy the attentional focus as a result of the saccade, before attention is disengaged in neurons that previously occupied the focus. Yet, intriguingly, this entire “attentional handoff” was complete before the saccade. The timing of the handoff in monkey V4 stands in contrast to human behavioral studies that indicate the behavioral handoff is more diffuse in time and impacts sensitivity both before and after the saccade (e.g., Jonikaitis et al. 2013), or, in some cases, can even occur entirely after the saccade (e.g., Golomb et al. 2011; but see Yao et al. 2016a). In a recent MEG study, Fabius et al. reported a similar “soft handoff” of information, with a brief period of post-saccadic overlap where information about low-level spatial frequency could be decoded from both pre- and post-saccadic processing areas (Fabius et al. 2020). While it is not yet clear what factors might cause the timing of the attentional handoff to vary, these studies indicate that attention is likely briefly split during the perisaccadic period, an idea supported by a recent set of studies comparing feature perception across tasks requiring saccadic remapping, covert shifts of attention while fixating, and covert splitting of attention while fixating, which found that feature-binding errors immediately after saccades were most consistent with attention being simultaneously split between two locations (Dowd & Golomb 2019, 2020; Golomb et al. 2014b).

2.3. Spatial remapping vs attentional remapping

Another level of the taxonomy important to clarify is the distinction between spatial remapping, involving neuronal receptive field shifts across the retina, and attentional remapping, which corresponds to a redistribution of attentional resources across the visual field representation to maintain a stable attentional topography or sustain a locus of spatiotopic attention, possibly without receptive field shifts. The early studies of neurophysiological remapping (LIP: Duhamel et al. 1992) (V4: Fischer & Boch 1981a,b; Tolias et al. 2001) focused exclusively on spatial selectivity. The generally-accepted interpretation was that if a neuron responded when a stimulus was presented pre-saccadically in the future RF, that was taken as evidence that the RF had transiently shifted, or remapped, in space, presumably due to corollary discharge from the eye movement (Mayo & Sommer 2010). A competing theory has proposed that attentional remapping is based on changing the target of top-down modulatory signals instead of remapping spatial RFs, that is, shifting attentional pointers in the brain between different locations in retinotopic visual or priority maps (Cavanagh et al. 2010). In the attentional pointer model, remapping occurs when the specific subpopulation of attentionally facilitated visual neurons in extrastriate cortex changes. This leads to a change in which neurons are attentionally modulated, without changing neuronal RF positions (Figure 4). In this model, the retinotopic trace reflects a delayed offset of attentional facilitation when the pointer shifts from one set of neurons to another, briefly leaving two distinct neuronal populations with spatially separated RFs in a facilitated state.

Figure 4. Spatial receptive field remapping vs attentional pointer remapping.

Left side: Schematic illustrating remapping mechanism where spatial receptive fields (RFs) transiently shift in anticipation of the saccade. Right side: Schematic illustrating alternative attentional pointer remapping mechanism. Gray boxes illustrate the locations of 3 neurons’ RFs (dashed circles), with current gaze position at the black dot, and the green dot representing the saccade target. The orange star indicates the stimulus probe, and the yellow shading indicates the attentional focus. Next to each box is a simplified diagram of the same 3 neurons (conceptualized here as LIP neurons), and the corresponding V1 neurons feeding feed-forward input (blue arrows indicate these connections). During the initial fixation period (top row), the stimulus falls in the RF of neuron 2. After the saccade (bottom row), the stimulus falls in the RF of neuron 1. During remapping (middle row), neuron 1 becomes active in anticipation of the saccade. The spatial RF remapping mechanism says this is because the RFs shift spatially to their future fields, which could be conceptualized as a remapping of which retinotopic V1 neurons feed into the LIP neurons, such that the neurons become transiently sensitive to a different portion of the visual field. The attentional pointer mechanism instead says that the RFs remain veridical, but the new set of neurons becomes facilitated in anticipation (i.e., the attention pointer remaps from neuron 2 to neuron 1). In both cases, the remapping signal could come from corollary discharge signals from an area such as thalamic MD.

In many neurophysiological studies of spatial remapping, attention is neither explicitly cued nor measured, making it difficult to tease apart these mechanisms. Attention is assumed to be deployed to the saccade target during saccade planning (Kowler et al. 1995; Rizzolatti et al. 1987), and it is also often assumed that the occurrence of a single salient visual stimulus will automatically capture attention as well. (Indeed, there is evidence that only attended stimuli are remapped; see Section 2.4.)

While the number of single-neuron studies explicitly examining attentional remapping is severely limited at this point, the results so far suggest that, at least under some experimental conditions, attentional remapping can occur in the absence of spatial remapping. Marino and Mazer (2018) recorded from neurons in macaque area V4 while animals performed a sustained spatiotopic attention task and found neurophysiological evidence of predictive attentional remapping. They observed pre-saccadic increases in attentional gain just before saccades that brought the RF into the attentional focus, and a decrease in gain when the saccade displaced the RF from inside the attentional focus to outside. Importantly, this study found no evidence of spatial remapping - no changes were observed in neurons’ spatial tuning properties, that is, the neuron’s RF was unchanged before, during and after the saccade. Yao and colleagues (Yao et al. 2016b, 2018) observed a similar peri-saccadic transfer of attentional gain without spatial remapping in MT neurons in monkeys performing a spatiotopic motion detection task and concurrently executing guided saccades.

Attentional remapping has been more commonly studied in human paradigms, due to intrinsic difficulties in accessing information at the spatial RF level. For example, Golomb and colleagues employed an attentional remapping paradigm where participants were instructed to maintain top-down spatial attention at a cued spatiotopic location, demonstrating the retinotopic attentional trace (Golomb et al. 2008; see also 2010a,b, 2011; Talsma et al. 2013; Yao et al. 2016a). Subsequently, Rolfs and Cavanagh (2011) reported evidence for anticipatory remapping of spatial attention, demonstrating pre-saccadic shifts in behavioral facilitation using a doublestep saccade task. Other studies have used exogenous cues to manipulate attention and probe facilitation before and after saccades (Jonikaitis et al. 2013; Mathôt & Theeuwes 2010a; Szinte et al. 2018). While these studies can naturally be interpreted in the attentional pointer framework, they don’t rule out shifting RF remapping, which is often posed as an explanation for other types of behavioral effects (e.g. Melcher 2007, discussed more in Section 4.1). Indeed, Cha and Chong (2014) manipulated top-down attention in a figure/ground perceptual aftereffects paradigm and concluded that complementary mechanisms of shifting RF and attentional remapping can coexist.

As noted above, while both spatial and attentional remapping can lead to pre-saccadic changes in attentional topography, the neural bases of these changes are distinctly different. In many cases, distinguishing between these different types of remapping has been difficult, either due to limited experimental conditions or because the behavioral tasks have insufficient control or measurement of attention. We suggest that this is an important direction for future research across neurophysiology, behavioral, and modelling-based investigations of remapping.

2.4. Is the type of remapping determined by task demands and/or context?

A common theme across the debates reviewed above is evidence for multiple types of remapping. So, what determines whether remapping is forward or convergent, predictive or lingering, spatial RF or attentional? One difficulty is that the experimenter’s choice of locations and timepoints to probe – and how the effects are measured – will inherently limit the types of remapping one could detect, and these choices vary widely across experiments. Below we consider a few factors which might influence the type of remapping observed.

(1). Is the type of remapping related to brain region, species, or experimental methodology?

So far, the evidence seems to suggest no. For the most part, the multiple types of remapping don’t cleanly map on to differences along any of these dimensions; there are examples of single brain regions that exhibit multiple types of remapping in both human fMRI and monkey neurophysiological experiments. However, it is rare for different forms of remapping to be systematically explored across brain regions within individual studies, and studies of different brain regions have often used significantly different experimental designs or parameters, leaving substantial room for investigation along these lines.

(2). Is the type of remapping influenced by stimulus-driven factors such as visual features and bottom-up attentional salience?

There is considerable evidence that bottom-up factors can influence remapping, but there is no consensus on whether/how these factors might map onto the taxonomies described above. Several studies have shown that only attended items – including salient stimuli that capture bottom-up attention – are remapped (Gottlieb et al. 1998; Joiner et al. 2011; but see Cha & Chong 2014). This finding has been mostly applied to studies investigating forward remapping, but a recent set of studies showed that attentional anticipation can produce convergent remapping effects (Neupane et al. 2016b, 2020). Other studies have also shown a compelling influence of scene context and low-level features (e.g., illumination, contrast or sparsity), at least for forward, predictive remapping. Churan et al (2011) showed that remapping in the SC was stronger when stimuli were presented on a dark background than a well-illuminated background, hypothesizing that in well-lit real world environments perceptual stability might depend on visual landmarks, but in darkness the visual system is forced to rely exclusively on remapping and corollary discharge mechanisms. The retinotopic attentional trace, on the other hand, has been shown to be insensitive to illumination and landmarks, suggesting that predictive remapping may be more sensitive to context and task-demands than the retinotopic trace is (Golomb et al. 2010b). Moreover, Marino and Mazer (2018) reported attentional remapping without corresponding spatial remapping in V4 neurons when using a dense mapping stimulus designed to provide constant visual stimulation and mimic cluttered natural vision conditions, while studies using sparser mapping stimuli consistently find both forward and convergent spatial RF remapping in V4 (Hartmann et al. 2017; Neupane et al. 2016a; Tolias et al. 2001), hinting at the possibility that the details of the visual stimulation conditions may be significant here as well. The use of a dense noise stimulus could potentially underlie another interesting conundrum, namely Marino & Mazer’s finding of attentional remapping without evidence of the retinotopic attentional trace, which is robustly observed in humans (e.g., Golomb et al. 2008), and also in macaque area MT (Yao et al. 2018) using single or sparse transient probe stimuli. Dense stimuli can recruit inhibitory or suppressive neural mechanisms not engaged by isolated single stimuli (e.g., Churan et al. 2011; Haider et al. 2010; Vinje & Gallant 2000), which could potentially impact the timing and extent of remapping processes. Remapping may also depend on the visibility of the saccade target (Marino & Mazer 2018), and the choice of saccade vector (Arkesteijn et al. 2019; Neupane et al. 2020), which could have particular implications for forward vs convergent remapping.

(3). Do goal-related factors such as task demands and top-down attention determine the type of remapping?

Remapping tasks also differ in terms of behavioral task design and top-down attentional demands. One challenge is that many physiological studies of remapping in nonhuman primates have used passive viewing tasks, measured visual responses of neurons with different receptive field placements relative to the saccade target or attentional focus, or relied on responses to task-irrelevant or even actively ignored probe stimuli, while behavioral studies in humans have tended to rely on explicit attentional manipulations or goal-directed tasks to measure differences in attentional facilitation (speeded RT or enhanced sensitivity) to stimuli presented at different visual field locations. In human behavioral studies there is some evidence that attentional manipulations can influence the relative strength and potentially the timing of post-saccadic perceptual stability. For example, Golomb et al. (2008) demonstrated that predictive attentional remapping isn’t automatic, but rather requires that observers sustain spatial attention at the spatiotopic location; when the task required attending only to the retinotopic location of a cue, only the retinotopic trace was found (i.e., there was no indication of attentional remapping). In a different task, Yao and colleagues failed to find evidence for either predictive remapping or the retinotopic trace interfering with spatiotopic performance, and suggested that differences in the nature of the task might have resulted in subjects adopting different attentional sets compared to the prior studies (Yao et al. 2016a), though this speculation has yet to be tested directly.

(4). Does expectation and predictability influence the type of remapping?

Another potential source of variance that could influence the type of remapping, or if remapping is even observed at all, in any given experiment, is the degree to which the stimulated locations – and saccade trajectories – are predictable, and whether implicit or explicit (i.e., cue-driven) expectations could play a role in the strength, timing, and/or type of remapping. If an experiment probes the same saccade trajectory repeatedly across hundreds of trials with similar timing, which is common in non-human primate (and some human) experiments, it is conceivable that expectation could alter the latency or efficiency of remapping. Similarly, expectations about an upcoming saccade target could enhance attentional effects at the saccade target location, which could in turn strengthen convergent remapping effects (Neupane et al. 2020), or vice versa -- stronger convergent remapping effects could enhance attentional effects at the target. While studies have varied in how predictable experimental designs are – with some studies being careful to minimize the predictability of saccades and stimuli and others going to great length to make experimental conditions completely predictable – it remains largely untested how these factors actually influence remapping, or how they interact with other more general cognitive factors, such as motivation and fatigue.

3. COMPUTATIONAL MODELS OF REMAPPING

As discussed above, experimental studies have demonstrated remapping effects in a number of brain regions, including both cortical and subcortical structures. Several other areas, including the pulvinar (Hall & Colby 2011; Rao et al. 2016a), the mediodorsal nucleus of the thalamus (MD; Sommer & Wurtz 2002) and the brainstem oculomotor nuclei (Sun & Goldberg 2016), while not exhibiting remapping effects, have been identified as possible components of a large remapping circuit. Given the inherent difficulties in simultaneously recording circuit-level activity from the many putative brain regions, theoretical and computational modeling is an essential part of understanding remapping. Two classes of model have been instrumental in the literature: conceptual or schematic models that provide an abstract framework for remapping, and biologically plausible models that attempt to accurately model remapping using information about the anatomical and functional properties of neurons in the putative remapping circuit. While we will briefly discuss the conceptual models, the majority of this section will focus on reviewing the current state of biologically-inspired computational models of visual remapping.

The earliest efforts to model remapping focused on forward spatial remapping, consistent with the initial experimental reports. Quaia and colleagues (1998) modeled peri-saccadic updating of the representations of saccade targets in the context of sequential saccade tasks. The model included several features that turned out to be essential components of more recent models. This includes circuit-level access to oculomotor plans, which in their model came from movement cells in the FEF, although in more recent studies this is usually modeled as a corollary discharge signal originating from the SC (reviewed in Rao et al. 2016a), and pre-saccadic forward (spatial) remapping to bridge between the pre- and post-saccadic visual representations, intended to account for differences in timing of motor command signals, oculomotor response times and visual response latencies. The model recapitulated Duhamel et al.’s (1992) forward remapping in LIP (n.b., it took another nine years before convergent remapping in V4 was reported by Tolias and colleagues (2001); this delay was likely due to the fact that convergent remapping can only be robustly detected by probing multiple visual field locations, which was not common practice in remapping studies prior to their 2001 report). Since the model focused exclusively on spatial properties and updating of saccade target locations, no effort was made to account for feature remapping (see Section 4.1). However, this early model correctly identified some important properties of the remapping circuit, namely modulation of a retinotopic visual representation by oculomotor signals coding the upcoming saccade vector. Importantly, Quaia and colleagues correctly noted that spatiotopic actions can be targeted without an explicit spatiotopic brain map and that remapping could reflect an implicit spatiotopic representation.

Sommer and colleagues were among the first to identify what is now generally agreed to be the essential components of the remapping circuit. Sommer and Wurtz (2002) showed that a corollary discharge signal encoding the direction of upcoming saccades was produced by SC neurons and transmitted to the FEF by way of the thalamus (MD), with inactivation of MD eliminating remapping in FEF neurons. Corollary discharge is thought to be a main signal contributing to visual stability through the remapping mechanism; visual reafference and proprioceptive information about current eye position are also believed to play a role (Wurtz 2008). A recent proposal by Sun and Goldberg (2016) suggests that the rapid predictive remapping signal driven by corollary discharge is supplemented by a slower mechanism that uses oculomotor proprioceptive representations from somatosensory cortex to construct a more accurate spatiotopic representation after the saccade.

Rao and colleagues (2016a,b) have recently developed a circuit-level neural network model of remapping. Their model posits that FEF is uniquely positioned to instantiate remapping, based on the following: (1) FEF visual and visuomotor neurons have the necessary retinotopic spatial selectivity, (2) FEF is the target of prerequisite SC-MD corollary discharge signals and (3) FEF projects directly or indirectly to the rest of the remapping network, including LIP and V4. Their model replicates prior experimental work and supports a functional role of remapping for perceptual stabilization. The definitive test of this model, as noted by the authors, would be to show that FEF inactivation eliminates both behavioral and physiological remapping. This prediction has yet to be tested experimentally.

The models discussed above generate dynamic changes in the spatial selectivity of neurons, i.e., pre-saccadic RF shifts. However, as discussed in Section 2.3, a competing conceptual model of remapping – Cavanagh and colleagues’ (2010) attentional pointer model – poses that the spatial properties of neurons can remain fixed, as long as the top-down attentional signal can shift or transfer between retinotopic neurons. Cavanagh’s model represents a more conceptual framework only loosely tied to the neurophysiological findings, but it has spurred important debates in the field. It’s worth noting that these models are not necessarily mutually exclusive – it is even possible that spatial remapping and attentional remapping, mediated by attentional pointers, could arise in the same brain areas and even the same neurons, depending behavioral context or timing. In fact, both models depend critically on access to oculomotor plans to redistribute either spatial selectivity or gain modulation and both model the source of these signals as corollary discharge.

Hamker and colleagues developed several detailed computational models based on the physiological and anatomical connectivity data (Ziesche et al. 2017; Ziesche & Hamker 2011); originally intended to provide a biologically realistic account of a number of well-known perisaccadic perceptual effects, like saccadic suppression of displacement, the model was recently extended to include top-down attentional signaling and used effectively to model recent behavioral and physiological studies related to attentional remapping (Bergelt & Hamker 2019). These models depend on multiplicative “planar gain fields” to generate pre-saccadic spatial RF shifts resembling the forward remapping in LIP first reported by Duhamel and colleagues (Duhamel et al. 1992). Andersen and Mountcastle (1983) coined the term planar gain field to describe observed interactions between spatial selectivity and gaze angle in area 7a, where Gaussian RF profiles, multiplicatively modulated by linear (1D) or planar (2D) gaze-angle dependent functions, generate a joint representation of stimulus and eye position. Gain field neurons do not necessarily exhibit remapping, nor are they really spatiotopic. Rather, they constitute an intermediate representation between retinotopic and spatiotopic and can be used to infer spatiotopic position. The original Hamker models (Ziesche et al. 2017; Ziesche & Hamker 2011) showed that gain field-like modulation by corollary discharge and proprioceptive signals (from MD and S1, respectively) were sufficient to generate the RF shifts observed in area LIP. Bergelt & Hamker (2019) extended the original models by modeling top-down attention as a spatiotopic or head-centered input signal directly modulating a subset of LIP neurons and interacting with the corollary discharge and proprioceptive signals. With this addition, the new model could generate both forward spatial remapping and attentional remapping effects, including both the predictive attentional shifts reported by Rolfs et al. (2011) and the post-saccadic retinotopic attention trace (Golomb et al. 2008), consistent with the dual-spotlight theory (Golomb 2019).

A recent report from Zhu et al. (2020) proposed that object pointers, closely related to attentional pointers, can be implemented with shifter circuits, a dynamic feed-forward routing circuit that can shift retinotopic labeled lines based on the state of a control signal (Anderson & Van Essen 1987; Olshausen et al. 1993). On face, this approach looks very different from the Bergelt & Hamker model, but both are premised on the idea that perceptual and attentional stability arises from modulation of retinotopically organized visual and/or priority maps by command signals from the oculomotor system and/or the attentional control system. These two models, i.e., planar gain fields and shifter circuits, are also not necessarily mutually exclusive. And, while experimental evidence of shifter circuits have been elusive, there is some theoretical support for the idea that the pulvinar could provide the necessary signals for a shifter circuit that could stabilize image representations in early visual cortex (Olshausen et al. 1993).

4. OPEN QUESTIONS AND FUTURE DIRECTIONS

4.1. Is feature/content information remapped?

One persistent open question about visual remapping is whether – and how – non-spatial information gets remapped. The answer to this question has both perceptual and physiological significance. Successful execution of visually-guided behaviors relies on our ability to simultaneously locate and recognize objects. Spatial remapping alone is insufficient to maintain perceptual stability, which requires stabilization of both object locations and their associated feature representations. Traditional thinking is that location and feature representations are functionally and even anatomically distinct aspects of visual processing; e.g., “what” and “where” pathways (Ungerleider & Mishkin 1982). But effective use of visual information during natural behaviors depends on combining “what” and “where” information. This challenge is commonly known as the “binding problem” (Holcombe 2009; Reynolds & Desimone 1999; Treisman 1996; von der Malsburg 1999; Wolfe & Cave 1999). The already difficult binding problem is even more complicated when considering the impact of eye movements (indeed, Cavanagh and colleagues (2010) referred to this as the “hard binding problem”). For real-world behavior, feature information must be bound to stable locations across eye movements.

Two basic solutions to the hard binding problem have been debated in the literature: one based on the premise that remapping simultaneously remaps both the features and location of objects, the other suggests that remapping, as we know it, is fundamentally spatial and that information about the appearance or identity of remapped objects must be refreshed or rebound to remapped spatial locations by an additional mechanism (e.g., Cavanagh et al. 2010).

In most of the brain regions where spatial remapping is found, neurons exhibit mixed spatial and feature selectivity, although dorsal stream areas, like LIP, tend to better represent spatial properties, and ventral stream areas, like V4, better represent features (Ungerleider & Mishkin 1982). When feature-selective neurons in these areas remap, feature selectivity has generally been presumed to remain constant, to the extent that only a handful of studies have even attempted to measure feature selectivity in remapped RFs (Subramanian & Colby 2013; Yao et al. 2016b; see discussion below). If this is the case, then spatial and feature tuning would automatically remap together. On the other hand, pointer-based remapping presumes that a spatial pointer shifts from one location to another, without regard for either the specific visual features present at the original or remapped location or the feature selectivity of the neurons being targeted by the pointer; such a mechanism could result in non-specific attentional facilitation of all neurons at the remapped location, regardless of their feature selectivity, or a delayed emergence of featural information as it gets updated or rebound to appropriate spatial locations. These two theories – automatic spatial-plus-feature remapping and spatial-pointer only remapping – represent two endpoints on a continuum, and while it is useful to consider the extremes, it is also important to recognize that remapping may be instantiated by a hybrid mechanism with aspects of each model. For example, it is possible that there is partial feature remapping, where limited low-level features are automatically remapped, enough to preserve a gist or coarse featural representation across the saccade (e.g., Fabius et al. 2020). And finally, the discussion from Section 2.4 applies here as well: the form or extent of feature remapping may vary depending on behavioral context, brain region, attentional state, or even species.

These two theories remain largely untested at this point, but they make specific predictions about the time course of remapping spatial and feature information. Specifically, the first model, where spatial and feature information are intrinsically bound at the neuronal level, predicts synchronous spatial and feature remapping, while the second model, where pointers remap, predicts spatial remapping is followed by a wave of “rebinding” activity, so feature remapping lags spatial remapping. (Though it’s technically possible that some neurons could remap spatial information only, while others remap feature information in parallel, which could lead to synchronous remapping through independent circuits.)

While neurophysiology would potentially provide the most direct evidence differentiating these options, as noted above, single-neuron remapping studies have almost exclusively focused on spatial remapping and largely ignored feature remapping, with two recent exceptions. Subramanian and Colby (2013) examined shape selectivity at the forward remapping location in LIP neurons using a guided saccade task similar to the one used in the original Duhamel et al. (1992) study. In a subset of LIP neurons studied (37%), they found a weak, but significant, correlation between stimulus selectivity in the current RF at fixation and in the future field at the forward remapping location, concluding that LIP remaps both spatial and shape information. However, this result is complicated by the fact that only about half of all LIP neurons exhibit robust shape selectivity in the absence of saccades (Sereno & Maunsell 1998). And interestingly, while the “best” stimulus was generally conserved between the RF and the future field, the overall pattern of selectivity (i.e., the overall shape-tuning curve) was generally not, suggesting that feature selectively may only partially remap and/or the feature remapping mechanism is not robust under these experimental conditions. Yao and colleagues (2016b), on the other hand, recorded from MT neurons and found negligible evidence for direction-related information in the remapped responses, interpreting their result as evidence in favor of the spatial-only attentional pointer hypothesis. However, MT is a somewhat controversial area for remapping – it exhibits memory trace remapping but not reliable predictive spatial remapping (Ong & Bisley 2011; Yao et al. 2016b) – so a lack of feature remapping in MT wouldn’t preclude the existence of feature remapping in other areas.

Using human fMRI and multivoxel pattern analysis, Lescroart, Kanwisher, and Golomb (2016) asked whether stimulus category information could be decoded from voxels at the remapping location. This study found no evidence of automatic remapping of category information, though it also failed to generalize previous spatial remapping effects detectable with fMRI (Merriam et al. 2007), raising doubts about the nature of the correspondence between predictive remapping signals observed with fMRI and those observed in single neurons (or behavior, for that matter). Intriguingly, an EEG study using a similar decoding approach found evidence of stimulus content remapping (Edwards et al. 2018), although they examined peripheral-to-fovea remapping, which, as discussed below, might be different than peripheral-to-peripheral remapping (Knapen et al. 2016; Williams et al. 2008). Other fMRI studies have found evidence for transsaccadic updating or spatiotopic adaptation of feature information (Baltaretu et al. 2020; Dunkley et al. 2016; Fairhall et al. 2017; Zimmermann et al. 2016), but the temporal resolution of fMRI precludes distinguishing automatic feature remapping from attentional pointer remapping in these paradigms. On the other hand, a recent study using MEG (which allows for higher temporal resolution) found that low-level feature information is not predictively remapped, but instead remains available from the pre-saccadic location for a period of time after the saccade, suggesting an alternative mechanism by which feature information could remain continuously available across the saccade (Fabius et al. 2020).

Finally, numerous human behavioral studies have probed feature remapping through indirect measures, again with variable results. Shafer-Skelton et al. (2017) examined feature-location binding (Golomb et al. 2014a; Treisman 1996) by asking whether an object presented prior to an eye movement preserves its binding across the saccade. They reported that object-location binding is preserved across a saccade, but only in retinotopic coordinates; they found no evidence of spatiotopic location-identity binding (Shafer-Skelton et al. 2017). This led the authors to conclude that feature-location binding may be performed in retinotopic coordinates, and then refreshed after each saccade based on the feedforward visual input, reminiscent of the attentional pointer remapping model discussed above. On the other hand, other human perceptual studies have been touted as strong evidence in favor of predictive remapping of features, specifically spatiotopic transfer of visual aftereffects and feature integration. Melcher (2007) was the first to report predictive feature remapping via spatiotopic aftereffects, but subsequent studies failed to replicate the original findings (Knapen et al. 2010; Mathôt & Theeuwes 2013; Wenderoth & Wiese 2008). Even more recent studies have again found support for spatiotopic aftereffects (He et al. 2017; Wolfe & Whitney 2015), and there is the suggestion that these effects may build up over time (Zimmermann et al. 2013). However, substantial variation in experimental methods makes it challenging to draw definitive conclusions (see Marino & Mazer 2016 Similarly, evidence of spatiotopic transsaccadic feature integration has been mixed and subject to controversy (Fabius et al. 2016; Harrison & Bex 2014; Hayhoe et al. 1991; Irwin et al. 1983; Melcher & Morrone 2003; Morris et al. 2010; Oostwoud Wijdenes et al. 2015; Paeye et al. 2017), though recent studies seem to support some degree of transsaccadic feature integration (Fabius et al. 2016; Ganmor et al. 2015; Oostwoud Wijdenes et al. 2015; Paeye et al. 2017; Wolfe & Whitney 2015). Further muddying the debate are questions about whether transsaccadic integration and aftereffects observed in these studies could be fully or partially driven by factors like attentional spread or remapping of attentional pointers, and therefore may not definitively indicate true feature remapping (e.g., Cavanagh et al. 2010).

Another confounding factor is the choice of location for spatiotopic stimuli and their controls: many perceptual studies that have found support for spatiotopic feature remapping have examined periphery-to-fovea remapping, which may be fundamentally different from remapping of features between peripheral locations – in periphery-to-fovea remapping the two locations differ in retinal eccentricity and the remapped location is the saccade target. This distinction may be relevant to the convergent versus forward remapping debate discussed in Section 2a; indeed, Zirnsak et al. (2011) compared the magnitude of the tilt aftereffect at the forward and convergent remapping locations in human observers and found stronger effects at the convergent location; in fact, they found almost no evidence at the forward remapping location. An intriguing possibility is that the degree to which feature information is remapped may depend on the specific type of remapping.

4.2. Remapping and visual stability

The functional significance of remapping – both in general and for the different subtypes of remapping – currently remains a matter of some debate. One suggestion is that forward remapping provides visual continuity across eye movements. A comparison of pre- and post-saccadic activity in forward remapping neurons can support detecting changes in the visual field during eye movements. This can also be achieved by comparing activity between remapping and non-remapping neurons after saccades. Either way, forward remapping could enhance transaccadic change detection by increasing sensitivity or speeding reaction times.

In the case of convergent remapping, the functional role seems less likely to be related to perceptual stability – the fact that each neuron remaps in a different direction, depending on the relationship between its RF and the saccade target, complicates making inferences about the causality of apparent visual field changes. At the population level, convergent remapping systematically shifts receptive fields towards the saccade endpoint (see Figure 2D). As a result, the number of visual neurons representing the region around the saccade endpoint increases right before the saccade. The increased density of the neural representation around the saccade target could increase both acuity and sensitivity around the endpoint. This would mean that the neural substrate of “attentional” facilitation at saccade endpoints differs from other types of attentional facilitation in retinotopic cortex, which are generally believed to be mediated by localized neuronal gain changes without changes in spatial selectivity (e.g., Reynolds & Heeger 2009; Lee & Maunsell 2009; although see Connor et al. 1996; Womelsdorf et al. 2006). In this context, convergent remapping would function to facilitate accurate saccade targeting. That said, given that some theories posit that transsaccadic change detection is most heavily influenced by processing of local information near the saccade target (saccade target theory of visual stability: McConkie & Currie 1996), it is possible that convergent remapping could indirectly aid perceptual stability in this sense.

The timing of remapping also carries functional implications with respect to visual stability. Predictive remapping is generally viewed as a positive source of visual stability, allowing perception to be stabilized by the time the saccade is completed. On the other hand, lingering retinotopic effects generally carry costs for visual stability, in the form of incorrect attentional foci (Golomb et al. 2008), feature-binding errors (Dowd & Golomb 2020; Golomb et al. 2014b), and poorer spatial memory and reaching precision (Golomb & Kanwisher 2012; Shafer-Skelton & Golomb 2017). However, one possible functional benefit of a dual-spotlight of attention could be allowing for a soft hand-off of feature information across saccades (Fabius et al. 2020).

Finally, the existence of both spatial and attentional remapping and the ongoing debate over their functions raises questions about the true significance of remapping: Is the function of remapping to provide perceptual stability? Or is its primary function to facilitate object-based or spatiotopic attentional targeting as the individual moves through the environment? Of course, this is not necessarily an either-or situation. However, it is surprising that after almost three decades of research on remapping, its functional role remains a matter of debate. The stability of visual and attentional representations in the brain is essential for natural visually guided behavior, so it seems clear that remapping is likely to be an important part of natural vision. Of the many open questions raised here, perhaps none are more important than determining, once and for all, the true functional role (or roles) of remapping.

Terms and Definitions List

Attentional handoff

Transfer of attentional state between neurons during attentional remapping; can be “soft”, resulting in a perisaccadic “dual spotlight” of attention

Attentional pointer

Conceptual idea of a location pointer directing spatial attention, to which object features/identity can be linked

Attentional remapping

Presaccadic shifts in attentional state that shift facilitation to neurons or locations that will be in the focus post-saccade

Binding problem

Linking spatial and featural information about single objects when encoded by separate populations of space- and feature-selective neurons

Convergent remapping

Spatial remapping towards the saccade endpoint; RFs shift to a location between the current RF and saccade target

Corollary discharge

A copy of the motor command directing the eye movement, a major signal triggering remapping; also known as efference copy

Dual Spotlight

When attentional handoff is asynchronous, two locations can be simultaneously highlighted during remapping; e.g. both remapped spatiotopic and RAT

Feature remapping

Idea that object features/identity (and neuronal feature selectivity) are automatically remapped when spatial RFs or attention remaps

Forward remapping

Spatial remapping parallel to the saccade vector, such that the RF shifts to the “future field”, anticipating the post-saccadic RF

Future Field (FF)

The visual field location corresponding to where the neuron’s RF would be after the eye movement

Head-centered Coordinates

Location relative to the head, invariant to changes in eye position. Sometimes used interchangeably with spatiotopic to mean non-retinotopic coordinates

Memory trace remapping

A variant of predictive remapping; stimulus is presented in FF prior to the saccade, but neuron responds after the saccade

Planar gain field

Multiplicative modulation of neuronal tuning to encode information about multiple stimulus dimensions (e.g., retinotopic location and gaze angle)

Predictive remapping

Remapping that begins prior to the saccade

Receptive field (RF)

Location in the visual field where a given neuron is sensitive

Retinotopic Attentional Trace (RAT)

Spatial attention that lingers at the previously relevant retinotopic location, for a brief period of time immediately after saccades

Retinotopic Coordinates

Location of object in eye-centered reference frame. The retinotopic coordinates of a stationary object change with each eye movement

Spatial remapping

Presaccadic shift in spatial RF position, starting 100-200ms before a saccade

Spatiotopic Coordinates

Location of object in world-centered reference frame. Spatiotopic coordinates are invariant to changes in eye position