Remapping locations and features across saccades: a dual-spotlight theory of attentional updating

Abstract

How do we maintain visual stability across eye movements? Much work has focused on how visual information is rapidly updated to maintain spatiotopic representations. However, predictive spatial remapping is only part of the story. Here I review key findings, recent debates, and open questions regarding remapping and its implications for visual attention and perception. This review focuses on two key questions: when does remapping occur, and what is the impact on feature perception? Findings are reviewed within the framework of a two-stage, or dual-spotlight, remapping process, where spatial attention must be both updated to the new location (fast, predictive stage) and withdrawn from the previous retinotopic location (slow, post-saccadic stage), with a particular focus on the link between spatial and feature information across eye movements.

1. Introduction

We make rapid, saccadic eye movements several times each second, with the resulting input to our visual system being a series of discrete, eye-centered snapshots. Yet the world does not appear to “jump” with each eye movement. Even though visual input is initially coded relative to the eyes, in “retinotopic” coordinates, we perceive objects in stable world-centered, “spatiotopic” locations. Most theories of visual stability across eye movements involve some sort of updating, or “remapping”, signal that helps align visual input from before and after a saccade [1–13]. Here we review what is currently known – and unknown – about this remapping process and its implications for visual perception.

The goals of this review paper are threefold: First, to offer a brief review of the current state of the literature regarding spatial remapping across saccades. Second, to describe a unified theory of remapping (the dual-spotlight theory), which focuses on the periods both immediately before and after a saccade. Third, to highlight new research on an often-overlooked aspect of remapping: the implications for non-spatial processing; i.e. perception of visual features and objects.

2.1. Spatial remapping across saccades

Predictive spatial remapping.

A seminal finding that has driven much of the past few decades of research was the discovery that certain visually responsive neurons begin to remap their activity in anticipation of an eye movement [1]. That is, in the 100–200ms before a saccade is executed, a neuron may respond to a stimulus presented outside its current receptive field, if it falls in its “future field” – where the receptive field would be after the eye movement, even though the eyes have not yet moved (Fig. 1). Predictive remapping has been found in many visual areas in monkeys [1–4], along with analogous results in human fMRI [5], ERP [6], and behavior [7]. Visual sensitivity and feature selective tuning are also predictively enhanced at the saccade target [14–16], i.e. the future fovea, but here we primarily focus on remapping of peripheral locations.

Figure 1. Dual-spotlight model of attentional remapping.

A: Real-world example illustrating sustained covert attention at a spatiotopic task-relevant location (dog), while saccading from soccer ball to baby. Orange dot indicates current fixation location (horizontal position also shown with cartoon eye icon below image); light orange arrows indicate planned or recently completed saccade trajectory. When viewer is initially fixating on soccer ball but attending to dog, the attended location (yellow circle) is in the upper right visual field. Saccading to the baby moves the dog into the upper left visual field. The dog’s spatiotopic position remains stable, but its retinotopic position has changed. From top to bottom, panels indicate early pre-saccade, later pre-saccade (predictive remapping period), early post-saccade (retinotopic trace period), later post-saccade. Red and blue circles correspond to other locations that may be attended during remapping, as defined in B,C. B: Hypothetical responses of two visual neurons with different spatial receptive fields. Yellow circle represents to-be-attended spatiotopic location. Before the saccade, the attended location falls within Neuron A’s receptive field; after the saccade, it falls in Neuron B’s. “Predictive remapping” is when Neuron B begins to respond in anticipation of the saccade. “Retinotopic attentional trace” is when Neuron A continues to respond for a period of time after the eye movement. Thus there is a period of time where both spatiotopic and retinotopic locations are facilitated. C: Corresponding locations for a behavioral study.

A recent source of controversy has been whether this predictive remapping is truly “forward” remapping, in the original sense of receptive fields shifting to the future field, or whether receptive fields instead demonstrate “convergent remapping”, shifting instead toward the saccade target [8, 9]. Other ongoing debates center on whether remapping is best thought of as a process by which receptive fields themselves shift, or rather reflects a shifting of “attentional pointers” [10], with at least one recent study suggesting that the remapping of attentional state may be dissociable from spatial receptive field remapping [11].

These important debates (reviewed elsewhere, including [12]) tend to focus on the nature of predictive spatial remapping; but the underlying assumption is that remapping does occur predictively, and that this predictive updating of visual information helps ensure visual stability once the eyes land. However, some recent evidence has challenged this notion, demonstrating that remapping may not be as fast or efficient as previously thought [13, 17–25], at least during remapping of spatial attention, and thus a full understanding of visual stability across saccades and remapping processes needs to take into account the period immediately after eye movements as well.

The post-saccadic retinotopic attentional trace

In a human behavioral task, Golomb and colleagues had participants sustain spatial attention at a spatiotopic location over a working memory delay, finding that attentional facilitation erroneously lingered at the retinotopic location of the cue for a brief time (100–200ms) after an eye movement before updating to the correct spatiotopic location [13]. This “retinotopic attentional trace” has been subsequently reported across a variety of behavioral tasks [17–20], in addition to both fMRI and ERP [21, 22], and model simulations [23], and is consistent with the idea that visual representations are natively coded in retinotopic coordinates [24] and updated imperfectly [25].

2.2. A unified (pre- and post-saccade) theory of spatial remapping

At first glance, the finding that spatial attention gets updated slowly and is still unstable after an eye movement seems to directly conflict with the evidence for predictive remapping. However, the two findings only conflict if we assume that remapping requires an instantaneous switch, or a single spotlight of attention that shifts from one location to the other. Instead, we propose an alternative theory of remapping that accounts for both effects (Fig. 1) and is rooted in our understanding of neural properties. This “dual-spotlight” theory of remapping is a conceptual theory that describes the remapping of attention, as it is primarily based off studies using behavioral tasks with an attentional component. As noted above, it is still an open question whether the remapping of attention reflects a novel form of remapping [11], and the dual-spotlight theory does not claim that attention is the sole mechanism of remapping; that said, the main implications of this theory – in terms of unifying pre- and post-saccade findings and the potential impact on feature perception – are worth considering in the broader context of remapping as well.

Fast arrival, slow departure: A dual-spotlight theory of attentional remapping.

When a saccade triggers the remapping process, attended locations are updated in two ways: a “turning on” of the new location, and a “turning off” of the old location. Importantly, these two processes do not have to occur synchronously, such that the first may be more rapid and the second slower or delayed in time. As a result, there can be a period of time where attentional facilitation is simultaneously at both locations. Preliminary behavioral evidence for the dual-spotlight remapping theory comes from demonstrations that both predictive remapping and the retinotopic trace can co-exist [19], and that during the few hundred ms following a saccade, facilitation ramps up at the spatiotopic location and ramps down at the retinotopic location, without spreading to intermediate locations [18]. Of course, it is possible that what appears to be dual spotlights could actually be a single spotlight that updates with variable timing across trials, given that the above studies rely on data averaged across trials. However, another set of studies using a different approach revealed a pattern of post-saccade feature-binding errors characteristic of split attention [26, 27; described more in the sections below], providing even more compelling evidence for the dual-spotlight account.

Moreover, the spatiotemporal dynamics of a two-stage or dual-spotlight process are consistent with known neural mechanisms. Neural investigations of covert shifts of attention (with the eyes fixated) have demonstrated that neurons in early visual cortex begin to facilitate the new location before disengaging from the previously attended location [28], and a similar pattern has been found across a saccade, where a new neuron whose receptive field is brought into the attended location ramps up before a previously active neuron ramps down [11]. Visual stability across saccades is also thought to be supported by two distinct sources of feedback with differing time scales: a rapid, predictive corollary discharge signal, and an oculomotor proprioceptive signal that stabilizes more slowly after a saccade [29]. Indeed, a recent computational model of remapping incorporating both of these signals naturally accounts for both the pre-saccadic predictive remapping and post-saccadic retinotopic trace effects [30].

While a number of open questions remain about the precise spatio-temporal dynamics and mechanisms underlying this theory (Box 1), what seems clear is that attentional remapping is not simply a single updating process with a variable time delay, but that the “turning on” and “turning off” processes are at least somewhat separable, which can result in a dynamic period where spatial attention is temporarily at both locations. In the next section, we explore the broader perceptual implications of such a process.

Box 1: Open questions about the dual-spotlight theory.

The dual-spotlight theory posits that spatial attention needs to be both updated to the new location (fast, predictive stage) and withdrawn from the previous retinotopic location (slow, post-saccadic stage) with each eye movement. Of the many open questions that remain to be addressed, a number center on the question: Are the updating and withdrawing two independent processes, or two components of a single process? While the findings discussed above seem to largely rule out the possibility that remapping is a single-spotlight updating process that’s simply variable in time, the degree of their independence remains to be seen. One possibility is that both the predictive component and the retinotopic trace component are automatically triggered together by the same updating signal(s), but the processes themselves overlap in time; this explanation seems favored by at least one computational model [30]. The other extreme is that the two processes are completely independent mechanisms, such that they can vary independently over time and are independently susceptible to individual, trial-by-trial, and/or task context differences. Intermediate possibilities include the idea that predictive spatiotopic remapping may be a task-dependent, active process, whereas the retinotopic trace is more of an automatic, passive decay of neural activity; this account was favored by the initial retinotopic trace findings that early spatiotopic facilitation was variable across experiments and found primarily when the spatiotopic location was task-relevant, whereas the retinotopic trace was found regardless of task demands [13, 17, see also 62]. Interestingly, however, other recent papers have failed to find early retinotopic effects in certain tasks [63]; the effect seems particularly controversial with inhibition of return [20, 64, 65]. Yao and colleagues also failed to find any post-saccadic performance costs stemming from interference from a retinotopic trace distractor, though they also failed to find pre-saccadic interference from distractors at the predictive remapping location, claiming that attention can update very rapidly, at least under certain situations (e.g., an intriguing difference is that their saccades were highly predictable in both space and time; [66]). Intriguingly, a recent study also found that in monkey area V4, the remapping of attention occurred in a manner somewhat consistent with the dual-spotlight theory, with new neurons becoming active before the old neurons disengaged, though this entire “attentional handoff” occurred before the saccade in their task [11]. Thus, it is possible that the efficiency of remapping (for one or both stages) may be dependent on factors such as cognitive control, motivation, reward, predictable context, and/or attentional demands.

2.3. Implications for non-spatial processing: feature and object perception

Saccadic remapping is by definition a spatial problem; the shifts induced by eye movements change the spatial perspective of the viewer, causing a mismatch between pre- and post-saccadic retinotopic positions. But in the real world, our visual systems don’t just process spatial information in isolation; visual scenes are full of rich feature and object information. What happens to these non-spatial representations during spatial remapping induced by saccades?

Are object features also remapped?

One of the greatest unresolved debates in the remapping literature is whether information about an object’s features and identity is remapped alongside the spatial signal, or whether remapping is purely a spatial process, such that feature and object information must be re-processed and re-bound to an updated spatial pointer after remapping [e.g., 10]. In the perceptual literature, spatiotopic transfer of visual aftereffects was initially touted as strong evidence in favor of predictive remapping of features [31], then contested [32–34]. However, more recent studies have again found support for spatiotopic aftereffects [35, 36], including the idea that these build up over time [37]. Evidence for spatiotopic transsaccadic feature integration has similarly been mixed [38–45]. Other behavioral studies have examined whether an object presented prior to an eye movement preserves its integrity across the saccade [46, 47]. Interestingly, a common source of discrepancy across all of these behavioral studies is whether effects are found in retinotopic or spatiotopic coordinates after a saccade [12]; given that the dual-spotlight theory predicts spatial pointers temporarily coexisting at both locations, if features are indeed remapped, perhaps this perceived discrepancy is not so unexpected after all.

Neurally, evidence for feature remapping has been ambiguous as well. A recent fMRI adaptation study used a similar approach to the behavioral tilt adaptation studies noted above, finding evidence for both retinotopic and feature-specific spatiotopic adaptation after a saccade [48]. Using multivariate pattern analysis of fMRI data, Lescroart, Kanwisher, and Golomb [49] investigated more directly whether stimulus category information could be decoded from remapped responses and failed to find evidence of automatic remapping of stimulus content, though this study also raises doubt about the type of remapping detectable with fMRI [5] and if it is the same as the predictive remapping signal in neurons (or behavior, for that matter). Intriguingly, an EEG study applying a similar approach did find evidence in support of remapping of stimulus content [50], though they tested the case of peripheral-to-fovea remapping, which might involve special processing [51, 52]. While neurophysiology could potentially provide the most direct evidence, this question has remained largely untested with this methodology; two recent studies have attempted to address this question, reaching opposite conclusions: Yao and colleagues found that remapped memory trace responses in MT do not contain information about motion direction [53], while Subramanian and Colby reported a small fraction of LIP neurons that showed properties that could be consistent with feature-selective remapping [54], although these latter results were not robust or completely consistent with true feature remapping.

Is feature and object perception affected by a saccade?

A related – and historically less investigated – question is whether feature perception and object recognition processes themselves are fundamentally affected by eye movements. It has long been known that people are susceptible to perceptual errors such as spatial mislocalization and saccadic suppression around the time of a saccade [reviewed in 55]. Visual memory across saccades may also be impaired [56], though the costs may be less severe in native retinotopic coordinates [25, 57].

Such costs may be expected consequences of unstable spatial representations during the remapping process. But given that spatial attention plays another fundamental role in visual perception – linking information about features and objects in the world – then theories of remapping should consider the consequences for feature/object perception as well. In the case of the dual-spotlight remapping theory, the critical postulation is that during dynamic remapping, attention can be simultaneously highlighting two different spatial locations. If true, this divided attention may carry interesting perceptual consequences beyond spatial cognition.

Spatial remapping and the feature-binding problem.

A principle long touted in the field of visual object perception is that spatial attention is critical for knowing “what” is “where” and for solving the famous Binding Problem [58]. Many theories of object perception center on the idea that object files are defined by location and spatial attention acts as the “glue” that binds an object’s features together [59]. If spatial attention is critical for feature-binding, but spatial attention is remapped across eye movements in a two-stage or dual-spotlight process, that begs the question: what happens to feature-binding and integrated object perception during dynamic remapping?

A recent study from our group [27] used a continuous-report paradigm to test feature perception following an eye movement, revealing that when subjects were supposed to report the color of a briefly presented spatiotopic target, the color they actually reported was systematically shifted in color space, toward the color of a simultaneously presented distractor occupying the retinotopic location of the cue. Probabilistic mixture modeling revealed that these perceptual errors consisted of both crude “swapping” errors (reporting the retinotopic color instead of the spatiotopic color) and subtler feature mixing (as if the retinotopic color had blended into the spatiotopic percept). Szinte et al subsequently showed a similar perceptual integration (motion integration) between spatiotopic and future retinotopic locations during the pre-saccadic period [60]. More recently, Dowd & Golomb extended this paradigm to probe multi-feature objects, using a joint continuous-report (i.e., reproduce both the color and orientation of a target) and joint probabilistic modeling to assess object integrity [61]; attentional updating after an eye movement triggered an increase in object-feature binding errors, including illusory conjunctions—e.g., mis-binding the color of the spatiotopic target item with the orientation of the simultaneously presented retinotopic distractor item [26].

Importantly, additional comparison experiments testing dynamic spatial attention outside the context of eye movements [27, 61] demonstrated that both within-feature mixing errors and the multi-feature breakdowns of object binding are specifically associated with situations in which covert attention is simultaneously split across two different locations (Fig 2), providing strong evidence that remapping involves a transient splitting of attention, as predicted by the dual-spotlight theory of remapping. An alternative single-spotlight process of remapping would predict that a single focus of spatial attention shifts from one location to another, such that at any given moment, attention is either at the updated (spatiotopic) location or stuck at the initial (retinotopic trace) location. If this were the case, we would expect remapping to involve primarily a shift of attention, which should result in a mixture of some trials in which attention had successfully remapped and some trials in which attention was still at the previous retinotopic location, which would induce a different pattern of feature-binding errors [27, 61]. The fact that the type of feature-binding errors found immediately after a saccade instead resembles those found during split attention suggests that both the remapped spatiotopic location and the lingering retinotopic trace location are being simultaneously selected by spatial attention during this period, at least on some trials. Moreover, this transient splitting of spatial attention induced by remapping affects not just spatial processing, but can have striking consequences for feature and object perception, such that features from two different spatial locations may be temporarily mixed together. It remains to be seen the practical consequences of these challenges on real-world perception (Box 2).

Figure 2. Implications of dynamic spatial attention on feature perception and object integrity.

Cartoon depicting simplified tasks and results from [27, 26, 61]. Top: When attention must be covertly shifted from one location to another (dark gray to black box), and the stimulus array is presented 50ms after the shift cue, subjects either report the correct features or misreport (swap) the features of the distractor at the initially attended location, but object integrity is preserved. Middle: When covert attention is simultaneously split across two different locations and subjects are post-cued to report one of them, feature reports may be distorted (e.g. blend of both colors), and object integrity is degraded (e.g. reporting color of one item and orientation of the other). Bottom: When covert attention must be maintained at a spatiotopic location (black) while executing an eye movement (white arrow) elsewhere, and the stimulus array is presented 50ms after the eye movement, interference is seen from the distractor at the retinotopic trace location (dark gray). Errors here are consistent with those seen during splitting of attention, suggesting spatial attention is temporarily highlighting both the spatiotopic and retinotopic trace locations after a saccade.

Box 2: Real-world implications.

If the visual system is susceptible to all sorts of perceptual errors around the time of an eye movement, what are the consequences for real-world vision? And why do we still perceive the world as stable? Our visual systems seem to solve the visual stability problem so seamlessly that most non- vision scientists aren’t even aware that we make multiple eye movements every second. One possibility is that remapping is but one of several mechanisms supporting visual stability across saccades, including retinal cues (e.g. stable visual landmarks and object correspondence, including the saccade target itself; [67–70]), as well as top-down, default assumptions or expectations about stability [71, 72]. Also, in the real world, visual objects of interest tend to remain intact as we saccade to directly fixate them, and thus predictive remapping of features and intact object integrity may be less crucial if we can simply re-process the object using the high-resolution fovea upon landing. Thus, perhaps it is not a coincidence that the timing of natural saccades leaves a few hundred milliseconds to re-stabilize before moving again; such that even the post-saccadic stage of remapping would be comfortably completed.

3. Conclusions

When predictive spatial remapping was first discovered [1], it spawned an exciting field of study built on the premise that remapping might help solve the fundamental challenge of visual stability across saccades. However, while much neurophysiological, behavioral, and neuroimaging work has followed over the decades, many fundamental questions and debates remain today. This brief review focused on two timely issues related to spatial and feature remapping: (1) that a view of remapping based solely on the predictive component is limited, as promoted by the more unified and comprehensive dual-spotlight theory of remapping; and (2) that a true understanding of visual stability – or instability – across saccades needs to account for non-spatial processing as well.