Visual stability based on remapping of attention pointers

Abstract

When we move our eyes, we easily keep track of where relevant things are in the world. Recent proposals link this stability to the shifting of receptive fields of neurons in eye movement and attention control areas. Reports of “spatiotopic” visual aftereffects have also been claimed to support this shifting connectivity even at an early level, but these results have not held up. Here we describe the process of updating visual location as predictive shifts of location “pointers” to attended targets, analogous to predictive activation seen cross-modally. We argue that these location pointers, the core operators of spatial attention, are linked to identity information and that such a link is necessary in order to establish a workable visual architecture and to explain frequently reported positive spatiotopic biases.

The challenge of eye movements

Because of our frequent eye and head movements, the image of the world moves constantly on our retina. Despite this instability, we know where things are around us, at least well enough to get by. Hurrying down the stairs we may slip but quickly grab hold of a handrail that we only glimpsed briefly a moment earlier. This knowledge might be represented in the brain as an explicit spatiotopic (see Glossary) map of the scene, where object locations are given in world-based coordinates that are independent of the orientations of our eyes, head, and body. This conjecture of an explicit spatiotopic map has been proposed several times [1-5], but has been vigorously challenged as unnecessary and unsupported [6-12]. This long debate has been completely transformed by two recent discoveries: remapping and spatiotopic visual aftereffects. In remapping, a neuron in attention and saccade control areas can be activated by a stimulus far outside its receptive field, even in the opposite visual field, if an impending saccade will bring that stimulus into the classical receptive field of the cell [13-18] (see Figure 1). This predictive transfer of activation has been attributed to shifting receptive fields and been proposed as a possible mechanism for visual stability. In the case of visual aftereffects, adaptation to a tilted grating, for example, causes a vertical test to appear tilted away from vertical in the direction opposite the adaptation grating. This negative aftereffect is typically seen at the same retinal location as the adapting stimulus, but if an eye movement intervenes between adaptation and test, it is also reported at the same spatial location as the adaptation [4, 19-22]. These spatiotopic aftereffects have been taken as evidence that remapping transfers low-level feature information around the time of a saccade. However, here we present evidence that challenges the shifting receptive field notion and the finding of spatiotopic aftereffects. We will argue that remapping is best seen as a transfer of activation to prepare for a predictable incoming stimulus, similar to that found in the somatosensory system when a stimulus is seen to be approaching a particular location on the body [23] or in the auditory system when a loud sound is anticipated [24]. In the visual system, in preparation for a saccade, this transfer of activation to the target’s upcoming location gives a head start for attention benefits at, or programming a saccade to that location. We show that remapping does not require briefly rewiring the receptive field to the new retinal location nor does it involve the transfer of any low-level features or gain settings. This simplified view of remapping helps organize current results in eye movements and attention within a common framework, suggesting that these maps are the functional core of spatial attention. We begin with a brief review of the remapping effect (see [7, 11, 12, 25] for detailed reviews).

Figure 1. Remapping.

a) the “+” indicates the fovea and a face is the target of an impending saccade. b) A second stimulus is flashed on just before the saccade, activating a cell (in LIP) with a receptive field, RF1, at the location of the new stimulus. c) The activation of the cell with classic receptive field on the right is transferred to a new cell whose receptive field, RF 2, is on the left; critically, its receptive field lies where the second stimulus will land after the saccade. This cell’s predictive activation gives a head start for the processing of this stimulus when it lands. d) The saccade has landed on the face and the truck lands in the pre-activated receptive field. In the physiological experiments [e.g., 13] the second stimulus is extinguished before the saccade lands so that any response from RF2 can be attributed solely to the pre-saccadic activity at the remote location and to the intended saccade and its corollary discharge or efference copy.

Remapping: Shifting receptive fields or shifting pointers?

The remapping of activity around the time of a saccade has been attributed to the shifting of receptive fields (Figure 2) [13-18] and has been proposed as the mechanism that keeps track of locations in space when the eyes move [11-18, 24]. Wurtz [11] has provided an elegant argument that corollary discharge of the impending eye movement command (also known as efference copy) transfers activity on a retinotopic map in order to keep track of the locations of objects in the world. Sommer and Wurtz [17] demonstrated that this remapping is at least partially under the control of signals from the superior colliculus (SC), closing the link between eye movements, corollary discharge, and visual stability. Wurtz [11] proposed that remapping does not lead to an explicit spatiotopic map but instead underlies the updating of the locations of targets of interest on a retinotopic representation. Only the activity for a few, relevant items is remapped [16] and Wurtz’s proposal is really a minimalist theory of visual stability – keep track of the locations of currently attended items and nothing else is required.

Figure 2. Simplified examples of two remapping mechanisms.

a) An upcoming saccade to a target at 10° with a briefly presented second target at 30°. This will activate some cells with classic receptive fields at 20° even though the secondary target does not land there until after the saccade. b) Shifting field. Each cell in a salience map (shown as LIP here) is connected to all locations on the retina through interneurons. Input from an oculomotor center (FEF here) turns on subsets of the input that correspond to the default location when fixating (saccade of 0°) or the appropriately offset location when a saccade is imminent (10° here). The shifted location to be activated depends on the cell and the saccade. The cell shown with its classical receptive field at 20° will now respond to the input at 30°. c) Activation transfer. Horizontal connections link all cells on a salience map (LIP here). An upcoming saccade will open all connections of the appropriate offset, transferring any activation from cells with pre-saccadic targets to locations where those targets will fall after the saccade. These connections must be learned [30, 31]. Here the secondary target activates the cell with receptive field at 30° and this activity is transferred to the 20° cell through the active interneuron.

We believe this proposal has great promise as it relies on activity in these areas to act as pointers to the locations of the targets, a role that fits with the characterization of the lateral intraparietal area (LIP), frontal eye fields (FEF), and SC as salience maps for spatial attention and maps of potential saccade targets [26-28]. We adapt Wurtz’s proposal here by linking remapping to the transfer of activity, rather than shifting receptive fields (see Box 1), and by claiming that it is the updating of attentional “pointers” (Fig. 3) that does the work: attended locations are all that need to be tracked, as feature information, which remains unchanged, can be dealt with independently (Box 2).

Box 1: Mechanisms of remapping.

We outline two current proposals for the remapping process. First, the shifting receptive field model, as suggested by Duhamel et al [13] and Sommer and Wurtz [18], requires broad input from all locations on the retina to each cell (Fig. 2b). A level of interneurons allows these inputs to be gated by the current saccade target, shifting all receptive fields from their default location to a location offset by the saccade vector. The default location of each cell is its classic receptive field location (which can be considered to be switched on by a saccade vector of 0°). This shift would maintain the stimulus specificity of the cell, if any, as it processed input from other retinal sites. Could this remapped receptive field mediate the transfer of visual adaptation from pre- to post-saccadic locations [4, 19-22] – the spatiotopic aftereffects reported by several authors? Not likely, as Melcher and Colby point out [25]. The remapped connection is brief, perhaps 50 to 100 msec, replaced shortly after the saccade lands with the cell’s default connection to its classical receptive field. Once the connection reverts to the default receptive field, stimulus information is arriving over unadapted connections.

Second, two neural models propose that learned horizontal connections in, for example, LIP, FEF or other salience map sites, determine which cell will be at a target’s post-saccadic location [30, 31]. The connection activates that cell without passing any stimulus information (Fig. 3b). To accomplish this, all cells in LIP must be connected together and the link between currently active cell, the efference copy (from FEF), and the predicted cell is learned across exposure to the pairings of pre- and post-saccadic activations that occur in LIP for each of the multitude of saccades occurring in our visual experience. Could this remapped activation mediate the transfer of visual adaptation from pre- to post-saccadic locations? Again, not likely, as the learned connections only transfer activation, they are not specific to stimulus features nor to any change in the sensitivity to those features.

Figure 3. Remapping and attention.

A network of areas form a target map that subserves spatial attention as well as eye movements [26-28]. a) Peaks of activity (in red) index the locations of targets and specify the retinotopic coordinates at which the target’s feature data are to be found in earlier visual cortices which are shown, highly simplified, as a stack of aligned areas divided into right and left hemifields with the fovea in the center. In object recognition areas, cells have very large receptive fields (heavy black outline) and depend on attention to bias input [67] in favor of the target and suppress surrounding distractors so that only a single item falls in the receptive field at any one time. b) Just around the time of a saccade, the activity peaks for the 3 attended targets shift to the locations the targets will have at the end of the saccade, (c) and, following the saccade, what had been the saccade target, at the right, now lands in the fovea.

Box 2 Object identity and remapping — the hard binding problem.

Attention has been characterized in a variety of ways but here we focus on the target or salience map [16, 26-28]. In this approach, the array of activations on areas involved in eye movement and attention control (LIP, FEF, SC) specify the location of the target’s features in early retinotopic visual cortices enhancing processing at those locations and suppressing surrounding features, a proposal quite like the master map of Treisman and Gormican [59]. The suppression of distractors surrounding the target is critical – if a distractor is also present in the receptive field and is not suppressed, its features are unavoidably mixed with those of the target producing crowding [60].

An activation peak on the target or salience map is the temporary pointer or location token [61, 62] for the target. But it does not specify which target nor provide any link to the target’s identity. Although several studies point to parietal and frontal areas for visual short term memory that might preserve location information [29, 63, 64] but feature information may be stored globally in earlier visual cortices [48, 49]. There is little information yet on the mechanism that links the representation of locations in the saccade and attention areas to these temporary memories let alone to the ventral regions that would establish those identities. Many have proposed temporary data structures that would manage all these connections between location and identity – object files for example, or FINSTs – and recent work suggests some structures that may underlie these functions [47]. However these links are established, whenever corollary discharge remaps the activation peak in saccade and attention areas, sending it on to its predicted postsaccadic location, it needs to take the link to the target’s identity with it so that the same identity will be continuously attributed to this target as the eyes move. Remapping does not solve this hard binding problem, it only adds additional challenges to it.

In their original paper [13], Duhamel, Colby, and Goldberg presented a target just before a saccade and discovered that some cells with classical receptive field at the anticipated, post-saccadic location of the target would respond even when the target was turned off before the saccade landed. The authors claimed that “the location of the receptive field is shown to shift transiently before an eye movement.” (p. 90) They concluded that parietal neurons have access not only to information in the fixation field but also to information at other retinal locations. This characterization of shifting fields has persisted across the years, both for LIP neurons [17] and FEF neurons [18].

With remapping, a stimulus produces a response from a cell whose receptive field is normally elsewhere and this remote activation is not unreasonably labeled a shifted receptive field. However, this characterization is inappropriate and we give two examples to explain why. First, there are analogous cross-modal anticipatory responses where no shifting receptive fields can be invoked. For example, some cells in area 5 of posterior parietal cortex, a somatosensory processing area [23], that respond to stimulation of a specific body part (e. g., the hand or shoulder) also show a change in firing rate when the experimenter is seen approaching that body part as though contact would be made. In this case, the somatosenory neuron is responding to a visual stimulus that is not just outside its receptive field, it is outside its modality. Not only would the rewiring have to be from each somatosensory neuron to all possible retinal locations, but the predicted contact location does not even correspond to the current location of the visual stimulus, but to its eventual contact point with the animal’s body. This anticipatory activation for cells with receptive fields at the location of future contact rules out any notion of a “shifting receptive field”. Second, remapping has also been seen in the case of FEF units that show short-term spatial memory responses [29]. Stimuli were flashed for 50 ms, 200 to 1000 ms before a saccade. A significant proportion of cells that did not respond to the flashed stimulus alone (the stimulus location was outside the cell’s receptive field) did respond if the saccade would have brought the stimulus onto its receptive field, just as in the classic remapping case. However, in this case there can be no rewiring of receptive field to earlier input streams to produce this remapping because the stimulus is long gone: there is no activity on the retina or in earlier cortices at the time of the remapping. The only source for the remapping is a transfer of activity from the currently active cells, holding the memory of its location, to its predicted postsaccadic location, maintaining a pointer to that remembered location in the world.

If, as these two examples indicate, receptive fields do not shift, how is the activation transfer produced? It may be generated by learned horizontal connections (Fig. 2) or by some more flexible mechanism yet to be discovered, analogous to the computation of expected touch location seen in the somatosensory system. Two separate groups [30, 31] have presented models in which horizontal connections in a salience map (for example, LIP) can transfer activation to the appropriate post-saccadic location (see Box 1). With the transfer of activation as the mechanism of remapping, it is clear that location counts, rather than content, as locations and not features are the critical properties represented in these particular parietal, frontal and subcortical regions. Once the activation shifts to the new location on the attention and saccade target maps (Fig 3), its downward projections can rapidly begin to facilitate processing at the expected post-saccadic locations in earlier retinotopic cortices.

Attention-based spatiotopy

The remapping hypothesis of spatiotopy proposed by Wurtz [11] and others underlines the potential of remapping to keep track of targets as the eyes move and so to derive a working spatiotopy of locations (Fig. 3) and, in particular, a working spatiotopy for spatial attention. Spatial attention shares spatial maps with saccade control centers in areas like SC, LIP and FEF [32] and the activity peaks on these maps do more than just indicate or point at each target’s location for purposes of programming a saccade. Each activation also indexes the location of that target’s feature information on other similarly organized retinotopic maps throughout the brain. Activity in these attention / saccade maps can project to corresponding locations in other regions, enhancing processing for information from the corresponding target location in these areas. The attentional benefits of the projections from these maps to other retinotopic visual cortices have been demonstrated in microstimulation studies [see 33 for a review]. When stimulating cells in FEF or SC with a movement field, for example, in the lower right quadrant, a high stimulating current triggers a saccade to that location but a slightly weaker stimulation that does not trigger a saccade generates either enhanced neural response for cells with receptive fields at that location [33] or lowered visual thresholds for visual tests at that location [33].

Duhamel et al [13] had originally argued that the remapping phenomenon was unrelated to attention because it did not occur when the monkey shifted attention to the saccade target but then withheld an eye movement. However, as Berman and Colby [12] point out, with an attentional shift alone, nothing will move on the retina and there is no need to remap anything. The shifting of attention to the saccade target that occurs just prior to the saccade [34, 35] is essential for acquiring information about the target but it is not the same as the shift of activity that accompanies the actual execution of the saccade. Interestingly, for the saccade target itself, the remapping must return the attentional “pointer” from the saccade target to the fovea, which is the future location of the target. This return of attention to the fovea (or nearby [36]) should occur only if the saccade execution is imminent and it should be synchronous with the shift of attention from the current locations of other targets of interest, such as recent transient flashes, to their predicted post-saccadic locations.

What is the evidence that attention remaps to its expected post-saccadic location? Direct evidence comes from studies of apparent motion, the perception of motion between two successive flashes. Apparent motion has been described as the consequence of dragging attention from an initial stimulus to the displaced location of a second [37], linking the two locations together as the changing location of a single target. If a saccade occurs between the first and second stimuli, apparent motion is seen in spatiotopic, not retinotopic coordinates [38, see Supplemental Material for a demonstration movie] suggesting that the attention pointer to the presaccadic location is correctly shifted to the target’s expected post-saccadic location, enabling the detection of the target displacement as apparent motion in world coordinates. Relative location information can be accumulated across saccades but only for up to about 3 locations [39], as would be expected for a capacity-limited, attention-based mechanism. Additional evidence of attention remapping has also come from cueing experiments where benefits of a cue presented before a saccade have been found in both retinotopic and spatiotopic locations following the saccade [40, 41].

The shifting of attention pointers can keep track of target locations but what about target identities? There is no mechanism built into attention remapping that represents or tracks identity and yet, it is obvious that we do know what is where after an eye movement, at least for attended targets. Change blindness, for example, has shown that changes across saccades are detected for attended items but not for unattended items [42, 43]. There is, as well, evidence for trans-saccadic spatiotopic memory [9, 44], priming [45], and integration [5]. These instances of transsaccadic maintenance of information may require mechanisms like object files [9, 46, 47] or non-spatial memory stores [48, 49] but there is no indication as yet how these mechanisms are linked to the location tracking processes described here (Box 2). Next, we review the nature of feature or identity information that might be maintained across saccades.

Feature-specific spatiotopy

A great deal of earlier research demonstrated just how little information is retained in world coordinates across saccades [9]. However, recent studies in brain imaging and visual psychophysics have claimed to find spatiotopic representations and link them to the remapping of visual feature information from presaccadic to postsaccadic target locations [4, 19-22, 50, 51]. In the case of functional brain imaging, two studies reported spatiotopic responses in extrastriate regions of occipital cortex that indicated a remapping of featural information. D’Avossa et al [50] found that activation in the classically retinotopic portion of MT was invariant in a spatiotopic reference frame and not in a retinotopic frame. McKyton and Zohary [51], in an fMRI study in the lateral occipital complex (LO), showed that object-specific adaptation was largely dependent on the position of objects on the screen and not their position on the retina.

Psychophysical adaptation experiments also showed spatiotopic aftereffects for tilt, shape and face adaptation [4], motion aftereffects [20], duration adaptation [21], and for binocular rivalry adaptation [22]. According to these studies, after adapting to a particular stimulus at one location and then making an eye movement, feature-specific negative aftereffects (a bias away from the adapting feature) are found at the spatial location where the adaptation took place, far removed from the location on the retina where adaptation took place. Since the classic literature on visual aftereffects had shown that effects are overwhelmingly centered at the same retinal location as the adaptation [52-53], these findings of aftereffects at spatiotopic locations attracted enormous attention and the importance of the findings was amplified by their potential link to the physiological findings of remapping. A subsequent study showed a tilt aftereffect in the remapped location briefly before a saccade [19] that again supported the notion that remapping may involve rewiring of information from earlier levels that could carry featural information.

However, these findings have not held up well in subsequent experiments. The report of spatiotopy in MT responses was challenged by a recent fMRI study that found no evidence of spatiotopy in either MT or LO [54] (or indeed anywhere from V1 through LO). The psychophysical experiments following up the original claims of spatiotopic aftereffects have not fared well either. Subsequent tests for adaptation to motion have not supported spatiotopic effects [55, 56]. Similarly, studies have found robust retinotopy, not spatiotopy, for the aftereffects of adaptation to face gender [57], for the aftereffects of orientation [58], and effects of adaptation to flicker on duration judgments [Bruno, A., Ayhan, I., & Johnston, A. (2009). Retinotopic adaptation on apparent duration is independent from perceived temporal frequency. 2009 Vision Sciences Society annual meeting]. The report of spatiotopy in binocular adaptation has not yet been re-examined but this spatiotopic effect is small (about 20% of the retinotopic effect) and is not significant until 10 seconds or more following adaptation, properties that do not seem in line with the rapid and reliable oculomotor remapping processes.

This absence of reliable transfer of low-level feature information (specifically the feature gain settings that underlie adaptation) across saccades weakens the claims for a link between the transsacaddic transfer of visual features and the oculomotor process of remapping in LIP and FEF. Physiological remapping that putatively underlies the stability of the visual world needs to be robust, rapid, and reliable. The pattern of spatiotopy reported for aftereffects is too variable to attribute these effects to oculomotor remapping. The properties that do transfer across saccades [5, 9, 44, 45] may be related to maintenance of a link between the target’s identity and its locations. This trans-saccadic memory [47] must call on other mechanisms beyond the updating of location (Box 2).

Summary

Spatiotopy as an explicit representation for world coordinates in the visual system had been abandoned for many years due to the failure of almost all tests of trans-saccadic fusion [55, 56]. The discovery of remapping in LIP [13], SC [15] and then FEF [14] and its characterization as shifting receptive fields led to a reawakening of interest in this topic and to explorations of spatiotopy in visual aftereffects and brain imaging. Reports of spatiotopic aftereffects added momentum to this new interpretation of visual stability by implicating a shifting of connections even through early levels of the visual system. However, as we showed here, remapping is more likely limited to the predictive transfer of activation for attended targets, a process that does not shift receptive fields or transfer low-level feature information such as changes in gain control settings that underlie adaptation effects [4, 19-22]. Whatever small spatiotopic aftereffects might exist are likely only superficially related to oculomotor processes, if at all.

Nevertheless, as pointed out by Wurtz [11] and others, remapping, however it is produced, can update target locations on a retinotopic map to keep track of targets as they eyes move. This activation transfer serves as an early warning to cells whose receptive fields are about to receive an attended target and suggest that these maps on which the transfer operates are the functional core of spatial attention. Additional mechanisms are needed to maintain target identity and link it to these location pointers as they are updated [47].

Outstanding Questions

1. Some cells show responses to a target outside their classical receptive field if an imminent saccade will bring that target into their receptive field. Do these remapped, anticipatory responses show any selectivity for the features of that target? Although cells in LIP, SC, and FEF show little feature specificity, those in LIP do show a weak directional selectivity [65]. Cells in V3A and other areas of visual cortex, which also show remapping responses [66], have a wider range of feature specificity. No experiments have yet tested whether the remapped responses of cells in LIP or these other visual areas show specificity to the features of the target. If the remapped responses do, in fact, reflect rewired connections to the retina, they should maintain the feature specificity of their classical receptive field. On the other hand, if only a predictive activation is involved, there is no requirement for any feature specificity in the remapped response.

2. Is unsupervised learning possible in Keith and Crawford’s [30] model for activation transfer (Box 1)? In their model, there is an expected outcome that drives learning to the appropriate connections. To be realistic, however, the learning in this model must be unsupervised, as it is in real brains.

3. What is the physiology underlying the temporary links, “object files”, that combine location and identity information of each target. Is the object file the same as visual short term memory [48, 49, 63, 64] and if so, how is it linked to object recognition areas [47]?

Glossary

Aftereffects

Prolonged exposure to visual patterns causes adaptation in the eye and brain. Aftereffects are the effects of this adaptation on subsequent perception. For example, the aftereffect of staring at a leftward-tilted line is that a vertical line will appear to tilt to the right.

Corollary discharge

See “efference copy”

Efference copy

When a signal is sent from the brain to the muscles to generate a movement (i.e., an efferent motor command), efference copy is the message that may also be sent to perceptual systems to predict or anticipate the sensory consequences of the movement as it is occurring.

FINST

The “Finger of Instantiation” [61] was suggested to provide a continuously-updated spatial index of an object, allowing it to be tracked over time and space.

fMRI

(Functional Magnetic Resonance Imaging). A technique for detecting oxygen intake in specific regions of the brain. Regions with more de-oxygenated blood are considered to have been recently active.

Frontal Eye Fields

An area of frontal cortex associated with eye movements and spatial attention, and also with saccadic remapping.

Gain setting

The specific relationship between the input and the output of a neuron or population or network of neurons. Gain settings can be modified by adaptation.

LIP

(the lateral intraparietal area). An area of parietal cortex also associated with eye movements and spatial attention, and also with saccadic remapping.

LO

(the lateral occipital visual area). Commonly associated with object segmentation and attention.

MT

(the Middle Temporal visual area, also known as V5). Commonly associated with processing of motion information.

Object files

This is the idea that attending to a given object creates a temporary representation combining what it is and where it is that is updated whenever the object moves or changes features [46].

Receptive field

A cell’s receptive field is the region of the retina where a visual stimulus can modulate the firing rate of the cell. Early in the visual system receptive fields are very small in size but grow increasingly large as the information progresses through additional stages.

Remapping

An eye movement will shift a stimulus from one location to another on the retina. Remapping is the activation of neurons representing the new retinal location of the stimulus, specifically when this activation occurs earlier than, or in the absence of, the actual arrival of the stimulus in that location when the eye movement occurs.

Retinotopy

The representation of spatial information in coordinates corresponding to where it falls on the retina.

Saccade

Fast movements of the eye that shift the higher-acuity region of the retina (the fovea) into alignment with a new location in space, allowing that location to be inspected in greater detail.

Salience map

A representation of space with areas of activation corresponding to locations that have some current or potential importance.

Spatiotopy

Representation of spatial information in world-based coordinates (as opposed to retinotopic coordinates).

Superior Colliculus

A mid-brain structure involved in eye movement control (also known as the optic tectum).

Transsacadic memory

Visual information that is retained in short-term memory buffer from one fixation to the next that allows for integration, comparison, and detection of changes.

Visual stability

The visual world does not appear to move when the eyes do, even though the eye movement shifts the image across the retina.