Attention and Active Vision

Abstract

Visual perception results from the interaction of incoming sensory signals and top down cognitive and motor signals. Here we focus on the representation of attended locations in parietal cortex and in earlier visual cortical areas. We review evidence that these spatial representations are modulated not only by selective attention but also by the intention to move the eyes. We describe recent experiments in monkey and human that elucidate the mechanisms and circuitry involved in updating, or remapping, the representations of salient stimuli. Two central ideas emerge. First, selective attention and remapping are closely intertwined, and together contribute to the percept of spatial stability. Second, remapping is accomplished not by a single area but by the participation of parietal, frontal and extrastriate cortex as well as subcortical structures. This neural circuitry is distinguished by significant redundancy and plasticity, suggesting that the updating of salient stimuli is fundamental for spatial stability and visuospatial behavior. We conclude that multiple processes and pathways contribute to active vision in the primate brain.

Spatial constancy and attention

Vision is an active process. The world that we see is not simply a direct impression derived from patterns of light on the retina. Instead, the brain constructs an internal representation of the world based on sensory input in conjunction with central cognitive and motor signals. Sensory activity in the brain is modulated by attention and memory and even by the intention to act. We focus here on how attention and motor intention work together to create the percept of a stable visual world.

Why does the world appear to stay still when we move our eyes? Attempts to understand our experience of spatial constancy have a long history. More than a century ago, Helmholtz (Helmholtz, 1866) hypothesized that the “effort of will” involved in making an eye movement simultaneously adjusts our perception to take that eye movement into account. He proposed that when a motor command is issued to shift the eyes in a given direction, a copy of that command, known as a corollary discharge, is sent to brain areas responsible for generating an internal image of the world. This image is itself shifted so as to stay in alignment with the new visual information that will arrive following the eye movement. This simple proposal seems to suggest that the entire image should be shifted when the eyes move.

But does the whole image really move? When we begin to think about brain mechanisms of spatial constancy, it seems improbable that every detail of a visual image could be updated with every eye movement. It is more likely that multiple mechanisms contribute to solving the problem of spatial constancy. This is where attention is critical. Much of the problem of spatial constancy may be solved by simply not attending to things. We don't see most of what happens around us; extensive research on the phenomenon of change blindness has shown that we are generally oblivious to unattended changes in our visual environment (O'Regan & Noe, 2001, Simons & Rensink, 2005). We don't see what we don't attend to and so for many objects there is no spatial constancy problem. For those things that we do attend to, updating portions of the internal image may be part of the solution. What does this mean in neural terms? In the following sections we describe a neural mechanism for spatial updating or “remapping,” and how it depends on attention. We begin with attentional effects at the single neuron level in parietal cortex and then discuss remapping of attended visual stimuli; the brain circuits that produce remapping; evidence for remapping in humans; and the neural basis of active vision.

Parietal Mechanisms of Attention

Activity in lateral intraparietal area neurons

Neurons in the lateral intraparietal area (LIP) are active under many conditions. When studied in a standard memory-guided saccade task (Hikosaka & Wurtz, 1983), neural activity can occur throughout the trial (Barash, Bracewell, Fogassi, Gnadt & Andersen, 1991a, Colby, Duhamel & Goldberg, 1996). The task requires the monkey to maintain fixation while a stimulus is presented in the receptive field. After the stimulus has been presented, the monkey continues to fixate during a delay period. At the end of the delay period, the fixation point is extinguished. This serves as a cue to the monkey to make a saccade to the location where the stimulus previously appeared. The monkey is typically required to maintain fixation at this new location for a brief period and then a reward is given. At the time of the eye movement, there is no stimulus on the screen so the monkey must use a memory trace of the stimulus location to perform the correct saccade. Neural activity in this task is typically measured in at least four main epochs: a baseline epoch (after fixation has begun and before the stimulus appears); a visual epoch (after the stimulus appears); a saccade epoch (around the time of the eye movement); and a delay period epoch encompassing the interval between the end of the visual stimulus and the occurrence of the cue that tells the monkey to make an eye movement. In the following sections we describe the effects of attention on neural activity in three of these task epochs.

Attentional modulation in anticipation of a stimulus

The baseline firing rate of neurons in area LIP increases when the animal is working in a task in which it expects that a behaviorally-relevant stimulus will appear. Two tasks have been used to demonstrate this attentional modulation of baseline activity (Colby et al., 1996). In the memory-guided saccade task, described above, a delay is imposed between the time that a stimulus appears and the time that the animal is allowed to look at it. During the delay, the animal has to remember where the stimulus appeared. We infer that the animal has to attend to the stimulus. In addition, when trials to a single location are presented consecutively, the animal can anticipate at the beginning of each trial where a behaviorally-relevant stimulus is about to appear. As shown in Figure 1, neurons in area LIP increase their baseline level of activity in this task (Colby et al., 1996). Even before the stimulus appears, the cell fires more in the memory-guided task (Figure 1B) than it does when the animal is simply fixating without the expectation of seeing an attention-demanding stimulus (Figure 1A). This increase in baseline activity is observed in the population of LIP neurons (Figure 1C) and reflects increased attention directed toward the spatial location where the animal anticipates that an important stimulus will appear.

Figure 1.

Attentional modulation of baseline activity in area LIP. The activity of an example LIP neuron is shown during performance of a fixation task (A) and a memory-guided saccade task (B). Rasters and histograms are aligned on the time when the monkey began to fixate the central fixation point, even before the appearance of the stimulus. Time lines from a sample trial, above the rasters, show the earliest time at which the stimulus appeared. V: vertical eye position, H: horizontal eye position, FP: fixation point, Stim: stimulus in receptive field (RF). In the fixation trials, the neuron does not respond until the stimulus appears. In memory-guided saccade trials, by contrast, baseline activity begins to build up before the stimulus appears. In the population (C), baseline activity was greater on average for the memory-guided saccade task (y axis) compared with the fixation task (x axis). From Colby et al. 1996.

Anticipatory attentional modulation during the baseline epoch is also found in a second task in which the monkey attends to the stimulus in the receptive field without ever generating an eye movement to it. In this peripheral attention task, the monkey initiates the trial by grasping a lever, which causes the fixation point to be illuminated. The animal must maintain fixation as long as the central fixation point is present. The stimulus appears in the receptive field as usual but now the animal has to attend to it without looking at it. When the stimulus dims slightly, the animal has to release the lever. In this task, as in the memory-guided saccade task, baseline activity in the fixation epoch preceding stimulus appearance is increased. In both tasks, baseline neural activity increases specifically when the animal expects a stimulus relevant for behavior to appear in the receptive field (Colby et al., 1996). The increase is not a change in the visual response itself but rather in the neuron's readiness to respond. Similar increases in baseline, pre-stimulus activity are present in ventral stream areas including areas V2 and V4 (Luck, Chelazzi, Hillyard & Desimone, 1997). Attentional modulation of baseline activity has also been observed in human frontal, parietal, and extrastriate visual cortex (Kastner, Pinsk, De Weerd, Desimone & Ungerleider, 1999, McMains, Fehd, Emmanouil & Kastner, 2007).

Attentional enhancement of visual responses

The visual response itself can also be modulated by attention. LIP neurons respond more strongly to the onset of an attended stimulus than they do to the same stimulus when the animal is simply fixating and is free to ignore the stimulus. This augmented visual response occurs in both the memory-guided saccade task and in the peripheral attention task (Bushnell, Goldberg & Robinson, 1981, Colby et al., 1996). As described above, the monkey maintains central fixation throughout the peripheral attention task and never makes an eye movement, either during the trial or during the intertrial interval. Nonetheless, the stimulus in this task is relevant for the animal's behavior because it must respond to a slight dimming of the stimulus. The visual response to the onset of the stimulus in the peripheral attention task is larger than the response to the same stimulus in the context of a simple fixation task. The increase in the number of spikes is comparable to the enhancement observed in the memory-guided saccade task (Colby et al., 1996). The interpretation is that in both cases the response enhancement reflects an increased level of attention to behaviorally-relevant stimuli. Both types of enhancement occur together in LIP neurons that have only visual responses and no saccade related activity. Attentional enhancement of visual responses is independent of motor planning. The role of area LIP in covert shifts of attention has been confirmed by reversible inactivation studies (Wardak, Olivier & Duhamel, 2004) and studies of neural activity in additional tasks that require covert attention (Balan & Gottlieb, 2006, Bisley & Goldberg, 2003, Bisley & Goldberg, 2006).

Attention modulates visual activity in many cortical areas, including dorsal stream areas V3A (Nakamura & Colby, 2000) and MT (Seidemann & Newsome, 1999, Treue & Martinez Trujillo, 1999, Treue & Maunsell, 1996); see (Treue, 2003) for review). The impact of attention on visual responses has been studied extensively in ventral stream area V4 (Armstrong & Moore, 2007, Connor, Preddie, Gallant & Van Essen, 1997, Fries, Reynolds, Rorie & Desimone, 2001, Hayden & Gallant, 2005, McAdams & Maunsell, 2000, Mitchell, Sundberg & Reynolds, 2007, Moore & Armstrong, 2003, Moran & Desimone, 1985, Reynolds & Desimone, 2003, Reynolds, Pasternak & Desimone, 2000). Attention can also modulate visual responses at still earlier stages of processing, including V1 and the lateral geniculate nucleus (Casagrande, Sary, Royal & Ruiz, 2005, McAdams & Reid, 2005, McAlonan, Cavanaugh & Wurtz, 2007, Motter, 1993). Imaging studies likewise have shown that attention modulates visual responsivity in the human brain, at both earlier and later stages of the visual hierarchy (Liu, Larsson & Carrasco, 2007, McMains et al., 2007, O'Connor, Fukui, Pinsk & Kastner, 2002). In sum, attention acts upon sensory signals at many levels to construct a selective representation of visual space.

Attention and delay period activity

Many LIP neurons are active during the delay period between the appearance of a visual stimulus and the generation of a saccadic response. Delay period activity in LIP is strongly modulated by the kind of motor response (saccade or reach) that the monkey might make to a stimulus (Cui & Andersen, 2007, Dickinson, Calton & Snyder, 2003, Lawrence & Snyder, 2006). This delay period activity before an eye movement has been variously interpreted as working memory (Gnadt & Andersen, 1988), expectation or evaluation of reward (Dorris & Glimcher, 2004, Platt & Glimcher, 1999, Sugrue, Corrado & Newsome, 2004), motor planning (Mazzoni, Bracewell, Barash & Andersen, 1996), accumulation of sensory evidence (Huk & Shadlen, 2005, Palmer, Huk & Shadlen, 2005) and decision making (Shadlen & Newsome, 2001). Many of the factors considered in these studies are consistent with an attentional interpretation as well. As the monkey plans a saccade toward a particular location, or remembers a location, or assesses the probability that a saccade to a location will yield a reward, it is also attending to that location. A parsimonious explanation of delay period activity is that it reflects attention to a spatial location (Colby & Goldberg, 1999, Maunsell, 2004). Activity in area LIP has been interpreted as a salience map, representing locations that are relevant for behavior (Goldberg, Bisley, Powell & Gottlieb, 2006, Gottlieb, 2007, Gottlieb, Kusunoki & Goldberg, 2005). The ongoing activity during the delay period can be characterized as a memory trace of an attended location. This memory trace can only be useful for guiding eye movements if it is maintained in an eye-centered representation. The question we are concerned with here is what happens to the memory trace of an attended location when the eyes move?

Remapping of Attended Visual Stimuli

We studied the fate of stimulus memory traces by developing a simple paradigm called the single step task (Duhamel, Colby & Goldberg, 1992a). The idea was to see if we could better understand, at the single neuron level, the process of adjusting the alignment between the new retinal image and the cortical representation of a previous stimulus. As shown in Figure 2, we first record the neuron's response to a visual stimulus presented in the receptive field while the animal fixates (Figure 2A). This response in the fixation task is the ordinary, expected visual response. The single step task reveals an unexpected response that indicates a dynamic updating of the internal visual representation. In the single step task (Figure 2B), the monkey fixates while a stimulus is presented briefly (50 msec) at a screen location that is well outside the neuron's receptive field. A subsequent eye movement to a new fixation point brings the receptive field onto the location of the flashed stimulus. The neuron responds following this eye movement even though the stimulus has been extinguished well before the onset of the saccade. We refer to this as a stimulus trace (or memory trace) response because the neuron is responding to a stimulus that is no longer there.

Figure 2.

Remapping in area LIP. In a simple fixation condition (A), the neuron responds to the stimulus in its receptive field. In the single step task (B), the stimulus is flashed outside the receptive field for 50msec, as indicated by the time line. The stimulus has disappeared before the eyes move to the second fixation point, FP2. The LIP neuron fires after the saccade brings the receptive field onto the previously stimulated location. In the stimulus-alone control (C), the stimulus is presented outside the receptive field and no saccade is made; this does not drive the neuron. In the saccade-alone control (D), the same eye movement is made but does not drive the neuron in the absence of the stimulus. Adapted from Duhamel et al. 1992a.

Most neurons in LIP are activated in this single step remapping task. Two control tasks demonstrate that this activity is a response to the updated memory trace of the stimulus. Neurons are not active when the stimulus is presented without any subsequent saccade (Figure 2C). Likewise, they are not active when the monkey makes the same saccade without any stimulus having appeared (Figure 2D). In the single step task, LIP neurons respond as though there were an actual stimulus in the receptive field. Activity in the single step task can only be a response to the updated memory trace of the stimulus: there is no physical stimulus in the receptive field at time one, before the saccade, and likewise no stimulus in the receptive field at time two, after the saccade. This response to the memory trace of a previous stimulus indicates that LIP neurons participate in updating an internal representation of space. We call this process “remapping” to emphasize that visual information is being shifted from the coordinates of the initial eye position to the coordinates of the next eye position. Remapping could thus contribute to maintaining the spatial alignment between the external world and its internal representation.

Remapping requires the combination of visual and motor signals

What produces this remapped response? The hypothesis is that a corollary discharge of the eye movement triggers a transfer of information. Neurons that initially represent the stimulated, attended location transfer their activity to another population of neurons, those which will represent that location after the saccade. There are two reasons to think that the triggering signal must be a corollary discharge rather than based on proprioception. First, at the single neuron level, remapping can occur before the saccade begins, before there can be any proprioceptive signal. Second, behavioral experiments have tested whether proprioception is necessary for performance on a task thought to require remapping, the double step task. These experiments show that performance on the task can be accurate even after all proprioceptive input from the eyes has been removed (Guthrie, Porter & Sparks, 1983). The conclusion is that information about the intention to make a saccade evokes a transfer of visual information.

In order to test whether this transfer of visual information depends on corollary discharge signals, we devised a task in which we could look for remapping in the absence of an eye movement. We asked whether a shift of attention alone would be as effective as an eye movement in evoking a remapped visual response. In other words, is a covert shift of attention alone sufficient to remap the internal representation of a stimulus? We tested this proposition in a variant of the peripheral attention task. As in the standard single step task, a stimulus was presented outside the receptive field of the neuron under study. Simultaneously, a new “fixation point” appeared on the screen. When the animal simply shifted its attention to the new location (in order to detect a slight dimming) without generating an eye movement there was no response from the neuron. In contrast, when the animal made a saccade to this target in the standard single step task, a remapped visual response was evoked (Colby, 1996).

The failure to trigger remapping with attention alone is critical, because it provides further evidence that the function of remapping is to maintain an accurate alignment between the visual world and its internal representation. With an attentional shift alone, nothing moves on the retina and there is no need to remap the internal representation. Indeed, it would be counterproductive to do so because such a shift would introduce a mismatch between the external world and the internal image of it. When a saccade is about to occur, however, parietal cortex can make use of the information about the intended eye movement to anticipate the retinal consequences of that saccade and update the stored representation of object locations.

Having described the basics of attentional processes and remapping in area LIP, we turn now to studies of the nature of the remapped signal and temporal and spatial aspects of remapping.

What remaps?

Saccade generation is the final trigger for remapping, yet attention seems to be a necessary prerequisite. Studies by Gottlieb and colleagues, described below, indicate that images are remapped only if they are first attended (Gottlieb, Kusunoki & Goldberg, 1998). Attention can be directed toward an object or location through either bottom-up or top-down mechanisms (see (Moore, 2006) for review). Sudden stimulus onsets automatically attract attention, and as such, are an example of bottom-up mechanisms of attention (Jonides & Yantis, 1988, Yantis & Jonides, 1984). By contrast, top-down mechanisms can be invoked when a stimulus is relevant for voluntary behavior. Gottlieb et al studied the effects of both bottom-up and top-down attention on remapping responses in area LIP. In the standard remapping tasks described above, the to-be-updated stimulus is never a target for the animal's behavior but it does always have a sudden onset. Gottlieb and colleagues first asked whether this sudden onset was important for remapping. They used a stable array task in which there were no sudden stimulus onsets. They found that LIP neurons did not remap when the receptive field was brought onto a stable, continuously visible stimulus embedded in a stable, continuously visible array. In other words, unattended stimuli were not remapped. If, however, the to-be-updated stimulus was briefly flashed within the stable array, making it salient, LIP neurons did show remapping activity. This observation indicates that attention, here brought about by bottom-up mechanisms, is a prerequisite for remapping.

Attention can also be controlled through top-down mechanisms. Gottlieb and colleagues went on to ask whether these top-down mechanisms also influenced remapping in LIP. They again used a stable array task with no sudden stimulus onsets. In this case, however, the animal was cued to make a sequence of saccades that would bring a stimulus into the receptive field of a neuron. They found that a neuron would remap the stimulus location only when it had been selected as the target and entered the receptive field as the result of a saccade. When an unattended stimulus entered the receptive field – a stimulus that was not used to guide the saccade - there was no remapping. From this set of experiments, they concluded that remapping depends on the salience of a stimulus, regardless of whether that stimulus has been made salient by bottom-up or top-down mechanisms. Their findings reinforce the notion that remapping is an attentional phenomenon. The necessity of attention indicates an economy of processing: the brain updates only what it needs to in order to maintain a stable image.

When does remapping occur?

Remapping depends on the conjunction of an attended stimulus and the intention to move the eyes. Beyond that, what have we learned about the timing of remapping? The intention to move the eyes of course precedes the actual eye movement. The transfer of visual signals that is triggered by the corollary discharge can in theory occur at any time after the decision to move the eyes. One of the most compelling benefits of remapping is that visual responses to a stimulus can occur before the normal visual latency of the neuron. These predictive signals are available well before reafferent visual signals are available. In other words, LIP neurons do not need to wait for signals to arrive from lower visual areas every time the eyes move to a new position. Instead, the remapped response takes advantage of a corollary discharge signals to yield a representation that is aligned with the anticipated new position of the eyes. Remapping is considered predictive if the response, when aligned on the saccadic eye movement, is earlier than the expected visual latency observed in a fixation task. For some predictive neurons, the location of the receptive field shifts even before the eyes begin to move. These presaccadic responses are a subset of predictive responses. Many neurons in area LIP exhibit predictive responses and thus anticipate the visual consequences of the eye movement. In a recent investigation of remapping in area LIP, we found that about two-thirds of neurons exhibited predictive responses (Heiser, Berman, Saunders & Colby, 2005). These predictive responses may be particularly useful for maintaining spatial constancy. The timing of remapping has been examined in detail in area LIP (Kusunoki & Goldberg, 2003) and in extrastriate area V3A (Nakamura & Colby, 2002). The V3A neuron illustrated in Figure 3 is strongly activated by a stimulus flashed at the new receptive field and has a presaccadic remapped response, more than 100 msec before the saccade begins (bottom panel, Time 1). This observation tells us that remapping is a rapid process and led us to ask about the temporal limits of remapping.

Figure 3.

Timing of remapping in V3A. Response of an example V3A neuron to a stimulus presented in the old receptive field (old RF, middle panel) or new RF (bottom panel) at four different timings. Time lines (top panel) show trial events. Data are aligned on stimulus onset. Inverted triangles show the mean time of the beginning of the saccade to the new fixation point. The neuron begins to respond at the new, future RF at Time 1, long before the eye movement begins. The neurons also responds simultaneously at the old, original RF at both Time 1 and Time 2, indicating a dual responsivity that encompasses the old and new RF locations. From Nakamura and Colby 2002.

What is the earliest that remapping can occur? In area LIP, we compared ordinary visual response latencies in a fixation task to response latencies in the single step task (Duhamel et al., 1992a). The standard way of measuring latency in the single step task is to align activity on the start of the eye movement. For these data, however, we also aligned single step activity on the appearance of the stimulus for direct comparison. The most rapid remapping was observed in a neuron with a normal, visual response latency of 70 msec (Figure 4). Its latency of response in the single step task was 83 msec when aligned on stimulus appearance (note that this is a response to a stimulus that is outside the receptive field). This is only 13 msec longer than the cell's normal visual latency. It means that in some cases, the whole circuit can accomplish its task within just a few msec. This observation helps us think about what the circuit might be. There is time for only a few synapses so the circuit should ideally be local and short.

Figure 4.

Timing of remapping and truncation in area LIP. The first panel (A) shows the visual response of an example LIP neuron while the monkey fixates. This neuron continues to respond even after the stimulus has disappeared. The second panel (B) shows the activity of the neuron during the single step task, when a leftward saccade brings the receptive field onto a location where a stimulus is present. Note that in this version of the single step task, the stimulus in the future receptive field remains illuminated. The single step data are aligned on stimulus appearance (B, left panel) for direct comparison with panel A, and show a rapid remapped response. The same data are also aligned on the beginning of the saccade (B, right panel), which shows that remapping in this neuron starts before the eyes begin to move. The third panel (C) shows activity of the neuron when an eye movement moves the receptive field away from the stimulated location. This eye movement causes a sharp truncation of activity, in contrast to the slow decay of activity after stimulus disappearance in panel A. From Duhamel et al. 1992.

How long do memory traces last?

An intriguing question about the temporal limits of remapping concerns how long memory traces are maintained. The longest that has been systematically tested is up to one second: the stimulus is flashed for 50 msec and the new fixation point appears 1 sec later (Duhamel et al., 1992a). In some neurons the remapped response after this delay was just as strong as when there was no delay. Given that remapping is essentially an attentional phenomenon, it may be that the memory trace remains active until some other event captures attention. This possibility remains to be tested.

Is remapping universal across directions?

Our daily experience tells us that spatial constancy is maintained no matter what size or direction of eye movement we make. We asked whether this universal insouciance was present at the single neuron level. For saccades of different sizes (10 deg vs. 20 deg) we found no difference in the strength of the remapped signal (Nakamura & Colby, 2002). For saccades in different directions the answer was more complicated (Heiser & Colby, 2006). We asked whether neurons in LIP remap when the eyes move in the four canonical directions. At the single neuron level, some cells actually do remap in all four directions. For most neurons, however, remapping is detectable only for some directions and the strength of remapping varies with saccade direction. Overall, neurons that exhibit remapping in multiple directions have more robust remapped responses. We also found that the prevalence of remapping is independent of receptive field location. Remapping was as common in neurons with peripheral receptive fields as in those with central receptive fields. Finally, we found that remapping is independent of the response properties exhibited in the memory-guided saccade task: visual and visuomovement cells in LIP are equally likely to exhibit remapping.

Despite the variability in single neuron responses, we found at the population level that all eye movement directions are equally good for remapping (Figure 5). This observation tells us that behavior should be equally good in all directions. Behavioral experiments suggest that this is true. For example, accuracy on the double step task does not differ for horizontal versus vertical initial saccades (Ram-Tsur, Caspi, Gordon & Zivotofsky, 2005). For the LIP population as a whole, we found no difference in the latency of remapped responses for different directions. We were particularly interested in whether neurons might respond differently when the stimulus trace had to be remapped within or across hemifields. We found that LIP neurons respond with similar time course and magnitude for either situation. We conclude that LIP neurons can access information from throughout the visual field and spatial updating is independent of saccade direction.

Figure 5.

Remapping in LIP is independent of saccade direction. A polar plot shows the distribution of preferred remapping directions in area LIP. Across the population, preferred directions are distributed throughout the visual field. From Heiser and Colby 2006.

Do receptive fields shift or expand?

When a stimulus trace is remapped, what happens to the receptive field of the neuron? There are multiple possibilities. One is that the receptive field may make a discrete shift: sensitivity to visual stimuli could shift entirely from the original receptive field location to the anticipated new location or “future field.” Another possibility is that the receptive field expands like a slinky to encompass both the original and future field location. We looked at these possibilities by placing stimuli at one of two locations, the current receptive field or the future receptive field (Nakamura & Colby, 2002). We found that V3A cells fall along a continuum. Some appear to have a discrete shift of the receptive field around the time of an intended saccade. They become less responsive at the old location while simultaneously becoming much more responsive at the future location of the receptive field. Other neurons appear to undergo a momentary expansion of the receptive field immediately before a saccade. These neurons responded to a stimulus at the future receptive field even when it was presented long before the saccade (Figure 3). This dual responsiveness suggests that the receptive field has expanded to encompass both locations.

Neurons in area LIP and in the frontal eye fields also exhibit this dual responsiveness (Kusunoki & Goldberg, 2003, Sommer & Wurtz, 2006). Recent behavioral experiments (Jeffries, Kusunoki, Bisley, Cohen & Goldberg, 2007) suggest that this dual responsiveness may relate to mislocalizations that occur around the time of the eye movement, such as those originally observed by Matin and Pearce (Matin & Pearce, 1965). A critical experiment by Sommer and Wurtz (2006) recently added to our understanding of how eye movements affect the structure of receptive fields in the frontal eye field. They found that even though a neuron may respond to stimuli at two locations, the receptive field does not expand indiscriminately between the two points. Specifically, frontal eye field neurons respond at the original and future receptive fields just before a saccade, but do not respond to a stimulus placed in between. This finding indicates that the receptive field expansion is not simply a continuous spread but is governed by the coordinates of the impending eye movement: as the eyes move from point A to point B, the neuron's sensitivity changes at the corresponding original and future field locations. This particular aspect of remapping has not yet been studied in other areas such as LIP, V3A, or the SC. It is not yet clear how these physiological observations relate to psychophysical findings (Melcher, 2007). The relationship between receptive field changes and perisaccadic perception remains an important question for future experiments.

Truncation

The effects of remapping are also evident when a saccade shifts a stimulus out of the receptive field. For the LIP neuron shown in Figure 4, two conditions are shown that are equivalent in terms of what happens on the retina. In panel A, the stimulus in the receptive field is turned off. The neuron continues to fire for some time. In panel C, the monkey makes an eye movement that moves the receptive field away from the stimulus. In both cases, the stimulus has been removed from the receptive field. But the effect on this parietal neuron is very different. Neural activity is abruptly truncated by the eye movement. This truncation tells us that an active process has shut down the cell's ongoing response. Truncation is the necessary obverse of remapping. It demonstrates that corollary discharge signals are used not only to increase sensitivity at the anticipated future receptive field but also to decrease sensitivity at the original receptive field. Truncation is another means by which the alignment between the internal representation of an attended stimulus and incoming visual signals is maintained.

Brain circuits that produce remapping

Where does remapping take place?

The above observations on remapping imply that receptive fields in cerebral cortex must be dynamic and not at all like simple labeled lines. Discovering the circuitry that produces remapping is essential for figuring out the neural mechanism that produces it. As a first step toward elucidating the mechanism, we asked where in the brain remapping takes place. Since the initial discovery of remapping in area LIP, many brain regions have been found to have neurons that remap stimulus traces. These include oculomotor areas, such as the frontal eye fields and the superior colliculus. Approximately 60% of visually-responsive neurons in the frontal eye fields exhibit remapping (Umeno & Goldberg, 2001), as do 30% of neurons in the intermediate layers of the superior colliculus (Walker, FitzGibbon & Goldberg, 1994).

We asked whether remapping is limited to these oculomotor and quasi-oculomotor areas, or whether it can be observed in areas thought to be primarily visual in function. We expected that if remapping actually has something to do with the perception of spatial constancy we might be able to detect it in regions that we think of as being more purely visual in function. We began in area V3A with our standard remapping task (Nakamura & Colby, 2002). We found that more than half of V3A neurons (52%) responded in the single step task. Remapped responses in area V3A are as robust as those in area LIP, and V3A neurons were not active in the control conditions. We went on to ask whether neurons at even earlier stages of the visual system hierarchy would remap stimulus traces (Figure 6A). Using the same tasks and conditions, we tested neurons in areas V3, V2, and V1 (striate cortex). We found that many neurons in area V3 respond in the single step task but that the proportion drops off rapidly in V2. In striate cortex, only 1 neuron out of 64 tested showed evidence of remapping. Two other trends are clear. First, the mean latency of the remapped response relative to saccade onset was much longer at lower levels. Second, in relation to the first trend, the proportion of neurons that remap predictively decreased markedly at lower levels of the hierarchy (Figure 6A, hatched shading). All of these findings suggest that earlier stages of the visual system are connectionally or computationally further from the source of the central signal that drives remapping. The next question for future research is whether remapping is in fact an entirely top-down process, in which the computation is carried out in LIP (or elsewhere), or whether it proceeds in parallel at multiple levels of the visual system. We conclude that remapping is not limited to neurons in oculomotor and closely allied brain regions but rather is present in dorsal stream visual areas as well.

Figure 6.

Remapping in early visual areas. In both monkey (A) and human (B), there is an increase in the proportion of remapped responses as one ascends the visual hierarchy. In monkey, the proportion of neurons with presaccadic remapping (hatched shading) also increases at higher levels of the hierarchy. From Nakamura and Colby 2002 and Merriam et al 2007.

Converging visual attention and motor signals

What are the neural circuits that generate remapping? In this section we consider the signals used to construct a dynamic, updated representation of space, and their neural underpinnings. As we have described, remapping must involve the convergence of at least two kinds of signals. The first is a visual signal, which indicates the location of a salient, attended stimulus. The second is a motor signal: a corollary discharge of the impending eye movement must be used to update the visual representation.

Where in the brain do these visual and corollary discharge signals converge? Area LIP is a likely candidate. Studies in human and monkey have demonstrated that parietal cortex is essential for behaviors that require spatial updating. One well-established behavioral measure of updating is the double step task (Hallett & Lightstone, 1976). This task allows us to measure the ability to keep track of a spatial location when the eyes move. In the double step task, the subject must make eye movements to two successively flashed targets, T1 and T2 (Figure 7). The key feature is that the second target (T2) disappears before the eyes leave the fixation point (FP). If the subject generates the sequence based only on the retinal location of the second target, the second saccade will be incorrect. For accurate performance of the sequence, the location of T2 must be updated in conjunction with the first saccade. Both humans and monkeys are able perform the double step task accurately (Baizer & Bender, 1989, Gnadt & Andersen, 1988, Goldberg & Bruce, 1990, Hallett & Lightstone, 1976, Mays & Sparks, 1980, Medendorp, Goltz & Vilis, 2006, Ray, Schall & Murthy, 2004).

Figure 7.

Impairment and recovery of updating behavior when direct cortical links are disrupted. Top panels show the double step conditions used to test updating behavior in split brain monkeys. In the across-hemifield condition (A), the second target (T2) appears in the right visual field when the eyes are at central fixation and therefore is initially represented by neurons in the left hemisphere (black T2). When the eyes reach the first target T1, the memory trace of T2 is now located in the left visual field and encoded by neurons in the right hemisphere (gray T2′). Updating in this condition must involve a transfer of visual information between cortical hemispheres. In the within-hemifield condition (B), T2 is in the right visual field both at FP and at T1; updating therefore involves communication within the same hemisphere. Behavioral testing revealed an initial impairment on across-hemifield sequences (C). The inset shows the six randomly interleaved sequences that were tested: trained central sequences (black) and novel within (green) and across (red) sequences. Eye traces (B) show double step performance in the first ten trials of the first testing session for monkeys EM (top) and CH (bottom). Dots indicate the locations of FP, T1 and T2; scale bar represents 10°. Summary data from the first session (D) show the second-saccade endpoints, which indicate a persistent impairment for monkey EM in both visual fields but a rapid improvement for monkey CH in the left but not the right field. Summary data from the final session of testing (E) show that both monkey ultimately achieved successful performance on across-hemifield sequences. Adapted from Berman et al 2005.

Neuropsychological studies have shown that accurate updating in the double step task depends on parietal cortex (Duhamel, Goldberg, Fitzgibbon, Sirigu & Grafman, 1992b, Heide, Blankenburg, Zimmermann & Kompf, 1995, Heide & Kompf, 1998). Patients with damage to parietal cortex are capable of generating slow double step sequences when both saccades are visually guided. In this case, the trajectory of the second saccade can be computed accurately using retinal information. When the targets are flashed in rapid succession, however, the second saccade is not visually-guided but must be based on the memory trace of the second target. Its accurate computation relies on spatial updating of this memory trace, to account for the intervening saccade to the first target. In this rapid version of the task, parietal patients fail to complete the double step sequence accurately. Impairment is not due to a simple motor deficit, because the patients can perform the slow, visually-guided version of the task. Rather, parietal damage is associated with a specific failure to update the memory trace of a visual stimulus on the basis of corollary discharge. Patients with damage to the frontal eye fields, by contrast, show more generalized impairments on the double step task, with deficits on the slow, visually-guided version as well as the rapid version of the double step task (Heide et al., 1995).

These results suggest that the parietal and frontal lobes may have distinct contributions to spatial updating, and that intact parietal function is critical for remapping. Consistent with these observations from lesion studies, a recent investigation has shown that double step performance is disrupted by transmagnetic stimulation over parietal cortex in normal humans (Morris, Chambers & Mattingley, 2007). The importance of parietal cortex for performance of the double step task has also been demonstrated in monkeys. Inactivation of area LIP causes decreased accuracy and increased latencies for the second eye movement in the task (Li & Andersen, 2001). It difficult to draw precise comparison between monkey inactivation studies and human patient data, as the correspondence between macaque area LIP and subregions of human parietal cortex is an area of active research (Konen & Kastner, 2008, Schluppeck, Curtis, Glimcher & Heeger, 2006, Schluppeck, Glimcher & Heeger, 2005, Sereno, Pitzalis & Martinez, 2001, Swisher, Halko, Merabet, McMains & Somers, 2007). Even if the homologue for area LIP were known in human, parietal lesions are unlikely to be confined to such restricted subregions. Nonetheless, these findings in humans and monkeys point toward parietal cortex as an essential site for the convergence of corollary discharge information and the visual representation of salient locations.

The anatomical connectivity and physiological properties of area LIP make it well suited to act as an central node in the network that constructs updated visual representations. LIP is a higher-order association area in the dorsal visual stream, with input from multiple lower-level visual areas. As described above, visual activity in LIP is strongly modulated by spatial attention. LIP therefore seems ideally suited to supply the visual information necessary for creating a remapped representation: it encodes the location of attended visual stimuli in eye-centered coordinates. The next question is – what is the origin of the corollary discharge signals that interact with these visual representations?

Corollary discharge pathways

What are the pathways for the transmission of corollary discharge signals? Recent research has identified a pathway for corollary discharge related to eye movements (see (Sommer & Wurtz, in press), for review). The pathway they investigated originates in the intermediate layers of the superior colliculus (SC). Here, presaccadic burst neurons send eye movement commands down to the brainstem nuclei that drive saccades. These neurons simultaneously send a copy of these commands, the corollary discharge, up to cortex via the thalamus. These corollary discharge signals are relayed to the frontal eye field (FEF) through the mediodorsal thalamus (Sommer & Wurtz, 2002). Disruption of this pathway interferes with updating. Inactivation of the mediodorsal thalamus causes a selective deficit in accuracy and precision of the second saccade in the double step task. This behavioral finding tells us that the pathway transmits a corollary discharge signal for accurate spatial updating. In an elegant study, Sommer and Wurtz showed that the disruption of this pathway also affects neural activity in FEF associated with remapping (Sommer & Wurtz, 2006). Specifically, inactivation of the mediodorsal thalamus reduces the strength of the remapped visual activity in FEF. These behavioral and physiological data indicate that the pathway from SC, via the mediodorsal thalamus, transmits corollary discharge signals to the cerebral cortex. Area LIP could easily receive corollary discharge signals from the FEF, as the two areas are strongly interconnected (Bullier, Schall & Morel, 1996, Chafee & Goldman-Rakic, 1998, Chafee & Goldman-Rakic, 2000, Petrides & Pandya, 1984, Schall, Morel, King & Bullier, 1995, Stanton, Bruce & Goldberg, 1995). Area LIP may also receive corollary discharge signals from the SC via the pulvinar, a pathway that has not yet been physiologically investigated (Clower, West, Lynch & Strick, 2001). There are likely to be multiple routes by which corollary discharge information can influence visual representations. This is suggested by the fact that inactivation of the mediodorsal thalamus causes only a partial deficit in double step performance (Sommer & Wurtz, 2002). As will be discussed below, there is behavioral and physiological evidence that corollary discharge signals can be transmitted through more than one pathway.

Are direct cortical links needed for remapping?

The idea that multiple pathways underlie remapping is emphasized by a set of experiments in lesioned animals. The goal of these experiments was to determine whether direct cortico-cortical links are necessary for remapping. The premise is that remapping involves communication between neurons in area LIP that encode a salient visual location before the eyes move, and those that will encode the location after the eyes move (Quaia, Optican & Goldberg, 1998). For example, we can consider a case in which a salient stimulus has appeared 10 degrees to the right of the center of gaze. The stimulus location is initially represented by LIP neurons in the left hemisphere, with retinotopic receptive fields at +10 degrees. After an eye movement of 20 degrees to the right, the same spatial location in the world will be represented by LIP neurons in the right hemisphere, with retinotopic receptive fields at -10 degrees. When the command is given to move the eyes, a corollary discharge could initiate a transfer of information from the “before” cells in the left hemisphere to the “after” cells in the right hemisphere.

This model suggests that direct communication between the “before” and “after” cells is important for updating. We hypothesized that this requires direct cortico-cortical links. In the example just described, we have an opportunity to examine the role of cortico-cortical links. In this example, the stimulus representation is updated across visual hemifields, and therefore the memory trace must be transferred from one cerebral hemisphere to the other. The cortical connections that would allow for this transfer are accessible to experimental manipulation: the forebrain commissures, comprised of the corpus callosum and anterior commissure, are the fiber pathways that link the cerebral hemispheres. We reasoned that if direct cortico-cortical links are necessary for spatial updating, then we should observe a profound deficit in across-hemifield updating following transection of these commissures.

We tested this hypothesis by assessing the behavioral and neural correlates of spatial updating in monkeys whose forebrain commissures were surgically transected (Berman, Heiser, Saunders & Colby, 2005, Heiser et al., 2005). We tested spatial behavior using two conditions double step task. In the across-hemifield updating condition, as described, the representation of the second target, T2, had to be updated from one visual hemifield to the other (Figure 7A). In the within-hemifield updating condition, T2 had to be updated from one location to another, within the same visual hemifield (Figure 7B). We tested the monkeys' performance on the across-hemifield and within-hemifield sequences after training on a “central” sequence that did not require across-hemifield updating (inset, panel C). We then simultaneously introduced the across-hemifield and within sequences. These two types of sequences were matched for saccade amplitude, counterbalanced for direction, and were equally new to the monkey, so that any differences in performance could be attributed to the type of updating required. This approach allowed a within-subject comparison of updating within and across hemispheres. We predicted that in the split-brain monkeys, spatial updating would be severely disrupted if not abolished in the across-hemifield condition, but unaffected in the within condition.

Initial testing confirmed this prediction: the split-brain monkeys' first exposures to the within and across-hemifield conditions revealed a striking and selective impairment for sequences that required updating across visual hemifields. Eye traces from the upper field show this initial double step deficit (Figure 7C). Traces from the central conditions show that the monkeys were very accurate in the execution of these trained sequences. Likewise, the monkeys were able to perform the within conditions with considerable accuracy, despite the fact that these sequences were entirely novel. In contrast, both monkeys made inaccurate movements on early across-hemifield sequences. On these trials, the trajectory of the second saccade deviated only slightly from a straight vertical saccade. Control experiments in an intact monkey demonstrated that the across-hemifield impairments were selective to the absence of the forebrain commissures. These initial behavioral data, summarized in Figure 7D, are consistent with the prediction that performance on across-hemifield sequences would be impaired when direct cortical connections were disrupted. To our surprise, however, the across-hemifield deficit was not permanent. This tells us that direct cortical links are not necessarily required for remapping.

Multiple pathways for the transfer of visual signals

We found that the split-brain monkeys' behavior recovered, very rapidly for some sequences although quite slowly for others. The essential result is that both monkeys were ultimately successful in performing double step sequences that required updating of the visual representations from one hemifield to the other (Figure 7E). Successful performance depended on experience with individual sequences. When we introduced new spatial arrangements of the target, we found that the across-hemifield impairment could be re-instated. Given continued experience, we found that performance again improved (Berman, Heiser, Dunn, Saunders & Colby, 2007, Berman et al., 2005). These behavioral data tell us that the route for updating across hemifields is not just cortex-to-cortex, via the forebrain commissures. The conclusion is that subcortical pathways must contribute to the recovered behavior.

If subcortical pathways are important for the recovery of across-hemifield performance, what is happening in cortical areas? We were particularly curious to know about the activity of LIP neurons in the split-brain monkeys. Are LIP neurons still in the loop? We asked whether LIP neurons are active when stimuli are remapped across hemifields. When we recorded in LIP, many neurons showed across-hemifield remapping. An example neuron is shown in Figure 8. In the within-hemifield condition of the single step task (panel A), this neuron exhibited a burst of activity that began even before the eye movement started. In the across-hemifield condition, we might have expected that the neuron would fail to show any updating activity, but instead the neuron fired briskly, with a burst of activity after the eye movement began (panel B). This demonstrates that LIP neurons are still in the loop; they remain part of the circuitry for across-hemifield updating in the absence of the forebrain commissures. This example neuron illustrates two additional findings about across-hemifield remapping in the split-brain monkeys. First, across-hemifield activity is diminished relative to within-hemifield activity. Second, across-hemifield activity begins later than within-hemifield activity; for this cell, presaccadic activity was observed only for the within-hemifield condition. These two differences were significant in the population of LIP neurons from the split-brain animals. By contrast, no such differences were found in the intact animal (Heiser et al., 2005). These observations indicate that across-hemifield remapping, while present in the split-brain animal, is less robust in the absence of direct interhemispheric links.

Figure 8.

Existence of across-hemifield remapping in the split-brain monkey. Activity of a single LIP neuron in the split-brain monkey exhibits remapping both within and across hemifields. Top panels (A,B) show the spatial configuration for the single step task, determined by the location of the neuron's receptive field (circle). In the within condition (A), a vertical saccade brings the receptive field onto a previously stimulated location in the same hemifield. In the across condition (B), a horizontal saccade brings the receptive field onto the stimulated location, which was in the left visual field at FP1 but in the right visual field at FP2. The neuron remapped the memory trace in both the within-hemifield (C) and across-hemifield conditions (D) of the single step task. In the within-hemifield condition, remapping begins even before the beginning of the eye movement, with an initial burst of activity, followed by a brief inhibition after the saccade and a subsequent second burst. In the across-hemifield activity, only the later burst of activity is present. The corresponding control conditions show that activity was minimal when the stimulus appeared alone (E,F) and when the saccade was generated with no stimulus present (G,H). Rasters and histograms are aligned on the beginning of the saccade. From Heiser et al 2005.

Our behavioral and physiological findings lead to two major conclusions. First, the forebrain commissures are important for rapid, robust updating of visual representations across hemifields. Second, these commissures are not strictly necessary. Alternate pathways can come online. There must be some neural reorganization, and our behavioral data suggest that this reorganization is experience-dependent: recovery does not generalize to any new double step sequence but seems to emerge as the monkey has experience with specific sequences. This aspect of our findings raises the possibility that, following the loss of the forebrain commissure pathway, the circuit for updating may need to recalibrate the relationships among neurons that represent visual locations before and after saccades. The conclusion is that the system for spatial updating is redundant and plastic, much like the system for saccade generation. Direct cortical links are the usual pathway for remapping of memory traces but not the only pathway.

Multiple pathways for the transfer of corollary discharge signals

The initial impairment for across-hemifield conditions suggests that the forebrain commissures indeed serve as the primary route for communicating visual signals between the cortical hemispheres at the time of an eye movement. Are these same commissures also the primary route for transmitting information about the impending eye movement, in order to initiate visuospatial updating? In another set of experiments, we investigated the role of cortico-cortical links in the transfer of corollary discharge signals that initiate spatial updating (Colby, Berman, Heiser & Saunders, 2005). We tested the performance of the split-brain monkeys on two new types of double step sequences. These sequences differed only in whether the corollary discharge signal stayed within one hemisphere or had to travel across hemispheres. Our expectation was that the split-brain monkeys would be impaired selectively on the sequences that required an interhemispheric transmission of the corollary discharge signal. Contrary to our expectation, we found that the split-brain monkeys could readily perform both types of double step sequences, regardless of whether the corollary discharge information traveled within or across hemispheres. The immediacy of successful performance indicates that corollary discharge signals are typically relayed to visual areas without reliance on the forebrain commissures. This finding is consistent with recent physiological studies of the corollary discharge pathway from SC to FEF. In addition to predominant ipsilateral connections, e.g., from right SC to right FEF (Sommer & Wurtz, 2002), there are also connections that cross, e.g., from right SC to left FEF (Crapse & Sommer, 2006). Similarly, anatomical studies of the pathway from SC to LIP suggest the presence of a crossed connection in addition to prominent ipsilateral connections (Clower et al., 2001). These data reinforce the idea that there are multiple pathways by which corollary discharge signals can interact with sensory and attentional signals.

Remapping in humans

Remapping in human parietal cortex

Several lines of evidence suggest that humans and monkeys use the same updating mechanisms for generating stable percepts. Behavioral studies have demonstrated that both species have similar abilities in eye movement tasks that require updating (Baizer & Bender, 1989, Dassonville, Schlag & Schlag-Rey, 1992). As discussed earlier, data from neuropsychological and inactivation studies suggest that the parietal lobe is crucial for these abilities in both species (Duhamel et al., 1992b, Heide et al., 1995, Li & Andersen, 2001, Morris et al., 2007). These findings imply that remapping also occurs in the human parietal cortex. We hypothesized that we could visualize remapping in humans using functional magnetic resonance imaging (fMRI) (Merriam, Genovese & Colby, 2003). In this section, we present functional imaging evidence that the human parietal cortex is involved in remapping.

In order to demonstrate remapping physiologically in humans, we designed an imaging experiment that followed the task parameters of the monkey experiments as closely as possible. The updating task used in the imaging experiment is shown in Figure 9 (panel A). Each trial began when the subject looked at one of two fixation points on the screen. On some trials, a stimulus appeared at the center of the screen, far from the location of gaze. The stimulus flickered at high contrast in an otherwise dark visual environment, making it a highly salient stimulus. As in the single-unit experiments, the stimulus was totally irrelevant to performance of the task. After 2 sec of stimulus presentation, the stimulus disappeared and a tone cued the subject to make a saccade to the other fixation point. There was no stimulus present at the time of the saccade so only a memory trace of the stimulus could be remapped. The second fixation point was positioned so that the eye movement to it brought the stimulus location into the opposite visual field (Figure 9A, second panel). In the example shown, gaze was initially directed to the right fixation point, and the stimulus appeared in the left visual field. Concurrent with stimulus offset, a tone instructed the subjects to move their eyes to the left fixation point. As a result of this eye movement, the stimulus location was then in the right visual field.

Figure 9.

Remapping in human parietal cortex. Panels in the top left (A) show the sequence of events in the single step task adapted for imaging. At the beginning of the trial, the subject fixates a right crosshair while a stimulus appears in the left visual field (LVF) and is on the screen for 2sec. We expected the stimulus to activate occipital and parietal cortex in the contralateral, right hemisphere (blue circle). The stimulus disappears and simultaneously a tone cues the subject to make a leftward saccade to the other crosshair. This saccade brings the location of the now-extinguished stimulus (dotted circle) into the right visual field. We expected that remapping of the stimulus trace would cause activation to shift from the right to the left hemisphere (hatched red circle). The predicted time course (B) of activation for this condition. Shaded region indicates the time that the stimulus is on and the vertical line at 2sec indicates the time of the auditory cue. Activation in the right parietal cortex was expected to follow a standard hemodynamic time course (blue trace); this represents visually-driven contralateral activation. Activation in the left hemisphere, due to the remapped stimulus trace, was expected to occur with a similar time course but shifted by 2sec because the saccade cue occurred 2sec after stimulus appearance (red trace). We also expected the remapped response to be lower in amplitude than the visually-driven response. Activation in a single subject (C) for a stimulus that is remapped from the left to the right visual field. Blue outline indicates the parietal region of interest. The stimulus elicits visually-driven activation in the contralateral (right) hemisphere occipital and parietal areas. Activation was also observed in the ipsilateral (left) parietal lobe, indicating that the visual representation was remapped in conjunction with the saccade. Time course of activation (D) evoked by the visual and remapped stimuli from parietal cortex in each hemisphere. Time courses are an average of 72 trials. The remapped response (red line) occurs later and is smaller than the visual response (blue line). BOLD-image raster plots of the responses from the same hemispheres for 72 successive trials, for the visual (E) and remapped (F) responses. On the y axis, each row represents a trial, and on the x axis, percent signal change is represented in pseudocolor plotted over time. Adapted from Merriam et al 2003.

The predictions from this experiment are straightforward. First, the stimulus should activate visually responsive cortical areas in the contralateral hemisphere (panel B, blue trace). Low level visual areas, such as V1 and V2, should become active because the stimulus has high contrast. Extrastriate and parietal areas should also become active because the sudden onset and high-frequency flicker of the stimulus make it salient. Second, following the eye movement, the memory trace of the stimulus should activate visual areas ipsilateral to the stimulus (panel B, red trace). We use the term remapped response to describe this ipsilateral activation in order to emphasize that it is not driven by direct visual stimulation. Third, this remapped response should be larger than the responses elicited by either an ipsilateral stimulus alone or an ipsiversive saccade alone. Both ipsilateral stimuli and ipsiversive saccades may induce minimal activation; receptive fields in extrastriate and parietal cortices can be large and sometimes encroach a few degrees into the ipsilateral side of space (Barash, Bracewell, Fogassi, Gnadt & Andersen, 1991b, Ben Hamed, Duhamel, Bremmer & Graf, 2001). There may also be some activation associated with the saccade because of retinal stimulation during the movement itself. These sources of activity should nonetheless be smaller than the remapped response. Fourth, remapped responses should have a characteristic shape and time course that distinguish them from visual responses. Specifically, remapped responses should occur later in time than visual responses, because the cue that triggers the eye movement occurs after the stimulus has been on the screen for two seconds. Remapped responses should also be lower in amplitude than visual responses; at the single neuron level, responses to memory traces are about half as large as responses to actual stimuli.

These four predictions were confirmed in our imaging data of human parietal cortex (Merriam et al., 2003). Our main result is illustrated in Figure 9. A left visual field stimulus strongly activated the region of interest in right parietal cortex (outlined in blue in Figure 9C). This contralateral activation in the right hemisphere reflects visual activity that is directly driven by the stimulus. We also observed activation in the ipsilateral parietal lobe (Figure 9C, left hemisphere). We interpret this ipsilateral activation as a response to the remapped trace of the stimulus. The time course of this ipsilateral activation indicates that it is indeed remapping (Figure 9D, left panel). The remapped activation in left parietal cortex (red trace) occurs later than the visual activation in this hemisphere (blue trace) and the amplitude is reduced, as predicted. We found a similar pattern of responses in right parietal cortex (Figure 9D, right panel). Control conditions demonstrated that activation observed in the remapping task cannot be accounted for by responses to the ipsilateral stimulus alone or responses to the ipsiversive saccade. These data show that remapped responses are detectable in human parietal cortex.

Remapping in human extrastriate and striate cortex

Remapping in monkeys, as described above, is not limited to parietal cortex. Remapping has been observed in the frontal eye field (Umeno & Goldberg, 1997, Umeno & Goldberg, 2001), the superior colliculus (Walker et al., 1994), and extrastriate visual cortex (Nakamura & Colby, 2002). Neurons in all these areas have spatially selective visual and perisaccadic responses and are modulated by spatial attention (Bruce & Goldberg, 1985, Ignashchenkova, Dicke, Haarmeier & Thier, 2004, Murthy, Ray, Shorter, Priddy, Schall & Thompson, 2007, Schafer & Moore, 2007, Thompson, Biscoe & Sato, 2005, Wurtz & Goldberg, 1971); for reviews see (Krauzlis, 2005, Schall, 2002, Treue, 2003). If remapping contributes to perceptual constancy, remapping should not be limited to brain regions with attentional and oculomotor functions. Rather, updated spatial information should reach visual areas that are involved in visual perception. We found remapping in early visual areas of monkeys, and so we next asked whether the same is true in humans (Merriam, Genovese & Colby, 2007). Two lines of evidence supported our hypothesis that remapping occurs in human early visual cortex. First, psychophysical studies have demonstrated that updated visual signals are required to integrate information about stimulus features across saccades (Hayhoe, Lachter & Feldman, 1991, Melcher, 2005, Melcher, 2007, Melcher & Morrone, 2003, Prime, Niemeier & Crawford, 2006). Second, several human fMRI studies have demonstrated strong top-down effects throughout occipital cortex. Multiple visual areas are activated in tasks that involve spatial attention (Brefczynski & DeYoe, 1999, Buracas & Boynton, 2007, Gandhi, Heeger & Boynton, 1999, Kastner et al., 1999, Martinez, Anllo-Vento, Sereno, Frank, Buxton, Dubowitz, Wong, Hinrichs, Heinze & Hillyard, 1999, McMains et al., 2007, McMains & Somers, 2004, Noesselt, Hillyard, Woldorff, Schoenfeld, Hagner, Jancke, Tempelmann, Hinrichs & Heinze, 2002, Ress, Backus & Heeger, 2000, Silver, Ress & Heeger, 2007, Tootell, Hadjikhani, Hall, Marrett, Vanduffel, Vaughan & Dale, 1998, Yantis, Schwarzbach, Serences, Carlson, Steinmetz, Pekar & Courtney, 2002). Many of these areas are also modulated by oculomotor signals (DeSouza, Dukelow & Vilis, 2002, Sylvester, Haynes & Rees, 2005, Sylvester & Rees, 2006, Vallines & Greenlee, 2006). These fMRI studies indicate that visual cortex has access to the corollary discharge signals needed for remapping.

We conducted a new fMRI experiment to test whether remapped visual signals are present in human extrastriate visual cortex. In the parietal experiment, the imaged area was restricted to parietal cortex in order to maximize detectability of remapped responses; the new experiment focused instead on extrastriate regions. As in the parietal experiment, the task was designed so that activation related to the memory trace of the stimulus would be remapped from one hemisphere to the other when the eyes moved. We predicted that the hemisphere that was initially ipsilateral to the stimulus would become active around the time of the eye movement, reflecting a remapped response. We found strong evidence for remapping in striate cortex and in each extrastriate visual area examined (Merriam et al., 2007). Further, we found that remapping was more robust in higher-order extrastriate areas (Figure 6B). This finding parallels our observations from macaque striate and extrastriate cortex (Figure 6A). Our results indicate that remapping is present in visual areas that are directly involved in visual perception.

The neural basis of active vision

In the dorsal stream of macaque cortex, we found that there is robust remapping in single neurons in areas V3A, V3, and V2 (Nakamura & Colby, 2002). Likewise, in humans, remapping is not limited to parietal cortex (Medendorp, Goltz, Vilis & Crawford, 2003, Merriam et al., 2003), but is also observed in early visual cortex (Merriam et al., 2007). These findings are significant because they tell us that extrastriate cortex is not simply engaged in passive elaboration of retinal signals. Rather, an active process is guiding the acquisition and maintenance of stimulus representations in extrastriate cortex (Maunsell, 1995). The function of such active processes in extrastriate cortex may be to narrow the task of stimulus representation to only those locations or stimulus features that are currently of importance for the organism (Huang, Treisman & Pashler, 2007). These observations return us to the close link between attention and updating. Psychophysical work on integration of information across saccades indicates that rather little information is maintained from one fixation to the next (Hayhoe et al., 1991, Henderson & Hollingworth, 2003, Irwin, 1991, Irwin & Andrews, 1996, Irwin & Zelinsky, 2002, Lachter & Hayhoe, 1995). Perceptual stability may result not from fusing complete images acquired in separate glances but from integrating information about only a few selected objects or potential targets. Furthermore, the spatial reference frame in which this limited information is integrated may reflect the demands of the specific task. Psychophysical results indicate that eye-centered (updated retinotopic), object-centered, and other spatial reference frames are used to integrate transsaccadic information (Karn, Moller & Hayhoe, 1997, Melcher, 2007, Melcher & Morrone, 2003, Pelz & Hayhoe, 1995). They also suggest that target selection and transsaccadic integration may take place at relatively early stages of the visual system (Hikosaka, Miyauchi & Shimojo, 1993, Shimojo, Tanaka, Hikosaka & Miyauchi, 1996).

We conclude that visual processing is dynamically modulated, even in early visual areas, by extraretinal signals such as attention and corollary discharge information. These findings emphasize the importance of studying visual responsivity in the context of behavior. The visual response properties of neurons in extrastriate and parietal cortex have commonly been studied during fixation or under anesthesia. Perception, however, normally takes place in the context of frequent eye movements. Understanding the dynamic nature of receptive fields around the time of saccades provides significant information about how we perceive the visual world in the natural environment. Specifically, predictive remapping allows spatial processing to proceed in advance of a saccade, as if the saccade had already taken place. Furthermore, it permits the maintenance of spatially accurate representations across saccades. This mechanism may be useful in constructing a stable percept of the visual world.