Corollary Discharge and Oculomotor Proprioception: Cortical Mechanisms for Spatially Accurate Vision

Abstract

A classic problem in psychology is understanding how the brain creates a stable and accurate representation of space for perception and action despite a constantly moving eye. Two mechanisms have been proposed to solve this problem: Herman von Helmholtz’s idea that the brain uses a corollary discharge of the motor command that moves the eye to adjust the visual representation, and Sir Charles Sherrington’s idea that the brain measures eye position to calculate a spatial representation. Here, we discuss the cognitive, neuropsychological, and physiological mechanisms that support each of these ideas. We propose that both are correct: A rapid corollary discharge signal remaps the visual representation before an impending saccade, computing accurate movement vectors; and an oculomotor proprioceptive signal enables the brain to construct a more accurate craniotopic representation of space that develops slowly after the saccade.

INTRODUCTION

A classic problem in psychology is understanding how the brain creates a stable and accurate representation of the location of objects in space for perception and action despite the fact that the eye is constantly moving. A retinotopic representation of the world in which visual objects are represented relative to the fovea, such as the map found in V1, by itself cannot accurately encode the spatial locations of objects because of the roving eye. Classically, two different solutions for the problem of spatial accuracy have been proposed. Herman von Helmholtz noticed that when a patient with a paralyzed muscle attempts to move an eye that cannot move, the world seems to jump in the opposite direction. From this he postulated, “our judgments as to the direction of the visual axis are simply the result of the effort of will involved in trying to alter the adjustment of the eyes” (von Helmholtz 1928). An implication of the von Helmholtz hypothesis is that the brain compensates for an intervening saccade by calculating the effect of that saccade on the visual representation of the object and can compensate for the change in eye position by adjusting the retinal representation to compensate for a planned or ongoing eye movement. In contrast, Sir Charles Sherrington (1918; Tozer & Sherrington 1910) hypothesized that proprioceptors in the extraocular muscles measure eye position, and the brain uses that signal for spatial localization. He wrote, “Muscular sense attributive to the extrinsic ocular muscles is therefore a source of certain of the space-attributes of visual perception” (Sherrington 1918, p. 338). An implication of the Sherrington hypothesis is that the brain uses a measurement of eye position to calculate the spatial location of a visual object in craniotopic coordinates, relative to the head, and therefore can ignore the derangements of retinal location evoked by eye movements. Both eye position and effort of will (today known as corollary discharge or efference copy) are measurable in the activity of neurons in the monkey parietal cortex, frontal eye field (FEF), and superior colliculus (SC). The role that each of these plays in the analysis of space for perception and action is still in dispute. In this review, we discuss evidence for both and propose that both are important, with corollary discharge playing a role in maintaining an accurate representation of space immediately before and after an eye movement by adjusting the retinal representation, and eye position playing a role well after the eye movement by establishing a craniotopic representation.

Of course, the goal of spatial processing is to represent objects in the world relative to some cyclopean, egocentric coordinate system, which simplifies the problem of running toward someone, throwing a ball to her, or reaching to touch her. The studies that we describe, however, use head-fixed subjects—either through a head post in monkey studies or through a bite bar in human studies—or do not state if the head was fixed or moving. Because of this, the only distinction we discuss is that between retinotopic and craniotopic representations. The more general question of creating an egocentric representation is beyond the scope of this review.

THE DOUBLE-STEP SACCADE TASK

The problem of spatial accuracy has been most effectively studied using saccadic eye movements to flashed targets. Although original models of saccadic programming postulated that saccades were made to a retinal location (Young & Stark 1963), Hallett & Lightstone (1976) showed that this was not necessarily true. If a subject were asked to make sequential saccades to flashed lights at a time when the second target appears and disappears before the first saccade occurs, the second saccade is accurate even though there is a dissonance between the retinal location and the saccade made to acquire it. This paradigm, the double-step saccade, has been a mainstay in the analysis of spatial perception and action. In a typical example (Figure 1), the subject must look at a fixation point (FP) while two stimuli flash, one at A and the second at B. The subject must make sequential saccades, from FP to A and then from A to B. Both saccade targets appear and disappear before the first eye movement. The first saccade is simple, because the vectors of the required eye movement ( $\stackrel{\rightharpoonup }{\text{FPA}}$) and the retinal location of the first target are identical. However, after the first saccade, the vector of the required eye movement $\stackrel{\rightharpoonup }{\text{AB}}$ is different from the original retinal vector. In order to make the proper eye movement, the brain must compute the proper saccade vector using what is essentially a vector subtraction: $\stackrel{\rightharpoonup }{\text{AB}}=\stackrel{\rightharpoonup }{\text{FPB}}-\stackrel{\rightharpoonup }{\text{FPA}}$.

Figure 1.

The double-step saccade task. (Top) The spatial locations of the fixation point (FP) and the two saccade targets (A and B). The solid arrows show the vectors of the two saccades, and the dashed arrow shows the vector from the fixation point to the retinal location of the B target. (Bottom) Appearance in time of FP, A, and B, and the horizontal (H) and vertical (V) eye movements of a normal human. Note that the targets were extinguished before any eye movements took place. This created a dissonance between the retinal coordinates of the stimulus at B and motor coordinates of the saccade to it. B was flashed in the left visual field but had to be acquired with a rightward eye movement. Figure adapted from Duhamel et al. (1992b) with permission.

Although the original studies implied that double-step saccades are as accurate as single visually guided saccades, and therefore that perisaccadic spatial perception is accurate, several experiments have suggested that this is not exactly true. Jeffries et al. (2007) trained monkeys to make double-step saccades to a second stimulus flashed for a duration of 100 ms in epochs before, during, and after the first saccade. Second saccades made to a target that flashed and disappeared within the 100-ms epoch before the first saccade were inaccurate, with an error of approximately 20% occurring when the target disappeared immediately before the saccade. The errors were always in the opposite direction of the first saccade as shown in Figure 2. The most accurate and least variable second saccades were made to the spatial location of a target that appeared 500 ms before the beginning of the first saccade. There was never any error in the first saccade, which was visually guided.

Figure 2.

Saccadic errors in the double-step task by a rhesus monkey. (a) Locations of initial fixation points (FP and FP′) at (−20°, 0°) and (+20°, 0°), respectively; first saccade target (A) at (0°, 0°); and second saccade targets (B and B′) at (−10°, 10°) and (10°, 10°), respectively. The red arrows represent the first saccade from FP′ to A and the second saccade from A to B′. The blue arrows represent the saccades in the other direction (to the left). (b) The graph plots horizontal saccadic error (eye position minus target position) against time from the disappearance of the stimulus to the beginning of the saccade. Data points are colored according to saccade direction. The red triangle is the mean error when the target disappeared 500 ms before the saccade. Error bars are standard errors of the means. The box denotes times at which the stimulus is on during the saccade. All the errors are in the opposite direction of the first saccade direction. Figure adapted from Jeffries et al. (2007) with permission.

Stimuli that flashed for 8 ms immediately before the first saccade are not perceived veridically, and saccades made to them are not accurate. Instead, visual objects that flashed for 8 ms within 25 ms of a saccade are perceived as if the visual world is compressed toward the saccade goal (Ross et al. 1997). When the second target in a double-step task appears for 2 ms, the second saccade to acquire it is distorted in the direction of the first saccade for most subjects (Dassonville et al. 1992). These results show that the accuracy and variability of the second saccade in a double-step task depend upon both the time and duration of the second target flash.

PREDICTIVE SACCADIC REMAPPING

The question of how neurons in the brain compensate for the intervening saccade in a double-step saccade task has been studied in a number of cortical and subcortical areas. When a monkey is fixating, a visual neuron responds when a stimulus appears in the neuron’s receptive field (Figure 3a). For the purposes of this review, we refer to this receptive field as the stable-fixation receptive field, studied when the eye has been stable for several hundred milliseconds and will remain so. When the monkey is fixating and planning a saccade, there are two special parts of visual space: the current spatial location of the stable-fixation receptive field, which in this review we call the current receptive field (CRF), and the part of space onto which the saccade will bring the stable-fixation receptive field, which we call the future receptive field (FRF) (Figure 3b). Mays & Sparks (1980a) described neurons in the SC that discharged appropriately during the double-step task, when the second saccade target lay in the FRF. Because the cells gave a tonic response from the appearance of the stimulus to the saccade in a simple saccade task and did not have a presaccadic burst, the authors described these cells as visual and not movement-related and called these cells quasi-visual, because they seemingly changed their receptive fields during the saccade. However, these cells were likely to have been the tonic visuomovement cells described by Walker et al. (1995), and their activity in the double-step task was likely a motor response. Goldberg & Bruce (1990) found that neurons in the FEF with visual responses but without any movement-related responses discharge even before the monkey made a saccade that would bring the target of the impending second saccade into the receptive field of the neuron. The authors postulated that this phenomenon implied a presaccadic shift of the receptive field that will excite the neuron before the eye moves. A similar effect can be seen in visual neurons in the lateral intraparietal area (LIP) (Barash et al. 1991, Goldberg et al. 1990), an area that provides the priority map to whose peak the oculomotor system can drive saccades (Bisley & Goldberg 2010).

Figure 3.

Predictive remapping in the lateral intraparietal area. (a) Tonic visual response to the appearance of stimulus (stim) in the current receptive field (CRF) while the monkey fixates. Figure shows the placement of the fixation point (FP), the current spatial location of the stable-fixation receptive field (CRF), and the stimulus (stim). (Bottom, a–c) Horizontal (H) and vertical (V) eye position traces, and time of stimulus presentation (dark bar). In the raster diagram, each dot is a spike. Successive trials are synchronized on the appearance of the stimulus. The histogram sums the spikes. The bar drawn at the left of each raster is 100 spikes/s. The two vertical lines signify the beginning and end of the stimulus presentation. The tonic activity of the cell well outlasts the stimulus. (b) Remapping response. The saccade brings the stable-fixation receptive field onto the spatial location of the stimulus, which we call the future receptive field (FRF). Figure shows the spatial location of the stable-fixation receptive field when the monkey fixates FP1 (CRF), and the FRF (when the monkey fixates FP2, having made the saccade described by the arrow). The raster diagram is synchronized on the beginning of the saccade. The cell discharges before the saccade. (c) Saccade truncates the tonic response. The monkey makes a saccade from FP1 to FP2 (arrow). The raster is synchronized on the beginning of the saccade. The cell stops responding after the saccade even though the stimulus remains in the CRF. Figure adapted from Duhamel et al. (1992a) with permission.

Duhamel et al. (1992a) showed that the perisaccadic shift of the receptive field, as first described in the double-step task, does not require that the monkey actually plan a second saccade. Approximately half [46% in the Duhamel et al. (1992a) sample] of neurons in LIP responded to stimuli in the FRF before the beginning of an impending saccade that would bring the spatial location of the stimulus into their retinal receptive fields, even though the monkey was not planning a saccade to the stimulus in the FRF, and would be punished by not getting a reward if it made a saccade to the stimulus in the FRF (Figure 3b). The neurons did not respond to a stimulus in the FRF if the monkey were not planning a saccade, nor did they respond if the monkey made the same saccade when there was no stimulus in the FRF.

This perisaccadic receptive field shift can be interpreted as a remapping of the retinal coordinate system, from one centered on the current fixation point (FP1 in the current example) to one centered on the spatial location that the center of gaze will occupy after the planned saccade (FP2). This remapping enables the cell to anticipate the visual response, which will be caused by the saccade as it brings the retinal receptive field onto the stimulus in the FRF, at which time the retinal coordinate system will return to the actual center of gaze. The reafferent response to the stimulus when it is actually brought into the stable-fixation receptive field by the saccade, as the FRF becomes the new CRF, is much stronger than the presaccadic predictive remapping response (compare pre- and postsaccadic responses in Figure 3b). The response to the stimulus in the FRF precedes the saccade; this ensures that there will not be a gap in the spatially accurate representation of the stimulus. This remapping explains how humans and monkeys can perform the double-step task (Figure 4). Consider the neuron whose stable-fixation receptive field lies at the spatial location CRF when the monkey is looking at FP. There is nothing in its receptive field. However, when the monkey plans the first saccade from FP to A, the remapping mechanism causes the neuron to respond to the B target, which occupies the spatial location FRF. This creates a peak on the LIP priority map that can drive the second saccade from A to B even though the B stimulus never appeared in the neuron’s stable-fixation receptive field.

Figure 4.

How remapping solves the double-step task. In the double-step task, the subject looks at a fixation point (FP), and stimuli flash sequentially at spatial locations A and B and disappear before the subject makes the first saccade FP to A. When the B stimulus appears, it is not in the stable-fixation receptive field of the A to B neuron, which occupies the spatial location CRF (current receptive field). Instead, it lies in the spatial location that the stable-fixation receptive field will occupy after the saccade from FP to A, the FRF (future receptive field). When the monkey plans the saccade from FP to A, the remapping mechanism causes the neuron to respond to the B target. This creates a peak on the LIP priority map that can drive the saccade from A to B even though the B stimulus never appeared in the neuron’s stable-fixation receptive field.

The intensity of the remapped response depends on whether the stimulus is the object of attention. The original remapping studies were all done with flashed stimuli. Because the abrupt onset of a task-irrelevant visual stimulus automatically evokes attention (Bisley & Goldberg 2003), the stimuli that evoked the remapped response were most likely the objects of attention. Unattended stimuli show very weak remapping responses in LIP (Gottlieb et al. 1998) and the FEF (Joiner et al. 2011).

Predictive remapping is not limited to cases in which monkeys make saccades to spots of light. Phillips & Segraves (2010) showed that when monkeys explore a natural scene, FEF visual and visuomovement neurons discharge before the monkey begins a fixation that will be terminated by a saccade to an object in the neuron’s receptive field.

At the same time that the neuron begins to respond to the stimulus in the FRF, the neuron becomes less responsive to a new stimulus flashed in the CRF (Kusunoki & Goldberg 2003). The decrement of activity does not occur with all saccades. LIP receptive fields are large (Ben Hamed et al. 2001), and when a monkey is planning a saccade that moves a stimulus from one part of the stable-fixation receptive field to another, there is no decrement in the response to the stimulus flashed close to the beginning of the saccade (Kusunoki & Goldberg 2003). Neurons that exhibit predictive remapping can also respond transiently to stimuli flashed at intermediate locations between the CRF and the FRF, across which the saccade sweeps the stable-fixation receptive field (Wang et al. 2016) although not to stimuli elsewhere in the visual field. In the immediate perisaccadic area, the neuron can respond to a stimulus in the CRF, the FRF, and the intermediate location. This creates an effective increase in the receptive field size, with an attendant imprecision in stimulus localization. The increase in receptive field size may be responsible for some of the perceptual and oculomotor inaccuracy around the time of the saccade. The decrement of activity does not occur with all saccades.

This perisaccadic expansion of the receptive field is in conflict with the recent claim by Zirnsak et al. (2014) that the receptive fields of FEF neurons are compressed to a location closer to the fovea than the FRF. The authors used a 25-ms flash appearing an average of 69 ms (SD = 35) before the saccade and averaged activity occurring from 50 to 350 ms after probe onset. By using averaging across such a large time interval and integrating activity that occurred before and after the saccade, the authors could not observe the transient receptive field expansion. Similarly, because Duhamel et al. (1992a) and Wang et al. (2016) did not exhaustively examine the spatial tuning of remapping, they could not exclude some component of compression of responses toward the saccade goal.

Predictive saccadic remapping has been found in a number of areas: V4 (Neupane et al. 2016), V3a and V2 (Nakamura & Colby 2002), the FEF (Umeno & Goldberg 1997), the SC (Walker et al. 1995), and the parietal reach area (Batista et al. 1999). Remapping also occurs in the parietal cortex of humans. Merriam et al. (2003) showed that when a stimulus appears and disappears in one hemifield while a human subject fixates, there is a blood-oxygen-level dependent signal only in the contralateral hemisphere. However, if the subject makes a saccade in the direction of the stimulus that brings its spatial location into the other hemifield, a response appears in the ipsilateral hemisphere. The ipsilateral response is later than the contralateral response and is evidence for remapping in humans.

Under certain circumstances, neurons in V4 also show compression of their responses to the saccadic goal (Tolias et al. 2001). However, this does not occur for all directions of saccades, and it never occurs when the saccade goes away from the receptive field (Neupane et al. 2016).

COROLLARY DISCHARGE

Because remapping can occur before the eye actually moves, it is clear that the remapping phenomenon occurs as a result of information about the eye movement being fed back to the sensory system. Feeding back the movement command not only enables spatially accurate saccades in the double-step task but also enables humans to distinguish sensations evoked by their own movements (reafferent signals) from sensations evoked by movement in the real world (Blakemore et al. 1998). An example of this is the fact that you can distinguish the retinal motion evoked by an eye movement when you make a saccade across a stable object as opposed to the same retinal movement evoked when an object moves in the real world while you fixate. In a more mundane example, the reason you cannot tickle yourself is that corollary discharge informs your somatosensory system that you are going to be touched, eliminating the surprise of the sensation, which would otherwise evoke the feeling of being tickled.

The first experimental evidence for this feedback from the motor system to the visual system came from Sperry (1950), who noticed that fish with one occluded eye and one inverted eye exhibited forced circling, as if the effort of swimming had an exaggerated effect on the fish’s visual perception. Lesions in the optic lobe obliterated the effect, and the fish swam straight again. Sperry postulated that an upper motor signal fed back to the visual system, and he called this signal the corollary discharge. From similar experiments, von Holst & Mittelstaedt (1950) postulated that the exact motor command going to the muscles fed back to the sensory system and called the signal an efference copy. Although the two terms are often used interchangeably, a distinction can be made that efference copy refers to a copy of the signal sent to the muscles, and corollary discharge refers to a copy of the motor plan as elaborated by a higher center. We use the term corollary discharge in this review.

The first physiological demonstration of corollary discharge affecting the activity of visual neurons was the observation that the baseline activity of visual neurons in the superficial layers of the monkey SC was inhibited by eye movements of any direction when the monkey was in total darkness (Goldberg & Wurtz 1972). Robinson & Wurtz (1976) showed that neurons in the monkey SC could distinguish between self-induced retinal motion and real-world induced retinal motion, although neurons in V1 could not do so (Wurtz 1969). Tonic visual neurons in LIP continue responding to the memory of a flashed stimulus well after the stimulus has disappeared (Figure 3a) but stop firing when a saccade brings the spatial location of the vanished stimulus out of the receptive field (Figure 3c). Remapping and saccade-induced truncation of a tonic visual response provide further evidence for the influence of motor plans on visual activity.

Sommer & Wurtz (2006) described the first network for corollary discharge, from the motor area that elaborated the command to the visual area whose activity was remapped by the corollary discharge. They discovered that the medial dorsal nucleus of the thalamus (MD), which receives projections from the SC and projects to the FEF, had presaccadic activity resembling that of the SC. When they inactivated MD, they did not produce an effect on saccadic accuracy or velocity. They did, however, eliminate remapping in FEF neurons, thus proving that the signal from MD to the FEF is a corollary discharge that effects the remapping.

PSYCHOPHYSICAL EVIDENCE FOR REMAPPING

If predictive remapping were behaviorally significant, one would expect that there would be psychophysical and neuropsychological evidence for it. Melcher (2007) used the tilt aftereffect to show a psychophysical correlate of remapping. After a 3-s adaptation to a strongly tilted adapted stimulus, a test stimulus that is presented at that location tends to be seen as tilted in the opposite direction. When both the adapting stimulus and the test stimulus were presented at the fovea, subjects showed a strong tilt aftereffect. When the adapting stimulus appeared at the fovea, the subject continued to fixate, and when the test stimulus appeared 10° in the periphery, there was no tilt aftereffect. If the subject were planning a 10° horizontal saccade and the test stimulus were shown at the saccade goal, the adapting stimulus evoked a tilt aftereffect. However, other tilt-aftereffect studies have not demonstrated a clear remapping to the future receptive field. Mathôt & Theeuwes (2013) did a similar experiment and showed only a retinotopic tilt aftereffect and not a craniotopic one. Zirnsak et al. (2011) showed that the tilt aftereffect was compressed to the saccade goal rather than to a future receptive field. Rolfs et al. (2011) failed to duplicate Melcher’s results, but they did show that when a subject performs a double-step saccade, attention (as measured by perceptual threshold) is remapped to both of the saccade goals before the first saccade. Despite the various contrasting results using the tilt aftereffect to assess spatial processing, it is clear that the majority of psychophysical studies show that around the time of the saccade, some changes in retinal responses can be described only by a perisaccadic shift of retinal excitability.

NEUROPSYCHOLOGICAL EVIDENCE FOR THE BEHAVIORAL SIGNIFICANCE OF REMAPPING

The best evidence for the role of remapping in the generation of spatially accurate behavior comes from the effects of lesions in humans and transient inactivation in monkeys. When patients with parietal lesions attempt a double-step task, their greatest deficit is with the second saccade even though they can perform simple, sequential visually guided saccades to the same targets quite well (Duhamel et al. 1992b, Heide et al. 1995, Rath-Wilson & Guitton 2015). Simple visually guided saccades into the contralateral field have decreased velocity and increased latency, and they are less accurate than saccades into the ipsilateral field (subpanel i of Figure 5b). However, in the double-step task, when the same targets appear and disappear before the beginning of the first saccade and the patients are given a brief time to make the saccades, they cannot compensate for the first saccade. Once they have made a saccade into the contralateral visual field, they cannot make a second saccade in the ipsilesional (unaffected) direction to foveate the spatial location of the second target that appeared in the ipsilesional (unaffected) field (subpanel ii of Figure 5b). It is unlikely the patients had difficulty perceiving the second target, because on some trials they neglected the first target and made a saccade directly toward the second target in the unaffected field.

Figure 5.

Double-step saccades of a patient with a right frontoparietal lesion. (a) Spatial locations of the fixation point (FP) and targets A and B used in the task. (b, i) Performance when the targets appear sequentially, remaining for 500 ms each. Because the B target appears after the patient has acquired the target at A, both saccades are simply visually guided saccades. (b, ii) Performance when both the targets flash before the first eye movement. The second saccade cannot rely on the retinal position of the target alone but must compensate for the intervening saccade. Figure adapted from Duhamel et al. (1992a, b) with permission.

Sommer & Wurtz (2002) used muscimol (a GABA_A agonist) to inactivate the medial dorsal nucleus of the thalamus temporarily. As described above, this inactivation eliminated remapping in FEF neurons. It did not affect the performance of simple visually guided saccades, but it evoked errors in the second saccade of the double-step task, thus providing further strong evidence that remapping contributes to the processes by which the brain accurately calculates saccade target locations.

REMAPPING OF MEMORY

Nearly all of the visual neurons [96% in the Duhamel et al. (1992a) sample] in LIP show a postsaccadic memory response (Figure 6). The neuron shown in Figure 6 has a tonic response when the monkey is fixating FP1 and the stimulus appears in the CRF (Figure 6a). When the monkey makes a saccade that moves the stable-fixation receptive field onto the stimulus in the FRF, the cell responds after the saccade, showing that this cell does not show presaccadic remapping. It does show a reafferent response that is time-locked to the saccade (Figure 6b). However, when the stimulus appears and disappears in the FRF before the saccade, the cell responds briefly, again after the saccade (Figure 6c) as if the memory trace of the stimulus were remapped. Because the stimulus has disappeared by the time saccade occurs, there is no reafferent response, merely the memory response.

Figure 6.

Remapping of memory in the lateral intraparietal area. Images at top of panels a–c show current receptive field (CRF) and future receptive field (FRF) locations, positions of the original fixation point (FP1) and the saccade target (FP2), and position of the visual stimulus that will be remapped. Horizontal (H) and vertical (V) eye position traces and stimulus (stim) appearance (black bar) plotted against time. Raster diagrams and poststimulus histograms are shown beneath eye position traces. Measured bar left of poststimulus histogram is 100 spikes/s. (a) Visual response to the stimulus when it appears in the CRF during fixation. (b) Reafferent response when the saccade brings the stimulus into the receptive field. The rectangle in the dashed circle shows stimulus location, and the arrow symbolizes the saccade. There is no remapping. (c) Remapping of the memory of the stimulus. The stimulus appears and disappears in the FRF before the saccade. The cell responds after the saccade. Figure adapted from Duhamel et al. (1992a) with permission.

In the FEF, the memory response can occur even when the stimulus appeared in the FRF on a previous trial (Umeno & Goldberg 2001). The neuron does not respond in a block of trials when the monkey makes a saccade from FP1 to FP2 and no stimulus appears in the FRF (Figure 7a). During a block of trials, when the stimulus appears in the FRF on every trial, the neuron responds with a predictive remapping response and a reafferent response (Figure 7b). When the two trial types are interleaved, the neuron gives the expected response in the continuous stimulus trials (Figure 7c), but in the interleaved saccade control trials (Figure 7d), the neuron responds as if the stimulus were still present, except that it no longer shows a predictive remapping response; it shows only the postsaccadic memory response. The memory takes up to 40 trials to disappear entirely. These data show that the FEF has access to a representation of visual space in craniotopic coordinates, which is turned into a vector representation when a saccade moves the spatial location of the vanished object into a neuron’s receptive field. The generation of this craniotopic memory must require some knowledge of eye position.

Figure 7.

Intertrial memory in the frontal eye field. The relevant boundaries of the receptive fields are shown but not the entire fields. On the left are rasters, poststimulus histograms, horizontal eye position (H), vertical eye position (V), and time of presentation of FP1, FP2, and the receptive field stimulus (stim) (when appropriate). On the right are the locations of the current receptive field (CRF), the future receptive field (FRF), the first and second fixation points (FP1 and FP2, respectively), and the receptive field stimulus when present. The arrows show the saccades from FP1 to FP2. (a) Blocked saccade control. The monkey makes saccades from FP1 to FP2, without any stimulus appearing in the CRF or FRF. (b) Blocked continuous stimulus trials. The monkey makes saccades with a stimulus in the FRF, and the neuron exhibits a predictive remapping response and a reafferent response. (c) Interleaved continuous stimulus task. Trials when the stimulus appears in the FRF are randomly interleaved (50% probability) with trials in which nothing appears in the FRF. In these trials, the stimulus is present and the cell shows a predictive and a reafferent response. (d) Interleaved saccade control. In these trials, the stimulus does not appear in the FRF, but the cell responds after the saccade. There is no predictive remapping in these trials. Figure adapted from Umeno & Goldberg (2001) with permission.

PSYCHOPHYSICAL EVIDENCE FOR A CRANIOTOPIC REPRESENTATION

The experiments described above, first showing that the oculomotor system could not rely on a retinal representation alone (Hallett & Lightstone 1976, Mays & Sparks 1980b), led the studies’ authors to assume that there was, in fact, a static representation of visual space in craniotopic coordinates. Remapping solves the problem without requiring that the brain maintain a static representation of the visual world in craniotopic coordinates.

However, Karn et al. (1997) showed remapping by itself could not explain certain aspects of oculomotor performance. They asked human subjects to fixate for one second, then presented them with a memory target that disappeared after 800 ms. Subsequently, the task required them to make two or five untargeted or visually guided saccades and, finally, to make a saccade to the spatial location of the remembered memory target location. The length of the memory period was the same for both the two- and five-saccade tasks. The subjects’ accuracy for the memory-guided saccade was slightly better when the subjects had to make only two rather than five visually guided saccades. The authors argued that if the representation of the remembered target depended upon remapping, the errors should have been additive and the memory-guided saccade after two saccades should have been far less inaccurate than the memory-guided saccades after five saccades. Instead, Karn et al. argued that under the conditions of their experiment, an attended memory target was represented craniotopically. Such a representation must require a measurement of eye position.

Poletti et al. (2013) also studied the effect of the number of intervening spontaneous and unconstrained saccades on spatial localization. In each trial, subjects searched for hidden objects in darkness (except for the glow of a dark monitor) while their saccades were counted. After a few saccades, a red circle flashed for 50 ms at the fovea. Then, the subjects had to search in darkness for a second hidden red circle that would flash briefly for 50 ms at their fovea when they found it. After they found the second circle, they had to make a saccade back to the spatial location of the first circle. Although the saccades to the remembered point were accurate, their variance increased as a function of the number of saccades. If the brain used only a corollary discharge to calculate target position, the dispersion of the saccades should increase linearly (Figure 8a). If the brain used only a craniotopic representation, dispersion should not increase with the number of saccades. Instead, the increased variance was best modeled as a combination of corollary discharge and proprioceptive signals, with corollary discharge dominating the spatial calculation for the first few saccades and the contribution of proprioception increasing over time. For the first two saccades, the increase of dispersion lies on the line predicted by the corollary discharge hypothesis, then diverges, with a smaller increase of dispersion per added saccade. The contribution of corollary discharge to the model decreases, and that of proprioception increases with the number of saccades (Figure 8b).

Figure 8.

Changing contribution of remapping and craniotopic representation to saccades to remembered targets.

(a) Decrease in variance with number of saccades. Filled circles represent actual data averaged across subjects. The blue dotted line represents estimate if only corollary discharge were used to calculate the spatial location of the remembered target. The green dotted line represents estimate if only proprioception were used to calculate the spatial location of the remembered target. The black line represents estimate for an optimal integration between a proprioceptive mechanism and corollary discharge. Asterisks mark significant deviations p < 0.001, two-tailed paired t tests, from the predictions of a purely efferent estimate, as given by the linear regression of the measurements obtained with the first three saccades (blue line).

(b) Optimal weighting of afferent and efferent estimates. As the number of saccades increases, proprioception is weighted more strongly and eventually becomes the predominant source of information. Error bars and shaded regions in panels a and b represent standard errors of the means. Figure adapted from Poletti et al. (2013) with permission.

E. Zimmermann et al. (2013) used the tilt aftereffect to study the temporal development of a craniotopic representation. After a 3-s adaptation period, they presented the saccade target 0 ms, 500 ms, or 1,000 ms before the time of the saccade signal. When the saccade target appeared with no delay the tilt aftereffect was retinotopic. However, the amount of craniotopic remapping of the tilt aftereffect increased with the length of time the subjects viewed the saccade target before they actually generated the saccade.

Suggestive evidence for a craniotopic representation comes from Rath-Wilson & Guitton (2015). They studied the performance of patients with lesions of the right parietal lobe in the double-step task. When the targets flash quickly as in the task used by Heide et al. (1995) and Duhamel et al. (1992b) and the patients made the second saccade after a short-latency single saccade into the damaged field, the results duplicated those of Heide et al. (1995) and Duhamel et al. (1992b). However, when the second targets were presented for a duration between 800 and 1,200 ms, as opposed to the 80 ms of the Duhamel at al. (1992b) and Heide et al. (1995) studies, or the patients acquired the first target in the rapid double-step task after multiple saccades, the patients’ performance improved. The authors interpreted this to mean that these results refuted the corollary discharge hypothesis. A more parsimonious interpretation, because they in fact confirmed the corollary discharge hypothesis for short latency single first saccade double-step trials, is that the long target display and long-latency saccades gave the patients enough time to access the sort of remembered craniotopic target representation invoked by Karn et al. (1997), E. Zimmermann et al. (2013), and Poletti et al. (2013). The result is consistent with the slow buildup of a craniotopic representation.

NEURAL EVIDENCE FOR CRANIOTOPIC REPRESENTATIONS

Unlike retinotopic representations, explicit craniotopic representations are quite unusual in the cerebral cortex. The great bulk of the visual representation is retinotopic, although gaze-independent visual neurons have been demonstrated in the ventral intraparietal area (Duhamel et al. 1997) in the depth of the intraparietal sulcus, an area thought to be important in the representation of perioral space (Colby & Goldberg 1999); the face area of premotor cortex, also important in the control of movements in perioral space (Fogassi et al. 1992); and V6 in the medial wall of the parietal cortex (Galletti et al. 1995). There are, however, a number of implicit representations of space that have been proposed. A neural population code model of LIP proposed by Graf & Andersen (2014) equates presaccadic target position with planned future eye position. However, under conditions of saccadic adaptation, visual and presaccadic activity in LIP are tuned for the stimulus location and not for the saccade amplitude (Steenrod et al. 2013), so presaccadic target position cannot be used rigorously to calculate postsaccadic eye position.

The most thoroughly studied of craniotopic representations is the phenomenon of gain fields. Andersen & Mountcastle (1983) discovered that the visual responses of neurons in parietal cortex were modulated by the position of the eye in the orbit (Figure 9), and from this, Andersen et al. (1985) posited that this modulation of eye position, the gain field, could be used to calculate craniotopic target position. Gain fields, an accepted mechanism by which two different and independent sensory signals are combined to compute an emergent signal (Dayan & Abbott 2001), provide a computationally tractable phenomenon, and a number of different computational techniques have used them to calculate target position in craniotopic coordinates (Pouget & Sejnowski 2001, Salinas & Abbott 1997, Zipser & Andersen 1988).

Figure 9.

Parietal gain field. (a) Area 7a receptive field mapped relative to coordinates of visual angle determined with the animal fixating straight ahead. (b) Method of determining spatial gain fields of 7a neurons in which the head is fixed and fixations (f) are at different locations on the screen. The stable-fixation receptive field (rf) moves with changing eye position. (c) Spatial gain field of single cell where eye position modulates the firing rate of the stable-fixation receptive field. Figure adapted from Andersen et al. (1985) with permission.

THE PERIPHERAL SOURCE OF THE EYE POSITION SIGNAL

There are two possibilities for the source of the eye position signal that modulates the visual response to create the gain fields. Andersen & Mountcastle (1983) postulated that it is a corollary discharge signal. There is an eye position signal on oculomotor neurons that leads the eye position (Robinson 1970), which arises from the neural integrator in medial vestibular and prepositus hypoglossi nuclei (Cannon & Robinson 1987). However, there is no evidence that this signal reaches the cerebral cortex. A cortical corollary discharge would be expected to create an explicit predictive eye position signal, but this has never been described.

A second source of a cortical eye position signal is an oculomotor proprioceptive signal. Two kinds of receptors have been implicated as possible ocular proprioceptive receptors. The first is the muscle spindle, which measures muscle length in the skeletomotor system, and is present in the eye muscles of a number of different species, including humans, sheep, giraffes, and chimpanzees but not in macaques (Cooper & Daniel 1949). Extraocular muscles in humans have spindle in amounts similar to that of the lumbricals, muscles in the hands involved in fine finger movements. The second is the palisade ending, or the myotendinous junction. Although for many years, palisade endings were considered proprioceptive (Donaldson 2000), there is some dispute about this. The nuclei of the nerves originating in the palisade endings lie in the oculomotor nucleus and travel in the oculomotor nerve (Lienbacher et al. 2011). They originate in a part of the nucleus known to innervate nontwitch, multiply innervated muscle fibers (Lienbacher & Horn 2012). They have cholinergic synaptic terminals, typical of motor neurons (L. Zimmermann et al. 2013), but some of their fibers go into the tendon where they are unlikely to affect any muscles but could serve as proprioceptive receptors.

THE CORTICAL REPRESENTATION OF EYE POSITION

Regardless of the source of the signal, the monkey clearly has a significant sensory representation of eye position in somatosensory cortex, in area 3a where the muscle spindles are represented (Wang et al. 2007) (Figure 10). The neurons are tuned for eye position eccentricity in a linear, monotonically increasing manner from around the center of the orbit and in a Gaussian manner for direction. All directions of eye position within the contralateral orbit are represented. When a monkey makes a saccade from a low gain field to a high gain field eye position, the neurons give a phasic and then a tonic response. When the monkey makes a saccade from a high to a low gain field position, there is an inhibition of discharge followed by a resumption of tonic activity. The signal arises from the contralateral eye. A transient retrobulbar block, which eliminates eye movements and presumably sensory signals from the deep orbital structures, eliminates the activity of cortical proprioceptive neurons. During smooth pursuit and the vestibuloocular reflex, where there are no saccades to evoke a phasic response, the neurons lag the actual eye position by an average of 60 ms (Xu et al. 2012). The oculomotor proprioceptive signal is unaffected by vision: The eye position effects of smooth pursuit (in which the retinal image changes with the eye movement) and the vestibuloocular reflex (in which the retinal image is stable) are identical. Using fMRI, Balslev & Miall (2008) found a bilateral representation of eye position in human somatosensory and motor cortex as well.

Figure 10.

Activity of a tonic eye position neuron in monkey somatosensory cortex. Nine raster diagrams, one at the center of the orbit and eight others positioned radially 15° from the center. The position of the raster in the diagram is related to the position of the eye in the orbit. Each tick is an action potential, and each line is a trial. Lines are synchronized on the end of the foveating saccade. Because the trial began with the appearance of the fixation point, the eye position before the saccade was uncontrolled. The histograms beneath each raster average—without smoothing—the activities of the raster above, with a bin width of 25 ms. Eye positions for each trial are superimposed beneath each raster [horizontal (H), blue; vertical (V), red]. Figure adapted from Wang et al. (2007) with permission.

Parietal neurons with gain fields behave as if they receive their input from proprioception rather than from a corollary discharge. Xu et al. (2012) studied the effect of a prior saccade on the visual responses of parietal neurons with gain fields. In the experiment shown in Figure 11, the monkey made a saccade from a high gain field orbital position to a low gain field orbital position, the conditioning saccade. A target for a memory-guided delayed saccade then flashed for 50 ms at times from 50 to 650 ms after the conditioning saccade. Xu et al. found two different effects of a prior saccade on gain fields. For stimuli flashed at 50 and 150 ms after the saccade, approximately two-thirds of the neurons showed eye position modulation consistent with the presaccadic eye position. Only at 250 ms after the saccade did the neurons give a response consistent with the postsaccadic eye position. The remainder of the cells gave eye position responses to stimuli flashed 50 and 150 ms after the saccade that were inconsistent with the steady-state gain fields and could not be predicted from the steady-state gain field responses. Both consistent and inconsistent cells gave the expected gain field response when the saccade target flashed 250 ms after the conditioning saccade. Despite the inaccuracy of the gain fields, when double-step saccade targets were flashed 50 ms after the prior saccade, the second saccades were quite accurate. However, because postsaccadic gain fields become accurate 250 ms after a saccade, it is impossible that they can be used to determine the spatially accurate trajectory of saccades in the rapid double-step task. The corollary discharge must be used to enable accurate performance of this task. Xu et al. postulated that gain fields may be used for calibration rather than for a motor targeting function. Alternatively, if spatial localization consists both of an early corollary discharge mechanism and of a late proprioceptive mechanism, it is possible that the gain fields can contribute to this later, more accurate mechanism, but more work will have to be done to establish this and to determine the role of gain fields in spatial behavior.

Figure 11.

Postsaccadic inaccuracy of gain fields in the lateral intraparietal area. (a) The two-saccade task. Dashed circle represents the receptive field of the neuron under study, and arrows represent directions of saccades. Single-cell responses to probes flashed at different times after a conditioning saccade in the (b) high-to-low and (c) low-to-high directions. Activity immediately following the conditioning saccade consistently indicates the presaccadic eye position. Activity is aligned on the end of the first saccade (dotted line), averaged across trials, and convolved with a 20-ms Gaussian filter. Colors indicate different timings of the probe. Rasters at top in panels b and c show spikes in the 50-ms (red) and 250-ms (blue) probe delay conditions. The solid curve (gray) shows the steady-state visual response at the postsaccadic orbital position during a memory-guided saccade task; for this curve, zero on the abscissa is the time of appearance of the saccade target. Figure adapted from Xu et al. (2012) with permission.

FUNCTIONAL EVIDENCE FOR THE ROLE OF OCULOMOTOR PROPRIOCEPTION IN PERCEPTION

Skavenski (1972) showed that normal subjects could perceive passive changes of eye position. He attached a contact lens to rotate a nonseeing eye in total darkness and asked the subject whether the eye had moved and if so, in which direction. The subjects reported eye movement regardless of whether the lids and conjunctiva were anesthetized with lidocaine. In a second experiment, he showed that one eye could track the position of the passively moved eye, showing that a proprioceptive representation of eye position could be used to control eye positioning.

Steinbach & Smith (1981) studied patients who had strabismus surgery involving cutting the tendons of extraocular muscles, which would have destroyed or deafferented the palisade endings in the tendons. The patients had small but significant errors in open-loop pointing when the operation was the patient’s first, but the errors significantly increased after multiple surgeries, which suggests that the repeated damage to myotendinous structures, including the palisade endings, leads to long-term errors in proprioceptive feedback. Roll et al. (1991) showed that vibrating the inferior rectus tendon resulted in the illusion that a target was moving upward in space, with a concurrent error in open-loop pointing.

Gauthier et al. (1990) showed that using a suction cup on the orbit, passive rotation of a patched eye produced an error of pointing to a fixed target that went with the direction of rotation, which suggested that passive stretch of an extraocular muscle produced a proprioceptive signal that contributed to a change in the perception of location. Bridgeman & Stark (1991), by using an open-loop pointing task to assess target location, attempted to separate corollary discharge from proprioceptive signals. They did this by applying continuous pressure on a patched eye that was either uncompensated (hence no corollary discharge) or pressure on an unpatched eye fixating that was compensated (by a motor command, and hence with a corollary discharge). They estimated that a quarter of the representation of a nonviewed spatial target came from proprioception and that the remainder came from corollary discharge.

Lewis & Zee (1993) examined a patient with a congenital trigeminal-oculomotor synkinesis, whose lateral movement of her left jaw (left lateral pterygoid contraction) produced an adduction of her left eye (left medial rectus contraction). They argued this patient’s synkinesis was due not to a motor signal from oculomotor sources but to an aberrant signal originating from trigeminal motor nuclei. Therefore, any signal arising from the movement of her covered left eye would be proprioceptive only and not associated with a corollary discharge. The patient’s perceived change in location of a target fixated by her uncovered fixating eye was opposite the direction of rotation of her covered synkinetic eye. The authors proposed that the difference between their findings and those of Gauthier et al. (1990) and Bridgeman & Stark (1991), both of whom invoked a muscle length stretching signal, were due to different effects on the myotendinous organ by passive versus active muscle contraction.

If the palisade endings do indeed provide the oculomotor proprioceptive eye-position signal, one must view with caution a number of experiments that used section of the trigeminal nerve to obliterate oculomotor proprioception and then showed preservation of normal function; this was demonstrated through double-step saccades (Guthrie et al. 1983) and open-loop pointing (Lewis et al. 1998). Nonetheless, monkeys with trigeminal nerve section do have a deficit with adaptive control: Monkeys with extraocular muscle palsies lost ocular alignment and saccadic conjugacy after trigeminal nerve section in a linear progression over 20 days (Lewis et al. 1994).

TWO DIFFERENT MECHANISMS FOR THE REPRESENTATION OF OBJECTS IN THE WORLD FOR ACTION AND PERCEPTION

How then does the primate brain accurately locate objects in the world when the retinal representation is inaccurate or unavailable? We suggest that psychophysical and physiological studies provide evidence for two mechanisms: The first is a rapid corollary discharge mechanism, which is approximate and calculates movement trajectories but not absolute locations. The neural activity that subserves the rapid mechanism is the predictive remapping of the visual representation by a corollary discharge. Calculation of the spatial location of an object that flashed immediately before a saccade is disrupted when the neural network involved in corollary discharge is damaged. Importantly, this rapid representation does not need or use eye position in craniotopic coordinates. The second is a slower mechanism, which uses oculomotor proprioception to establish an accurate craniotopic representation. The intertrial memory in the FEF may be a physiological example of this process, utilizing a craniotopic representation in memory. The gain field mechanism is likely to be used to generate this slow, accurate craniotopic mnemonic representation, but future work will have to establish that gain fields are in fact used to do so.

A PROBLEM FOR THE FUTURE

The literature reviewed here makes the assumption that corollary discharge and remapping are the determinants of visuospatial perception despite a moving eye. The Duncker illusion questions the universality of this idea. When humans (Zivotofsky et al. 1996) and monkeys (Zivotofsky et al. 2005) pursue a small spot moving orthogonally to a flow field, they perceive the motion of the spot and the trajectory of their pursuit eye movement to be distorted diagonally away from the direction of the flow field (a video example of the Duncker Motion Illusion is found at https://youtu.be/QUbJKakfmZw). Their pursuit eye movements are accurate, not distorted like their perception of the movements. If a saccade target flashes above the pursuit trajectory during the pursuit and the subject makes a memory-guided saccade to it at the end of the pursuit epoch, the saccade overshoots its target when the flow field moved toward the target and undershoots its target when the flow field moved away from the target. The subjects are therefore compensating for a pursuit trajectory that they perceived but did not follow, choosing to use their erroneous perception rather than either corollary discharge or oculomotor proprioception to determine the location of the saccade target in space. Spatial perception is by no means a solved problem.

SUMMARY POINTS.

1. The primate eye is always moving, so the retinal location of an object is not useful in determining where it is in space. In order for the brain to calculate the spatial location of an object so one can reach and touch it, or throw a ball at it, the brain must compensate for eye movements.

2. Two mechanisms have been proposed to solve this problem. The first mechanism is corollary discharge: The motor plan that will drive the movement is also fed back to the visual representation of the object to compensate for the impending movement.

3. Neurons in many brain areas, including the lateral intraparietal area (LIP), the frontal eye fields (FEFs), the superior colliculus (SC), pre-striate area V4, and the parietal reach region, remap their retinotopic visual receptive fields to compensate for an impending saccade. In the FEF, this remapping is done by a corollary discharge signal motor signal sent from the SC to the medial dorsal nucleus of the thalamus and thence to the cortex.

4. The second mechanism is an eye position signal used to calculate the position of the object in craniotopic coordinates (relative to the center of gaze when the subject is looking straight ahead). The eye position signal may come from the representation of oculomotor proprioception in area 3a of primary somatosensory cortex. This representation lags the actual eye position by 60 ms.

5. Neurons in LIP have their retinotopic receptive fields modulated by eye position (the so-called gain field). Gain fields can be used to calculate target position in craniotopic coordinates. Because gain fields are inaccurate for 150 ms after a saccade, they cannot be used to solve all aspects of the spatial accuracy problem.

6. Psychophysical evidence suggests that there are two different representations of space: a rapid retinotopic one and a slower craniotopic one. Remapping of a retinotopic signal can affect the former; using an eye position signal, perhaps oculomotor proprioception, can affect the latter.

FUTURE ISSUES.

1. How accurate is the remapping process?

2. How do stimulus characteristics—for example, timing and shape—affect localization?

3. Do gain fields require an intact somatosensory cortex?

4. Does transient inactivation of oculomotor proprioception have any effect on behavior?

5. What is the neural mechanism that translates the gain field signal back to the vector signal necessary to generate most eye and skeletal movements?

6. How is the change from an early remapping strategy to a late craniotopic strategy accomplished? What is its actual time course?