Visual Perception and Corollary Discharge

Abstract

Perception is dependent not only on sensory input but on the state of the brain receiving that input. A classic example is perception of a stable visual world in spite of the saccadic eye movements that shift the image on the retina. The long standing hypothesis is that the brain compensates for the disruption of visual input by using advance knowledge of the impending saccade, an internally generated corollary discharge. A possible neuronal mechanism for this compensation has been previously identified in parietal and frontal cortex of monkeys, but the origin of the necessary corollary discharge remained unknown. Here we consider recent experiments that identified a pathway for a corollary discharge for saccades that extends from the superior colliculus in the midbrain to the frontal eye fields in the cerebral cortex with a relay in the medial dorsal nucleus of the thalamus. We first review the nature of the evidence used to identify a corollary discharge signal in the complexity of the primate brain and show its use for guiding a rapid sequence of eye movements. We then consider two experiments that show this same corollary signal may provide the input to the frontal cortex neurons that alter their activity with saccades in ways that could compensate for the displacements in the visual input produced by saccadic eye movements. The first experiment shows that the corollary discharge signal is spatially and temporally appropriate to produce the alterations in the frontal cortex neurons. The second shows that this signal is necessary for this alteration because inactivation of the corollary reduces the compensation by frontal cortex neurons. The identification of this relatively simple circuit specifies the organization of a corollary discharge in the primate brain for the first time and provides a specific example upon which consideration of the roles of corollary activity in other systems and for other functions can be evaluated.

INTRODUCTION

No one would argue that our perception is the passive interpretation of what falls on our sensory receptors. This is particularly the case for the images falling on our retina. The acid test for this view is the simple challenge forced upon our brain several times per second as our eyes flit about to different points in a visual scene. Although the retinal images continually shift, we do not perceive them as jumping about; rather, our perception of the visual world remains stable. This is perhaps one of the most striking and persuasive arguments for active visual perception, be it referred to as preemptive perception, enactive perception, or simply active perception.

The aspect of this problem that we address in this paper is the possible contribution of the signals used for the generation of eye movements to the modification of perception, specifically the problem of the stability of our visual perception in the face of saccadic eye movements. We have explored this issue by building on the substantial knowledge of the visual and oculomotor system in the best model of the human visual system, that of the old world monkey, Macaca mulatta. Our experiments initially centered on identifying an internal representation of a planned saccadic eye movement, a corollary discharge, and then showed how this corollary is used for the planning of successive eye movements. More recently we have extended these observation to consider how this corollary discharge might contribute to mechanisms underlying visual perception. The details of this series of experiments have been reported previously (Sommer and Wurtz, 2002, 2004a, 2004b, 2006) so that what we consider here are the salient points of the research findings and the relation of these findings to active visual perception.

Identifying a corollary discharge in the primate brain

The central idea of a corollary discharge is that the same signal that leaves a sensory motor area as a movement command, to eventually reach the motor neurons for producing a movement, is also directed towards other parts of the brain as a corollary discharge as shown conceptually in Fig. 1A. This corollary discharge does not generate the movement but instead is directed to regions of the brain that require information about the impending movement. The assumption is that in these other brain areas this signal is used both for planning subsequent movements and for interpreting reafference, i.e. the sensory events that will result from the movement (von Holst and Mittelstaedt, 1950).

Figure 1.

The concept of a corollary discharge and it implication for saccadic eye movements. A. The corollary discharge signal originates in the same sensorimotor structure as the command to make the movement. B. The corollary discharge considered here is conveyed through a pathway running from superior colliculus through mediodorsal thalamus (MD) to the frontal eye field.

The problem of identifying the neurons in the brain carrying the corollary discharge signal, however, is not trivial. How do we distinguish whether a neuron from which we are recording is carrying the movement command signal or the corollary discharge signal, if by definition, both signals look identical? In the monkey brain a corollary discharge had not previously been identified so there were no established ground rules. We therefore developed a set of four criteria that if met would satisfy us that we were recording from neurons carrying a corollary discharge (Wurtz and Sommer, 2004).

The first criterion was that the corollary discharge should originate from a brain structure known to be involved in the generation of the movement but travel toward brain areas that were not on the final output pathway. The structure we identified as a candidate for generating saccadic eye movements was the superior colliculus (SC) lying on the roof of the midbrain where neurons in the intermediate layers are active before saccades. These neurons project down to areas of the midbrain and pons that lie on the pathway to the extraocular muscle nuclei which drive the eye movements (Sparks and Hartwich-Young, 1989). An additional projection from the SC is upward to the medial dorsal nucleus of the thalamus (MD), which then projects to the frontal eye field (FEF) of frontal cortex (Fig. 1B). Anatomical evidence for this pathway came from retrograde labeling and anterograde degeneration studies (Benevento and Fallon, 1975; Goldman-Rakic and Porrino, 1985) and from transynaptic retrograde labeling (Lynch et al., 1994).

A critical point that will pervade all of our subsequent investigations of this corollary discharge pathway is that it has a relay in MD thalamus. This is an experimental gift because it allows us to alter the transmission of the corollary discharge by manipulating MD relay neurons without perturbing either the source of the corollary, the SC, or its target, the FEF, both of which are important oculomotor structures in their own right. An analogy is to study the influence of a computer on a printer; one would learn little by hitting each machine with a hammer, but one would learn a lot by cutting the cable that connects them. But the presence of a relay node in MD also generates a problem: how do we find it? Neurons in MD may receive input from, and send output to, many different parts of the brain, yet we want to identify only those MD neurons that get input from SC and project to the FEF. Back to our analogy, this is like blindly grabbing a tangle of cables behind one’s desk and trying to determine which one connects the computer to the printer. We solved this problem by using the classical neurophysiological techniques of antidromic and orthodromic stimulation (Sommer and Wurtz, 1998, 2002, 2004a). We selected for study only those MD neurons that could be orthodromically driven (through synapses) from the SC and antidromically driven (backfired through its own axon) from the FEF (Fig. 2A). This does not exclude the possibility that these neurons might project to other frontal cortical areas in addition to FEF or receive from other areas besides SC, but at the least we could say definitively that all of the neurons we studied did receive input from SC and project to FEF, identifying them as relay neurons in this pathway. In sum, we isolated signals coming from a known oculomotor structure, the SC, but which traveled up to cortex rather than toward the muscles, thus satisfying the first criterion.

Figure 2.

Establishing the neuronal correlates of a corollary discharge for saccades. A. Method used to identify the neurons in MD that both receive input from SC and project to FEF. Every MD relay neuron was double-identified: it was both antidromically activated from the FEF (showing that it projected to FEF) and orthodromically activated from the SC (showing that it received input from the SC). Dashed line connections indicate other types of potential connections that may be present. B. Temporal characteristics of the corollary discharge. Once an MD relay neuron was isolated it was studied by having the monkey perform a delayed saccade task. The monkey looked at a fixation spot, then a target appeared in the periphery, and after a delay period of 500–1000 ms the fixation spot disappeared (Cue to Make Saccade), which was the prompt to make a saccade to the target. The example neuron is a visuomovement MD neuron with a burst of activity starting before the saccade plus a small visual response. C. Spatial characteristics of the corollary discharge. Schematic of a typical MD relay neuron movement field. The movement field (dashed oval) indicates which saccades (thick arrows) were accompanied by the burst of activity, and which were not (thin arrows). Peak activity was always associated with saccades to the contralateral visual field.

The second criterion was that the signals should represent temporal and spatial parameters of the movement. We studied the activity of the MD neurons while monkeys made saccades to visual targets (Sommer and Wurtz, 2002). The temporal characteristics of many MD neurons were appropriate to represent the upcoming saccade: three quarters of them had a presaccadic burst of activity (Fig. 2B). The median start of this burst was 66 ms before saccadic initiation. Note that because this activity preceded the saccade, it could not have resulted from proprioceptive feedback from eye muscle contraction, which had not yet begun. As to the MD neurons’ spatial characteristics, like SC neurons, the MD neurons were directionally tuned; most neurons had peaks of activity related to a limited range of saccadic amplitudes and directions (Fig. 2C). A critical point that we will return to later is that best direction for the MD neurons was for saccades into the contralateral visual field.

The third criterion is that eliminating the neuronal activity should not impair movements in simple tasks that did not require a corollary discharge. That is, the neurons conveying corollary discharge should not lie in the pathway producing the movement. A priori, we did not know whether or not the signals sent from SC up to FEF were actually leading to saccade generation via a circuitous route, rather than just providing a corollary discharge. We resolved this uncertainty by inactivating the MD neurons (Fig. 3A) and determining whether the monkey could still make saccades to visual targets. Following inactivation the accuracy, latency, and peak speed of the saccades was not altered (Sommer and Wurtz, 2002, 2004b). Thus, the MD-mediated signals do not seem to be used for generating simple, single saccades.

Figure 3.

Evidence that the corollary discharge from SC is necessary for guiding rapid sequential eye movements. A. Inactivation of the corollary discharge pathway was accomplished by injecting muscimol at the sites of previously recorded MD relay neurons. B. The double-step task requires a corollary discharge for correct performance. The monkey first looked at a fixation spot which then disappeared as two targets were flashed sequentially and the monkey then made two saccades to the target locations. Due to the reaction time of the saccades, all stimuli were gone before the saccades started. With corollary discharge intact, the first saccade would go rightward and the second saccade would go straight up from there (solid arrows). Without the corollary discharge, the first saccade would go rightward, but the monkey would not know that its eyes were at a new position, would make the second saccade as if it were still looking at the center of the screen, and the second saccade would be diagonal (dashed arrow). C. The means (and SDs) of the initial fixation locations, first saccade endpoints, and second saccade endpoints for an example experiment. The only significant change was that predicted by loss of corollary discharge: there was a shift of second saccade endpoints in the contraversive (rightward) direction. After Sommer and Wurtz (2004b)

The fourth criterion is the most critical and requires that eliminating the corollary discharge signal should disrupt performance in tasks that do require corollary discharge. The task we used that requires corollary discharge is the double-step task, in which the monkey had to make successive saccades to two flashed targets (Fig. 3B). This task is widely used as an assay for the presence of corollary discharge in oculomotor research (including in patients with cortical lesions, Duhamel et al., 1992b, and thalamic lesions, Bellebaum et al., 2005) because execution of the second saccade (the upward, solid arrow in Fig. 3B) requires knowledge of where the eye was displaced after the first (horizontal) saccade. No visual feedback is available about where the eye is after the first saccade because the saccades begin after the targets disappear and the experiment can be done in the dark. Proprioception probably has little influence in the online control of saccades (Lewis et al., 2001) and is unnecessary for performing a similar double-step task (Guthrie et al., 1983). After the MD inactivation, if the corollary discharge is reduced, the monkey should make a second saccade as if it never made the first, i.e. as if the saccade was made from the original fixation point diagonally to the right (dashed arrow in Fig. 3B). Following inactivation of MD (Fig. 3A), the second saccade endpoints were in fact shifted to the right as shown in a sample experiment (Fig. 3C) as predicted by loss of corollary discharge. Over the series of injection experiments, the size of the horizontal shift was highly significant and in the predicted direction. The magnitude of the effect was small, however, about 11% of the maximum possible shift. Further trial-by-trial analyses of the data showed that the precision of the second saccades also was disrupted as predicted by loss of corollary discharge, and this effect was larger (about 50% of maximum possible; Sommer & Wurtz 2004b). Overall, though, a take-home message was that the deficits were partial, not total. We think this is related to incomplete inactivation of MD or to the existence of other remaining pathways to cerebral cortex which are not fully identified.

In contrast, when the saccades were made to targets in the ipsilateral visual field we found no deficit; second saccades showed no average shifts or reduced precision (not shown). This is a critical control because, as we noted above, the MD neurons were most active before saccades to the contralateral field. We therefore found what we predicted knowing the activity of the MD neurons: deficits only when the first saccade (the one that had to be represented by corollary discharge for directing the second saccade) went into the contralateral visual field.

Overall, we conclude that the signals in the pathway from SC to FEF via MD meet the four criteria we regard as necessary to identify a corollary discharge. The signals originate from the SC, a known visual-motor area; they convey the timing and spatial parameters of the upcoming saccade; blocking the corollary pathway does not affect saccades in a simple task not requiring a corollary discharge; and blocking it does disrupt saccades in a double-step task that requires a corollary discharge.

A Neuronal Basis for Stable Visual Perception

Now that we believe that we have identified a corollary discharge, we next tried to see if the same corollary might be contributing to the stability of perception during saccadic eye movements. For this next step we built on the previous experiments and insights of Duhamel, Colby and Goldberg (1992a) who studied the remarkable ability of some neurons in the cerebral cortex to change their response to a visual stimulus just before a saccadic eye movement occurred. They argued that these changes could underlie the stability of visual perception in spite of saccadic eye movements, and that the alteration of activity resulted from input from a corollary discharge. Their experiments were in parietal cortex, but similar characteristics were subsequently found for neurons in the FEF by Umeno and Goldberg (1997). We therefore address the question, is the corollary discharge signal that we find ascending from SC to FEF the signal that is used to alter the visual responses of FEF neurons during saccadic eye movements.

First we determined that the alterations reported by Umeno and Goldberg (1997) could be found in those FEF neurons that we showed by orthodromic or antidromic stimulation to be part of the SC FEF circuit. The critical characteristic of the alteration of activity with saccades in these neurons is that before the onset of the saccade, the neurons become visually responsive at the site where the receptive field (RF) would fall after the saccade was made. We will refer to such neurons as having a shifting RF. The spatial location relative to the fovea where the classical RF shifts to is called the future field (FF), because that is where the RF will be in the future, after the saccade. When the saccade is subsequently made, the FF becomes the RF. The hypothetical relation of the anticipatory shift to stability of perception across saccades is that, if the FF and postsaccadic RF samples match, visual stability is perceived; if not, visual displacement is perceived.

Figure 4 illustrates this shifting receptive field phenomena for an FEF neuron that is connected to the SC. The monkey first looked at a fixation point (Fix Spot, Fig 4A and B), and then subsequently made a saccade from that point to a target point (Target). A visual probe stimulus was then flashed at one of several times before or after the saccade (Fig. 4A) and at one of two places (Fig. 4B): at the center of the RF or at the FF. Only one probe was flashed per trial. Figure 4C (left box) shows the visual response to a stimulus flashed long before the saccade in the RF. There was no response in the FF. In contrast, if a probe was flashed just before the saccade began, it elicited a visual response at the site of the FF (Fig. 4C, right box). For many neurons, like this one, the emergence of a FF occurred even for probes flashed hundreds of ms before the saccade (Fig. 4C, second record from left, probe was flashed ~200 ms before saccade). Note that in this neuron the activity remained at the RF site even as the activity at the FF increased. In other neurons the activity in the RF decayed as the activity in the FF developed.

Figure 4.

Shifting Receptive Field (RF) in a FEF neuron. A. Outline of the temporal sequence for the two fixations points (Fix spot and Saccade Target) and eye position. The boxes under the eye position trace indicate the timing of the probe stimulus flashes. B. Spatial organization. The monkey’s task was to look at the fixation spot (Fix) and then make a saccade to the Target placed far from the RF. Meanwhile, a visual probe was flashed in either the receptive field (RF) or the future field (FF). (C) Shifting RF of an example FEF neuron. Firing rate (mean and SE) is plotted as function of time and aligned with events represented in diagrams from A above. Long before the saccade (−70 ms column), the probe elicited a strong response in the RF (upper graph) but not in the FF (middle graph). After Sommer and Wurtz, 2006

Contribution of the Corollary Discharge to Perceptual Stability

This is where a corollary discharge enters the picture. If the shift of the RF occurs before the saccade and is related to the amplitude and direction of the saccade (as it is), it must rely on information about the upcoming saccade. Such information is exactly what is carried by the corollary discharge we have identified. But does the shift depend upon this corollary discharge? We answered this question in two sets of experiments. First, we recorded from FEF neurons with shifting RFs to see if the spatial and temporal characteristics of their shifts were consistent with the corollary discharge signals in the SC-MD-FEF pathway. Inconsistency would seem to reject our hypothesis outright that the corollary causes the shift. Second, we directly tested the causal relation between the corollary discharge and the shifting RFs by inactivating the pathway to see if the shift was altered.

On the first point, if neurons with shifting RFs in FEF use the corollary discharge ascending from the SC through MD, then the characteristics of the shifts should be predictable from the spatial and temporal activity of the corollary recorded in MD. Spatially, these neurons encode the vector of the next saccade; the activity indicates the eccentricity and direction of the impending saccade. Therefore the shifting RFs should jump as if driven by this vector of the impending saccade (Figure 5A, top) instead of spreading as if the RFs simply expanded or were driven by continuous, instantaneous eye position (Figure 5A, bottom). We tested the nature of the shift (Sommer and Wurtz, 2006) by placing visual probes at the midpoint between the previously used RF and FF locations (Figure 5A). For the dozen neurons in FEF that we tested, we found no significant change in activity at the midpoint in any of them, nor in the population activity. We concluded that there is no evidence of a spread of activity; the results were consistent with a jump.

Figure 5.

The spatial and temporal properties of shifting RFs and their relation to the corollary discharge. A. Spatially, input from a saccadic corollary discharge would predict that shifting RFs jump (upper diagram) as indicated by a significant (*) increase of activity at the FF (arrow) but no significant difference (n.s.d.) at the midpoint. Alternatively, shifting RFs could spread (lower diagram), as indicated by a significant increase at both the midpoint and the FF (arrows). B. Temporally, our hypothesis predicts that shifting RFs are synchronized with saccades. Shift activity (in the FF) for an example neuron is aligned with probe onset (left) and saccade initiation (right). Rasters of action potentials from each trial are sorted by increasing saccadic latency from bottom to top, and eye position traces for each trial (horizontal component) are shown below. Arrow indicates average visual latency of this neuron. After Sommer and Wurtz, 2006

The activity of SC neurons also have a distinct temporal component: the neurons increase their firing ~100 ms before the saccade to create a burst that peaks near saccade initiation. If the corollary discharge from the SC is acting on the FEF neurons to produce the shifting RFs, we would expect the shift of the visual response to be better synchronized with the onset of the saccade than to the onset of the stimulus. In other words, a neuron would detect a stimulus in the FF but would withhold its response (in terms of emitting action potentials) until triggered to do so by the saccade-related input. Although it sounds strange, such a saccade-triggered visual response would be crucial to ensure that shifting RFs are registered by the brain only once a saccade is guaranteed to occur (the point of no return for saccade generation is ~100 ms before saccade initiation). As expected, the results for a sample neuron (Figure 5B) show that the activity at the FF site is better synchronized to the onset of the saccade than to the onset of the FF stimulus (Duhamel et al., 1992a; Sommer and Wurtz, 2006). This was true for our sample of neurons with the shift of activity beginning on average 174 ms after probe onset, greatly exceeding the visual latency for the same neurons of 86 ms. Thus for both the spatial and the temporal characteristics of the shifting RFs in the FEF neurons, the corollary discharge from the SC is sufficient and appropriate for causing shifting RFs.

The second question is whether the corollary discharge is necessary to produce the shift. To test for this, we first recorded from an FEF neuron that was visually responsive, that was connected to the SC, and that showed shifting receptive fields. We then inactivated neurons in the MD relay that carried the corollary discharge from SC to FEF as we described above (Fig. 3A). We quantified the same FEF neuron’s visual sensitivity at the RF and FF (as shown in Fig. 4B) before and after inactivating MD. If shifting RFs depend on corollary discharge passing through MD, the shift (as indicated by activity at the FF) should decrease.

Figure 6 shows the result of the inactivation for one FEF neuron. Before inactivation (gray outlined plots), the neuron showed the shift of activity as indicated by an increased visual response at the FF 70 ms before the saccade. During the inactivation (Fig 6, black plots), this visual activity at the FF declined by 70%. Thus without the corollary discharge, the response at the FF was greatly reduced. Overall, the average reduction in FF activity was 50%, with no significant change in the classic visual response (at the RF). Again, we have no assurance that we inactivated all of the corollary discharge pathway or that the pathway through MD is the only pathway.

Figure 6.

Inactivation of the corollary discharge reduces the RF shift. Recording from a single neuron in FEF while MD relay neurons that convey corollary discharge are reversibly inactivated. In the top row are responses to probes flashed in the RF; in the bottom row are responses to probes flashed in the FF. The grey filling highlights the differences in activity during vs. before inactivation. Visual responsiveness in the FF just before the saccade – i.e. the shift activity – decreased significantly during inactivation of MD. From Sommer and Wurtz, 2006.

A critical aspect of these MD inactivations was that the reduction in the response we found in the FF was for saccades made into the contralateral visual field and not for those made into the ipsilateral visual field. As noted above, this contralateral preference is just what we would expect because the neurons in MD that carry the corollary discharge have their peak response before saccades to the contralateral visual field. Though our sample of neurons in which we did the entire experimental sequence of recording and inactivation was limited due to the technical challenges inherent in the experiment (eight FEF neurons), the results were strikingly consistent across injections (Sommer and Wurtz, 2006).

Implications for Stable Visual Perception

We think that the experiments on neurons in frontal cortex of the monkey summarized here provide some insights into the contribution that a corollary discharge might make to the stability of visual perception. We have tried to show how a corollary discharge, which we previously established for the control of sequential saccadic eye movements, contributes to the neuronal activity that might underlie the stability of our visual perception. Our logic rests on the assumption that shifting visual receptive fields, first discovered by Duhamel, Colby, and Goldberg (1992a), contribute to the stability of perception during saccadic eye movements. This is not proven yet, but the shifting RF phenomenon is probably the best candidate mechanism for stabilizing visual perception yet discovered. Shifting RFs require a corollary discharge input and we have now shown that the corollary discharge from the SC that we established for the oculomotor sequences probably also underlies the shifts. We showed this in two sets of experiments. First, we demonstrated that the spatial and temporal characteristics of the shift were consistent with the characteristics of a corollary discharge input. The RFs of the FEF neurons shifted by a jump of activity, not a spread, compatible with vector input of a corollary discharge, and the shift was synchronized to the saccade consistent with the burst of presaccadic input from the corollary discharge. Second, we showed that in addition to being an appropriate input, the corollary discharge was necessary for the shifting RFs: inactivating the corollary discharge pathway in MD significantly reduced the magnitude of the shift. Furthermore, the deficit was lateralized just as predicted by the contraversive saccade representation in the corollary discharge pathway. Thus we think that these two experiments show that the shifting RFs in the FEF rely on input from a corollary discharge arising through the SC-MD-FEF pathway.

Several key questions remain. First, as already noted the MD inactivation caused deficits that were consistent and strong but did not eliminate the shift. Hence, there may be other corollary pathways that contribute to the corollary discharge input to the FEF neurons. In addition, the original discovery of shifting RFs was in the parietal lobe, not FEF. We do not know the corollary input to the parietal neurons although direct interaction between FEF and parietal cortex is well known (Chafee and Goldman-Rakic, 1998, 2000; Clower et al., 2001; Ferraina et al., 2002).

The most critical issue, however, is the assumption that the shifting RFs do in fact mediate the sense of spatial constancy across saccades. While this hypothesis is compelling, since shifting RFs by their nature seem to predict the consequences of a saccade, it has suffered from an apparent lack of testability because a causal test would be to perturb the shift and see if perceptual stability changed predictably. It has been hard to imagine how one could manipulate the shift while leaving other properties of the FEF neurons unchanged. A major implication of the present result is that such an experiment is now within reach. Our next step will be to determine if silencing the SC-MD-FEF pathway, which impairs shifting RFs, in turn creates a percept that the world lurches with each saccade.

Implications for Active Perception

The hallmark of these observations is that the neurons in the frontal cortex that have a clear visual response are modulated by a now identified corollary discharge emanating from the circuits generating saccadic eye movements. It is essentially a visual response that is gated by saccadic eye movements.

To return to the topic of this set of articles, this impact of motor planning on visual processing would seem to be the best model to date in the primate brain of one of the meanings of preemptive perception, namely, that the corollary discharge for the impending action has a preemptive effect on visual processing (see Bodis-Wollner, this volume). More generally, the interaction of the corollary discharge for the impending movement and the visual processing would seem to be consistent with the idea of enactive perception that entails the continuing interaction of planning actions and perception (but see discussion by Jacob, this volume).

While we think the understanding of the alteration of visual processing described in these experiments on corollary discharge can provide observational underpinnings to discussions of preemptive or enactive perception, we also recognize that these experiments may have limited general applicability. What we have demonstrated is an effect of the saccadic corollary discharge on the visual processing that is disrupted by the saccade. There is the possibility, however, that this is a highly specific interaction and that generalizing from this interaction to broader issues of preemptive and enactive perception may be premature. But the conceptual and technical methods used in these experiments might well provide the means of addressing these more general questions experimentally.