Predictive adjustment of the perceived direction of gaze during saccadic eye movements

Abstract

When we look at a stationary object, the perceived direction of gaze (where we are looking) is aligned with the physical direction of eyes (where our eyes are oriented) by which the object is foveated. However, this alignment may not hold in a dynamic situation. Our experiments assessed the perceived locations of two brief stimuli (1 ms) simultaneously displayed at two different physical locations during a saccade. The first stimulus was in the instantaneous location to which the eyes were oriented and the second one was always in the same location as the initial fixation point. When the timing of these stimuli was changed intra-saccadically, their perceived locations were dissociated. The first stimuli were consistently perceived near the target that will be foveated at saccade termination. The second stimuli once perceived near the target location, shifted in the direction opposite to that of saccades, as its latency from saccades increased. These results suggested an independent adjustment of gaze orientation from the physical orientation of eyes during saccades. The spatial dissociation of two stimuli may reflect sensorimotor control of gaze during saccades.

Introduction

During a saccade to an object of interest, we feel that we are seeing the visual world as stable despite displacement of the retinal image caused by eye movements. Researchers have often used a brief stimulus flashed around the time of a saccade to probe into the dynamics of neural processing sequence of visual stability (Morrone et al. 1997; Ross et al. 1997, 2001; Diamond et al. 2000). The pattern of mislocalization of stimuli or the resulting space distortion found in these studies has been of specific interest since it might be attributable to the predictive relocation of the receptive field of visual neurons in many cortical areas (Duhamel et al. 1992; Nakamura and Colby 2000; Sommer and Wurtz 2002, 2006), which may help to avoid long visual processing delays as well as to mediate visual stability across saccades (Jeffries et al. 2007; Kusunoki and Goldberg 2003). Actually, recent studies have provided supportive indications that perisaccadic space distortion is linked directly to the process of recovering accurate spatial representations from distorted retinotopic visual maps, arising from predictive relocation of receptive fields (Hamker et al. 2008; Richard et al. 2009; VanRullen 2004) (see also Krekelberg et al. 2003).

When the predictive shift of visual receptive fields is part of the processes that stabilize the visual world, our visual experience as to when or where we are looking may also be influenced by this mechanism (e.g., Morrone et al. 2005). Recently, Hunt and Cavanagh (2009) have shown that when human subjects saccaded to a clock with a fast-moving hand, they saw a time that predated their eye movements by comparable duration to the saccade. This means that at the beginning of a saccade the subjects were already seeing a moving clock hand that was falling on the peri-foveal region of the retina. As they suggested, if the receptive fields are shifted in a predictive manner, the perceived direction of our gaze may also be influenced by them, such that we experience that our gaze is directed to the target before our eyes arrive there. However, their analyses were focused on the times perceived on the clock when the subjects judged that they were looking at it, and thus no concrete measure representing the timecourse of gaze adjustment during saccades was given.

Here, we examined a simple experimental situation in which horizontal saccadic eye movements were fed back to gazing subjects. The physical direction of the eyes was adjusted to align with the current position of a light spot on an oscilloscope across a saccadic period, on which a pulse stimulus was superimposed at different times. From the timecourse of perceived positions of the pulse stimuli on the oscilloscope screen, we found that, independently from the physical orientation of the eyes, gaze was always directed in the target position from the beginning toward the end of saccades.

Methods

Four observers were used in the experiment. The subjects were seated in a dimly lit room to view an oscilloscope screen binocularly (viewing distance: 30 cm), with the head fixed by chin and forehead rests. The subjects conducted 13-degree (°) rightward horizontal saccades over a gray oscilloscope screen, where eye movements represented by electro-oculographic (EOG) signals were displayed as horizontal motion of a light spot (Fig. 1A). The scale of the light spot on the oscilloscope screen was carefully adjusted such that the horizontal distance between start and end positions of the light spot was equal to the corresponding change of the orientation of eyes between separate fixations. By this method, the start and end positions of the light spot could be located in a rectangular starting grid taped with white lines and on the target position (a red LED), respectively (Fig. 1A).

Fig. 1.

A Schematic illustrations of two stimuli on an oscilloscope screen (gray circles). Left panel image of two stimuli in physical space when they are triggered at 6°. The screen displays a path of a light spot as a horizontal line, on which a vertical pulse stimulus S1 is superimposed. Before initiating each trial, the resting EOG potential was located in a starting grid (a white rectangle). Green circle on the left represents a light-emitting diode (LED) to provide S2. A red LED on the right is to provide a reaction time stimulus to trigger a saccadic eye movement, and serves to indicate the target position to which the saccade is made. Right panel impression of two stimuli perceived in the same trial as in the left. Note that the perceived position of S1 is greatly displaced from the real position, while S2 is also displaced although it lags behind S1. B Typical records of EOG signals obtained from one subject for one experiment when two stimuli were triggered at 6°. (a) EOG signals. (b) Pulses to trigger two stimuli. (c) Reaction time stimuli. In all panels, data for 47 trials are aligned with the onset times of eye movement (0 ms) and superimposed

Before initiating a trial, the subject was asked to gaze in the direction of the center of the starting grid; the experimenter then adjusted the position of resting EOG potential to be in the grid to minimize misalignment of the physical eye position with the light spot position, arising from baseline fluctuation of EOG signal. After a warning period of between 1,500 and 2,500 ms, the saccade target was illuminated transiently (1 ms), and the subjects saccaded to it as soon as it was illuminated. This procedure was repeated on a trial-by-trial basis (Fig. 1B).

During the saccadic period, two different stimuli with 1 ms durations were displayed simultaneously: a vertical pulse stimulus (S1) superimposed on the instantaneous position of the light spot and a stationary stimulus (S2) flashed in the starting grid (a green LED shown in Fig. 1A). Both stimuli were triggered simultaneously when the light spot passed one of five different amplitude levels (1, 2, 6, 10, 13°) (Fig. 1B). In physical space, the S1 was always presented at the location to which the eyes were instantaneously directed between separate fixations, while the S2 remained stationary in the starting grid across the saccadic periods. The subjects reported the perceived positions of the two stimuli on a trial-by-trial basis.

The physical eye (Ep) position was recorded using an EOG system (AVH-10, Nihon-Koden) by placing Ag–AgCl skin electrodes at the outer canthi of both eyes. A ground electrode was placed just above the eyebrows in the center of the forehead. The EOG signals were low-pass filtered at 50 Hz, displayed on an oscilloscope screen and used as a common triggering source of two stimuli, S1 and S2. The short exposure (1 ms) of both stimuli effectively minimized motion blur during saccades, allowing the subjects to specify the apparent positions of the two stimuli. The perceived positions of the two stimuli were pointed out by the subject in each trial by adjusting the sensor head position of a linear potentiometer (Novotechnik, TLH1000, length 0.8 m) fixed sideways along the oscilloscope screen. Data collection consisted of repetition of a subset of ascending and descending trials, each of which involved five consecutive trials for one of five triggering conditions. This repetition lasted an hour. The number of trials executed by one subject per experiment averaged 250. Target presentation and data collection were controlled using custom software programmed in LabVIEW (National Instruments). Off-line analyses were performed using MatLab (The Mathworks). The onsets of eye movement were scored on the basis of 5% of the peak of first-time derivative of EOG signals.

Results

The subjects conceived that the motion of the light spot between separate fixations was too rapid to track, and thus the path of the light spot was not perceived intra-saccadically (right panel in Fig. 1A). However, the same subjects reported that the S1 superimposed intra-saccadically on the light spot was clearly perceived not in the physically correct direction, but in the direction of the target, across saccadic periods (Figs. 1A, 2a). This means that during saccades the light spot itself was actually seen, but its entire timecourse was shifted to be seen as a single dot near the target location.

Fig. 2.

Spatial relationships among three positional variables (S1, S2, Ep) as functions of time. a Physical eye position (Ep) when two stimuli were triggered at five different amplitudes. Note that when the trigger level was set at 1° (blue dots), both stimuli were frequently triggered before saccades. b, c The perceived positions reported for S1 and S2 in b and c, respectively. In all panels, plots for trials with five different triggering levels of 1, 2, 6, 10 and 13° are represented by blue, red, green, magenta and light-blue dots, respectively. Note that just around the onset of saccades with a trigger level of 1°, both stimuli were frequently invisible, and thus corresponding plots are not given in b and c. The time when eye movement was initiated (solid vertical lines) is defined as 0 ms. Gray areas denote time width of mean ± standard deviation of ending times averaged across all trials. All plots were derived from a single subject

The spatial relationship among the reported positions for S1, S2 and the corresponding Ep positions on the same timecourse during saccadic periods was as follows (Fig. 2a–c). When the trigger level was set at 1°, two stimuli were triggered occasionally before eye movements owing to noisy resting EOG potentials. These were perceived near the veridical positions (Fig. 2b, c). However, in a limited timecourse from before (>−5 ms) to just after the instant of saccades (<5 ms), neither stimulus was apparent. In the remaining trials at 1°, both stimuli triggered after the onset of saccades were perceived again, while their positions were greatly displaced toward the target, where the gaze would be directed at the end of movement. Thus, a difference in the perceived positions of the two stimuli was manifested when the trigger level was greater than 1°. The S1 was continually perceived near the target position irrespective of the triggered amplitudes, and matched the veridical position when the trigger level was set at 13° (Fig. 2b). On the same timecourse, by contrast, perceived positions of S2 once specified near the target position at 1° shifted in the direction opposite to that of saccades and then matched their veridical positions near saccade termination when the trigger level was set at 13° (Fig. 2c).

Figure 3 shows averaged plots for S1, S2 and the corresponding Ep positions as a function of time in panels A, B and C, respectively. The data obtained from four subjects are superimposed in each panel. The same positional dissociation of the two stimuli (S1, S2) during saccadic periods was apparent across four subjects. In panel D, a highly negative correlation coefficient was seen between the plots for positions S2 and Ep across the saccadic period (r = −0.995, P < 0.001, n = 20). Similarly, a highly positive correlation coefficient was confirmed between the plots for position Ep and the differences between positions S1 and Ep (r = 0.981 P < 0.001, n = 20).

Fig. 3.

Comparisons of the pattern of three variables (Ep, S1, S2). These are an equivalent representation of Fig. 2, but were averaged separately at each of five different triggering conditions. a–c Plots for Ep in a, perceived positions of S1 in b and of S2 in c. Data collected from four subjects are plotted separately by different colors. Movement times averaged across five tasks performed by all subjects were 45±1 ms. d Correlations among three positional variables (S1, S2, Ep). Closed circles denote plots for the perceived positions of S2 and the corresponding eye positions of Ep, and open diamonds denote the plots for S2 and the differences between the perceived positions of S1 and the corresponding eye positions of Ep. All plots were derived from the averaged records appeared in panels a, b and c

Discussion

During fixations, gaze is directed to the object that falls on the fovea, but this may not hold during saccadic eye movements. In our task, the eyes were always directed to a light spot and thus rotated in the same direction with the same velocity as the motion of the light spot. Nevertheless, perceived position of a stimulus superimposed on the light spot disagreed with its physical positions over the saccadic period; rather, it was predictive as if the gaze were directed in the target position across the saccadic periods. Independently from the physical orientations of eyes, our gaze during a saccade may be continuously directed to the target where the fovea will be placed once a saccade is completed, rather than what is actually falling on the fovea. This predicted adjustment of gaze direction may also lead to seeing a target before our eyes foveate it. Hunt and Cavanagh (2009) have shown that when subjects shifted their gaze to a clock with 8° saccades, the reported time was 39 ms earlier than the time when the eyes foveate the clock, implying that the subjects were seeing the target from nearly the beginning of saccades. They speculated that this may be an associative phenomenon with the predictive remapping of receptive fields of visual neurons. It is well established that the receptive fields of neurons in multiple retinotopically organized areas, including striate and extrastriate cortex (Nakamura and Colby 2000), lateral interparietal cortex (Duhamel et al. 1992) and the frontal eye fields (Sommer and Wurtz 2002, 2006), begin to shift just before the instant of a saccade. This means that responsiveness of neurons with nothing in their receptive field begins to increase without actual foveation of the target, if the saccade will bring it into the expected receptive field location at the end (Kusunoki and Goldberg 2003). Owing to this predictive shift of receptive field(s) of visual neurons, our ability to see the target during saccadic periods may be facilitated before our eyes arrive there.

The other important aspect of remapping is transfer of memory traces of salient stimuli. When the eyes move, the neurons that initially encoded the stimulus are thought to transfer their activity to neurons where the receptive field will encompass the stimulus location after the eye movement (Colby and Goldberg 1999). This type of remapping is stimulus-selective, involving only a salient object, causing its spatial location to be transferred across saccades in anticipation of future saccadic landing position (Melcher 2008). Our finding about the positional constancy of gaze direction during a saccade may be attributed to this continuous, object-based remapping for the time series of salient stimuli S1. The pattern of perceived position of S2 may be explained in the same context. It has been reported that a brief stimulus triggered pre-saccadically near the initial fixation point is displaced toward the target, whereas that triggered intra-saccadically is displaced toward its veridical position as its latency from saccade initiation increases (Morrone et al. 1997; Ross et al. 1997, 2001; Diamond et al. 2000). The later part of distortion, as it seems to be similar to the backward displacement of S2 during a saccade, may be attributed to the predicted remapping of salient stimuli, with the consequence that we see the visual world still being recovered from distorted retinotopic visual maps, arising from the predictive relocation of receptive fields until our eyes arrive there (Hamker et al. 2008; see also Krekelberg et al. 2003).

Our findings may be considered within a theoretical framework of the corollary discharge (CD) model (Martin 1976); if perceived positions of S1 reflect the pattern of CD signals, our results (Fig. 2) will agree with the model prediction: an error in stationary stimuli (=S2) originating from a spatial misalignment of CD (=S1) signal with the resulting saccade (=Ep). Although the CD signal in the model is a hypothetical neuronal signal, converging evidence from primate neurophysiology and human psychophysics provides additional concrete indications: the CD signal is an extra-retinal component to drive suppression of image motion on the retina caused by the saccade per se (Burr et al. 1994; Diamond et al. 2000), remapping of stimuli and/or shift of the receptive fields of visually responsive cells prior to a saccade (Duhamel et al. 1992; Sommer and Wurtz 2002, 2006), or all of them leading to the perception of a stable visual world across saccades (Wurtz 2008). All these studies have supported the idea that the CD signal from the oculomotor system allows the visual system to predict and discount spatial changes produced by an eye movement executed in the natural environment (see von Holst and Mittelstaedt 1950; Sperry 1950; Taylor 2007). Our findings about the spatiotemporal relationship among three positional variables (S1, S2, Ep) may capture the interaction between oculomotor and visual signals on a retinotopic visual map, as would be predicted from CD theory.

One might argue that attention shifts to the saccadic target before eye movement (Deubel and Schneider 1996) and leads to a predictive relocation of gaze. However, the shift of attention and shift of gaze are independent, as we can orient our attention without preparing a saccade (Kowler et al. 1995; Juan et al. 2004). Actually, when our subjects tried to shift their attention toward the target without preparing a saccade, they could not see a light spot near the target position, but could always see it near the initial fixation point (data not shown). Even in the preparatory period of saccades, attention can be voluntarily deployed away from the saccadic target (Montagnini and Castet 2007), suggesting that the initial shift of attention to the saccadic target is not always the critical step for saccadic relocation of gaze. It is known that the remapping of visual neurons in the posterior parietal cortex may reflect the attentional shift that normally precedes eye movements, although an attentional shift alone cannot produce a shift in the stored representation (Colby 1996), implying that only when an intended eye movement is about to occur do we see evidence for a remapped representation. Our predictive relocation of gaze, as it occurred during saccadic periods, was clearly movement-related but not, at least, merely attention-related. It is likely that while eye movements and attention normally coincide, the underlying neural mechanisms are distinct and subserve different cognitive functions.

The saccadic system is highly adaptive and can rapidly adjust to changes in the oculomotor musculature and plant (Kommerell et al. 1976), and fatigue (Barash et al. 1999). Moreover, the exposure to a visual stimulus has a persisting effect on sensory detection of subsequent stimuli (Tanaka and Sagi 1998). This so-called memory trace may have some biased effect on the perception of the subsequent stimuli, without the observer consciously recalling the previous stimuli (Squire 1987). In the repetitive ascending and descending trials used in this study, such memory traces might be established on a trial-by-trial basis, and affect the subjective judgment of the spatial location of the subsequent stimuli. Our future studies will be elaborated to minimize these biased effects on perception.