Time course of spatiotopic updating across saccades

Significance

Humans make frequent eye movements—about three to four times per second. Eye movements create changes in sensory input that the visual system should dissociate from changes in the outside world. Still, visual perception is introspectively undisrupted, but appears continuous. It has been hypothesized that the visual system anticipates the sensory changes based on a predictive signal from the oculomotor system. However, psychophysical studies suggested that this anticipation develops slowly: too slow for natural vision. Here, we examined the speed of this anticipation more closely using psychophysics and a motion illusion. We observed fast anticipatory updating, quantifiable in human behavior. The time scale at which the anticipation is reflected in behavior is compatible with typical fixation durations in natural viewing.

Abstract

Humans move their eyes several times per second, yet we perceive the outside world as continuous despite the sudden disruptions created by each eye movement. To date, the mechanism that the brain employs to achieve visual continuity across eye movements remains unclear. While it has been proposed that the oculomotor system quickly updates and informs the visual system about the upcoming eye movement, behavioral studies investigating the time course of this updating suggest the involvement of a slow mechanism, estimated to take more than 500 ms to operate effectively. This is a surprisingly slow estimate, because both the visual system and the oculomotor system process information faster. If spatiotopic updating is indeed this slow, it cannot contribute to perceptual continuity, because it is outside the temporal regime of typical oculomotor behavior. Here, we argue that the behavioral paradigms that have been used previously are suboptimal to measure the speed of spatiotopic updating. In this study, we used a fast gaze-contingent paradigm, using high phi as a continuous stimulus across eye movements. We observed fast spatiotopic updating within 150 ms after stimulus onset. The results suggest the involvement of a fast updating mechanism that predictively influences visual perception after an eye movement. The temporal characteristics of this mechanism are compatible with the rate at which saccadic eye movements are typically observed in natural viewing.

Humans sample the visual world by making fast, ballistic eye movements: saccades (1). Because acuity is not homogenous across the visual field (2), the fovea is directed to those locations that need to be inspected in closer detail. Saccades are made frequently—roughly every 200 ms to 300 ms (Fig. 1C; ref. 3)—causing stimuli to fall on different locations on the retina several times per second. Still, feedforward processing of visual information in the brain is even faster: It is possible to decode stimulus specific representations within 100 ms after stimulus onset (4), and humans can discriminate a peripheral object and make a saccade toward it in 120 ms (5, 6). However, given that the visual system is largely retinotopically organized (7), saccades repeatedly create temporal discontinuities and spatial instabilities in the retinotopic representations, posing a problem for continuity in visual processing. However, introspectively, most humans perceive a continuous and stable visual world without these distortions generated by saccades.

Fig. 1.

Experiment 1, design and results. (A) High phi example (48). An annulus of random low-pass filter noise is presented around the point of fixation. The annulus starts rotating slowly (inducer), clockwise (CW) or counterclockwise (CCW). Then, the random noise texture is replaced rapidly by four different textures, 20 ms per texture (transient). The transient induces the percept of a large rotational step in the opposite direction from the inducer. The percept of a backward step is illusory because, on average, the change of textures does not contain global motion in CW or CCW direction. (B) Perceived step direction with high phi as a function of inducer duration. Observers indicate whether they perceived a CW or CCW step when the transient was presented. Their responses were recoded to forward (1) or backward (−1) with respect to the rotation direction of the preceding inducer. More negative numbers reflect a stronger bias to perceive backward steps, and thus a stronger high phi. High phi increases with longer inducers but is already apparent after brief inducers. (C) Example distribution of fixation durations in natural viewing tasks (based on ref. 3). Comparing B and C, it can be noted that high phi can be induced within the temporal limits of a typical fixation. (D) Gaze-contingent conditions in experiment 1. The two conditions proceeded almost identically, with the only exception that the annulus remained static until saccade onset (saccade static, black) or started rotating immediately upon onset (saccade preview, yellow). Subjects maintained fixation until the annuli appeared. The dotted lines in the first panel were not actually visible but merely illustrate that the stimuli could appear at two locations (equal probability). The eye indicates gaze position in each panel. Arrows on the annuli illustrate that the annulus rotated in that phase of the trial. Median saccade parameters in rows 2 and 3 were obtained from the trials that were included in the analysis. (E) Model estimates of the average perceived step direction, where the error bars represent the 95% CI of the estimates obtained with nonparametric bootstrapping.

How is perceptual continuity established? One prominent hypothesis is that the visual system anticipates the change in sensory input caused by a saccade, based on a corollary discharge from the oculomotor system that carries information about the upcoming saccade (8, 9). Close to saccade onset, a subset of neurons responds to different retinotopic locations than they do under stable fixation (10–15). This anticipatory remapping of receptive fields could give rise to a transient nonretinotopic representation called “spatiotopic updating” (16, 17). Spatiotopic updating has been used to explain both the subjective impression of a continuous stream of visual perception across saccades (18, 19) and the objective psychophysical evidence for transsaccadic integration of orientation, color, motion, or higher-level features (20–33). In these studies, a presaccadic probe affected perception of a postsaccadic stimulus at the same spatiotopic location.

Because the oculomotor system executes about three to four saccades per second, spatiotopic updating should operate within a small time window to facilitate perceptual continuity across saccades. Within a single fixation, presaccadic information should be updated and be available directly after the saccade. Concerning the postsaccadic availability, different experiments demonstrated that spatiotopic updating primarily affects perception immediately after saccades (20, 34–36). However, concerning the presaccadic updating of visual information, spatiotopic representations have been estimated to develop surprisingly slowly, requiring fixation durations of more than 500 ms (37–41). This raises a question: If visual processing is fast—content specific representations in 100 ms—and the saccade system is fast—250 ms between two saccades—why is spatiotopic updating slow?

We hypothesized that the apparent slow speed of spatiotopic updating resulted from the nature and interpretation of the psychophysical tasks that have been used. The tilt aftereffect (TAE) is one such example (37, 38), although updating of the TAE is not without controversy (42, 43). The TAE is a perceptual aftereffect where the perceived orientation of a test stimulus is changed after prolonged exposure of another oriented grating, the adapter. When the test stimulus is presented with an orientation away from the adapter, perceptual reports tend to be even farther away from the adapter (44). Because the TAE is a slow process—still increasing in magnitude after 10 min (45)—it might not be a particularly sensitive paradigm to investigate fast visual processing across saccades. To investigate spatiotopic updating, the TAE has been tested in a spatiotopic reference frame where a saccade was made between the presentation of the adapter and the test stimulus. The time course of spatiotopic updating was inferred to take a long time because the TAE increases in strength when saccades are delayed. This increase continues for delays up to 1,000 ms. Similar results were obtained for delayed saccades with saccadic suppression of intrasaccadic displacement (40) and perisaccadic mislocalization (41). However, although the effects were strongest for the longest delays, they were already apparent even for short delays. Finally, it should be noted that, in most transsaccadic experiments, like these with the TAE, two essentially different stimuli are presented before and after the saccade, violating the assumption of a stable, continuous visual world across the saccade. Indeed, psychophysical evidence shows that, when visual stimuli are continuous across saccades, observers perceive the continuity, whereas, if reliable intrasaccadic changes are made to the stimuli, observers expect stimuli to change during a saccade (46). To study visual continuity, the experimental stimulus should also be continuous (47).

To test spatiotopic updating within the time window of 250 ms before saccade onset, we used our recently developed psychophysical, gaze-contingent paradigm (20) with a fast motion illusion: high phi (48). This paradigm allows for the examination of the complete time course of spatiotopic updating. In high phi, subjects see an annulus with a random low-pass filtered texture. This annulus rotates slowly (inducer), after which its texture is sequentially replaced by four different random textures (transient). This creates an illusory transient percept of a large rotational step in the opposite direction from the preceding inducer. Previous experiments with high phi have shown that high phi can be experienced with inducers as brief as 50 ms (Fig. 1B). In our previous study, we observed that it is possible to induce the illusion in a spatiotopic reference frame, when testing with long inducer previews (>500 ms).

Here, we presented an inducer in the peripheral visual field (inducer preview) and asked subjects to make a saccade to the center of the inducer as soon as it appeared, i.e., visually guided saccades. After the saccade, the inducer continued to rotate briefly (postsaccadic inducer), followed by the transient. If the rotational motion of the inducer preview is spatiotopically updated across the saccade, the rotational information of the preview should be added to the rotational information of the postsaccadic inducer, resulting in stronger high phi. Alternatively, if the rotational motion of the inducer preview is not (yet) spatiotopically updated, the strength of high phi is only related to the postsaccadic inducer. To test whether spatiotopic updating can indeed be observed within the temporal regime of visually guided saccades (3), we kept the duration of the inducer preview as long as (experiment 1) or shorter than (experiment 2) the saccade latencies of our subjects. Thus, we were able to dissociate whether spatiotopic updating itself is slow, or whether updating occurs at a shorter time scale but previous paradigms were not sensitive to this fast process.

Results

Rapid Spatiotopic Updating.

In experiment 1, we measured the strength of high phi in four conditions (SI Appendix), two transsaccadic conditions (Fig. 1D) and two additional conditions where subjects maintained fixation to control for a spatial invariant effect (see Control for Spatially Invariant Effect). The direct test for spatiotopic updating is the comparison between the two transsaccadic conditions. In the “saccade preview” condition, subjects were presented the inducer before saccade onset, whereas, in the “saccade static” condition, subjects were presented a static annulus before saccade onset. After the saccade, the annulus rotated briefly for 20 ms or 50 ms in both conditions, followed by the transient. Subjects indicated whether they perceived a large clockwise or counterclockwise step. We analyzed responses with a logistic linear mixed effects model, with condition and postsaccadic inducer duration as fixed effects. The estimated intercept of the model gives the log odds of the transient being reported as a forward rotational step in the saccade preview condition. The other estimated coefficients (β) are relative to this intercept (see Fig. 3A). A negative coefficient indicates a higher probability of perceiving the transient as a backward rotational step.

Fig. 3.

Bootstrapped coefficient estimates of the mixed effects model from experiment 1 (A) and experiment 2 (B). Estimates are obtained with nonparametric bootstrapping (2,000 samples). Error bars represent empirical 95% CI of the coefficient estimates. In experiment (Exp.) 1, the coefficient estimates of saccade static, fixation preview, and fixation static are relative to the saccade preview condition. In experiment 2, the intercept refers to trials with 10 ms of preview and 10 ms of inducer.

Longer durations of the postsaccadic inducer lead to more frequent percepts of backward rotational steps [β = −0.36/10 ms, 95% CI = [−0.41, −0.31], F(1,7957) = 80.98, P < 0.001]. This shows that high phi rapidly increases in strength with longer inducers, similar to the results of previous experiments (20, 48). Importantly, if the inducer is previewed in the periphery before saccade execution (saccade preview, Fig. 1E, yellow solid line), high phi is stronger than in the saccade static condition after the saccade [Fig. 1E, black solid line; β = 0.63, 95% CI = [0.33, 0.91], F(1,7957) = 17.54, P = 0.001]. The preview effect can be interpreted as a spatiotopically transferred effect of the inducer preview: The visual system updated the location of the rotating inducer to a spatiotopic reference frame before the saccade. As a result, the inducer preview and the postsaccadic inducer jointly biased perception after the saccade, inducing a stronger high phi. We estimate that the preview resulted in an approximate 17.5 ms (95% CI = [10.7, 27.3] ms) “head start” in visual processing after saccades with latencies of 150 ms, by taking the ratio of the coefficient of the saccade static condition (β = 0.63) and the coefficient of the postsaccadic inducer (β = −0.36/10 ms). This preview effect generalizes to annuli that cover different and more peripheral portions of the visual field (inner, outer radius = [2.6, 5.0]° and [6.0, 9.25]°), as observed in a control experiment with different subjects (SI Appendix).

Control for Spatially Invariant Effect.

The observed spatiotopic preview effect could potentially be explained by a general, spatially invariant induction of high phi. Such an effect should also be observed without the execution of a saccade. Therefore, we measured high phi in two conditions without saccades, where subjects maintained fixation at the center of the screen and either an inducer (fixation preview) or static annulus (fixation static) was presented in the periphery before the annulus was presented around fixation (SI Appendix, Fig. S1). The results of the fixation preview (Fig. 1E, yellow dashed line) condition demonstrate that a spatially invariant effect cannot fully account for the observed spatiotopic effect, because the illusion was less strong in the fixation preview condition than in the saccade preview condition [β = 0.37, 95% CI = [0.11, 0.63]; F(1,7957) = 10.13, P = 0.006]. However, high phi in the fixation preview condition was slightly stronger than in the fixation static (SI Appendix, Fig. S1A) condition [F(1,7957) = 7.85, P = 0.015]. In short, we observed a limited spatially invariant effect, but this cannot fully account for the transsaccadic preview effect.

Duration of Presaccadic Preview and Strength of Postsaccadic Bias.

In experiment 1, the inducer preview biased postsaccadic perception of the same stimulus when it was presented in the same spatiotopic location. In general, the strength of high phi depends on inducer duration. We examined whether the strength of the preview effect similarly depends on preview duration. In experiment 1, the duration of the inducer preview coincides with saccade latency. We constructed a second mixed effects model, using only data from the saccade preview condition. Preview duration and postsaccadic inducer duration were fixed effects, and we included random effects per subject for the fixed effects and inducer rotation direction. We compared this model to a null model without a fixed effect for preview duration. Preview duration did not improve the model fit [χ² (1) = 0.82, P = 0.36], so it seems that the preview effect was not modulated by preview duration. However, if the preview effect is perceptual in nature, it should be related to the strength of the preview. To test the limits of the preview effect, in experiment 2, we uncoupled preview duration and saccade latency for even shorter preview durations than in experiment 1.

In experiment 2, each preview consisted of a mixture of a static annulus followed by an inducer preview (Fig. 2A and SI Appendix). The data were analyzed with a mixed effects model, with fixed effects for preview duration and postsaccadic inducer duration and random effects per subject. The model with preview duration as a fixed effect was a better fit for the data than the model without it [χ² (1) = 8.99, P = 0.003]. In this model, a longer preview duration results in more frequent percepts of a backward step [Fig. 2B; β = −0.05/10 ms, 95% CI = [−0.07, −0.02], F(1, 3799) = 13.99, P < 0.001]. In addition to the effect of the inducer preview, the postsaccadic inducer also induced a strong bias, similar to experiment 1 [β = −0.30/10 ms, 95% CI = [−0.35, −0.25], F(1, 3799) = 91.90, P < 0.001]. The estimated coefficients are displayed in Fig. 3B. In sum, in both experiment 1 and 2, we observed spatiotopic updating within 150 ms after stimulus onset. Moreover, the duration of the preview increases the strength of the spatiotopic effect.

Fig. 2.

Experiment 2, design and results. (A) Subjects fixated a fixation target for 250 ms to 500 ms. An annulus appeared in the periphery. The annulus remained static for 0, 50, 100, or 150 ms, and then started rotating. The annulus continued to rotate throughout the saccade and 20 or 50 ms after (postsaccadic inducer). If subjects moved their eyes before the annulus started rotating, it started rotating when gaze was detected >3° away from the fixation target. After the postsaccadic inducer, the texture of the annulus was replaced by four different, random textures (20 ms per texture). Subjects indicated whether they perceived the change in textures as a step in CW or CCW direction. Responses were recoded to “backward” and “forward” with respect to the rotation direction of the preceding inducer. (B) Estimated perceived step direction from the mixed effects model as a function of inducer preview (y axis) and postsaccadic inducer (x axis). Brighter colors indicate more frequent percepts of backward steps. The range of the colormap goes from 0.25 to −0.75 to optimize color contrasts for the range of plotted values.

Discussion

We examined spatiotopic updating of visual information across saccades. The current experiments demonstrate a fast updating mechanism in the visual system that predictively influences perception after an eye movement. We observed a direct link between postsaccadic perception and the strength of the presaccadic stimulus for stimuli that covered the parafovea after a saccade; in a control experiment (SI Appendix), we also observed this link for larger stimuli (inner, outer radius = [6.0, 9.25]°), in the same eccentricity range typically used in spatiotopic updating experiments (∼5° to 10° in the periphery). The time scale on which this link is established is compatible with typical fixation durations observed in natural viewing (3) and represents a behavioral index of spatiotopic updating expressed as a perceptual bias in the direction of the presaccadic visual information, comparable to a 17.5-ms head start in visual processing.

The current study differs in two important aspects from the studies with tilt adaptation to assess the time course of spatiotopic updating (37–39). First, the stimulus we used to assess spatiotopic updating is fast in nature. High phi can be induced in the order of tens of milliseconds, whereas tilt adaptation is typically induced in the order of hundreds of milliseconds (45). Second, the stimulus feature that had to be updated (inducer rotation direction) was stable and continuous across saccades, enabling the assessment of perceived visual continuity in an environment where the assumption of continuity across saccades is true (12, 46).

Rapid spatiotopic updating is plausible when considering the speed of processing in the human visual system, which contains stimulus specific representations rapidly after stimulus onset—in the order of 100 ms—as demonstrated in psychophysical studies (5, 6) and neuroimaging studies (4). This rapidly acquired information is used by the visual system to predict the sensory changes induced by saccades. It facilitates postsaccadic visual processing by anticipating the postsaccadic retinal input based on presaccadic input (49). Three fMRI studies support this idea by showing spatiotopic and feature-specific repetition suppression (50–52). Repetition suppression in neurophysiological measures is observed when the same stimulus is presented twice (53). Hence, repetition suppression in spatiotopic coordinates can be interpreted as a neurophysiological measure of the visual system regarding the postsaccadic stimulus to be “the same” as the presaccadic stimulus, even though it was presented at different retinotopic coordinates. Although these effects are in line with the current findings, the time scale of fMRI studies is limited by the slow blood–oxygen level-dependent response. Interestingly, a recent EEG study provides more direct neurophysiological correlate of our behavioral findings (54). Edwards et al. (54) used time-resolved decoding of a postsaccadic stimulus while varying the correspondence between the presaccadic and postsaccadic stimuli. The postsaccadic stimulus could be decoded faster when it matched the presaccadic stimulus than when it was different from the presaccadic stimulus. This indicates that information about the presaccadic stimulus affects the neural responses to the postsaccadic stimulus in a way that suggests more efficient processing when the two stimuli match. The current results show that this fast facilitation in postsaccadic visual processing is not only reflected in neurophysiological measures but can be quantified in human behavior.

Still, although we observed spatiotopic updating on a short time scale, we would not generalize the results to all stimuli in the visual field. The reason for this caution is that, while there is ample evidence in favor of spatiotopic updating of visual information, there are also studies that fail to observe this with either behavioral measures (42, 43, 55) or fMRI (56). One important restriction on spatiotopic updating seems to be that it is limited to attended stimuli; passive visual stimulation does not automatically result in spatiotopic updating (47, 57). The introspective feeling of visual continuity thus could arise from a match between the predicted postsaccadic retinal image and observed retinal image of an attended stimulus (49, 58).

Predicting upcoming stimuli is a fundamental characteristic of the brain, as stated by theories of predictive coding (59). Anticipating the consequences of an upcoming saccade is a frequently recurring example of a scenario where the principles of predictive coding are applied (60–62). This anticipation could be implemented as a forward model (63), where a corollary discharge from the oculomotor system enables the dissociation between internal and external changes in retinal input (64). Here, we observed effects of a spatiotopic prediction on postsaccadic perception within the temporal regime of the typical latencies of visually guided saccades. With these findings, rapid spatiotopic updating of visual information is a plausible mechanism that contributes to perceptual continuity across saccades in natural viewing.

Methods

Subjects.

Fifty-two subjects (age: M = 22.6, range = [18, 37], 26 female) with normal or corrected-to-normal acuity participated after giving written informed consent (n = 20 in experiment 1, n = 12 in experiment 2, n = 20 in SI Appendix, Control Experiment). The sample size of experiment 1 was based on the effect sizes of our previous study with high phi (20). The sample size in experiment 2 was lower because we planned to make fewer statistical comparisons with fewer experimental conditions. This study was approved by the local ethical committee of the Faculty of Social Sciences of Utrecht University. All subjects were naïve to high phi before the experiments and completed a screening procedure (SI Appendix, Experimental Procedures) to ensure they could reliably report the motion direction of a rotating annulus. Moreover, we verified whether subjects perceived backward steps with high phi after a long inducer (500 ms; SI Appendix, Experimental Procedures and Fig. S2). One subject was excluded from the dataset of experiment 1 because of a failure to meet this criterion (SI Appendix, Data Analysis).

Setup.

Stimuli were displayed on a 48.9° by 27.5° Asus RoG Swift PG278Q, an LCD twisted nematic monitor with a spatial resolution of 52 pixels per degree, and a temporal resolution of 100 Hz (AsusTek Computer Inc.). The Ultra Low Motion Blur backlight strobing option of the monitor was enabled (maximum pulse width) for higher temporal precision (65). Eye position of the left eye was recorded with an Eyelink 1000 at 1,000 Hz (Sr Research Ltd.). The eye tracker was calibrated using a nine-point calibration procedure. All stimuli were created and presented in Matlab 2016a (The Math Works, Inc.) with the Psychophysics Toolbox 3.0 (66) and the Eyelink Toolbox (67). Visual onsets and eye movement data were synchronized using photodiode measurements (SI Appendix, Data Analysis).

Stimuli.

Stimuli were annuli (inner radius ≈ 3°, outer radius ≈ 6°) with random grayscale textures, created by low-pass filtering random black (0.09 cd/m²) and white (88.0 cd/m²) pixels with a pillbox average (radius = 1.24°). For rotating annuli, the rotational velocity was 20°/s. Fixation targets were black dots (radius ≈ 0.2°) with a gray point in the center (radius ≈ 0.075°). All stimuli were presented on a uniform gray background (44.1 cd/m²). We tested the spatial generalizability of the preview effect observed in experiment 1 by repeating the saccade conditions using stimuli with different radii (SI Appendix).

Analysis.

Before the statistical analysis, eye movement data were preprocessed (SI Appendix, Data Analysis) and visual onsets were aligned to the eye movement data based on photodiode measurements (SI Appendix, Data Analysis and Fig. S5). We analyzed the perceived step direction (i.e., the probability of a “forward step” response: p_forward) with a logistic linear mixed effects model (68). $p_{forward}=2/(1+{\text{e}}^{-(\text{X}\beta +\text{Zy})})-1$, where X is the design matrix, β is a vector with the fixed effects coefficients, Z is the random effects design matrix, and y is the random effect coefficients. All estimates of fixed effects coefficients are reported relative to the intercept condition, here the saccade preview condition with an inducer of 10 ms (Fig. 1D). In experiment 1, the mixed effects model contained fixed effects of inducer duration and condition, and random effects of inducer duration, condition, and inducer rotation direction per subject (SI Appendix, Statistics Experiment 1). Condition was modeled as a categorical variable, and inducer duration was modeled as a continuous variable. We only allowed inducer durations between 10 ms and 60 ms. We did not include the interaction between condition and inducer duration, because a model comparison showed that, all other things kept equal, the interaction did not improve the model [χ²(3) = 4.16, P = 0.245]. We compared conditions with planned contrasts. Reported P values for planned contrasts are corrected with the Holm−Bonferroni method (69). In experiment 2, the model contained fixed effects for presaccadic inducer duration and postsaccadic inducer duration, and random effects of presaccadic inducer duration, postsaccadic inducer duration, and rotation direction per subject (SI Appendix, Statistics Experiment 2). Both inducer durations were modeled as continuous variables. We used nonparametric bootstrapping to obtain 95% CI of the estimated fixed effects coefficients. Two thousand bootstrap samples were constructed by stratified sampling from the original dataset, with stratification according to the fixed effects but not the random effects. Trials were sampled with replacement. Bootstrapped coefficient estimates and 95% CI are displayed in Fig. 3. Individual variations across these estimates are displayed in SI Appendix, Fig. S3.

Saccade Latencies.

We set out to investigate spatiotopic updating across saccades’ unconstrained latencies. Saccade latencies in natural viewing conditions are typically around 250 ms (3). In experiment 1, the average median saccade latency was 146 ms (range = 111 ms to 177 ms across subjects). In experiment 2, the average median saccade latency was 136.8 ms (range = 112 ms to 178 ms across subjects).

Data Availability.

All scripts and data are publicly available at Open Science Framework (70).