Consequences of the oculomotor cycle for the dynamics of perception

Summary

Much evidence indicates that humans and other species process large-scale visual information before fine spatial detail. Neurophysiological data obtained with paralyzed eyes suggest that this coarse-to-fine sequence results from spatiotemporal filtering by neurons in the early visual pathway. However, the eyes are normally never stationary: rapid gaze shifts (saccades) incessantly alternate with slow fixational movements. To investigate the consequences of this oculomotor cycle on the dynamics of perception, we combined spectral analysis of visual input signals, neural modeling, and gaze-contingent control of retinal stimulation in humans. We show that the saccade/fixation cycle reformats the flow impinging on the retina in a way that initiates coarse-to-fine processing at each fixation. This finding reveals that the visual system uses oculomotor-induced temporal modulations to sequentially encode different spatial components and suggests that, rather than initiating coarse-to-fine processing, spatiotemporal coupling in the early visual pathway builds on the information dynamics of the oculomotor cycle.

Introduction

A popular view of the early visual system compares the eye to a camera, where the camera lens represents the eye’s optics and the film acts as the retina. Aside from obvious simplifications, there are two important aspects in which this analogy falls short. First, unlike the film in a camera, our retina and visual system are primarily sensitive to temporal modulations [1–5], to the point that even gross changes in the scene escape detection when they occur too slowly [6]. Therefore, while a camera needs a stationary image on its film to operate optimally, we see best when the input to our retina varies with time. Second, while a photographer usually takes great care to ensure that the camera does not move, our eyes are never at rest.

Two major types of eye movements continually alternate during examination of a static scene. Every few hundreds of milliseconds, saccades rapidly shift gaze toward new regions of interest [7]—sometimes even by very small amounts (microsaccades) [8,9]—enabling examination of interesting stimuli with the foveola, the retinal region with highest acuity. In the “fixation” periods in between saccades, the eyes jitter incessantly following seemingly erratic trajectories which resemble Brownian motion [10,11], a behavior often termed ocular drift [12]. Although not always considered in studies of visual functions, eye movements are always present and, thus, intertwined with the neural processing stages that encode visual information.

To date, most studies of this interaction focus on the challenges that eye movements pose to the establishment of visual representations [7,13,14]. In contrast, considerably less work has been dedicated to the analysis of a possible beneficial consequence of oculomotor activity, the reformatting of the visual input in the joint space-time domain [15,16]. By modulating the signals experienced by retinal receptors, all eye movements transform the spatial pattern of luminance into a spatiotemporal flow on the retina. This transformation converts spatial variations into temporal changes, a format that neurons in the visual system greatly prefer [17,18]. Because of their very different characteristics, saccades and drifts deliver signals with correspondingly different spatiotemporal statistics to the retina. Thus, the actual input to the visual system strongly depends on the observer’s eye movements.

These considerations raise the interesting hypothesis that eye movements may enable selection of information not just in the spatial domain, by reorienting the fovea, but also in the temporal domain, by reformatting luminance signals so to deliver selected aspects of spatial information within the temporal range of neuronal sensitivity. In previous research, we have shown that ocular drift yields a specific, beneficial frequency equalization during viewing of natural scenes [19]. However, little is known about how saccades reformat the retinal stimulus and about the effects of the normal alternation between rapid/large and slow/small eye movements. What are consequences of the saccade/fixation cycle for retinal stimulation? Do these stereotypical input changes affect perceptual dynamics? And if so, what do these effects imply for the encoding of spatial information? Here we show that this oculomotor cycle plays a key functional role: it initiates a coarse-to-fine processing sequence at each visual fixation.

Results

Retinal input dynamics and modeling predictions

We started by recording oculomotor activity while human observers freely examined pictures of natural scenes (Fig. 1A). As is typical, subjects frequently relocated their gaze by means of saccades (average inter-saccadic duration ± standard deviation: 264 ± 78 ms), which spanned a wide range of amplitudes, from a fraction of a degree to several degrees (mean amplitude 4.4° ± 1.3°), and reached velocities as high as 900°/s. As expected from previous studies [7,12,20], the eye remained in motion in the periods in between saccades, when it drifted with apparently random trajectories at average speed of almost 1° /s (Fig. 1 B). Thus, eye movements continually transformed spatial information into temporal modulations on the retina, yielding motion signals that would be immediately visible had they originated from the motion of objects in the scene rather than the observer’s eye movements. This recurrent alternation between macroscopic and microscopic eye movements results in a stereotypical sequence of retinal stimulation, with the brief abrupt changes caused by saccades at fixation onset followed by the slower luminance changes caused by ocular drifts as fixation progresses (Fig. 1C).

Figure 1. Natural input to the retina and predicted neural dynamics.

(A) Normal alternation between saccades (yellow segments) and periods of fixational eye movements (red circles) during viewing of natural scenes (image size: 17° × 12. 8°). The size of each circle represents fixation duration. A portion of the eye movement trace (orange) is expanded on the right panel (top), together with the luminance signal experienced by a retinal receptor (bottom). Eye movements continually modulate the input flow onto the retina, effectively redistributing spatial information in the joint space-time domain. (B) Mean instantaneous speed of the retinal image (note log scale) and (C) mean rate of change in the input luminance experienced by a foveal receptor during a fixation-saccade-fixation sequence. Time zero marks saccade end. (D–F) Neural modeling. (D) The responses of model V1 simple cells were simulated as their receptive fields moved following recorded oculomotor traces. (E) Models consisted of rectified filters with separable spatial (top) and temporal (bottom) kernels. (F) Responses of neurons tuned to low (1 cycles/deg, blue line) and high spatial frequency (10 cycles/deg, red line). Solid lines represent medians of activity. Shaded regions enclose the range between the first and third quartile.

We examined the impact of this time-varying input on the responses of modeled simple cells in the primary visual cortex (V1). Models were exposed to the same visual input experienced by the subjects in our experiments, i.e., the spatiotemporal luminance signals impinging onto the retinas given the recorded sequences of eye movements (Fig. 1D, E). We simulated the mean instantaneous firing rates of neurons with peak sensitivity at either low (1 cycles/deg) or high (10 cycles/deg) spatial frequencies. Apart from the difference in spatial frequency tuning, these two neuronal populations were identical; yet they responded very differently during the course of post-saccadic fixation (Fig. 1F). At fixation onset, the temporal transient caused by the preceding saccade elicited a strong response in units tuned to low spatial frequency, but had a modest impact on high-frequency neurons. In contrast, later in fixation, once the effect of the preceding saccade had terminated, the two sets of neurons gave comparable responses, even though the images presented to the observer contained approximately 100 times more power at low than high spatial frequencies. Thus, modeled neural responses transiently followed the power distribution of natural images immediately after a saccade, and then settled into a more balanced regime with equal responses in different frequency channels during the course of fixation.

To understand the factors responsible for these neural dynamics, we examined the frequency content of the visual input transients resulting from different types of eye movements. When a static scene is observed in the complete absence of oculomotor activity, the input to the retina is a static image, so its power is confined to 0 Hz, the temporal DC frequency axis (Fig. 2A). Eye movements transform this static input into a spatiotemporal stimulus on the retina, redistributing its power across nonzero temporal frequencies in a way that depends on the specific characteristics of eye movements (Fig. 2B). Since drifts and saccades move the eyes in very different ways, one may expect considerably different power redistributions driving neural responses during the early and late phases of post-saccadic fixation.

Figure 2. Power spectra of the retinal input signals resulting from different types of eye movements.

(A, B) Eye movements transform spatial information in the scene into spatiotemporal modulations on the retina, redistributing the power at 0 Hz of a static scene (A) across non-zero temporal frequencies (B). The characteristics of this transformation depend on the type of eye movements. (C, D) Spectral density of the visual input present during fixational drift. Both the full spatiotemporal distribution (C) and sections at individual temporal frequencies (D) are shown. Sections in D are compared to the ideal 1/k² power spectrum of natural images (Natural). During viewing of natural scenes, fixational drift equalizes power across a broad range of spatial frequencies. (E, F) Spectral density of the visual input given by saccades. Unlike ocular drift, saccades yield modulations that follow the spectral density of natural images over a wide range of temporal frequencies.

We first examined the spatiotemporal power redistribution present during late fixation, when input signals were only modulated by ocular drift (Fig. 2C, D). We have recently shown that the amount of DC power that drifts convert into nonzero temporal frequencies increases with spatial frequency in a way that counterbalances the spectral density of natural scenes [19], effectively a form of matching between the characteristics of eye movements and those of the natural world. Because of this interplay, ocular drift produces a striking spatiotemporal reformatting of the input to the retina during viewing of natural scenes. Unlike the power-law spectral density of the images displayed on the monitor [21] (black line in Fig. 2D), the temporal modulations resulting from ocular drift contained equal power over a broad range of spatial frequencies (Fig. 2C, D), an effect known as spectral whitening.

The abrupt input changes caused by saccades possessed very different spectral densities (Fig. 2E, F). Unlike drift, saccades converted spatial power into temporal power at a similar rate across a broad range of spatial frequencies. Therefore, saccade transients preserved the uneven spectral distributions of the scenes presented to the observers: at all relevant nonzero temporal frequencies, the power of the visual input to the retina declined with the square of spatial frequency, like the spectrum of natural images. Because of this effect, the total temporal power (the sum across all nonzero temporal frequencies) transiently delivered by saccades was larger than that resulting from ocular drift at low spatial frequencies, and the two become comparable around 10 cycles/deg.

The spectral redistributions in Fig. 2 explain the dynamics of neural activity in our model (Fig. 1F). They show that the normal alternation between saccades and fixational drift sequentially exposes neurons to modulations that contain different kinds of spatial information during the course of normal post-saccadic fixation. At fixation onset, because of the integration memory of the neuronal firing (the temporal sensitivity profile in Fig. 1E), neurons are effectively stimulated by an input signal dominated by low spatial frequencies, as it occurs in natural images. Later during fixation, ocular drift enhances high spatial frequencies, and the same neurons are now driven by a signal in which power has been equalized across spatial frequencies.

To make quantitative predictions of the perceptual consequences of these retinal dynamics, we modeled the post-saccadic course of contrast sensitivity in a standard detection experiment. Our modeled V1 neurons were now exposed to a grating embedded within a natural noise field, as their receptive fields translated following recorded sequences of eye movements (Fig. 3A). A standard decision-making model continually accumulated the evidence provided by neural activity during post-saccadic fixation and determined contrast thresholds by comparing the mean instantaneous firing rates simulated in the presence and absence of the grating (Fig. 3B).

Figure 3. Predicted dynamics of contrast sensitivity.

(A) The responses of V1 neurons were modeled in a detection task with gratings embedded in a naturalistic noise field (top). Models were tuned to the grating’s orientation and spatial frequency (either 1 or 10 cycles/deg), and their receptive fields translated following sequences of recorded eye movements (two fixations separated by a saccade; bottom). (B) At each time during post-saccadic fixation, a standard decision-making model determined the presence or absence of the grating based on the cumulative neural response (averaged across all simulated cells) from fixation onset (η(t); top). The contrast yielding a hit rate of 0.75 and false alarm rate of 0.25 (d′ = 1.35) was selected as threshold (bottom). (C) Average responses with or without the grating at 1 (top) and 10 cycles/deg (bottom). Shaded regions mark the post-saccadic period of fixation considered by the model in B. The darker shade represents the period of early fixation in which the preceding saccade influences neural responses. Note how responses to the high spatial frequency target remain well-separated from the noise response not only in early fixation, but also in late fixation. (D) Schematic representation of the impact of different types of eye movements on visual detection. For any given contrast, the separation between the distributions of cell responses in the presence and absence of the grating transiently increases following saccades at low spatial frequencies, but remains more similar throughout the course of fixation at high spatial frequencies. (E) Model predictions. With a 1 cycle/deg grating, contrast sensitivity does not increase beyond the early-fixation level provided by the saccade transient. Sensitivity to 10 cycles/deg improves more gradually, as it also weights the amplified response that persists during late fixation because ocular drift.

In keeping with the spectral analysis in Fig. 2, saccades and ocular drift yielded different modulations during exposure to noisy gratings at 1 and 10 cycles/deg. These modulations effectively varied the Signal-to-Noise Ratio (SNR) in the input to the decision-making stage, as they changed the level of stimulus-driven response relative to the fixed internal noise of the model. Responses to a 1 cycles/deg grating were clearly distinct from those generated by the noise field alone only at fixation onset (early fixation; Fig. 3C, top). In contrast, because of the high-frequency enhancement given by ocular drift, neural responses at 10 cycles/deg remained distinguishable from noise during both early and late fixation (Fig. 3C, bottom). In fact, for these neurons, the separation between the responses with and without the target was comparably large in early and late fixation. Thus, at 1 cycle/deg, detection was greatly facilitated immediately after a saccade and did not benefit from integration of firing rates past early fixation, whereas at 10 cycle/deg continued temporal integration was beneficial, as it further separated the response distributions elicited in the presence and absence of the grating (Fig. 3D). Because of these effects, modeled sensitivity remained virtually constant at low spatial frequencies during the course of fixation, but continued to increase at high spatial frequencies (Fig. 3E).

These results suggest distinct perceptual contributions from different types of eye movements. Three specific predictions emerge: (1) the transients introduced by saccades are primarily responsible for vision at low spatial frequencies, a process that occurs rapidly following fixation onset; (2) the modulations resulting from ocular drift continually enhance high spatial frequencies at later times during post-saccadic fixation; (3) more broadly, the normal alternation between saccades and fixational drift gives rise to a stereotypical time-course of visual perception, leading to an oculomotor-induced coarse-to-fine analysis of the scene.

Experimental validation of theoretical predictions

We tested our predictions by assessing the consequences of the normal saccade-drift cycle on visual sensitivity in human observers. We first determined whether, following a saccade, spatial vision evolves as predicted by Fig. 3E. In two further experiments, we then separately analyzed the contributions of ocular drift and saccades by selectively eliminating their input modulations.

Although eye movements are known to both suppress and enhance visual functions [22,23], standard laboratory measurements of contrast sensitivity typically ignore or attempt to minimize the effects of the luminance changes resulting from oculomotor activity. As a consequence, little is known about visual dynamics during normal inter-saccadic fixation. As in the simulations of Fig. 3, we measured the post-saccadic evolution of contrast thresholds at low and high spatial frequencies (1 and 10 cycles/deg, respectively; Fig. 4C). To maintain control of the spatiotemporal stimulus impinging onto the retina, while ensuring foveal exposure to the normal modulations resulting from the saccade/fixation cycle, we synchronized stimulus delivery to eye movements and tested sensitivity at different times (100 and 800 ms) after a saccade had swept the retina across a naturalistic noise field (Fig. 4A–B). In keeping with our predictions, sensitivity to low spatial frequencies changed little during the course of fixation: after 800 ms exposure, thresholds measured at 1 cycle/deg were similar to those measured after only 100 ms (p > 0.24, paired two-tailed t-test; Fig. 4D). In contrast, prolonged stimulus exposure was beneficial at high spatial frequencies, leading to a more-than-twofold elevation of sensitivity at 10 cycles/deg (p < 0.027), an effect present in each individual observer (p ≤ 0.043; parametric bootstrap [24]). Thus, performance improved over the course of fixation at high but not low spatial frequencies. These results confirm the general prediction of our model: visual perception follows a coarse-to-fine dynamics during normal post-saccadic fixation over a naturalistic field.

Figure 4. Post-saccadic dynamics of contrast sensitivity.

(A–B) Experimental procedure. (A) Subjects reported the orientation (±45°) of a grating at either 1 (low frequency; top) or 10 cycles/deg (high frequency; bottom) embedded in a full-screen naturalistic noise field (1/k² spectrum; shaded triangle). (B) Subjects initially fixated ~7° away and performed a saccade toward the future grating location at the appearance of a cue. The stimulus appeared during the saccade and was displayed for either 100 or 800 ms following fixationonset. (C) Contrast sensitivity (N=4). Sensitivity increased with longer stimulus exposure for high but not low spatial frequencies. (D) Relative change in sensitivity with exposure duration. In both C and D, colored symbols are individual subject data, black symbols are group means, and error bars represent SEM. Asterisks mark statistically significant differences (p < 0.03; paired two-tailed t-test).

Having established the presence of the predicted post-saccadic dynamics of sensitivity, we examined whether these dynamics were driven by the luminance modulations caused by eye movements, as predicated by our theoretical analyses. To specifically test the impact of ocular drift and saccades at individual spatial frequencies, we repeated the experiment of Fig. 4 with presentation of isolated gratings and manipulated the oculomotor modulations of retinal input signals by means of gaze-contingent display control procedures.

To isolate the influence of ocular drift, we compared contrast thresholds measured at late fixation (800 ms post-saccadic exposure) in the presence and absence of retinal image motion. It is known that high-acuity discriminations are impaired under retinal stabilization [25,26]. These previous studies, however, did not separate the possible contributions of different types of fixational eye movements—ocular drift and small saccades (microsaccades)—, and it has been recently demonstrated that microsaccade centering of the stimulus within the foveola facilitates fine spatial vision [27]. Here, we discarded all trials containing microsaccades and specifically focused on the impact of the luminance modulations caused by ocular drift. To selectively eliminate drift fluctuations while preserving the initial saccadic transient, we stabilized the stimulus on the observer’s retina at the detection of saccade end. That is, following the initial saccade, we continually adjusted the position of the stimulus on the display according to the subject’s eye movements, so that it remained at a fixed position on the retina [26,28].

Retinal stabilization acted differently on the time-course of sensitivity at high and low spatial frequencies (Fig. 5B). As predicted, drift cancellation had minimal impact at low spatial frequencies: thresholds measured at 1 cycle/deg were very similar and statistically indistinguishable during normal viewing and under retinal stabilization (p >0.43). In contrast, removal of retinal image motion greatly affected sensitivity at high spatial frequency, attenuating the improvement measured during the course of normal unstabilized fixation by about 40% in both observers. These results confirm our modeling prediction that continual post-saccadic integration of drift luminance modulations selectively enhances high-frequency sensitivity as fixation progresses.

Figure 5. Perceptual consequence of fixational drift.

(A) Experimental procedure. Following the initial saccade, stimuli were either displayed at a fixed location on the monitor (Normal) or at a fixed location on the retina (Stabilized). In this latter condition, stimuli moved on the screen under real-time control to compensate for the subject’s eye movements. (B) Experimental results. Contrast sensitivity in the two viewing conditions at 1 (low) and 10 cycles/deg (high). Eliminating drift modulations impairs sensitivity at high (left panel), but not low (right panel), spatial frequencies. Each graph shows data from two observers. Error bars represent 95% confidence intervals. (*) p = 0.0035; (**) p=0.0005; parametric bootstrap test.

To isolate the influences of the luminance transients caused by saccades, we relied on gaze-contingent control of stimulus contrast. As in the previous experiments, subjects performed a saccade toward a cued location. Here, however, the abrupt change in the retinal input resulting from the saccade (Normal condition) was replaced by a gradual increment in the contrast of the grating timed with saccade landing (No-transient condition). That is, the 800 ms period of fixed-contrast stimulus exposure in the experiments of Figs. 4 and 5 was here preceded by a 1.5 s contrast ramp starting at fixation onset (Fig. 6A). Thus, in the no-transient condition, subjects were exposed to the stimulus for much longer time, receiving almost the double of the energy experienced in the normal condition, but were deprived of saccadic transients.

Figure 6. Perceptual consequences of saccade transients.

(A) Experimental procedure. The stimulus was either displayed during the saccade (Normal condition) or gradually increased in contrast after saccade end (No-transient condition). The graphs show the dynamics of stimulus contrast in the two conditions. (B) Experimental results. Contrast sensitivity in the presence and absence of saccade transients for gratings at 1 and 10 cycles/deg. Small colored symbols are individual subject data, large gray and black symbols are group means. Error bars represent SEM. (*) p <0.02; paired two-tailed t-test; N=4.

As predicted, the consequences of saccades were complementary to those of fixational drift (Fig. 6B): elimination of the saccade transient impaired sensitivity in each individual observer (p ≤ 0.0035 ; parametric bootstrap). On average across observers, removal of the saccade transient lowered sensitivity by approximately 40% at 1 cycle/deg (p=0.017; two-tailed paired t-test), but had little impact at 10 cycles/deg (p=0.93), resulting in a much larger change in sensitivity at low spatial frequency than high spatial frequency (p = 0.047; two-tailed paired t-test). In other words, at low spatial frequencies, saccades cannot be traded for ocular drifts by increasing the duration of fixation, as drift modulations are not beneficial in this range regardless of their duration. These findings confirm our theoretical prediction that saccade transients are critical for low spatial frequency vision.

Discussion

Species with a high-acuity fovea, like humans and other primates, redirect their gaze to examine objects of interest, but also continue to move their eyes during the post-saccadic periods in which visual information is acquired and processed. Our results show that this behavioral cycle converts spatial information into temporal modulations in a way that generates a coarse-to-fine dynamics of visual analysis within each individual fixation. This transformation bears fundamental implications for the way space is represented in the visual system and can be regarded as a primary function of oculomotor activity. Eye movements enable selection of visual information not just in the spatial domain, by properly redirecting the fovea, but also in the temporal domain, by determining what aspects of the visual scene fall within the temporal range of neuronal sensitivity.

The evidence presented in this study comes from three distinct components. First, spectral analyses of signals impinging onto retinal receptors showed that saccades and drift redistribute the power of the external stimulus in complementary ways on the retina. Second, modeling of V1 neurons predicted different consequences of these two regimes on visibility, with low spatial frequencies emerging rapidly after a saccade and high spatial frequencies benefiting from continued integration during the course of fixation. Third, psychophysical experiments validated these predictions by controlling retinal stimulation during the saccade/fixation cycle in human subjects. We found that, as predicted by our model, the evolution of contrast sensitivity during natural post-saccadic fixation exhibits oculomotor-dependent phasic and tonic dynamics at low and high spatial frequencies, respectively. Note that in everyday viewing, these dynamics occur at each fixation and follow the reduction in visibility and the spatial updating phenomena present before and during saccades [29,30].

A large body of evidence supports a coarse-to-fine hierarchy of visual analysis [31–35]. However, little consideration has been given to the temporal characteristics of the retinal stimulus itself, which results from the interaction of largely static visual scenes with the dynamics of eye movements. Our results indicate that this interaction is a critical component in establishing the coarse-to-fine processing sequence. In the laboratory, where observers are often required to maintain fixation, sharp stimulus onsets commonly replace saccades. Under natural viewing conditions, however, the saccade-fixation cycle is the primary source of temporal modulations. Note that, because of the impossibility of completely eliminating oculomotor transients, both sets of data in Figs. 5 and 6 likely underestimate the real effects of eye movements. This happens because also the residual temporal modulations caused by imperfections in retinal stabilization in Fig. 5 and by the contrast ramp in Fig. 6 are expected to emphasize high and low spatial frequencies, respectively. Similarly in Fig. 4, while our experiments provided natural stimulation in the fovea, the sharp peripheral onset of the stimulus during the saccade might have enhanced performance. Thus, eye movements may normally exert even stronger influences than those revealed by our data.

These findings provide critical insights into the strategy followed by the visual system to represent space. While the spatial format of neural representations (e.g., retinotopic vs. spatiotopic [14,36]) has been intensely debated, considerable less attention has been dedicated to the multidimensional reformatting of visual input signals given by eye movements, i.e., the conversion of spatial patterns into temporal modulations on the retina [15,16]. Our findings show that the visual system takes advantage of the stereotypical evolution in the spectral redistribution of the input flow to sequentially represent space in the temporal domain during natural post-saccadic fixation. That is, spatial information is not just encoded by the position of neurons within maps, but also by the temporal structure of neural responses, a dynamics governed by oculomotor activity and, in large part, under the observer’s control. Saccades and drifts can be regarded as two fundamental computational elements of this strategy, as they deliver useful temporal modulations in complementary spatial frequency ranges.

This view implies important mechanisms for decoding visual signals beyond those commonly postulated. Since neural firing in thalamic afferents carries distinct kinds of information during early and late fixation, the input to the cortex needs to be interpreted according to the time elapsed from fixation onset. Neural responses immediately following a saccade are primarily driven by low spatial frequencies and presumably contribute to perception of the general structure of the scene. Later responses from the same neurons will in contrast be driven by a whitened input, and will likely signal luminance discontinuities and finer scales. These considerations support a fundamental role for the magnocellular pathway in the representation of the gist of a scene [37,38] via its saccade-induced response, establishing a scaffolding for the later addition of fine spatial detail during fixational eye movements.

Our results also provide a functional explanation for the different dynamics of neurons sensitive to low and high spatial frequencies in the early visual system. It is well established that magnocellular and parvocellular neurons in the retina and lateral geniculate nucleus tend to prefer separate spatial frequency ranges, with magnocellular neurons sensitive to lower frequencies than parvocellular ones [39]. Notably, these neurons also exhibit different temporal preferences: magnocellular neurons tend to be more responsive to temporal transients than parvocellular neurons [40]. But why should neurons sensitive to separate spatial frequency ranges also differ in their temporal characteristics? The results in Fig. 3 suggest an answer to these questions. In a system designed to encode space in time—i.e., to operate at non-zero temporal frequencies—, analysis of low, but not high, spatial frequencies tends to be more reliable immediately following a saccade than later during fixation. In other words, an observer that optimally accrues spatial information from the oculomotor-shaped retinal input flow should: (a) heavily rely on saccade transients to extract low spatial frequencies; and (b) continue integrating throughout the course of fixation to extract high spatial frequencies. Thus, the incessant presence of eye movements implies distinct strategies for efficiently acquiring information in different frequency ranges, and the spatiotemporal characteristics of magnocellular and parvocellular neurons are well suited to carry out these functions.

The previous observations suggest that several neurophysiological findings with anesthetized and paralyzed animals need to be reevaluated in the light of their implications for natural viewing conditions. Various studies have observed temporal changes in the responses of cortical neurons consistent with a progressive refinement of visual sensitivity, such as a sharp increase in preferred spatial frequency after stimulus presentation [41–43]. These cortical changes, which likely originate from delays in thalamic afferents [44], are believed to represent the neural mechanisms responsible for a coarse-to-fine perceptual dynamics. However, the view emerging from our study is that these responses constitute a direct consequence of a system optimally tuned to its input, once eye movements are taken into account: spatiotemporally-coupled neural dynamics are not the primary cause for a coarse-to-fine evolution, but are necessary to operate synergistically with oculomotor activity and efficiently extract spatial information from oculomotor transients. We predict that a coarse-to-fine perceptual course would still occur even in the absence of these neural delays as a consequence of the normal saccade/drift cycle, as demonstrated by our model in which all simulated neurons possessed identical temporal profiles.

Studies of visual perception often focus on the costs associated with gaze shifts, and how the visual system avoids their potentially disruptive consequences. Here, we show that these gaze shifts have computational advantages. We have shown that the incessant alternation between macroscopic and microscopic eye movements leads to a dynamic reformatting of the visual flow which enables encoding of space in the temporal domain. This space-time conversion acts as a first processing stage that operates together with the response selectivity of retinal neurons to implement a hierarchical scheme of visual analysis. An interesting hypothesis now emerges. In principle, control over eye movements could go beyond foveation to select the scale of visual processing and emphasize task-relevant ranges of spatial frequencies. Further work is needed to test whether humans and other species employ this form of active control.

Experimental Procedures

Stimuli consisted of natural scenes (free-viewing experiment, Figs. 1 and 2) and Gabor patches (forced-choice experiments, Figs. 4–6). They were displayed on a gamma-corrected fast-phosphor CRT monitor (Iiyama HM204DT) at either 100 Hz (free-viewing) or 200 Hz (forced-choice). Stimuli were observed monocularly, with the left eye patched. Twenty-one emmetropic subjects participated in the study following the procedures approved by the Boston University Charles River Campus Institutional Review Board.

Eye movements were recorded by means of a Dual Purkinje Image eyetracker (Fourward Technology) and sampled at 1 KHz, while a custom dental-imprint bite bar and a headrest prevented head movements. Movements with speeds larger than 3°/s and amplitudes exceeding 3’ were classified as saccades. Saccade amplitude was defined as the length of the vector connecting the two locations at which velocity became greater (onset) and lower (offset) than 2°/s.

Thresholds in Figs. 4–6 represent Michelson contrasts yielding 75% performance. The stimulus contrast changed at each trial following the PEST algorithm [45]. Gaze-contingent manipulation of the retinal input was achieved by means of a custom-developed system for flexible gaze-contingent display control [46], an extensively-tested system than enables precise synchronization between eye movements and changes in the display. All trials in which the saccade landed more than 1° away from the center of stimulus or in which blinks or microsaccades occurred during stimulus presentation were discarded (see Supplemental Experimental Procedures for details).

Frequency analyses in Fig. 2 were conducted by means of the eye movement displacement probability—the probability that the eyes moves by x in the interval t—, an approach described in previous publications [19,47] that enables high-resolution spectral estimation. The mean instantaneous firing rates of simple cells were modeled by means of standard space-time separable filters designed on the basis of neurophysiological data (Fig. 1E). Modeledreceptive fields moved following recorded traces of eye movements (Fig. 1E and Fig. 3A). In Fig. 3, a decision-making stage continually integrated the responses of modeled neurons tuned to the grating’s frequency and orientation. At each time during post-saccadic fixation, contrast thresholds were selected as the contrast values yielding hits and false alarm rates of 0.75 and 0.25, respectively (d′ = 1.35).