Spatiotemporal Content of Saccade Transients

Summary

Humans use rapid gaze shifts, known as saccades, to explore visual scenes. These movements yield abrupt luminance changes on the retina, which elicit robust neural discharges at fixation onsets. Yet, little is known about the spatial content of saccade transients. Here we show that saccades redistribute spatial information within the temporal range of retinal sensitivity following two distinct regimes: saccade modulations counterbalance (whiten) the spectral density of natural scenes at low spatial frequencies and follow the external power distribution at higher frequencies. This redistribution is a consequence of saccade dynamics, particularly the speed/amplitude/duration relation known as the main sequence. It resembles the redistribution resulting from inter-saccadic eye drifts, revealing a continuum in the modulations given by different eye movements, with oculomotor transitions primarily acting by regulating the bandwidth of whitening. Our findings suggest important computational roles for saccade transients in the establishment of spatial representations and lead to testable predictions about their consequences for visual functions and encoding mechanisms.

eTOC Blurb

Humans use saccades (rapid gaze shifts) to explore visual scenes. Mostofi et al. show that these eye movements flexibly reformat the luminance flow on the retina in a way that selectively discards redundant information present in natural scenes. These results reveal a matching between saccade dynamics and the characteristics of the natural world.

Introduction

Humans perform rapid eye movements, known as saccades, more frequently than their hearts beat. Every few hundreds of milliseconds, a saccade shifts gaze toward a new location in the scene, so that it can be inspected with the fovea, the tiny region of the retina that affords high visual acuity. Although this region only covers a minuscule fraction of the visual field—less than 0.001%—it has disproportionate importance for visual functions, as manifested by the devastating consequences of foveal impairments [1, 2].

Much research has focused on how saccades affect visual function. At the perceptual level, it has long been observed that saccades are coupled with a transient attenuation in visual sensitivity, an effect known as “saccadic suppression” [3–7]. Saccades have also been associated with mislocalizations and distortions in the perception of space [8–11], enhancements in visual sensitivity following their offsets [12–15], as well as other extraretinal effects [7,16–18]. At the neural level, the sudden changes in the visual input caused by saccades tend to elicit strong responses at fixation onset in neurons at the early stages of the visual system [19–22]. Furthermore, phenomena associated with saccade preparation [23–26], as well as modulatory signals related to motor commands [27–30] have been observed.

Despite the massive impact of saccades at both the perceptual and neural levels, relatively little attention has been paid to the information content of the luminance signals that these movements deliver to the retina. As they relocate gaze, saccades yield complex spatiotemporal modulations that depend on both the dynamics of the movement and the statistics of the visual scene. These signals presumably play an important role in the strong responses of neurons following saccades. Previous analysis of another type of eye movements, ocular drift, the incessant inter-saccadic wandering of the eye, has shown that the motion of the eye may act as an information processing stage. As ocular drift transforms spatial patterns into temporal modulations, it counterbalances the power distribution of natural scenes, yielding an input signal to the retina that attenuates statistical redundancies in natural scenes and enhances luminance discontinuities [31, 32], effects traditionally attributed to center-surround interactions within the receptive fields of retinal ganglion cells [33–35]. It is unknown whether saccades also serve computational functions in the processing of visual information, beyond simply presenting new stimuli to the retina.

Here we show that saccade dynamics—specifically the main sequence, the well-known relations among duration, velocity, and amplitude [36]—lead to a flexible reformatting of the visual flow, which selectively discards redundant information present in natural scenes. The bandwidth of this phenomenon increases for small saccades, with microsaccades approaching the previously reported effects for ocular drift. These results reveal a form of matching between saccade dynamics and the characteristics of the natural world and show that the luminance signals delivered by movements as different as saccades and ocular drifts are part of a continuum. These findings have important implications for the visual functions of saccades, their motor characteristics, and the mechanisms of spatial encoding.

Results

The visual signals impinging onto retinal receptors are never stationary. Eye movements transform an external spatial scene into a temporally changing flow of luminance to the retina, even when no motion occurs in the scene itself (Figure 1A–C). In the frequency domain, this transformation corresponds to a conversion of the spatial power of the image into temporal power in the retinal input. For a static scene, if the eye did not move, all the power of the incoming luminance flow would be confined to the zero temporal frequency plane (Figure 1D). However, oculomotor behavior redistributes this spatial power across non-zero temporal frequencies (Figure 1E). The specifics of this redistribution depend on the characteristics of eye movements.

Figure 1. Oculomotor influences on the retinal input.

Observers freely examined pictures of natural scenes, while their eye movements were measured at high spatial and temporal resolution.

(A) An example of recorded eye movements superimposed on the observed image.

(B) The corresponding horizontal and vertical traces of eye positions as a function of time.

(C) Eye movements transform a static visual scene into a time-varying luminance flow on the retina. An example of the resulting visual flow: the images represent the input at various times as the eye moves.

(D-E) This transformation corresponds to a redistribution of the stimulus spatial power in the joint space-time frequency domain. The power of the retinal stimulus would (D) be concentrated at 0 Hz in the absence of image motion, but (E) is shifted to nonzero temporal frequencies by eye movements.

(F-G) Saccade characteristics. (F) Distribution of saccade amplitude averaged across observers (N = 14; bin width 0.2°). The vertical dashed line marks the mean of the distribution. The inset zooms in on the range of small amplitudes (bin width 3′). (G) Saccade main sequence. Each dot corresponds to one saccade, and the red line represents the linear regression of the data averaged across subjects. Shaded areas in both F and G represent SEMs across subjects.

Here we focus on the space-time conversion resulting from saccades. Rather than restricting our analysis to the instantaneous modulations present during saccades, we consider, more broadly, how shifting gaze from one point to the next via a saccadic movement redistributes the power of an external static scene across temporal frequencies on the retina (i.e., from ω = 0 to ω ≠ 0; Figure 1D–E). These input changes strongly drive neural responses after saccades, irrespective of possible influences from saccadic suppression. In the following, we will refer to the luminance flow resulting from exploring the external scene via eye movements as the visual (or retinal) input. This signal should not be confused with the stationary pattern of luminance given by the scene by itself.

We recorded the eye movements of 14 subjects as they freely examined images of natural scenes. As expected, our observers made frequent saccades (~3 saccade/s; average inter-saccadic interval across subjects ± standard deviation: 248 ± 54 ms). Saccades covered a broad range of amplitudes, from just a few minutes of arc to over 10° (average saccade amplitude: 3.1 ± 0.79°; Figure 1F) and a very wide range of velocities, with peak speed ranging from ~20°/s to more than 500°/s. Most observers executed primarily saccades around 2–3°, except for a few who exhibited a preference for smaller amplitudes. Irrespective of these individual preferences, all observers exhibited tightly stereotyped relations among peak velocity, saccade duration, and amplitude, as well established in the literature [36] (Figure 1G). The resulting fast and abrupt motion of the retinal image contrasts with the slow/smooth motion present in between saccades. In these fixation intervals, the eye drifted following seemingly random trajectories, with a mean instantaneous speed of 1°/s, a value consistent with previous measurements [37].

We first examined how saccades redistribute power at each individual spatial frequency. That is, we estimated the redistribution resulting from saccades along the temporal frequency axis, given unit power at each spatial frequency. This is computationally equivalent to the power spectrum of the visual flow delivered by the recorded saccades had they occurred over white noise scenes. This step is important because the spatiotemporal transformation resulting from saccades is non-linear in the temporal domain but linear in space. This implies that we can understand the impact of saccades on any possible spatial scene from the modulations saccades generate at each individual spatial frequency.

Since saccades of different sizes follow different velocity profiles and, therefore, yield luminance modulations with distinct characteristics on the retina, we divided the recorded saccades into subsets with similar amplitudes, so that all saccades within each subset differed by no more than 1° in size. We then examined each subset separately. The data in Figure 2 refer to a common range of saccades, those with amplitudes between 2° and 3°. A very specific pattern is clearly visible in these data, with two distinct regimes present and a transition at ~0.3 cycles/deg.

Figure 2. Power redistribution caused by saccades.

(A) Proportion of the spatial power in the stimulus that saccades make available at each spatial and temporal frequency on the retina. Data represent averages across observers and refer to saccades with 2 – 3° amplitudes. Note the presence of two distinct regimes. The horizontal dashed lines mark the temporal frequencies further examined in B.

(B) Power present at four temporal frequencies (dashed lines in A). For each temporal frequency, the critical spatial frequency separating the two regimes is defined as the frequency at which power deviates by 3 dB from the k² interpolation (k denotes spatial frequency).

(C) The critical spatial frequency as a function of temporal frequency. Shaded areas in B and error bars in C represent one SD across subjects.

See also Figure S1.

At high spatial frequencies, the abrupt changes caused by saccades yield luminance modulations with approximately constant power across spatial frequencies. This ‘saturation regime’ is consistent with previous reports [15] and conforms to the standard way of thinking about saccades as producing abrupt luminance steps. Unexpectedly, however, a strikingly different regime emerges at lower spatial frequencies. In this band, the power of the luminance modulation increases with increasing spatial frequency, a behavior not previously reported in the literature. We will refer to this regime as ‘whitening’ for reasons that will be clear in Figure 4.

The presence of two regimes is particularly evident in the temporal frequency sections in Figure 2B, which better show how the saccade-induced conversion of spatial power into temporal modulations depends on the spatial frequency of the stimulus. At every non-zero temporal frequency section, the gain of this conversion first increases proportionally to the square of the spatial frequency until a critical spatial frequency is reached, and then delivers equal power across spatial frequencies. Note that the critical spatial frequency that divides these two regimes varies relatively little with temporal frequency, i.e., by less than a factor of 3 as temporal frequency varies from 1 to 30 Hz (Figure 2C).

The data in Figure 2 refer to saccades in a single amplitude range (2–3°). However, saccades vary greatly in amplitude, prompting the question of how saccade size influences the power redistribution. To investigate this point, Figure 3A–B shows the spatiotemporal power made available by saccades in three amplitude ranges: microsaccades with amplitudes of only a fraction of a degree; small saccades of 1–2°; and relatively large saccades with amplitudes of 5–7°. Comparison across different saccade amplitudes shows that, qualitatively, all saccades transform the spatial patterns entering the eyes into temporal modulations on the retina in a similar manner. They all result in the two distinct regimes of whitening and saturation in separate spatial frequency bands. In the whitening regime at low spatial frequencies, power always increases proportionally to the square of spatial frequency; and in the saturation regime at higher spatial frequencies, all saccades deliver the same amount of power irrespective of their amplitudes.

Figure 3. Impact of saccade amplitude.

(A-B) The analysis of Figure 2 is here conducted for saccades in three different amplitude ranges. Graphical conventions are as in Figure 2. The full spatiotemporal spectra are in A, and sections at selected temporal frequencies in B.

(C) Changes in power with saccade amplitude. Each curve represents the total power within the temporal range of human sensitivity [38] by saccades of a given amplitude. Shaded areas represent SEM across subjects. For comparison, the panel also shows the power of the modulations resulting from the inter-saccadic eye drifts recorded both in the experiments of this study and during examination of a 20/20 line in a Snellen chart (data from Intoy and Rucci [40]). (D) Critical spatial frequency, the frequency separating whitening and saturation regimes, as a function of saccade amplitude. This value was estimated from the curves in C, following the procedure described in Figure 2C. Experimental data are well approximated by the function 1/2A, where A is the saccade amplitude. Error bars represent one standard deviation across temporal frequencies.

As summarized by the data in Figure 3C, two important changes occur as saccade amplitude varies. First, the critical spatial frequency that separates the two regimes of whitening and saturation also varies. As the saccade amplitude decreases, the critical frequency shifts toward higher spatial frequencies, resulting in a broader whitening range. Second, below the critical frequency, the power of the input signals increases with saccade amplitude. Saccades with amplitudes above 7° give substantially stronger signals than those elicited by 1° saccades within the bandwidth of temporal sensitivity. This effect is a consequence of the faster and wider displacements resulting from larger saccades, which convert more spatial power into temporal power, as shown by comparison of the spectral distributions at individual temporal frequencies in Figure 3B.

The data in Figure 3C were obtained from the spatiotemporal spectra in Figure 3A, by weighting and integrating temporal frequencies based on the known temporal characteristics of human temporal sensitivity, as measured in the absence of the retinal image motion caused by eye movements [38]. Over the range of saccade amplitudes examined in this study, the critical spatial frequency varied by approximately one order of magnitude, from ~1 cycle/deg for microsaccades (<0.5 deg) to less than 0.1 cycles/deg for saccades larger than 7° (Figure 3D). Interestingly, the critical spatial frequency is well-approximated by 1/(2A), where A is the saccade amplitude. This is the frequency of a grating for which a single lobe is equal to the saccade amplitude.

Thus, all saccades, irrespective of their amplitudes, yield luminance modulations that qualitatively follow the same pattern of power redistribution, but the boundary between the two regimes of whitening and saturation and the amount of low-frequency power depend on the saccade amplitude.

For comparison, Figure 3C also shows the power redistributions resulting from ocular drifts. Previous studies have shown that, over a wide range of spatial frequencies, the proportion of spatial power in the scene that ocular drift makes available in the form of temporal modulations also increases proportionally to the square of the spatial frequency [39]. The luminance flow from drift measured in our experiments closely followed this behavior, with power increasing up to ~10 cycles/deg, similar to what reported in previous free-viewing experiments [31]. Like for saccades, this cut-off frequency is not hard-wired: it has been observed that it changes across tasks [40], yielding broader ranges of whitening in high-acuity tasks (see dashed and solid gray curves in Figure 3C).

Thus, surprisingly, the luminance modulations resulting from eye movements as different as saccades and ocular drifts follow similar spectral distributions. Both types of eye movements yield two distinct modulation regimes in separate spatial frequency ranges, with the lower band characterized by an increase in power with spatial frequency. The primary difference across distributions is the critical spatial frequency, which changes according to the eye movements: it is higher for ocular drift than for saccades. In both cases, the critical frequency varies with the scale of the movement, the equivalent diffusion constant for drift and the amplitude for saccades. Both distributions can be captured by a unified, general model, with a single parameter (the critical spatial frequency separating the two regimes) that varies according to the type of movement.

So far, we have focused on the proportion of the stimulus power redistributed by saccades. As mentioned above, the data in Figure 2,3 correspond to the power spectra of the retinal signals available while viewing scenes with constant spectral density, such as white noise images. In the presence of a scene with different statistics, saccade modulations depend on the spatial power available in the scene itself, which in general will not be equal across spatial frequencies. It is particularly interesting to examine the interaction between eye movements and natural scenes.

It is well known that the power of natural scenes decreases approximately proportionally to the square of the spatial frequency [41]. Below the critical spatial frequency, saccade modulations counterbalance this distribution, yielding a luminance flow with equalized spatial power, i.e., a whitened retinal input (Figure 4A–B; hence the term ‘whitening regime’ to indicate this band). Above the critical spatial frequency, the proportion of redistributed power remains constant, so that the luminance transients from saccades follow the spectral density of natural images, decreasing proportionally to the square of spatial frequency. These general characteristics hold for saccades with all amplitudes. However, larger saccades yield a narrower whitening band, and more power within this band reaches the temporally relevant range of sensitivity (Figure 4C). In contrast, all saccades deliver equal power irrespective of their amplitude, in the saturation regime. Thus, during natural viewing, the normal saccade-drift alternation yields a cyclical modulation of the range of whitening to which retinal neurons are exposed. This cycle emphasizes distinct spatial frequency ranges depending on saccade amplitude.

Figure 4. Interaction between the power redistribution due to saccades and natural scenes.

(A-B) Power spectrum of the luminance modulations from saccades (2°−3°) during viewing of natural images. Both the full space-time distribution (A) and sections (B) at selected temporal frequencies (horizontal lines in A) are shown. Note that below the critical spatial frequency saccades counterbalance the power spectrum of natural scenes. At higher spatial frequencies, visual signals follow the spectral density of natural images (the dashed lines, k⁻², in B). Graphical conventions are as in Figure 2.

(C) Power in the temporal range of human sensitivity delivered by saccades with various amplitudes. The shaded areas represent SEM.

Why do saccades deliver luminance transients with these characteristics? An intuition can be gained from the changes in the visual signal experienced by a retinal receptor as a saccade relocates gaze. If the scene does not vary in space, as when looking at a uniform field (all power at 0 spatial frequencies), the receptor experiences no change in its input signal. If, in contrast, the same saccade occurs over a textured, non-uniform field (power available at non-zero spatial frequencies), the visual input changes, and the extent of the modulation depends on the specific structure of the scene. Consider, for example, a saccade of amplitude A that shifts gaze over an orthogonal grating at spatial frequency k. For small k, the variance of the luminance signal experienced by the retinal receptor will grow as k increases. The largest response will be achieved for k = 1/2A, the frequency that, on average, yields the largest luminance difference between the starting and ending locations of the saccade. For larger k, the variance of the input modulation saturates, as the receptor covers more than half a period during the saccade. Thus, as spatial frequency increases, the amount of change in the modulation first increases and then saturates, yielding the two regimes described in Figure 2. This example also provides an intuition for the influence of saccade amplitude. The frequency of the grating that yields the largest luminance change decreases with increasing saccade amplitude; while, all saccades, irrespective of their amplitudes, yield signals with similar variance with stimuli at higher spatial frequencies.

Although intuitive, the description above neglects important detail. A deeper understanding can be gained on the basis of the dynamics of saccades, specifically the well-established relations among duration, peak velocity, and amplitude known as the saccade main sequence [36]. A fundamental consequence of these relations is that all saccades can be well described by scaling and stretching a prototypical waveform, u(t) (Figure 5A). That is, any saccade trace, s(t), can be approximated as: s(t) ~ A u(t/ψ), where A is the amplitude ψ the duration, and u(t) is a standard template for saccade trajectories. This implies that the spectral redistribution of the luminance flow of any saccade, S(k,ω), can be directly estimated from U(k,ω), the redistribution caused by the template u(t): S(k,ω) = ψ² U(Ak, ψω). Note that, as saccade amplitude varies, two factors cooperate to attenuate the influence of the temporal scaling factor, ψ: (a) the relation between saccade velocity and amplitude, specifically the higher speed reached by larger saccades (Figure 1G), which maintains ψ relatively close to unity (Figure 5B); and (b) the spectral characteristics of U, notably its ~1/ω² behavior (Figure 5D), which partially counterbalances the scaling. Thus, saccades with different amplitudes primarily differ by an amplitude-dependent compression of the spatial frequency axis, as shown in Figure 3.

Figure 5. Some factors responsible for the power spectra of saccade transients.

(A) Saccades of different amplitudes (colored solid lines) can be well approximated by scaling in size and time a prototypical saccade waveform u t (dashed lines).

(B) The spatial (A) and temporal (ψ) scale factors resulting from least-squares fitting of saccades with different amplitudes (x-axis). Note that Ψ remains close to 1, a consequence of the saccade main sequence. The red line marks the linear regression.

(C) Spatial distribution of power made available at 8 Hz in three cases: an instantaneous displacement over orthogonal gratings (yellow curve); the prototypical saccade waveform over orthogonal gratings (blue curve); the prototypical saccade waveform over a white noise pattern, with texture in all orientations (red curve). The x-axis represents the frequency of the grating(s).

(D) Same as in C for the temporal distributions at 1 cycle/degree.

What is then U(k,ω) ? Figure 5C shows a temporal section (8 Hz) when the stimulus is simply a grating orthogonal to the saccade. If saccades were instantaneous displacements, the changes in luminance would only depend on their starting and ending positions, so that power would oscillate with the frequency of the grating (yellow curve in Figure 5C). Real saccades, however, deliver a signal that depends not just on the end points of the trajectory, but also on their speed dynamics, yielding more evenly distributed power at high spatial frequencies (blue curve in Figure 5C). Furthermore, more complex stimuli, like natural scenes, contain Fourier components with all possible orientations, not just orthogonal to the saccade direction; these oblique components act as if they had a lower spatial frequency (down to 0 cycles/deg for a grating parallel to the saccade). In these circumstances, the total power available at any spatial frequency also includes these non-orthogonal components, further flattening the power distribution in the high-frequency range (red curve in Figure 5C). Thus, while several factors contribute, the temporal reformatting of the visual flow resulting from saccades is primarily shaped by saccade dynamics and the main sequence.

Discussion

Eye movements are an incessant presence during normal visual functions. Since high visual acuity is only restricted to a minuscule portion of the visual field, humans need to shift their gaze several times every second, executing billions of saccades throughout their lifetimes. Like all eye movements, saccades temporally modulate the stimulus on the retina, effectively transforming patterns of luminance into a spatiotemporal flow impinging onto retinal receptors. Here we have shown that this transformation results in a stereotypical redistribution, which counterbalances the power spectra of natural scenes over a frequency range that systematically depends on saccade amplitude. For very small saccades, this redistribution approaches that given by inter-saccadic eye drifts, so that the luminance modulations delivered by both slow/smooth and rapid/jerky eye movements follow a similar pattern, well-described by a unified model.

Estimating the power redistribution resulting from saccades is not straigtforward. The visual flow delivered by these movements is inherently non-stationary, and their brief duration limits the temporal resolution that can be achieved via standard methods of spectral analysis. Standard methods are also limited in the lowest spatial frequency range that can be practically examined, as spatial resolution is inversely related to image size. To circumvent these difficulties, here we averaged the power spectrum of the visual flow over the course of saccades and followed the factorization approach developed in Kuang et al. [31]. This method assumes independence between eye movements and the pattern of luminance to gain spectral resolution. It enables power estimation at any desired spatial frequency, if the spectral distribution of the external scene is known. Since saccades tend to be directed toward meaningful regions of the scene, the assumption of independence may not strictly hold within selected regions of the retina, notably the fovea. It is, however, a reasonable assumption when considered across the entire retina, as saccade characteristics cannot depend on the pattern of luminance at every retinal location. Indeed, results obtained with the factorization method were virtually identical to those obtained—at lower resolution—by means of a more standard method of spectral analysis (see Figure S1), confirming the validity of the approach.

These findings have implications at multiple levels. At the methodological level, a direct consequence is that they challenge the common assumption that equates saccade temporal transients with instantaneous steps in contrast. Studies of visual perception often replace saccades with abrupt stimulus onsets, implicitly assuming that the two types of stimulus presentations are functionally equivalent. However, these transients differ considerably in terms of the spatial information they provide. Unlike saccades, contrast steps do not yield a whitening range: the proportion of spatial power that is made available in the form of luminance modulations remains constant across all spatial frequencies. Thus, perceptual and neural responses measured with contrast steps in the laboratory may differ in important ways from those elicited by saccades during natural viewing. For example, one would expect the responses of neurons with peak sensitivity within the whitening range to be more similar after saccades than following steps in contrast. Furthermore, stronger correlated activity should be expected in the responses of retinal ganglion cells to natural scenes when saccades are replaced by abrupt onsets of the same images.

With regard to neural encoding, our results suggest that saccades act as a computational stage in the processing of visual information. Theories of efficient sensory encoding [33, 42] were traditionally formulated without taking eye movements into account, under the premise that the statistical properties of retinal stimulation and the observed external images were identical. This, however, is not the case during natural viewing. Previous studies have observed that the incessant inter-saccadic motion of the eye removes part of the statistical redundancies of natural scenes before any neural processing [39]. Our data add to this previous body by showing that, surprisingly, saccades and drift yield luminance transients that differ only quantitatively, not qualitatively. Saccades, like ocular drift, also possess a whitening region, but restricted to a lower range of spatial frequencies, up to approximately 1 cycle/deg for small saccades.

Interestingly, retinal ganglion cells tend to exhibit sensitivity functions complementary to the input reformatting given by saccades. These neurons typically possess: (a) a frequency range within the saccade saturation region in which sensitivity increases with spatial frequency; and (b) a lower range that overlaps with the saccade whitening region, in which sensitivity flattens, deviating from the responses of ideal decorrelating filters [43, 44]. These observations suggest a possible synergy between the structure of saccade transients and the sensitivity of ganglion cells during viewing of natural scenes, with the two elements cooperating to attenuate correlations in neural responses immediately following saccades, before the influences from ocular drifts emerge. This interaction would facilitate encoding of the visual scene under examination by emphasizing how it differs from the general, predictable characteristics of natural scenes.

The reformatting of the visual flow resulting from saccades may also provide an explanation for discrepant findings between studies conducted in the presence and absence of eye movements. Neurophysiological recordings often use temporally modulated stimuli. With such stimuli, saccades will spread temporal power in a similar way to what occurs with a stationary scene, but starting from the frequency of the modulation. This spread is similar to the mechanism proposed to be responsible for boosting sensitivity to temporally-modulated low spatial frequency gratings relative to stationary ones [45]. This effect tends to maintain more power within the bandwidth of temporal sensitivity in a low range of spatial frequencies. Thus, saccade modulations may play a role in the shift in neural tuning toward lower spatial frequencies measured in experiments in which eye movements occur [46, 47] relative to studies with anesthetized, paralyzed monkeys [48, 49].

At the perceptual level, multiple considerations emerge. First, it is worth pointing out that a contribution of saccade transients to visual representations is not in contradiction with the notion of saccadic suppression, the lack of awareness to the inter-saccadic motion of the retinal image [3, 50, 51]. As noted above, the spectral analyses reported here apply to the luminance flow given by the entire saccadic gaze shift, not just to the signals present during the movement itself. That is, they include the changes in visual input from one fixation to the next, which elicit strong responses after the termination of saccadic suppression [19–22]. Furthemore, contrast sensitivity is only moderately suppressed during saccades [4, 24, 52], and information acquired around the occurrence of saccades may influence visual processes [53, 54]. In fact, our results are informative in terms of the mechanisms of saccadic suppression. Because of the whitening regime, saccades yield weaker signals at low spatial frequencies than commonly assumed. This attenuation may contribute to make a moderate suppression in sensitivity sufficient to prevent visibility of retinal image motion during saccades.

An important consequence of our findings is the possibility that saccades can be used for selecting information not just in space (by positioning the fovea), but also in spatial frequency, by using saccade amplitude to control the spatial frequency range that is made available within the temporal frequency range of retinal sensitivity. While the primary function of saccades is to enable sequential exploration of the scene by centering objects of interest on the high-acuity fovea, our data show that a similar selection process is also possible relative to spatial frequency. In terms of absolute power, the strength of the luminance flow to the retina increases with saccade amplitude in the region of whitening (Figure 4C). In relative terms, once the total amount of power on the retina is normalized, the bandwidth of the whitening region decrease with saccade amplitude. Mechanisms of contrast gain control, which presumably emphasize relative differences within power distributions rather than overall power, occur starting from the retina [55–58]. Thus, saccade amplitude could be effectively controlled to emphasize the frequency range relevant to the task at hand. Targeted experiments are needed to determine whether this power modulation translates into a perceptual benefit and, if so, whether this phenomenon plays a role in saccade planning.

Irrespective of whether or not saccades are controlled to select spatial frequency, our data show that a stereotypical evolution in the bandwidth of whitening continually occurs during the natural alternation between saccades and fixational drifts. Visual responses are driven by a signal that contains strong low-spatial frequency power and narrow whitening bandwidth immediately after a saccade, and stronger high-spatial frequency power and broad whitening later during fixation. This observation is in agreement with the proposal that eye movements initiate, during normal viewing, a coarse-to-fine dynamics of visual analysis [15], a processing hierarchy in which analysis of the overall gist of the scene precedes elaboration of fine details [59–64].

These dynamics also imply different optimal strategies for extracting spatial information in separate frequency ranges. Given the vast discrepancy in saccade/drift power at low, but not high, spatial frequencies, low spatial frequency information is predominantly available immediately after a saccade, whereas high spatial frequency information is available throughout the post-saccadic period of fixation. The response characteristics of magnocellular and parvocellular neurons, with their different tuning in both space and time, appear well suited to capture these time-courses, and evidence exists in favor of a role of the magnocellular pathway in encoding the initial gist of a scene [65, 66]. In this view, the delays in thalamic afferents [67] and refinements in neuronal selectivity [68–70] that also occur under anesthesia and oculomotor paralysis are not the primary causes for a coarse-to-fine evolution of visual processing, but rather reflect the natural tuning of a system designed to operate on the visual signals given by the saccade/drift alternation. This view also implies that saccades act as clocks in the establishment of visual representations, as the information conveyed by each neuronal impulse varies with the amount of time elapsed since the end of a saccade.

In sum, our results show that the characteristics of saccades, particularly their dynamics and the velocity-amplitude relation, lead to luminance modulations on the retina that counterbalance the power spectra of natural scenes up to an amplitude-dependent cut-off frequency. The resulting conversion of spatial patterns into temporal signals is similar to that previously observed for inter-saccadic eye drifts, but now compressed in spatial frequency and with greater power at low spatial frequency. These findings suggest that saccades are not merely a means to center the high-acuity fovea on the objects of interest. Rather, they appear to play important roles in processing visual information before neural computations take place, by removing broad-scale correlations in natural scenes, enhancing low-frequency vision, and setting the stage for a coarse-to-fine processing dynamics. Saccade characteristics are altered in a variety of neurological disorders [71]. Our findings point at the need to examine the visual signals emerging in these conditions and their possible relations with the observed perceptual deficits.

STAR⋆METHODS

RESOURCE AVAILABILITY

Lead Contact

Further information and requests for resources should be directed to and will be fulfilled by the Lead Contact, Michele Rucci (mrucci@ur.rochester.edu).

Materials Availability

This study did not generate new unique reagents.

Data and Code Availability

The dataset generated during this study is available at the Mendeley Data repository, http://dx.doi.org/10.17632/5yvjwnpggg.2. This study used standard, custom-built MATLAB programmed scripts that are available from the Lead Contact upon request.

EXPERIMENTAL MODEL AND SUBJECT DETAILS

Human subjects

Fourteen subjects (6 females and 8 males; age range: 21–31 years) with normal, non-corrected vision participated in the study. All observers were naive about the purpose of the experiment and were compensated for their participation. Informed consent was obtained from all participants following the procedures approved by the Boston University Charles River Campus Institutional Review Board and the Declaration of Helsinki.

METHOD DETAILS

Apparatus

Gray-scale images of natural scenes were displayed on a fast-phosphor CRT monitor (Iyamaya HM204DT) at 800×600 pixel resolution and 200 Hz refresh rate. Stimuli were observed monocularly with the right eye while the left eye was patched. Subjects were positioned at a fixed distance (126 cm) from the monitor, so that each pixel subtended ~1′. The subject’s head was immobilized by a custom dental-imprint bite bar and a head-rest.

Eye movements were measured by means of a Generation 6 Dual Purkinje Image (DPI) eye-tracker (Fourward Technologies). This method is known to provide high linearity and reaches a resolution of approximately 1′ [72]. The specific unit used in the experiments had been extensively tuned and tested to minimize its level of noise and increase its sensitivity. The analog output signals delivered from the eye-tracker were first low-pass filtered at 500 Hz and then sampled at 1 kHz. The digital traces were recorded for off-line analysis.

Data Collection

Observers were instructed to freely examine and memorize pictures of natural scenes. The images were presented sequentially, each for 10 s. Data were collected via separate blocks of consecutive trials, each trial consisting in the presentation of one image, in experimental sessions that lasted approximately one hour each. Before a block of trials, preliminary set-up procedures, described in detail in previous publications, ensured optimal eye-tracking. These procedures included tuning the eye-tracker and executing a gaze-contingent calibration procedure that enables conversion of the eye-tracker output signals into degrees of visual angle with high precision [18, 73]. The duration of each block of trials never exceeded 10 minutes, and brief breaks between successive blocks allowed the subject to rest.

QUANTIFICATION AND STATISTICAL ANALYSIS

Analysis of Oculomotor Data

Eye movement traces were processed as described in previous publications [40]. Only periods with optimal, uninterrupted tracking and no blinks were selected for data analysis. Traces were segmented into complementary periods of fixation and motion events based on a speed threshold of 3°/s. The instants in which speed became, respectively, higher and lower than this threshold defined the start and end points of the event, respectively, and its amplitude was defined as the modulus of the vector connecting gaze positions at these two instants. Consecutive events in which eye speed returned lower than threshold for less than 15 ms were merged into a single movement, a method that took care of possible post-saccadic overshoots [74]. Events with amplitude of at least 3′ were classified as saccades.

Power spectrum estimation

Spectral analyses were conducted using the factorization method described in previous publications [31]. This method enables high-resolution estimation of the spectral density, under the plausible assumption of statistical independence between the pattern of luminance and eye movements:

$$S\left(k,\omega \right)=I\left(k\right)Q\left(k,\omega \right)$$ (1)

where k and ω indicate spatial and temporal frequencies, respectively; I represent the power spectrum of the external scene (∝ k⁻² for natural scenes); and Q(k,ω) is the spatiotemporal power redistribution resulting from eye movements. This term is given by the Fourier Transform of the displacement probability q(x,t), the probability that the eye moved by x in an interval t.

To estimate the average redistribution given by saccades, we first grouped them according to their amplitudes. For each recorded saccade, ξ(t), we selected a 512 ms segment centered on the time of peak velocity and eliminated confounding influences from the eye drifts preceding and following the saccade by replacing these trajectories with periods of equal duration in which the eye was assumed to remain immobile. We then estimated Q directly in the frequency domain, as:

$$Q\left(k,\omega \right)={\langle {\left|\int e^{-2\pi j{k}^T\xi \left(t\right)}{e}^{-2\pi j\omega t}dt\right|}^2\rangle }_{\xi }$$ (2)

By separating retinal image statistics and eye movements, this method enables spectral estimation with high temporal resolution at any desired spatial frequency k. Reported data are the spectra estimated individually for each subject and then averaged across observers. They are summarized in two dimensions (space and time) by taking radial averages across spatial frequencies (k = ∥k∥). For comparison, the power spectrum of the retinal flow given by ocular drift was estimated following the same approach over uninterrupted, non-overlapping drift segments of 1024 ms.

To facilitate comparisons of the amount of useful power made available by eye movements, in Figure 3C and Figure 4C, we integrated across temporal frequencies after weighting spectral density distributions by the profile of human temporal sensitivity. We used the filter developed by Kelly [38], which incorporates the way temporal sensitivity changes with spatial frequency:

$$G(\omega ,k)=\left(6.1+7.3{\left|\log \left(\frac{k}{3}\right)\right|}^3\right)\omega k{e}^{-\frac{\omega +2k}{22.95}}$$

This filter was derived from measurements obtained under retinal stabilization, a condition that minimizes oculomotor influences. At each individual spatial frequency, it was normalized by the sum of its values over temporal frequency to summarize the power delivered by saccades within the range of temporal sensitivity.

KEY RESOURCES TABLE

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Antibodies
Bacterial and Virus Strains
Biological Samples
Chemicals, Peptides, and Recombinant Proteins
Critical Commercial Assays
Deposited Data
Original data	This paper	http://dx.doi.org/10.17632/5yvjwnpggg.2
Experimental Models: Cell Lines
Experimental Models: Organisms/Strains
Oligonucleotides
Recombinant DNA
Software and Algorithms
Matlab code to process data	this paper & Mathworks	https://www.mathworks.com/
Other

Highlights.

By formatting the input flow, saccades play an important computational role in vision

At low spatial frequencies, saccades discard redundant information in natural scenes

The bandwidth of this effect (whitening) increases as saccade amplitude decreases

The whitening bandwidth oscillates during the natural saccade/fixation cycle