Temporal Encoding of Spatial Information during Active Visual Fixation

Summary

Humans and other species continually perform microscopic eye movements, even when attending to a single point [1-3]. These movements, which include microscopic drifts and microsaccades, are under control of the oculomotor system [2, 4, 5], elicit strong responses throughout the visual system [6-11], and have been thought to serve important functions [12-16]. The influence of these fixational eye movements on the acquisition and neural processing of visual information remains unknown. Here, we show that during viewing of natural scenes, microscopic eye movements carry out a crucial information-processing step: they remove predictable correlations in natural scenes by equalizing the spatial power of the retinal image within the frequency range of ganglion cells' peak sensitivity. This transformation, which had been attributed to center-surround receptive field organization [17-19], occurs prior to any neural processing, and reveals a form of matching between the statistics of natural images and those of normal eye movements. We further show that the combined effect of microscopic eye movements and retinal receptive field organization is to convert spatial luminance discontinuities into synchronous firing events, thus beginning the process of edge extraction. In sum, our results show that microscopic eye movements are fundamental to two goals of early visual processing —redundancy reduction [20, 21] and feature extraction— and, thus, that neural representations are intrinsically sensory-motor from the very first processing stages.

Results and Discussion

To determine the influence of fixational eye movements, we examined the visual input to the retina during periods of visual fixation. The eye movements of human observers were recorded during free-viewing of pictures of natural scenes, a condition in which saccades occurred frequently (average intersaccadic duration: 248 ±54 ms) and possessed highly variable amplitude (Fig. 1C). As expected from previous studies [22], microsaccades were rare in this condition and their frequency varied substantially across subjects, ranging from approximately one microsaccade every 2 s to one every 20 s (average rate: 0.17 ± 0.15 microsaccades/s). Nevertheless, the eye was continually in motion, as it drifted with apparently random trajectories in between saccades (Fig. 1D and E). Across all subjects, the probability distribution of the center of gaze during the intersaccadic periods covered an area of 363 ± 151 arcmin². Thus, the projection of the scene moved significantly on the retina during fixation.

Figure 1.

Microscopic eye movements modulate visual input signals during normal fixation. (A) An example of recorded eye movements superimposed on the observed image. The enlarged region shows the eye movements occurring during a period of fixation in relation to the size of photoreceptors. (B) The resulting spatiotemporal stimulus impinging on a region of the retina (top) and on a single photoreceptor (bottom). Note the luminance fluctuations in the input signal. (C–E) Characteristics of eye movements during free-viewing of natural scenes. Average distributions of saccade amplitudes (C), ocular drift speeds (D), and lengths (E) over N = 14 observers. The insert in C shows the range of microsaccades. Data in E represent the arc lengths of the trajectories followed by the eye during the intersaccadic periods. Black triangles indicate means. Errorbars represent standard deviations.

To examine the impact of fixational instability on visual input signals, we reconstructed the spatiotemporal stimulus on the retina, i.e., the visual input resulting from scanning the image according to the subject's eye movements. We then quantified the characteristics of this signal via spectral analysis (Fig. 2).

Figure 2.

Fixational modulations of luminance equalize spatial power. (A) Spatiotemporal frequency content of the retinal stimulus during intersaccadic fixation. (B–C) Sections at several temporal and spatial frequencies. At nonzero temporal frequencies, the power of the retinal stimulus was constant over a wide range of spatial frequencies. Theoretical predictions from a Brownian motion model of eye movement (BM) are also shown. (D) Comparison between the power spectrum of the images observed by subjects (black line) and the power available in the form of temporal modulations in the retinal stimulus (the sum over all nonzero temporal frequencies) during drift (red) and during periods of fixation which included both drifts and microsaccades (blue). The dotted line represents the prediction of the Brownian motion model. Shaded areas in B–D represent one standard deviation.

Fig. 2A shows the results obtained during the intersaccadic periods of ocular drift. With spatial frequency held constant, power declined as a function of temporal frequency (Fig. 2B). However, with temporal frequency held constant at a nonzero value, power was equalized over a wide range of spatial frequencies (Fig. 2C). This was strikingly different from the spatial characteristics of the images displayed on the monitor: as is typical for pictures of natural scenes [23], the spectral density of the images in our database declined steeply with spatial frequency (k^-1.9; k denotes spatial frequency). In sum, the effect of fixational eye movements was to redistribute the spatial power of the scene across temporal frequencies so that the power available in the form of temporal modulations (the sum of all power at resolvable temporal frequencies) was constant up to approximately 10 cycles/deg (Fig. 2D).

These results were robust with respect to the method used for spectral analysis. Fig. 2C shows that equalization of spatial power also occurred when eye movements were modeled by the best-fitting Brownian motion process, a case in which the influence of eye movements can be expressed in closed form (see Supplemental Experimental Procedures (SEP)). Furthermore, very similar results were obtained during periods of fixation which also included microsaccades in addition to ocular drift (Fig. 2D). While the fixations recorded in these experiments were on average too short to allow spectral estimation below 4 Hz, power equalization could be assessed down to 1 Hz when observers were instructed to maintain prolonged fixation on a single point in the image. In all cases, fixational eye movements attenuated the low spatial frequencies, which predominate in natural images, yielding a retinal stimulus with “whitened” spatial distributions in the temporal domain. This transformation is significant for neural coding, as it occurs within the range of peak spatiotemporal sensitivity of neurons in the retina and lateral geniculate nucleus [24-26] (i.e., 0.1-10 cycles/deg and above 2 Hz).

To determine the factors responsible for the spatial whitening, we investigated the dependence of input signals on the probability distribution of eye movement. Under the assumption that fixational eye movements are independent of the observed image I, the spectrum of the retinal stimulus is given by:

$$S(k,\omega )=I(k)Q(k,\omega )$$ (1)

where I(k) represents the spectral density of the image, and Q(k, ω) is the Fourier transform of the probability, q(x,t), that the eye moves by x in an interval t (SEP). This equation provided excellent approximation of the input power spectrum estimated in the experiments (Fig. 2D). It shows that the spectral distribution of the retinal stimulus depends both on the image and the way the eye moves.

The probability density function of ocular drift and its frequency distribution are reported in Figs. 3A and B, respectively. For a wide range of nonzero temporal frequencies, Q(k,ω) increased proportionally to the square of the spatial frequency. This dependence counterbalanced the spectral distribution of natural scenes (I(k) in Eq. 1), yielding a retinal stimulus whose power did not depend on spatial frequency. Thus, whitening of the retinal stimulus originated from the interaction between the statistics of natural images and the spatiotemporal structure of fixational eye movements.

Figure 3.

Whitening of the retinal stimulus depends on the characteristics of natural images and fixational eye movements. (A) Probability distribution q(x,t) of eye movement during intersaccadic fixation. The color of a pixel at coordinates (x,t) represents the probability (in log scale) that the eye moved by x in an interval t. (B) Frequency content of q(x,t). For a wide frequency range, energy increased proportionally to the square of spatial frequency (dashed lines), a dependence that counterbalanced the power spectrum of natural images. (C–D) Whitening was lost with presentation of images which lacked scale-invariance and was greatly attenuated when eye movements were artificially enlarged. In (C), natural images were filtered to yield various spectral densities: the red curve is the unfiltered case, and corresponds to the scale-invariant spectrum. In (D), eye movements were scaled by different factors: the red curve, a scale factor of 1, corresponds to normal eye movements. Each curve represents the sum over all nonzero temporal frequencies. Shaded areas represent one standard deviation.

Eq. 1 predicts that whitening of the retinal stimulus by normal fixational instability will only occur during viewing of images with approximately scale-invariant spectral density (I(k) ∝ k⁻²), like natural images. Fig. 3C tests this prediction. It shows the input temporal power measured when the same eye movement traces scanned images whose power spectra declined according to other power-law functions (I(k) ∝ k^−α, for α = 3,4,5). No power equalization occurred with these images. Whitening was also greatly attenuated when images of natural scenes were scanned by abnormally large eye movements (Fig. 3D). This happened because normal eye movements only counterbalanced the spectral density of natural scenes up to approximately 10 cycles/deg (Fig. 3B). Artificial amplification of eye movements shifted this whitening range to lower spatial frequencies, away from the region in which ganglion cells have their greatest sensitivity. Thus, fixational eye movements match the characteristics of natural scenes to the sensitivity of retinal neurons.

The data in Figs. 2 and 3 show that fixational eye movements carry out a specific conversion of spatial patterns into temporal modulations. What are the implications of this reformatting of the retinal input for the neural encoding of visual information?

To gain some intuition, it is convenient to decompose the stimulus on the retina into a sum of two separate signals (Fig. 4A): a static image, in which the intensity of each pixel represents the average luminance experienced by a retinal receptor over the fixation period; and a dynamic signal containing the changes in luminance caused by eye movements. These two signals have very different correlation properties. The static signal corresponds to the power in the retinal stimulus at zero temporal frequency. Most of this power is at low spatial frequencies: correspondingly, pairs of pixel have extensive luminance correlations. In contrast, the dynamic signal corresponds to the power which eye movements shifted to nonzero temporal frequencies and equalized over a broad range of spatial frequencies. Since equalization of spectral power is equivalent to elimination of second-order correlations, this signal no longer contains the long-range correlations attributable to the average second-order statistics of natural images. The surviving correlations are specific to the scene being observed and emphasize the luminance discontinuities that occur at edges and object boundaries.

Figure 4.

Implications of spatial whitening for neural encoding. (A) The retinal stimulus during fixation can be decomposed into a blurred image (the power at temporal DC) and the spatially decorrelated modulations caused by eye movements (see Movie S1). (B–C) Influences of eye movements on the response characteristics of parvocellular (B) and magnocellular (C) retinal ganglion cells. Contrast sensitivity functions measured with immobile retinas [24, 31, 32] (Static) are compared to equivalent functions which incorporate the effect of the redistribution of input power caused by eye movements (FEM). The two panels on the right show spatial sections at the original peak temporal frequencies (dashed lines). (D) Snapshots of activity in an array of computer-simulated transient neurons during a recorded trace of fixational eye movements (FEM) and when the same scene was displayed without retinal image motion (Static). The intensity of each pixel represents the mean instantaneous firing rate of a neuron with receptive field centered at the pixel coordinates. Notice the enhancement of important edges in the scene during fixational instability. See Movie S2 for full simulations of neural activity. (E) Neurons were modeled as linear filters replicating the spatial and temporal contrast sensitivity of a ON-center parvocellular cell recorded in the macaque's lateral geniculate nucleus [26].

Both parvocellular and magnocellular retinal ganglion cells possess peak sensitivity at nonzero temporal frequencies and will, therefore, be strongly influenced by the dynamic component of the visual input. Neurons with transient responses —which are not sensitive to zero temporal frequency— will effectively see a spatially whitened input. Their responses will, therefore, be less influenced by low spatial frequencies and enhance high spatial frequencies significantly more than expected from their contrast sensitivity functions measured in neurophysiological recordings with immobile retinas (Fig. 4B-C; see SEP). More sustained neurons will be driven by both the static and dynamic signals. Their tonic responses will depend on the local mean luminance conveyed by the static signal. Superimposed on this, phasic modulations in their responses driven by the dynamic, spatially-whitened component of the visual input will emphasize high spatial frequencies. These synchronous modulations are likely to elicit strong responses in cortical neurons [27, 28]. Interestingly, in physiological recordings, these modulations would be mistaken for correlations arising from neural noise, unless careful experimental measurement of microscopic eye movements was undertaken [10]. They are instead generated by the jittering retinal stimulus itself.

These surprising effects were demonstrated by computer simulations of populations of transient parvocellular neurons. Whereas a static (no eye movements) flash of a natural image led to spatially extensive pools of coactive neurons, neural responses during viewing of the same image with normal fixational eye movements emphasized object contours (see Movie S2). This pattern of activity occurred even though the receptive fields of model neurons were circularly symmetric and did not possess a preference for oriented stimuli. Thus, fixational eye movements facilitated the extraction of important features in the scene.

The observed whitening of the retinal input may provide an explanation for the lack of perceived blurring during fixational instability [29]. With normal eye movements, a predominance of low spatial frequency power only occurs at zero temporal frequency (the static component of the visual input in Fig. 4A), a range in which retinal ganglion cells are only marginally responsive. At nonzero temporal frequencies, fixational eye movements enhance high spatial frequency power, an effect which also supports a contribution of fixational eye movements to the perception of fine spatial detail [12, 15] and explains the perceptual impairments reported in the absence of retinal image motion [16]. Furthermore, these results radically alter standard views about the goals of early neural processing in the visual system. While center-surround interactions [17-19] and nonlinearities [30] in retinal neurons might contribute to spatial decorrelation in ganglion cells with sustained responses, transient neurons are driven by signals that are already spatially whitened. Thus, the band-pass spatial sensitivity of transient neurons actually introduces correlations in their responses, which further emphasize edges in the image.

In sum, our findings show that the incessant fixational motion of the eye reshapes the spatiotemporal stimulus on the retina into a format that facilitates subsequent neuronal processing. This transformation takes advantage of the temporal tuning of visual neurons by (a) spreading the energy of the image into the temporal domain, and (b) equalizing the resulting spatial distribution within the range of peak neuronal sensitivity. Much more than just a means for refreshing neural activity, fixational eye movements are a crucial mechanism for encoding a spatial sensory domain in time before any neuronal processing occurs.

Experimental Procedures

Fourteen naive subjects with normal vision participated in the experiments. Subjects were instructed to memorize 60 gray-scale pictures of natural scenes. Images were displayed sequentially, each for 10 s, on a Iyama HM204DT CRT monitor at 1024×768 pixel resolution and 100 Hz refresh rate. Each pixel subtended 1′, an angle similar to that covered when the image was originally acquired. Stimuli were observed monocularly, with the left eye patched, in a dimly illuminated room.

Eye movements were recorded by means of a Dual Purkinje Image eyetracker (Fourward Technology) and sampled at 1 Khz. A head rest and a custom dental-imprint bite bar prevented head movements. Recorded traces were segmented into periods of fixation and saccades based on eye velocity. Movements with speeds larger than 3°/s and amplitudes exceeding 3′ were classified as saccades or microsaccades depending on whether their amplitudes were larger or smaller than 30′. Saccade amplitude was defined as the length of the vector connecting the two locations at which velocity became greater (onset) and lower (offset) than 2°/s.

For each period of fixation, we created a movie of the retinal stimulus by translating the image following the recorded eye trajectory, so that each frame was centered at the current gaze location. Power spectra of these movies were evaluated by means of Welch algorithm with 50% block overlap (window lengths: 512 ms and 256′). Spectra were estimated separately for each subject and then averaged. 3D power spectra were also rotationally averaged across all directions in each spatial frequency plane (k_x,k_y) to yield 2D maps of spatial and temporal frequency. Data in Fig. 2 were obtained by selecting fixations longer than 512 ms. Results were virtually identical (other than a lower temporal resolution) when fixations longer than 128 ms were selected for analysis.

Highlights.

• In natural viewing, microscopic eye movements decorrelate (i.e., whiten) the visual stimulus.

• Whitening is due to a match between natural scene characteristics and fixational eye movements.

• Elimination of input redundancy starts before any neural processing takes place.

• In a continually moving eye, the process of edge extraction starts in the retina.