Stereopsis for rapidly moving targets

Abstract

Stereoscopic depth perception is possible with luminance-defined target velocities at least as high as 600° s⁻¹, up to the limit of 30 Hz imposed by the high-temporal frequency cut-off of the eye. The limitation for perceiving depth from stereo disparity of moving targets is not their velocity but the temporal frequency bandwidth of the eye, which is affected by adaption state. Stereoacuity for a depth shift in a horizontally moving grating depends not on spatial disparity between corresponding luminance points in spatial units of arc min, but on the spatial shift as a fixed proportion of the period of the grating, in other words, on the phase angle difference between the two eyes, as is also the case for obliquely orientated, stationary gratings. Phase differences explain not only the classic Pulfrich stereophenomenon but its equivalent with dynamic visual noise, and a new effect in which depth results from interocular phase differences in luminance modulation.

This article is part of a discussion meeting issue ‘New approaches to 3D vision’.

Jenny Read's fascinating contribution to this volume makes it clear that prey catching in a variety of animals could involve stereopsis. But stereopsis for a rapidly moving target raises several problems, not least the problem of correspondence, given the relatively sluggish temporal response of photoreceptors. The typical impulse response of cones (their response to a brief flash of light) rises gradually to a peak, and then subsides. Crucially, the time to reach the peak depends on the adaptation states of the receptor and thus on the background light level. The time to peak can be anything from tens of milliseconds at high light levels to hundreds of milliseconds at low levels [1,2]. This means that a fast-moving target will leave behind it a trail of activity without any precise location. Nevertheless, human stereoacuity thresholds for periodic gratings (vertically oriented, horizontally moving from left to right) is not degraded by velocities up to 640° s⁻¹ provided that their temporal frequency does not exceed 30 Hz [3]. The spatial frequency of a grating is the number of cycles of the grating in a degree of visual angle. The temporal frequency is the number of bars of the grating that will pass a fixed point in a second. If we measure the luminance of the grating at a fixed point as it passes, the luminance will cycle sinusoidally. The angle of the sinusoidal modulation is referred to as the phase angle. If there is a disparity between the gratings in the two eyes this will produce a difference in phase angle, which can have a maximal value of 180°. It turns out that the best way of characterizing the limits of stereopsis for gratings so that it is independent of spatial frequency and velocity, is the phase angle difference between the eyes. This comes out at about 5°, which can—depending on spatial frequency and velocity—correspond to an interocular temporal delay of as little as 450 µs [3].

Although the approximately 5° phase difference limit has been measured for periodic sinusoidal gratings, the same principal can be presumed to apply to a single moving bar, or any other stimulus that can be decomposed by Fourier analysis into its component sinusoids. With a square wave grating, for example, it is likely that the upper temporal frequency limit would be the same as for its lowest frequency component (the fundamental), and would be in the region of 30 Hz, depending on adaptation level.

It may be helpful to clarify that the upper temporal frequency limit we are discussing here is for luminance-defined targets such as moving sine-wave gratings, not the limit for temporal modulations in disparity itself, which are much lower, for example, the 6 Hz limit for oscillation in depth of a grating defined by a random-dot stereogram [4], or the 8 Hz limit for a moving triangle-wave modulation of disparity, also in a random-dot sterogram [5]. These much lower limits reflect the temporal frequency limits of the process underlying the computation of the interocular cross-correlation underlying the detection of disparity [5].

One dimensional motion is tilted in axes {x,t}. Similarly, if we consider stereoacuity for static gratings that are tilted in space, we have axes {x,y} and, once again, in this space the limit for stereoacuity is set by the phase angle difference, not just the horizontal component of the disparity. Over a range of tilts for a Gabor patch ranging from 0° to 80°, the stereoacuity threshold was found to be constant over tilt angle when expressed as a phase shift, though not when expressed as a horizontal disparity [6].

Stereopsis from interocular temporal delay is demonstrated by the Pulfrich [7] stereophenomenon, in which a neutral density filter over one eye causes the perceived position of a moving target to be shifted in depth. Pulfrich (who incidentally had only one functioning eye) accepted the explanation by the optical engineer Fertsch at Carl-Zeiss, that the filter caused a transmission delay in the covered eye by delaying the response of the photoreceptors in the covered eye. The delay meant that the covered eye was signalling an earlier position of the target along its trajectory, and thus an instantaneous disparity. Pulfrich quoted from Richard Wagner's ‘Parsifal’: ‘Du siehst mein Sohn zum Raum wird hir die Zeit’ (You see my son, that here time is changed into space).

The key word here is ‘instantaneous’. The instantaneous position of a moving object is an abstraction, and cannot be measured by any physical system, let alone the eye with its relatively sluggish response time in the millisecond order of magnitude. The neutral density filter complicates this by decreasing the retinal illumination and delaying the peak response still further, as the eye attempts to maintain sensitivity by integrating photons over time. The Pulfrich effect could thus be due to greater persistence of the signal in the covered eye, which would push its average position at any time back along the trajectory. The effect of differential persistence can be mimicked by making the target in one eye larger in one eye along the trajectory, but with its leading edge still aligned with the target in the other eye. This produces a depth effect, which can be cancelled by a neutral density in the opposite eye [8].

However, depth-from-delay can also be demonstrated without differential persistence, using a haploscope with control of the phase angle between the two eyes. Initially, this was done with continuously moving targets, but David Lee pointed out that a depth shift could also be seen with a stroboscopically illuminated target [9], where the target moves in a series of discrete steps. This raises a problem for the simple Fertsch–Pulfrich ‘time to space’ theory, because delay cannot alter the retinal position of the target, only the time taken for that position to be signalled to the brain. A possible explanation is that the brain would fuse the nth position of the target in the delayed eye with the (n − 1)th position in the other eye. To see if this was the explanation, Morgan & Thompson [10] used sinusoidally moving, stroboscopically illuminated targets and counteracted the depth effect caused by a ND filter with a phase difference between the eyes. Below stroboscopic sampling rates of about 30 ms, the equivalent temporal delay required to null the depth effect was less than half the sampling interval, contradicting the predictions of the ‘shifted pairing’ hypothesis. It thus appeared as if the relevant measure of disparity is not the position of the samples but the spatial disparity between the underlying motion trajectories. Using both a Vernier and a stereo method, Morgan [11] showed that targets in apparent (stroboscopic) motion appeared aligned when they had the same underlying motion trajectory, even if that trajectory were sampled at different times and places, provided that the temporal sampling interval did not fall below approximately 40 Hz (depending on adaptation state, [12]). Burr & Ross [13] elegantly confirmed this by measuring a depth effect from temporal delays of as little as 160 µs or 2 arcsec, in spatially corresponding positions.

Interpolation of the perceived position of a target in sampled motion would be expected in any system with a limited temporal frequency response. A moving target sampled at 1000 Hz could not in any way be distinguished by the visual system from one sampled at 100 Hz, provided the eye is not tracking the target, which was controlled in at least some experiments [11,14]. In Fourier terms a discretely sampled moving target contains the spatio-temporal spectrum of the underlying trajectory, with the addition of components arising from the sampling. The higher the sampling rate the greater the separation between the trajectory components and the sampling replicas [15]. If the sampling replicas are of sufficiently high frequency, the human visual system will not be sensitive to them, and only the underlying trajectory will be visible. However, if the replicas fall within the visible range of frequencies they will interfere with the original signal and interpolation will break down. This happens in human vision when the sampling rate falls below a critical value, in the region of 20 Hz at typical experimental background luminance levels. When interpolation fails, it can be restored by external filtering to remove the closest replicas, or by reducing the mean luminance of the signal (for a review of these and other methods see a previous article in this journal; [14]).

An exception to the interpolation rule is the little-noted report of perceived depth in a moving bar target that had the same motion trajectory in the two eyes [16]. The moving bar was subjected to a sinusoidally modulation of its luminance at approximately 10 Hz, with the modulation being out of phase in the two eyes. Thus, the instantaneous spatial positions of the bar was the same in the two eyes, but one eye was temporally lagging in its luminance modulation relative to the other. This luminance modulation produced a perceive depth shift of the moving bar, as if the temporally lagging bar (in its luminance modulation) were spatially leading its partner in the other eye. Once again, the threshold phase angle for seeing the effect was about 5°. To form a picture of this phenomenon, imagine that the targets had been photographed with the camera shutter open for several seconds. Each target would produce a sinusoidal grating with the phase-lagging grating shifted in the direction of motion (figure 1). The fused image would be seen as shifted in depth in the same direction as the previously moving target. Temporal smearing in the eye and brain means that it is a kind of storage device, so it is not entirely surprising that a phase lag in luminance modulation produces depth. A full model for the effect has yet to be developed (it would be a good student project), but it is easy to see by example that a mechanism sensitive to interocular phase would be affected by temporal luminance modulation. Consider, for example, a quadrature pair of receptive fields, in corresponding retinal positions, which would respond maximally to a 90° disparity of a stationary Gabor target [17]. This would also respond with greater amplitude to 90° out-of-phase temporal modulation than to in-phase (0°) (figure 2).

Figure 1.

Morgan & Watt [16] used separate moving bars to the left and right eyes in a dichoptic display. The movement vector of the bars is shown by arrows. These vectors were identical in the two eyes, and so was the instantaneous position of the bars. However, the luminance of the two bars was modulated sinusoidally so that that over time they would have traced out two sinusoidal gratings, with a phase difference between the eyes. Apparently, the brain was sensitive to the temporal modulation, because the phase difference produced an apparent depth shift in the moving bar. A pair of quadrature receptive fields in the two eyes [17] would respond with greater modulation depth to out-of-phase (90°) luminance modulation than to in-phase (0°). (Online version in colour.)

Figure 2.

The left-hand panel is an x,t plot of a single row (x) of the random-noise pattern evolving over time (t, vertical axis) with a sinusoidal modulation of its luminance. The right-hand panel shows the corresponding power spectrum, which contains only temporal frequencies of ±ω but a broad band of spatial frequencies, and thus of velocities. When such a stimulus is viewed dichoptically with a phase lag in the temporal modulation between the two eyes, it is seen to rotate in depth around a vertical axis (modified from [18]).

An interocular phase limit for stereopsis also appears using dynamic visual noise as a stimulus. Tyler [19] and Ross [20] reported that dynamic visual noise viewed with an interocular delay appeared to swirl around in depth like a transparent cylinder rotating around a vertical axis. At first sight this is mysterious since the noise does not appear to be horizontally moving until the delay is applied. But since the spatio-temporal spectrum of the noise is broad band, it must contain horizontally moving components, which could in theory be susceptible to stereo-tuned mechanisms. Horizontal movement of a horizontal Fourier component is invisible, so we can predict that noise containing only horizontal components will not support the stereophenomenon, while noise containing vertical components can. This was verified by Morgan & Tyler [21] using a cylindrical lens as a filter (a piece of shower stall, as it happens).

To measure the frequency dependence of the noise stereophenomenon, Morgan & Fahle [18] used band-limited dynamic visual noise composed of a checkerboard array, the elements of which were sinusoidally modulated in luminance over time, with the phase of each element randomly chosen from a uniform distribution (figure 1). The threshold phase angle for reporting the direction of rotation (clockwise or anticlockwise) was in the region of 5–10°, which corresponded to temporal delays in the millisecond region at frequencies of approximately 10 Hz. As in Morgan and Castet's experiment [3] there was an upper temporal frequency limit for the luminance modulation above which the depth effect could not be seen. In Morgan and Castet's experiment this was approximately 30 Hz; in the dynamic noise stereophenomenon it was lower: approximately 6 Hz in one observer and approximately 19 Hz in the other. The different limits in the two experiments is not remarkable. The dynamic visual noise stimulus contains many different spatio-temporal components and orientations, of which the relevant component for stereopsis is only a weak part. Measurements of contrast sensitivity for flickering gratings show a rapid fall off above 10 Hz. The observers in the dynamic visual noise experiment were presumably too insensitive to higher temporal frequency components to be able to use them for stereopsis.

Sensitivity to high-temporal frequencies increases with retinal illuminance. This is assumed to be because the speed of the phototransducation response increases with retinal illuminance [1]. It is clear from the temporal frequency limits on dynamic stereoacuity that prey catching will, therefore, be best at high levels of illumination. Hovering birds of prey, looking for the tiniest movement of their prey on the ground, prefer to hunt on sunny days. Temporal frequency limits also explain why bad light stops play in cricket. Bad light does not slow down the action of the bowler, but it does slow down the batsman's retina. The only remaining mystery is why some fielders persist in wearing fashionable sunglasses late into the evening.

Data accessibility

This article has no additional data.

Authors' contributions

M.M.: writing—original draft.

Conflict of interest declaration

I declare I have no competing interests.

Funding

Funding was provided by the Leverhulme Trust (grant no. RPG-2021-020).