High temporal precision for perceiving event offsets

Abstract

Characterizing the temporal limits of the human visual system has long been a central goal of vision research. Spanning three centuries of research, temporal order judgments have been used to estimate the temporal precision of visual processing, with nearly all the research focusing on onset asynchrony discriminations. Recent neurophysiological work, however, demonstrated that neural latencies for stimulus offsets are shorter and less variable than those following event onsets, suggesting that event offsets might provide more reliable timing cues to the visual system than event onsets. Here, we tested this hypothesis by measuring psychophysical thresholds for discriminating onset and offset asynchronies for both stationary and moving stimuli. In three experiments, we showed that offset asynchrony thresholds were indeed consistently lower and were less affected by stimulus variations than onset asynchrony thresholds. These findings are consistent with neurophysiology and suggest a possible role of offset signals as reliable timing references for visual events.

1. Introduction

One of the longstanding aims of vision research has been to establish both spatial and temporal limits of visual perception. While spatial acuity currently receives a more thorough treatment in the literature, better characterization of temporal limits is equally important, particularly when it comes to advancing our understanding of visual function in dynamic environments. This was recognized by Exner (1875) who studied observers' ability to make fine temporal order discriminations, reporting thresholds of only 44 ms for overlapping stimuli. These thresholds decreased to just 17 ms when the stimuli were spatially displaced – a stimulus condition perceived as apparent motion. Subsequent research by Sweet (1953) showed that further optimization of stimulus parameters, including peripheral presentation, can decrease temporal order discrimination thresholds for two light flashes to 5 ms or less. Later work was able to match this performance in the fovea, but the measurements were still confounded by motion cues (Westheimer & McKee, 1977). In sum, studies exploiting motion cues to aid temporal order discriminations yielded asynchrony thresholds between 2 and 5 ms (Sweet, 1953; Westheimer & McKee, 1977; Wehrhahn & Rapf, 1992). Remarkably, for spatially overlapping color targets, detection of temporal order asynchronies of less than 1 ms has been demonstrated (Yund & Efron, 1974).

While impressive, these highly precise temporal order discriminations might not accurately reflect the true temporal limits of the visual system. As many of the aforementioned papers readily admit, these measurements are confounded by non-temporal cues, such as apparent motion caused by asynchronous stimulus onsets (e.g., Westheimer & McKee, 1977) and visual persistence of the trailing target (e.g., Yund & Efron, 1974). To directly address these confounds, Zanker & Harris (2002) investigated the temporal precision of vision in a task that is uncontaminated by non-temporal information. In their task, observers were asked to indicate whether the middle of three LED flashes was closer to the first or third flash – essentially, a temporal analogue of the spatial bisection task. The thresholds measured in this task, as well as supporting experiments, were all contained in the 20 – 40 ms range. These experiments likely reflect a more pure estimate of temporal precision in the human vision, and are comparable to the temporal resolution estimates obtained by flicker-fusion measurements (Kelly, 1972).

The neural correlates of psychophysically estimated temporal precision are most likely contained in the timing of the neural responses to abruptly occurring stimuli, specifically the variability of neural latencies to changing visual input. It is important to note that these changes in neural activity can occur in response to both onsets and offsets of environmental events. Despite this potential contribution of neural offsets, the bulk of the research has focused on onset latencies (Nowak & Bullier, 1997). A notable exception is work by Bair and colleagues (2002), which examined both the magnitude and variability of onset and offset latencies for neurons in macaque LGN, V1, and MT/V5. For all studied neurons, the results revealed that onset latencies were actually greater than offset latencies. The results were most dramatic for V1 simple cells, whose offset latencies were 22 ms shorter than onset latencies. More importantly for the present study, offset latencies were both less dependant on stimulus parameters and, at times, less variable than onset latencies. This result suggests that stimulus offset responses may contain more precise information for determining the relative timing of environmental events – a hypothesis tested in the present paper.

A limited number of studies have examined this question in human observers. Kreegipuu and Allik (2007) showed that offset latencies for visually evoked potentials in response to moving stimuli are shorter than onset latencies – a result in accord with Bair et al. (2002). However, their concurrent reaction times analysis yielded the opposite result: reaction times for the detection of motion onset were shorter than those for motion offset. The authors do not offer an explanation of this discrepancy, indicating that future work is necessary. Yund and Efron (1974) examined temporal order discriminations for both onset and offset asynchronies and found that offset discriminations were more precise, with offset asynchrony discrimination thresholds less than 0.5 ms. This remarkable performance is most likely due to the visual persistence of the target that offset last – i.e., it likely does not reflect the true temporal precision of vision (Zanker & Harris, 2002).

The paucity of behavioral research examining the precision of offset discriminations cannot be justified by their limited ecological validity. Given the continuity of our visual experience, event onsets are paired with event offsets. For example, the appearance of an occluding object is paired with the disappearance of the occluded object. Dynamic visual scenes involve frequent appearances and disappearances of objects, particularly at a local receptive field level. If we consider this environmental pairing of onsets and offsets with neurophysiological findings demonstrating shorter latencies for event offsets (Bair et al., 2002), it can be argued that, within our visual system, event offsets are the first sign of change (Clifford, 2002).

In the present study, we investigated temporal precision for perceiving stimulus onset and offset asynchronies under various stimulus conditions, using methods that should be less affected by non-temporal cues present in many previous studies. The results revealed that offset asynchrony thresholds were consistently lower and were less affected by stimulus variations than onset asynchrony thresholds.

2. General Methods

Stimulus patterns were created in MATLAB with the Psychophysics Toolbox (Brainard, 1997) and Video Toolbox (Pelli, 1997), and were shown on a linearized monitor (800 × 600 resolution, 200 Hz). To achieve a high video refresh rate, we used a high-speed PROCALIX monitor (Totoku, Irving, TX) driven by a MP 960 graphics card (VillageTronic, Berlin, Germany). This combination provided a 200 Hz frame rate, allowing us to measure temporal asynchrony thresholds with high temporal precision. Viewing was binocular at 83 cm (yielding 2 × 2 arcmin per pixel). A fixation cross was continuously presented in all experiments. Fixation was aided with a chin/forehead rest. Ambient illumination was 3.4 cd/m². Background luminance was 42.3 cd/m². Four observers participated in the study (author DT, who was an observer in all experiments, and three naive individuals). All procedures complied with institutionally reviewed guidelines for human observers.

3. Experiment 1: detection of stimulus onset and offset asynchronies

The aim of Experiment 1 was to measure temporal acuity for detecting stimulus onset and offset asynchronies and their dependence on spatial separation between stimuli.

3.1. Methods

Three observers participated in Experiment 1. Stimuli were two horizontal Gabor patches (2σ = 1°, spatial frequency = 1.25 cycles/°, temporal frequency = 0 Hz, contrast = 73%; spatial phase randomly selected on each trial). Gabor patches were presented on each side of the fixation cross, with the center-to-center separation varying between 3.3° and 20° in different conditions.

In the offset condition, each trial was initiated by the observer, which was then followed 880 – 1120 ms later (duration randomly drawn from a uniform distribution) by the onset of both Gabor patches. Both Gabor patches were shown for 880 – 1120 ms, at which point one of the Gabor patches disappeared, while the other Gabor patch remained on the screen for a brief period of time (between 5 and 65 ms, with the exact value depending on the observer and the condition), creating an offset asynchrony (Figure 1). In the onset condition, the sequence of Gabor events was the same except that the stimulus asynchrony was present in the stimulus onset (i.e., the sequence of Gabor stimuli shown in Figure 1 was presented in reverse). The range of asynchronies in the onset task varied between 5 and 110 ms. Observers' task was to indicate which Gabor patch appeared/disappeared first by pressing one of two keys. Auditory feedback followed correct responses. For both onset and offset asynchrony conditions, we tested 3 spatial separations, yielding a total of 6 conditions.

Fig. 1.

A schematic illustrating the time course of a single offset asynchrony trial. Observers' task was to indicate which Gabor patch disappeared first by pressing a designated key (corresponding to the left Gabor in this example). For the purposes of this illustration, the duration of the offset asynchrony is exaggerated. See Experiment 1 Methods section for additional details.

Observers first performed 5 blocks of 72 trials at different onset/offset asynchronies for each condition (2160 trials per observer). These sessions served both as task practice and as a way to select the appropriate range of stimulus asynchronies for the main experiment. In the main experiment, each observer was tested at 6 different asynchronies, again completing 5 blocks of 72 trials per condition (2160 trials per observer). Note that use of a 200 Hz monitor allowed 5 ms temporal spacing of stimulus asynchronies. Resulting data was fit with a cumulative normal function. To obtain just noticeable difference (JND) estimates, we computed the difference between asynchrony values yielding 75% and 25% of the decisions that the right Gabor patch was first (dashed lines in Figure 2A) and dividing the resulting value by 2.

Fig. 2.

Results of Experiment 1. (A) Psychometric functions for an observer (DT), showing the percentage of trials in which the right Gabor patch was identified as appearing/disappearing first as a function of temporal asynchrony in the condition where the spatial separation was set to 3.3°. Solid and empty symbols show results for offset and onset asynchrony trials, respectively. Data were fit with a cumulative normal function, from which we estimated the JND value by computing the half-difference between asynchrony values yielding 75% and 25% points. (B) Group data showing JNDs for onset and offset asynchrony discriminations as a function of spatial separation. Error bars are SEM.

3.2. Results

Figure 2A shows data and fits for onset and offset asynchrony discriminations of Gabor patches separated by 3.3° for one observer. The JND for the offset task (5.4 ms) was about half that for the onset task (12.4 ms). This pattern of results was observed across observers and over the 6-fold range of separations tested in this study (Figure 2B). Increases in the spatial separation yielded an increase in JNDs for both tasks, with a larger absolute increase for onset discriminations (22 ms vs. 9 ms). Note that this approximately two-fold increase in onset JNDs with increasing separation preserved the ratio between onset and offset results: averaged over all separations, onset JNDs were about 120% higher than offset JNDs.

4. Experiment 2

Experiment 1 examined temporal order discriminations for pairs of static stimuli. In Experiment 2, we measured temporal acuity for detecting motion onset and offset asynchronies and their dependence on the relative direction of target stimuli. We focused on motion because of ecological importance of detecting motion onsets and offsets – e.g., noticing when a moving object suddenly stops.

4.1. Methods

Four observers participated in Experiment 2, including three that participated in Experiment 1. Stimuli were two horizontal Gabor patches (2σ = 1°, spatial frequency = 1.25 cycles/°, temporal frequency = 6 Hz, contrast = 73%, initial phase was randomly selected). Gabor patches were presented on each side of the fixation cross, with 4° center-to-center separation.

Each block of trials started with a presentation of two stationary Gabor patches. In the offset condition, each trial was initiated by the observer, followed 650 – 850 ms later by simultaneous onset of vertical Gabor motion. Motion continued for 650 – 850 ms, at which point one of the Gabor patches stopped moving, while the other Gabor patch continued moving for a brief period of time (between 10 and 100 ms), creating a motion offset asynchrony. Here, only the grating pattern within each Gabor patch moved, i.e., its spatial envelope was always stationary. After the trial ended, stationary Gabor patches remained on the screen until the next trial was initiated, with their spatial phase matching the last frame of motion. In the onset condition, the sequence of events was the same except that the asynchrony occurred at the motion onset. The range of asynchronies in the onset task varied between 10 and 120 ms. For both onset and offset asynchrony conditions, we investigated conditions where Gabor patches were moving in the same (both up or both down) or in the opposite directions (one up, one down), yielding a total of 4 conditions. Each condition was tested in a separate block. In order to prevent a build-up of the motion after-effect, each Gabor's motion direction was reversed on subsequent trials. This also ensured that stimulus motion direction was fully predictable from trial to trial. The observers' task was to indicate which Gabor patch started/stopped moving first by pressing one of two keys. Auditory feedback followed correct responses.

As in Experiment 1, observers first performed 5 blocks of practice trials at different onset/offset asynchronies for each condition (1440 trials). In the main experiment, each observer was tested at 6 different asynchronies, again completing 5 blocks of 72 trials per condition. Resulting data was fit with a cumulative normal function, from which we estimated JND value (see Figure 2A).

4.2. Results

Figure 3A–B shows one observer's data for motion onset (A) and offset (B) asynchrony discriminations of Gabor patches moving either in the same or opposite directions, indicating that the relative motion direction had a larger effect on onset than on offset discriminations. The same pattern of results is apparent in the group data, with the offset and onset JNDs increasing by 30% and 55% when the Gabor targets moved in the opposing directions (Figure 3C, interaction: F(1, 3) = 9.1, p = 0.057). As in the Experiment 1, offset discriminations were more precise than onset discriminations (F(1, 3) = 15.1, p = 0.03) although the motion offset advantage was considerably smaller (36% in Experiment 2 vs. 120% in Experiment 1).

Fig. 3.

Results of Experiment 2. Psychometric functions for an observer (DT) showing results in the offset (A) and (B) onset asynchrony task. Circles show the data for the conditions where target Gabors moved in the same direction. Triangles depict the data for the conditions with target Gabors moving in the opposing directions. (C) Group data showing JNDs for onset and offset asynchrony discriminations as a function of relative direction of target Gabor patches. Error bars are SEM.

5. Experiment 3

In the first two experiments, we investigated temporal order discriminations for pairs of stimuli that differed only in their timing and spatial location – i.e., other stimulus parameters, such as contrast, were identical. In naturalistic environments, however, temporal order judgments typically involve stimuli that are considerably more heterogeneous. In such situations, a reliable timing cue is, therefore, one that is less affected by stimulus variations. Motivated by this particular feature of real world stimuli, we measured the effect of stimulus heterogeneity on onset and offset judgments. Specifically, we measured temporal acuity for detecting stimulus onset and offset asynchronies and their dependency on variations in contrast and/or spatial frequency.

5.1. Methods

Three observers who participated in Experiment 1 also participated in Experiment 3. Stimuli were eight horizontal Gabor patches (2σ = 1°, temporal frequency = 0 Hz). The Gabor patches were evenly distributed along an imaginary circle with a 6° radius, whose center was the fixation cross (Figure 4A). Four task-irrelevant Gabor patches presented along oblique axes were continuously presented and were fixed in contrast (29%) and spatial frequency (1.265 cycles/°), while their spatial phase was randomly chosen on each trail. These Gabor patches were added to increase visual clutter (and thus better mimicking real world scenes) and to minimize apparent motion associated with asynchronous onsets or offsets of target Gabor patches. The remaining four task-relevant Gabor patches were presented on each trial along cardinal axes and were, in some conditions, varied in contrast and/or spatial frequency.

Fig. 4.

Results and stimuli of Experiment 3. (A) Screen snapshots of stimuli used in Experiment 3. The top right panel is the stimulus in the homogeneous condition. The panels to the right and below illustrate ±0.5 log unit random variations in contrast and spatial frequency, respectively. The bottom right panel shows a stimulus varied in both contrast and spatial frequency. (B) Group data showing temporal asynchrony thresholds for onset and offset asynchrony discriminations as a function of contrast and/or spatial frequency variation. (C) “Threshold cost” for onset and offset asynchrony discrimination across contrast, spatial frequency, and joint contrast + spatial frequency variation conditions. Threshold cost was calculated by subtracting the onset and offset thresholds for the homogeneous stimulus (the leftmost points in B) from the thresholds estimates for the conditions with ±0.5 log unit variation (the rightmost points in B). Error bars are SEM.

In the offset condition, each trial was initiated by the observer, followed 880 – 1120 ms later by the onset of all four task-relevant Gabor patches. These stimuli were presented for 880 – 1120 ms, at which point one of the task-relevant Gabor patches disappeared, while the other task-relevant Gabor patches remained on the screen for a brief period of time (between 10 and 100 ms), creating an offset asynchrony. This sequence of events is a four Gabor analogue of that shown in Figure 1. In the onset condition, the sequence of task-relevant Gabor events was the same except that the stimulus asynchrony was present in the stimulus onset – one task-relevant Gabor patch appeared before the other three. The observers' task was to indicate which task-relevant Gabor patch appeared/disappeared first by pressing one of four keys. Auditory feedback followed correct responses.

Each observer completed 7 onset and 7 offset asynchrony conditions. In the baseline (homogeneous) condition, task-relevant Gabors had the same contrast (29%) and spatial frequency (1.265 cycles/°) as the task-irrelevant Gabors presented along the oblique axis (Figure 4A, top left corner). In the remaining 6 conditions, task-relevant Gabors were varied in spatial frequency (2 conditions), contrast (2 conditions) or both contrast and spatial frequency (2 conditions). Spatial frequency and/or contrast variation was introduced by sampling from a uniform distribution defined as ± 0.5 or ± 0.25 log units from the anchor values of 1.265 cycles/° and 29%. For example, 1.265 cycles/° ± 0.5 log units yielded spatial frequencies between 0.4 and 4 cycles/°.

Observers first performed 5 blocks of 72 trials at different onset/offset asynchronies for each condition (5040 trials per observer). These sessions served both as task practice and as a way to select the appropriate range of stimulus asynchronies for the main experiment. In the main experiment, each observer was tested at 6 different asynchronies, again completing 5 blocks of 72 trials per condition (5040 trials). The resulting data was fit with a Weibull function, with the temporal asynchrony yielding 72.4% correct performance taken as the threshold.

5.2. Results

Consistent with Experiment 1 findings, Experiment 3 results revealed an approximately two-fold difference between onset and offset asynchrony thresholds (Figure 4B, F(1, 2) = 152.7, p = 0.007). For both onset and offset detections, introduction of variability in target contrast and/or spatial frequency resulted in increased thresholds (F(2, 4) = 33.8, p = 0.003). Importantly, the effect of stimulus heterogeneity differed in the onset and offset conditions (F(2, 4) = 18.5, p = 0.01), Specifically, this threshold increase was larger in the onset task for two conditions where contrast was varied (Figure 4C), while approximately equal increase was observed in the condition where spatial frequency was the only stimulus feature varied.

6. Discussion

In three experiments, we found that observers exhibit better temporal sensitivity to asynchronous event offsets than to asynchronous event onsets. Additionally, stimulus variations, including spatial separation, motion direction and contrast variability, had a larger disruptive effect on onset discriminations than on offset discriminations. These results are consistent with neurophysiological findings indicating that offset latencies are less variable than onset latencies (Bair et al, 2002), and support our hypothesis that event offsets may provide more reliable timing cues to the visual system than event onsets. These results are in accord with one past study that contrasted onset and offset asynchrony (Yund & Efron, 1974), although their offset asynchrony thresholds were compromised by color mixing cues. Analogous results have been reported for auditory onset and offset discrimination: Yund and Efron (1974) reported that for both monaural and dichotic stimuli pairs, observers demonstrated better sensitivity to offset asynchrony sensitivity than onset asynchrony. Pastore (1983) confirmed these auditory findings, with the additional result that absolute thresholds were tied to stimulus duration. Taken together, these visual and auditory findings suggest that the more precise temporal encoding of offset timing might be a common property of sensory systems, suggesting a general mechanism for determining the relative timing of events.

In our experimental design, we took several precautions to eliminate possible contamination due to apparent motion cues associated with asynchronous onsets or offsets of Gabor patches. In other words, we wanted to ensure that the observers were not relying on the direction of apparent motion to make temporal order judgments. Specifically, in two experiments, target stimuli were horizontal gratings presented along the horizontal meridian, minimizing apparent motion. Moreover, spatially separating stimulus targets by up to 20° (Exp 1), measurements of motion onsets and offsets where moving stimuli were both preceded and succeeded by static Gabor patches (Exp 2), and continuous presentation of task-irrelevant Gabor stimuli spaced out between task-relevant targets (Exp 3) all served to minimize any corruption by motion information. This conclusion was confirmed by subjective impressions of naïve observers who reported not seeing apparent motion. Indeed, the absolute thresholds measured here are comparable to the “pure” temporal precision measurements reported by Zanker and Harris (2002), and much higher than those measured in studies where apparent motion cues were present (Exner, 1875; Sweet, 1953; Westheimer & McKee, 1977; Wehrhahn & Rapf, 1992). It is important to note that the presence/absence of apparent motion was not the only difference among these studies, so it is possible that other stimulus and/or task differences contributed to variations in temporal precision measurements.

Our results raise two related questions: Why is vision more sensitive to small temporal asynchronies in event offsets, and what are, if any, possible functional implications of such a result? As stated above, our psychophysical results are in accord with the neurophysiological findings of Bair et al. (2002) who found that in macaque LGN, V1 and MT offset latencies to grating stimuli are shorter, less variable and less stimulus-dependent than onset latencies. While it is very intuitive to think of the neural response onsets as the neural correlates of event timing, response offsets can be just as informative: most abrupt visual changes will result in both neural response onsets of neurons preferring the new stimulus, and response offsets of neurons losing their preferred stimulus. Given that response offsets have shorter latencies and are less dependent on stimulus parameters, it is possible that neural offsets are exploited as both a faster and more reliable timing reference signal (Bair et al., 2002; Clifford, 2002). Still unknown, however, are mechanism(s) that underlie this greater utility of offset signals. One possibility is contrast normalization – a mechanism that maximizes neural sensitivity to the prevailing level of visual stimulation (Heeger, 1992). Thus, contrast normalization should increase sensitivity to visual events that involve abrupt disappearances of objects, while it is less helpful in situations where an object abruptly appears on a featureless background. This hypothesis predicts a larger difference between onset and offset discriminations in cases where onsets and offsets involve appearances and disappearances of stimuli then for motion onsets and offsets of an otherwise stationary stimulus. Indeed, that seems to be case in our results (compare Experiment 2 results with those from Experiments 1 and 3).

Finally, it is important to consider limitations of our findings. We measured the temporal precision with which observers can make onset and offset asynchrony judgments. These results, however, can only determine the accuracy of onset and offset discriminations, and not differences between onset and offset latencies. It will be important for future research to investigate whether psychophysically estimated differences between onset and offset latencies are consistent with single neurons results reported by Bair et al. (2002). A second limitation pertains to the difficulties in defining onsets and offsets in a psychophysical experiment. While it is relatively trivial to determine which stimulus changes result in response onsets and offsets for a particular neuron (given that visual neurons are usually tuned to particular stimulus characteristics), determining whether a psychophysical event is an offset or an onset is not always straightforward. For example, an abrupt appearance of a Gabor patch on a gray field (as in our Experiment 1) better characterized as an onset event (Gabor patch onset) than an offset event (gray field offset). However, if the visual event consists of a stationary object that abruptly starts moving (as in our Experiment 2), its characterization as an event onset seems less certain because it could also be labeled as an offset of a stationary stimulus. We speculate that this ambiguity might be another reason why the differences between onset and offset results in the motion onset/offset experiment (Figure 3) are considerably smaller that those measured with appearing/disappearing stimuli (Figure 2).

In summary, in a series of psychophysical experiments we demonstrated that observers exhibit higher temporal precision for perceiving event offsets than event onsets. These results are in accord with recent neurophysiological observations, and suggest a possible functional relevance of offset signals as a timing reference for visual processing.