Action and Perception Are Temporally Coupled by a Common Mechanism That Leads to a Timing Misperception

Abstract

We move our eyes to explore the world, but visual areas determining where to look next (action) are different from those determining what we are seeing (perception). Whether, or how, action and perception are temporally coordinated is not known. The preparation time course of an action (e.g., a saccade) has been widely studied with the gap/overlap paradigm with temporal asynchronies (TA) between peripheral target onset and fixation point offset (gap, synchronous, or overlap). However, whether the subjects perceive the gap or overlap, and when they perceive it, has not been studied. We adapted the gap/overlap paradigm to study the temporal coupling of action and perception. Human subjects made saccades to targets with different TAs with respect to fixation point offset and reported whether they perceived the stimuli as separated by a gap or overlapped in time. Both saccadic and perceptual report reaction times changed in the same way as a function of TA. The TA dependencies of the time change for action and perception were very similar, suggesting a common neural substrate. Unexpectedly, in the perceptual task, subjects misperceived lights overlapping by less than ∼100 ms as separated in time (overlap seen as gap). We present an attention-perception model with a map of prominence in the superior colliculus that modulates the stimulus signal's effectiveness in the action and perception pathways. This common source of modulation determines how competition between stimuli is resolved, causes the TA dependence of action and perception to be the same, and causes the misperception.

Introduction

Visual stimuli compete to be the target for an eye movement, but they also compete to be the object for perception. Action and perception should solve these competitions in a temporally coordinated fashion. Otherwise, we might make a movement away from an object before it can be perceived, or we might delay making a movement toward an object we need to see (Gibson, 1979). However, visual information for action and perception is processed in different areas (Ungerleider, 1982; Milner and Goodale, 2008). These areas are thought to interact (Schenk and McIntosh, 2010; Cloutman, 2013), e.g., motor processing is known to influence attention and thus perception (Kowler, 2011), but it is not known whether this interaction implies a temporal coupling between action and perception (Gibson, 1979; Warren Jr, 1990; Prinz, 1997; Goodale, 2001). Thus, whether how and where action and perception are temporally coupled remains to be elucidated. To address these questions, we studied temporal coupling between action and perception in the classic gap/overlap paradigm.

The time course of motor preparation has been extensively studied by measuring the saccadic reaction time (SRT) in a task with variable temporal asynchrony (TA) between the time of fixation point offset and the time of target onset. Saslow (1967) first described the gap effect, the dependence of SRT upon TA: compared with the synchronous condition, SRT is reduced when the fixation point turns off before the target turns on (gap condition), and is increased when the fixation point turns off after the target turns on (overlap condition).

The subject's perception in Saslow's task has not been reported in the literature. Although a motor dependence upon temporal asynchrony does not imply a perceptual dependence, the time and content of perception might also depend upon the temporal asynchrony. In that case, the temporal asynchrony dependencies of action and perception might be either loosely or tightly coupled. A tight coupling would suggest that temporal coordination occurs through a common neural mechanism.

We looked for temporal coupling in a gap/overlap dual-task in which subjects made saccades to targets appearing with variable temporal asynchrony, and also reported (by a button press) whether they perceived a gap or an overlap between the stimuli. Saccadic reaction time, perception report reaction time (PRT), and perception were analyzed as functions of temporal asynchrony. We found that saccade reaction and PRTs shared the same time-change dependence upon temporal asynchrony. We also found a novel temporal illusion: subjects misperceived small overlaps as gaps. This misperception is surprising, because in many experiments target onset and fixation point offset are synchronous, and the assumption has been that these events are perceived as simultaneous.

The tight temporal coupling between action and perception supports the hypothesis of a common structure coordinating their temporal asynchrony dependencies. We propose that a map of stimulus prominence in the SC modulates each stimulus's effectiveness in the action and perception pathways. This common source of modulation determines how competition among stimuli is resolved, causes the same temporal asynchrony dependence of action and perception, and causes the misperception.

Materials and Methods

Participants.

The recordings were made in 12 healthy adult human subjects of either sex (mean age 40.1, range 27–62 years). Three of them were authors; the others were naive to the purpose of the experiment and were paid for their participation. All subjects underwent a neuro-ophthalmological examination before the recording: participants were negative for clinical and neurological pathologies or visual impairment, except very mild refractive error in some cases (subjects did not wear their glasses during the experiment). Correction of vision was not necessary for the recording in any of our conditions, because the targets were bright and contrast was high. Subjects were not under the influence of drugs or alcohol. Experimental protocols were approved by the NIH Institutional Review Board concerning human subjects, and were performed in accordance with NIH guidelines for human subject research and with the Declaration of Helsinki. All volunteers provided written informed consent.

Visual stimuli and apparatus.

Visual stimuli were red laser spots (3 mm diameter, subtending a visual angle of 0.16°) projected onto a screen positioned 105 cm from the subject's eye. The fixation point was a dedicated laser pointing at the center of the screen; the target dot was another laser moved by two orthogonal mirror galvanometers (General Scanning). The luminance of the spots and the screen were measured with a Konica Minolta LS-100 luminance meter. Recordings took place in a dimly lit room (screen luminance 0.009 cd/m²), except for one subject who was recorded with the room lights on (screen luminance 36.17 cd/m²), because in dim light his pupils dilated too much for the eye tracker to follow. The brightness of the fixation point was 46.35 cd/m² in the dimly illuminated condition (Michelson contrast, C = 0.9996) and 84.84 cd/m² with the room lights on (C = 0.4022). Target brightness was 46.69 cd/m² in dim light (C = 0.9996), and 92.13 cd/m² with the lights on (C = 0.4362). The experiment was performed by a real-time data acquisition and control system (REX version 8.0; Hays et al., 1982).

Horizontal and vertical eye movements were recorded from the right eye of all subjects using an infrared iView X Hi-Speed tracker (SensoMotoric Instruments) sampling at 1 kHz. Only the horizontal movements were analyzed. Viewing was binocular, except in one subject with diplopia, in whom the nonrecorded eye was patched. The subject's head was stabilized in the tracker by a chin rest, a forehead rest, and a strap around the head. Each recording session began by calibrating the stimulus display and the eye monitor.

Statistics.

All statistics were computed with MATLAB. Correlations used Pearson product-moment correlation coefficients. Saccade and perception reaction times were compared with orthogonal linear regression (Total Least Squares Method MATLAB package; Petráš and Bednárová, 2010), with bootstrap confidence intervals. Statistical significance was tested with one-way ANOVA (anova1 function) for difference among means, or the two-sample Kolmogorov–Smirnov test (kstest2 function) for difference between distributions. Significance level was considered as p < 0.05. Unless otherwise indicated, all averages are given as mean ± SEM.

Psychometric functions.

We varied TA with the method of constant stimuli in a yes/no choice paradigm (gap/overlap). There was a transition between responding gap for negative TA and responding overlap for positive TA. Psychometric functions (PFs) were fitted (MATLAB Psignifit3 software package, from http://psignifit.sourceforge.net; Frund et al., 2011) to the perceptual responses (percentage overlap responses as a function of TA) using Bayesian inference [Markov Chain Monte Carlo (MCMC) method] (Wichmann and Hill, 2001). As the timing of gap and overlap responses might be due to different mechanisms, perceptual report reaction time distributions might not be a simple Gaussian distribution, e.g., the PF might be asymmetric above and below the threshold. Therefore, we evaluated several sigmoidal functions for the PF. These were based on Cauchy, Gauss, left Gumbel, right Gumbel, and logistic functions (Wichmann and Hill, 2001). Each dataset (from each subject and the pool of subjects, and for each task) was fitted with all five functions. Each subject's threshold was determined by the fit with the smallest deviance (i.e., log-likelihood ratio) to his/her data (Wichmann and Hill, 2001). Thus, each subject's threshold came from the PF that gave the best fit to his/her own data, which avoids bias if the data are skewed.

Design and procedure.

The experimental design was similar to the gap/overlap paradigm of Saslow (1967), but without an auditory warning signal. A central spot came on, acting as the fixation point (FP). Duration of the FP was randomly drawn from the set [1600, 1700, 1800, 1900, 2000 ms]. Later, the target (TG) was turned on at an eccentric location. The relative timing of offset of the FP with respect to the onset of the TG defined a specific TA. Onset of the TG could follow offset of the FP after a short time (TA < 0, gap condition), it could be synchronous with the offset of the FP (TA = 0, synchronous condition), or it could precede the offset of the FP (TA > 0, overlap condition). Gap, synchronous, and overlap conditions were presented pseudorandomly during each session. We tested 13 TA times [0 and ±50, 100, 150, 200, 350, 500 ms]. There were five TG eccentricities [4, 6, 8, 10, and 12°] and two directions (left or right). Duration of the TG was randomly chosen from the set [1000, 1050, 1100, 1150, 1200 ms]. The next trial started 100 ms after the TG was turned off. Subjects took part in three sessions in 1 d or across 2 d: the main experiment and two control sessions. Each session took ∼30 min. Every 5–6 min during the session subjects were given 1–2 min of rest time; hydrating eye drops were provided if requested. During each session trials were defined by randomly selecting three parameters (TA, eccentricity, and direction of the TG) from the above sets.

Session one: main experiment, saccade + perception task.

Each trial began with the onset of the FP. The subject was instructed to look at the FP as soon as it came on and to maintain fixation until the appearance of the eccentric TG. Then, the subject was asked to make a saccade to the TG as quickly and accurately as possible. After looking at the TG, the subject was requested to push one of two buttons on a handheld box, indicating whether he/she saw a gap (perceiving a period when no lights were on) or an overlap (perceiving that both lights were on at the same time). The subject received no feedback about his/her performance. The button press did not terminate the trial, and subjects were instructed to fixate the TG until its disappearance.

Session two: control session, saccade-only task.

The visual stimuli were the same as in the main paradigm, but subjects were only required to look at the FP and TG in turn. No perceptual report or button press was required.

Session three: control session, perception-only task.

The stimulus sequence was the same as in the previous sessions, but subjects were asked to maintain fixation on the FP (or where it had been if it went off), without making a saccade when the eccentric target appeared. After TG onset, they pressed a button to indicate whether they perceived a gap or an overlap.

Data analysis.

Data were analyzed off-line with algorithms developed by our laboratory in MATLAB (The MathWorks). Eye position data were obtained by smoothing the raw data with a Savitzky–Golay filter (sgolay function with order four and frame length 21 (Savitzky and Golay, 1964). Eye velocity was obtained from raw eye position data with a Savitzky–Golay differentiating filter (order two and frame length 25). Saccades were detected when velocity crossed 0.3 times the SD of the eye velocity trace during fixation. Saccade end was defined as when the eye velocity decreased below the same threshold. In the saccade + perception (SP) and saccade-only (SO) sessions, latency (interval between onsets of TG and saccade), peak velocity, amplitude (degrees of visual angle traversed between start and end of the saccade), and duration (time between saccade start and end) were measured for the first saccade after the onset of the target in each trial. Only trials without artifacts (such as blinks or saturation of eye tracker signal) were analyzed. Trials where the first saccade was made away from the target, where saccadic latency was <80 or >500 ms, or where peak speed was <25 or >800°/s, were excluded. If the subject did not maintain fixation within ±1°, the trial was excluded. In the SP session, if the subject did not look at the FP at the start of the trial, or made no saccade after the TG onset, or did not press a button, or pressed the button before the saccade, the trial continued but was considered incorrect and was not analyzed. In the perception-only (PO) session, trials with saccades or trials where the subject was not looking at the FP were ignored. We typically acquired ∼270 trials. After excluding trials according to the above criteria, ∼220–260 valid trials remained.

Saccadic and perception report reaction times were measured from TG onset. Whenever subjects pushed a button in the SP and PO sessions both the perception (gap or overlap) and its time were recorded. If the subject pressed a button twice, or pressed first one button and then another, we took as valid only the first button press after the target appeared. In the SP session, a valid response had to follow a valid saccade.

Each session was analyzed separately. For each TA, we calculated the average saccadic and perception reaction times and the number of “gap” and “overlap” responses. Data were first analyzed for each subject. As there were no qualitative differences among subjects, we also analyzed the pool of all subjects. We did not find any significant differences in saccadic or perception reaction times; or the psychometric function's transition threshold; or the percentage of correct responses, between left and right targets; or for different target amplitudes (ANOVA, p > 0.05), so all target amplitudes and directions were pooled. Thus, each TA was repeated ∼20 times per session.

Results

Saccade and perception reaction times depend on TA

The SP dual task (Fig. 1) was compared with two single-task control sessions, SO and PO. We found that saccadic reaction times in SP and SO tasks shared the same TA dependence (Fig. 2A, single subject; E, pool of all subjects). These results were independent of stimulus eccentricity or direction (ANOVA, p > 0.05). As expected, the saccadic reaction time's dependence on the TA reproduced Saslow's saccadic gap effect in both SP and SO tasks (Fig. 2A,E). In our experiment saccade latencies were ∼75 ms longer than in Saslow's. This may be explained because both of his subjects were younger than ours (latency is shortest at ∼20 years of age; Munoz et al., 1998), and they were given a brief auditory warning signal (which can shorten saccadic latency; Ross and Ross, 1980). In addition, our targets were very unpredictable, which can lead to longer reaction times (Dorris and Munoz, 1998; Schiller et al., 2004; Rolfs and Vitu, 2007). Nonetheless, the range of the effect (longest–shortest reaction time) was ∼120 ms in both studies.

Figure 1.

A, B, Task conditions and example of saccade and perception responses. A, In the gap condition (left), the FP turned off (red dashed line) before the peripheral TG turned on (green dashed line). In the overlap condition (right), the TG turned on before the FP turned off. B, Example saccades from gap (left) and overlap (right) trials of subject six (S6) in the SP task. Button press (orange triangle) shows when the subject indicated his perception.

Figure 2.

Saccade and perception reaction times, and perceptual transition from gap to overlap, depend on TA. Left column shows results from a single subject. Right column shows the results from averaging across all 12 subjects (Pool). A, E, Show SRTs. B, F, Show PRTs. C, G, Orthogonal linear regression comparing reaction time changes (i.e., times are shifted so that the mean value is zero at TA = 0). (Dashed line indicates x = y, b = regression slope with [95% CI].) D, H, Psychometric function, m = threshold (i.e., TA at inflection point of function) [95% CI]. Horizontal lines indicate the chance levels. Note that the threshold is not at TA = 0, as expected in the case of a faithful perception; instead there is a misperception of small overlaps as gaps (in H, dots and bars are mean ± 1 SEM of the 12 subjects).

Here we show for the first time that the perception reaction time also depends on TA (Fig. 2B, single subject; F, pool of all subjects; cf. Table 1). The perception reaction time has three main components: the time to form the conscious perception on which the report is based (gap/overlap), the time to select which button to press, and the time to plan and execute that button press (manual reaction time). The temporal asynchrony of our stimulus may affect both the time to form the conscious perception (gap/overlap) and the time to make the selection (which button to press). In general, in visual studies the last component is considered constant. We also make this assumption.

Table 1.

Regression statistics

	Saccade + perception task	Perception-only task
Orthogonal regressions
Single subject (Figure 2C)
Slope [95% CI]	1.03 [0.73, 1.44]	1.12 [0.83, 1.53]
Pearson's correlation	r = 0.876, p = 0.00009	r = 0.854, p = 0.0002
Pool of subjects (Figure 2G)
Slope [95% CI]	0.98 [0.77, 1.13]	0.83 [0.48, 1.09]
Pearson's correlation	r = 0.942, p = 0.0000	r = 0.833, p = 0.0004
Psychometric functions
Single subject (Figure 2D)
Threshold: mean [95% CI]	105.3 [104.0, 106.4]	100.5 [99.3, 101.4]
Width: mean [95% CI]	291.1 [289.0, 293.7]	208.2 [206.8, 210.0]
Guess rate	0.022 ± 0.001	0.039 ± 0.001
Lapse rate	0.084 ± 0.000	0.083 ± 0.000
Pool of subjects (Figure 2H)
Threshold: mean [95% CI]	74.4 [73.5, 75.7]	66.0 [63.6, 67.5]
Width: mean [95% CI]	581.6 [580.6, 582.5]	535.6 [534.5, 536.4]
Guess rate: mean ± SEM	0.006 ± 0.000	0.005 ± 0.000
Lapse rate: mean ± SEM	0.023 ± 0.000	0.007 ± 0.000

Confidence intervals for orthogonal regression come from 2000 bootstrap resamplings. To estimate the parameters of the psychometric function [its mean, width, and low (guess rate) and high (lapse rate) asymptotes] with a Bayesian technique, a priori assumptions must be made about the distribution of the parameters (Kuss et al., 2005). We used a prior with a normal distribution, Gauss (μ = 50, σ = 150), for the threshold parameter. (Changing the threshold parameter's a priori μ from −50 to 50 ms changed the threshold value by 3 ms or less.) The width parameter was given a γ distribution prior, γ (shape = 1.01, scale = 2000). The lapse and guess rate were assumed to be small, and were given priors based on the β distribution, β (α = 1, β = 20).

A gap effect has been demonstrated for manual reaction times (Ueda et al., 2014). This might explain the reduction in the perception reaction time that we observed in the gap condition. However, manual reaction times do not show an increase in latency during an overlap task. Indeed, it has been demonstrated that maintenance of a foveal stimulus does not increase the manual reaction time (Ueda et al., 2014). Thus, changes in the manual reaction time could not explain the increase of the perception reaction time in the overlap condition.

Nonetheless, it has been shown that the eye and the hand coordinate temporally when they move toward the same target (Horstmann and Hoffmann, 2005). Saccade reaction times are reduced when a hand movement points at the same target (Lünenburger et al., 2000). However, this coordination cannot account for the coupling we see in action and perception, because eye–hand coordination has not been demonstrated in conditions where the eye and the hand do not move toward the same target. A shared spatial end point is supposed to be the basis of the eye–hand coordination (Gielen et al., 1984), but the saccade reaction time reduction that occurs when the gaze supports target-directed hand movements has not been found in our saccade + perception condition. Instead, we found an increased saccade reaction time in the dual task. Consequently, we infer that the perception reaction time's dependence upon TA results from the TA dependence of the time of conscious perception, and not from changes in the button-selection time or the manual reaction time.

We evaluated the similarity of the saccade and reaction time dependence on the TA by comparing the reaction times after shifting them to cross zero at TA = 0. This compares the change in reaction time as a function of TA. In the dual-task condition, the changes of saccadic and perception reaction times with the TA were tightly coupled (see Table 1). In the single subject (Fig. 2C) and the pool of subjects (Fig. 2G) the slopes of the lines for PRT versus SRT in the SP task (dark green lines) or in the PO and SO tasks (light green lines) were not different (because their 95% confidence intervals overlap).

This temporal coupling could have been induced by the experimental instruction to push the button after the saccade in the SP session. However, the same dependence was seen when comparing PO and SO sessions, in which the two performances were unrelated by the task's instruction and were executed in different sessions. Thus, the experimental design cannot be responsible for their coupling. The coupling in all three sessions strongly suggests that the same neural substrate regulates both saccadic and reaction time dependence on TA. Moreover, we think that the change in perception reaction time results from the time of perception (all other factors being independent of TA). Thus, we infer from the tight coupling in Figure 2, C and G, that the TA dependence of saccadic reaction time and perceptual reaction time results from the same neural mechanism.

Misperception of small overlaps as gaps

Increasing the TA delays the time of perception. Nonetheless, the delayed perception should still represent the true visual condition (i.e., perceive gap for TA < 0, overlap for TA > 0, and respond at chance when TA = 0). To test this, we fitted the relationship between percentage overlap responses and TA with a psychometric function (Fig. 2D,H). Surprisingly, the location of the transition from perceiving a gap to perceiving an overlap did not occur at the synchronous condition (TA = 0 ms), as expected if perception faithfully reflected the visual scene. The average transition location, or threshold, for the pool of subjects was m = 74 ms (95% CI = [73, 76]) in the SP, and 66 ms [64, 68] in the PO task. Thus, synchronous TA and small overlaps (less than ∼100 ms) were misperceived as gaps.

The average transition location, or threshold, was computed for each subject (Fig. 3). The thresholds for subjects ranged from approximately −15 to 125 ms. The threshold distributions have a large range of shifts. This suggests that there is not a common shift with a wide variability across subjects; rather, the shift depends upon the subject and the experimental task, and is rather precise. We think that this wide range rules out the possibility that an early visual mechanism causes the common increase in saccade and perception reaction time with TA.

Figure 3.

Psychometric function transition thresholds for each subject. Thresholds for gap/overlap responses for the SP and PO tasks. Seven subjects' thresholds shifted to an earlier time in the PO task, and two subjects' thresholds shifted to a later time. Only one subject showed a negative threshold, i.e., S8, in both the SP and PO tasks, responded overlap even though there was a gap. The variability across subjects suggests the transition threshold is not random, but depends upon the subject and the task. The wide range of TA thresholds makes it unlikely that a low-level, early visual process can account for the shift of the thresholds. (Dots and error bars are mean ± 1 SEM from the last 1000 MCMC chains.)

Before concluding that the misperception of small overlaps as gaps is caused by the mechanism that creates the TA dependence of saccade and perception reaction times, we must discount known mechanisms that might have caused the misperception (i.e., visual masking and metacontrast, flicker fusion, an after effect, contrast sensitivity's dependence on eccentricity, ambiguity, saccadic suppression and perisaccadic perceptual phenomena, and microsaccades). Here we will exclude each of these possible alternatives. Visual masking and metacontrast refer to the reduced visibility of a first stimulus when followed by the presentation of a second stimulus in close spatial and temporal proximity (Enns and Di Lollo, 2000). The perceptual behavior that we observed in our experiment cannot be due to visual masking or metacontrast; first, the eccentricity of visual stimuli in our experiment (4–12°) was too large to induce visual masking or metacontrast. Second, visual masking and metacontrast occur usually for very brief stimuli (≤50 ms), whereas the duration of the fixation point in our experiment ranged from 1600 to 2000 ms and the duration of the target from 1000 to 1200 ms.

When the frequency of an intermittent stimulus is maintained above a critical threshold, the light of the stimulus can be perceived as persistent—a phenomenon called flicker fusion. The lights in our experiment cannot be considered properly as intermittent; furthermore, flicker fusion would have led to an overestimation of the overlap condition, the opposite of what we found.

A perceptual after effect also would have led to an overestimation of the overlap condition. Moreover, an after effect would be expected to have led to a correlation between central fixation time exposure and perceptual response, but that was not found in our experiment.

As target eccentricity increases, the contrast sensitivity threshold increases monotonically (Pointer and Hess, 1989). Thus, the perception of a peripheral stimulus that is near the threshold could be impaired proportionally with eccentricity. However, our stimuli had very high contrast (C ∼0.4–1.0), and perceived contrast for high contrast stimuli (C ≥ 0.5) does not change with eccentricity (Cannon, 1985). This suggests that perception here should not have been affected by eccentricity.

Detection reaction time for suprathreshold stimuli increases with eccentricity (from 0 to 15°), by ∼0.5 ms/° (Strasburger et al., 2011). This effect might delay reaction time and perception of our most eccentric target (at 12°) by 6 ms. However, this cannot account for the 66–74 ms shift in the perceptual threshold that we found for the subject pool (Fig. 2H), or the wide range of shifts we found across subjects (Fig. 3). To further rule this out, we looked at the dependence of the perception reaction time on eccentricity in our SP and PO experiments. Pooling across subjects, neither saccadic nor perception reaction time nor perceptual threshold depended on eccentricity or direction (ANOVA, p > 0.05).

Ambiguity might cause an increased error rate in the perceptual response when the visual input condition is more difficult to recognize. If this were the case, ambiguity would have led to a Gaussian distribution of the error rate for the TA, centered on the synchronous condition, but that would not explain a shift to later TA.

Saccades should cause the image of the visual world to smear across the retina, but a perisaccadic perceptual suppression precedes the onset of saccades by ∼50 ms (Diamond et al., 2000). Saccadic suppression is supposed to be a global effect, decreasing contrast sensitivity at all locations (Knoll et al., 2011). This suppression is thought to affect the dorsal magnocellular pathway, which is involved in the perception of motion, sparing the ventral parvocellular stream (Burr et al., 1994). Despite its relation to motion, saccadic suppression may also affect perception. Could saccadic suppression partially account for our results? In the SP condition the average saccadic reaction time corresponding to the 0–100 ms TA was 304 ± 15 ms, which was long enough so that the visual information would not have been obscured by saccadic suppression. Moreover, if saccadic suppression were responsible for the misperception, we would have expected saccades with the shortest reaction times to have more incorrect responses, because those saccades are closest to the stimulus onset. However, we did not find this effect (ANOVA, p > 0.05).

We also exclude other perisaccadic phenomena, such as space and time compression and remapping of receptive fields. For example, saccades cause a compression or reversal of the interval between two spatially extended peripheral bars (Morrone et al., 2005). However, for very short times, their subjects perceived a time compression and a temporal inversion (the second bar was seen as first). If this misperception had occurred in our experiment, subjects would have misperceived short gaps as overlaps, the opposite of what we observed.

Furthermore, TAs ≤ 100 ms were also misperceived as gaps in the PO condition (no saccade), thus excluding a proper perceptual saccadic suppression. However, the target onset in the PO task might have induced a motor plan for a saccade that was later inhibited. There is no evidence, yet, that motor planning in the absence of a saccade causes saccadic suppression. Nonetheless, target onset in the PO task induces a covert attention shift (i.e., without a saccade). Covert attention shifts have been associated with visuomotor activity in the SC (Ignashchenkova et al., 2004), and visuomotor activity has been related to saccadic suppression (Churan et al., 2012). Thus, even if a saccade is not executed, there could be a perceptual consequence of the SC visuomotor activity associated with target onset. It is also known that attention increases perceptual sensitivity at the target (Carrasco, 2011). We do not know if these effects would be enough to suppress perception.

Microsaccades reduce the visual response in both V1 (by ∼17%; Hass and Horwitz, 2011) and SC (by ∼30%; Hafed and Krauzlis, 2010). However, the rate of microsaccades (normally ∼1/s) dips by 80% between 50 and 250 ms after the peripheral target appears, and does not increase until ∼300 ms after target appearance (Engbert and Kliegl, 2003). Thus, even if microsaccades have a perceptual effect, their occurrence would be too rare to explain the misperception for overlaps of ∼100 ms in the pool of subjects.

Based on the above considerations, we infer that the perceptual response reaction time dependence upon the TA underlies the misperception.

Dual-task interference

Dual-task interference occurs when limited neuronal resources have to be shared, which may cause reaction times to increase; this is called the psychological refractory period (Pashler, 1994). We found that saccade and perception reaction times were both increased in the dual-task condition relative to the single-task condition, which supports the idea of a common modulator. For pooled data, the saccadic reaction time difference, SP − SO, was 21.3 ± 4.5 ms (mean ± SEM, p < 0.001, Wilcoxon sign rank test); the perception reaction time difference, SP − PO, was 46.5 ± 6.5 ms (p < 0.001). Interestingly, the dual-task condition also induced a rightward shift of the psychometric curve (8 ms in the pooled data; Fig. 2H). Averaging the shift across individual subjects gave quartiles [25%, 50%, 75%] for the range of shifts of [−1.3, 11.4, 39.8 ms]. This difference is surprising, because the visual stimuli were exactly the same in the SP and PO tasks.

Therefore, the unexpected dual-task interference with the gap/overlap transition also suggests that the common mechanism underlying the timing of action and perception influences what is perceived.

Model of TA-dependent time change and perception

We implemented a functional model of action and perception by combining models for perception and attention with a collicular mechanism (Fig. 4). This model does not predict reaction times, but rather the change in reaction times caused by temporal asynchrony. Some of the interactions in this model require global connections, which could arise in the brain through either intrinsic projections or feedback networks (Lee and Hall, 2006; May, 2006), but they will be represented here in a feed forward model for simplicity.

Figure 4.

Functional diagram of the TA-dependent time-change model of perception. Each block represents a 1D map of space (i.e., retinal eccentricity), divided into an attentional system (green) and a perceptual system (purple). Action and perception depend upon attention, which is modulated by prominence. The SC (blue) combines bottom-up and top-down information to determine which retinotopic locus has the most prominent stimulus for action and perception. Prominence affects the attention gain modulating the visual input to the attention system. Enhanced vision is the common input for the perceptual system (in prefrontal cortex) and for the stimulus priority map (in lateral intraparietal cortex). The priority map selects the target, which leads to an action (red) when the signal at that locus crosses a threshold. The object map in the perceptual system receives the enhanced vision signal when the stimulus is stable in time (determined by comparing the template and pretemplate signals across a brief time). The object map is integrated. When the integrator signal crosses a fixed threshold it enters the perceptual map (perception). Conscious perception occurs when the signal in the perceptual map crosses a second, dynamic threshold. Thus, prominence couples the time of action and perception, and affects what is perceived, by modulating the enhanced vision signal. Asterisk indicates convolution. Steps indicate thresholds. Π and ÷ indicate multiplication and division.

Model basis

Models of perception formation and selection function by accumulating evidence until a threshold is reached (Ditterich, 2010). One such model for perception (Fig. 4, purple boxes) was proposed by Ma et al. (2006). Their model looks at the conscious visual perception of briefly presented stimuli. Although perception usually reflects the time course of the physical stimuli, there is a minimal duration of a perception (Efron, 1970), which is accounted for in their model. This MHK model (named from the authors' initials) represents visual perception by interactions among pools of neuronal populations. Each member of the pool corresponds to some feature of interest, such as the tilt of a line; here, the feature of interest is eccentricity. Thus, each layer is a spatial map. The stimulus is considered to be an object if it persists, as judged by comparing two pools of neurons that are delayed versions of each other (Fig. 4, pretemplate and template). Once a stimulus enters the object pool of neurons it gets integrated in the next layer. An object enters the perception map if its integrated signal crosses a fixed threshold. The object enters conscious perception if the activity in the perception map crosses a dynamic threshold that depends upon what is already in the object pool. This dynamic threshold is the key parameter of the model that determines a subject's perceptual transition time (see below).

The MHK model works well for many experiments where attention is not involved. To introduce attentional effects, we added a divisive normalization mechanism (Reynolds and Heeger, 2009; Louie et al., 2013). This RH model (named from the authors' initials) modulates the visual input by an attentional gain field. It then uses an averaging mechanism to provide a divisive normalization of the input strength (Fig. 4, green boxes). In the RH model, the attention gain field is assigned, not computed. Here, we replace the assigned gain field with a spatial prominence map in the superior colliculus (Fig. 4, blue box). Prominence is the likelihood that a stimulus will be selected for action and perception. The activity on the map evolves as evidence about the competing stimuli is accumulated, eventually crossing a threshold (see Discussion for more on this concept).

Model equations

The model is described by first-order ordinary differential equations and parameters taken from MHK (Ma et al., 2006) and RH (Reynolds and Heeger, 2009) models with a few modifications. Parameters of the new model are listed in Table 2. The following equations are a complete description of the model. See the original papers for explanations of where these equations come from. The simulations were run in MATLAB (The MathWorks).

Table 2.

Model parameters

Time constants (s)		Gains		Constants
τV1	0.020	αd	0.200	θ0	30.000
τS	0.040			δ	0.003
τPT	0.060	αPT	1.000	αθ	3.000
τPTi	0.010	αPTi	0.022	σ	0.500
τT	0.200	αT	15.000	σD	0.100
τO	0.020	αO	0.500	θt	0.000
τI	0.200	αI	20.000	θCPmax	4.8
τp	0.020	αP	1.000	Prominence factor	1.100
τAF	0.075	αAF	1.000	Number cells/map	9
τAF2	0.075	αCP	0.450	MATLAB solver	ode23
				MaxStep	0.002

Perception model values taken from Ma et al., 2006. Similar values were chosen for the attention model. Thresholds have been adjusted for our stimuli. Prominence factor was applied for the simulation of Sweet's experiment (Sweet, 1953), to account for the subjects' focus on where the foveal light would flash. For the simulation in Figure 6B, θ₀ was reduced from 30 to 0.005, because of the very short flash duration. All times are in seconds.

The model consists of a set of 1D neuronal maps, each map consisting of nine units representing different eccentricities. The variables in the equations below are thus for a representative unit within a map. The visual input, I(t), arrives in the A_V1 map and is modulated by synaptic depression, S(t) (MHK Eqs. 16.3a–b) as follows:

where S(t) is given by the following:

In the original model, there were no multiple stimulus presentations, so the value of synaptic depression simply grew throughout the simulation until it reached a value of 1. We modified Equation 1b so that it can account for multiple stimulus presentations by making S decay when there is no input:

At this point, we interpose the RH equations to create an enhanced visual signal with an attention map with second-order dynamics. First, an attention kernel, K_a, which is +1 at the center and −1 everywhere else, is convolved with A_V1 to provide a competitive map for attention, A_Fi:

(where * indicates convolution). The first-order attention map, A_F, is given by the following:

The first-order attention map is inverted to give A_Fp:

Next, we compute the second-order attention map, A_F2:

A new map, A_F2p, representing prominence in our model, is given by the following:

where M_P is the SC map of prominence from top-down influences, and []₊ indicates that negative values are set to zero. The ratio (3:1) is somewhat arbitrary, but prevents the prominence map from going to zero when A_F2 is zero. The suppressive drive, S_D, from Equation 6 of RH is given by the following:

where a spatial map kernel, K_s, defined by a Gaussian (N(0, σ)), is convolved with the product.

Now, we can compute the normalized population response, R_p (Eq. 5 from RH), which we call here the enhanced vision signal (Fig. 4), because it includes the effect of the prominence map:

where θ_t is a threshold and σ_D is the minimum suppression factor.

We now describe the perceptual part of the model (Ma et al., 2006, Eqs. 16.4–16.10). In this model, a stimulus becomes eligible for entering perception if it persists across time, as defined by comparing its appearance in the past (held in the template map) with its appearance now (in the pretemplate map). If the stimulus persists long enough, it can enter the object map. The object map is integrated and, when the information exceeds a threshold, it can enter the perception map. Conscious awareness of an object occurs when the activity in the perception map crosses a dynamic threshold.

The enhanced vision signal becomes the input to the rest of the perception model, starting with the pretemplate map (A_PT):

where A_PTi is a delayed version of the integrated sum of the template map (A_T):

where the sum over θ is over the map's neurons. The template map equation is as follows:

Object representation, in the map A_O, is given by the following:

The object map projects to a leaky integrator, A_I, which is given by the following:

To enter the perception map, A_p, the sum of the object map activity has to get over a threshold, θ_p, which is itself a function of the input:

The MHK model defines something as being consciously perceived when it crosses a threshold that is approximately half the size of the activity in AP:

In our model, the dynamic threshold is instead adjusted so that the transition threshold for gap/overlap matches our results.

Simulations

The columns in Figure 5A show simulations of our model for three values of TA. In the bottom row, the output of the visual area is shown. Its response follows the visual stimuli with a short time constant. The row above, prominence, is a rapidly adapting map that shows the emphasis given to each location. The product of these two maps results in the enhanced output. In the object map and above, the time constants are much longer, and include a nonlinear operation (a threshold). It is in these maps that the TA dependence emerges. Note that, in the gap and overlap tasks, it is the presence of the fixation point that prolongs the activities in the object and integrator maps, causing the delay in the perception of the target in the overlap task (cf., Fig. 5A, solid green line relative to dashed green line in top row).

Figure 5.

Simulations of perception and reaction time change as function of the TA. A, Simulation of model with TAs of −350 (left), 70 (middle), and 350 (right) ms. Each row represents one of the critical blocks of the model. Time-domain response of two units from the map in the model block are shown: one unit at the location of the fixation light (magenta) and one at the location of the target light (blue). This explanation follows the organization of the model in Figure 4, from bottom to top. The bottom row shows the vision map (the stimulus is its input). The blue vertical line in each graph shows the time of onset of the TG. The next row up indicates the prominence map in the SC. The next row up shows the attentional enhancement of the visual signal caused by the prominence signal. The object row is a low-pass filtered version of the enhanced visual signal. The next row is the integral of the object; when the integral passes the first threshold (black dashed line) it enters the perceptual map. To be consciously perceived (indicated by the thick parts of the traces), the signal must pass a dynamic threshold (black dashed line). The latter threshold was adjusted to put the change of perception from gap to overlap at 70 ms, thus overlaps from 0 to 70 ms will be misperceived as gaps. The target first appears at Time = 0 (blue vertical line in all parts). The green vertical dashed line (P) in each graph in the top row is a reference for when the target was first perceived for a TA of −350 ms. Comparing this line with the time when the target is perceived at the other TAs (green vertical solid lines) shows that the time to perceive the target becomes longer as the TA moves from gap to overlap. Critically, the time of the threshold between gap and overlap is dependent upon the threshold in the perceptual map. Thus, thresholds for different subjects would be modeled simply by adjusting this threshold. B, Simulation of reaction time change as a function of TA. Colored lines indicate the saccade and perception reaction time changes in our experiments (same data as in Fig. 2E and F, but shifted vertically to cross zero at TA = 0). Thick black line is the change of the time of perception of the target in our model. According to our hypothesis, this line determines the change of the PRT (light and dark orange lines). As the underlying mechanism of the TA dependence is common to both action and perception, the perception output of the model must also reproduce the output of the action branch, as indicated by the similarity of the black line with the change in the SRT (light and dark blue lines).

Our experiments showed that, when the overlap was in the range [0, 100 ms], the subjects perceived a gap. In the center column of Figure 5A, where 70 ms of overlap is misperceived as a gap, note that the visual stimuli clearly overlap (vision, bottom row), but because of the different times at which the foveal target (magenta) and the peripheral target (blue) get over the dynamic threshold in the perception map, the perception does not show both at the same time; thus, the model perceives a gap. Moreover, fixation-related activity attributed to the prominence map in the superior colliculus (Fig. 5A, prominence) delays the time of perception of the peripheral target as the TA increases (percept), causing short overlaps to be misperceived as gaps (center column).

The misperception is a consequence of both the coupling between action and perception and the existence of a threshold in the perception map. If there were coupling but no threshold, or a threshold but no coupling, there would be no misperception.

The simulated increase in the time of perception of the target as a function of TA (Fig. 5B, black line) follows the TA dependence of the perception reaction time (orange lines). In our hypothesis, the prominence map couples perception and action. This is confirmed by the simulation, which shows that the simulated time change for perception (black line) is also similar to the change in saccadic reaction time (blue lines).

Simulations of other studies

The model can also explain earlier observations related to the perception of competitive stimuli. As those data were not used in the design of the model, these simulations are predictions by the model. When equally salient stimuli appear simultaneously at the fovea and in the periphery, the central stimulus is perceived earlier than the peripheral one. Thus, to be perceived as simultaneous, the peripheral stimulus had to precede the foveal stimulus (Sweet, 1953). In his experiment the central stimulus had a greater prominence than the peripheral stimulus because of the allocation of attention at the foveal location at the beginning of each trial, the greater saliency of the stimulus conveyed by the fovea, and the foveal-peripheral inhibition within the SC. Thus, in this case, the perceptual processing of the more prominent central stimulus is accelerated by the vision enhancement from the SC, causing an earlier perception of the central stimulus with respect to the peripheral one. To overcome this advantage and have both flashes perceived as simultaneous, the peripheral stimulus needs to appear earlier (Fig. 6A).

Figure 6.

Simulation of prior perceptual experiments. A, A peripheral target (blue) turned on 12 ms before a foveal target (magenta; Sweet, 1953). Both lights flashed for 15 ms. The subject perceived a simultaneous presentation of the lights, because of the greater prominence of the foveal target, which accelerated its rise to the threshold necessary to enter the perceptual map. Prominence map, M_p, is 1 everywhere but at the fovea, where it is 1.1, to reflect the fact that the subjects were instructed to look at the foveal fixation point before the flash. B, Two flashed lights (12 ms duration) appeared simultaneously (Alpern, 1954). The brighter light (magenta) had greater prominence, and was perceived earlier. The shift in times of perception is the consequence of the thresholds for the integrator and for conscious perception. (Layout same as in Fig. 5A.)

The model also reproduces the earlier perception of the brighter of two simultaneous flashes (Alpern, 1954). In this experiment the more intense stimulus has a greater prominence in the SC, which causes it to cross the perceptual threshold first (Fig. 6B). For the same reason, our model is also consistent with earlier observations reporting an enhanced perception in the periphery when the central stimulus is removed before target onset (Huestegge and Koch, 2010). In this case, removal of the central stimulus reduces the prominence at the fovea and increases it in the periphery.

Discussion

We showed for the first time that there is a gap effect on perceptual reaction times in a classic gap/overlap task. Furthermore, we found a tight correlation between the dependence on temporal asynchrony of saccade reaction time and perceptual report reaction time, suggesting a common modulator. Surprisingly, we also found that subjects misperceived small overlaps as gaps. Model simulations showed that a prominence map in the superior colliculus could act as a common modulator for saccade and perception signals, and determine what is perceived.

Common modulator

What area acts as the common modulator for coupling the time of action and perception and as the mechanism causing the misperception? One possibility is that the area responsible for the saccadic reaction time gap effect also affects perception. Several neuronal mechanisms have been proposed to account for the gap effect (Munoz and Wurtz, 1993; Munoz and Fecteau, 2002; Pratt et al., 2006). Cortical areas play a role in influencing saccadic reaction time (Pratt et al., 2006; Sumner et al., 2006), but they are not sufficient to resolve competition between stimuli (McPeek and Keller, 2004). Thus, it is unlikely that cortical areas are the common modulator for the TA dependencies of action and perception. In contrast, the SC, a midbrain nucleus representing a major input to the brainstem saccade generator, is necessary to resolve competition among stimuli and to initiate saccades (Munoz and Fecteau, 2002). Excitatory and inhibitory long-range lateral interactions that could manage the competition between simultaneous stimuli have been demonstrated in the SC intermediate layer anatomically (Behan and Kime, 1996; Sooksawate et al., 2011), pharmacologically and neurophysiologically (Meredith and Ramoa, 1998; Munoz and Istvan, 1998; Phongphanphanee et al., 2014). Indeed, neurons in the SC rostral intermediate layer are active during fixation of foveal stimuli and inhibit saccade-related caudal neurons (Munoz and Wurtz, 1993; Munoz and Istvan, 1998). Finally, activity of these rostral neurons correlates with the gap effect (Dorris and Munoz, 1995). Thus, release or maintenance of this fixation-induced inhibition explains the reduction or increase of saccadic reaction time in the gap and overlap conditions, respectively. Thus, the SC is the site that resolves the competition for action among foveal and peripheral stimuli, and is responsible for the gap effect on saccadic reaction time.

Now we argue that the SC could also be responsible for modulating perception reaction time. The SC intermediate layer is connected with areas involved in the attentional network. Thus, in addition to its pivotal contribution in controlling eye movements, the SC also plays a role in orienting covert visual spatial attention (Goldberg and Wurtz, 1972; Ignashchenkova et al., 2004; Krauzlis et al., 2013). Attention, captured exogenously by the sudden onset of a salient visual stimulus (Carrasco, 2011), facilitates the processing of sensory information at the selected location (Posner, 1980; Carrasco, 2011). Moreover, exogenous attention is a necessary antecedent for conscious perception (Chica and Bartolomeo, 2012). The intermediate layer of the SC is also a multimodal integration area, where multiple sensory inputs, cognitive inputs, and reward information from the basal ganglia converge (Meredith and Stein, 1986; Hikosaka et al., 2000; May, 2006). The pooled inputs can result in excitatory as well as inhibitory interactions, depending on which SC cell type is stimulated and the combination of sensory modalities (Meredith and Stein, 1986). Thus, the intermediate layer of the SC could represent locations in space that stand out, or are particularly noticeable, because of a combination of stimulus salience (bottom-up physical properties across modalities) and cognitive importance or attentional priority (top-down context).

We suggest that the SC, by integrating all this information, provides a representation of the prominence of a given stimulus. Prominence indicates the likelihood that a stimulus at one location will be selected for action and perception, thus resolving competition among stimuli. This is consistent with recent findings suggesting that when stimulated the SC acts first as a linear accumulator, and later as a winner-take-all mechanism (Phongphanphanee et al., 2014). This function is particularly crucial to the prominence of central compared to peripheral stimuli, determining the necessity of orienting toward a different location and thus influencing the reaction time to that stimulus. The SC would also signal the prominence of different stimuli to different brain areas by enhancing or reducing the activity related to those stimuli along other pathways. Thus, the SC could act as the common modulator for temporally coupling action and perception.

Note that we draw a distinction between the prominence map in the SC and the attention priority map in the lateral intraparietal area (LIP; Ipata et al., 2009). Although they both represent important areas for attention and action, we suggest that the SC map is unique in three ways: it integrates multimodal information (Meredith and Stein, 1986), it receives input directly from the basal ganglia (Hikosaka et al., 2000), and it has foveal-peripheral inhibition (Munoz and Wurtz, 1993). Thus, whereas the LIP is involved in choosing among different possible peripheral targets, the SC also resolves the competition between foveal and peripheral targets. The relative prominence of these stimuli is then used by other areas to decide not only where to go but whether or not to move. Reciprocal connections between the SC and the LIP would allow the priority map and the prominence map to cooperate in picking the winning target.

Temporal asynchrony-dependent time-change model

Prominence can also affect perception. We hypothesize that the SC modulates the activity of specific locations in the attentional network in relation to the prominence of the corresponding stimuli. As a consequence, the SC affects the attention-mediated enhancement of the perceptual processing. Perception probably requires an accumulation of evidence about the stimulus until a threshold is crossed (O'Connell et al., 2012). Attention, by enhancing the visual signal, causes the object to cross the perceptual threshold sooner. Thus prominence, by modulating attention up or down, can cause the time of perception to be advanced or delayed.

The premise of our TA-dependent time-change model is that SC activity provides the attention gain field, which in turn influences perception. Inactivating the SC alters performance in a visual attention task, but leaves unchanged the attention-related effects in two visual areas: medial temporal (MT) and medial superior temporal (MST) cortices (Zénon and Krauzlis, 2012). Therefore, the SC contribution to spatial attention must occur through mechanisms different from the known attentional changes of activity in visual cortex. For example, SC and MT/MST might project to a third area, where attentional effects on perception occur. In our hypothesis, the SC would exert a multiplicative effect on the visual inputs to that third area.

The prominence at a location, as indicated by the SC, would then determine not only the likelihood that one stimulus would be identified as the target for action, but also when it would be perceived. Furthermore, as prominence is the result of a competition between stimuli, it determines what will be perceived. For example, when a central stimulus is fixated, foveal-related neurons in the SC intermediate layer inhibit caudal neurons that would respond to the onset of a peripheral stimulus. The prominence of the peripheral stimulus would thus be reduced by the presence of the competitive central stimulus. This foveal-peripheral dynamic delays the identification of the peripheral object as the target for action, increasing the saccadic reaction time. In our model, the same dynamic also influences the attentional modulation at the stimulus location, which affects perception. Therefore, the presence of a competitive central stimulus reduces the SC output to the attentional network at the location of a peripheral target. This, in turn, delays the perceptual processing of the peripheral target and, consequently, the time of its conscious perception. Time of action and perception are thereby coupled. The delay of the perception of the target causes small overlaps to be misperceived as gaps.

Model predictions

Our model makes several predictions. Predictions of the effect of attention at the central location and the effect of stimulus salience were shown in Figure 6. The integration/threshold mechanism also allows behavior to vary depending on the context in which the stimuli are presented. When competitive stimuli have equal visual salience, but are also associated with an auditory stimulus in temporal and spatial proximity, saccadic reaction times are reduced (Frens and Van Opstal, 1998; Colonius and Arndt, 2001): our model predicts that the visual/auditory stimulus should also be perceived earlier. Correspondingly, factors that increase saccadic reaction time, e.g., cognitive load (Pashler, 1994) and microsaccades (Watanabe et al., 2014), should also delay the perception of that object. Superior colliculus lesions, which greatly prolong saccadic reaction times, should also prolong the time to form a perception.

Conclusion

Action and perception show a similar gap effect because they share a low-level prominence map in the SC. Prominence determines when and where to make the saccade, and also modulates the gain of the corresponding signal in the attention-perception system. This temporally couples action and perception. Furthermore, when there is competition between the fixation point and the target, this mechanism requires more time to identify the target for the saccade, and thus more time to perceive the target. This results in the misperception of small overlaps as gaps.