Attentional trade-offs maintain the tracking of moving objects across saccades

Abstract

In many situations like playing sports or driving a car, we keep track of moving objects, despite the frequent eye movements that drastically interrupt their retinal motion trajectory. Here we report evidence that transsaccadic tracking relies on trade-offs of attentional resources from a tracked object's motion path to its remapped location. While participants covertly tracked a moving object, we presented pulses of coherent motion at different locations to probe the allocation of spatial attention along the object's entire motion path. Changes in the sensitivity for these pulses showed that during fixation attention shifted smoothly in anticipation of the tracked object's displacement. However, just before a saccade, attentional resources were withdrawn from the object's current motion path and reflexively drawn to the retinal location the object would have after saccade. This finding demonstrates the predictive choice the visual system makes to maintain the tracking of moving objects across saccades.

tracking moving objects in peripheral vision is crucial for successful behavior in any dynamic environment, from simple household activities to extreme sports, from searching busy visual scenes to obstacle avoidance during locomotion. When driving, for example, we effectively avoid collisions with other cars by covertly tracking them while keeping our eyes mostly fixated on cars ahead of us (Fig. 1A). A number of studies have investigated tracking while the eyes fixate a static object. Using a variety of behavioral measures, including localization performance (Shioiri et al. 2000), visual sensitivity (Shioiri et al. 2002; Atsma et al. 2012), and deviations in saccade direction (Barborica and Ferrera 2004), these studies show that covert visual attention is deployed not only to the moving object's current position but also along its predicted future path (Fig. 1B; Shioiri et al. 2000, 2002; Atsma et al. 2012; but see Yantis and Nakama 1998). Although the execution of saccades modifies the projections of objects onto the retina, preventing saccadic eye movements during object tracking appears to impair performance (Intriligator and Cavanagh 2001). Tracking improves when eye movements are allowed and is more accurate if the tracked objects keep the same position in space across the saccade (Howe et al. 2011), suggesting the involvement of attentional updating preceding the eye movement (Fig. 1C; Cavanagh et al. 2010; Higgins and Rayner 2014).

Fig. 1.

Attentive tracking while driving. A: we can easily keep track of an ambulance overtaking on the right with our eyes mostly fixated on the car in front of us. To do so, we shift attention to the different positions the ambulance will have across time. B: during fixation, we attentively track the ambulance at its current tracked position (n, left) and possibly anticipate its next tracked position (n + 1, right). C: however, when we move our eyes, for example, to a speed limit sign on the right, the ambulance's retinal position changes, passing from the right to the left visual field. To prevent any disruption in the tracking process, attentional resources at the current (left) and possibly at the next tracked (right) position should be “remapped”: before the eyes start to move, attention should be deployed to where the tracked object will fall on the retina after the saccade (orange circles).

Indeed, just before a saccade across a static display, attention is allocated simultaneously to the current retinal position of an attended object and, predictively, to the retinal position it will occupy after the eye movement (Rolfs et al. 2011; Jonikaitis et al. 2013). With the use of a corollary discharge signal (Holst and Mittelstaedt 1950; Sperry 1950) of the imminent saccade motor program (Sommer and Wurtz 2002), visuomotor areas of the brain appear to shift priorities to process the object's future retinal location (Duhamel et al. 1992; Gottlieb et al. 1998; Merriam et al. 2003; Mirpour and Bisley 2012). This “remapping” allows the maintenance of an attention pointer at the relevant position in space, despite changes in retinal coordinates due to the upcoming saccade (Cavanagh et al. 2010; Rolfs et al. 2011; Jonikaitis et al. 2013).

Here we asked whether and how predictive updating of attention occurs when tracking moving objects in dynamic displays. To do so, we developed a novel procedure in which participants covertly tracked the position of an object in an apparent motion display. The results show first that attentively tracking a moving object (while steadily fixating) involves the continuous displacement of covert attention from the object's current to its predicted next position. Second, we show that the preparation of a saccade results in a rapid disengagement of attentional resources from the moving object's current position in space and a transfer of those resources to the retinal position the tracked object would soon occupy after the saccade (the remapped position). These results suggest that briefly before a saccade the visual system trades off attentional resources between the current and future locations of relevant moving objects to allow continuous tracking after the saccade.

METHODS

Participants.

Six students of the Université Sorbonne Paris Cité participated in the experiments (age 22–27, 2 females, 6 right-eye dominant, 1 author). All participants except one author (M. Szinte) were naive as to the purpose of the study, and all had normal or corrected-to-normal vision. Informed consent was obtained before participation, and the protocols for the study were approved by the Université Paris Descartes Review Board, CERES, in accordance with French regulations and the Declaration of Helsinki.

Setup.

Participants sat in a quiet and dimly illuminated room. The experiment was controlled by an Apple iMac Intel Core 2 Duo computer (Cupertino, CA). Manual responses were recorded via a standard keyboard. The dominant eye's gaze position was recorded and available online using an EyeLink 1000 Desktop Mounted (SR Research, Osgoode, ON, Canada) at a sampling rate of 1 kHz. The experimental software controlling the display, the response collection, as well as the eye tracking was implemented in Matlab (MathWorks, Natick, MA), using the Psychophysics and EyeLink toolboxes. Stimuli were presented at a viewing distance of 60 cm on a 21-in gamma-linearized (Brainard 1997; Pelli 1997; Cornelissen et al. 2002) Compaq P1220 CRT screen (Houston, TX) with a spatial resolution of 1,024 × 768 pixels and a vertical refresh rate of 120 Hz. A thick black cardboard covered the CRT screen to create a circular aperture of 39 cm of diameter.

Procedure.

The study was composed of a threshold, a fixation, and a saccade task completed by all participants in six to eight experimental sessions (on different days) of about 90–120 min each. Participants started with a training phase in which they were familiarized with the three different tasks. After the training phase, participants started each experimental session with two threshold blocks followed by two to three blocks of the fixation or the saccade task (blocks were randomly interleaved). Participants ran a total of five blocks of the fixation task and seven to nine blocks of the saccade task (each block lasted about 20–30 min with a pause and eye movement calibration embedded every 10 min).

Each trial began with participants fixating a central fixation target (FT) forming a black and white “bull's eye” (radius 0.4°) on a gray background (45 cd/m²). When a participant's gaze was detected within a 2.5° radius virtual circle centered on the FT, the FT changed from white to orange. This color change signaled to participants that they had achieved fixation and that the next trial would start momentarily. After 200 ms of correct fixation, the FT's color changed back to white and the trial began. The trial display was composed of three red and three green Gaussian blobs (3° radius, σ = 1°, 75% contrast, mean luminance = 14 cd/m²) located at the six corners of a hexagon with 6° side length (at 30, 90, 150, 210, 270, and 330° of rotation) centered on the FT (Fig. 2A).

Fig. 2.

A: we used an ambiguous apparent motion display composed of 3 red and 3 green blobs (equiluminant) arranged at the 6 corners of a hexagon (6° side length). Every 600 ms, we flipped the color of each blob producing ambiguous circular clockwise or counterclockwise apparent motion of the blobs around the fixation target (FT). Participants tracked clockwise following the red blob highlighted at the beginning of every trial by a brighter color (blob to track). The tracked blob luminance progressively reached the equiluminant level, initiating clockwise tracking. B: in each blob, we presented dots moving either randomly (Noise) or briefly coherently (each dot's direction was drawn from a von Mises distribution centered on the main direction) in 1 out of 4 different directions (Motion pulse; right, up, left, down). C: across trials, motion pulses were presented randomly in any blob relative to the tracked object while participants either kept their eyes on a FT or prepared a 12° saccade toward a saccade target (ST) cued at a random 1 of 6 potential locations. We determined the allocation of attention by assessing discrimination performance at the tracked position (n), its next tracked position (n + 1), as well as every other position in the display (n + 2, n ± 3, n − 2, n − 1). Saccades were always made in the direction of the tracked blob, so that positions n − 2 and n ± 3 correspond to the remapped position of the tracked and the next tracked position, respectively.

Every 600 ms, we flipped the color of each blob (red blobs became green and vice versa) to induce ambiguous circular clockwise or counterclockwise apparent motion of the blobs around the FT (corresponding to a linear angular velocity of 100°/s). Participants were instructed to track clockwise the apparent motion of one red blob identified at the beginning of each trial with a brighter color (the tracked blob started at a random corner of the hexagon). To disambiguate the direction of rotation of the apparent motion of the blobs, we gradually changed the mean luminance of the tracked blob from 20 cd/m² at the beginning of a trial, to 18 cd/m² after the first clockwise step, and to 16 cd/m² after the second clockwise step (Fig. 2A, top). After these initiation steps the tracked blob's mean luminance returned to the level of the other nontracked blob (equiluminant red and green blob at a mean luminance of 14 cd/m²) and participants continued clockwise tracking for one to nine (each number was equally likely) consecutive apparent motion steps.

In parallel to this apparent motion display, we presented six random dot kinematograms (RDKs) composed each of half black (0 cd/m²) and half white (90 cd/m²) 0.1° radius dots, restricted in 1° radius apertures centered on the blobs. Dots moved in random directions (see “Noise” in Fig. 2B) at a constant speed of 5°/s (mean limited life-time of 150 ms, minimum life-time of 83 ms). On each trial, one of the RDKs underwent a coherent motion pulse for 100 ms in which dots moved in one of four possible directions (right-0°, up-90°, left-180°, or down-270°; see “Motion pulse” in Fig. 2B), randomly chosen on each trial. During the motion pulse, the speed of the dots jumped to 15 or 20°/s, adjusted for each participant during training to increase performance if necessary. Motion pulses were displayed randomly from 50 to 150, 250 to 350, or 450 to 550 ms after an apparent motion step.

Following the presentation of a motion pulse, trials finished randomly either at the end of the apparent motion step directly preceding the motion pulse, one step after it, or two steps after it. For both the noise and the motion pulse, we drew the motion direction of each dot from a circular normal distribution (von Mises) in which we varied the dispersion around the main motion direction, κ. We used a κ of 0 for the noise (uniform distribution across all directions) and an adjusted value of κ for the motion pulse from 0.1 (very disperse) to 10 (very coherent); see threshold task below.

At the end of each trial, participants reported the main direction of the motion pulse using the keyboard (pressing either the right, up, left, or down arrow key). A positive feedback sound was played after the trial if the reported direction was correct (irrespective of the position of the motion pulse). Participants earned 10 euros per hour of testing. Moreover, to encourage them to keep track of the indicated red blob for the entire duration of the trial, we informed participants about their performance after each experimental session and payed a bonus of up to 30 euros depending on their ability to correctly discriminate motion pulses presented at the tracked position. Bonus payments were calculated by determining a participant's performance at the end of all their experimental sessions and applying the formula ([performance at tracked position] − [chance level])/([threshold level] − [chance level]) × 30 euros, where chance level and threshold level were 25 and 79.4% correct, respectively. Thus they received 30 euros if performance at the tracked location reached threshold level or above and no bonus payment if performance was at chance level. Participants earned bonus payments between 15 and 20 euros.

Fixation task.

Participants completed between 771 and 949 trials of the fixation task during which they were instructed to track a red blob in the clockwise direction while keeping the eyes steady on the FT. Correct fixation within a 2.5° radius virtual circle centered on the FT was checked online. Trials with fixation breaks were repeated at the end of each block in a random order (each participant repeated between 29 and 128 trials).

Motion pulses occurred during an apparent motion step following the tracking initiation steps (see above), either at the tracked position (n, i.e., the position where the tracked blob is at the time of the motion pulse), one step or two steps forward (n + 1, and n + 2, respectively, at 60 and 120° clockwise), one step or two steps backward (n − 1 and n − 2, respectively, at 60 and 120° counterclockwise), or three steps forward and backward (n ± 3, ±180°) from the tracked blob. We informed participants explicitly that the motion pulse could appear at any time and equally often in any of the six blob positions and instructed them to report the main direction of the motion pulse with a keypress.

Saccade task.

Participants completed between 1,092 and 1,223 trials of the saccade task during which they were instructed to track a red blob in the clockwise direction while preparing a 12° rapid eye-movement toward a saccade target (ST). Correct fixation as well as correct saccade landing within a 2.5° radius virtual circle centered, respectively, on the FT and ST was checked online. Trials with fixation breaks or incorrect saccades were repeated at the end of each block in a random order (each participant repeated between 30 and 126 trials).

On each trial, the appearance of the ST cued a randomly chosen corner of a 12° side length hexagon centered on the FT and rotated of 30° relative to the apparent motion display (Fig. 2A, bottom). The ST appeared randomly between 300 and 0 ms (in steps of 50 ms) before the onset of the motion pulse and corresponded to the jump of the FT to the ST location and a concurrent replacement of the FT by a black dot (0 cd/m²) of the same size. This timing ensured that most motion pulses be played entirely during saccade preparation (mean saccade latency, the delay between ST and saccade onset, across participants was of 299.6 ± 11.9 ms). As in the fixation task, we presented motion pulses at the tracked position (n), one step (n + 1 and n − 1), two steps (n + 2 and n − 2), or three steps forward and backward (n ± 3) from the tracked position.

The number of apparent motion steps before the ST varied randomly across trials. However, participants were informed that the ST would always be located 30° clockwise relative to the current tracked blob position. This procedure made the saccade task easier to perform. More importantly, it ensured that motion pulses presented in the opposite direction of the saccade (n − 2 and n ± 3) always corresponded to the remapped locations of the tracked blob (n) and of its next predicted position (n + 1), respectively. We informed participants explicitly that the motion pulse could appear at any time and equally often in any of the six blob positions and instructed them to report the main direction of the motion pulse with a keypress.

Threshold task.

To avoid possible effects of task learning across experimental sessions and to adjust the difficulty of the task for each participant, threshold task blocks preceded fixation and saccade task blocks at the beginning of each session. During threshold task blocks, participants were instructed to track clockwise an indicated red blob while keeping the eyes steady on FT. Contrary to the fixation and the saccade tasks, participants were explicitly instructed that although the onset of the motion pulse was unpredictable, it always appeared at the tracked position (n).

We used multiple staircases to adjust the difficulty of the direction discrimination task. Each threshold task session consisted of two blocks of 72 trials each. In each block, we ran simultaneously two staircases starting at two extremes of the dispersion parameter κ. In each staircase, three consecutive correct responses caused a decrease in κ while an incorrect answer caused an increase. For each participant and session individually, we determined the threshold as the reciprocal measure of the κ value leading to correct main motion direction discrimination in 79.4% of the trials. To do so, we fitted cumulative Gaussian functions to performance gathered in both threshold task blocks across the different staircases. We used this threshold in both the fixation and the saccade tasks for all the different positions tested (n, n + 1, n + 2, n − 1, n − 2, n ± 3), such that any spatial allocation of attention would lead to a local increase in performance.

Data preprocessing.

Before proceeding to the analysis of the behavioral results, we scanned offline the recorded eye data. We detected saccades based on their velocity distribution (Engbert and Mergenthaler 2006) using a moving average over 20 subsequent eye position samples. Saccade onset was detected when the velocity exceeded the median of the moving average by 3 SDs for at least 20 ms. We included trials if a correct fixation was maintained within an 2.5° radius centered on FT (all tasks), if a saccade started at FT and landed within a 2.5° radius centered on ST (saccade task only), and if no blink occurred during the trial (all tasks).

In the saccade task, we included only trials for which saccades started before the tracked blob jumped to the next location, n + 1. This selection ensures that motion pulses presented at the positions n − 2 and n ± 3 were located on the retina at the positions that, respectively, n and n + 1 will have after the saccade (their remapped positions). We included 4,598 trials (99.7% of the online selected trials, 90.1% of all trials) in the fixation task and 4,727 trials (68.8% of the online selected trials, 64.9% of all trials) in the saccade task.

Behavioral data analysis.

For each participant, we computed performance (percentage of correct discrimination of motion pulse direction) as a function of 1) the position of the motion pulse relative to the position of the tracked blob (fixation and saccade task), and 2) the time of the offset of the motion pulse relative to the onset of the saccade (saccade task). In the saccade-locked analysis (see Figs. 4 and 5), we binned trials in a 100-ms moving average (corresponding to the motion pulses duration), stepping every 50 ms from 250 to 0 ms before the saccade onset (we also computed results for the last 50 ms preceding the saccade). Next, we drew (with replacement) 10,000 bootstrap samples from the original values and computed 10,000 means. Finally, to compare performance between two conditions, we subtracted the mean of the first condition from that of the second and derived two-tailed P values from the distribution of these differences. For the figures, we also determined within-participants bootstrapped 95% confidence intervals by first normalizing each participants' mean across conditions and scaling the variance of the bootstrap distribution by M/(M − 1), where M is the number of within-participants conditions in the analysis (Morey 2008).

Fig. 4.

Saccade task. A–C show the discrimination performance, averaged across participants, computed for different time widows of motion pulse offset relative to saccade onset (gray bar). Motion pulses were presented either at the tracking side (n and n + 1; A), the control positions (n − 1 and n + 2; B), or the remapping side (n − 2 and n ± 3; C). Each data point is the mean across 6 participants. Lines are spline fits interpolating between time windows. Shaded areas represent within-participants 95% CIs, determined with a bootstrap procedure (10,000 iterations). The icons in A–C, top left corner, illustrate the position tested. Asterisks below the time axis show significant differences between the 2 positions in a given graph, evaluated separately for each time window.

Fig. 5.

Comparison of saccade and fixation tasks. A: each panel shows the average performance across the 6 participants in the saccade task (red) and the fixation task (blue), computed for different time windows of motion pulse offset relative to the saccade onset. Time windows last 100 ms each and are separated by 50 ms, ranging from 250 ms (top left) to 100 ms (bottom right) before saccade onset (bottom insets illustrate the time window for each panel). Conventions are as in Fig 3A; asterisks show significant differences between tasks. B: heat map illustrating performance differences between saccade and fixation tasks, with a color scale going from red (saccade task significantly superior to fixation task) to blue (saccade task significantly inferior to fixation task), with central beige indicating no difference. Sectors represent positions relative to the tracked object n and circles represent temporal windows going from 250 ms (inner circle) to 100 ms (outer circle) before saccade onset. Differences are interpolated (bicubic) both in temporal and spatial directions, and asterisks show significant differences between tasks for each of the spatiotemporal bins.

To determine attentional peak angles for different motion pulse times within an apparent motion step (Fig. 3B), we computed for each participant individually the means of Gaussian functions fitted to early and late motion pulses at different positions relative to the tracked blob. Early pulse tests were computed from trials with motion pulses displayed from 50 (onset) to 150 ms (offset) and from 250 to 350 ms (motion pulse mean time of 200 ms, corresponding to a mean angular rotation of 20°), while late pulse tests were computed from trials with motion pulses displayed from 250 to 350 ms and from 450 to 550 ms after an apparent motion step (mean of 400 ms, corresponding to mean angular rotation of 40°). Using this procedure, we determined the mean angles between the tracked blob position (n, or 0°) and its next step (n + 1, or 60°) and the attention peak both during early and late pulses and compared these angles with the locations that an object moving at a constant velocity would have at these instants.

Fig. 3.

Fixation task. A: performance with a scale ranging from 0% in the center to 100% at the outer edge of the graph. Motion pulse location is expressed relative to the tracked position n, rotated for all trials such that n is the top right sector. Blue dots represent means across the 6 participants; the blue line interpolates between test positions. B: a Gaussian function describes performance as a function of motion pulse location relative to the tracked object, and was separately fitted for the presentation of early (blue) or late motion pulses (light blue); see methods. The peak of the attentional focus is given by the mean angular rotation relative to the tracked object n. C: estimated attentional peak as a function of the mean presentation time of the motion pulse for each participant individually (gray lines) and their average (black line) The dashed line represents the (implied) angular velocity of the tracked object (100°/s). The blue shaded area (A) and error bars (B and C) represent within-participants 95% confidence intervals (CIs) determined with a bootstrap procedure (10,000 iterations).

RESULTS

Our goal was to determine the allocation of attention when tracking a moving object both during fixation and just before a saccade. We used an ambiguous apparent motion display to isolate effects of attention from possible effects of changes in visual stimulation (Shioiri et al. 2000; Verstraten et al. 2000; Shioiri et al. 2002). The display was composed of six colored blobs forming the six corners of an imaginary hexagon, rotating in ambiguous apparent motion around a FT (Fig. 2A). Participants continuously tracked a single red blob in a clockwise direction (Fig. 2A, top), identified by an initial brightness difference that decremented to the luminance level of the other blobs within the first two motion steps. This guidance then ended and subsequent tracking had to be maintained with attention. During sustained tracking, we probed participants' visual sensitivity to a brief pulse of coherent motion presented in one of the colored blobs (Fig. 2B). Visual discrimination performance is highly sensitive to spatial covert attention (Posner 1980; Carrasco 2011), allowing us to study attentional deployment to the tracked position (n), as well as one step (n + 1 and n − 1), two steps (n + 2 and n − 2), and three steps (n ± 3) forward (+) and backward (−) from the tracked position (Fig. 2C). Importantly, any differences in performance between the positions tested can be attributed to the tracking-dependent different allocation of attentional resources, as all motion pulses appeared after the initiation of tracking (when blobs were equiluminant).

Attentive tracking during fixation.

Figure 3A shows the discrimination performance in the fixation task, averaged across six participants, for the different positions of the motion pulse relative to the tracked object n. Overall, performance was best at the tracked object than at any other location (n: 76.6 ± 3.3%, P < 0.001) and worst at the opposite location (n ± 3: 39.3 ± 2.9%, P < 0.01). Moreover, attention spread across the positions adjacent to the tracked object, with better performance one step forward than one step backward (n + 1: 65.5 ± 4.1% vs. n − 1: 45.4 ± 5.9%, P < 0.0001) as well as better performance two steps forward than two steps backward (n + 2: 60.6 ± 4.1% vs. n − 2: 52.5 ± 3.6%, P < 0.0001). These results suggest that participants preferentially allocate attentional resources to the tracked position of the moving object and along its future path than at the other locations.

Moreover, as motion pulses were presented at different times during the apparent motion step of the tracked object (see methods), we were able to estimate the shift of the focus of attention during tracking. To this end, we determined the peak of the attentional allocation as the angular distance, relative to the tracked object, at which participants' performance was highest for pulses occurring either early (mean motion pulse time of 200 ms) or late (400 ms) during the apparent motion step of the display (Fig. 3B). Figure 3C shows the attentional peak as a function of the mean presentation time of the motion pulse (early vs. late after the apparent motion step) for each participant separately (gray lines) and on average (black line). The peak of the attentional focus closely followed the angular velocity of the apparent motion display (dotted line) as indicated by an absence of a significant deviation from the expected 100°/s linear displacement of the attentional focus from the tracked position n to the next, n + 1 (early motion pulses: 21.4 ± 4.9 vs. 20°, P = 0.76; late motion pulses: 41.6 ± 3.2 vs. 40°, P = 0.55). During fixation, therefore, participants track a moving object by smoothly and predictively shifting attention from its current to its future position.

Attentive tracking when preparing a saccade.

Next, we assessed the allocation of attention when, instead of fixating, participants were cued to make a saccade while tracking the object. We found that when participants prepared a saccade, performance at the tracked position n was still better than at the position one step forward (across all time windows n: 47.7 ± 7.4% vs. n + 1: 40.5 ± 5.2%, P < 0.005) and one step backward (n vs. n − 1: 41.6 ± 5.6%, P < 0.05). However, preparing a saccade modified the allocation of attentional resources in the direction opposite that of the saccade with performance at the tracked position n being now worse than at the remapped tracked position n − 2 (n vs. n − 2: 60.4 ± 6.1%, P < 0.001) and marginally higher than performance at the remapped location of the next tracked position n ± 3 (n vs. n ± 3: 42.3 ± 5.4%, P = 0.052).

In Fig. 4, we show the temporal dynamics of performance in the saccade task along with the average discrimination performance of six participants in the saccade task computed for four different time windows (relative to the saccade onset) in which the motion pulse offset could occur. Results are plotted in Fig. 4, A–C, for pairs of positions that share the same eccentricity from the ST, either on the tracked side of the display (Fig. 4A), at two control positions in between (Fig. 4B), or at the remapping side (opposite direction to the saccade, Fig. 4C). These graphs illustrate that attentional resources are allocated toward the remapping side with performance at position n − 2 surpassing performance at all other positions (0.001 > P > 0.0001) and the benefit does not extend to nearby positions such as n ± 3 and n − 1 (P < 0.001).

Contrary to previous studies of attentional remapping (Rolfs et al. 2011; Jonikaitis et al. 2013), our procedure here was designed to evaluate the updating of attention in dynamic displays. Performance in the saccade task seems to provide evidence for the updating of attention to the remapped tracked position n − 2; however, the allocation of attention before a saccade must be compared with the baseline attentional profile observed during fixation to reveal the scale of this effect.

Figure 5A shows the comparison between discrimination performance at the different positions tested in the saccade (in red) and in the fixation task (in blue) for the four different time bins preceding the execution of the eye movement. Whereas in the fixation condition participants correctly reported the direction of the motion pulses at the tracked position n in more than three quarters of the trials, they correctly reported the motion direction of the same pulses in less than half of the trials when preparing a saccade in the direction of the tracked object (n before saccade: 48.7 ± 5.2% vs. during fixation: 76.6 ± 3.3%, P < 0.0001). Similarly, compared with the fixation condition, performance dropped for the two subsequent positions of the tracked objects (n + 1 before saccade: 40.5 ± 3.5% vs. during fixation: 65.5 ± 4.1%, P < 0.0001; n + 2 before saccade: 45.9 ± 3.1% vs. during fixation: 60.6 ± 4.1%, P < 0.0001).

This disengagement of attention from the tracked positions is not simply a general effect of adding a secondary task, i.e., making a saccade. First, such an effect of saccade preparation would have degraded performance at all locations (which it did not, see below). Second, performance at the tracked position n increased during saccade preparation (Fig. 4A) and reached a similar level of performance as during fixation during the very last 50 ms preceding the saccade (from 50 to 0 ms before saccade onset: 71.2 ± 6.7% vs. fixation: 76.6 ± 3.0%, P = 0.41; results not shown in Figs. 4 and 5).

Next, we examined performance at the remapped positions, which, before the saccade, have no functional relation to the explicit tracking task. These positions (n − 2 and n ± 3) are not the upcoming location of the target in the world but are the expected next location of the target on the retina. As these locations are irrelevant for the tracking task, any performance advantages here must reflect reflexive compensation for the upcoming eye movement. Specifically, as saccades were always made towards targets in the direction in between the tracked object (n) and its future location (n + 1), motion pulses presented in the opposite direction of the saccade at n − 2 and n ± 3 corresponded to the remapped positions of the tracked positions n and n + 1, respectively (Fig. 2B). Indeed, for these remapped positions, we found that participants discriminated motion pulses significantly better during saccade preparation than when they performed the tracking task during fixation. Most notably, in the last 100 ms preceding a saccade (Fig. 5A, bottom right), participants attended to both the remapped tracked position n − 2 (from 100 to 0 ms before saccade: 64.8 ± 4.6% vs. fixation: 52.5 ± 3.6%, P < 0.005) and the remapped upcoming tracked position n ± 3 (from 100 ms to 0 ms before saccade: 48.3 ± 6.5% vs. fixation: 39.3 ± 2.9%, P < 0.05). These effects follow the typical time course of remapping; we only found a significant improvement of performance at the remapped positions if the motion pulse appeared within 100 ms of saccade onset (Kusunoki and Goldberg 2003; Rolfs et al. 2011; Jonikaitis et al. 2013) and not for motion pulses presented earlier during saccade preparation (before saccade: 61.8% > n − 2 > 56.6% vs. during fixation: 52.5%, P > 0.11; before saccade: 41.7 > n ± 3 > 39.2% vs. during fixation: 39.3%, P > 0.10).

In both tasks, we presented the motion pulses while participants fixated the center of the screen. The preparation of a saccade profoundly modified the deployment of attention both in the proximity of the tracked object (“tracking side”) and in the opposite direction of the saccade (“remapping side”). Figure 5B highlights this predictive transfer of attentional resources by visualizing the statistical comparison of the two tasks (two-tailed P values) for each position relative to the tracked object (represented as sectors) and different times preceding the saccade progressing towards more eccentric circles. When a moving object was tracked, the preparation of a saccade resulted in a pronounced withdrawal of attention from the tracked object's current path in space (bluish colors at n, n + 1, n + 2), followed by a remapping of attention to the path's future retinal location (reddish colors at n − 2, n ± 3 in the outer circle of the graph).

Note that we cannot completely exclude the possibility that the drop of performance at the tracked side is due to the proximity of the onset of the ST. Indeed performance was lowest at the nearest positions to the ST (n and n + 1) in the first 150 ms following its onset (Fig. 6). To investigate the impact of the onset of the ST on the allocation of attention, we analyzed the evolution of visual performance relative to ST onset (instead of saccade onset). Specifically, we computed the percentage of correct discrimination of the motion pulse direction in the saccade task for each position of the motion pulse relative to the tracked position at four different times after ST onset. As in the saccade-locked analysis (Figs. 4–5), trials were binned in 100-ms moving windows but stepping from 200 ms (motion pulses ending 200 ms after ST onset) to 350 ms after ST onset.

Fig. 6.

ST-locked analysis of the saccade task. A–C show the discrimination performance, averaged across participants, computed for the different time windows of the motion pulse offset relative to the ST onset. Motion pulses were presented either at the tracking side (n and n + 1; A), the control positions (n − 1 and n + 2; B), or the remapping side (n − 2 and n ± 3; C). Conventions are as in Fig 4.

The ST-locked analysis yielded roughly the same results as the saccade-locked analysis with the exception that the remapping benefits when compared with the fixation task were less sharply spatiotemporally tuned. Figure 6 shows the temporal dynamics across four different time bins after ST onset of the discrimination performance in the saccade task; Figure 7 shows the comparisons between the fixation and saccade task.

Fig. 7.

Comparison of saccade and fixation tasks, locked to ST onset. A: each panel shows the average performance across the 6 participants in the saccade task (red) computed for different time windows of motion pulse offset relative to the movement cue (ST). Time windows last 100 ms each and are separated by 50 ms, ranging from 100 ms (top left) to 250 ms (bottom right) after the ST (bottom insets illustrate the time window for each panel) For comparison, fixation task performance (blue) is plotted in each panel. Conventions are as in Fig. 3A and Fig. 5; asterisks show significant differences between tasks. B: heat map illustrating performance differences between saccade and fixation tasks, with a color scale going from red (saccade task significantly superior to fixation task) to blue (saccade task significantly inferior to fixation task), with central beige indicating no difference. Sectors represent positions relative to the tracked object n and circles represent temporal windows going from 100 ms (inner circle) to 250 ms (outer circle) after AT onset. Differences are interpolated (bicubic) both in temporal and spatial directions and asterisks show significant differences between tasks for each of the spatiotemporal bins.

In this analysis, performance at the tracked position n was better than at any other location (n: 51.3 ± 6.4%, P < 0.05) with the exception of the remapped position n − 2 (n vs. n − 2: 62.0 ± 4.7%, P < 0.001). Interestingly when we compared performance for locations close to the ST (positions n and n + 1), we found that performance was lowest during the first 200 ms following the onset of the ST; it then increased continuously over time with the difference between the tracked (n) and the next tracked (n + 1) positions becoming significant in the time window for 150 to 250 ms after ST onset (P < 0.05, see Fig. 6A). The onset of the movement cue (ST) may have initially perturbed attentional tracking of the moving object in the saccade task, a result that is in accordance with several studies showing that participants are unable to voluntarily monitor one eccentric location while preparing a saccade to another (Deubel and Schneider 1996; Montagnini and Castet 2007).

However, does the onset of the ST explain the withdrawal of attention from the tracked side? To answer this question it is necessary to take into account the attentional profile we found for tracking during fixation, in which no movement cue (ST) appeared. Indeed, during the first 200 ms following ST onset, participants correctly reported the direction of the motion pulse at the tracked position n in only two-fifths of the trials (n from 100 to 200 ms after ST onset: 40.6 ± 5.6% vs. during fixation: 76.6 ± 3.3%, P < 0.0001). Similarly, performance dropped at the two next positions of the tracked object (n + 1 from 100 to 200 ms after ST onset: 37.5 ± 4.4% vs. during fixation: 65.5 ± 4.1%, P < 0.0001; n + 2 from 100 to 200 ms after ST onset: 44.4 ± 2.1% vs. during fixation: 60.6 ± 4.1%, P < 0.0001). This disengagement of attention, however, was maintained across time at each of these locations: at the tracked position n (40.6% > n > 64.9% vs. during fixation: 76.6 ± 3.3%, P < 0.0001), the next tracked positions n + 1 (37.5% > n + 1 > 46.1% vs. during fixation: 65.5 ± 4.1%, P < 0.0001), and n + 2 (44.4% > n + 2 > 47.3% vs. during fixation: 60.6 ± 4.1%, P < 0.05).

These ST-locked results allow us to determine whether there are any effects of exogenous attention driven by the onset of the ST. Specifically, the exogenous attention attracted by the ST onset would draw attention away from the tracked locations and these costs should have peaked at ∼100 ms after ST onset and then disappeared shortly thereafter (e.g., Nakayama and Mackeben 1989; Fuller et al. 2008). Although we found that performance was worst at the tracked locations (n and n + 1) at the time exogenous attention would normally peak (Fig. 6A), we also found that the cost at these locations remained significant up to 350 ms after ST onset (Fig. 7). This result argues against exogenous attention to the ST as the main source of the withdrawal of attention before the saccade. More importantly, an exogenous shift of attention to the ST could not explain the increase of attention to the remapped locations.

An alternative explanation for the increase in performance in the opposite direction of the saccade is that it reflects a release of a performance bias towards the tracked positions. We showed that, during fixation, performance is better at (or, biased toward) the tracked positions than at the opposite positions. The preparation of the saccade might have released this bias, leading to a relative increase of performance at all untracked locations. In this scenario, the increase at the remapped locations would not be due to remapping but simply due to a return to baseline at the untracked locations following the loss of the positive bias at the tracked location. We can rule out this alternative, based on three results. First, it would predict an equalization of performance across all locations just before the saccade, which we did not find: instead, performance was best at the remapped locations just before saccades. Second, we compared performance at the remapped tracked position (n − 2) before the saccade with the average of the six tested locations during fixation (representing participants overall performance). We found that performance at the remapped tracked location is better than the average of the six positions during fixation only in the last 100 ms preceding the saccade (n − 2 from 100 to 0 ms before saccade: 64.8 ± 4.6% vs. across positions during fixation: 52.5 ± 3.2%, P < 0.005; other time windows, P > 0.05), a result that seems incompatible with a mere release of a performance bias. Finally, as pointed out earlier, performance is best at n − 2 during the last 100 ms before saccade, when participants have almost recovered their high level of performance at the tracked locations (50 ms before the saccade, performance was no longer different from that during fixation). Together, these results favor a genuine attentional trade-off, boosting performance at the remapped location of the tracked object.

DISCUSSION

We studied the deployment of visual attention during the covert tracking of a moving object by assessing participants' sensitivity to subtle pulses of coherent motion in an apparent motion display. We found that, during fixation, participants' performance was better at the object's current and immediate future location (n, n + 1) than anywhere else in the visual display (Fig. 8, A and B). In agreement with previous studies (Shioiri et al. 2000, 2002; Atsma et al. 2012), our results suggest that the tracking of a moving object is based on a smooth shift of attention from where the object is at the moment to where it soon will be.

Fig. 8.

Summary. A driving situation (A) illustrates the allocation of attention during fixation (B) and just before a saccade (C). During fixation, we found that attention is allocated to the currently tracked and the next tracked positions of the ambulance (nonblurred yellow circles in B). However, when participants plan a saccade to the speed limit sign (white arrow in A), attention is first withdrawn from the tracked positions (blurred yellow circles in C) to later be transferred (orange arrow in C) to the remapped positions of the tracked and the next tracked positions of the ambulance (nonblurred orange circles in C) These locations in the world are irrelevant for the cognitive demands of the task and are only meaningful in the context of the reflexive preparation for the saccade and its consequences for the projection of the tracked object on the retina.

The allocation of attention during tracking changed drastically when participants prepared a saccade. Performance at the current position of the object (n) as well as at the positions it would move to next (n + 1 and n + 2) dropped severely in the presaccadic interval. However, just before the eyes started to move, visual performance increased selectively at locations in the opposite direction of the saccade (Fig. 8, A–C). This performance advantage is incompatible with any explanation that is based on the cognitive demands of the task. Instead, as these retinal locations will fall upon the object once the saccade has landed, this spatially specific (restricted to the remapped but not to intermediate locations) reallocation of attention appears to reflexively anticipate the consequences of the saccade.

This study reveals a presaccadic drop in visual performance at a behaviorally relevant spatial location. We have previously observed predictive updating of attention for both salient objects that captured spatial attention involuntarily (Jonikaitis et al. 2013) and for locations attended as a target in a sequence of saccades (Rolfs et al. 2011), but in neither of these studies did we observe a presaccadic drop in visual performance at the relevant location. Previous research has shown that participants are unable to voluntarily monitor one eccentric location while preparing a saccade to another (Deubel and Schneider 1996; Montagnini and Castet 2007), which has been interpreted as a competition of voluntary attention and saccade preparation for shared resources. Our findings suggest an updating of voluntary covert attention to the postsaccadic retinal coordinates of the attended objects.

Taken together, our results suggest that attentive tracking relies on predictive mechanisms, allocating attention to the moving target's current and future positions during fixation and transferring these resources in anticipation of the visual consequences of the saccade. This transfer can only be inferred, however, as we have no way of tracking the origin of the attentional resources deployed to the remapped location. However, a growing number of studies are consistent with a predictive trade-off of attentional resources, bolstering our proposal.

Specifically, several visuomotor areas play crucial roles in both the deployment of visual attention and the generation of saccades, including the frontal eye field (FEF) and lateral intraparietal area (LIP), as well as the superior colliculus (SC) at the subcortical level (Awh et al. 2006; Moore 2006; Bisley 2011; Krauzlis et al. 2013), with a division of labor at the level of neural subpopulations (Ignashchenkova et al. 2004; Juan et al. 2004; Thompson et al. 2005; Gregoriou et al. 2012). Using functional MRI in humans, Culham et al. (1998) showed that attentive tracking of moving objects activated FEF and the intraparietal sulcus (IPS; the human homologue of LIP) more than twice as much as passive viewing of the same stimuli. Attentional load strongly modulates the activation in IPS but very little in FEF (Culham et al. 2001; Jovicich et al. 2001; Howe et al. 2009; Jahn et al. 2012). This difference has been interpreted to suggest that IPS is directly involved in tracking, whereas FEF may perform more general tasks such as the control of eye movements during tracking (Culham et al. 2001; Howe et al. 2009).

Little is known about the neural mechanisms of a predictive shift of covert attention along the extrapolated path of a tracked object. Single-cell recordings suggests that both LIP and FEF may encode such information. In LIP, neurons encode the inferred direction of an invisible moving target (Eskandar and Assad 1999, 2002). In FEF, probes flashed ahead of the extrapolated motion path of an invisible target generate the strongest responses in FEF neurons and probes flashed behind that position resulted in inhibited responses (Xiao et al. 2007). Moreover, when monkeys prepare a saccade towards a moving target, many FEF neurons encode the location the target will have upon saccade landing, as opposed to its current retinal location (Cassanello et al. 2008). Indeed, both FEF and LIP have been causally linked to visual performance in covert attention tasks (Moore and Fallah 2001, 2004; Wardak et al. 2004, 2006; Balan and Gottlieb 2009), and FEF activity is known to drive increases in firing rates in visual cortex, mimicking the consequences of selective attention (Moore et al. 2003; Moore and Armstrong 2003; Armstrong et al. 2006; Armstrong and Moore 2007). Whereas no study has explored predictive shifts of covert attention in early visual areas, recent single-cell recordings in monkeys (Niebergall et al. 2011) and event-related potentials in humans (Drew et al. 2009; Doran and Hoffman 2010; Störmer et al. 2013) have shown that visual responses in striate and extrastriate cortical areas show enhanced responses to tracked compared with untracked objects. Finally, during smooth pursuit, visual performance is highest at the current position of the target (Lovejoy et al. 2009) or even ahead of it (van Donkelaar and Drew 2002; Khan et al. 2010), both of which require predictive shifts of attention to compensate for the neural delays of the visuomotor system. Together, these findings hint at a possible neural architecture underlying the attentional enhancement in visual performance along the predicted motion path: attention-related areas index the current and future locations of a tracked object and facilitate visual processing at these locations by altering visual signals in striate and extrastriate cortex. Note that a similar mechanism could originate in the SC, or any other crucial player in the control of covert attention, provided it encodes the predicted future locations of a moving object (but see Keller et al. 1996).

How does this potential mechanism handle our result that attention remaps in anticipation of the retinal changes caused by saccades? The updating of attentional resources before saccades has been linked to the predictive remapping of neural activity observed in many attention-related visuomotor areas of the primate brain (Cavanagh et al. 2010; Rolfs et al. 2011; Jonikaitis et al. 2013). Predictive remapping, first discovered in LIP (Duhamel et al. 1992), is the anticipatory response of visual neurons that will encode visual stimulus after a saccadic eye movement. Neural responses compatible with predictive remapping have also been observed in FEF (Umeno and Goldberg 1997; but see Zirnsak and Moore 2014; Zirnsak et al. 2014) and SC (Walker et al. 1995) and even the motion-sensitive area medial superior temporal (Inaba and Kawano 2014). Thus, the same areas that may predict future locations of attentively tracked moving objects have been shown to anticipate the retinotopic consequences of eye movements. In LIP, anticipatory increases in neural activity occur only for neurons that will encode attended locations following a saccade (Gottlieb et al. 1998), providing a potential source for the visual performance benefits at the remapped location observed here. Two recent findings in FEF may help explain the drop in performance at the tracked object's current location. First, Zirnsak et al. (2014) showed that during saccade preparation, most neurons in FEF become more sensitive to stimuli in the vicinity of the target. Indeed, the enhanced responses of FEF neurons to probes along the extrapolated motion path of a tracked object diminish during saccade preparation (Xiao et al. 2007). This substantial shift of neural resources to the ST may explain the severe drop in visual performance observed in the current study, but it does not account for the enhancement of performance at the remapped location. Second, LIP neurons that will encode unattended or empty locations after the saccade predictively decrease their activity (Mirpour and Bisley 2012). Critically, this finding was made in a visual search task, which requires the voluntary control of visual attention to keep track of previously attended and unattended locations. We suggest that, in situations that tax attentional resources extensively, such as the combination of visual exploration with attentive tracking tasks, remapping trades off attentional resources between an object's current and future retinal location. The behavioral procedure introduced here may set the stage for a better understanding of the neural processes involved in attentive tracking during active visual behavior.

In summary, our results suggest that preparing a saccade results in a strong attentional disengagement at the object's current and subsequent locations on its movement path. Instead, attention shifts to the object's remapped location, facilitating tracking across the saccade. The resulting trade-off in visual sensitivity is quite pronounced, starting some time before the saccade, and should be observable at the neural level. Critically, it results in the participant's insensitivity at the tracked locations, pointing to an acute performance limit in many sensorimotor tasks performed in dynamic visual environments, such as driving, sports, and other fast-paced activities. This transient inattention demonstrates the choice the visual system makes to sustain attentional continuity across the saccade.

GRANTS

This research was supported by an Alexander von Humboldt Foundation Fellowship (to M. Szinte), National Eye Institute Grant R01-EY-016200 (to M. Carrasco), l'Agence Nationale de la Recherche and European Research Council Grants (to P. Cavanagh), and Deutsche Forschungsgemeinschaft Emmy Noether Grant RO 3579/2-1 (to M. Rolfs).

DISCLOSURES

No conflicts of interest, financial or otherwise, are declared by the author(s).

AUTHOR CONTRIBUTIONS

Author contributions: M.S., M.C., P.C., and M.R. conception and design of research; M.S. performed experiments; M.S. analyzed data; M.S., M.C., P.C., and M.R. interpreted results of experiments; M.S. prepared figures; M.S. and M.R. drafted manuscript; M.S., M.C., P.C., and M.R. edited and revised manuscript; M.S., M.C., P.C., and M.R. approved final version of manuscript.