Decoding Trans-Saccadic Memory

Abstract

We examine whether peripheral information at a planned saccade target affects immediate postsaccadic processing at the fovea on saccade landing. Current neuroimaging research suggests that presaccadic stimulation has a late effect on postsaccadic processing, in contrast to the early effect seen in behavioral studies. Human participants (both male and female) were instructed to saccade toward a face or a house that, on different trials, remained the same, changed, or disappeared during the saccade. We used a multivariate pattern analysis of electroencephalography data to decode face versus house processing directly after the saccade. The classifier was trained on separate trials without a saccade, where a house or face was presented at the fovea. When the saccade target remained the same across the saccade, we could reliably decode the target 123 ms after saccade offset. In contrast, when the target was changed during the saccade, the new target was decoded at a later time-point, 151 ms after saccade offset. The “same” condition advantage suggests that congruent presaccadic information facilitates processing of the postsaccadic stimulus compared with incongruent information. Finally, the saccade target could be decoded above chance even when it had been removed during the saccade, albeit with a slower time course (162 ms) and poorer signal strength. These findings indicate that information about the (peripheral) presaccadic stimulus is transferred across the saccade so that it becomes quickly available and influences processing at its expected new retinal position (the fovea).

SIGNIFICANCE STATEMENT Here we provide neural evidence for early information transfer across saccades. Specifically, we examined the effect of presaccadic sensory information on the initial neuronal processing of a postsaccadic stimuli. Using electroencephalography and multivariate pattern analysis, we found the following: (1) that the identity of the presaccadic stimulus modulated the postsaccadic latency of stimulus relevant information; and (2) that a saccadic neural marker for a saccade target stimulus could be detected even when the stimulus had been removed during saccade. These results demonstrate that information about the peripheral presaccadic stimulus was transferred across the saccade and influenced processing at a new retinal position (the fovea) directly after the saccade landed.

Introduction

Humans make up to four saccadic eye movements per second to direct the high-resolution fovea to locations of interest in the visual environment (Schiller, 1998). As efficient as this process may seem, it is unclear how the successive snapshots of information are combined across saccade sequences. One possibility is that information is carried across the saccade to compensate for the eye movement so that it can integrate with the information on the next fixation (McConkie and Rayner, 1976; Trehub, 1977; Paeye et al., 2017). However, many studies have demonstrated that changes to a scene made during a saccade are rarely detected (Grimes, 1996, O'Regan et al., 1999, Simons and Rensink, 2005), indicating that little, if any, perceptual information is transferred across saccades. Even though much is lost, information about attended items may be preserved (Higgins and Rayner, 2015). For example, Grimes (1996) demonstrated that changes to more salient, attention-grabbing objects of a scene were noticed in 40% more trials than changes to background objects. Furthermore, information preservation across a saccade has been demonstrated behaviorally in motion perception (Fracasso et al., 2010; Szinte and Cavanagh, 2011), detection of line interception (Prime et al., 2006; Paeye et al., 2017), object completion (Hayhoe et al., 1998), color biasing (Wittenberg et al., 2008), and identification performance (De Graef and Verfaillie, 2002). These studies have indicated that presaccadic information is available within 20–140 ms following the saccade landing. Even though behavioral signatures of trans-saccadic memory have been reported before, the early neurophysiological correlates of this information transfer remain largely unexplored. A series of studies from one group (Dimigen et al., 2012; Niefind and Dimigen, 2016; Kornrumpf et al., 2017) found a relatively late marker with electroencephalography (EEG) for parafoveal-to-foveal information transfer in reading (from 140 ms lasting until 300 ms). However, the initial postsaccadic processing was unaffected. The dissociation between the trans-saccadic information findings for behavioral versus neuroimaging measures motivated our interest in the effect of presaccadic sensory information on the early neuronal processing of postsaccadic stimuli.

We used EEG and multivariate pattern analysis (MVPA) to address this question. Specifically, we hypothesize that the accuracy and/or latency of decoding a postsaccadic stimulus will depend on whether the presaccadic stimulus at the same spatial location was either the same or different (on separate trials). The time course of decoding accuracy indicates when there is sufficient information to identify the stimulus. Trans-saccadic information transfer should improve decoding accuracy and/or decrease the latency of the peak decoding performance when the same stimulus was present before the saccade, compared with when a different stimulus was present. In contrast, if there is no memory or information transfer across saccades, the decoding of a postsaccadic stimulus should operate identically, regardless of the presaccadic information.

We also included a condition where no stimulus was present after the saccade. Early studies reported the presence of spatiotopic, persisting target information even when the stimulus had been removed during saccade (Wolf et al., 1980; Jonides et al., 1982). These studies were later overturned when the phosphor persistence was properly controlled and no effect was found (Jonides et al., 1983). Nevertheless, trans-saccadic integration studies have suggested the presence of a spatiotopic information transfer, a “ghost” illusion, that may in some cases be perceptual as well as memory based (Wolf et al., 1980; Deubel et al., 1996; Ganmor et al., 2015; Wolf and Schütz, 2015; Paeye et al., 2017). We therefore examined the timing and strength of information in the postsaccadic time period in additional trials where the target had been removed during saccade.

Materials and Methods

Participants.

Seventeen volunteers, including author G.E. (10 females; age range, 19–40 years), participated in the experiment. All participants had normal or corrected-to-normal vision. Three participants were rejected from the analysis (leaving 14 participants), because we were unable to efficiently decode between the neural signals of faces and houses when the stimuli were presented in the peripheral visual field (for further details, see the Criteria section).

Stimuli.

Stimuli were presented with a 16 inch Sony Triton Monitor (resolution, 1024 × 768; refresh rate, 85 Hz) at a distance of 50 cm from the participants. The stimuli were designed and presented using MatLab 2009a and the Psychophysics Toolbox extension (Brainard, 1997). The stimuli were presented in separate saccade and fixation blocks. Fixation blocks contained the trials used to train the multivariate pattern classifier, and the trials in the saccade blocks were fed to the classifier as the test set. Each block contained 20 trials in a randomized order. Five fixation blocks were interleaved with six saccade blocks across one run of 15 min. Participants performed four runs in total. Therefore, there were 480 saccade trials and 400 fixation trials presented to each participant.

Fixation blocks.

There were two fixation conditions: “central” and “peripheral” (Fig. 1a). In the central condition, participants fixated a fixation marker, which was presented 6° to the right of the screen center. After 200 ms, either a face or a house (3.4° height, 3.3° width) was then presented for 500 ms, replacing the fixation marker. Low-level stimulus features (i.e., global luminance, contrast, spatial frequencies, and 2D Fourier power spectrum) of the two possible images were equalized by spectral normalization (Senoussi et al., 2016). In the peripheral condition, the trials began with the same fixation marker as in the central condition. After 200 ms, a face or a house was presented in the periphery (10° to the left of the fixation marker) for 500 ms while the participants remained fixated on the fixation marker. To keep the participants' attention on the image in both conditions, participants were required to perform a one-back task to determine whether the image presented in trial n was the same as or different from the image presented in n − 1 (regardless of stimulus position).

Figure 1.

Saccade and fixation task stimulus. a, Fixation conditions. In both conditions, subjects fixate between the two dashed lines for 200 ms. An image (a house or a face) was then presented either centrally or 10° to the left of fixation for 500 ms. The black bar across eyes of the face is for publishing purposes only; the bar is not present in experimental stimuli. Participants are required to keep their fixation regardless of the position of the stimuli and report whether the image in trial n is the same or different to the image in trial n − 1. In this example only the face stimulus is shown; however, there was equal likelihood of the presentation of the house stimulus. b, Saccade conditions. In all conditions, participants fixate on the empty space between the two dashes. After 200 ms, an image was presented 10° to the left of the fixation point. The image could be a house or a face. Participants remain fixated on the fixation point while attending to the image for 500 ms until the fixation point disappears, which cued subjects to saccade to the image. In the same condition, participants' saccade would land on the same stimulus, whereas in the change condition participants' saccade would land on a different image. These images would be presented for 45 ms after saccade offset. In the disappear condition, the image would disappear as soon as the saccade was detected, so that the saccade would land on an empty space. Subjects were instructed to respond “same” if they landed on the same image, or “different” if they landed on the changed image.

Saccade blocks.

There were three saccade conditions: “same”, “change”, and “disappear” (Fig. 1b). In each condition, the trial began with a fixation point presented 6° to the right of the screen center. After 200 ms, a grayscale image (either a face or a house, 3.4° height, 3.3° width) was presented to the left of the screen, 10° from the fixation point. The fixation point was removed 500 ms later, which cued participants to perform a saccade toward the image. In the same condition, the image remained on screen throughout the saccade and for 45 ms after saccade landing. In the change condition, the image changed during the saccade (which was detected on-line) so that the participant's saccade would land on a new image, which remained on screen for 45 ms postsaccade. The change was made once the saccade crossed a 1.5° boundary to the left of the fixation dashes. The trial was restarted if participants initiated saccade before the removal of the fixation dashes. The new image would be a house if a face were presented before saccade (Fig. 1a) or the new image would be a face if a house were presented before saccade. In the disappear condition, the image would disappear once the saccade was initiated, meaning that the saccade would land on the blank, gray background. Phosphor persistence was measured using an oscilloscope and photodiode, and the signal from a light spot on a black background was found to drop to <1% of its peak luminance by a maximum of 11 ms after stimulus offset. Therefore, stimulus, which was light and dark on a gray background, had faded from the phosphors long before the saccades landed in the disappear condition (mean saccade duration, 51.5 ms). All saccade trials were performed with a leftward saccade. The participants' behavioral task was to determine whether the image presented before the saccade was the same as or different from the image presented after the saccade. Importantly, participants were not made aware that the saccade target could disappear during saccade.

Experimental design and statistical analyses.

The experiment was performed as a within-subjects design; each participant completed all conditions of the experiment. There were three test conditions in the saccade block and two training conditions in the fixation block, which are described in detail above. Using the training conditions, we analyze the participants' EEG signal for different neural signatures in the three test conditions. The postsaccadic neural signals were expected to be different depending on whether an image remained the same, changed, or disappeared during a saccade. Repeated-measures analyses (ANOVA and paired t tests) were used to compare neural signatures across the three saccade conditions. One-sample t tests were performed in each condition to determine those with a significant difference from chance (50%). Multiple comparisons were based on Bonferroni-corrected p values from paired t tests performed at each time-point (Shaffer, 1995). Behavioral analyses were also performed using repeated measures. Specific details of each analysis are presented below.

Behavioral data analysis.

Analysis of the saccade task and fixation task was performed in MatLab 2016a. For the saccade task, the mean correct responses and reaction times (RTs) were calculated across participants for each saccade condition. A correct response was defined as responding “same” in the same condition, “different” in the change condition, and “same” in the disappear condition where the participant might perceive the presaccadic stimulus as still present (Wolf et al., 1980), but a priori would have no reason to report the opposite stimulus. A fixed-effect one-way ANOVA was performed on the percentage correct responses and mean reaction times across participants and conditions. Responses were performed in a designated response period 300 ms after each trial was concluded. A designated response time window was used to reduce motor response noise during the postsaccadic time period. This constraint ruled out their use for reaction time analyses. For the fixation task, subjects performed a one-back task, and the mean correct responses were calculated across participants. A t test was conducted for the group analysis.

Eye-tracker data acquisition and analysis.

The timing of the stimulus sequence on saccade trials was locked to the detection of the saccade initiation. We used an Eyelink 1000 Plus eye-tracking system (SR Research) to record participants' eye movements throughout the runs. Calibration was conducted at the beginning of each run and again during the runs if the Eyelink 1000 Plus lost the ability to track the pupil. Participants' saccade onsets and offsets were extracted and combined with the EEG triggers for analysis of the EEG signal. We found a consistent delay of 4.28 ms between saccade onset and the EEG triggers for saccade onset embedded in the EEG signal.

EEG acquisition and preprocessing.

EEG and EOG were recording using a BioSemi system, with 64 active electrodes and 3 ocular electrodes at 1024 Hz. The continuous EEG data were preprocessed off-line using MatLab and the EEGLAB toolbox (Delorme and Makeig, 2004). First, the continuous data were notch filtered ∼50 Hz to remove electrical artifacts, then they were bandpass filtered between 0.1 and 80 Hz, and finally downsampled to 256 Hz. Saccade trials were analyzed in the following two epochs: the presaccade epoch was locked to the stimulus onset and covered 200 ms before stimulus onset until 300 ms after stimulus onset; the postsaccadic epoch was time locked to the saccade offset and covered 200 ms before saccade offset until 300 ms after saccade offset. Fixation trials were analyzed around stimulation onset, with 200 ms prior to onset and 300 ms post-onset. The data collected 200–100 ms before stimulus onset or saccade offset for each trial were used to baseline the remaining data in the trial. Individual electrodes with artifacts were interpolated by the mean of the adjacent electrodes, and manual rejection of epochs with artifacts was performed (average rejected epochs of saccade trials, 87.5 of 480 epochs; SD, 14.5; average rejected epochs of fixation trials, 32.4 of 400 epochs; SD, 6.3).

Multivariate pattern analysis.

MVPA distinguishes among electrophysiological signals associated with distinct brain states. Here, a linear classifier (Crouzet et al., 2015) was used to distinguish between processing face and house stimuli in the peripheral or foveal region of the visual field at specific time-points. Importantly, we designed our experiment to ensure that the classifier decoded between only face and house information (present either peripherally or foveally in the visual field). The training conditions purposefully did not include any saccade, enabling our analysis to focus on contextual information transfer rather than motor-related discrepancies between conditions.

First, we tested the accuracy of the classifier on our two training sets (central or peripheral fixation trials) separately, at each time-point independently. The classifier performance was tested using a Monte-Carlo cross-validation procedure (n = 50). On each cross-validation iteration, we randomly selected 90% of fixation trials to train the classifier and tested the classifier on the remaining 10% of fixation trials. There were always an equal number of face trials to house trials in the training set. To increase the signal-to-noise ratio, we subaveraged every three trials in the training set and in the test set (Isik et al., 2014; Grootswagers et al., 2017). Averaging across three trials was performed on each iteration in each condition separately after the trial order in each condition was randomized. We averaged at each time-point, essentially creating one time course from three trials. On each cross-validation, the signal of each electrode was normalized across trials (z-score) at each time-point. Once the classifier was trained to distinguish between the electrophysiological signals elicited by face versus house trials, a label was provided by the linear classifier at each time-point for each set of subaveraged trials in the test set. After 50 iterations, the percentage of correct labeling was calculated per participant. For group analysis, classifier performance was averaged at each time-point across participants and presented with nonparametric 95% confidence intervals. Classification accuracy was considered to be above chance (50%) by Bonferroni-corrected p values (Shaffer, 1995) from t tests performed at each time-point.

Next, we trained the classifier using the peripheral fixation trials, and test it with the first 300 ms of saccade trials, before any saccade (at the beginning of the saccade trials, subjects are attending to a face or house stimulus presented in the periphery, comparable to the stimulation in peripheral fixation trials). As described above, the classifier was trained on each time-point of the peripheral fixation trials and tested at the corresponding time-point from the stimulation onset of the saccade trials. The data were randomized and subaveraged across three trials within each trial type (as described above). Importantly, the z-score normalization was performed on the training and test dataset separately. For each time-point within the first 300 ms of each saccade trial, the classifier would then label the trial as face or house according to the learned patterns of the peripheral fixation trials (chance = 50%). Classification accuracy for each participant was determined from 10 iterations of randomizing and subaveraging the data in each trial type. Group analysis was performed as described in the above paragraph.

MVPA of the postsaccadic time-period was performed using the central fixation condition. The postsaccadic time-period was defined from saccade offset to 300 ms after saccade offset. This time-period is most similar to the central fixation condition, when processing is occurring at fixation. Importantly, here we used the classifier trained at the time-point of peak decoding accuracy for the central training set (at 140 ms) and then tested this classifier on every time-point of the saccade trials from saccade offset. This method was used because we were uncertain of the “reference” time at which information would become available across the saccade (and, as the analysis revealed, this time could be different for the different saccade conditions same, change, and disappear). The following multivariate analysis per subject and group analysis were performed as described above. Latency of the postsaccadic decoding of the same and the change condition was quantified by selecting the peak of the classification performance for each condition in each participant after saccade offset. A t test was performed to determine whether there was a significant difference between processing latency for the same versus the change condition.

The disappear condition was used to test for trans-saccadic information presence without postsaccadic stimulation, and, as a comparison, we used the peripheral fixation condition because it was exactly the same as the disappear condition, but without saccade. Here, we trained the classifier at the time-point of peak decoding accuracy for the central training set (at 140 ms) and test this classifier on every time-point of the peripheral training trials from stimulation offset (plus 51 ms, to simulate the saccade latency in the disappear condition). This analysis enabled a direct comparison between information within the visual system with saccade (disappear condition) and information in the visual system without saccade (peripheral fixation condition).

The final MVPA analysis performed on the postsaccadic time-period examined how information generalized across time. The classifier was trained on every time-point of the central fixation conditions and tested at every time-point of the postsaccadic period of the saccade conditions. This resulted in a matrix of decoding accuracy values, where the diagonal relates to corresponding time-points between training and test trials.

Criteria.

Subjects were removed from analysis if the classifier trained on peripheral fixation trials was unable to decode between face and house stimuli (presented peripherally) before saccade. Specifically, the average classification performance was derived from 100 to 300 ms after stimulation onset, and if the 95% confidence interval included chance (50%), then the subject was removed from following analysis (three participants were removed). Chance classification performance before saccade could have occurred for several reasons, including poor signal-to-noise ratio or lack of proper attention to the peripheral stimuli. This rejection criterion was used because our main question of interest (the potential transfer of stimulus information across the saccade) only makes sense when information is actually present and detectable before the saccade.

Results

Behavioral data

Participants performed two tasks during the experiment. In the saccade task, the participants indicated whether the stimulus before saccade was the same as or different from the stimulus they perceived after saccade. In the fixation task, participants performed a one-back task to indicate whether the image presented in trial n was the same or different to the image presented in n − 1, regardless of the spatial position of the image (central or peripheral).

All participants performed the matching task in the saccade conditions correctly above chance (>50%, p < 0.0001, one-sample t tests). In the group analysis, participants reported that the stimuli were the same in 95.1% (SEM, 2.4) of the same trials. They reported the change in 96% (SEM, 1.3) of the change trials. In the disappear condition, participants reported that they saw the same image in 95.7% (SEM, 2) of the trials, when in fact there was no image present. There was no evidence for a difference in participants' performance across conditions (F_(2,39) = 0.05, p = 0.9482, ANOVA). However, the participants were not given an option to report “neither” or “nothing,” so we cannot conclude that they actually perceived a postsaccadic persisting, spatiotopic image, nor can we rule it out. RTs were calculated from the beginning of the response period that followed the 300 ms no-response interval and did not differ across the conditions (F_(2,39) = 0.12, p = 0.8843, ANOVA), with participants performing the task with a mean RT of 191.2 ms (SEM, 16.5) for the same condition, 199.4 ms (SEM, 17.5) for the change condition, and 188 ms (SEM, 17.6) for the disappear condition. However, it is important to note that responses were recorded during a response period after the conclusion of the trial and therefore do not reflect “true” reaction times.

All participants performed the one-back fixation task significantly above chance (50%; p < 0.0001, one-sample t tests). Across the group, participants performed the one-back task correctly on 93.9% (SEM, 1.7) of trials, which was significantly above chance (t₍₁₃₎ = 26.3667, p < 0.0001, one-sample t test).

EEG data

Using EEG and MVPA, we examined whether saccade target information affects postsaccadic target processing. If the saccade target changed during the saccade, we expected an alteration in processing latency and/or processing performance of the new target after saccade. We further hypothesized that participants could perceive the saccade target momentarily after saccade landing, even when it was removed during saccade, and that we would find a reflection of this illusory percept in the EEG signal.

Classifier accuracy

First, we tested the accuracy of the classifier after being trained on the two separate sets of training trials, peripheral and central. Using a leave-10%-out Monte-Carlo cross-validation procedure (see Materials and Methods), we found that the classifier worked effectively at labeling the test set trials when it was trained using the peripheral trials and the central trials. When the classifier was trained on the peripheral fixation trials, the percentage performance showed two peaks at 179 and 246 ms with a performance of 59.3% and 59%, respectively (Fig. 2a). This decoding accuracy on peripheral stimuli is similar to that found previously by Carlson et al. (2011), even though our peripheral stimuli were 3° further from fixation than in their study. The decoding topographies of the peripheral classifier suggest that frontal, central, and occipital electrodes contribute to both decoding performance peaks. The peak at 179 ms is slightly lateralized on the right, which is expected because the face and house stimuli were presented to the left of fixation. For the classifier trained on the central trials, classification performance peaked at 89.5% at 140 ms after stimulation onset (Fig. 2b). This classifier was expected to perform considerably better than the classifier trained on peripheral stimuli.

Figure 2.

a, Peripheral training stimuli: classifier trained on 90% of peripheral fixation trials at each time-point individually and tested at each corresponding time-point with the remaining 10% of the trials. b, Central training stimuli: classifier trained on 90% of central fixation trials at each time-point individually and tested at each corresponding time-point with the remaining 10% of the trials. Note: the classification scales change from a to b. stim, Stimulation.

Presaccadic decoding using fixation trial training

We trained the classifier at each time-point of the peripheral fixation conditions and tested the classifier using the corresponding time-point of the saccade conditions for the first 300 ms after stimulation onset (before saccade). We found that the classifier was able to distinguish between face and house processing activity in the periphery before saccade in all the saccade conditions (Fig. 3). Note, three subjects were removed due to poor peripheral classification; therefore, successful classification of the peripheral trials was expected. The same condition peaked at 164 ms with a classification performance of 59.5%, the change condition peaked at 156 ms at 58.3%, and the disappear condition peaked at 183 ms at 57.6%. According to the 95% confidence intervals, we were unable to differentiate among the three conditions, as expected since before the saccade they are fully identical. These classification accuracies are quite similar to the training performance in the peripheral fixation condition.

Figure 3.

Classification of the presaccadic time-period of saccade trials. Classifier trained on peripheral fixation conditions at each time-point and tested on the corresponding time-point within the presaccadic time-period from stimulation onset to 300 ms. The solid horizontal line indicates chance level (50%), 95% confidence intervals, and the Bonferroni-corrected p values depicted. Bonf, Bonferroni; stim, stimulation.

Postsaccadic decoding using fixation trial training

The first classifier used to decode the postsaccadic time period was trained at the time-point of best performance (at 140 ms; Fig. 2) in the central fixation trials. The dotted vertical lines in Figure 4 show the 140 ms offset relative to the saccade landings. Figure 4 shows the performance of this classifier for each time-point of the after saccadic time-period, from saccade offset up to 300 ms following saccade offset. The classifier for face versus house becomes significantly different between the same and change conditions at 92 ms after saccade (t₍₁₃₎ = 2.628, p = 0.0485, Bonferroni-corrected paired t test). This is the earliest evidence for trans-saccadic transfer found using EEG. Classification between face and house stimuli then peaked in the same condition at 123 ms at 68.9% (Fig. 4a). In contrast, classification in the change condition peaked 28 ms after the same condition at 151 ms at 72.6% (Fig. 4b). The 28 ms time difference between the peak classification for the same and change conditions was consistent across subjects (t₍₁₃₎ = 6.8399, p < 0.0001, paired t test). There was no overlap between the 95% confidence intervals around the times of the peak classification for the same and the change conditions (95% CI same, 116.01–131.95; 95% CI change, 138.44–159.10), further demonstrating a lag for peak classification in the change condition compared with the same condition even when accounting for between-subject variability. The increased latency found for the change condition suggests that there is a relative processing advantage for the same condition trials, reflecting information transfer across the saccade (decoding after saccade in the same and change trials can only be distinguished by their presaccadic history). There was no difference in accuracy between the two conditions (t₍₁₃₎ = 1.6294, p = 0.1272, paired t test).

Figure 4.

Classification of postsaccadic time-period from saccade offset. The classifier was trained on the “fixation” condition trials with central stimuli. a, Classification performance between face and house for same saccade condition. b, Classification performance between face and house for the change condition. c, Classification performance between face and house for the disappear condition and “peripheral fixation” condition after stimulus offset (plus 51.5 ms to simulate saccade). The solid horizontal line indicates chance level (50%), and the vertical dotted line is the peak performance time-point of the fixation trials (140 ms), which is used to train the classifier. 95% confidence intervals and Bonferroni-corrected p values are depicted. Note that the classification performance scale is different for a/b and c. Bonf, Bonferroni; sacc, saccade; stim, stimulation.

The pattern for the disappear condition was less clear (Fig. 4c, blue), but classification did rise above chance for several time-points, first reaching significance at 162 ms with 54.4% performance. The classifier performance was based on labeling each trial according to the stimulus, face or house, that was presented before saccade. Above chance classification therefore indicates that significant information about the stimulus presented before saccade was still available after the saccade, even though no stimulus was on the screen. Classification for the disappear condition was consistently later (t₍₁₃₎ = 3.3676, p = 0.0061, paired t test) and weaker (t₍₁₃₎ = 8.2416,p < 0.00001, paired t test) than that for the same condition.

Importantly, we were able demonstrate that a saccade was necessary to retain decodable information in the visual system. We were unable to distinguish between face and house information in the EEG signal of the peripheral fixation condition when the stimulus was removed from the screen (plus 51.5 ms to simulate the saccade duration) and no saccade was performed (Fig. 4c, gray). This peripheral fixation condition is identical to the disappear condition except for the saccade and the slightly different memory task (one-back instead of same vs different). Nevertheless, the decoding performance dropped in the absence of the saccade.

For completion, we next trained the classifier successively on all time-points of the central fixation trials and tested at all time-points of that condition (central fixation) as well as during the postsaccadic time-period of the three saccade conditions. The diagonal in each panel of Figure 5 indicates when the time-point of the training trials matches the time-point of the test trials (e.g., train at 200 ms and test at 200 ms). Zero indicates the saccade offset in saccade conditions. This alignment is based on the assumption that saccade offset corresponds to stimulus onset in the fixation condition. Warmer colors off of the diagonal indicate that the training data from these time-points enables the classifier to generalize to other time-points within the test data. The horizontal line on each panel indicates the 140 ms peak performance chosen for the analysis presented in Figure 5. As expected, classification performance was strongest along the diagonal (with a peak at 140 ms) for the Monte-Carlo procedure of the central fixation trials. The leftward shift of the strongest performance observed in the same condition suggests that the stimulus in the postsaccadic period of the same condition was more rapidly processed (Fig. 5a,b). Similarly, the corresponding rightward shift of peak performance observed in the change trials indicates relatively delayed processing. Note that apparent latency differences relative to the “central fixation” condition may be contingent on our choice to use the saccade offset to mark the onset of postsaccadic processing. Postsaccadic processing may begin during saccade or even during saccade planning or during saccade, and this would effectively imply that processing latencies for both the same and change saccade conditions are underestimated in our analysis. Regardless, the relative rightward shift in the performance pattern between the same and change conditions (Figs. 4a,b, 5b,c, also visible as a 28 ms shift in peak classification latency when using only the 140 ms classifier) is independent of the choice of postsaccadic time reference, since this choice affects all saccade conditions equally. Finally, a rather different pattern was found in the disappear trials (Fig. 5d). In this case, the best classifier for each delay does not fall along the diagonal as it does approximately for the other two saccade conditions. Instead, there are patches of significant classification performance at later times, mostly resulting from classifiers trained between 120 and 170 ms. The trans-saccadic information thus seems to have a longer time course with a lower signal strength when there is no postsaccadic stimulus.

Figure 5.

a, Training the classifier on each time-point of 90% of central fixation trials and test on every time-point of the remaining 10% of central fixation trials. b–d, Training the classifier on each time-point of central fixation trials and test on every time-point after saccade offset of the following: same trials (b), change trials (c), and disappear trials (d). Note that the range of classification performance changes in each panel.

Using postsaccadic signals to train the classifier

We also performed an alternative analysis where we trained the classifier to discriminate same and change trials based on postsaccadic EEG signals instead of our fixation trial training of house versus face applied to same and change trials individually. This additional analysis answered a number of critical questions. (1) Why did we find evidence for trans-saccadic transfer at early time-points and others did not? (2) Could the change of the stimulus pattern (house to face or vice versa) have generated any EEG consequences, either from the change itself or from the effect it would have on corrective saccades? (3) Can the presaccadic stimulus be classified based on training with postsaccadic traces?

Question 1

Why do we find evidence for early transfer whereas others did not? Ours is not the first study to examine the transfer of information across saccades using neuroimaging (Dimigen et al., 2012; Kaunitz et al., 2014; Dunkley et al., 2016; Fairhall et al., 2017; Zimmermann et al., 2016; Niefind and Dimigen, 2016; Kornrumpf et al., 2017); however, this is the first neuroimaging study to find evidence for early information transfer that correlates with findings in psychophysical experiments (Wittenberg et al., 2008; Fracasso et al., 2010; Vetter et al., 2012). Our classifier, built on steady fixation stimuli, gave the advantage of pinpointing early postsaccadic information that may have been hidden under saccade-related noise in previous neuroimaging studies. To demonstrate the improvement offered by our training on fixation trials, we replicated the leave-10%-out analysis performed by Kaunitz et al. (2014). To do this, we use only postsaccadic signals to train the classifier to distinguish between same and change trials after saccade offset. Here, we find evidence for only a late information transfer across the saccade, as follows: significant discrimination between same and change trials occurred at 248 ms rather than at 92 ms, as in our original analysis using the fixation trial classifier to analyze these same postsaccadic EEG signals (Fig. 6a). We assume that the extra saccade-related information incorporated into this alternative classifier masks the detection of the emerging difference between same and change. With regard to the fMRI studies on trans-saccadic integration (Dunkley et al., 2016; Fairhall et al., 2017; Zimmermann et al., 2016), we note that fMRI does not have the temporal sampling advantage of EEG, which may have resulted in the inability to locate the rapid trans-saccadic signal found in the present study.

Figure 6.

Using postsaccadic signals to train the classifier. a, Classification between same and change in postsaccadic time-period from the time-point corresponding to the mid-saccade transient in the change condition. The training classifier on each time-point of 90% of same and change trials and the test classifier on the corresponding time-point of the remaining 10% of same and change trials. b, Classify presaccadic EEG signal using postsaccadic traces. Classifier trained on peak decoding time-point in the postsaccadic time-period for each condition separately and then tested using the presaccadic time-period of the corresponding condition. Solid horizontal line indicates chance level (50%). 95% confidence intervals and Bonferroni-corrected p values are depicted. Bonf, Bonferroni.

Question 2

Could the differences in the stimulus sequence between same and change generate the delayed classification in the change trials? In the change trials, the initial stimulus is switched with the alternative stimulus during the saccade. This could have two consequences that may delay the classification on change trials relative to same trials. First, the stimulus exchange on change trials may trigger a transient signal that could mask the processing of the postsaccadic stimulus; and second, the change of stimulus might drive different pattern of corrective saccades once the saccade lands. We again used the results of the leave-10%-out analysis of purely postsaccadic EEG signals (replication of the study by Kaunitz et al., 2014) to rule out an influence from either of these two factors. This analysis showed that there is no extra transient-related signal in the change condition compared with the same condition because the presence of this signal would have supported classification at the delays where transients are typically picked up in EEG signals. The EEG response to a transient would become evident at ∼100 ms in the n1–p1 complex (Näätänen and Picton, 1987) in the change trials, and if present, this would have contributed to a significant classification accuracy at that time delay. However, the same and change conditions do not differ significantly until 248 ms after the stimulus change (Fig. 6a; Bonferroni corrected). This indicates that the significant difference between the same and change conditions found in our original analysis reflects the postsaccadic processing of an expected target versus an unexpected target and is not a nonspecific disruption or extra noise effect from the change transient. The same logic also rules out any impact of differences in corrective saccades on the EEG signals in the time frame where the house and face are classified. It is also a clear demonstration of how our classifier trained on face-versus-house at fixation and applied separately to same and change trials is able to robustly analyze postsaccadic processing.

Question 3

Can the presaccadic stimulus be classified based on training with postsaccadic traces? As a further assurance that the information in the postsaccadic EEG signals corresponds to the presaccadic stimulus, we trained the classifier at the peak decoding time-point in the postsaccadic time period for each condition separately (same, 123 ms; change, 151 ms; disappear, 162 ms) and then tested the classifier at each time-point in the presaccadic period from stimuli onset to 300 ms in the corresponding condition. We found results similar to those we reported when using the classifier trained on fixation trials, as follows: all conditions classify significantly above chance with same peaking at 164 ms (58.25%), change peaking at 156 ms (59.67%), and disappear peaking at 187 ms (54.86%; Fig. 6b). The peak for the classifier trained on the disappear condition was delayed and weaker. This is not unexpected as the disappear classifier is trained on a time-period when no sensory stimulation is present, whereas the same and change classifiers are trained on an EEG signal during stimulus processing.

Discussion

We found that presaccadic stimulation affects early postsaccadic processing, indicating that information transfers across saccade and interacts with initial stimulus processing at the new fixation. We provide the following two examples of information transfer across saccade: (1) the identity of the presaccadic stimulus modulated the latency at which we could decode the identity of the postsaccadic stimulus; and (2) a postsaccadic neural marker was found for a saccade target stimulus that had been removed during saccade.

The decreased postsaccadic processing latency of the same condition compared with the change condition demonstrates that postsaccadic processing does not start anew from saccade offset; some internal information is retained and influences the processing speed of the postsaccadic stimulus. Furthermore, we were able to find evidence for postsaccadic processing of a face or house when the stimulus was no longer present. The only explanation is that information pertaining to the presaccadic stimulus remains available within the visual system; this could reflect a potential mechanism intended to aid with postsaccadic processing (since in ecological conditions, the presaccadic target tends to remain present throughout and after the saccade).

We were able to find this evidence of early transfer of information across the saccade because of changes to the classification procedure we used compared with that used in earlier studies (Kaunitz et al., 2014). Specifically, the classification was performed separately on the same and change trials using nonsaccade fixation trials for training. This ensured that saccade-related signals that differed in change and same trials could not directly drive any classification performance—same and change trials were never compared for classification. The earlier demonstration of trans-saccadic transfer had used a classifier trained on postsaccadic traces and the extra saccade-related signals may have masked the evidence for early transfer.

The processing advantage of the same stimulus in our primary finding is consistent with the preview facilitation (Boucart et al., 2016) and trans-saccadic memory research (Higgins and Rayner, 2015). Despite the low spatial resolution of information in the periphery, saccade target preview benefits object identification (Henderson and Anes, 1994; Schotter et al., 2013) and face identification (Crouzet et al., 2010; Boucart et al., 2016), and increases reading speed (Rayner et al., 2011). Within the preview facilitation research, trans-saccadic information is related to visual short-term memory (Higgins and Rayner, 2015). Visual short-term memory can last for a few seconds and therefore may also contribute to information transfer across the saccade (Hollingworth et al., 2008).

However, it is also possible that the mid-saccade stimulus change resulted in a cost to the visual system that could explain the processing latency effect between the same and the change condition. We tested and rejected this alternative by showing that there was no discrimination between same and change trials until 248 ms when using a classifier trained on the postsaccadic EEG signals. This argues against any signals in the earlier postsaccadic trace that could interfere with the house versus face classification differently in same and change trials. If they could, they would also have supported a same versus change discrimination at the earlier time period.

Memory transfer across saccades may be aided by predictive remapping of attended objects (Melcher and Colby, 2008; Melcher, 2009; Cavanagh et al., 2010; Howe et al., 2011; Rolfs et al., 2011; Jonikaitis et al., 2013; Zimmermann et al., 2013, Harrison and Bex, 2014; Ganmor et al., 2015; Wolf and Schütz, 2015; Wolfe and Whitney, 2015; Paeye et al., 2017). With each eye movement, attention remains on objects of interest within the visual field. To retain attention at the appropriate spatial location after the saccade, receptive fields at the remapped location are activated in preparation for the arrival of the expected attended stimulus (Gottlieb et al., 1998; Melcher and Colby, 2008). Schneider and Deubel (1995) found that visual discrimination was most accurate when the discrimination stimulus was also the saccade target, demonstrating attention allocation to new saccade targets. Melcher (2009) further demonstrated active remapping of attended objects with evidence that the processing of a presaccadic grating influenced the perception of a postsaccadic grating presented at the same spatiotopic position. Within our study, the receptive fields processing the fixation marker before saccade should be activated to receive the peripheral face or house stimulus after saccade. The increased processing latency found between same and change conditions indicates an expectation of the original stimulus after the saccade. The remnant postsaccadic information in the disappear condition also supports this notion.

Evidence of receptive field remapping has been found within saccade centers, such as the frontal eye fields, intraparietal sulcus, and superior colliculus (Goldberg and Bruce, 1990; Duhamel et al., 1992; Umeno and Goldberg, 1997). fMRI studies have also demonstrated that the intraparietal sulcus and the frontal eye fields have retinotopic representations of visual attention and saccade targets (Sereno et al., 2001; Hagler et al., 2007; Kastner et al., 2007).

Predictive coding may also contribute to information conservation and transfer across saccades (Vetter et al., 2012). Predictive coding models propose that our perception is built from feedforward sensory information and cortical predictions fed back from higher cortical areas (Rao and Ballard, 1999). Cortical predictions are internal models of our expected stimulation from our environment. Recent studies have demonstrated that predictive codes can transfer across saccade and affect postsaccadic processing, evidenced both behaviorally (Vetter et al., 2012) and in neuroimaging findings (Fairhall et al., 2017; Edwards et al., 2017). Vetter et al. (2012) found detection benefits for postsaccadic targets that were predictable by their relationship with the presaccadic stimulus. In neuroimaging, predictable postsaccadic stimulation resulted in a decrease in BOLD activity in the early visual cortex (Fairhall et al., 2017; Edwards et al., 2017), commonly accepted as a marker for predictive processes (Alink et al., 2010; Kok et al., 2012).

Interestingly, evidence for information transfer is present in the latency of pattern classification, rather than in the classification performance, as seen in previous studies (Kok et al., 2012). Decreased stimulus-processing latency using internal predictive codes seems logical, yet others have been unable to relate processing latency and stimulus predictability (Todorovic et al., 2011). We did not find a significant difference in reaction times between our conditions, but this is principally due to our use of a response window 300 ms after stimulus offset.

Notably, we may have found a possible neural correlate of the illusory ghost phenomenon reported by Wolf et al. (1980). Wolf et al. (1980) found that a target removed during saccade was still perceived on saccade landing. Jonides et al. (1982) replicated this illusion but later found that the phosphor persistence of CRT monitors could explain the percept (Jonides et al., 1983). Phosphor persistence was measured at 11 ms in our study, 40.5 ms shorter than the average saccade duration for our participants. Yet, we were still able to decode between the perceptions of face and house after saccade on the disappear condition. However, the difference in the postsaccadic decoding ability of face and house percepts between same and disappear conditions is significant (both in terms of accuracy and latency). The late classification may indicate that the illusory percept has a long time course with lower signal strength. It is also possible that we are not exploiting the optimal processing strategy for illusory percepts: the classifier is trained with feedforward sensory stimulation, yet illusory percepts may be created from internal templates of sensory information that do not result in the same patterns of activity. For example, imagining a sound results in a correlated but significantly reduced classification performance compared with listening to that sound (Vetter et al., 2014).

Information about the peripheral presaccadic stimulus is transferred across the saccade so that it becomes available and influences processing at a new retinal position (the fovea) when the saccade has landed. Presaccadic information was found to interact with postsaccadic stimulus processing and to remain available when no stimulus was present after saccade.