Evidence for Optimal Integration of Visual Feature Representations across Saccades

Abstract

We explore the visual world through saccadic eye movements, but saccades also present a challenge to visual processing by shifting externally stable objects from one retinal location to another. The brain could solve this problem in two ways: by overwriting preceding input and starting afresh with each fixation or by maintaining a representation of presaccadic visual features in working memory and updating it with new information from the remapped location. Crucially, when multiple objects are present in a scene the planning of eye movements profoundly affects the precision of their working memory representations, transferring limited memory resources from fixation toward the saccade target. Here we show that when humans make saccades, it results in an update of not just the precision of representations but also their contents. When multiple item colors are shifted imperceptibly during a saccade the perceived colors are found to fall between presaccadic and postsaccadic values, with the weight given to each input varying continuously with item location, and fixed relative to saccade parameters. Increasing sensory uncertainty, by adding color noise, biases updating toward the more reliable input, which is consistent with an optimal integration of presaccadic working memory with a postsaccadic updating signal. We recover this update signal and show it to be tightly focused on the vicinity of the saccade target. These results reveal how the nervous system accumulates detailed visual information from multiple views of the same object or scene.

SIGNIFICANCE STATEMENT This study examines the consequences of saccadic eye movements for the internal representation of visual objects. A saccade shifts the image of a stable visual object from one part of the retina to another. We show that visual representations are built up over these different views of the same object, by combining information obtained before and after each saccade. The weights given to presaccadic and postsaccadic information are determined by the relative reliability of each input. This provides evidence that the visual system combines inputs over time in a statistically optimal way.

Introduction

Each saccade radically alters the pattern of input to the retina. Details of the presaccadic input are maintained within the limited store of visual working memory (Irwin, 1991, 1992; Rensink, 2000; Bays and Husain, 2008; Hollingworth et al., 2008; Ma et al., 2014), permitting an imperfect comparison that may fail to detect even large-scale changes to the environment (Rensink et al., 1997; Henderson and Hollingworth, 2003). But what is the fate of previous object representations when no change is detected?

On the one hand, the preceding information could simply be discarded, replaced by input from the new fixation. Conventional support for this possibility comes from observers' inability to combine pieces of information displayed separately, before and after a saccade, into a single percept (Irwin et al., 1983; O'Regan and Lévy-Schoen, 1983; Rayner and Pollatsek, 1983; Irwin, 1996); this has commonly been taken as evidence against information accumulation across fixations. Furthermore, integration of features represented in the activity of retinotopic visual neurons requires a computationally challenging remapping to counteract the shift in object coordinates (Duhamel et al., 1992). The neurophysiological basis for such remapping, based on shifting receptive fields, is currently in question (Zirnsak and Moore, 2014; Zirnsak et al., 2014).

On the other hand, several recent studies have reported situations in which presaccadic input can impinge on postsaccadic awareness through remapping of attentional “pointers” to maintain visual stability (Mathôt and Theeuwes, 2011), via transsaccadic memory of shape information (Hayhoe et al., 1991; Demeyer et al., 2009, 2010, 2011; Fracasso et al., 2010), by enhanced feature individuation (Harrison and Bex, 2014), or as intrusions of irrelevant presaccadic signals into perception (Wittenberg et al., 2008; Golomb et al., 2014). Nonetheless, it remains unclear how presaccadic visual features are integrated with postsaccadic input when both inputs refer to the same object in the external world.

Critically for this investigation, eye movements influence information storage in working memory (Irwin, 1992, 1996). In particular, visual features of an intended saccade target are stored with enhanced precision compared with other objects (Bays and Husain, 2008; Shao et al., 2010), indicating a change in the allocation of memory resources as attention moves to the upcoming gaze location (Kowler et al., 1995; Deubel and Schneider, 1996; Zhao et al., 2012). Enhancing storage resolution of one stimulus typically comes with a cost to other stimuli, which are remembered more poorly (Bays et al., 2011; Gorgoraptis et al., 2011; Melcher and Piazza, 2011). In this study we investigated how shifting allocation of resources affects the updating of internal representations of multiple visual objects by presenting stimuli at different locations relative to presaccadic and postsaccadic gaze.

We directly examine the contribution of working memory to the perception of stable object features by making small changes to object colors during a saccade. Rather than being overwritten by new input, we find that the presaccadic color strongly contributes to postsaccadic perception. Our results are consistent with an optimal updating of presaccadic representations with postsaccadic visual input, which takes into account the reliability of each source of information.

Materials and Methods

Participants and apparatus.

In total, 59 participants (21 males, 38 females) aged 18–52 years (M = 24.7 years) who were naive to the research question took part in the study after giving informed consent. This study was approved by the University College London Research Ethics Committee. All participants had normal color vision and normal or corrected-to-normal visual acuity. Experiments 1A, 2, and 3 each had 16 participants, while 11 others participated first in Experiment 1B and then in Experiment 1C. Participants were seated with their head in a forehead and chin rest such that their eyes were 60 cm in front of a 21” CRT monitor (refresh rate 140 Hz). Eye position was tracked at 1000 Hz with a desk-mounted EyeLink 1000 (SR Research).

General methods.

Each trial started with a gray background and a white fixation cross, 10° of visual angle to the left or right (randomly selected) of the screen center (Fig. 1a). A presaccadic display was presented when stable fixation was determined. The presaccadic display consisted of three colored disks (radius 1°) displayed 4° above the fixation cross and separated horizontally by 10° (Fig. 1a). The colors of the disks were randomly drawn from a color wheel, with a minimum separation of 30°. Colors on the wheel were selected from a circle in CIE L*a*b* space (L* = 50, a* = b* = 20, radius = 60); the CIE color values were sampled nonuniformly to maximize perceptual uniformity of the wheel (this optimization was performed before the study, based on reports of two observers not part of the main study).

Figure 1.

Probing transsaccadic working memory. a, Experimental configuration of Experiment 1A. While fixating (dotted red circle) a cross, participants were presented with a presaccadic display consisting of three colored disks. When the fixation cross jumped, they were required to saccade to its new position on the opposite side of the display. On PRE+POST trials, the colors were shifted a small distance in color space during this saccade; participants were unaware of these changes. This postsaccadic display was followed by a blank display, then one item location was indicated, and participants reported the color they remembered at that location. Note that there was no postsaccadic display in the PRE condition; instead, the blank interval started as soon as the saccade was detected. b, The color space. Arrows correspond to the CCW color shift illustrated in a. c, Frequency distributions of the discrepancy between the reported and presaccadic colors, for conditions with PRE+POST saccadic displays. Different plots show the three different item positions. Zero corresponds to the presaccadic color and dashed lines correspond to postsaccadic colors. Solid lines indicate the medians of the two response distributions (orange = CW shift; red = CCW shift). Increasing separation of the two distributions, left to right, indicates increasing bias toward the postsaccadic color. d, Response bias as a function of item position for the different postsaccadic display durations (PRE+POST condition with color change). Dashed lines correspond to presaccadic and postsaccadic colors. e, Response variability for conditions with PRE+POST displays, when item colors shifted during the saccade. f, Response variability as a function of item position for conditions with PRE+POST displays presenting the same color (red) or with presaccadic display only, when the observer maintained fixation at fix1 (light blue) or made an eye movement from fix1 to fix2 (dark blue). Error bars indicate ± 1 SE.

After 500 ms, the fixation cross jumped to the opposite side of the screen. Participants were instructed to move their gaze to the new position as soon as possible after the jump. In conditions with no postsaccadic display (PRE-ONLY condition), the colored disks were removed from the display as soon as a saccade was detected. In conditions with a postsaccadic display (PRE+POST condition), the colors of all items were shifted on saccade detection by 20° on the color wheel [clockwise (CW) or counterclockwise (CCW), randomly selected on each trial] and displayed for 500 ms (except for Experiment 1A, see below).

Next, a blank display containing only the fixation cross was presented for 1000 ms. Then the previous location of one of the three items, selected at random, was indicated by a white square, and participants were asked to report the color they remembered at that location by selecting it on a color wheel with a computer mouse.

If a participant's gaze deviated >5° from the fixation cross at any point during stimulus presentation, or if they failed to follow the fixation cross after the jump within 500 ms, the trial was aborted and restarted with new parameters. The experiment was followed by a structured debriefing in which we determined whether participants had perceived any color changes. Because the changes happened during saccades, most participants did not perceive the color changes; participants who reported seeing a color change at any point in the experiment were excluded from the analysis (N = 9).

Experiment 1.

In Experiment 1A, the three colored disks were positioned at −10°, 0°, and 10° relative to the display center, i.e., one aligned with the presaccadic fixation, one aligned with the postsaccadic fixation, and one intermediate between them. Each participant completed 280 trials in total: 70 with no postsaccadic display (PRE-ONLY condition) and 70 each with postsaccadic displays of 250, 500, or 750 ms duration (PRE+POST condition). These conditions were randomly interleaved.

Experiment 1B controlled for the effect of item location on color perception without any eye movements. It was identical to the PRE-ONLY condition of Experiment 1A except that the fixation cross did not jump to the opposite side of the screen and participants did not make a saccade. Each participant completed 180 trials.

Experiment 1C controlled for simple effects of eye movement on color perception. It was the same as Experiment 1A except that item colors did not change between presaccadic and postsaccadic displays, which made the displays identical. Each participant completed 180 trials.

Experiment 2.

To examine the effect of item location on saccadic integration, the position of the items relative to fixation was varied in Experiment 2. There were three different item configurations (Fig. 2a). In config1 the middle item was aligned with the presaccadic fixation cross while in config3 this item was aligned with the postsaccadic fixation cross; config2 was identical to the standard configuration. The fixation crosses were positioned at −8 and 8° relative to the display center and the interitem distance was 8° in all configurations. Items stayed at the same location throughout a trial. Participants performed 360 trials in total, distributed equally between the different configurations.

Figure 2.

Manipulating stimulus configuration. a, Three different item configurations. Responses were analyzed according to the relative location of the target item within the three items (rel1, rel2, and rel3), and according to the absolute location of the target item with respect to presaccadic fixation (ext1, fix1, int, fix2, ext2). b, Response bias as a function of relative item location for three different configurations. Dashed lines correspond to presaccadic and postsaccadic colors. c, Response bias plotted as a function of absolute item location. d, Response variability as a function of absolute item location for the PRE-ONLY (blue) and PRE+POST display condition (red). Error bars indicate ±1 SE.

Experiment 3.

To examine the effect of reliability of color perception on saccadic integration, we added color noise to the display in Experiment 3 (Fig. 3a). In presaccadic displays either 5% (low noise) or 55% (high noise) of pixels were replaced with colors chosen at random from the color wheel. The color noise in the postsaccadic display was fixed at 30%. Participants performed a session of 360 trials with each of the two presaccadic noise levels, in a counterbalanced order.

Figure 3.

Altering input reliability. a, Examples of presaccadic and postsaccadic displays in the low- and high-noise conditions of Experiment 3. b, Response variability as a function of item position for the PRE-ONLY condition with low (dark blue) or high noise (light blue). c, Response bias as a function of item position for the PRE+POST condition, when the presaccadic display contained low (red) or high noise (orange). Error bars indicate ±1 SE.

Analysis.

The error on each trial was defined as the angular deviation between the color reported by the participant and the color of the target item in the presaccadic display. For the PRE+POST conditions, errors on trials where the colors shifted CCW were first inverted, so that positive errors were always in the direction of the color change. Bias of responses toward the postsaccadic color was then assessed by the median error. Response variability was measured by the SD of errors. Participants made occasional very large (>90°) errors: these could result from lapses of attention or confusion over which item in memory was indicated by the probe (Bays et al., 2009). To exclude these influences, errors >90° were rejected from analysis (<11% of total trials).

Errors for different items were analyzed according to the item's position with respect to fixation: fix1 indicates the item aligned with the presaccadic fixation, fix2 the item aligned with the postsaccadic fixation, and int the item intermediate between the two. In Experiment 2, there were two more possible locations, ext1 and ext2, and we also performed a separate analysis based on the item's relative position within the group (rel1, rel2, or rel3). See Figure 2a for an illustration of these different configurations and labels. Differences in bias and variability across items and conditions were examined with ANOVA and post hoc t tests. If Mauchly's test of sphericity was significant we adapted a Greenhouse–Geisser correction.

Estimating postsaccadic variability.

Assuming optimal integration (Ernst and Banks, 2002), the variability of representation of the postsaccadic update can be inferred from the observed bias in responses (measured in the PRE+POST condition) and the observed variability of the presaccadic display (measured in the PRE-ONLY condition). We estimated this postsaccadic variability using Bayesian inference (Gelman and Shalizi, 2013).

We considered the observed participant biases W, scaled to range 0 (PRE color) to 1 (POST color), and presaccadic SDs S₀ as noisy samples (with normal and log-normal distribution, respectively) of population values w and s₀. The posterior distributions of w and s₀, based on noninformative priors, are as follows:

Under optimal integration, w and s₀ are related to the postsaccadic SD s₁ by the following:

This weighting defines the linear combination of presaccadic and postsaccadic input that minimizes variance of the resulting estimate (Ghahramani et al., 1997). Combining Equations 1 and 2 we can calculate the posterior distribution of s₁. Our estimate of postsaccadic variability is obtained from the maximum of this distribution (i.e., the MAP estimate).

Influence of retinotopic target.

The item at position fix2 is at the same retinal location after the saccade as the item at fix1 before the saccade. To examine whether responses were biased by a previous color in the same retinotopic location, on trials where fix2 was probed we calculated the mean squared deviation (MSD) between the participant's response and the color of the item at fix1. To obtain the MSD that would indicate no effect of fix1 color, we performed a Monte Carlo simulation: we generated 1000 new fix1 color values per fix2 trial at random, respecting the minimum separations between colors that applied in the real experiment, and calculated the MSD in the same way as above. For comparison, we also calculated the MSD between the response on fix2 trials and the presaccadic color of the (real) fix2 item (i.e., the target item). T tests comparing retinotopic MSD with spatiotopic and simulation MSD tested if these MSDs were different.

Results

To examine influences of presaccadic input on postsaccadic perception, we presented participants with color stimuli that subtly changed their hue during an eye movement (Fig. 1a,b; Experiment 1A). Although unaware of this manipulation (see Materials and Methods), when asked to report the color they had observed at a specified location, observers' responses were centered on values that fell between the presaccadic and postsaccadic colors (Fig. 1c). The weight, or bias, toward one or the other color depended strongly on item location (F_(2,26) = 34.4, p < 0.001). Figure 1, c and d, illustrates that responses for fix1 (the item closest to the presaccadic fixation) were biased toward the presaccadic color, responses for fix2 (closest to postsaccadic fixation) were biased toward the postsaccadic color, and responses for int (intermediate location) showed an intermediate bias (fix1 vs int, t₍₄₁₎ = 5.92, p < 0.001; fix1 vs fix2, t₍₄₁₎ = 11.8, p < 0.001; int vs fix2, t₍₄₁₎ = 3.14, p = 0.003). There was no main effect of postdisplay duration (250–750 ms) on these biases (F_(2,26) = 0.97, p = 0.39) or on response precision (Fig. 1e; F_(2,26) = 0.35, p = 0.71), which was highest for the item closest to the postsaccadic fixation (fix1 vs int, t₍₄₁₎ = 0.08, p = 0.94; fix1 vs fix2, t₍₄₁₎ = 5.49, p < 0.001; int vs fix2, t₍₄₁₎ = 4.95, p < 0.001).

To explore the possible role of noise in generating these biases, we examined how the variability of color representations for the different items changed before, during, and after saccade generation. In a condition in which no saccade was made, and participants simply maintained fixation at fix1, a recall test showed a significant precision advantage for the item closest to fixation (Fig. 1f, light blue symbols; Experiment 1B, fix1 vs int, t₍₁₀₎ = 2.47, p = 0.033; fix1 vs fix2, t₍₁₀₎ = 2.84, p = 0.017; int vs fix2, t₍₁₀₎ = 1.42, p = 0.19). In contrast, when a saccade was made to location fix2, but the colored items were removed during the saccade, we observed a shift in representational precision toward the saccade target region (Fig. 1f, dark blue symbols; Experiment 1A, PRE-ONLY condition), such that responses were less variable for the item closest to the saccade target than for the intermediate item, even though none of the items were visible at the termination of the saccade (fix1 vs int, t₍₁₃₎ = 3.76, p = 0.002; fix1 vs fix2, t₍₁₃₎ = 0.82, p = 0.43, int vs fix2, t₍₁₃₎ = 2.79, p = 0.015). This is consistent with previous studies that have observed a reallocation of memory resources toward the saccade target immediately before a saccade (Bays and Husain, 2008; Shao et al., 2010).

When the same colors were presented both before and after the saccade (Fig. 1f, red symbols; Experiment 1C), we observed a further shift of representational precision toward the postsaccadic fixation. The results now mirrored performance in the no-saccade condition (light blue symbols), with variability determined by eccentricity from the new postsaccadic fixation location (fix1 vs int, t₍₁₀₎ = 0.09, p = 0.93; fix1 vs fix2, t₍₁₀₎ = 2.20, p = 0.052, int vs fix2, t₍₁₀₎ = 1.92, p = 0.084). We found no significant difference between this condition with unchanging colors and the previous condition with a color change during the saccade (Experiment 1A vs Experiment 1C; all items: t_(15.6) < 0.81, p > 0.43), indicating that the color-change manipulation affected only the content and not the precision of item representations.

Relative or absolute stimulus position

The results of the first experiment indicate that the location of an item is a critical factor in determining both the bias toward presaccadic or postsaccadic input and the variability in internal representation after a saccade. These effects could reflect the relative position of each item in the stimulus set (e.g., because of configurational effects on the distribution of attention, or because the items vary in the degree of perceptual crowding induced by their neighbors; Whitney and Levi, 2011) or they could be a fixed outcome of the position of items with respect to the start and end points of the saccade.

To investigate this, we varied the absolute position of the items in space, while keeping their relative positions unchanged (Fig. 2a; Experiment 2). Figure 2, b and c, plot bias as a function of relative position in the stimulus set, and absolute position in space, respectively. We observed strong effects of stimulus configuration when comparing across items at the same relative coordinates (item × configuration ANOVA: item, F_(2,20) = 8.27, p = 0.002; configuration, F_(2,20) = 8.51, p = 0.002; item × configuration, F_(4,40) = 3.58, p = 0.014). However, testing between the different configurations for items at the same absolute position in space, we found only one significant effect (for fix2, t₍₁₀₎ = 2.77; p = 0.020; fix1, t₍₁₀₎ = 0.40; p = 0.70; int, F_(1,10) = 0.002; p = 0.97), and this effect was in the opposite direction to that expected for an effect of relative position (which would be a bias closer to the presaccadic color for fix2-config3 than for fix2-config2). These results suggest that the absolute location of items with respect to the saccade primarily determines how presaccadic and postsaccadic information is integrated.

Similar results were obtained for response variability (effect of configuration in relative coordinates: item, F_(2,20) = 5.78, p = 0.011; configuration, F_(2,20) = 2.12, p = 0.15; item × configuration, F_(4,40) = 3.20, p = 0.023; in absolute coordinates, t₍₁₀₎ < 1.5, p > 0.17). Figure 2d plots mean variability for conditions with (red symbols) and without (blue symbols) a postsaccadic display. The effects of position and of postsaccadic input on precision were consistent with those observed in Experiment 1. Additionally, the shift of memory resources in the direction of the saccade target, observed as a precision advantage for fix2 over int in the condition with no postsaccadic display, is seen to be tightly focused on the saccadic target, as it does not benefit the item ext2 lying further in the direction of the saccade (ext2-config3 vs fix2-config3, t₍₁₀₎ = 4.15, p < 0.001; ext2-config3 vs fix2-config2, t₍₁₀₎ = 4.67, p < 0.001).

Reliability of presaccadic information

The results of Experiments 1 and 2 show that biases in perception toward presaccadic or postsaccadic input vary considerably for stimuli at different positions relative to a saccade, and that this coincides with significant changes in working memory variability. In Experiment 3 we tested whether the bias could be altered experimentally by manipulating uncertainty in the sensory input. We introduced color noise into both presaccadic and postsaccadic displays (Fig. 3a). In the low noise condition, the presaccadic display had less noise than the postsaccadic display, whereas in the high noise condition it had more.

The effect of noise condition on presaccadic representations was tested in trials with no postsaccadic display (Experiment 3, PRE-ONLY condition). Consistent with results above, variability in the reported color was higher for the intermediate item than those at presaccadic or postsaccadic fixation (Fig. 3b; fix1 vs int, t₍₂₇₎ = 5.79, p < 0.001; fix1 vs fix2, t₍₂₇₎ = 0.37, p = 0.72, int vs fix2, t₍₂₇₎ = 5.09, p < 0.001). As expected, variability in reported color was higher for the high noise condition than for the low noise condition (t₍₄₁₎ = 5.09, p < 0.001). The effect of noise on recall precision was also found to vary with item location. This is likely to reflect unequal allocation of attention, as attention enhances the ability to distinguish a visual signal from noise (Dosher and Lu 2000; Lu and Dosher, 2004; Pratte et al., 2013). Quantitatively, the largest effect of noise condition on SD was observed for the item at the intermediate position (mean difference = 0.11, t₍₁₃₎ = 2.77, p = 0.016), with a smaller effect at fix1 (mean difference = 0.10, t₍₁₃₎ = 5.15, p < 0.001) and no significant effect at fix2 (mean difference = 0.05, t₍₁₃₎ = 2.04, p = 0.062).

Figure 3c plots bias for low and high noise conditions, as assessed on trials with both presaccadic and postsaccadic displays (Experiment 3, PRE+POST condition). For the intermediate item, responses were biased significantly more toward the postsaccadic color if the presaccadic display had high noise than if it had low noise (int, t₍₁₃₎ = 2.18; p = 0.048; fix1, t₍₁₃₎ = 1.34, p = 0.20; fix2, t₍₁₃₎ = 0.01, p = 0.99). This indicates that, for the item most affected by the addition of visual noise, the reliability of the presaccadic information influenced the integration of presaccadic and postsaccadic input.

Postsaccadic variability

The results of Experiment 3 show that transsaccadic updating is influenced by noise in sensory input, with the perceived color biased toward the less noisy of the presaccadic and postsaccadic inputs. This behavior is characteristic of an optimal integration mechanism that seeks to minimize error in the integrated estimate (Ernst and Banks, 2002). In this case, we propose that the integration is between an internal representation of the presaccadic colors, maintained in working memory, and an update signal arising from postsaccadic sensory input. We estimated the variability of this postsaccadic signal, assuming that the reported color was the result of an optimal linear combination of the presaccadic and postsaccadic colors determined by the variability of their individual representations.

Figure 4a shows the postsaccadic variability estimated from data in Experiments 1A, 2, and 3 [the different noise conditions for Experiment 3 did not result in significantly different postsaccadic estimates (fix1: t₍₁₃₎ = 1.18, p = 0.26; int: t₍₁₃₎ = 1.17, p = 0.26; fix2: t₍₁₃₎ = 0.57, p = 0.58) and so were pooled for this plot]. All three experiments resulted in very similar estimates of the postsaccadic representation variability (all ts < 0.31, p > 0.76). Compared with presaccadic variability (bounded by shaded area), the inferred precision of the postsaccadic signal varied over a wide range, with a very high precision representation of the item closest to postsaccadic fixation and less precise representations of items at greater eccentricities.

Figure 4.

Postsaccadic variability and updating. a, Estimated postsaccadic response variability as a function of absolute item position for the three experiments with a PRE+POST condition with color change. The shaded area indicates the range of presaccadic response variability. b, Illustration of spatiotopic and retinotopically matched colors. The spatiotopic match to the postsaccadic item fix2 is the presaccadic color of fix2; the retinotopic match to fix2 is the presaccadic color of fix1. c, MSD between the reported color and the retinotopic match, spatiotopic match, or a randomly chosen color for each experiment. Note that reported colors were no closer to retinotopically matched colors than to randomly chosen colors, indicating no influence of the retinotopic item on report. Error bars indicate ±1 SE.

Retinotopic versus spatiotopic updating

So far, we have examined perceived color with respect to the presaccadic and postsaccadic colors displayed at the same location in external space based on the assumption that visual updating should operate in spatiotopic (i.e., world-centered) coordinates (Burr and Morrone, 2011). However, other studies have observed an influence on perception of visual features presented at the same retinotopic (eye-centered) position as a target or cue (Mathôt and Theeuwes, 2013; Golomb et al., 2014). In Experiments 1A and 3 this would be observed as an influence of the presaccadic color at fix1 on report of the item at fix2 (Fig. 4b; in Experiment 2, the retinotopic match is between presaccadic rel1 and postsaccadic rel3).

For all three experiments, the mean squared error (MSE) between the reported color and the presaccadic color at the same retinotopic location was substantially larger than for the presaccadic color at the same spatiotopic location (Fig. 4c; t > 35.0, p < 0.001). Furthermore, there was no significant difference between MSE for color at the matched retinotopic location and MSE for randomly drawn color values (t < 1.06, p > 0.31; see Materials and Methods). Therefore, we find that postsaccadic updating operates on visual features matched in external space, with no evidence for an influence of the retinotopic color on responses.

Discussion

In this study we investigated how the internal representation of a visual scene develops across eye movements. By subtly changing the color of visual objects during a saccade we could assess how the postsaccadic representation was influenced by the different color hues presented before and after the saccade. We found that following a saccade, the presaccadic content was not overwritten; instead the postsaccadic perception reflected a combination of presaccadic and postsaccadic input. How the two sources of information were weighted in the integrated representation depended on an object's location in space (Fig. 1). Specifically, the weighting was determined by the location of the object along the saccade axis, relative to the start and end points of the saccade, and not the object's location within the configuration of surrounding items (Fig. 2).

Externally manipulating the reliability of the presaccadic input influenced the integration. If the presaccadic input was noisier than the postsaccadic input, the representation was biased more toward the postsaccadic input then if the presaccadic input was less noisy (Fig. 3). We interpret this as optimal integration of a stored presaccadic representation in memory with a postsaccadic update signal, i.e., the percept is a weighted combination of the two sources of information, where the weights are determined by the relative reliability, or precision, of the sources (Ghahramani et al., 1997; van Beers et al., 1999; Ernst and Banks, 2002; Ernst and Bülthoff, 2004; see also Niemeier et al., 2003; Niemeier et al., 2007 for evidence of optimal integration of presaccadic and postsaccadic spatial information).

Consistent with previous studies, working memory precision was influenced by an upcoming saccade such that, at the time of the saccade, the item closest to the saccade target was represented with increased reliability (Bays and Husain, 2008; Shao et al., 2010). This finding is consistent with a shared resources model of working memory, whereby limited memory resources are distributed over all presented items, and the allocation of resources depends on the salience or goal relevance of the items (Bays and Husain, 2008; Bays et al., 2011; Gorgoraptis et al., 2011; Melcher and Piazza, 2011; Ma et al., 2014). Allocating more resources to the saccade target provides a more reliable representation of the target; this may support visual stability via a mechanism that uses the presaccadic representation in working memory to identify the target in the shifted retinal image after the saccade (Currie et al., 2000; Hollingworth et al., 2008).

Accompanying this shift of resources toward the saccade target, receptive fields in the parietal cortex, frontal eye fields, and other levels of the visual system shift in the direction of the saccade (Duhamel et al., 1992; Sommer and Wurtz, 2006). A recent observation suggests that this shift is not a fixed translation of each receptive field, but characterized as a compression of the receptive fields toward the saccade target (Zirnsak and Moore, 2014; Zirnsak et al., 2014). The resulting increase in receptive field density may support the high-resolution representation of the saccade target and objects in its vicinity, as observed in this study.

It has been suggested that shifts of receptive fields could be responsible for the inadvertent mixing of information presented at retinotopic and spatiotopic target locations. Golomb et al. (2014) presented participants with a precue at one of four locations before a saccade, then presented four color stimuli and asked them to report the color at the precued location. They found that the color reported for this spatiotopically cued location was biased toward the color presented at the same position as the cue on the retina.

Such confusion of retinotopic and spatiotopic information occurring in natural vision could have profound effects on our ability to perceive the visual world. Therefore, we examined in our task whether perception of a target color was influenced by preceding color information presented at the same retinotopic location. Our results showed no influence on postsaccadic perception of the presaccadic color at the same retinotopic position as the target (Fig. 4c). This indicates that any confusion between retinotopically coincident stimuli is minimal under ecological conditions, where the same objects are present both before and after the saccade. Our results show evidence for integration across saccades of visual inputs matched solely in spatiotopic coordinates.

Several studies have reported situations in which presaccadic information influenced postsaccadic target perception. For example, presaccadic adaptation to a tilt has an aftereffect on the perceived orientation of a postsaccadic stimulus (Melcher, 2007), although it is debated whether this occurs when adaptor and stimulus are at the same spatiotopic location, or solely for retinotopically matched inputs (Mathôt and Theeuwes, 2013). Harrison and Bex (2014) found spatiotopic transsaccadic feature integration in a visual crowding task. When flankers were presented at the same spatiotopic location before and after the saccade (but the target only afterward), the critical spacing to enable feature individuation was smaller than when the flankers were only presented after the saccade (Harrison and Bex, 2014).

For color stimuli, perception of a single, briefly presented (40 ms) postsaccadic stimulus was influenced by a separate presaccadic colored stimulus at the same spatiotopic position, even though participants were instructed to ignore the presaccadic color (Wittenberg et al., 2008). Unlike the present study, in this case the target for report was a new object appearing some time after the saccade was complete; effect sizes—representing an undesirable intrusion of irrelevant presaccadic information into perception—were consequently small. In contrast, the present study examined the more ecological situation in which the same stimuli appeared fixed in world coordinates before and after the eye movement. We found that presaccadic and postsaccadic input contributed overall on an approximately equal basis to the final percept, consistent with a true integration of two valid sources of information about the stimulus. The effect observed by Wittenberg et al. (2008) could be from a misapplication of the same integration mechanism.

We found that manipulating the reliability of the presaccadic input (by adding color noise) altered the relative weights given to presaccadic and postsaccadic input in the integrated representation. Importantly, this implies that the information carried over the saccade not only comprised a presaccadic estimate of color input, but also a measure of the reliability of that input that could be compared with the reliability of postsaccadic information. Working memory could be the medium that carries this information. This is consistent with the finding that observers have insight into the precision of their memories, as demonstrated by the confidence they ascribe to their estimates (Rademaker et al., 2012) and those estimates they choose as best remembered (Fougnie et al., 2012). Understanding the nature and neurophysiological basis of working memory representations of confidence is an important topic for future research.

In the experiment in which presaccadic reliability was manipulated, we observed a significant effect on the weighting of presaccadic and postsaccadic inputs for the intermediate item (int), and not for the other two items. In the case of the item closest to the saccade target (fix2), the reliability of the presaccadic representation was not significantly affected by color noise, most likely because of the strong allocation of visual attention to this item before the saccade (Deubel and Schneider, 1996). Therefore, it is predictable that we did not observe a significant effect of the noise manipulation on bias for this item. For the item closest to the initial fixation position (fix1), increased presaccadic noise did lead to significantly more variability in the representation, albeit less than for the intermediate item, but this decrease in reliability did not result in a significant bias toward the postsaccadic input. This could represent evidence for a suboptimal integration in the case of this item location, perhaps because of a failure to assess or accurately store the reliability of the presaccadic item's representation. However, we did not obtain positive evidence for a deviation from optimal integration for this or any other item: that would be demonstrated by a significant difference in estimated postsaccadic signal between low- and high-noise conditions, which was not observed.

Based on the variability in presaccadic representations and the observed weighting of presaccadic and postsaccadic input, we could infer the variability of the postsaccadic update signal. While reports based only on presaccadic input showed strong advantages both for the initial fixation location and the target of the saccade (Fig. 1f, dark blue), the inferred representation of postsaccadic input displayed very high resolution solely at the saccade target, i.e., the new fixation location (Fig. 4a). In addition to the improvement in resolution from foveation, this suggests that, immediately following a saccade, attentional resources are tightly focused on the new center of gaze. This high-resolution representation of the region corresponding to the previous saccade target could support a detailed comparison with presaccadic input that determines whether a gaze correction is necessary.

Two observations support the view that visual working memory is the medium by which presaccadic information is maintained across saccades for integration. First, the capacity to consciously retrieve information after a saccade is equivalent to the capacity of visual working memory (Irwin, 1992, 1996). Second, integration occurs in spatiotopic coordinates, while low-level perceptual integration mechanisms would be expected to operate retinotopically. Nonetheless, examples such as the tilt aftereffect (Melcher, 2007; see also Fracasso et al., 2010) indicate that working memory is not the only mechanism by which visual information can survive a saccade. We therefore leave open the possibility that some other mechanism of transsaccadic maintenance with similar properties to working memory is responsible for the integration observed here.

The present work shows that information about stable objects is accumulated across a saccade, taking into account the reliability of presaccadic and postsaccadic information. Future work could examine the process of integration under more naturalistic conditions, involving longer sequences of saccades and complex visual scenes.