Optimal trans-saccadic integration relies on visual working memory

Abstract

Saccadic eye movements alter the visual processing of objects of interest by bringing them from the periphery, where there is only low-resolution vision, to the high-resolution fovea. Evidence suggests that people are able to achieve trans-saccadic integration in a near-optimal manner; however the mechanisms underlying integration are still unclear. Visual working memory (VWM) is sustained across a saccade, and it has been suggested that this memory resource is used to store and compare the pre- and post- saccadic percepts. This study directly tested the hypothesis that VWM is necessary for optimal trans-saccadic integration, by introducing memory load during a saccade, and testing subsequent integration performance on feature similar and dissimilar stimuli. Results show that integration performance was impaired when there was an additional memory task. Additionally, performance on the memory task was affected by feature-specific integration stimuli. Our results suggest that VWM supports the integration of pre- and post- saccadic stimuli because integration performance is impaired under VWM load.

1. Introduction

Humans make up to three saccadic eye movements every second, selecting potential fixation targets with peripheral vision, and bringing them into foveal focus after the saccade. Due the non-homogenous distribution of photoreceptors in the retina (Rovamo, Virsu, & Näsänen, 1978), this means that with every saccade, our visual input switches from low-resolution pre-saccadic information to high-resolution post-saccadic information. However, despite this constant flux of low to high resolution information, we do not notice these differences in acuity across eye movements, and instead maintain a remarkably stable percept of the world.

One factor that might contribute to perceptual stability is trans-saccadic integration. Despite early arguments against the existence of trans-saccadic integration (O'Regan and Lévy-Schoen, 1983, Rayner and Pollatsek, 1983), many studies have now demonstrated that trans-saccadic integration of pre- and post-saccadic stimuli occurs for features such as orientation and form (Demeyer et al., 2010, Melcher, 2005, Melcher, 2007), colour (Oostwoud Wijdenes, Marshall, & Bays, 2015), location information (Prime, Niemeier, & Crawford, 2005), and stimulus position (Cicchini, Binda, Burr, & Morrone, 2013), as well as providing evidence for the fusion of pre- and post-saccadic stimuli (Paeye, Collins, & Cavanagh, 2017). By considering the pre- and post-saccadic information to be two separate, independent sources of sensory information, two recent studies used maximum likelihood estimation (MLE) (Ernst & Bülthoff, 2004) to show that integration occurs in a near-optimal manner, when comparing observed integration performance with predicted integration performance based on the performance on individual conditions alone (Ganmor et al., 2015, Wolf and Schütz, 2015). However, while there is solid evidence that integration occurs, it is unclear what mechanisms may underlie or facilitate this process.

Many studies have provided evidence for the existence of a trans-saccadic memory resource, which supports the maintenance of pre-saccadic information for subsequent comparison or integration with post-saccadic information. Early evidence for the existence of trans-saccadic memory came from a study demonstrating that people are able to identify an object faster after the saccade if they have been shown a preview of the object before the saccade (Henderson, Pollatsek, & Rayner, 1987), suggesting that the representation of these stimuli is retained across eye movements. It was subsequently proposed that integration relies on a limited-capacity memory resource that does not rely on the absolute position of objects in the visual field (Irwin, 1991). Numerous studies have since tested the properties of this trans-saccadic memory, demonstrating that subjects can remember both object identity and position across saccades (Irwin & Andrews, 1996), that trans-saccadic memory performance is improved at locations near the saccade target (Irwin and Andrews, 1996, Irwin and Gordon, 1998), and that the amount of information that can be accumulated across eye movements is affected by the capacity of trans-saccadic memory (Irwin & Andrews, 1996). Trans-saccadic memory and visual working memory (VWM) seem to share many similarities, suggesting that VWM is the memory resource underlying observed trans-saccadic memory effects (Hollingworth, Richard, & Luck, 2008). For example, both trans-saccadic memory and VWM have a similar capacity of 3–4 objects (Irwin, 1992, Luck and Vogel, 1997, Prime et al., 2007), or are alternatively a flexible limited-capacity resource (Ma, Husain, & Bays, 2014), and this capacity is determined by the number of objects and not the number of features within the objects (Irwin and Andrews, 1996, Luck and Vogel, 1997). The role of VWM in trans-saccadic memory was seen to aid the comparison of pre- and post-saccadic stimuli, and to establish object correspondence across fixations. Early theories such as the saccade-target theory (Irwin, 1992) suggested that VWM aids trans-saccadic integration by storing information about a target before the saccade, and then retrieving this information after the saccade: this is then used to compare and integrate the pre- and post-saccadic target representations (Hollingworth et al., 2008). It has also been suggested that VWM across saccades helps to maintain object correspondence across saccades by correcting oculomotor plans so that the eye lands on a post-saccadic object that matches the pre-saccadic target (Hollingworth et al., 2008), that a colour held in VWM can bias saccade targeting (Hollingworth & Luck, 2009), and that VWM can be spatiotopically remapped across saccades (Zerr et al., 2017). More direct evidence that VWM plays an important role in trans-saccadic integration comes from a study directly testing the link between VWM and the integration of pre- and post-saccadic features (Prime et al., 2005). In this case, participants had to judge the intersection point of pre- and post-saccadic oriented bars – a task that required memory of orientation and location information across saccades. The study found that participants were able to integrate this orientation and location information, suggesting that information is retained from one fixation to the next, and used for subsequent integration.

An alternative account of how pre- and post-saccadic information may be combined comes from studies of perceptual fusion, where a post-saccadic stimulus is directly overlayed with the pre-saccadic stimulus to form a composite image. Early studies found no evidence for any form of fusion (Bridgeman and Mayer, 1983, Irwin et al., 1990, O’Regan and Levy-Schoen, 1983), leading researchers to suggest that any interaction between pre- and post-saccadic information should be due to the retention and comparison of the pre-saccadic stimulus with the post-saccadic stimulus (Demeyer et al., 2009, Irwin, 1991). Recent evidence however has suggested that trans-saccadic fusion can occur under specific circumstances (Paeye et al, 2017), re-opening the possibility for some form of low-level feature transfer that occurs across saccades, which may not actively require encoding and comparison via memory resources. If trans-saccadic integration is supported by trans-saccadic fusion, it might not rely on memory resources.

Thus, it seems to be the case that there is some sort of trans-saccadic memory resource that retains information across saccades, and this resource is likely VWM. Additionally, people use this trans-saccadic memory to integrate and consolidate information across saccades. However, while these studies have examined the role of VWM in maintaining information across saccades, and in planning saccades, they did not directly test whether VWM is necessary for the optimal integration of pre- and post-saccadic information across the saccade. This study aimed to determine whether visual working memory supports trans-saccadic integration by introducing memory load during the saccade. Participants performed two tasks: a standard working memory task, and an integration task, where participants had to perceptually discriminate stimuli that were presented either before the saccade (peripheral trials), after the saccade (foveal trials), or throughout the whole saccade (trans-saccadic trials). The perceptual discrimination performance on the peripheral and foveal tasks alone was used to predict performance if optimal integration occurred, and this was then compared with observed performance in trans-saccadic trials. If VWM is required for integration, introducing the memory task concurrent to the integration task should deplete the memory resources available for integration, and sub-optimal integration performance should be observed. This experiment also explored whether the memory underpinning integration is a general or feature-specific resource by testing memory items that were either closely related to the integration task, or feature dissimilar (orientation or colour).

2. Methods

2.1. Participants

20 participants (6 male, 14 female), aged between 19 and 29 took part in the study. All were naive as to the purposes of the experiment, and had normal or corrected-to-normal vision. All participants had normal colour vision according to the Ishihara test for colour blindness (Ishihara, 1960). Participants were paid or given course credit for their time. Ethics approval was obtained from the local ethics commission of the Department of Psychology of Marburg University (proposal number 2015-35k), and experiments were conducted in accordance with the Declaration of Helsinki.

2.2. Equipment

Stimuli were presented on a 91 × 51 cm back projection setup with a PROPixx projector from VPixx Technologies and screen from Stewart Filmscreen. The screen had a resolution of 1920 × 1080 and a refresh rate of 120 Hz, with a viewing distance of 106 cm. Background luminance was 92 cd/m² and the screen was calibrated to ensure a linear gamma correction. Eye movements were recorded with an Eyelink 1000 (SR Research Ltd., Ontario, Canada) with a sampling rate of 1000 Hz. Experimental software was written in Matlab using the Psychophysics Toolbox (Brainard, 1997, Pelli, 1997). Participants responded using a standard keyboard and mouse.

2.3. Stimuli

The central fixation target was a combination of a bulls-eye and a cross-hair shape (Thaler, Schutz, Goodale, & Gegenfurtner, 2013). On trials with both the memory and integration tasks, the fixation target was black. For trials with a memory task alone, the fixation target was white, and for trials with the integration task alone, the fixation target was a random colour generated in DKL colour-space (Derrington, Krauskopf, & Lennie, 1984) with a set Cartesian value of 0.4 in the L + M axis, 0.6 on the L − M axis, and 0 on the S axis, and randomised polarity to avoid the build-up of afterimages. The saccade target that appeared either before or after the perceptual integration stimulus was a black dot with diameter 0.18°, and luminance 3.36 cd/m².

2.3.1. Orientation stimuli in the integration task

Saccade targets in the orientation integration task were oriented Gabors, presented at a randomly determined orientation (from 0 to 180°) on each trial. The Gabors had a standard deviation of 3.2°and a spatial frequency of 2c/°. They were overlayed with band pass filtered noise with a central frequency of 2c/° and a Gaussian standard deviation of 1°. Peripheral stimulus contrast was 25%, foveal contrast was 21%. These values were used to equate peripheral and foveal performance and were based on threshold values obtained in a pilot study.

2.3.2. Colour stimuli in the integration task

Saccade targets in the colour integration task were coloured discs of 3.2° in diameter. The colour of the colour stimulus was randomly chosen on each trial from a set of colours generated in CIE L^*a^*b space (radius 60, luminance 52).

2.3.3. Stimuli in the memory task

All memory stimuli were 1° in diameter. Orientation memory items were oriented Gabors at 100% contrast, with a spatial frequency of 2c/°. Stimulus orientation was randomly determined on each trial and could range from 0 to 180°. A colour wheel of was generated from colours in CIE L^*a^*b space (radius 60, luminance 52), resulting in 734 colours. Colour memory items were randomly selected from this colour wheel on every trial. For memory response on colour items, the colour wheel was centrally placed on the screen, with an outer diameter of 17.9°, and inner diameter of 10.2°.

2.4. Procedure

The experiment was comprised of two memory tasks (orientation or colour) and two integration tasks (orientation or colour). Orientation and colour integration were tested in separate blocks of 80 trials per block. Trans-saccadic integration was assessed in three different types of trials (peripheral, foveal or trans-saccadic trials). These trials were interleaved in a block such that a trial could contain the memory task alone (Fig. 1A), the integration task alone (Fig. 1B), or any combination of memory and integration tasks (Fig. 1C). To balance the ultimate number of trials in memory and integration conditions, each block contained twice as many integration as memory trials. Participants completed 2 sessions of 2 h each, resulting in 14–20 blocks per participant with a total number of 1120–1600 trials over all blocks, or 56–80 trials per condition (for example peripheral orientation integration with orientation memory) before exclusions.

Fig. 1.

Events in a trial. A: Memory task. B. Integration task. C. Integration + memory task. Here one example of a combination of colour memory and a trans-saccadic trial with orientation integration is shown. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

2.4.1. Memory task

In trials containing the memory task alone (Fig. 1A), participants started a trial by depressing the space-bar. After a random delay, the memory item (either oriented Gabor or colourd circle) appeared for 500 ms at one of eight equidistant locations at 3° from the fixation target. After the memory item has disappeared, participants were required to make a saccade to a saccade target (black dot) which appeared at 15° left or right on the screen. To equate the time between the disappearance of the memory item and response across the integration + memory task and the memory task alone, there was a pause between the detected saccade onset and the memory response prompt. This pause was calculated as the median time taken between saccade onset and response on the integration task in the integration + memory task in each block. After this pause, participants were asked to report the memory item. For orientation memory, a black outline of a circle appeared at the location of the memory item, and a bar appeared in the centre. Participants responded to the orientation of the stimulus by using a mouse to rotate a bar to match the perceived orientation, and confirmed with a mouse click. For colour memory, the black outline of a circle appeared at the location of the memory item together with a colour wheel. Participants used the mouse pointer to pick the remembered colour of the memory item.

2.4.2. Integration task

In trials in which there was an integration task but no memory task (Fig. 1B), participants fixated the central fixation target, and then pressed the space bar to begin. The pre-saccadic stimulus then appeared at 15° left or right on the screen. In peripheral and trans-saccadic trials, the pre-saccadic stimulus was the oriented Gabor or coloured circle with a small black dot in the centre (saccade target); in foveal trials, the small black dot alone. After the initiation of a saccade (defined as the eye having moved more than 1.5° from the centre of the screen), the saccade stimulus changed to the post-saccadic stimulus, which was presented for the same duration as the pre-saccadic stimulus; in peripheral trials this was a small dot, in foveal trials and trans-saccadic trials, this was the oriented Gabor or coloured circle. Participants then responded to either the orientation or colour of the saccade stimulus, using the same method as the response to the memory stimuli.

2.4.3. Integration + memory task

For trials in which there was both the memory and integration task (Fig. 1C), participants completed both tasks as described above: they were first presented with the memory items, then completed the integration task, and subsequently reported the memory item.

2.4.4. Exclusions

Trials were excluded on the following bases across all conditions: if the saccade latency was below 50 ms (to avoid anticipatory saccades); if the saccade latency was more than 2 standard deviations above the median latency and if the saccade landing position was more than 2 standard deviations from mean saccade landing position for each participant. Saccade latency was measured as the time from saccade stimulus onset to saccade onset, as calculated by the Eyelink saccade detection algorithm. 4.6% of trials were excluded for technical reasons. In total 81% of trials were included for analysis, constituting 28,320 trials across all participants.

2.5. Analyses

For orientation judgments, perceptual performance was measured as the smallest angular distance between the presented stimulus orientation and reported stimulus orientation. For colour judgments, perceptual performance was measured as the smallest angular distance between the presented stimulus colour and reported stimulus colour. To equate the scales of colour and orientation measures, colour judgment errors were divided by 2. To remove extreme outliers, any errors +−2SD of the mean error were omitted from the distribution. To quantify performance, a cumulative Gaussian distribution function was fitted to the distribution of errors for each condition (Fig. 2). The fit of the cumulative distribution function was more robust than fitting a standard mixture model (Bays et al., 2009, Zhang and Luck, 2008) for conditions with a smaller number of data-points. The just noticeable difference (JND) was measured as the standard deviation of this fitted distribution. The analysis was used to quantify perceptual performance in both the integration and the memory task.

Fig. 2.

Gaussian cumulative probability function fitted to the error measurements for one example condition, for one subject. Measured values are shown in black, with the fitted curve in gray.

2.5.1. Bayes factor calculations

All Bayes factors were calculated using the BayesFactor package in R using default priors. Bayes factors for t-test analyses used a weakly informative Jeffreys prior on variance and Cauchy prior on effect size. For ANOVA analyses, a g-prior was placed on variance and Jeffrey’s prior on effects. For mixed model analyses, Bayes factors were calculated using the same fixed and random effects as the frequentist model, with a default Inverse gamma prior on the regression and Jeffreys prior on effects (Rouder, Morey, Speckman, & Province, 2012). Bayes factors for main effects were calculated as the ratio of evidence for the model containing only that factor vs the null model (intercept and random effects only). Interactions were calculated as the model containing main effects with no interaction term vs the full model.

2.5.2. Predicted integration performance

For the integration stimuli, maximum likelihood estimation (MLE) was used to determine the predicted performance if foveal and peripheral information was optimally integrated (per Wolf & Schütz, 2015). The individual reliabilities for each condition (foveal, peripheral, integration) were calculated using the equation:

$$\mathit{rel}=\frac{1}{{\mathit{JND}}^2}$$ (1)

Predicted integration performance can then be calculated as the sum of the reliabilities of foveal and peripheral performance alone (Ernst & Bülthoff, 2004):

$$\mathit{re}{l}_{\mathit{int}}=re{l}_{\mathit{per}}+re{l}_{\mathit{fov}}$$ (2)

The JND for this predicted performance is then calculated as:

$${\text{JND}}_{\mathit{int}}=\sqrt{\frac{1}{\mathit{re}{l}_{\mathit{int}}}}$$ (3)

Predicted integration performance can then be compared to observed integration performance to determine whether optimal integration is occurring. To quantify the relationship between predicted and observed integration performance, we can calculate the benefit to integration as the difference between the best individual performance (either peripheral or foveal) and observed trans-saccadic performance, divided by the difference between best individual performance and predicted trans-saccadic performance:

$$\mathit{Integration}Benefit=\frac{{\mathit{JND}}_{\mathit{best}single}-{\mathit{JND}}_{\mathit{int}(obs)}}{{\mathit{JND}}_{\mathit{best}single}-{\mathit{JND}}_{\mathit{int}(pred)}}$$ (4)

2.6. Participant exclusions

According to the MLE model, the benefit of integration is maximal when the performance on individual peripheral and foveal performance is equated, and it decreases with increasing difference between peripheral and foveal performance. Studies investigating the principles of maximum-likelihood estimation usually try to equate performance in the single conditions to maximize the potential benefits of integration (Alais and Burr, 2004, Gepshtein et al., 2005, Gu et al., 2008, Bentvelzen et al., 2009, Jones, 2016, Rohde et al., 2016). To this end, we also tried to equate peripheral and foveal performance as much as possible by reducing foveal contrast as measured in a pilot experiment. However, this procedure did not equate performance for all individual participants on every condition. Therefore, we excluded participants whose performance on individual foveal and peripheral trials was not well matched. To quantify the match, for each participant, for each integration condition (colour or orientation) and each memory condition (no memory, colour memory and orientation memory), we calculated the ratio of the difference between best single performance (peripheral or foveal) and predicted performance vs. the difference between the worst single performance and predicted performance. This ratio ranges from 0 when the predicted integration performance equals the best single performance to 1 when the single performances are identical and the predicted integration benefits are maximal. We excluded those conditions with a score below 0.2, which had the most extreme differences between individual conditions. For orientation integration, 14 of 60 conditions were excluded (4 no memory, 4 colour memory, 6 orientation memory), and for colour integration 12 of 60 conditions were excluded (4 for no memory and colour memory, 3 for orientation memory). As the experimental design was primarily concerned with within-subjects differences between memory conditions, we excluded any subject for whom one memory condition did not meet the criteria (7 participants were excluded for orientation integration, and 10 for colour integration). For orientation integration, a further 3 subjects were excluded as JNDs on all conditions were more than a standard deviation from the mean JND across all subjects and conditions, indicating a high rate of guessed responses. For both orientation and colour integration, 10 subjects were included for final analysis for each condition (5 participants were included for both conditions). See Supplementary materials for further details on participant exclusions.

2.7. Control variables

2.7.1. Stimulus durations and memory delays

In order to compare integration and memory performance across conditions, it is important that pre- and post-saccadic stimulus durations and memory delays are similar. The following values are for all participants. For the integration only condition, the median saccade latency was 214 ms with a standard deviation of 92 ms (trans-saccadic condition: 198 (64); foveal condition: 253 (114); peripheral condition: 196 (71)). A one-way ANOVA revealed a significant difference in saccade latencies in the integration only conditions: F (2,60) = 54.32, p < 0.0001, B₁₀ = 161,704,531,318. There was a difference between the foveal condition and other conditions, as in the foveal condition, the saccade was “cued” by the saccade target, which was a small black dot, whereas in the peripheral and trans-saccadic conditions, the saccade was cued with the larger gabor (perceptual stimulus), leading to slower latencies in the foveal condition. However, we believe it does not affect performance, and if anything would cause an over-estimation of integration performance in predictions, which does not contribute to our overall results. For the integration plus memory condition, the median saccade latency was 193 ms with a standard deviation of 68 ms (trans-saccadic condition: 182 (51); foveal condition: 216 (85); peripheral condition: 183 (49)). There was again a significant difference between conditions in the integration plus memory: F (2,60) = 10.89, p = <0.0001, BF₁₀ = 269.8. For the memory only condition, the median memory delay was 4663 ms with a standard deviation of 1225 ms, and for integration plus memory, the median memory delay was 4461 ms, standard deviation 1271 ms. There was no evidence for a difference in memory delay between conditions: F (3,80) = 0.75, p = 0.53, BF₁₀ = 0.43.

2.7.2. Saccade amplitudes

Means and standard deviations for saccade amplitudes were calculated across all participants for each condition. Integration only 14.87 deg (0.41): (trans-saccadic condition: 14.88 deg (0.41); foveal condition: 14.91 deg (0.38); peripheral condition: 14.81 deg (0.47)). Integration plus memory: 14.87 deg (0.40) (trans-saccadic condition: 14.87 deg (0.42); foveal condition: 14.87 deg (0.38); peripheral condition: 14.87 deg (0.41)). Memory only: 14.9 deg (0.40). A one-way ANOVA revealed no significant difference in mean amplitude across all conditions: F (6,140) = 0.12, p = 0.99, and there was strong evidence that amplitudes did not differ: BF₁₀ = 0.016.

3. Results

Integration performance was calculated for both orientation and colour integration, with concurrent colour or orientation memory task or without memory task.

Fig. 3 shows JNDs for integration performance for foveal, peripheral and trans-saccadic trials, and predicted performance based on the peripheral and foveal performance alone.

Fig. 3.

Average JNDs in the integration task for orientation integration (A) and colour integration (B). Peripheral trials (green), foveal trials (blue), trans-saccadic trials (purple), and predictions (grey) are shown for conditions when there was a concurrent memory task (colour or orientation) or no concurrent memory task. Error bars represent 95% confidence intervals. Conditions in which there was a significant difference between the predicted and observed trans-saccadic performance are marked with an asterisk. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

3.1. Orientation integration

Fig. 3A shows performance on orientation integration across different memory conditions. Without concurrent memory task, observed integration performance was similar to predicted performance; with orientation or colour memory task, integration performance was more similar to the best single performance (peripheral or foveal). We used different methods to assess the quality of trans-saccadic integration.

3.1.1. Orientation: benefit from integration

To determine whether there was a benefit from integration in each memory task, we calculated the benefit from integration as the difference between the best individual performance (either peripheral or foveal) and observed trans-saccadic performance, divided by the difference between best individual performance and predicted trans-saccadic performance (Eq. (4)). This value is 0 if the observed integration performance is equal to the best single performance and 1 if the observed integration performance is equal to the predicted integration performance. For each condition, one-tailed t-tests with a Holm correction for multiple comparisons were used to test whether the benefit to integration was larger than 0 (showing that a benefit occurred). There was a significant benefit for the no memory condition: t(9) = 2.99, p = 0.023, BF₁₀ = 8.78, but not for orientation memory: t(9) = 1.29, p = 0.23, BF₁₀ = 1.03 (although there is weak anecdotal evidence for an effect), or colour memory: t(9) = 0.71, p = 0.25, BF₁₀ = 0.56. This indicates that there was only conclusive statistical evidence in favour of a benefit from trans-saccadic integration without concurrent memory task.

3.1.2. Orientation: trans-saccadic vs predicted performance

To determine whether the memory task affected integration vs predicted performance alone, mixed models were used to test the difference between observed integration performance and predicted integration performance with and without memory. The model contained fixed effects of trans-saccadic performance (observation or prediction) and memory condition (orientation memory, colour memory or no memory), and random effect of participant. There was a significant effect of trans-saccadic performance: F (1,9) = 7.5, p = 0.023, BF₁₀ = 26.56; a significant effect of memory condition: F (2,36) = 9.44, p = 0.0005, BF₁₀ = 6.36; but no significant interaction between integration and memory: F (2,36) = 0.98, p = 0.39, BF₁₀ = 0.31. To determine if there was a difference between colour and orientation memory, we ran a separate mixed model with the memory condition containing just colour or orientation memory, with fixed effects of trans-saccadic performance (observation or prediction) and random effect of participant. There was a significant effect of trans-saccadic performance: F (1,9) = 6.82, p = 0.028, BF₁₀ = 16.5, but no significant effect of memory condition: F (1,18) = 3.97, p = 0.06, BF₁₀ = 0.61, and no significant interaction: F (1,18) = 0.33, p = 0.57, BF₁₀ = 0.4. This indicates that the detriment to integration performance changed across memory conditions, however while there was a trend towards a feature-specific effect, there was no statistical evidence for a difference between orientation and colour memory items.

3.1.3. Orientation: trans-saccadic vs best single performance

In addition to comparing observed and predicted trans-saccadic performance, one can compare the observed trans-saccadic performance to the best single performance in peripheral or foveal conditions. To test whether the addition of memory load affected the gain of trans-saccadic integration, we used a mixed model with fixed effects of eye-movement condition (trans-saccadic vs best single condition) and memory condition (colour, orientation or no memory), and random effect of participant. There was a significant effect of memory condition, with weak evidence for the effect: F (2,36) = 7.05, p = 0.0026, BF₁₀ = 2.7, and a significant effect of eye-movement condition: F (1,9) = 8.66, p = 0.016, BF₁₀ = 44.6, but no significant interaction between memory and eye movement condition: F (2,36) = 0.82, p = 0.45, BF₁₀ = 0.36. To determine whether there was any difference between colour and orientation memory, we ran a mixed model as above with fixed effects of memory condition (orientation or colour) and eye-movement condition, and random effect of participant. There was no significant effect of eye-movement condition: F (1,9) = 3.5, p = 0.094, although there was weak evidence for an effect BF₁₀ = 2.6; memory condition: F (1,18) = 2.8, p = 0.11, BF₁₀ = 0.6; or the interaction between eye-movement and memory condition: F (1,18) = 0.32, p = 0.58, BF₁₀ = 0.36. This indicates that there was only weak evidence for a difference between trans-saccadic and best single performance for either colour or orientation memory, and there was no difference between memory conditions.

In sum, trans-saccadic integration was impaired by the memory task: observed trans-saccadic performance matched predicted optimal trans-saccadic performance only without memory task, but was significantly reduced with orientation or colour memory. Trans-saccadic performance was only significantly better than the best peripheral or foveal performance without memory task. This indicates that for orientation integration, the addition of a memory task specifically affects the ability to integrate peripheral and foveal information but this impairment does not seem to be specific for a certain feature.

3.2. Colour integration

Fig. 3B shows performance in the colour integration task across different memory conditions. Again, integration performance was similar to the predictions without concurrent memory task, and worse than predicted with either a concurrent orientation or colour memory task, while peripheral and foveal performance remained unaffected.

3.2.1. Colour: benefit from integration

Benefit from integration was calculated for colour integration for the no memory, colour memory and orientation memory conditions, and as with orientation integration, was tested using one-tailed t-tests with a Holm correction. There was a significant benefit from integration for the no memory condition: t(9) = 3.37, p = 0.012, BF₀₁ = 14.4, for orientation memory: t(9) = 3.4, p = 0.012, BF₀₁ = 14.82, but not for colour memory: t(9) = −0.74, p = 0.76, BF₁₀ = 0.20. This indicates a benefit from integration when there was no memory task, an orientation memory task but not a colour memory task.

3.2.2. Colour: trans-saccadic vs predicted performance

A mixed model with the same fixed and random effects as previously described was used to determine the difference between observed and predicted integration performance. First we tested the effect of different memory conditions. There was a significant effect of trans-saccadic performance: F (1,9) = 45.55, p = 0.0001, BF₁₀ = 6512, a significant effect of memory condition: F (2,36) = 17.77, p < 0.0001, BF₁₀ = 29.08, and a significant interaction between trans-saccadic performance and memory conditions: F (2,36) = 3.73, p = 0.034, although with only weak evidence for this interaction: BF₁₀ = 1.8. As there was a significant interaction between trans-saccadic and memory conditions, we conducted post-hoc pairwise comparisons with a Tukey adjustment for multiple comparisons to test the difference between individual memory conditions. This revealed no significant difference between observed and predicted performance without a memory task: t(9) = −2.73, p = 0.16 (Bayesian paired-sample t-test however revealed a weak to moderate effect: BF₁₀ = 3); or with orientation memory: t(9) = −3.37, p = 0.065, BF₁₀ = 5.9 (moderate evidence), but a significant difference with the addition of a colour memory task: t(9) = −6.25, p = 0.0015, BF₁₀ = 94.35.

This indicates that the addition of a memory item affected trans-saccadic performance, and this is a feature-similar effect for colour but not orientation memory, although there is moderate evidence to suggest that orientation memory also played a role.

3.2.3. Colour: trans-saccadic vs best single performance

To test whether integration was occurring for each memory condition, we compared trans-saccadic performance with the best single performance, as described above. There was a significant effect of memory condition: F (2,36) = 13.72, p = <0.001, BF₁₀ = 116; and of eye-movement condition: F (1,9) = 10.17, p = 0.011, BF₁₀ = 7.42; and a significant interaction: F (2,36) = 4.12, p = 0.025, BF₁₀ = 2.35. To test the effect of the individual memory conditions we ran post-hoc multiple comparisons with a Tukey correction. There was a significant difference between integration and best single performance for the no memory condition: t(9) = −3.11, p = 0.013 (Bayesian paired-sample t-test: BF₁₀ = 13.21), and for orientation memory: t(9) = −2.98, p = 0.016, BF₁₀ = 10.23, but not for colour memory: t(9) = 0.43, p = 0.68, BF₁₀ = 0.31. This indicates that trans-saccadic performance is better than either single percept, and that memory again induces a feature-specific impairment in trans-saccadic integration. These results suggest that for colour integration, as with orientation integration, completing a memory task affects the ability to integrate peripheral and foveal information across saccades; however the detriment to colour integration shows a feature-specific effect.

3.3. Peripheral and foveal performance

To determine whether peripheral and foveal performance was affected by the addition of a memory task, we tested the effects of memory on peripheral and foveal performance for both colour and orientation integration. We included all participants for this analysis to ensure that the exclusions used to equate performance did not bias the results, as exclusions were performed in order to equate peripheral and foveal performance. We used a linear mixed model with fixed effects of integration condition (peripheral or foveal), and memory condition (memory or no memory), and random effect of participant. For orientation integration, there was no significant effect of memory condition: F (1,32) = 4.21, p = 0.05, and no evidence for an effect of memory condition: BF₁₀ = 0.43; and there was no interaction between integration condition and memory condition: F (1,32) = 0.90, p = 0.35, BF₁₀ = 0.38. For colour integration, there was no significant effect of memory condition: F (1,38) = 2.3, p = 0.13, BF₁₀ = 0.39; and no interaction between integration condition and memory condition: F (1,38) = 0.56, p = 0.46, BF₁₀ = 0.31. This indicates that the addition of a memory task did not affect either peripheral or foveal performance alone, so memory load affects the ability to integrate pre- and post-saccadic information rather than affecting either percept individually.

3.4. Trans-saccadic integration affects memory performance

Since the trans-saccadic integration and memory task could be subject to dual-task trade-offs, we also analysed performance in the memory tasks. Fig. 4 shows performance on the memory tasks, both with and without the intervening integration task.

Fig. 4.

Performance on memory tasks. A: Orientation memory performance for no concurrent integration task (grey), concurrent orientation integration task (blue) and concurrent colour integration task (red). B: Colour memory performance for no concurrent integration task (grey), concurrent orientation integration task (blue) and concurrent colour integration task (red). Error bars are 95% confidence intervals. Significant differences are indicated with an asterisk. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

To investigate whether completing an integration task in the memory interval had any effect on memory performance, a mixed model was used to compare performance across integration task (colour, orientation or none) for colour memory. The model had a fixed effect of integration task, and random effect of participant. There was a significant main effect of integration task: F (2,13) = 26.59, p < 0.0001, BF₁₀ = 25.81. Post-hoc pairwise comparisons with a Tukey adjustment indicate a significant difference between the no integration condition and the colour integration condition: t(13) = 7.07, p < 0.0001, (Bayesian paired-sample t-test: BF₁₀ = 12.37) but not in the orientation integration condition: t(13) = 1.55, p = 0.57, BF₁₀ = 0.38. A mixed model as above was used for orientation memory: there was a significant main effect of integration task: F (2,13) = 7.04, p = 0.0085, BF₁₀ = 12.13; post-hoc comparisons showed a difference between no integration and orientation integration: t(13) = 3.75, p = 0.0064, BF₁₀ = 402.3, but not for colour integration: t(13) = 1.45, p = 0.34, BF₁₀ = 0.39. This indicates that the addition of the integration task significantly affected memory performance for feature-specific conditions.

3.5. Interference between integration and memory stimuli

To further explore whether the feature-specific detriments to memory and integration were due to any bias or interference between the stimuli, we conducted an analysis to determine whether the presentation of one stimulus biased the response to the other stimulus for each subject and each stimulus combination (orientation integration with orientation memory, orientation integration with colour memory, colour integration with orientation memory, and colour integration with colour memory).

To calculate the effect of memory on the integration response, for each trial, we calculated the smallest angular distance between the presented memory and integration stimuli, and then rotated and flipped the presented stimuli such that the memory stimulus was zero, and the integration stimulus was always situated a positive distance from the memory stimulus. The responses to these stimuli were transformed accordingly (Fig. 5A). We then calculated the angular distance between the memory and integration stimuli (between 0 and 90°) (Fig. 5B x-axis), and for each difference value we plotted this same difference value, resulting in a slope of 1. The corresponding responses to the integration stimuli were then also plotted as a function of the distance between the shown stimuli. A linear regression was fitted to these responses (Fig. 5B, y-axis). If the memory stimulus biased the response to the integration stimulus, the integration responses should be closer to the memory stimulus (zero), and the fitted regression should show a slope shallower than 1. This same method was used to determine the effect of the integration stimulus on the memory response. Fig. 5B shows an example plot from one subject for orientation integration with orientation memory. Fig. 5C and D shows the average slope of the fitted regression for each stimulus combination.

Fig. 5.

Interference between memory and integration stimuli. A: Diagram representing the rotation of shown stimuli and responses such the memory stimulus is zero, and the integration stimuli and responses are always a positive distance from 0. B: Example plot from one participant for orientation integration with orientation memory. The distance between the memory stimulus and integration stimuli/responses (y axis) are plotted against the distance between the shown stimuli (memory and integration stimuli) on the x axis. C: Mean slopes of the fitted regression for the difference between memory stimulus and integration responses. The horizontal black line represents the slope of the difference between memory and integration stimuli. D: Mean slopes of the fitted regression for the difference between integration stimulus and memory responses. All error bars are 95% confidence intervals.

To determine whether the presentation of a feature similar/dissimilar memory item affected the response to the integration stimulus, we used paired-samples t-tests with a Holm correction for multiple comparisons, to see if the slope of the fitted regression for the difference between the memory stimulus and integration response differed from one. There was no significant effect for any condition(Fig. 5C): orientation integration with orientation memory: t(9) = −1.26, p = 1, BF₁₀ = 0.58; orientation integration with colour memory: t(9) = 0.78, p = 1, BF₁₀ = 0.40; colour integration with colour memory: t(9) = 0.46, p = 1, BF₁₀ = 0.34; colour integration with orientation memory: t(14) = 1, p = 1, BF₁₀ = 0.46. Similarly, we tested whether the integration stimulus biased the memory response, and again there were no significant effects (Fig. 5D): orientation integration with orientation memory: t(9) = −1.04, p = 1, BF₁₀ = 0.48; orientation integration with colour memory: t(9) = 0.51, p = 1, BF₁₀ = 0.34; colour integration with colour memory: t(9) = −0.49, p = 1, BF₁₀ = 0.34; colour integration with orientation memory: t(9) = −0.10, p = 1, BF₁₀ = 0.31.

This suggests that the detriment seen to both integration and memory performance in feature-similar conditions is not due to a reporting bias where the reporting of two feature-similar items causes a bias in response, and is not due to an interference effect where the features of similar items are merged or averaged. This rather suggests a feature-specific interference effect that affects the maintenance of feature-similar stimuli in VWM, which may be due to capacity limitations or selective processing strategies for similar stimuli (Lin & Luck, 2009).

4. Discussion

This study aimed to determine whether visual working memory (VWM) is necessary for integration of pre- and post-saccadic stimuli, and whether there are any feature-specific interactions between integration and memory stimuli. The results clearly show that there is a relationship between VWM load and participants’ ability to integrate: integration performance was impaired by both memory items for orientation integration, and by colour memory only for colour integration. Conversely, memory performance was affected by feature-similar integration stimuli only. This shows that the addition of memory load during a saccade adversely affects integration performance.

These results also support previous work that has argued for the existence of trans-saccadic memory (Irwin, 1991, Irwin, 1996, Irwin and Gordon, 1998), as well as studies that have found a direct relationship between VWM and the integration of pre- and post- saccadic information (Hollingworth et al., 2008, Prime et al., 2007). In addition, this study compared integration performance to the predictions of maximum-likelihood-estimation (Ernst and Bülthoff, 2004, Ganmor et al., 2015, Wolf and Schütz, 2015) and showed that optimal integration across a saccade does not occur when the amount of available VWM resources are depleted. Interestingly, when the integration and memory tasks were being completed together, memory performance was unaffected for feature-dissimilar items, but there was a detriment in memory performance for feature-similar items. For the feature dissimilar items, this could be suggestive of a dual-task trade-off (Cowan and Morey, 2007, Logie et al., 1990, Woodman and Luck, 2004) – memory resources may have been used to retain the memory items, and thus were not available to encode and transfer information about the pre-saccadic stimulus. This lack of effect on memory is similar to the findings of Prime et al (2007), that memory is not affected by an intervening saccade (although they did not include an additional integration task in their paradigm); however the greater detriments seen in the orientation conditions are consistent with recent evidence suggesting that memory for orientation features is disrupted by a saccade, compared to fixation (Jeyachandra, Nam, Kim, Blohm, & Khan, 2018). In our case, the maintenance of the memory item could have been weighted more highly, thus depleting the memory resources required for integration. This suggests that VWM may draw from one unified resource (Frick, 1988) that is allocated preferentially to competing task demands (Hayhoe, Bensinger, & Ballard, 1998). The detriment to integration performance with the addition of memory load may also be indicative of a set-size effect occurring: the limited capacity of VWM may be unable to encode and retain the memory item, the pre-saccadic stimulus, the post-saccadic stimulus and to perform the required trans-saccadic comparison at the same time. This would explain why the addition of a memory item did not cause any detriment to the individual peripheral or foveal performance: peripheral or foveal information alone could be considered to take up one memory slot, so the addition of the memory item would not exceed memory capacity. However, the trans-saccadic stimulus might inherently take up double the memory capacity of either individual stimulus, so the addition of the memory item may have reduced resources such that both pre- and post- saccadic information could not be stored across the saccade. It must be noted that this experiment did not aim to look at set-size effects, so future studies could use multiple memory items to test this hypothesis. This could alternatively reflect the addition of memory load in a resource model of VWM: as more items are added, the amount of VWM allocated to each drops proportionally (Bays and Husain, 2008, Ma et al., 2014). The difference between the memory slot explanation and the resource model is that it would not be the number of items per se that would determine memory performance, rather it is the precision of recall of these items that determines limits on memory capacity (Ma et al., 2014). This would suggest that in this study, rather than the capacity limits of VWM being exhausted by the presentation of both memory and integration items, that the retention of all items caused a decrease in precision of each of them (Keshvari et al., 2013, van den Berg et al., 2012): this could explain why there was a deficit in both integration and memory performance. Resource models also suggest that the allocation of VWM to an object decreases the noise in the representation of that object (Wilken & Ma, 2004): trans-saccadic integration could rely on this reduction of noise across pre- and post-saccadic stimuli to produce a more reliable integrated signal. It could also be the case that this observed detriment to integration is due to a capacity-limited ability of VWM to maintain both the memory and integration items in VWM (Fougnie and Marois, 2006, Fougnie and Marois, 2009), or alternatively it could arise from interference between the maintenance of one item (memory item) and the encoding or retrieval of the other (integration item) (Cowan & Morey, 2007). This may support a maintenance interference hypothesis, whereby the maintenance of two items causes interference between two stimuli being held in a shared, limited-capacity memory resource (Fougnie & Marois, 2009). If this is the case this may suggest that the interference arises not from encoding or retrieval of the memory items, but rather from the storage of items in this limited-capacity system (Fougnie & Marois, 2009).

4.1. How does VWM support trans-saccadic integration?

The results of this study clearly indicate that VWM is necessary for optimal integration, but this then raises the question of how VWM actually supports integration. The first, and perhaps most obvious explanation is that VWM aids integration in a predictive manner: information about a stimulus at an upcoming saccade location can be pre-saccadically encoded and stored in VWM for post-saccadic comparison. This is in line with theories such as saccade target theory, suggesting that pre-saccadic stimuli are encoded into VWM, and post-saccadically matched with similar stimuli in the vicinity of the saccade target (Currie et al., 2000, Irwin et al., 1994). The results of this study suggest that this integration and comparison of stimuli is not an automatic process, rather that this process requires memory resources to retain and subsequently integrate pre- and post-saccadic stimuli. As the pre-saccadic information needs to be encoded and maintained throughout the duration of the saccade preparation and execution, this may suggest that integration begins during the early stages of a saccade. Indeed, this is supported by evidence suggesting that the visual system uses pre-saccadic information to predictively process the post-saccadic stimulus (Fabius, Fracasso, & Van der Stigchel, 2016), and by studies showing that there is a benefit to object identification when there has been a pre-saccadic preview of that object (Henderson et al., 1987), and supports proponents of a predictive integration process (Herwig and Schneider, 2014, Mathot and Theeuwes, 2011, Melcher and Colby, 2008). It has also been shown that VWM can be spatiotopically remapped across saccades (Zerr et al., 2017): maintaining memory items in spatiotopic coordinates could facilitate the comparison of the percept of a location pre- and post-saccadically.

An additional interesting facet of this story may come from the link between attention, integration, and VWM. Attention has been shown to facilitate the encoding and maintenance of information in VWM (Awh and Jonides, 2001, Heuer and Schubö, 2016, Hollingworth and Henderson, 2002, Johnson et al., 2008, Schmidt et al., 2002), and has also been shown to aid in the recall of items across a saccade (Prime et al., 2007), suggesting a role of attention in maintenance of trans-saccadic information. Attention itself has also recently been linked with the facilitation of trans-saccadic integration (Stewart & Schütz, 2018). This study showed that the presentation of an attentional distractor impeded integration performance when presented around the time of saccade onset: this disruption to attention may also have affected the maintenance of the pre-saccadic stimulus in working memory. It may thus be the case that attention and VWM are both important for integration, but these processes may not be entirely dissociable, as VWM is reliant on attentional processes for active maintenance of information.

It is likely that integration occurs by the allocation of attention to the upcoming saccade target (Deubel and Schneider, 1996, Kowler et al., 1995), and that VWM resources are also directed to that location (Ohl & Rolfs, 2017), potentially preferentially given the task-relevance of the object (Bays et al., 2011, Melcher and Piazza, 2011). The pre-saccadic target properties are then encoded into VWM, and can be used to predict the appearance of the post-saccadic target, and this information can be used for facilitation of processing features post-saccadically, or is retained for comparison and integration with the post-saccadic target. This integration is different from the simple overlay of pre- and post-saccadic stimuli proposed by a perceptual fusion account (Irwin, 1991, Paeye et al., 2017). Perceptual fusion would result in a composite image of pre- and post-saccadic information that does not require VWM. At present, there is no evidence that fusion is crucially involved in integration: a recent study showed that numerosity information can be integrated near-optimally even when low-level stimulus features are changed during the saccade, which should hamper perceptual fusion (Hübner & Schütz, 2017). Here we showed in addition that trans-saccadic integration relies on VWM for higher-level comparison of stimuli, resulting not in some form of low-level overlay of pre- and post-saccadic information but in further perceptual benefits as measured by an increase in the reliability of the trans-saccadic percept.

4.2. VWM, integration and features

A secondary aim of this study was to determine whether trans-saccadic integration was affected more by feature-similar than feature-dissimilar memory items, and the results show that orientation integration was affected by both orientation and colour memory, while colour integration was affected by colour memory only. This suggests that for orientation integration, memory load interferes with integration regardless of the features contained within the memory item, which points to VWM as a general resource being used for the transferal and comparison of pre- and post-saccadic information. This is consistent with previous findings that trans-saccadic memory is used for the integration of entire object representations (i.e. coloured letters), rather than the separate features of the objects (i.e. colours or letters separately) (Irwin, 1996).

However, feature-specific detriments were seen in memory performance and in colour integration performance when the stimuli shared the same features. This could be due to some form of feature-specific interference between the two items, such that there is more interference for similar items in memory after the encoding stage (Fougnie & Marois, 2009), or there is an interference in neural channels processing and storing memory items with similar features (Cohen, Konkle, Rhee, Nakayama & Alvarez, 2014). Alternatively, it has been suggested that performance effects might not be due to any interactions between the representations of the stimuli features, but rather due to a strategic difference in processing similar stimuli, for example forming an inhibitory segregation of similar items so that they are distinguishable (Lin & Luck, 2009). Our analysis in Fig. 5 shows no evidence that the reported orientation or colour of either integration or memory stimulus was systematically biased by the presence of the other stimulus, and this supports the idea that differences in feature-specific maintenance may not be due to any merging of stimulus features, but may rather be due to additional processing resources being required to maintain these items. These feature-specific effects could also reflect a misbinding of sequentially presented feature information, and interference between multiple features held in VWM (Gorgoraptis, Catalao, Bays, & Husain, 2011).

Why might there be a difference between colour and orientation stimuli? Orientation integration was disrupted by both memory items, whereas colour memory showed feature-specific effects. The simplest explanation for this is that the colour tasks were easier than the orientation tasks due to the ability to categorise and assign verbal labels to colour stimuli (while people may also be able to assign verbal labels to cardinal orientations, this would only produce four such labels (horizontal, vertical, 45° left or right), whereas many more colours could potentially be represented by their own label). It is arguable that in testing colour, we were not testing purely visual working memory, and that the report of colour integration items received support from both verbal working memory and long term memory resources (Olsson & Poom, 2005), or that colour objects are able to be verbally rehearsed to maintain those object representations in attention and thus VWM (Awh & Jonides, 2001). Any of these options could reduce the load on VWM for colour stimuli, and therefore reduce the efficacy of a memory item, which could in turn mean that less interference will occur in VWM if the colour stimulus is also being represented by supporting memory resources. While we did not explicitly control for task difficulty between colour and orientation tasks, we are also not directly comparing JNDs between the tasks, so the relative differences should not affect the overall results of the study.

4.3. Implications for perceptual stability

This study found that integration performance was depleted by the addition of a single memory item: this raises the question of how these findings may relate to a theory of integration as a useful mechanism supporting perceptual stability. So far in all of the studies on trans-saccadic feature integration, the primary task of the subjects was to respond to those features. It might be that integration is a very specific effect that occurs primarily when discerning specific features is task relevant. For an example, you may want to look at a particular object to better discern some fine detail on that object. Before making the saccade, attention would shift to that area (Deubel and Schneider, 1996, Kowler et al., 1995), and the pre-saccadic representation of the object would be actively maintained in VWM. Here, the need for actively reconciling the pre- and post-saccadic percepts would be important, as there would be a strong pre-saccadic representation of the object that you want to scrutinise in more detail. In this case, both your attention, and memory would also play an active role in ensuring that the pre-saccadic representation of the object is reconciled with the post-saccadic representation; this is consistent with both the current results, and our previous work showing that attention is necessary for integration (Stewart & Schütz, 2018). This account would also be consistent with our feature-specific interactions between memory and integration, and may suggest that feature-level effects are especially important. This would be in contrast to a scenario were one is just casually scanning a scene with no particular purpose, where there may not in fact be integration, or integration may occur to a lesser extent. Here, perception may follow either pre-saccadic predictions or post-saccadic vision more closely, as there would be no need to reconcile this finer object detail across these saccades. This would fit with theories of perceptual stability that predict that stability occurs via the allocation of attentional pointers to relevant areas in the visual field (Cavanagh et al., 2010, Mathot and Theeuwes, 2011), or that VWM resources can be flexibly allocated depending on saliency (Melcher & Piazza, 2011), or task demands (Bays et al., 2011), and may be an interesting avenue for future research.

5. Conclusion

This study has shown that visual working memory plays a role in trans-saccadic integration – completing a concurrent memory task reduces integration performance, suggesting that VWM resources are required for the integration of pre- and post- saccadic stimuli. Moreover, when completing a concurrent memory and integration task, memory performance is affected for feature-similar items.

Appendix A. Supplementary data

The following are the Supplementary data to this article: