Visuospatial Working Memory as a Fundamental Component of the Eye Movement System

Abstract

Humans make frequent movements of the eyes (saccades) to explore the visual environment. Here we argue that visuo-spatial working memory (VSWM) is a fundamental component of the eye movement system. Memory representations in VSWM are functionally integrated at all stages of orienting, from selection of the target, to maintenance of visual features across the saccade, to processes supporting the experience of perceptual continuity after the saccade, to the correction of gaze when the eyes fail to land on the intended object. VSWM is finely tuned to meet the challenges of active vision.

The human eye has a small region of high acuity at the center of vision, the fovea, that supports fine-grained perceptual processing of objects. We make frequent saccadic eye movements to orient this region to objects of interest in the world. Saccades introduce two problems for the visual system. The first is one of control. How are goal-relevant objects selected as the targets of eye movements over other objects in a scene? The second is one of continuity. Each saccade generates a brief perceptual disruption as the eyes rotate, and the visual information presented on the retina is displaced spatially. How does the eye movement system establish the correspondence between objects visible before and after the saccade to generate the experience of perceptual continuity?

Recent research indicates that two forms of working memory play a central role in solving these problems: visual working memory (VWM) and spatial working memory (SpWM), with the combined system termed visuo-spatial working memory (VSWM). VWM is a limited capacity system for the active representation of the visual appearance of relevant objects (Luck & Vogel, 2013; Ma, Husain, & Bays, 2014)². SpWM is a limited capacity system for the active representation of the locations of relevant objects (Awh & Jonides, 2001). Before a saccade, selection of the saccade goal is strongly guided by the current content of VWM, and a mandatory shift of spatial attention to the saccade target leads to the automatic encoding of the specific features of that object into VSWM. During the saccade, these representations are used to bridge the perceptual gap created by the saccade. After the saccade, VSWM is used to establish object and location continuity and to correct possible errors in saccade landing position. Thus, we argue that VSWM should be conceptualized as part of a closely integrated system for orienting gaze.

VWM and Oculomotor Control

To behave adaptively, we must direct our gaze in a goal-driven manner. For example, when cooking, one must fixate each of the ingredients and utensils as they become relevant (Land & Hayhoe, 2001). Recent developments indicate a central role for VWM in this type of eye-movement control (or oculomotor control), encapsulated in an experiment by Bahle, Matsukura, and Hollingworth (in press; Figure 1). Participants searched for a target object in a scene. Simultaneously, they maintained a secondary color in VWM for a memory test. Saccade target selection was guided by a representation of the search target, with the eyes directed very efficiently to that object. But it was also guided by the incidental content of VWM: a critical distractor in the scene was more likely to be fixated when it matched the secondary color than when it did not. First, this experiment illustrates that the strategic maintenance of a target representation in VWM introduces strong control over where the eyes are oriented. In more simplified displays, saccade target selection can be limited, almost exclusively, to objects matching VWM content (Beck, Hollingworth, & Luck, 2012). Second, the active maintenance of a representation in VWM is often sufficient to implement guidance; gaze is directed to memory-matching objects that are known to be irrelevant (Soto, Humphreys, & Heinke, 2006). Third, guidance by VWM tends to dominate other forms of guidance, particularly at the early stages of visual search. In Bahle et al., differential fixation of the critical distractor was observed from the very first saccade on the scene and was not influenced by whether or not the distractor appeared in a plausible location for the target, indicating that VWM-based guidance is implemented before guidance based on scene gist recognition (c.f. Wolfe, Võ, Evans, & Greene, 2011). Finally, guidance can be implemented by multiple items in VWM simultaneously: oculomotor selection was influenced both by a VWM representation of the search target and by a VWM representation of the secondary color. The finding of multiple-item guidance (see also Beck et al., 2012) contrasts with recent claims that only one item can be maintained in an active state in VWM and guide selection (Olivers, Peters, Houtkamp, & Roelfsema, 2011).

Figure 1.

A) Method in Experiment 1 of Bahle et al. (in press). Participants saw a target object and a label that cued the relevant object in the upcoming scene (i). Then they saw a color to be remembered for a memory test at the end of the trial (ii). Participants searched for the target object (iii) and reported the orientation of a small letter ‘F’ superimposed upon it. Finally, they completed a forced-choice, within-category color memory test (iv). The color of the memory item was manipulated so that it did or did not match the color of a critical distractor in the scene (pumpkin). Eye movements were recorded. B) Scan paths (lines represent saccades; circles represent fixations) for the 10 participants who saw this scene item in the mismatch condition (top panel) and the 10 participants who saw this item in the match condition (bottom panel). Note that gaze started at the center of the scene and was ultimately directed to the target object. However, the critical distractor was more likely to be fixated when it matched the secondary color in VWM. Across participants and scene items, mean probability of critical distractor fixation was .18 in the mismatch condition and .40 in the match condition. C) Method in Experiment 1 of Schut et al. (2017). Participants saw a memory array of 1, 2, or 3 common shapes (i) and remembered the width/height ratio of each (i.e., the extent to which the shape was stretched on the vertical or horizontal dimension). During the retention interval, they either did or did not complete a saccade task (ii) that required them to execute a saccade to an object that was briefly enlarged. Finally, the participants manipulated the width/height ratio of a test item until it matched the corresponding memory item (iii), allowing an estimate of memory precision. D). Memory precision as a function of the number of memory items and as a function of whether there was or was not an intervening saccade task. Note that the saccade task introduced a drop in memory precision approximately equivalent to the addition of one object to the VWM load.

In sum, what one happens to be representing in VWM has a substantial influence over where gaze is directed. One locus of this interaction appears to be relatively early within the visual processing stream. Hollingworth, Matsukura, and Luck (2013) examined the influence of VWM-match on rapidly generated, reflexive saccades to single targets. VWM influenced both the latency and the accuracy of saccades generated in less than 150 ms. A plausible mechanistic implementation of such effects was outlined in Schneegans, Spencer, Schöner, Hwang, and Hollingworth (2014). In this model, VWM maintenance involves sustained activation of sub-populations of neurons in sensory cortex. This activity interacts with the first, feedforward sweep of sensory input to increase the perceptual salience of items matching VWM content (Gayet et al., 2017) and thereby biasing the competition between objects for selection.

Presaccadic Encoding into VWM

When saccadic competition has been resolved and the eye movement is programmed, the visual system needs to generate a robust visual representation that can survive perceptual disruption and interference from postsaccadic sensory input. Because this transsaccadic representation depends on the VWM system (for an extensive review, see Irwin, 1992b), it tends to be limited to a subset of scene information, and there is a strong bias to select objects at the saccade target location (Currie, McConkie, Carlson-Radvansky, & Irwin, 2000). Specifically, visual attention shifts to the location of the impending saccade target (e.g., Deubel & Schneider, 1996), leading to the preferential encoding of visual information from that region into VSWM, both in terms of the probability of encoding (Irwin, 1992a) and the precision of the target representation relative to other remembered items (Bays & Husain, 2008).³ In addition to a close relationship between saccades and VWM encoding, saccade preparation also prioritizes the retention of objects already maintained in VWM (Hanning, Jonikaitis, Deubel, & Szinte, 2016; Ohl & Rolfs, 2017), with the selective retention of items that were originally encoded at the saccade target location.

Further, several recent studies have indicated that oculomotor selection effects on VWM encoding and maintenance are automatic and are specifically related to saccade preparation (Ohl & Rolfs, 2017; Schut, Van der Stoep, Postma, & Van der Stigchel, 2017; Shao et al., 2010; Tas, Luck, & Hollingworth, 2016). In Schut et al. (Figure 1), the precision of VWM for shapes was assessed either with or without an intervening eye movement task. The demand to execute a saccade introduced substantial interference with VWM maintenance. The drop in precision was approximately equivalent with the loss of one object’s worth of information, presumably caused by saccade target encoding into VWM. These selection effects are specific to VWM and to the situation in which a saccade must be executed to a visible object. Saccades produced no interference with WM for verbal stimuli in Schut et al. And, in a similar study by Tas et al. (2016), there was no interference with VWM if participants executed a saccade to empty space or if participants were required to shift attention covertly to a peripheral object, without executing a saccade. This latter finding indicates an important dissociation between covert and overt attention and is explained naturally by the idea that the functional relationship between attention and VWM encoding is produced by the demands of saccade execution. Covert shifts of attention do not introduce a perceptual disruption or shift in retinal input. It is only when a saccade has been programmed and will be executed that VWM encoding is required to bridge perceptual disruption and maintain object continuity across retinal displacement.

VSWM across Saccades Supports Perceptual Continuity

The role of VSWM in transsaccadic continuity has been central to accounts that posit a central role for a representation of the saccade target object (e.g., Currie et al., 2000; Deubel, Schneider, & Bridgeman, 1996; Irwin, McConkie, Carlson-Radvansky, & Currie, 1994). Under this view, transsaccadic object continuity and the experience of stability is accomplished by a saccade-target mapping operation, in which VWM for pre-saccadic target properties is compared with post-saccadic sensory input near the fovea. This mapping operation occurs in the context of a strong bias to assume that world has remained stable (Atsma, Maij, Koppen, Irwin, & Medendorp, 2016; Deubel et al., 1996).

What is the nature of the saccade target information functional in these processes? Both the locations and surface feature properties of objects are used to compute target correspondence: spatial displacement of the saccade target across the eye movement (Bridgeman, Hendry, & Stark, 1975) and significant changes in surface features (Tas, Moore, & Hollingworth, 2012) both interfere with the perception of a single, continuous object. In addition, the mapping operation is inherently predictive, both for location and for surface features. In the former case, the attended retinotopic locations of a few objects are updated to account for the spatial displacement (Boon, Belopolsky, & Theeuwes, 2016; Rolfs, Jonikaitis, Deubel, & Cavanagh, 2011), although attention may linger briefly at the old retinal location immediately after the saccade (Golomb, Pulido, Albrecht, Chun, & Mazer, 2010). This type of attentional remapping can be considered as dependent on the SpWM system given evidence indicating a close relationship between spatial attention and SpWM (Awh & Jonides, 2001). In the feature domain, Herwig and Schneider (2014) showed that the representation of the saccade target is influenced by the expected appearance of that object when the eyes land. Specifically, they trained participants to associate different spatial frequencies for the saccade target before and after the saccade. This association then came to bias perceptual experience of the target before the saccade, toward the expected postsaccadic spatial frequency.

Given that a representation of the saccade target is maintained across the saccade, to what extent is this integrated with perceptual information activated when the eyes land? Several recent studies have indicated some level of perceptual integration. For instance, when colors were shifted imperceptibly during a saccade (Oostwoud Wijdenes, Marshall, & Bays, 2015), participants tended to report a color that was between the pre- and postsaccadic values. Interestingly, the weights given to pre- and postsaccadic representations in the integrated representation appear to be influenced by sensory uncertainty (e.g., as introduced by acuity differences or the level of visual noise), with more weight given to the more reliable representation (Ganmor, Landy, & Simoncelli, 2015; Wolf & Schütz, 2015).

For such integration to play a functional role in the perception of object continuity, it needs to occur immediately after the saccade. To measure the time-course of integration, Fabius, Fracasso, and Van der Stigchel (2016) used a motion illusion (the High Phi illusion, Wexler, Glennerster, Cavanagh, Ito, & Seno, 2013), in which an annulus with a random texture rotates slowly, and is then replaced with several different textures (transient). With sufficient rotation duration, participants report the transient as a large rotational jump in the opposite direction. In Fabius et al., the texture rotated in the periphery and participants executed a saccade to it. Participants observed the illusion if the transient was presented as soon as the eyes landed, indicating that the presaccadically acquired information influenced perception immediately after the saccade.

VWM Supports Selection and Gaze Correction after the Saccade

Eye movements often fail to land on the saccade goal, and a corrective saccade is needed to orient gaze to the original target. The process of gaze correction serves as a microcosm of the processes discussed so far, illustrated in experiments by Hollingworth, Richard, and Luck (2008). Participants executed a saccade to a target disk in a circular array of discs that differed only by color. On some trials, the array rotated so that the eyes landed between the target and an adjacent distractor disk. This was meant to simulate the common situation in which the eyes miss the saccade target and there are several candidate objects near the landing position. Memory for the color of the target allowed participants to efficiently correct their gaze, and the effect of color match was stronger than the effect of relative proximity of the landing position to each of the two objects: feature correspondence dominated spatial correspondence. Moreover, a secondary VWM load impaired gaze correction, and the features for the secondary task interfered with correction when they were associated, postsaccadically, with the distractor (Hollingworth & Luck, 2009), implicating VWM in solving the gaze correction problem. Thus, after the primary saccade, the VWM representation of the saccade target is used to establish correspondence, and if there is no appropriate object at the fovea, this representation is used as a template to guide an extremely rapid visual search operation, supporting selection of the original saccade target. For such a mechanism to work, the presaccadic attentional shift should be determined by the planned saccade target location and not by the actual saccade landing position. Indeed, this appears to be the case (Deubel & Schneider, 1996). For example, when there is strong competition for oculomotor selection and the eyes generally land between a target and a distractor, spatial attention is nevertheless allocated to the object that is the goal of the saccade (Van der Stigchel & de Vries, 2015). This distribution of attention would support the encoding of target properties for saccades that will not ultimately land on that object.

Conclusion

VSWM is functionally integrated with oculomotor mechanisms at all stages of the orienting process (see Figure 2).

Figure 2.

Proposed function of VWM during eye movement orienting (note that we do not depict SpWM representations). A) During the task of baking, a lemon is needed, and the person forms a VWM representation of a canonical lemon as the search target to guide selection. B) Spatial attention (orange circle) shifts to an object that is a relatively close match to the search target representation, the butter is selected as the target of the next saccade, and the features of that object are encoded into VWM. C) A saccade is executed but fails to land on the target. The VWM representation of the saccade target is used to establish object correspondence and to localize the target, leading to a rapid corrective saccade. D) Detailed inspection of the fixated object reveals that it is not the target, and attention shifts to another object that is also a relatively close match to the search target representation, the features of the next saccade target are encoded into VWM, and so on.

Before the saccade:

• The saccade goal is selected, to a significant degree, by the current content of VWM, with attention biased toward memory-matching objects;

• The presaccadic shift of spatial attention to the target leads to the encoding of target features into VWM in a manner that predicts the postsaccadic appearance of that object;

• Attentional pointers in SpWM are predictively updated to the future retinal locations to maintain attention on goal-relevant objects.

During the saccade:

• The content of VSWM is used to bridge the perceptual gap created by the saccade in a format that is resistant to masking from postsaccadic sensory input.

After the saccade:

• Surface feature and position information in VWM and SpWM are used to establish the correspondence between a few relevant objects that were visible before and after the saccade, particularly for the saccade target.

• In addition, the VWM representation of the target can be integrated with new perceptual input to form a composite representation.

• If the eyes fail to land on the target, the VWM representation is used as a search template, guiding a corrective saccade to the target in manner similar to the original selection of that object.

We argue that many of the basic properties of VSWM can be understood as arising from the optimization of oculomotor control. For example, the relationship between spatial attention and VWM encoding can be understood as reflecting the demand to bridge the perceptual gap introduced by saccades. The close relationship between visual attention and SpWM can be understood arising from a need to maintain attention on relevant locations across saccadic disruption and delay. The guidance of attention by multiple VWM representations can be understood as reflecting the simultaneous demand to select the ultimate target and to establish correspondence and correct gaze for each of the individual objects fixated during search (which may not always be a precise match to the target). In sum, the oculomotor system does not simply exploit a general WM system; VSWM is finely tuned to meet the demands of active vision.