Visual working memory modulates within-object metrics of saccade landing position

Abstract

In two experiments, we examined the influence of visual working memory (VWM) on oculomotor selection, testing whether the landing positions of rapidly generated saccades are biased toward the region of an object that matches a feature held in VWM. Participants executed a saccade to the center of a single saccade target, divided into two colored regions and presented on the horizontal midline. Concurrently, participants maintained a color in VWM for an unrelated memory task. This color either matched one of the two regions or neither of the regions. Relative to the no-match baseline, the landing positions of rapidly generated saccades (mean latency < 150 ms) were biased toward the region that matched the remembered color. The results are consistent with the hypothesis that VWM modulates early, spatially organized sensory representations to bias selection toward locations with features that match VWM content. In addition, the results demonstrate that saccades to spatially extended objects are sensitive to within-object differences in salience.

Theories of attention typically posit that features of task-relevant objects are maintained in VWM and that this maintenance biases selection in favor of objects in the visual field that share those features.^1–6 Recent evidence suggests that this VWM template interacts with selection relatively early within the visual processing of a scene to modulate rapid oculomotor orienting. In a study by Hollingworth, Matsukura, and Luck,⁷ participants remembered a sample color and executed a saccade to a single, sudden-onset disk drawn in a color that did or did not match the memory sample. In the memory-match condition, orienting saccades landed closer to the center of the saccade target and were generated more rapidly compared with saccades in the no-match condition. These effects were observed in the absence of stimulus competition and despite the fact that the remembered color did not predict the saccade target color and that the saccade target color did not predict the correct response on the memory test at the end of the trial. Mean saccade latencies in this study were well under 150 ms, consistent with the hypothesis that VWM modulates the initial sensory response following stimulus onset.

VWM has even more substantial effects on oculomotor selection in competitive contexts. When a distractor object was added to the saccade target display by Hollingworth, Matsukura, and Luck,⁷ memory match placed strong constraints on the object to which gaze was directed, with a memory-matching distractor attracting a substantial proportion of saccades, despite instructions to avoid fixating distractors. Moreover, memory match modulates the landing position of averaging saccades in the global-effect paradigm, in which participants execute a saccade to one of two closely spaced objects. In Hollingworth, Matsukura and Luck⁸ (for complementary results, see Ref. 9), participants executed a saccade to the outer object in a closely spaced pair of objects appearing on the horizontal midline. When neither object matched the color category of a secondary VWM task, saccades tended to land at the midpoint between the two objects, the global effect.^10–12 When either the target or the distractor matched the remembered color category, the distribution of landing position was shifted systematically toward the matching object. Similar interactions between remembered and visible stimuli have been observed in the effects of spatial working memory on saccade metrics.¹³ As in Hollingworth, Matsukura and Luck,⁷ VWM modulated the metrics of saccades with latencies near the lower limit of human capabilities. Together, these results disconfirm earlier claims that very rapidly generated saccades are computed solely on the basis of bottom-up sensory information.^14,15 The perceptual salience of an object is a joint function of its physical attributes and the match between those attributes and the content of VWM.

If VWM modulates early saccade target selection, what are the representational units over which this modulation occurs? In the studies reviewed above, modulation could have occurred through the influence of VWM on the relative contribution of particular spatial locations to the calculation of the saccade vector. Alternatively, modulation could have occurred through the influence of VWM on selection at the object level: for example, by influencing the salience of a single saccade target object or by influencing the competition between discrete object representations for selection. Because the targets in these experiments were solid, colored disks, space and objecthood were perfectly confounded. An object-based locus of modulation would be consistent with claims that top-down factors influence competition maximally at higher levels of the visual system (for a review, see Ref. 16). However, it remains possible that effects of VWM influence earlier processes, before visual input is parsed into object regions.

This assumption was instantiated in a recent computational model by Schneegans and colleagues.⁵ In this approach, VWM and feature-based attention interact directly with a low-level, retinotopic sensory representation of feature values over space. This contrasts with models of saccade target selection that do not include feature-based guidance^17–21 and with models claiming that strategic guidance is implemented relatively late in the visual system via direct modulation of priority.²² In the Schneegans et al. model, when a color is held in VWM, that maintenance modulates the initial sensory response, facilitating the perceptual registration of memory-matching colors across the visual field. A stronger sensory representation at memory-matching locations then biases eye movement behavior in systems instantiating spatial attention and saccade targeting. Specifically, the presence of a memory-matching feature in the display increases the weight of that spatial location in the computation of the saccade vector (averaged across all active locations in the motor field). Thus, saccade landing position should be biased toward memory-matching features in the display in a manner that is, at least to some extent, independent of the perceptual objects to which those features belong.

In the present study, we had participants execute a saccade to the center of a single object composed of two colored regions (Fig. 1). The saccade target appeared either to the left or right of fixation, centered on the horizontal midline. This saccade task was performed during the retention interval of a color VWM task, and the relationship between the remembered color and the colors of the saccade target regions was manipulated. On a subset of trials, either the top or bottom region of the saccade target matched the category of the color retained in memory; we examined whether the landing positions of rapidly generated saccades would deviate toward the matching region.

Figure 1.

Sequence of events on a trial of Experiment 1. Participants first fixated a central cross. A color memory disk was presented for 300 ms (A), followed by an ISI of 700 ms (B). The saccade target stimulus was then displayed, and participants attempted to execute a saccade tothe center of the object (C). After the target was fixated, there was a 200-ms delay, followed bythe memory test consisting of two test disks (D), the original color and a foil drawn from thesame color category. This example trial shows a saccade target stimulus with a memory-matching region on the top.

Several aspects of the design ensured that any observed deviation of saccade landing position could not have been caused by VWM-based modulation at the level of discrete object representations. First, despite the differences in region color, the saccade target stimulus had strong cues for perception as a single object: The two regions aligned spatially and were enclosed by a shared black contour. Second, there was only one object in the display, eliminating any need to resolve competition for selection as the saccade target. In addition, the paradigm was designed so that any deviation of landing position toward the memory-matching region could not have been caused by strategic orienting to that region. First, a color match was irrelevant to the explicit goal of the saccade task, which was to execute an eye movement to the center of the entire object. Second, the match between the remembered color and the relevant region was sometimes exact (the same color) and sometimes inexact (a different color from the same category). In the latter case, the saccade target color became the foil in the two-alternative forced-choice memory test at the end of the trial. Across the experiment, the color of a matching region therefore did not predict the correct response on the memory test, precluding any benefit from strategically attending to matching regions.

In addition to probing the locus of VWM-based modulation in saccade target selection, the present study also informs our understanding of the visual cues governing the landing position of saccades directed to spatially extended objects. One prominent account holds that the bounding shape of an object is the critical cue governing within-object landing position and that the selection of saccade target location is not influenced by the internal features of the object.^23,24 Melcher and Kowler²³ found that saccades tended to land at an object's center of gravity defined by its global shape, but landing position was not significantly influenced by the distribution of luminance intensity within the object. The generality of this account is limited, to some extent, by the fact that participants were instructed explicitly to execute a saccade to the center of the target shape and were under no time pressure, potentially providing them sufficient time to intentionally minimize the influence of within-object features. In the present study, we were able to test the effect of region salience within the object (as a function of the memory-matching region) on saccade landing position for the type of rapid, reflexive saccades that typically would be induced by abrupt onset stimuli.

Experiment 1

Participants maintained a color in memory as they executed a saccade to an object that appeared on the horizontal midline (Fig. 1). First, a color disk was presented, to be remembered for a within-category memory test at the end of the trial. Next, a rectangular saccade target object appeared, divided into two colored regions. Participants were instructed to generate a saccade as quickly as possible to the center of the target. The match between the sample color and the colors of the saccade target regions was manipulated. In the no-match condition, neither region matched the category of the memory color, providing a baseline measure of saccade metrics and dynamics. In the memory-match condition, one of the two regions matched the memory color category, and the other did not. The location of the matching region (top/bottom) was manipulated.

Method

Participants

In both experiments, participants were between 18 and 30 years of age, reported 20/20 uncorrected vision, and received course credit or pay for their participation. Sixteen participants completed Experiment 1.

Stimuli

The stimuli for the memory and saccade tasks appeared against a gray background with a central white fixation cross subtending 0.3°. The memory sample display (Fig. 1A) was a 2.28°-diameter colored disk at the center of the screen, with color category selected randomly from red, green, and blue. Within a color category, the color value was selected randomly from four similar colors (for precise color values, see Ref. 7). In the memory-test display (Fig. 1D), two color disks were presented 3.26° to the left and right of the central fixation cross. One color was the same as the sample color (correct alternative), and the other was drawn randomly from the remaining three colors in that category (foil). The positions of the two alternatives were determined randomly. This within-category discrimination task minimized the role of verbal encoding by ensuring that the test could not be performed successfully by retention of a simple verbal label (e.g., “red”).

The saccade-task display (Fig. 1C) contained one rectangular object consisting of two colored regions, with one region above and one region below the horizontal midline. The entire object subtended 0.98° horizontally and 2.61° vertically. Each colored region subtended 0.91° horizontally and 1.24° vertically. There was a 1-pixel (0.03°) black border around the object and a 2-pixel (0.07°) black border separating the two regions. The inclusion of the black regions was designed to give the impression of two colored surfaces on a single, black object. The target was displayed either to the left or right of central fixation, and eccentricity was selected randomly within a range of 4.6–7.0°.

When one region matched the memory category, that match to the remembered color was either exact or inexact (a different color selected randomly from the same category). In the latter case, this color became the foil color in the subsequent memory test. Because a category-matching color in the saccade display was equally likely to be the correct color or the foil color in the memory test, participants could derive no benefit from strategically attending to the colors in the saccade display.²⁵

Apparatus

Stimuli were displayed on a 17-in CRT monitor with a refresh rate of 120 Hz. The right eye was monitored by an SR Research EyeLink 1000 eye tracker sampling at 1000 Hz. A chin and forehead rest minimized head movement and maintained a viewing distance of 70 cm. Manual responses were collected by a serial button box. Screen events, eye events, and manual responses were coordinated by E-prime software.²⁶

Design and procedure

Upon arriving for the experiment session, participants provided informed consent and received task instructions. The eye tracker was calibrated at the beginning of the session, and it was recalibrated during the experiment if the position estimates deviated from the calibration points by more than approximately 0.75°.

The experimenter initiated the trial as the participant maintained central fixation. After a delay of 400 ms, the memory sample disk was presented for 300 ms (Fig. 1A), followed by a blank (fixation cross only) delay of 700 ms (Fig. 1B). Then, the saccade target object was presented (Fig. 1C). Participants were instructed to move their eyes “as quickly as possible to look directly at the center of the rectangle.” When a fixation was detected in the target region, the target display remained visible for an additional 200 ms. It was then replaced with the memory test display (Fig. 1D). Participants pressed one of two buttons to indicate whether the left or right disk was exactly the same color as the sample presented at the beginning of the trial. Button response terminated the trial. Incorrect responses on the memory test were followed by the word “incorrect” presented in red at the center of the screen for 500 ms. The next trial was initiated when the participant had returned gaze to central fixation.

Participants completed a practice session of 12 trials, followed by two experiment sessions of 192 trials each. Half of the trials had one memory-matching region (memory match), and half had no memory-matching region (no match). The memory-match trials were divided evenly between exact and inexact match and were also divided evenly between trials with the matching region on the top or on the bottom. Finally, half of the trials presented the target in the left hemifield and half in the right hemifield. Trials from the different conditions were randomly intermixed.

Data analysis

A saccade was defined as a change in the position estimate consistent with an eye movement velocity of >30°/s or acceleration >8000°/s². Trials were eliminated from the analysis if the participant was not fixating within 1° of the center cross when the target stimulus appeared (10.8% of trials), if saccade latency was greater than 400 ms or less than 60 ms (5.5% of remaining trials), or if the first saccade did not land within 2° of the target center (7.4% of remaining trials). A total of 19.1% of trials was eliminated. Elimination of these trials did not alter the pattern of results.

The effects of memory match on saccades were not reliably influenced by whether the match was exact or inexact, and this factor was collapsed. In addition, saccade metrics and dynamics in the match trials did not significantly vary as a function of subsequent accuracy on the color memory task, so all trials were included.

Results

Landing position

Of primary interest was the landing position of the first saccade following the onset of the saccade-task stimuli. Landing position was coded relative to the center of the target object. On the horizontal dimension, landing positions short of the target center were assigned negative values and landing positions beyond the target center positive values. On the vertical dimension, landing positions above the target center were assigned positive values and below target center negative values. Mean landing position data are reported in Figure 2. There was a general tendency for hypometric saccades, consistent with the general finding that saccades tend to undershoot the target.²⁷ In addition, there was a tendency for saccades to land slightly below the center of the object, as indicated by the mean landing position of saccades in the baseline, no-match condition. Critically, saccade landing position in the memory-match condition was influenced by the location of the matching region. Mean landing position when the top region matched the remembered color fell above the no-match baseline position, and this relationship was reversed when the bottom region matched the remembered color.

Figure 2.

Mean landing position error (relative to the target center) as a function of memorymatch and match location for Experiment 1. The main graph plots mean landing positionsuperimposed over the saccade target stimulus. The inset graph expands the outlined region ofthe main graph. Error bars are standard errors of the means.

To generate a measure of vertical deviation toward the memory-matching region, mean landing position for the memory-match trials was conditionalized on each participant's mean landing position for no-match trials. Deviations from baseline toward the memory-matching region were assigned positive values and deviations away assigned negative values. Overall, there was a mean vertical deviation toward the memory matching region of 0.07 °, which differed reliably from zero (t(15) = 5.06, P < 0.001). Although the absolute magnitude of the bias was relatively small, it was observed consistently: 15 of the 16 participants exhibited a landing-position bias toward the memory-matching region.

Latency

Consistent with previous reports,^7,8 saccade latency was reduced when the target stimulus contained a region with a color that matched the content of VWM. Mean saccade latency was 154 ms in the no-match condition and 148 ms in the memory-match condition (t(15) = 3.00, P = 0.009).

Color memory

Overall, mean accuracy on the memory test was 78.1%, and no participant performed below 67% correct. Accuracy differed as a function of the match between the remembered color and the saccade task colors, consistent with previous findings.^7,8 Mean accuracy was 78.2% for the no-match condition, 81.2% for the exact-match condition, and 75.5% for the inexact-match condition. The difference between exact and inexact match reached significance (t(15) = 3.04, P = 0.008). Recall that an inexact match always became the foil color in the memory test, so this pattern indicates that, in some trials, participants reported the saccade-task color rather than the memory-task color. This is not particularly surprising, because saccade target properties are encoded into VWM before each saccade.^28,29 potentially creating ambiguity regarding which of the two colors in VWM was to be reported.

Discussion

In Experiment 1, the landing positions of rapid saccades to a single target object were biased by the location of a memory-matching region within the object. This effect was observed despite instructions to generate a saccade to the center of the object and despite the fact that the object center was always on the horizontal midline, requiring a simple horizontal saccade to reach it. This latter feature of the design may have limited the absolute magnitude of the effect, which was small (less than 0.1 °), but the bias was observed consistently across participants. The results suggest that VWM match modulates the weighting of particular locations in the computation of the saccade vector and that this modulation occurs, at least to some extent, independently of the object containing those features. Moreover, the result show that within-object landing positions are not necessarily determined exclusively by the bounding contours of the object and are influenced by the relative salience of internal regions.

Experiment 2

In Experiment 2, we replicated Experiment 1 and extended the method to a different target stimulus. The target was an outlined disk divided into two semi-disks, one above and one below the horizontal midline (Fig. 3). The shared circular contour and black bounding region gave an extremely strong impression of a single-object stimulus. However, the spatial extent of the colored regions, above and below the midline, was reduced, and thus we expected to replicate the results of Experiment 1 but with a reduction in the absolute magnitude of the biasing effect.

Figure 3.

Sequence of events on a trial of Experiment 2. The method was the same as inExperiment 1, except that the saccade target (C) was a disk rather than a rectangle and thememory stimuli (A and D) were squares rather than disks. This example trial shows a saccadetarget stimulus with a memory-matching region on the bottom.

Methods

Participants

Twelve new participants completed the experiment.

Stimuli

The stimuli were the same as in Experiment 1, with the following exceptions. The saccade target was a disk divided into upper and lower semi-disk regions (Fig. 3C). The diameter of the disk was 1.3°. In addition, the memory stimuli were changed from disks to squares (2.3° × 2.3°). The manipulated relationship between the memory color and the colors of the two saccade target regions was the same as in Experiment 1.

Apparatus, design, and procedure

The apparatus, design, and procedure were the same as in Experiment 1.

Data analysis

Trials were eliminated from the analysis if the participant was not fixating within 1° of the center cross when the target stimulus appeared (7.1% of trials), if saccade latency was greater than 400 ms or less than 60 ms (3.4% of remaining trials), or if the first saccade did not land within 2° of the target center (6.0% of remaining trials). A total of 13.2% of trials were eliminated. Elimination of these trials did not alter the pattern of results.

The effects of match were not reliably influenced by whether the match was exact or inexact, and this factor was collapsed. In addition, saccade metrics and dynamics in the match trials did not significantly vary as a function of subsequent accuracy on the color-memory task, so all trials were included.

Results and discussion

The results of Experiment 2 replicated the principal findings in Experiment 1.

Landing position

Mean landing-position data are reported in Figure 4. As in Experiment 1, there was a general tendency in the no-match condition for hypometric saccades that landed slightly below the center of the object. When one region matched memory, landing position deviated from the baseline position toward the matching region.

Figure 4.

Mean landing-position error (relative to the target center) as a function of memorymatch and match location for Experiment 2. The main graph plots mean landing positionsuperimposed over the saccade target stimulus. The inset graph expands the outlined region ofthe main graph. Error bars are standard errors of the means.

Each participant's mean landing position for the memory-match trials was again conditionalized on the participant's mean landing position for no-match trials, with deviations from baseline toward the memory-matching region assigned positive values and deviations away assigned negative values. Overall, there was a mean vertical deviation toward the memory-matching region of 0.03°, which differed reliably from zero (t(11) = 4.02, P = 0.002). Eleven of the 12 participants exhibited a landing-position bias toward the memory-matching region.

Latency

Replicating Experiment 1, saccade latency was reduced when the target stimulus contained a feature that matched the content of VWM. Mean saccade latency was 145 ms in the no-match condition and 140 ms in the memory-match condition (t(11) = 2.92, P = 0.01).

Color memory

As in Experiment 1, mean accuracy on the memory test was 78.1%, and no participant performed below 65% correct. Mean accuracy was 78.1% for the no-match condition, 80.7% for the exact match condition, and 75.8% for the inexact match condition. The difference between exact and inexact match was reliable (t(15) = 2.26, P = 0.045).

General discussion

In both experiments, single objects were presented as saccade targets, divided into two colored regions. The landing positions of saccades directed to the centers of the objects were systematically biased toward the within-object region matching the color of a concurrent VWM load. The results suggest that VWM modulation of selection is not necessarily limited to processes operating over discrete object representations. Consistent with the assumptions of the model of Schneegans, Spencer, Schöner, Hwang, and Hollingworth,⁵ VWM match appears to modulate the relative weighting of individual locations in the computation of the saccade vector. In the model, the locus of this interaction is assumed to occur at relatively early, retinotopically organized regions of visual cortex. Specifically, the model consists of a low-level sensory field of feature values across space that is connected to two separate processing streams: a spatial pathway implementing spatial attention and saccade generation and a surface-feature pathway implementing color working memory and feature-based attention. Each representational system is implemented as a dynamic neural field, in which peaks of activation along a metrically organized dimension (such as locations in space or values in hue space) serve as representational units. When a particular color value is held in VWM, this activation feeds back broadly to the low-level sensory field, facilitating the initial sensory registration of stimuli with that color value across the visual field. The VWM-modulated sensory response drives activity in the spatial attention and saccade generation fields, biasing the central tendency of their activation distributions toward the memory-matching region and thus generating a saccade with a landing position that is biased toward that location.

Of course, we do not claim that saccade target selection is insensitive to object-level cues. There exist several demonstrations that object structure plays an important role in selection operations^{23,24,30–32} (but see Ref. 33). These studies have focused on the role of external contour in governing the precise landing positions of saccades to spatially extended objects. In particular, Melcher and Kowler²³ proposed that landing position is established by pooling across all of the locations within the bounding contour, producing saccades that tend to land at the object's center of gravity. In addition, they claimed that this operation does not consult the perceptual features at each of the pooled locations, so that selection is insensitive to within-object feature differences, such as differences in luminance. The current data demonstrate quite clearly that, contrary to this latter proposal, saccades to spatially extended objects are sensitive to the relative salience of within-object regions. The results are particularly strong given that the differences in salience were established not by physical discontinuity but by a color match between one region and the content of VWM. It seems likely that effects larger than those in the current study could be obtained by manipulating the physical luminance of within-object regions. In the study by Melcher and Kowler,²³ the effect of within-object features may have been minimized because of the design and demands of their task. Participants were instructed to execute a saccade to “the target as a whole,” and they were placed under no time constraint (saccade latencies typically exceeded 500 ms). Thus, participants may have intentionally discounted within-object features when preparing the saccade. In contrast, the rapidly generated saccades in the present study, reflecting a natural orienting response to an abrupt onset stimulus, showed clear effects of within-object feature salience. In general, it is likely that the precise landing position of saccades to spatially extended objects is determined both by global shape and by local, internal feature differences.