Competition in saccade target selection reveals attentional guidance by simultaneously active working memory representations

Abstract

The content of visual working memory (VWM) guides attention, but whether this interaction is limited to a single VWM representation or functional for multiple VWM representations is under debate. To test this issue, we developed a gaze-contingent search paradigm to directly manipulate selection history and examine the competition between multiple cue-matching saccade target objects. Participants first saw a dual-color cue followed by two pairs of colored objects presented sequentially. For each pair, participants selectively fixated an object that matched one of the cued colors. Critically, for the second pair, the cued color from the first pair was presented either with a new distractor color or with the second cued color. In the latter case, if two cued colors in VWM interact with selection simultaneously, we expected the second cued color object to generate substantial competition for selection, even though the first cued color was used to guide attention in the immediately previous pair. Indeed, in the second pair, selection probability of the first cued color was substantially reduced in the presence of the second cued color. This competition between cue-matching objects provides strong evidence that both VWM representations interacted simultaneously with selection.

Visually-guided behavior requires that attention is directed strategically to goal-relevant objects. Most theories of attention implement strategic guidance by means of a template representation in visual working memory (VWM). Indeed, attention is directed toward objects that match VWM content (e.g., Wolfe, Horowitz, Kenner, Hyle, & Vasan, 2004), even when doing so is counterproductive (e.g., Soto, Heinke, Humphreys, & Blanco, 2005). Although this basic relationship is well established, the architecture of interaction between VWM and attentional selection is currently under debate. Olivers, Peters, Houtkamp, and Roelfsema (2011) proposed that only one item in VWM can guide attention at any given time, a single-item template hypothesis (SIT). In this view, although VWM can maintain multiple items in prefrontal regions, only one of these items is able to interact with sensory processing to bias selection. In contrast, we have proposed that multiple items in VWM can guide attention simultaneously (Beck, Hollingworth, & Luck, 2012), a multiple-item template hypothesis (MIT). Several different items can be represented via sustained activity in visual-sensory cortex (Emrich, Riggall, Larocque, & Postle, 2013), so it is plausible that multiple VWM representations will interact with sensory processing to bias selection simultaneously.

Early evidence supporting the SIT came from attention capture paradigms (Downing & Dodds, 2004; Houtkamp & Roelfsema, 2006). A remembered item often failed to capture attention when included as a distractor during search for a different target, suggesting that only the current target template representation influenced selection. In a related method using a static search target, attention was captured by a memory-matching distractor when a single color was held in VWM but not when multiple colors were held in VWM (van Moorselaar, Theeuwes, & Olivers, 2014), again indicating a limit on the number of items interacting with selection. However, Hollingworth and Beck (2016) found reliable memory-based capture when more than one item was maintained in VWM, and capture magnitude scaled with the number of matching distractors in the display, consistent with the MIT.

Capture paradigms test whether non-target items in VWM automatically influence selection. However, they do not test whether participants can strategically maintain multiple items in the template state, the critical evidence needed to distinguish the SIT and MIT. This latter test requires that participants are asked to search on the basis of multiple target representations. In Beck et al. (2012), participants searched for items matching two different target colors. They were instructed to select items matching the two colors either sequentially or simultaneously. The sequential instruction led to long sequences of selection of a particular color and a saccade latency switch cost when switching between colors, consistent with template reconfiguration. However, the simultaneous instruction led to frequent switches between the two target colors. Critically, there was no switch cost, suggesting that both colors were maintained in an “active” state that guided selection simultaneously.

In the present study, we implemented a strong converging test, contrasting novel predictions of the two theories for multi-target search. Consider the situation in which a participant must select between two objects that each match one of two cued target colors. If both target colors are maintained in an active template state in VWM, as held by the MIT, the two objects should both act as strong candidates for selection, since they should both receive top-down prioritization. In contrast, the SIT predicts minimal competition in this situation. If the template is limited to one active VWM representation, then top-down prioritization will be applied to only one of the two cue-matching items. Selection of the item matching the single active template color should be efficient: specifically, no less efficient than selection when only one of the two items matches a cued color.

We tested these predictions in a new sequential search task that allowed us to control the history of selection within a trial. Each trial began with a dual-color cue (e.g., red and blue) followed by two pairs of colored objects presented sequentially (Figure 1). Participants fixated one item in each pair, with the instruction to always saccade to a cue-matching object. The first pair contained one cue-matching object (e.g., red) and one distractor object (e.g., yellow). The key data came from selection in the second pair. In the same condition, the cue-matching color in the first pair was repeated, along with a novel distractor color. In the switch condition, the other cue-matching color was presented (e.g., blue), along with a novel distractor color. Finally, in the critical both condition, the objects in the second pair were both cue-matching but different colors (e.g., red and blue), and participants could select either object.

Figure 1.

Example sequence of trial events for Experiment 1A and 1B. Each trial began with a cue stimulus (100ms) presented as a mini checkerboard (1.34×1.34 dva), with two squares (0.63×0.63 dva) for each of the two colors. After a 900-ms delay, two disks (0.67 dva) appeared simultaneously (first pair): one cue-matching and one distractor. Once participants fixated the cue-matching object, the distractor disappeared, and after a 200-ms delay, two new objects appeared simultaneously (second pair). The second pair could contain a same-color cue-matching object with a new distractor (“same”), a new cue-matching object with a new distractor (“switch”), or two objects that each matched a different cue color (“both”). These three conditions were equally probable. Once participants fixated the target in the second pair, the other second-pair object and first-pair target disappeared. Each object had a central vertical or horizontal line (0.04×0.17 dva; light grey, like the background), selected randomly, and participants indicated whether the lines in the cue-matching objects had the same or different orientations. When both objects in the second pair were cue-matching, the line orientation in both objects was the same. In Experiment 1A, the color of the cue-matching object in the first pair changed to grey during the saccade to it, so that there was no direct perceptual match with an object in the second pair. Experiment 1B was the same except that the objects retained their colors until they offset.

Under the SIT, after successful fixation of the cue-matching object in the first pair, the color of that object should be in the “active” state; it had just been used for selection. Thus, in the second pair, this first-cued-color should be selected again efficiently, and the selection process should be similar in the same and both conditions. In particular, the second-cued-color in the both condition should be in an “accessory” state that does not interact with selection and should produce competition no greater than a novel color distractor in the same condition. Under the MIT, however, each object in the both condition should act as strong candidate for selection, as both match a template color, increasing the probability that gaze is directed to the second-cued-color. In addition to this main test, we sought to replicate the switch cost results obtained by Beck et al. (2012). Because both colors should be maintained in a state that guides selection, the MIT predicts minimal switch costs when comparing the same condition with the switch condition. However, the SIT predicts a significant cost on switch trials, as the template must be reconfigured.

Method

Participants

Thirty-two University of Iowa students participated for course credit: 16 in each experiment.

Stimuli, Apparatus, and Procedure

Stimuli were presented on an LCD monitor (100Hz) at a distance of 77cm. Eye position was recorded at 1000Hz using an Eyelink1000 eyetracker. Saccades were defined using a combined velocity (>30°/s) and acceleration (>9500°/s²) threshold. Fixation on an object was defined as twelve consecutive samples within the surrounding interest area (2.01 degrees visual angle, hereafter dva).

The procedure is illustrated in Figure 1. The cued colors varied from trial to trial, ensuring that cue representation depended on VWM. Each of the four colors (red, yellow, green, blue, see supplementary materials for CIE values) appeared equally often in the cue. Each pair of objects appeared 4-6 dva from the previous fixation positon and were 40° apart. The second pair was presented within the range of 90°-270°, if 0° represents the trajectory of the previous saccade, so that saccades to the second pair were always progressive. If a distractor was fixated in either pair, the trial terminated with an error message. The secondary line orientation task was included to replicate the demands of a visual search task, requiring discrimination of the properties of each object.

In Experiment 1A, the color of the cue-matching object in the first pair changed to dark grey during the saccade so that there was no direct perceptual match with an object in the second pair. Experiment 1B was the same except that the cue-matching object in the first pair retained its color until it was removed upon fixation of the target in the second pair.

The session began with 20 practice trials on which participants simply fixated cue-matching objects. Then they completed a second practice block (24 trials) implementing the full design. Participants completed 10 blocks of 46 experimental trials. In total, there were 168 trials in each of the three conditions (same, switch, both), randomly intermixed across the practice and experimental blocks.

Results

Participants were excluded for manual response accuracy less than 75% in one or more conditions (Exp1A: N=4; Exp1B: N=3) yielding 12 participants in Experiment 1A and 13 participants in 1B. Accuracy for remaining participants was high (Exp1A: M=93%; Exp1B: M=90%) with no meaningful differences between trial types.¹ The primary measure was selection probability, defined as the first object fixated after the onset of a pair.² For the first pair, selection probability for the cue-matching object was high (Exp1A: M=82.6%; Exp1B: M=82.0%). This is similar to a comparable task in which there was only one template color (76.9%, unpublished data), indicating that participants in the present study efficiently used the dual-color cue to guide selection.

Same versus Both Trials: Evaluation of Competition between Template Colors

The key results concerned selection probability in the second pair (Figure 2). For same trials, participants frequently selected the first-cued-color again (Exp1A: M=76.5%; Exp1B: M=75.1%). The key question was whether, on both trials, selection probability of the first-cued-color would be similarly high (consistent with the SIT), or whether competition from a simultaneously active, second-cued-color would reduce that probability (consistent with the MIT). Not only was the probability of first-cued-color selection reduced from same to both trials [Exp1A: t(11)=7.54, p<.001, η_p²=.84; Exp1B: t(12)=4.94, p<.001, η_p²=.67], selection in the both condition was roughly equivalent between the two cue-matching colors. Specifically, selection probability for the first-cued-color did not significantly differ from 50% in either experiment (both ps>.46). This indicates that both colors were maintained in a similar state vis-à-vis attentional guidance.³

Figure 2.

Probability of selecting the different types of objects (Cued1: same cue-matching color used in the first pair; Cued2: cue-matching color not used in the first pair; Dist2: novel distractor color) presented in the second pair split by trial type (Switch, Same, or Both). A) Selection probability results from Experiment 1A. B) Selection probability results from Experiment 1B. Error bars indicate within-subjects 95% confidence intervals (Morey, 2008).

A possible alternative explanation for selection in the both condition is that, on some trials, participants implemented a strategy of purposefully switching the active template color after selecting the target in the first pair. Although possible, we think this alternative is unlikely. First, it would require that participants exerted effort to switch template colors despite the absence of any possible performance benefit, since the second pair contained the first-cued-color on two-thirds of trials. Moreover, the second pair appeared only 200 ms after fixation of the target in the first pair. Thus, this alternative would require an extremely rapid and precisely timed strategic switch, with no perceptual support in the display (particularly when the participants continued to fixate the first-cued color in Exp 1B) and with no prospect for a benefit in performance.

We also examined saccade latency for same and both trials. For latency analyses, we excluded saccades with latencies <90ms and >600ms (Exp1A: 2.2%; Exp1B: 1.4%) and restricted the analysis to the initial eye movement after the second pair appeared, using only saccades that landed within the interest area of the cue-matching object (Exp1A: 63.2% retained; Exp1B: 62.6% retained). Saccades to either cue-matching object on both trials (Exp1A: M=175ms; Exp1B: M=155ms) were slightly faster than saccades to the cue-matching object on same trials (Exp1A: M=182ms; Exp1B: M=157ms). These differences were statistically reliable [Exp1A: t(11)=2.92, p=.01, η_p²=.44; Exp1B: t(12)=2.26, p=.04, η_p²=.30], but numerically very small. Note that the slightly faster latencies on both trials is likely to have been caused by the fact that participants were free to saccade to either object. In cases where competition leads to increased saccade latency (e.g., Hollingworth, Matsukura, & Luck, 2013), there is a task-defined target and distractor; competition from the distractor must be suppressed in order to reliably select the target. In the current both condition, however, no distractor suppression was required, as both objects were possible targets, allowing participants to saccade to whichever object first exceed threshold for saccade initiation.

Same versus Switch Trials: Evaluation of Switch Costs

Selection probability for the cue-matching color did not differ between same and switch trials for either experiment (both ps>.24). Critically, saccade latency was not increased for switch trials compared with same trials for either Experiment 1A (switch: M=183ms, same: M=182ms) or Experiment 1B (switch: M=159ms, same: M=157ms; both ps>.41), replicating our previous finding (Beck et al., 2012).

Discussion

In a novel, sequential search task we observed substantial competition for selection between two objects that both matched a target color, suggesting that both colors were maintained in an “active” state in VWM and influenced selection. Additionally, as in Beck et al. (2012), we observed no cost when switching from one cued color to another. The present approach provides a particularly strong test of the SIT and MIT, as it examined whether participants are capable of maintaining multiple VWM representations in the template state. In addition, the reliable difference in selection probability between the same and both conditions provides novel evidence for simultaneous guidance. The results converge with several other studies indicating that multiple representations in VWM interact with perceptual selection (Beck et al., 2012; Hollingworth & Beck, 2016; Roper & Vecera, 2012). A similar resolution has emerged in the traditional literature on visual search. Wolfe (2007) proposed that search could be guided by only one feature value on a dimension. However, several recent studies have demonstrated simultaneous guidance by multiple values (Grubert & Eimer, 2015, 2016; Irons, Folk, & Remington, 2012; Moore & Weissman, 2010; Stroud, Menneer, Cave, & Donnelly, 2011). In this literature, the target values are fixed across the experiment, and guidance is likely to depend on LTM rather than VWM (Carlisle, Arita, Pardo, & Woodman, 2011). The two literatures therefore converge on a common principle of multiple-item guidance that appears to span guidance by active VWM representations and guidance by LTM.

The present results also have implications for general theories of working memory. Competing theories diverge on whether the “active” component of working memory, or “focus of attention”, is limited to a single representation/chunk (McElree, 2006; Oberauer, 2002) or spans multiple representations (Cowan, 2001). The need for a single-item “focus of attention” has been argued to arise from the need for item-level selectivity in cognitive operations (Oberauer & Hein, 2012). In vision, this type of discrete selection is ultimately instantiated by the oculomotor system via fixation. However, the VWM system that guides oculomotor selection has the capability to maintain multiple active representations, allowing for flexibility in strategic attentional control.