Simultaneous Control of Attention by Multiple Working Memory Representations

Abstract

Working memory representations play a key role in controlling attention, making it possible to shift attention to task relevant objects. Visual working memory has a capacity of 3–4 objects, but recent studies suggest that only one representation can guide attention at a given moment. We directly tested this proposal by monitoring eye movements while observers searched for one or two different colors in arrays containing two or four different colors. First, we identified behavioral signatures of template use: When observers implemented a single color template, they sequentially searched many consecutive items of a color (long run lengths), and they exhibited a delay prior to switching gaze from one color to another (switch cost). In contrast, when searching two colors simultaneously, observers exhibited short run lengths and no switch costs, consistent with the simultaneous guidance of attention by the two cued colors. Thus, multiple working memory representations can guide attention concurrently.

The mechanisms of selective attention fall into two classes, those involved in determining relevant sources of information (the control of attention) and those responsible for enhancing the processing of relevant sources and inhibiting competing sources (the implementation of selection –Luck & Vecera, 2002). Multiple systems contribute to the control of attention, including bottom-up salience computations, trial-by-trial priming, associative learning, and long-term knowledge (Chun & Turk-Browne, 2006; Kristjansson, 2008; Torralba, Oliva, Castelhano, & Henderson, 2006; Van der Stigchel et al., 2009). However, the guidance of attention toward task-relevant objects is thought to depend primarily on the use of working memory representations (Soto, Hodsoll, Rotshtein, & Humphreys, 2008). Guidance by working memory makes it possible for attention to “change gears” rapidly, because information can be loaded into visual working memory (VWM) in as little as 50 ms (Vogel, Woodman, & Luck, 2006), which leads to changes in the control of attention in 200 ms or less (Vickery, King, & Jiang, 2005; Wolfe, Horowitz, Kenner, Hyle, & Vasan, 2004).

The typical storage capacity of VWM is 3–4 items (Cowan, 2001; Luck, 2008). Consequently, one might expect that observers could maintain 3–4 simultaneous search templates, which would be useful in many natural tasks (e.g., finding either an orange or an apple on the counter). However, several researchers have argued that not all working memory representations are equal (Cowan, 2001), with only a single object in a fully active state (McElree, 2001; Oberauer, 2002). This view has led to a theory of attentional control in which only a single VWM representation can control attention at any given time (Olivers, Peters, Houtkamp, & Roelfsema, 2011). A similar claim is made by the Boolean map theory of attention (Huang & Pashler, 2007), which proposes that the visual input can be subdivided into to-be-attended and to-be-ignored regions on the basis of just one feature value.

These proposed limits on VWM control could reflect a fundamental bottleneck in the architecture of the brain. It is possible that, despite the ability to represent multiple objects, only one control signal can be sent from working memory processes to attentional mechanisms that implement visual selection. However, an architectural division of this kind is difficult to reconcile with evidence that VWM and perceptual processes are closely integrated. VWM representations can be stored within the visual system itself (Luck, 2008), including within primary visual cortex (Harrison & Tong, 2009; Serences, Ester, Vogel, & Awh, 2009). If multiple VWM representations are active within the visual system, it should be possible for multiple VWM representations to simultaneously control the operation of attention. Consistent with this possibility, a recent study found that observers could search selectively for targets matching two different templates (Stroud, Menneer, Cave, & Donnelly, 2011). However, the targets remained constant over the entire session, making it likely that attention was guided by long-term memory rather that working memory, and it is also possible that observers switched back and forth between the templates rather than concurrently searching for both targets. Thus, it is unknown whether observers can use two working memory representations simultaneously to guide attention.

The present study addressed this fundamental issue by measuring the pattern of eye movements as observers searched for a target presented in either of two colors. If observers maintain only one search template at a time, they should tend to search many items of one color before switching to search the other color, with a brief pause as they switch from one control signal to the other. However, if observers can keep two templates active concurrently, then they should go back and forth between the two colors, with no delay when switching from one color to the other.

Experiment 1

Before testing observers’ ability to search two colors concurrently, we examined the pattern of eye movements when the task explicitly encouraged observers to search one color and then switch to another color. That is, Experiment 1 was designed to reveal the “signatures” of a single attentional template during search.

Each search array contained 12 red circles and 12 blue circles (Figure 1). Observers searched for a target circle with a gap on the top or bottom and reported gap location. In the 80/20 condition, the target was 80% likely to be red and 20% likely to be blue (or vice versa). These probabilities encouraged observers to search the 80% color first and then, if the target had not been found, switch to the 20% color. We also included a 50/50 condition in which the target was equally likely to be red or blue and a 100/0 condition in which the target was 100% likely to be red (or blue). These control conditions assessed the limits of attentional control when target color was maximally and minimally uncertain.

Figure 1.

Example search arrays and trial events for Experiment 1. Landolt-C objects were 0.33° in diameter, had a line width of 0.10°, and a gap measuring 0.07°. Half were red (u’=0.479, v’=0.514, 8.12 cd/m²) and half were blue (u’=0.180, v’=0.158, 8.96 cd/m²), presented on a light grey background (42.31 cd/m²). Objects were assigned randomly to locations within a 5 × 5 grid (excluding the center location) and jittered within each cell by ±0.96° vertically and ±0.82° horizontally. Observers began each trial by directing gaze to a central fixation region (1.55°) for 300–500 ms, followed by the presentation of the cue square (0.65°) and search array until the observer’s response. There was a 1000-ms inter-trial delay.

In the 80/20 condition, we predicted that observers would fixate many items of the 80% color consecutively, more than predicted by choosing colors independently on the basis of the 80% and 20% probabilities. We further predicted that observers would switch to the 20% color if they did not find the target in the 80% color, which would require updating the search template and therefore produce a delay in making the next saccade (analogous to switch costs in the taskswitching literature, Monsell, 2003).

Methods

Observers

Twelve observers (18–30 years of age; 7 female) from the University of California, Davis completed the experiment. They reported normal color vision and normal or corrected-to-normal acuity.

Stimuli and Procedure

Stimuli were presented on a CRT monitor at 70 cm. Each search array contained 24 Landolt-C objects—12 red and 12 blue—against a gray background (see Figure 1 for stimulus parameters). There was one target (top or bottom gap) and 23 distractors (left or right gap). Observers reported target gap location via button press. The gaps were so small that gap position discrimination required object fixation, and the task therefore implicitly required observers to translate covert attentional control into overt shifts of gaze.

A cue square appeared at fixation throughout the search task to provide a constant reminder of the relevant color. The cue indicated the 100% or 80% color. It was black in the 50/50 condition. Cue color and probabilities were blocked (in counterbalanced order), and observers were informed of both at the beginning of each block. There were two blocks of 42 trials in the 100/0 and 50/50 conditions and four blocks of 52 trials in the 80/20 condition. The first two trials in each block were considered buffer trials and were excluded from all analyses.

Eye movements were recorded using an Eyelink 1000 eye tracker (2000 Hz). Saccades were defined by a combined velocity (>30°/s) and acceleration (>9500°/s²) threshold.

Results and Discussion

Manual response accuracy was uniformly high (mean = 99% correct) across all conditions.

Selectivity of search

Observers used the cue to limit gaze to the most likely target color. As shown in Figure 2A, manual correct reaction time (RT) was fastest in the 100/0 condition and slowest when the target appeared in the 20% color of the 80/20 condition. All pairwise differences were significant (p<0.05) except the difference in RT between the 20% trials and the 50/50 condition. The same pattern was observed for the time required for gaze to reach the target item (all ps<0.05; Figure 2A) and for the number of items fixated prior to fixating the target (all ps<0.05; Figure 2B). The strong correspondence between eye movement measures and manual RT validate the use of eye tracking to probe search efficiency in this paradigm. All subsequent eye movement analyses are limited to fixations prior to the target fixation.

Figure 2.

A) Mean time to the manual response (dark grey) and mean time to target fixation (light grey). B) Stacked bars indicate mean number of uncued- (dark grey), cued-color (light grey), or total (dark grey + light grey) objects fixated per trial. Since neither color was cued in the 50% condition, the columns represent mean number of blue (right diagonal) or red (left diagonal) objects fixated per trial. Error bars represent within-subject confidence intervals (Morey, 2008).

Selectivity was nearly perfect in the 100/0 condition, with almost all fixations directed to the cued color instead of the uncued color (t(11)=22.12, p<0.001 for the difference between the cued and uncued colors). In the 80/20 condition, gaze was directed to the 80% color much more often than to the 20% color when the target was the 80% color (t(11)=7.00, p<0.001) and somewhat more often when the target was the 20% color (t(11)=4.57, p<0.001). In the 50/50 condition, gaze was directed nearly equally to the red and blue items (t(11)=0.90, p=0.39). Thus, observers used the color probability information to control the search process.

Run length

Next, we examined whether observers maintained a consistent color template in the 80/20 condition, producing several consecutive fixations of the same color. In this condition, one strategy would be to search the cued-color items exhaustively and then switch to the uncued-color items (if the target had not been found). Alternatively, observers might select each successive saccade target independently, with an 80% probability of selecting the 80% color and a 20% probability of selecting the 20% color. To distinguish these possibilities, we examined the number of items of a particular color fixated consecutively (run length). The former strategy should result in a greater mean run length than the latter. Monte Carlo simulations were used to assess whether the observers conformed to these strategies (see online supplementary materials). We examined the initial run at the beginning of each trial. In addition, we focused on trials in which the target was in the 20% color and observers started by searching the 80% color, as this circumstance required a color switch for target detection (similar results were obtained when the target was the 80% color).

The mean initial run length was 6.61, which was significantly greater than the run length of a simulated observer (4.63) that independently selected each saccade destination according to the 80/20 probabilities, (t(11)=2.203, p=0.05). Observed run length was also significantly greater than the initial run length of 2.07 in the 50/50 condition (t(11)=5.12, p<0.001). Run length for the 50/50 condition was nearly identical to the value expected if red and blue items were fixated randomly (2.00). Thus, run length can be a signature of search template use.

Monte Carlo analyses demonstrated that the optimal strategy in the 80/20 condition would be to first search all 12 items of the 80% color, but the mean initial run length (6.61) was significantly less than 12 (t(11)=5.98, p<0.001). Although observers maintained a template of the 80% color, they tended to switch to the 20% color sooner than was optimal. This may reflect limits on the ability to keep track of which items have already been searched (Horowitz & Wolfe, 1998; Peterson, Kramer, Wang, Irwin, & McCarley, 2001) or it may reflect a tendency to engage in suboptimal probability-matching strategies (Vulkan, 2000).

Switch cost

We next examined the process of switching from one template to another. As shown in Figure 3, the duration of the fixation immediately before switching ("pre switch") was significantly greater than both the duration of the previous fixation ("minus one"; t(11)=3.12, p=0.01) and the duration of the first fixation on the new color ("post switch"; t(11)=2.16, p=0.05). Importantly, this switch cost was not found in the 50/50 condition, in which observers often switched randomly from one color to the other. The difference in switch costs between the 80/20 and 50/50 conditions was confirmed by an ANOVA with factors of condition (80/20 versus 50/50) and fixation position (minus one versus pre-switch), which yielded a significant interaction (t(11)=2.61, p=0.02) but no significant main effects. Thus, observers produced a switch cost only when they were actively selecting on the basis of color, providing a second signature of a search template use.

Figure 3.

Mean fixation duration for fixations immediately prior to a switch from fixating one color to fixating the other color. In the 80/20 condition (circles), this is restricted to a switch from a run of cued-color items to a run of uncued-color items. In the 50/50 condition (triangles), this represents a switch from a run of one color (e.g., red) to a run of the other color (e.g., blue). A run is defined by 2+ objects of the same type fixated consecutively. Error bars represent within-subject confidence intervals (Morey, 2008).

Summary

Together, these results provide strong evidence that observers can form a strong search template when the task encourages it, limiting gaze almost perfectly to cued-color items when the cue is 100% valid and limiting gaze primarily to cued-color items when the cue is 80% valid. Moreover, observers consecutively searched many items of the cued color, and they exhibited a cost when they switched from searching one color to the other. When color was nonpredictive, however, observers ignored color and switched randomly between red and blue with no switch cost. These patterns serve as signatures of search template use that can be applied to the main question of the study: Can observers maintain two templates concurrently in VWM?

Experiment 2

In Experiment 2, a cue was presented before the onset of the search array, and the cued color(s) changed randomly from trial to trial (Figure 4). Consequently, it was necessary to store the cued color(s) in VWM and use the VWM representation to guide attention (Chelazzi, Miller, Duncan, & Desimone, 1993; Vickery et al., 2005; Wolfe et al., 2004; Woodman, Luck, & Schall, 2007). Each search array contained 8 items of each of 4 colors, and either one or two of these colors was indicated by the cue. The target was always a cued color. We tested whether observers would concurrently search two of the four colors when two colors were cued or, alternatively, search multiple items of one color and then switch to the other color (as in the 80/20 condition of Experiment 1).

Figure 4.

Example search arrays and trial events for Experiment 2. There were 32 objects total: 8 red (u’=0.414, v’=0.443, 18.75 cd/m²), 8 blue (u’=0.193, v’=0.259, 18.60 cd/m²), 8 yellow (u’=0.305, v’=0.535, 18.67 cd/m²), and 8 green (u’=0.141, v’=0.510, 18.60 cd/m²), presented on a grey background (39.65 cd/m²). Objects were assigned randomly to locations within a 20.85° × 15.82° region with a minimum inter-item distance of 2.10° and a minimum distance from center of 1.96°. The cue square subtended 0.79° and was composed of four smaller squares each subtending 0.36°. On single-cue trials, all four squares in the cue were the same color; on dual-cue trials, two diagonally opposed squares were presented for each of the two cued colors. Observers began each trial by directing gaze to a central fixation region (1.55°) for 300–500 ms, after which the cue appeared in the center of the screen for 500 ms, beginning 1000 ms prior to search array onset. There was a 1200-ms inter-trial delay.

Because observers may be able to strategically control whether they maintain two simultaneous templates or switch between one template and another, we gave observers explicit instructions about which strategy to use. In half of the trial blocks, they were instructed to search one color and then switch to the other. In the other half, they were instructed to search both of the two cued colors concurrently. The goal was to determine whether they could actually search both cued colors concurrently when instructed to do so.

Method

The methods were identical to those of Experiment 1, except as noted (Figure 4). Eleven new observers (8 female) searched arrays containing 8 red, 8 blue, 8 yellow, and 8 green items.

To ensure that observers focused on the cue color(s) and did not simply search for an item with a top/bottom gap, one item of an uncued color contained a top/bottom gap. The target was therefore defined as having the cued color and a top/bottom gap. The cue stimulus indicated either a single color or two colors and was composed of four smaller squares. The target was always a cued color.

When two colors are cued, observers might form a single template that covers a broad area of color space including both colors, but this is possible only when the cued colors are linearly separable from the uncued colors (D'Zmura, 1991; Duncan & Humphreys, 1989). Color values were selected to form a quadrangle in color space (see Figure 4). On separable trials, the two cued colors were on one side of the quadrangle (e.g. red and yellow) and the two uncued colors were on the other (e.g., green and blue), making it possible to form a single template that was closer to the two cued colors than to the uncued colors. On nonseparable trials, the two cued colors were diagonally opposed in color space (e.g., red and green) and the uncued colors were diagonally opposed along the orthogonal direction (e.g., blue and yellow), so that no single color value was closer to the cued colors than to the uncued colors. This method is well established to preclude the use of a single template for both cued values (D'Zmura, 1991; Duncan & Humphreys, 1989).

Single-cue, separable dual-cue, and nonseparable dual-cue trials were randomly intermixed. Observers were instructed to search one color at a time in half of the trial blocks (sequential-search condition) and to search the two colors simultaneously in the other half (simultaneous-search condition). Block order was counterbalanced. Observers performed 16 blocks of 32 trials, yielding a total of 120 single-cue, 240 separable dual-cue, and 120 nonseparable dual-cue trials.

Results and Discussion

Overall search performance

Manual response accuracy was uniformly high across all conditions (mean = 97% correct). Search RT was lower on single-cue trials than on dual-cue trials (Figure 5A), (t(10)=18.27, p<0.001), reflecting the fact that attention could be limited to 8 items on single-cue trials versus 16 items on dual-cue trials. The single-cue trials replicated the results from the 100/0 condition of Experiment 1 and will not be discussed further.

Figure 5.

A) Mean time to manual response (solid-fill bars) and to target fixation (diagonal-fill bars). B) Stacked bars indicate mean number of uncued-color (solid-fill bars), cued-color (diagonal-fill bars), or total (solid-fill + diagonal-fill) objects fixated per trial. Error bars represent within-subject confidence intervals (Morey, 2008).

For dual-cue trials, there was no significant effect of instructions on manual RT, elapsed time to target fixation, or the number of items fixated prior to target fixation (Figure 5; all ps>0.22). For all three of these measures, there was no difference between separable and nonseparable trials (all ps>0.25). A Bayes factor analysis (Rouder, Speckman, Sun, Morey, & Iverson, 2009) indicated that the null hypothesis was substantially more probable than the hypothesis that separable and nonseparable trials differed (odds ratio of 4.0 for RT, 3.4 for time to target fixation, and 2.3 for number of fixations). Consequently, all subsequent analyses collapse across the separable/nonseparable factor.

Selectivity and speed of search

Observers fixated fewer items on single-cue trials than on dual-cue trials (Figure 5B), reflecting the smaller number of potential targets on single-cue trials. Fixation durations were 18 ms faster on single-cue trials than on dual-cue trials (t(10)=9.00, p<0.001). Thus, there was an advantage to having a single target color. This does not imply that observers were unable to maintain multiple templates; it may simply indicate that maintaining multiple templates requires additional resources.

Observers were just as selective and just as fast when instructed to search both colors simultaneously as when instructed to search the two colors sequentially (Figure 6). To quantify selectivity, we computed the proportion of fixations that were directed to the cued color (cued÷(cued+uncued)). These values did not differ significantly between the sequential and simultaneous conditions (t(10)=0.52, p=0.62), and Bayes factor analysis indicated that the null hypothesis was 3.9 times more probable than the alternative hypothesis. Fixation durations were nearly identical in the sequential and simultaneous conditions (t(10)=1.34, p=0.21), with the null hypothesis 2.0 times more likely than the alternative hypothesis. Thus, attempting to search both cued colors simultaneously produced no disruption in the ability to search rapidly or selectively.

Figure 6.

A) Mean number of cued-color (solid-fill bars) and uncued-color (diagonal-fill bars) objects fixated per trial, collapsed across separable and nonseparable dual-cue trials. B) Mean fixation duration for fixations on cued-color items, collapsed across separable and nonseparable dual-cue trials. Error bars represent within-subject confidence intervals (Morey, 2008).

Run length

Figure 7 shows representative scan paths from individual trials on which red and green were cued in the sequential-search and simultaneous-search conditions. The observer was highly selective in both conditions, limiting gaze to the two cued colors. The observer searched seven consecutive red items and then five consecutive green items in the sequential-search example, but went back and forth between red and green items multiple times with short runs in the simultaneous-search condition. To quantify this difference, we computed mean initial run length, as in Experiment 1. Mean run length was significantly smaller in the simultaneous-search condition (2.4) than in the sequential-search condition (3.4), (t(10)=2.72, p=0.02). Mean initial run length for the simultaneous-search condition was quite close to the value expected by random selection of the two cued colors (1.8), whereas the mean run length for the sequential-search condition (3.4) was significantly greater than would be expected by random selection (t(10)=4.27, p=0.002).

Figure 7.

Shown here are sample scanpaths for a sequential trial (above) and a simultaneous trial (below) for which the observer was cued to search both red and green. The order of fixations is indicated by the number next to each fixation. The sequential trial shows a run of seven red items followed by a run of five green items. The simultaneous trial shows the same observer switching more frequently between the red and green items (run of two green items, run of four red items, one green item, then one red item).

Switch cost

There was a significant switch cost (increase in the pre-switch fixation duration relative to the preceding fixation) in the sequential-search condition (23-ms difference; t(10)=2.57, p=0.03), indicating that observers actively switched from one template to the other in this condition (Figure 8). However, there was no significant switch cost in the simultaneous-search condition (1.5-ms difference; t(10)=0.23, p=0.82; Bayes factor = 4.4 in favor of null hypothesis), indicating that observers were not switching between templates in this condition. An ANOVA with factors of instruction condition and fixation position yielded a significant interaction (t(10)=2.35, p=0.04), indicating a smaller switch cost in the simultaneous-search condition than in the sequential-search condition.

Figure 8.

Mean fixation duration for fixations immediately prior to a switch from searching one cued color to searching the other cued color. This analysis was restricted to trials on which the target color was not initially search, and it was limited to a switch from a run of non-target cued-color items to a run of target-color cued-color items. A run is defined by 2+ objects of the same type fixated consecutively.

One possible alternative explanation for a lack of switch cost is that, in the simultaneous condition, the preparation for switching was spread out over several fixations instead of being limited to the “pre switch” fixation. If this were the case, and the switch cost in the simultaneous condition was incorporated into many fixations, the overall mean fixation duration should be greater in the simultaneous condition than in the sequential condition. As seen in Figure 6B, however, there was no difference in mean fixation duration across the simultaneous and sequential conditions.

Summary

These results indicate that observers can either activate two templates sequentially or activate them both simultaneously, depending on task instructions. When asked to search the colors sequentially, observers exhibited relatively long runs of a given color, and a switch cost when they shifted from one color to the other. When asked to search the colors simultaneously, they shifted back and forth between the two colors more frequently, and there was no switch cost when they shifted from one color to the other. In addition, the overall speed and selectivity of search was virtually equivalent for these two search tasks. These results demonstrate that people are able to maintain two active representations in VWM that guide attention concurrently.¹

General Discussion

These results directly demonstrate that people can use multiple attentional templates to simultaneously guide search toward relevant objects. In Experiment 1, we identified two signatures of the use of a single attentional template to guide search: a) long runs of fixations on items that match the template, and b) a switch cost when they shift from one template to another. In Experiment 2, we used these signatures along with other measures to demonstrate that observers can maintain two concurrent templates when asked to do so. Searching two colors concurrently led to no impairment in the time required to find and report the target compared to searching for the two target types sequentially. Moreover, observers shifted gaze back and forth between the two cued colors with short run lengths in the simultaneous-search condition, and no switch cost was present when they shifted from one color to the other. Thus, searching for two distinct features concurrently led to no cost relative to search for these features sequentially, and the gaze patterns indicated that both templates were concurrently active.

These results provide an important constraint on the architecture of cognition. Specifically, they demonstrate that the multiple representations that are concurrently stored in VWM (Cowan, 2001; Luck, 2008) can simultaneously be linked to the control of attention. In other words, there is no single-channel bottleneck in top-down attentional control. Instead, multiple VWM representations may interact directly with the flow of sensory information through the visual system, which matches the fact that visual perception and VWM operate within the same regions of visual cortex (Harrison & Tong, 2009; Serences et al., 2009).

It is difficult to know with certainty why previous studies failed to find evidence of multiple simultaneous templates (see reviews by Huang & Pashler, 2007; Olivers et al., 2011). The present study found that observers could voluntarily decide whether to search for the two cued colors sequentially or simultaneously, so it is possible that previous studies simply failed to induce the participants to activate the templates simultaneously. Indeed, one previous study found a switch cost that was comparable to that the cost observed in the sequential condition (Dombrowe, Donk, & Olivers, 2011). Moreover, the present results indicate that there is a cost to maintaining multiple templates, and this may have motivated observers in previous studies to use other strategies (e.g., using singleton detection mode in the study of Eimer & Kiss, 2010). In other cases, the task required observers to link particular features with particular locations (Adamo, Pun, Pratt, & Ferber, 2008; Parrott, Levinthal, & Franconeri, 2010; Wolfe et al., 1990), and this may be more difficult than merely activating two features. In any case, the present results demonstrate that people can, under some circumstances, activate multiple search templates simultaneously, even if there are limits on the situations in which they can do so.