Selective Maintenance in Visual Working Memory Does Not Require Sustained Visual Attention

Abstract

In four experiments, we tested whether sustained visual attention is required for the selective maintenance of objects in VWM. Participants performed a color change-detection task. During the retention interval, a valid cue indicated the item that would be tested. Change detection performance was higher in the valid-cue condition than in a neutral-cue control condition. To probe the role of visual attention in the cuing effect, on half of the trials, a difficult search task was inserted after the cue, precluding sustained attention on the cued item. The addition of the search task produced no observable decrement in the magnitude of the cuing effect. In a complementary test, search efficiency was not impaired by simultaneously prioritizing an object for retention in VWM. The results demonstrate that selective maintenance in VWM can be dissociated from the locus of visual attention.

Visual working memory (VWM) supports the brief maintenance of several visual representations across disruptions in perceptual input.¹ Visual attention is a mechanism that selects a location or set of locations containing relevant perceptual information within a scene.² Understanding the relationship between VWM and visual attention is one of the most active areas of research in perceptual psychology and neuroscience, and current evidence suggests that they interact closely in several domains (for reviews, see Awh, Vogel, & Oh, 2006; Desimone & Duncan, 1995; Gazzaley & Nobre, 2012; Jonides et al., 2008; Theeuwes, Belopolsky, & Olivers, 2009). First, visual attention influences the consolidation of perceptual information into VWM (Irwin & Gordon, 1998; Schmidt, Vogel, Woodman, & Luck, 2002; see also Averbach & Coriel, 1961; Sperling, 1960). Where one attends determines, to a large extent, what one encodes into VWM. Second, VWM content correlates with feature-specific activation in low-level sensory regions of the brain (Harrison & Tong, 2009; Serences, Ester, Vogel, & Awh, 2009). Visual attention and VWM therefore may operate upon an overlapping, visual-sensory substrate. Third, VWM content modulates the perceptual competition among objects in the visual field. Objects that match feature values in VWM tend to attract attention (Hollingworth & Luck, 2009; Olivers, Meijer, & Theeuwes, 2006; Soto, Heinke, Humphreys, & Blanco, 2005; Soto, Humphreys, & Heinke, 2006), suggesting a role for VWM in feature-based perceptual selection (Desimone & Duncan, 1995). Finally, visual attention supports the working memory representation of spatial locations (Awh & Jonides, 2001; Theeuwes, Olivers, & Chizk, 2005), with disruptions of visual attention disrupting spatial memory (Awh, Jonides, & Reuter-Lorenz, 1998). Although most researchers have characterized attention and working memory as overlapping and interactive, some have gone further to speculate (Theeuwes et al., 2009) or even claim outright (Cowan, 1995) that “working memory” and “attention” are just two terms to describe a single system. In a similar vein, several researchers have argued that visual attention and VWM reflect a common selective mechanism that differs only in whether it operates over the representation of currently visible objects or previously visible objects (Chun, 2011; Gazzaley & Nobre, 2012).

Visual attention is selective, by definition. VWM is also selective, in that it retains a small set of task-relevant visual information. Understanding the functional relationship between attention and VWM therefore depends on understanding how selectivity in VWM arises. Entry into VWM is strongly influenced by visual attention. Once objects are represented in VWM, additional selective operations are required. To meet the demands of real-world tasks, the content of VWM must be managed strategically to keep task-relevant information active. Consider a visual search task. The features of the search target (i.e., the search template) must be preferentially maintained in VWM throughout the search event and protected from the interference generated by perceptual processing of other objects in the scene. When the target of search changes, a new set of features must be loaded into VWM and preferential maintenance transferred to the new template. How is such selective maintenance, or prioritization, implemented? This issue has attracted substantial interest: It is critical to understanding the general relationship between attention and VWM, and it strikes to the heart of how VWM is managed to adapt to the changing informational demands of visual behavior.

Consistent with recent trends, the dominant position in the literature is that objects are selectively maintained in VWM by the application of visual attention (Gazzaley & Nobre, 2012; Griffin & Nobre, 2003; Landman, Spekreijse, & Lamme, 2003; Makovski & Jiang, 2007; Makovski, Sussman, & Jiang, 2008; Matsukura & Hollingworth, 2011; Matsukura, Luck, & Vecera, 2007). The bulk of the evidence supporting this view comes from a retention-interval cuing paradigm (often termed the “retro-cuing” paradigm). In the initial study of this type, Griffin and Nobre (2003) had participants view a brief memory array of colors followed by a single test color. The task was change detection. During the retention interval, a cue appearing 1500–2500 ms after the offset of the memory array specified the location of one item that was likely to be tested. Change detection accuracy for valid-cue trials was superior to neutral-cue trials, indicating that the cued item had been preferentially retained.

Griffin and Nobre (2003) argued that the spatial cue allowed visual attention to be directed to the memory representation of the cued item, and the act of sustaining attention on the cued item was the mechanism of selective maintenance. Specifically, keeping visual attention focused on the cued item was proposed to enhance the perceptual-level representation of that item in VWM in the same way that sustained visual attention to a location in the visual field enhances the perceptual processing of sensory input. At the heart of this proposal is the idea that selection in VWM is equivalent to selection in visual perception, except that in the former, visual attention operates over the representations of stimuli that are no longer visible (Chun, 2011; Gazzaley & Nobre, 2012). Subsequent studies have suggested that sustained attention protects the cued item from the interference generated by other items in VWM (Matsukura et al., 2007) and from the interference generated by subsequent perceptual processing (Makovski et al., 2008). A functional role for visual attention in VWM prioritization is bolstered by evidence that a cue to retain an object in memory modulates activation in extra-striate visual cortex (Lepsien & Nobre, 2007) and that similar networks of brain regions are activated by cues to attend within currently visible arrays and previously visible arrays (Nobre et al., 2004).

This account of VWM prioritization is certainly plausible. However, the evidence to date is incomplete, because it is not known whether sustained attention is required for VWM prioritization. No study has tested whether selective maintenance is possible in the absence of visual attention. A test of this type is particularly important, because a related theoretical account has been challenged by experiments manipulating the availability visual attention. Wheeler and Treisman (2002) claimed that the binding of features in VWM requires sustained attention, proposing that visual attention is a common mechanism for feature binding in vision and in working memory. However, subsequent studies that directly manipulated attention observed robust feature binding in its absence (Gajewski & Brockmole, 2006; Johnson, Hollingworth, & Luck, 2008; see also Allen, Baddeley, & Hitch, 2006).³ For example, Johnson et al. (2008) added a difficult visual search task to the retention interval of a change-detection paradigm. The search task, which precluded sustained visual attention to any given item, had a generally detrimental effect on change detection accuracy, but it did not specifically impair binding memory, which remained well above chance. Thus, feature binding in VWM does not appear to require sustained attention. No equivalent test has been applied to the claim that prioritization in VWM depends on sustained attention.

A second reason to question the idea that VWM prioritization requires sustained attention is based on a consideration of the demands of real-world visual tasks. Returning to the example of visual search, efficient performance of a search task depends on strategically maintaining the target features in VWM throughout the event. Simultaneously, visual attention must be shifted serially from object to object within the array. The attentional demands of these two components of the task appear to be in conflict. If selective maintenance requires sustained attention, then one could either prioritize the template features or conduct serial search, but not both. To support efficient search, one would need a mechanism of prioritization that was dissociable from the current locus of attention. Other common tasks, such as perceptual comparison, introduce a similar demand to dissociate VWM prioritization from visual attention. To decide which apple in a bin is the largest and most appetizing, one must preferentially maintain the features of one apple in VWM while shifting visual attention to other apples in the bin so as to encode and compare their features to the remembered apple.

In the present study, we tested whether sustained visual attention is necessary for selective maintenance in VWM by introducing a demanding visual search task to the retention-interval cuing paradigm. Participants viewed a memory array of colors. After the spatial cue to preferentially retain a particular item, participants engaged or did not engage in visual search. In the memory-only trials, participants could sustain attention on the cued location or item, and we expected to replicate the cuing advantage observed in previous studies. In the dual-task condition, however, the search task limited participants’ ability to sustain attention on the cued location or item. If selective maintenance in VWM requires sustained visual attention, then prioritization of the cued object should be impaired or prevented on search trials, reducing or eliminating the cuing effect. Contrary to this prediction, in Experiments 1 and 2, we observed a robust cuing effect, and the magnitude of the cuing effect was independent of the availability of visual attention. As a converging approach (Experiment 3), we presented the memory stimuli sequentially at fixation and used a non-spatial cue, so that the cued item could not be selected by directing visual attention to any particular location. Nevertheless, a robust cuing effect was observed. Finally, in Experiment 4, we probed the efficiency of visual search when participants were or were not simultaneously prioritizing an object for retention in VWM. Search efficiency was independent of VWM prioritization. Thus, just as feature binding in VWM can be dissociated from visual attention (Delvenne, Cleeremans, & Laloyaux, 2010; Gajewski & Brockmole, 2006; Johnson et al., 2008), selective maintenance in VWM can be dissociated from the locus of visual attention. We discuss alternative mechanisms of VWM prioritization and implications for theories of the relationship between attention and working memory.

Experiment 1

The primary task in Experiment 1 was color change detection (Figure 1). Participants saw a memory array of six colors at the beginning of the trial. At the end of the trial, one test color (same or changed) was displayed at the location of a memory array item. There was a 700-ms ISI between the offset of the memory array and the presentation of the retention-interval cue. This was long enough to ensure that sensory persistence was no longer available at cue onset (Irwin & Yeomans, 1986). The cue was visible for 500 ms. It was either valid (pointing to the location of the memory item that would be tested) or neutral (pointing to all six locations). In the memory-only condition, the cue was followed by a 2500-ms blank ISI before the test display. In the dual-task condition, the cue was followed by a 500-ms ISI and a 2000-ms search array. The search array consisted of 8 boxes, each with one side missing. The target had a gap on the left or the right, and distractors had a gap on the top or bottom. This type of search task, in which the target differs from distractors in the spatial arrangement of features, requires serial shifts of attention to the items in the search array (e.g., Woodman & Luck, 2003). To equate the overall retention interval between the dual-task and memory-only conditions, the search array was visible for 2000 ms regardless of when the participant made the target discrimination response.

Figure 1.

Sequence of events in a trial of the dual-task condition of Experiment 1. Participants began the trial by repeating aloud a set of four digits (not pictured). After pressing a pacing button, there was a delay of 500 ms, followed by the depicted events. A memory array of six different colors was presented for 250 ms, followed by a 700-ms ISI, a 500-ms cue display (valid/neutral), a 500-ms ISI, a 2000-ms search array, and a test display. The figure depicts a changed trial. Participants responded during the search array to indicate target gap location (left/right). Then, they responded to the test display to indicate same or different color. The memory-only condition was identical, except that the search array was replaced by 2000 ms of blank ISI.

The cuing effect was calculated as the difference in color memory accuracy between valid-cue trials and neutral-cue trials. The magnitude of the cuing effect was compared for the dual-task condition and the memory-only condition.

Participants

Sixteen University of Iowa undergraduates participated for course credit. All reported normal or corrected-to-normal vision. One participant responded to the search display on less than 15% of trials and was replaced.

Stimuli

Stimuli were presented on a gray background with a small, black fixation ring (0.3° diameter) that was visible throughout the trial. The color memory squares (1.1° × 1.1°) appeared at six locations evenly spaced around a virtual circle with a radius of 2.3°. Each color was selected randomly without replacement from a set of ten: red, blue, green, yellow, black, brown, orange, cyan, pink, and violet. The cue was a black line (.09° width, 0.5° length) that extended from the central fixation ring toward one of the six stimulus locations. On valid-cue trials, the cued location was selected randomly from the set of six. On neutral-cue trials, all six cue lines were displayed.

The eight search stimuli were squares (.86° × .86°) with one side missing. The seven distractors had the gap on either the top or bottom (randomly selected). The one target had the gap on the left or right (randomly selected). Target location was randomly selected. Each search element was centered 3.6° from the center of the screen. The search elements were organized into four pairs (above, below, left, and right) to promote a strategy of serial shifts of attention from pair to pair. The center-to-center distance between two elements in a pair was 1.9°. The screen regions occupied by the search and memory arrays did not overlap.

The memory test display was composed of a single color square that appeared at one of the locations occupied by a memory square. On valid-cue trials, the test item appeared at the cued location. On neutral-cue trials, the test location was randomly selected. For “same” trials, the test color was the same as the color appearing in that location in the memory array. For “changed” trials, the test color was randomly selected from the four colors that had not appeared in the memory array.

Apparatus

Stimuli were displayed on a 17-inch CRT monitor with a resolution of 800 × 600 pixels at a 100-Hz refresh rate. Stimulus presentation was synchronized with the monitor’s refresh cycle. A viewing distance of 80 cm was maintained by a forehead rest. Responses were collected by a serial button box. The experiment was controlled by E-prime software (Schneider, Eschmann, & Zuccolotto, 2002).

Procedure

Each trial began with the visual presentation of four randomly chosen digits. Participants repeated the digit sequence aloud throughout the trial to suppress verbal encoding of the memory stimuli. After initiating digit repetition, the participant pressed a pacing button to begin the trial. There was a 500-ms delay (fixation ring only). In the dual-task condition (Figure 1), the memory array was presented for 250 ms, followed by a 700-ms ISI, the cue display for 500 ms, a 500-ms ISI, the search display for 2000 ms, and finally the test display. Participants made two responses. First, they responded during the presentation of the search display to indicate target gap location (left/right). Second, they responded to the test display to indicate “same” or “changed”. The same two buttons were used for both responses. The memory-only condition was identical to the dual-task condition, except the search array was replaced by a 2000-ms blank ISI (for a total of 2500-ms ISI between cue and test), and participants responded only to the test display.

Participants were given both written and oral instructions. They first completed a practice block (12 trials) of the memory-only condition and a practice block (12 trials) of the dual-task condition. This was followed by four blocks of experimental trials, two of dual task and two of memory only, interleaved (e.g., ABAB). The assignment of task condition to the first block was counterbalanced across participant groups. Each block began with four buffer trials that were excluded from analysis, followed by 80 experiment trials. The latter were divided evenly among the four conditions created by the 2 (cue: valid, neutral) × 2 (change: same, changed) design. Trials from the four conditions were randomly intermixed. Participants completed a total of 320 experiment trials.

Data Analysis

In the search task, participants responded to the search display during the 2000 ms that it was visible on 97.2% of trials. Target discrimination accuracy was 88.9%. Mean correct RT was 868 ms. None of these measures varied as a function of cue type.

In all experiments, limiting the change detection analysis to trials on which the search response was correct did not influence the results, so all trials were included. For the color memory task, trials in Experiment 1 were eliminated if the participant responded within 250 ms of the onset of the test display (1.1% of search trials), as these were assumed to be delayed responses to the search task.

Results and Discussion

Accuracy, collapsing across same and changed trials, is displayed in Figure 2 as a function of search task and cue validity. The raw percentage correct data are reported in Table 1. In all four experiments, an analysis over d′ produced exactly the same pattern of results and the same pattern of statistical significance.⁴

Figure 2.

Change detection accuracy (collapsed across same and change trials) for Experiment 1 as a function of search task and cue validity. Error bars are standard errors of the means.

Table 1.

Mean percentage correct data in the change detection task for Experiments 1–4.

	Valid		Neutral
	Same	Changed	Same	Changed
Experiment 1
Memory Only	76.1	96.1	42.7	91.1
Dual Task	77.2	90.7	38.4	82.0
Experiment 2
Memory Only	71.2	69.0	58.8	65.1
Dual Task	73.9	54.9	57.7	55.9
Experiment 3
Serial Position 1
Memory Only	70.0	68.7	54.8	73.6
Dual Task	73.8	64.9	53.8	65.2
Serial Position 2
Memory Only	62.5	68.8	49.0	68.1
Dual Task	71.7	65.0	51.5	62.8
Serial Position 3
Memory Only	71.2	72.5	61.8	72.6
Dual Task	70.5	67.1	66.5	68.7
Serial Position Collapsed
Memory Only	69.3	70.7	56.7	71.4
Dual Task	73.1	65.5	59.0	64.9
Experiment 4
Dual Task	73.2	59.7	57.4	59.7

The pattern of data in Figure 2 suggests that there was a cuing effect in both the memory-only and dual-task conditions. This observation was confirmed by a 2 (dual task, memory only) × 2 (cue: valid, neutral) repeated measures ANOVA. First, the search task produced a small but reliable decrement in memory accuracy, F(1,15) = 14.8, p = .002. This is not surprising given that the search task introduced additional stimuli that could have interfered with the content of VWM. Second, there was a reliable cuing effect, with higher accuracy in the valid condition than in the neutral condition, F(1,15) = 168.8, p < .001. Critically, these two factors did not interact, F(1,15) = 3.24, p = .09. The numerical trend was toward a larger cuing effect in the dual-task condition (23.7 percentage points) than in the memory-only condition (19.2 percentage points).

On memory-only trials, when attention could be sustained at the cued location, participants prioritized the cued item. This replicates previous retention-interval cuing studies. However, participants used the cue just as efficiently in the dual-task condition, when they could not sustain attention at the cued location. Selective maintenance in VWM does not appear to require sustained visual attention.

Experiment 2

We assume that the cued color was retained in a visual format in Experiment 1. However, prioritization of a cued item could be implemented by recoding it in a nonvisual format, such as a verbal or categorical code. Articulatory suppression was used to eliminate verbal coding of the stimuli in Experiment 1. To reduce the possibility of categorical coding of color, we used a within-category discrimination task in Experiment 2 that required memory for the precise visual properties of the array items (Figure 3). Participants saw three color patches in the memory array. On changed trials, the changed item was a similar color drawn from the same color category as the corresponding memory item. Thus, encoding the stimuli as ‘red’ or ‘green’ would not have sufficed; successful change detection required encoding and maintaining a specific color value within a particular color category.

Figure 3.

Sequence of events in a trial of the dual-task condition of Experiment 2. The method differed from Experiment 1 in two key respects: A within-category, change-detection task was used, and the memory set size was reduced to three items.

Participants

Sixteen new University of Iowa undergraduates participated for course credit. All reported normal or corrected-to-normal vision.

Stimuli

The stimuli were the same as those in Experiment 1, with the following exceptions. Due to difficulty of the change detection task, three memory array stimuli were presented instead of six. They appeared in a triangular configuration around central fixation. The color of one square was drawn from a set of four similar reds, the color of the second from a set of four similar blues, and the color of the third from a set of four similar greens. The assignment of color category to stimulus locations was randomly determined. The choice of the particular color value in each category was also randomly determined. The cue pointed either to one stimulus location or to all three. On color change trials, the test color was chosen randomly from the remaining three color values within the relevant category.

Procedure

The procedure was the same as in Experiment 1 (including articulatory suppression), except the memory array was presented for 350 ms rather than for 250 ms to allow for precise encoding of color values.

Data Analysis

In the search task, participants responded to the search display during the 2000 ms that it was visible on 98.7% of trials. Target discrimination accuracy was 86.9%. Mean correct RT was 842 ms. None of these measures varied as a function of cue type.

For the color memory task, 1.3% of search trials and 0.5% of memory-only trials were eliminated due to a response within 250 ms of the onset of the test display.

Results and Discussion

Accuracy, collapsing across same and changed trials, is displayed in Figure 4 as a function of search task and cue validity. The raw percentage correct data are reported in Table 1.

Figure 4.

Change detection accuracy (collapsed across same and change trials) for Experiment 2 as a function of search task and cue validity. Error bars are standard errors of the means.

As in Experiment 1, the search task produced a reliable decrement in memory accuracy, F(1,15) = 13.3, p = .002. In addition, there was a reliable cuing effect, F(1,15) = 38.4, p < .001. These two factors did not interact, F < 1. The cuing effect was 8.1 percentage points in the memory-only condition and 7.6 percentage points in the dual-task condition. Replicating Experiment 1, participants were able to use a cue to prioritize an item in VWM independently of whether they could sustain attention on that item.

Experiment 3

In Experiment 3, we modified the task to place additional limitations on the utility of visual attention in VWM prioritization. In previous retention-interval cuing studies, the memory items have appeared at different spatial locations, and the cue has indicated the location of the to-be-tested item. This method is amenable to the application of visual attention, which operates by selecting a particular location, or group of locations (Hollingworth, Maxcey-Richard, & Vecera, 2012), within a spatially arrayed perceptual representation (e.g., Posner, 1980). Experiment 3 replicated the within-category task of Experiment 2 with two modifications that limited participants’ ability to select an item in memory by attending to a particular location. First, the memory stimuli were presented sequentially at the center of the screen and could not be distinguished on the basis of location. Second, the cue indicated the color category of the relevant item. For example, the valid cue for a particular red item was the word “red”. A feature-based cue does not provide any direct means to engage a spatially selective mechanism. Given the results of Experiments 1 and 2, we expected that participants would nevertheless show robust prioritization of the cued item and that the cuing effect would be independent of search condition.

Participants

Twenty-four new University of Iowa undergraduates participated for course credit. All reported normal or corrected-to-normal vision. Two were replaced, because they failed to perform above chance on the change detection task.

Stimuli

The stimuli were the same as those in Experiment 2, with the following exceptions. Each color square was presented alone at the center of the screen. Color squares subtended 1.71° × 1.71°. The valid cue was a word specifying the relevant color category (“red”, “blue”, or “green”). The neutral-cue word indicated that all color categories could be tested (“all”). The test color was presented at the center of the screen. Finally, the central fixation ring was eliminated, as all memory stimuli were presented at the center.

Procedure

The procedure was the same as in Experiment 2, except that the three color stimuli were presented sequentially, each for 250 ms, separated by 250-ms ISIs. In addition, the ISI between cue and search array was increased to 700 ms. Each of the four blocks contained 72 experiment trials, divided evenly among the four conditions created by the 2 (cue: valid, neutral) × 2 (change: same, changed) design.

Data Analysis

In the search task, participants responded to the search display during the 2000 ms that it was visible on 94.8% of trials. Target discrimination accuracy was 91.2%. Mean correct RT was 821 ms. None of these measures varied as a function of cue type.

For the color memory task, 0.7% of search trials and 0.1% of memory-only trials were eliminated due to a response within 250 ms of the onset of the test display.

Results and Discussion

Accuracy, collapsing across same and changed trials and serial position, is displayed in Figure 6 as a function of search task and cue validity. The raw percentage correct data are reported in Table 1.

Figure 6.

Change detection accuracy (collapsed across same and change trials) for Experiment 3 as a function of search task and cue validity. Error bars are standard errors of the means.

There was an effect of serial position in the memory sequence, F(2,46) = 10.3, p < .001. Mean change-detection accuracy was 65.7% for the first item, 62.4% for the second item, and 68.9% for the third item. However, serial position did not interact with variables of interest (search task; cue validity), and we collapsed serial position for the main analysis.

The search task did not produce a reliable decrement in memory accuracy, F(1,23) = 1.69, p = .21. However, there was again a reliable cuing effect, F(1,23) = 36.0, p < .001. These two factors did not interact, F < 1. The cuing effect was 6.0 percentage points in the memory-only condition and 7.3 percentage points in the dual-task condition. Despite the fact that the memory stimuli could not be selected on the basis of location, there remained a reliable cuing advantage of a magnitude similar to that observed in Experiment 2. Thus, the mechanism by which the cued item was selected in VWM does not appear to depend on directing visual attention to a particular location, either within a representation of the currently visible scene or within a representation of the previously visible memory items. Moreover, the cuing effect was independent of the availability of visual attention after the cue, replicating the first two experiments.

Experiment 4

To retain the idea that visual attention is necessary for VWM prioritization, one might argue that, in Experiments 1–3, visual attention was split between the memory representation of the cued item and particular locations within a perceptual representation of the search array. That is, attention might have been sustained on the memory representation of the cued object and simultaneously shifted to stimulus locations during the search task. Alternatively, one might argue that visual attention maintains a unitary focus but can be shifted rapidly between the cued memory representation and locations in a representation of the search array. In this view, visual attention may serve a refreshing/rehearsal functional in VWM that can be interrupted periodically to execute shifts of attention during search. Although these alternatives are possible, they are speculative and would need independent support to be considered plausible. In addition, one would need to specify how, mechanistically, visual attention would operate over memory and perceptual representations simultaneously or how it could be efficiently shifted between them.

Nevertheless, we conducted Experiment 4 to test these alternative accounts. Both predict that prioritizing an object in VWM should influence the deployment of attention during the search task. If visual attention must be split or shifted between the cued memory representation and the search array, then search efficiency should be impaired relative to a baseline search condition with no demand to prioritize an object in VWM. To test this prediction, we compared visual search efficiency in two conditions. The dual-task condition was identical to that in Experiment 2 (Figure 3), except set size in the search task was manipulated to estimate search efficiency. The search-only condition included only the search task. Thus, the design was conceptually similar to previous experiments, except we probed the effect of memory prioritization on search rather than the effect of search on memory prioritization.

Participants

Sixteen new University of Iowa undergraduates participated for course credit. All reported normal or corrected-to-normal vision. One failed to perform above chance on the change detection task and was replaced.

Stimuli

The stimuli were the same as those in Experiment 2, with the exception of the search display. The search array was composed of one, two, three, or four pairs of stimuli (set sizes 2, 4, 6, and 8). Set size 8 was identical to the search array used in previous experiments. For set sizes smaller than 8, the assignment of stimulus pairs to the four locations (top, bottom, left, right) was determined randomly.

Procedure

The sequence of events in the dual-task condition was the same as in Experiment 2. In the search-only condition, the memory task events were eliminated: After pressing the pacing button to start the trial, there was a 500-ms delay, followed by the search array for 2000 ms. Search display offset terminated the trial. As in the dual-task condition, the search array remained visible for 2000 ms regardless of when the participant responded. In both conditions, only responses made during the time that the array was visible contributed to the response time measure.

The search-only and dual-task conditions were blocked. Each of the four blocks contained 96 experiment trials. In the dual-task condition, these were divided evenly among the 16 conditions created by the 2 (cue: valid, neutral) × 2 (change: same, changed) × 4 (search set size) design. In the search-only condition, they were divided evenly among the four search set sizes.

Data Analysis

Participants responded to the search display during the 2000 ms that it was visible on 99.1% of trials. Target discrimination accuracy was 98.2%. These measures did not differ between the two tasks (search only, dual-task) or between the valid- and neutral-cue trials of the dual-task condition.

For the color memory task, 0.6% of dual-task trials were eliminated due to a response within 250 ms of the onset of the test display.

Results and Discussion

As in previous experiments, change detection accuracy in the dual-task condition was higher in the valid-cue condition (66.4%) than in the neutral-cue condition (58.5%), F(1,15) = 16.1, p < .001, despite the demand to conduct search during the retention interval. The raw percentage correct data are reported in Table 1.

The search task data were of primary interest. First, we examined search efficiency as a function of memory task (search only, dual task), collapsing across the VWM cuing manipulation in the dual-task condition. The data were entered into a 2 (task) × 4 (set size) repeated measures ANOVA. There was a reliable main effect of set size, F(3,45) = 164.0, p < .001, but no main effect of memory task, F(1,15) = 1.17, p = .30, and no interaction, F < 1. The memory task had no observable impact on visual search. In fact, the numerical trend was toward slightly faster search in the dual-task condition than in the search-only condition. The absence of a memory effect on search was not observed because participants ignored the memory task. Change detection performance was numerically higher in Experiment 4 than in the dual-task trials of Experiment 2, and the cuing advantage was also numerically larger than that observed in Experiment 2.

To provide a test targeted at the cuing manipulation, we estimated search slope (via linear regression) for each participant for three types of trials: search only, dual-task valid, and dual-task neutral. These data are illustrated in Figure 7. Of central importance, mean slope was no higher for dual-task valid trials (36.3 ms/item) than for search-only trials (37.8 ms/item), F < 1, indicating that the prioritization of the cued object did not have any observable influence on search efficiency. We had sufficient power to detect a slope difference of 5.6 ms/item. Further, there was no slope difference between dual-task valid trials and dual-task neutral trials (33.1 ms/item), F < 1. In sum, visual search performance was independent of the VWM cuing manipulation and of the VWM task in general. Therefore, it is highly unlikely that, in Experiments 1–3, participants performed the dual-task condition by dividing or shifting visual attention between the cued item in memory and items in the search array.

Figure 7.

Mean search RT in Experiment 4 as a function of set size and memory condition.

The Experiment 4 results are consistent with two earlier studies that examined the effect of a secondary VWM task on visual search (Woodman, Luck, & Schall, 2007; Woodman, Vogel, & Luck, 2001). A concurrent VWM task interferes with search if the search target changes each trial and VWM is required to update and maintain the relevant target template (Woodman et al., 2007). However, if the search target is consistent across the experiment, the role of VWM in template maintenance is minimized (presumably because the target attribute can be off-loaded to LTM), and a concurrent VWM task does not impair search efficiency (Woodman et al., 2001). In Experiment 4, the search target feature (L/R gap) was consistent across the entire experiment, and search efficiency was unimpaired by a secondary VWM task.

General Discussion

In the present study, we tested the hypothesis that selective maintenance in VWM requires sustaining visual attention on the representation of a task-relevant item. In all four experiments, participants were able to prioritize the retention of a cued item in VWM. However, the effectiveness of prioritization was independent of whether visual attention could or could not be sustained on that item. In addition, prioritization remained robust when spatially overlapping memory items precluded selection by means of attending to a particular location. Finally, the deployment of visual attention during serial search was independent of VWM prioritization. These findings contrast with theories of VWM prioritization that depend on sustained visual attention (Griffin & Nobre, 2003; Makovski et al., 2008; Matsukura et al., 2007), with broader theories that posit a common selective mechanism in visual perception and VWM (Chun, 2011; Gazzaley & Nobre, 2012), and with even broader theoretical approaches that treat attention and working memory as coextensive (Cowan, 1995, 2001). Although VWM and attention certainly interact, they should not be considered as two sides of the same coin.

We have shown that sustained visual attention is not required for VWM prioritization. Perhaps visual attention operates early within the retention interval, before the search task, and the effect of attending to the cued item persists without having to sustain attention on that item during the remainder of the trial. For example, attention might act to transfer a sensory-level representation of the cued object to VWM systems that are resistant to subsequent perceptual interference, such pre-frontal and medial temporal systems (see below). Even if this were the case, however, it would still be inconsistent with the idea that selective maintenance in VWM is equivalent to the deployment of visual attention. The prioritization of the cued object, which continued throughout the search task, was clearly not the result of sustaining attention on that object; the prioritized object and the current locus of visual attention were dissociable. Thus, although we cannot rule out the possibility that visual attention plays a functional role in prioritization at some point during the trial, the data eliminate the hypothesis that selective maintenance in VWM is equivalent to the deployment of visual attention.

The results of Experiment 3 further constrain any account in which selection in VWM is proposed to operate by directing attention to a particular spatial location, as the memory items were not differentiated by position. The Experiment 3 results might be consistent with a sustained-attention account if attention were oriented on the basis of a feature value rather than on the basis of location. However, then one would have to posit that focal attention to a feature value can be implemented simultaneously with focal attention to a series of locations during the search task, as the cuing effect in Experiment 3 was unreduced in the dual-task condition. There is no existing evidence to suggest that this is possible. In vision alone, feature-based selection can be applied independently of spatial selection only under highly limited circumstances (Treue & Martinez Trujillo, 1999; Zhang & Luck, 2009), with several studies suggesting that feature-based selection is generally constrained to the currently selected region of space (Anllo-Vento & Hillyard, 1996; Hillyard & Münte, 1984).

Clearly, prioritizing an item in VWM is an inherently selective operation. Would it not be valid to argue that, whatever the means of prioritization, it should be called “attention”? Certainly, if attention is defined broadly as any selective operation, then the prioritization of a task-relevant object in the present study is an example of attention. Similarly, one could describe the entire VWM system as “attention,” since it maintains a selected subset of task-relevant information. But this does not advance our understanding of the mechanisms involved. Without positing a specific mechanism of selection, describing a given operation as “attention” does no more than restate the fact that the operation is selective. Thus, a broad hypothesis such as, “selection in VWM depends on attention,” is uninformative of mechanism, circular, and unfalsifiable. One can test the hypothesis that selection in VWM depends on the well-defined mechanism of visual attention. One cannot test the hypothesis that selection in VWM depends on attention.

It is important to note that our data do not rule out the possibility that visual attention can be used to prioritize items in VWM. Our data indicate only that prioritization can occur without sustained attention, suggesting, at the very least, that there is an alternative and equally efficient means to implement selective maintenance in VWM. The functional consequence is that VWM prioritization and visual attention can be dissociated, with different objects simultaneously selected in VWM and in visual perception. One possibility for VWM prioritization in the absence of visual attention is that task-relevant VWM representations are preferentially maintained in prefrontal regions classically associated with working memory retention (Courtney, 2004; Goldman-Rakic, 1996), buffered from new perceptual processing in posterior visual areas and thus protected from perceptual interference (Miller, Erickson, & Desimone, 1996). However, this possibility is limited somewhat by recent evidence that prefrontal regions may not represent the content of VWM directly, with their involvement in working memory operations limited to control signals for selective encoding and rehearsal (for a review, see Postle, 2006). A second possibility is that selective maintenance could be achieved by storing task-relevant object information in medial temporal lobe (MTL) systems that would offer robust retention despite changes in low-level sensory events. There is now strong evidence that MTL supports object memory over timescales characteristic of VWM (Barense, Gaffan, & Graham, 2007; Ezzyat & Olson, 2008; Hannula, Tranel, & Cohen, 2006; Olson, Page, Moore, Chatterjee, & Verfaellie, 2006). In addition, evidence that the locations of objects in VWM are coded primarily in an abstract, scene-based coordinate frame (Hollingworth, 2006, 2007; Hollingworth & Rasmussen, 2010; Jiang, Olson, & Chun, 2000; Olson & Marshuetz, 2005) is consistent with a MTL contribution to the type of change detection task used here. Such coding would presumably be a precursor of the LTM representation of objects. Given that spatial activation in MTL systems depends strongly on fixation position in primates (Rolls, 1999), it is at least plausible that visual attention could play a role, early in the retention interval, in the formation of a MTL representation of the cued object.

The present results contribute to a growing body of evidence that visual attention and VWM are often dissociable. Converging evidence comes from a recent study by Maxcey-Richard and Hollingworth (2012). The availability of visual attention was manipulated by requiring participants to make series of saccades after receiving a cue to prioritize a particular item in memory. Specifically, participants directed gaze to real-world objects that appeared sequentially within a depiction of a natural environment. One object was followed by an auditory cue indicating that it was likely to be tested. After the cue, participants executed a series of eye movements to subsequent objects in the series. Because visual attention is mandatorily and exclusively directed to the impending saccade target location (e.g., Hoffman & Subramaniam, 1995), attention could not have been maintained on the cued object as the eyes were directed to subsequent objects in the scene. Yet, robust prioritization was observed, as in the present study. The Maxcey-Richard and Hollingworth method created a situation similar to common real-world tasks, the performance of which often depends on dissociating the item preferentially maintained in VWM from the current locus of visual attention. It appears that people are perfectly capable of selectively remembering one apple while shifting visual attention and gaze to different apples in the bin. Similarly, people should be perfectly capable of selectively maintaining features of the search target in VWM while shifting attention and gaze to a series of candidate objects during visual search.

The Experiment 4 results converge with other studies indicating that visual search efficiency is dissociable from VWM retention (unless frequent template updating is required, Woodman et al., 2007), and thus that the two do not necessarily depend on a common selective mechanism (Woodman et al., 2001). Evidence that attention tends to be attracted to items that match features in VWM (e.g., Soto et al., 2005) has been used to argue for a common selective mechanism in perception and in memory, but there is strong evidence that this relationship can be eliminated or reversed (Downing & Dodds, 2004; Han & Kim, 2009; Woodman & Luck, 2007). Visual attention has been proposed as a common mechanism for feature binding in perception and in VWM (Wheeler & Treisman, 2002), yet robust binding in VWM is observed in the absence of visual attention (Gajewski & Brockmole, 2006; Johnson et al., 2008). Attending to an item and representing that item in VWM are not necessarily equivalent operations.

Conclusion

The idea that selective maintenance in VWM is equivalent to sustained visual attention is attractive, but it neglects the flexibility, complexity, and representational diversity of the systems involved. There are multiple manifestations of selection in visual processing. There are multiple means to store visual information for brief periods of time. Functionally, this allows for dissociation between selective maintenance in VWM and selection in vision: One object can be preferentially retained in VWM as different objects are selected perceptually.

Figure 5.

Sequence of events in a trial of the dual-task condition of Experiment 2. The method differed from Experiment 2 in two key respects: Memory stimuli were presented sequentially at the center, and the cue was a word specifying the relevant color category (e.g. “red”) or all categories (“all”).