The Strategic Retention of Task-Relevant Objects in Visual Working Memory

Abstract

The serial and spatially extended nature of many real-world visual tasks suggests the need for control over the content of VWM. We examined the management of VWM in a task that required participants to prioritize individual objects for retention during scene viewing. There were five principal findings: 1) Strategic retention of task relevant-objects was effective and was dissociable from the current locus of visual attention; 2) strategic retention was implemented by protection from interference rather than by preferential encoding; 3) this prioritization was flexibly transferred to a new object as task demands changed; 4) no-longer-relevant items were efficiently eliminated from VWM; and 5) despite this level of control, attended and fixated objects were consolidated into VWM regardless of task relevance. These results are consistent with a model of VWM control in which each fixated object is automatically encoded into VWM, replacing a portion of the content in VWM. However, task-relevant objects can be selectively protected from replacement.

Initial research on visual working memory (VWM) focused on delineating the basic attributes of the system, with particular emphasis on understanding its capacity and representational format (for a review, see Luck, 2008). Now that the basic representational properties of VWM are fairly well understood, considerable research has turned to the function of VWM (Ballard, Hayhoe, & Pelz, 1995; Hollingworth, Richard, & Luck, 2008; Hyun & Luck, 2007), particularly as the content of VWM has been shown to be directly related to the demands of the current task (Droll & Hayhoe, 2007; Droll, Hayhoe, Triesch, & Sullivan, 2005; for reviews, see Hayhoe, 2008; Tatler & Land, 2011). Perhaps the most important—and certainly the most frequently studied—function of VWM is to enable perceptual comparison when the visual representation of a scene is interrupted by some form of discontinuity. VWM retains object information across disruptions such as saccades, blinks, and occlusion, so that objects can be compared and the correspondence between an object visible before and after the disruption established (Currie, McConkie, Carlson-Radvansky, & Irwin, 2000; Hollingworth & Franconeri, 2009; Hollingworth et al., 2008).

VWM is used to compare two states of an object, but VWM is also necessary to compare two different objects that are separated in space, even if those objects are visible simultaneously. This is particularly applicable to behavior in complex environments. Often, the perceptual details of objects entering into a comparison cannot be perceived simultaneously, requiring a saccade from one object to the next. VWM is needed to store information about the first object so that it can be retained across the perceptual disruption as the eyes are shifted to the second object entering into the comparison. Extended retention of a particular VWM representation is frequently required to support multiple comparisons. For example, choosing the best flower for a bouquet requires selectively retaining visual features of the current “best candidate” flower in VWM as attention is directed to multiple subsequent flowers. Similarly, visual search requires selectively maintaining a target template in VWM as attention and gaze are directed to a series of objects in the scene.

To perform these types of extended, spatially separated comparisons, VWM would need to have four key properties. First, VWM representations must survive shifts of visual attention.¹ Attention is shifted covertly to each object before the eyes fixate it (Deubel & Schneider, 1996; Hoffman & Subramaniam, 1995; Kowler, Anderson, Dosher, & Blaser, 1995). This is both mandatory and exclusive; attention cannot be maintained at one location as a saccade is being prepared to a different location (Deubel & Schneider, 1996; Hoffman & Subramaniam, 1995). If VWM representations did not survive shifts of attention and gaze, one would be unable to perform even a simple perceptual comparison between two sequentially fixated objects; the representation of the first object would not survive the shift of attention to the second object. Despite early claims that VWM representations disintegrate upon the withdrawal of attention (Rensink, 2000; Wheeler & Treisman, 2002), a range of studies has demonstrated robust VWM retention across shifts of attention and gaze (Gajewski & Brockmole, 2006; Hollingworth, 2004; Hollingworth & Henderson, 2002; Irwin & Gordon, 1998; Johnson, Hollingworth, & Luck, 2008). Second, task-relevant objects must be preferentially retained in VWM. For example, the current “best candidate” flower must be selectively retained so that it is not displaced by the processing of subsequent objects as attention and the eyes are shifted to other flowers. Third, selective retention should be flexible. As task demands evolve and the object relevant to comparison operations changes (e.g., the goal of visual search changes), prioritization must be transferred to the newly relevant object. Finally, no-longer-relevant information should be eliminated from VWM. Reallocation of prioritization would be facilitated by the elimination from memory of previously relevant objects.

In the present study, we focused on the mechanism of control serving to selectively retain task-relevant objects. Several studies suggest that after objects are consolidated into VWM, participants can exert selective control over which objects are retained (Astle, Summerfield, Griffin, & Nobre, 2012; Griffin & Nobre, 2003; Landman, Spekreijse, & Lamme, 2003; Makovski, Sussman, & Jiang, 2008; Matsukura & Hollingworth, 2011; Matsukura, Luck, & Vecera, 2007). In these retention-interval cuing studies², participants encode a set of simple objects into VWM in preparation for a change detection test. During the blank retention interval, a spatial cue indicates an item that is likely to be tested, providing incentive to retain this item preferentially. Retention-interval cuing is effective: Cued items can be protected from degradation during the retention interval, leading to improved change detection performance.

Authors in the retention-interval cuing literature have argued that the mechanism of selective maintenance is visual attention (Gazzaley & Nobre, 2012; Griffin & Nobre, 2003; Makovski et al., 2008; Matsukura et al., 2007). The cue allows visual attention to be directed to and sustained upon a representation of the cued object. Directing visual attention to the cued object enhances the perceptual-level representation of that object in the same manner that attention facilitates perceptual processing of sensory input (Gazzaley & Nobre, 2012). In addition, attention has been proposed to protect the cued representation from passive decay (Matsukura et al., 2007), from interference generated by other items stored in VWM (Matsukura et al., 2007), or from interference generated by perceptual processing of the test display (Makovski et al., 2008). These views are consistent with a theoretical commitment to the idea that selection in visual perception and selection on VWM depend on a common mechanism, visual attention (Chun, 2011; Gazzaley & Nobre, 2012). They are also broadly consistent with the claim that VWM and attention constitute a single system (Cowan, 1995).

Attentional accounts provide a plausible explanation for selective maintenance in the retention-interval cuing literature. However, this form of control is unlikely to be functional in real-world visual tasks that depend on VWM, potentially limiting the generality of sustained visual attention as a mechanism of prioritization in VWM. As discussed above, the sequential fixation of objects during real-world viewing makes it impossible to sustain attention on a particular object location, as visual attention precedes the eyes to each object before it is fixated. Our primary goal in the present project was to determine whether task-relevant objects can be selectively retained in VWM as attention and gaze are directed to subsequent objects in a scene. If selective retention is not possible across shifts of attention and gaze, this would confirm a central role for visual attention in the control of VWM, consistent with claims from the retention-interval cuing literature. If task-relevant objects can be selectively retained despite shifts of attention and gaze to other objects, then strategic prioritization does not necessarily require sustained attention, indicating that selection in visual perception and VWM are dissociable.

We addressed four additional issues important for understanding strategic control of VWM. First, we probed the mechanism of prioritization. A task-relevant object could be prioritized either by differential encoding (e.g., by encoding more detail from task-relevant objects) or by differential maintenance (e.g., by protecting task-relevant objects from interference and replacement). Second, we examined control over entry into VWM. One possible means of control would be to block the consolidation of task-irrelevant objects so that they do not interfere with the representations of task-relevant objects. We tested whether objects known to be of low relevance can be denied access to VWM. Third, we probed whether prioritization can be flexibly reallocated from a previously relevant item to a newly relevant item. Finally, we examined whether items that are no longer relevant to the current task are actively eliminated from VWM.

Experiment 1

In the general method, participants viewed a series of objects in a scene and selectively retained one of the objects in memory for a test at the end of the trial. The scene depicted a workshop (see Hollingworth, 2004). On each trial, participants first saw the background scene (Figure 1). Then, a set of objects was presented sequentially within it. Each object was presented at a different location for 1000 ms and then removed. The removal of one object coincided with the presentation of the next object in the series. There were 10 possible objects that could appear in the sequence. Two token versions of each object were created (e.g., two different hammers, two different screwdrivers, etc.). These are displayed in Figure 2. On each trial, between 6 and 10 objects were presented in the sequence (randomly chosen without replacement from the set of 10). The token version of each object was chosen randomly.

Figure 1.

Background workshop scene.

Figure 2.

Panels A and B show the two token versions of the ten manipulated objects (aerosol can, bucket, electric drill, fire extinguisher, hammer, lantern, scissors, screwdriver, watering can, wrench).

A sample portion of the study sequence is displayed in Figure 3. Participants were instructed to generate a saccade to each object as it appeared in the scene, to remain fixated on that object while it was visible, and to shift gaze to the next object when it appeared. At the end of this study sequence, one test object was displayed within a green box in its original position in the scene. The participant responded to indicate whether this object was the same token version as the object presented during the study sequence (e.g., the same screwdriver) or a different token version. Thus, the basic method was a change-detection task in which the to-be-remembered stimuli were presented sequentially within a real-world scene. Token change detection ensured that the method probed memory for the visual details of the objects (Hollingworth, 2003, 2004; Hollingworth & Henderson, 2002).

Figure 3.

Sequence of key events in a trial of Experiment 1. Each trial began with central fixation on the background workshop scene, followed by between one and five randomly selected filler objects presented sequentially (not pictured). The last five objects in the sequence (pictured) could be tested at the end of the trial. Each object was presented for 1000 ms, and its removal coincided with the presentation of the next object in the sequence. In the figure, these objects are numbered 5 through 1, with Object 5 being the fifth object before the test and Object 1 the last object before the test. Each object was accompanied by a tone. A low-pitched tone (black symbol) indicated that the current object was unlikely to be tested. A high-pitched tone (red symbol), indicated that the current object was likely to be tested. After the last object in the sequence was displayed, participants fixated the center of the scene. Finally, the test object was displayed, and participants reposed to indicate “same” or “changed” (different token). In the present example trial, the last five objects before the test were, in order, aerosol can, fire extinguisher, electric drill, hammer, and lantern. A cue-position-2 trial is illustrated, in which the second object before the test was cued by the high-pitched tone. Cuing was valid in this example, as the cued object was tested at the end of the trial. The correct response for this trial was “changed”.

The prioritization of task-relevant objects in VWM was probed using an auditory cuing manipulation. The appearance of each object in the study sequence coincided with a tone. One object in the sequence was cued by a high-pitched tone as ‘to-be-remembered’. All other objects were presented with a low-pitched tone. At the end of the sequence, the high-pitched tone (cued) object was six times more likely to be tested than a low-pitched tone (non-cued) object. Memory for the cued object was compared with object memory at that same serial position in a neutral control condition in which no object was cued. If prioritization in VWM requires sustained visual attention, then memory for the cued item should not differ significantly from memory in the neutral condition. If, however, task-relevant items can be preferentially retained in VWM despite shifts of attention and gaze to subsequent objects, then memory for cued objects should be superior to memory at the same serial position in the neutral condition.

How might participants exert control over VWM to selectively retain the cued object? One possibility is that participants block the entry of unlikely-to-be-tested objects into VWM (Yotsumoto & Sekuler, 2006). If so, then we should observe that objects appearing after the cued object should be remembered poorly, possibly even at chance. Alternatively, object information may automatically enter VWM as a consequence of shifting attention and gaze to the object (Hollingworth et al., 2008; Olson, Moore, & Drowos, 2008). Under this account, each fixated object automatically enters limited-capacity VWM, replacing the memory representation of a previously attended item or items. In the present paradigm, participants might prioritize the cued object by selectively protecting it from being replaced by the consolidation of subsequently fixated objects. Because it is likely that the capacity of VWM during natural scene viewing is limited to no more than two objects, this account predicts superior memory for the object most recently attended and fixated within the scene (as each fixated object is, at least temporarily, maintained in VWM) and for the cued object (strategically protected from replacement).

Method

Participants

Forty participants from the University of Iowa community completed the experiment for course credit or pay. Each was between the ages of 18 and 30 and reported normal vision.

Stimuli

A workshop scene was created from a 3-D model. The 10 objects that could appear within the study sequence were: bucket, watering can, wrench, lantern, scissors, hammer, aerosol can, electric drill, screwdriver, and fire extinguisher. Two token versions of each object were created (see Figure 2). The token versions typically differed in the visual details of the object. For example, one screwdriver had a blue and yellow handle, and the other had a green and black handle. Token versions were created to match closely in size and orientation. Both token versions appeared at the same location within the scene. The objects were rendered so that when they appeared, they were naturally integrated within the scene, and each appeared at a plausible location.

The same background workshop scene and object set was used on every trial. Scene and object repetition does not necessarily introduce significant proactive interference (Hollingworth, 2004; Irwin & Zelinsky, 2002). Scene images subtended 16.9° × 22.8° visual angle at a viewing distance of 70 cm. Target objects subtended 3.3° on average along the longest dimension on the picture plane.

Apparatus

The stimuli were displayed on a 17-in CRT monitor with a refresh rate of 120 Hz. The position of the right eye was monitored by a SR Research Eyelink 1000 eye tracker sampling at 1000 Hz. A chin and forehead rest maintained a viewing distance of 70 cm. Manual responses were collected with a serial button box. The experiment was controlled by E-prime software (Schneider, Eschmann, & Zuccolotto, 2002).

Procedure

After calibrating eye position, all trials began with fixation of a central, green box for 750 ms. None of the objects was visible. During this time, four consonants were played through the computer speakers, randomly selected from the set: C, G, H, J, K, L, Q, R, T, V. This verbal WM load minimized verbal encoding of the objects (Murray, 1968). After a 1000-ms delay, between six and ten objects (randomly selected without replacement) were presented sequentially. The appearance of the first object co-occurred with the disappearance of the green central box. Each object appeared embedded in the scene for 1000 ms and then disappeared simultaneously with the appearance of the next object. Each object was removed from the scene after it had been presented to control the amount of time the participant could fixate and encode each object. If all objects had remained on the screen, it would have been possible for the participant to either revisit cued objects or continue to fixate them after another object appeared. Eye movements were monitored to ensure participants complied with the instructions. The experimenter observed gaze position during each trial and corrected the participant if the participant failed to shift gaze to each object in turn, but this was almost never required, as participants consistently followed the instruction to shift gaze to each new object.

After all of the six to ten objects had been presented, the original background workshop scene was displayed for 1000 ms, and participants directed gaze back to the center. Then, one of the objects appeared in its original location within a green box. This test object was always one of the last five objects presented in the scene. The test object was either identical to the object presented during the trial or a different token version of that object (e.g., a white bucket rather than a green bucket). Participants responded ‘same’ or ‘changed’ by button press. Half of the trials were ‘same’ trials, and half were ‘changed’ trials.

On each trial, 100 ms following the onset of every object, either a low- or a high-pitched tone played for 500 ms. The tone indicated the likelihood that the object would be tested at the end of the trial. Cuing conditions are depicted in Figure 4. A low-pitched tone suggested that the item was not likely to be tested. A high-pitched tone (cue) indicated that the object was ‘to-be-remembered’ and likely to be tested. Although the pitch of the tones was not counterbalanced, Experiment 3 showed that participants can effectively purge items that were once cued by a high-pitched tone, indicating that a high-pitched tone was not simply more arousing than a low-pitched tone.

Figure 4.

Illustration of the three cuing conditions in Experiment 1. In cue-position-2 and cue-position-3 conditions, either the second or third object before the test, respectively, was cued by the high-pitched tone (represented by red border) and had a 60% probability of being the tested item. In the neutral condition, none of the objects was cued, and all five were equally likely to be the tested item.

On 80% of the trials, one object in the sequence was cued by a high-pitched tone. On the remaining 20% of trials (neutral condition) none of the objects was cued (i.e., all were accompanied by a low-pitched tone). When the object was accompanied by a high-pitched tone, it was the object tested on 60% of trials. On the remaining 40% of trials in this condition, the tested object was one of the other four objects appearing before the test (10% each). Thus, the cued object was six times more likely to be tested than any of the other individual objects. On trials containing a high-pitched tone, it accompanied the second (cue condition 2) or third object before the test (cue condition 3). In other words, the high-pitched tone was always followed by one or two items before the test. Participants were unable to predict when the ‘to-be-remembered’ object would occur, because they never knew how many objects would be in the trial, as each sequence randomly varied from six to ten objects.

After the participant responded ‘same’ or ‘changed’, one consonant was played. The consonant was either a member of the original set of four consonants or a different, fifth consonant. Participants responded ‘same’ or changed’. Accuracy was above 89% across all experiments. Accuracy on the verbal memory task was not related to accuracy on the visual memory task, so the visual memory data reported here and in subsequent experiments include all trials. An error in the Experiment 1 program created inaccurate feedback for the verbal memory test on approximately 10% of trials. The error was corrected in Experiment 2. The participant’s response to the test consonant terminated the trial. The next trial was initiated by the experimenter after checking calibration of the eye tracker.

The experiment consisted of 12 practice trials, drawn randomly from the full experimental design, and 100 experiment trials. Eighty of the trials were divided equally between the two cuing conditions (cued object appearing either 2 or 3 objects before the test). Of the 40 trials in each cue condition, 24 trials were valid cue trials, and the remaining 16 trials were invalid trials, equally divided among the other four test positions. The neutral condition consisted of 20 trials, with 4 trials at each of the five test positions. Trials from the different conditions were randomly intermixed. The entire session lasted approximately 50 min.

Results and Discussion

Effects of cue

We first examined whether participants could prioritize task-relevant objects for retention. The results are illustrated in Figure 5. Performance for the cued item was compared with performance at the same serial position in the neutral condition. Accuracy was reliably higher for a cued item at serial position 2 (86.6%) compared with serial position 2 in the neutral condition (73.8%), t(39) = 3.73, p < .001. Accuracy was also reliably higher for a cued item at serial position 3 (81.7%) compared with serial position 3 in the neutral condition (67.5%), t(39) = 3.78, p < .001. Cued items were selectively retained in VWM despite the fact that attention and gaze were shifted to subsequent objects in the sequence.

Figure 5.

Percentage correct on the token change detection task for each cuing condition in Experiment 1. The cue-position-2 condition refers to the condition in which the second object before test was cued as most likely to be tested. The cue-position-3 condition refers to the condition in which the third object before test was cued as most likely to be tested. The neutral condition refers to the condition in which none of the objects was cued as most likely to be tested. Error bars are standard errors of the means.

Next, we assessed memory for the last object in the sequence. When an object was cued, participants knew that the last object was always unlikely to be tested. Yet, the last object in the sequence was remembered at a rate of 85% correct. This was only slightly lower than, and not significantly different from [t(39) = 1.36, p = .18], accuracy in the neutral condition (90% correct). Thus, objects appearing after the cued item were not blocked from entry into VWM. In addition, there was no reliable difference between memory for the cued item and the last item when serial position 2 was cued, t(39) = 0.11, p = .456, or when serial position 3 was cued, t(39) = 0.67, p = .254. The most recently attended object was reliably retained in VWM along with the cued object that participants were strategically prioritizing. Thus, control over VWM was not implemented by blocking the consolidation of subsequently attended objects into VWM. Despite its low probability of being the tested item, the most recently fixated object was reliably maintained in VWM. Rather, these results are consistent with idea that attended and fixated objects are automatically consolidated into VWM (Hollingworth et al., 2008; Olson et al., 2008).

Effects of serial position

In the neutral condition, there was a reliable effect of serial position, F(4,195) = 13.28, p < .001. This effect is consistent with the serial position effects found in other scene memory studies (Hollingworth, 2004; Irwin & Zelinsky, 2002; Melcher, 2006; Tatler, Gilchrist, & Land, 2005; Zelinsky & Loschky, 2005). For cuing trials, accuracy for cued objects at serial position 2 (86.6%) was reliably higher than at serial position 3 (81.7%), t(39) = 2.82, p = .004, indicating some degree of forgetting, even for a cued object.

The results of Experiment 1 demonstrate that task relevant objects can be prioritized for retention in VWM despite the fact that shifts of attention and gaze to other objects precluded sustained attention on the cued item. This provides preliminary evidence that selective retention in VWM does not require sustained visual attention. Before endorsing this conclusion, however, an alternative explanation must be addressed. In the Experiment 1 method, the cue appeared 100 ms after the onset of the object, which remained visible for a further 900 ms. The prioritization observed in Experiment 1 may have been attributable to differential encoding of the cued object rather than to selective retention of an item already consolidated into VWM. Consistent with this possibility, Swallow and Jiang (2010) found that the scene encoding was enhanced when accompanied by a task-irrelevant, but salient, onset stimulus. The salient, high-pitched tone cue in Experiment 1 could have generated a similar selective encoding effect for the cued object.

Experiment 2

To test this alternative account, we contrasted the two cuing conditions depicted in Figure 6. Each object in the sequence was followed by a pattern mask that obscured the object (Figure 7). In the simultaneous-cue condition, the cue was presented while the object was visible (as in the Experiment 1), 100 ms after the appearance of the object. In the post-cue condition, the cue was presented 100 ms after the mask onset. In the post-cue condition, prioritization of the cued object could only operate over an object representation that had already been encoded into VWM, because the cue was presented only after the object had been masked. If prioritization was achieved in Experiment 1 by preferential encoding, then performance should be impaired in the post-cue condition relative to the simultaneous-cue condition. Alternatively, if prioritization was achieved by selective retention of the cued object, performance in the post-cue condition need not be impaired.

Figure 6.

The two cuing conditions in Experiment 2. In the simultaneous-cue condition, the cue was presented 100 ms after the onset of the object (as in Experiment 1). In the post-cue condition, the cue was presented 100 ms after the onset of the mask.

Figure 7.

Illustration of the object masks used in Experiment 2 with the aerosol can (top row) and electric drill (bottom row) as examples.

Method

Participants

Sixteen new participants from the University of Iowa community completed the experiment. They received course credit or were paid. All participants reported normal vision.

Stimuli

The stimuli were identical to Experiment 1 with the following exceptions. The object mask (depicted in Figure 7) was composed of a patchwork of small colored shapes and completely covered each of the objects. Each object mask was rectangular and had an area approximately 20% greater than the objects themselves.

Procedure

The procedure was identical to Experiment 1, with the following exceptions. The presentation of every object was followed by an object mask for 1100 ms at the location of that object (see Figure 7). The mask was then followed by the presentation of the next object. After the last object of the trial, the object mask was followed by the blank scene with the central fixation box for 1000 ms and then the test object until response.

Experiment 2 was divided into two blocks. In the simultaneous-cue block, the tone occurred 100 ms after the onset of each object, as in Experiment 1. In the post-cue block, the tone occurred 100 ms after the onset of each mask. The tone that occurred during the mask signaled the status (likely to be tested, not likely to be tested) of the object currently covered by the mask. Participants were given the relevant cuing instructions at the beginning of each block. Block order was counterbalanced across participant groups. The serial positions that could be tested were limited to the last three objects before test (rather than the last five). Each block consisted of 6 practice trials and 42 experiment trials. All cues were valid, and there was no neutral condition. In both blocks, each of the three last serial positions in the sequence was validly cued and tested on 14 trials.

Results and Discussion

The central issue in Experiment 2 was whether memory accuracy would be impaired in the post-cue condition relative to the simultaneous-cue condition. The data are reported in Figure 8. Overall, performance was very high, exceeding 84% correct in all conditions. There was no main effect of cue type, F < 1, indicating no reliable difference in performance for items cued while visiblet compared with items cued after the object had been obscured by a mask. The absence of an advantage for the simultaneous-cue condition suggests that the prioritization of task relevant objects in this task is implemented primarily after an object has been consolidated in VWM.

Figure 8.

Percentage correct in the token change detection task in each of the cue conditions of Experiment 2. Error bars are standard errors of the means.

Discussion of Experiments 1 and 2

In Experiments 1 and 2, task-relevant objects were prioritized for retention in VWM. This occurred despite the fact that after the cue, participants were required to execute a series of eye movements to subsequent objects in the series. The prioritization effect reflected selective maintenance of the cued object and was not attributable to differences in encoding. It is well established that visual attention precedes the eyes to each saccade target location (Deubel & Schneider, 1996; Hoffman & Subramaniam, 1995). Visual attention could not have been sustained on the cued object or location as participants executed saccades to other objects. The results therefore demonstrate that selective maintenance in VWM can be dissociated from the locus of visual attention. Consequently, selection in visual perception and VWM are unlikely to depend on a single, common mechanism.

The results complement other work showing that the content of VWM is closely associated with the informational demands of the current task (Ballard et al., 1995; Droll & Hayhoe, 2007; Droll et al., 2005). For example, Hayhoe, Bensinger, and Ballard (1998) found that changes to objects were more salient when the changed object was immediately relevant to the task. In these studies, it is likely that task-related selection occurred at encoding, with preferential consolidation of relevant objects into VWM. It is well established that the locus of attention and gaze is directly related to the microstructure of the unfolding task (for a review, see Land & Hayhoe, 2001), and visual attention facilitates the consolidation of perceptual information into VWM (Irwin & Gordon, 1998; Olson et al., 2008; Schmidt, Vogel, Woodman, & Luck, 2002; Sperling, 1960). Subsequent work (Droll et al., 2005) demonstrated that selective encoding extends to task-relevant features of individual objects. Thus, task-related control of VWM is implemented both by selective encoding, as shown in the work of Hayhoe and colleagues, and by selective retention, as shown here and in the literature on retention-interval cuing.

A possible concern with Experiments 1 and 2 is that performance may not have depended entirely on the VWM system and that LTM may have contributed to the task. A second possible concern is that our conclusions depend on evidence from a single scene item and a set of 10 objects. Addressing each of these, the prioritization effects observed here have been replicated in a modified paradigm (Hahn et al., 2012) using an object RSVP stream and a different set of objects. Thus, the results generalize to a different stimulus set and to a paradigm more closely related to traditional VWM tasks.

Experiment 3

In Experiment 3, we examined two additional components of flexible management in VWM: 1) the reallocation of prioritization from a no-longer-relevant object to a newly relevant object and 2) the elimination of the no-longer-relevant item from VWM. Trials included either one or two cues. In the one-cue condition, only the object at serial position 4 was cued. In the two-cue conditions, the first cue at serial position 4 was followed by a second cue at serial position 2 or 3. Participants were instructed that in the event of a second cue, the second cue indicated the object that was most likely to be tested, and the previously cued object was the least likely to be tested of all the objects in the trial. These instructions encouraged participants to utilize the second cue and purge the previously protected object from VWM. Participants could not anticipate whether a given trial would have one or two cues, so they had strong incentive to utilize the first cue. Memory for the originally cued object was tested on a very small proportion of trials, allowing the assessment of memory for the previously prioritized object without providing significant incentive for participants to persist in prioritizing that object.

Method

Participants

Twenty-four new participants from the University of Iowa community completed the experiment. They received course credit or were paid. All participants reported normal vision.

Stimuli

The stimuli were identical to those in Experiment 2.

Procedure

The procedure was identical to Experiment 2 (including the use of post-object masks), with the following exceptions. Each trial consisted of 5 or 6 total objects, randomly selected. The tone (high or low) was presented 100 ms after the onset of each object. All of the last 5 serial positions had the potential to be tested.

The trials were divided between one-cue and two-cue conditions. In both, the first cue was at serial position 4. In the one-cue condition, this was the only item cued. In the two-cue condition, the second cue was either at serial position 3 (described hereafter as second-cue-3), with 0 intervening items between the first and the second cue, or at serial position 2 (second-cue-2), with 1 intervening item between the first and the second cue. Participants were instructed that a high-pitched tone indicated the object most likely to be tested, but if a second high-pitched tone occurred on a trial, that second cued object was now most likely to be tested, and the previously cued object was the least likely to be tested of all the objects.

The experiment consisted of 136 experiment trials and 12 practice trials randomly selected from the full design. Half of the 48 trials in the one-cue condition were valid trials. The other 24 trials were evenly distributed between the other 4 serial positions. The remaining 88 trials were evenly distributed among the two-cue conditions (second-cue-2 and -3). Half of the trials in the two-cue conditions were valid, 9% probed the originally cued item, and 41% were invalid trials probing one of the 3 non-cued locations. In two-cue trials in which the second cue was at serial position 3, serial positions 1, 2 and 5 were the 3 non-cued locations. In two-cue trials in which the second cue was at serial position 2, serial positions 1, 3 and 5 were non-cued locations.

Results

The results are displayed in Figure 9.

Figure 9.

Percentage correct in the token change detection task in each of the cue conditions of Experiment 3. Error bars are standard errors of the means.

Initial prioritization of the first cued object

Before continuing to the main analyses, we determined whether participants attempted to prioritize the first cued object. In the one-cue condition, performance for the initially cued object (serial position 4) was reliably higher than performance at serial position 3 in that condition, t(23) = 2.95, p = .004. Participants successfully prioritized the item at serial position 4 when that item was the only object cued in a trial, and thus we can be confident that in the two-cue conditions, participants initially prioritized the object at serial position 4.

Reallocation of prioritization to the second cued object

To test whether prioritization was successfully transferred to the second cued object, we compared performance at the second cued position with the same serial position on one-cue trials. Performance at serial position 3 was reliably better on second-cue-3 trials (76.3%) compared with one-cue trials (61.8%), t(23) = 3.88, p < .001. Performance at serial position 2 was numerically higher on second-cue-2 trials (77.1%) compared with one-cue trials (72.9%), although the difference did not reach statistical reliability, t(23) = 1.05, p = .153.

As a converging analysis, we compared performance for the second cued object and the first cued object on two-cue trials. In the second-cue-2 condition, performance at the originally cued position 4 was reliably worse than at the second cued position 2, t(23) = 3.37, p = .001. In the second-cue-3 condition, performance at the originally cued position 4, was significantly lower relative to the second cued position 3, t(23) = 2.53, p = .009. Again, these data suggest that prioritization was successfully reallocated to the second cued object.

De-prioritization of the first cued object

Next, we examined the fate of the first cued object on two-cue trials. Memory at serial position 4 was compared for trials with a second cue (de-prioritized) and trials with no second cue (prioritized). Memory for a de-prioritized item was significantly impaired relative to a cued item in the same serial position that was not followed by a second cue, t(23) = 2.84, p = .005. Thus, when the object at serial position 4 remained prioritized, memory was significantly better than when protection was reallocated to a second object, suggesting that at least some of the information that would have been stored from the first cued object was eliminated when prioritization switched to the second cued object. This result highlights the fact that VWM resources are limited, as is the capacity for prioritization: The prioritization of the second cued object was achieved at the expense of memory for the first cued object. This effect was large in absolute terms. Memory performance at serial position 4 was ~75% correct in the one-cue condition but was barely above chance in the two-cue conditions (less than 60% correct), suggesting almost complete purging of the previously prioritized object.

Was the previously prioritized object purged so effectively that memory fell below baseline level? We examined memory for a de-prioritized object relative to objects that were never prioritized in the first place. In the second-cue-2 condition, there was no reliable difference between memory performance for the de-prioritized object (serial position 4) and non-cued items at serial positions 3 [t(23) = 1.56, p = .066)] and 5 [t(23) = 1.43, p = .082]. Similarly, in the second-cue-3 condition, there was no reliable difference between memory for the de-prioritized object (serial position 4) and non-cued items at serial position 5, t(23) = 1.27, p = .108. However, in each of these cases, the numerical trend was toward lower performance for a de-prioritized object compared with non-cued objects at flanking serial positions. Thus, there is suggestive, but certainly not conclusive, evidence that memory for the de-prioritized object was inhibited relative to objects that were never cued.

Discussion

Participants successfully transferred prioritization during a trial from a first cued object to a second cued object. In addition, the no-longer-relevant item was essentially purged from VWM. When an object is no longer relevant to the task, participants were able to “release” the protection afforded to that object. Note that what was “released” cannot be visual attention, as visual attention could not have been sustained on the cued object to begin with.

There are several candidate mechanisms by which no-longer-relevant objects could be eliminated from VWM. First, the object presentation could persist in VWM until overwritten by newly consolidated items. In this view, ultimate removal would be dependent on new perceptual input (Makovski et al., 2008). Second, the object representation might suffer interference from other items already consolidated into VWM (Matsukura et al., 2007). Third, the object representation might be actively eliminated in a manner that is not dependent on competition from currently or newly consolidated items. Consistent with these latter possibilities, Williams and Woodman (2012) demonstrated that participants can eliminate no-longer-relevant information from VWM in the absence of subsequent perceptual input. During the retention interval of a traditional change detection task, participants were given a cue to forget either the left- or right-hand portion of the array. Maintenance activity, as indexed by the contralateral delay activity (CDA) component of the ERP waveform, significantly decreased for the cued hemifield, suggesting the elimination of object information. This occurred in the absence of new perceptual input, making it unlikely that the effect was due to overwriting. Overwriting remains a plausible means of task-related elimination in the present study, however, as there was significant new perceptual input following the second cue.

The results provide converging evidence that the locus of the task-related control in these experiments was VWM. If cuing led to a stronger LTM trace, either actively or passively by virtue of association with a salient high-pitched tone (Swallow & Jiang, 2010), then one would expect that this memory would persist despite subsequent changes in the task. The fact that memory for a de-prioritized object showed no benefit (and perhaps even a deficit) relative to objects that were never cued suggests that memory for the de-prioritized object was initially maintained within a temporary store.

General Discussion

The present study examined how the content of VWM is managed strategically. Participants fixated a series of objects presented sequentially within a workshop scene. Each object was followed by a tone indicating whether the item was likely to be tested or unlikely to be tested. At the end of the sequence, participants responded to indicate whether a test object was the same or a different token version. In Experiment 1, we examined the selective retention of task-relevant objects in VWM when prioritization must bridge shifts of attention and gaze to subsequent objects. Both the cued item and the most recently fixated item were remembered significantly more accurately than other items in the sequence. The former effect demonstrates that strategic retention cannot require sustained visual attention, as attention is allocated exclusively to each object before it is fixated, and participants fixated multiple objects after the cued item. The latter suggests that objects appearing after the cued object were not strategically blocked from entry into VWM. Strategic prioritization of previously attended objects therefore appears to be implemented by protecting task-relevant objects from replacement as newly attended objects are attended and consolidated into VWM.

In Experiment 2, we examined the mechanism of prioritization. There was no significant difference in memory performance between conditions in which the relevant object was cued while visible or cued only after it was masked. Therefore, the prioritization observed here operated primarily via the protection of task-relevant objects in VWM from subsequent interference (Makovski et al., 2008; Matsukura et al., 2007) and was not caused by differences in encoding.

Experiment 3 tested the flexible reallocation of prioritization from an object cued early in the trial to an object cued later in the trial. Participants were able to re-assign prioritization to the second cued item. In addition, participants effectively purged the no-longer-relevant object from VWM, exhibiting quite precise control over VWM content. In fact, there was a trend toward poorer memory for a no-longer-relevant object compared with objects that had never been cued in the first place.

The Relationship between Selection in VWM and Visual Attention

The present work was founded on the assertion that in order to support real-world perceptual comparisons, the selective retention of task-relevant objects in VWM must be dissociable from the locus of visual attention. However, this assumption contrasts with the current consensus in the VWM literature that selective maintenance in VWM requires sustained attention (Griffin & Nobre, 2003; Makovski et al., 2008; Matsukura et al., 2007). It is also conflicts with recent claims that selection in visual perception and VWM depend on a common mechanism: visual attention (Chun, 2011; Gazzaley & Nobre, 2012). In the present experiments, the locus of visual attention, which precedes the eyes to each fixated object, was dissociated from the item to be prioritized in VWM. Nevertheless, robust prioritization was observed. Thus, selective maintenance in VWM does not necessarily require sustained visual attention. Selection in vision and VWM are unlikely to be controlled by a single, shared mechanism.

This is not to say that visual attention and VWM are unrelated. In fact, they interact closely. It is well established that visual attention influences the consolidation of perceptual information into VWM (Irwin & Gordon, 1998; Olson et al., 2008; Schmidt et al., 2002; Sperling, 1960). In addition, the content of VWM biases attention toward items matching currently active features (Hollingworth & Luck, 2009; Olivers, Meijer, & Theeuwes, 2006; Soto, Heinke, Humphreys, & Blanco, 2005; Soto, Humphreys, & Heinke, 2006). Moreover, it is still possible that visual attention can serve to prioritize items in VWM. The present data indicate only that visual attention is not required to do so. Alternative means of prioritization and protection include the selective representation of the cued items in prefrontal regions (Courtney, 2004; Goldman-Rakic, 1996), buffered from the interference generated by new sensory input (Miller, Erickson, & Desimone, 1996), and the selective encoding of cued items in medial temporal regions (Barense, Gaffan, & Graham, 2007; Ezzyat & Olson, 2008; Hannula, Tranel, & Cohen, 2006; Olson, Page, Moore, Chatterjee, & Verfaellie, 2006) that would likewise offer robust retention despite changes in low-level sensory events.

The present findings could be interpreted as inconsistent with general theories of working memory claiming a very close relationship between working memory and attention (Awh & Jonides, 2001; Cowan, 1995, 2001; Jonides et al., 2008; McElree, 2006; Theeuwes, Belopolsky, & Olivers, 2009). Evaluation of such theories is complicated by differences in the definition of attention, however. In the realm of spatial memory (Awh & Jonides, 2001), it is clear that the proposed form of attention functional in memory maintenance is visual attention. And visual attention certainly provides a means to reference external locations after stimulus removal, providing a clear, mechanistic link between attention and memory. In more general theories of WM (Cowan, 1995; Jonides et al., 2008) the attention mechanisms proposed as functional in WM maintenance are not as clearly specified. In some cases, authors appear to implicate visual attention in the maintenance of non-spatial properties of objects. In other cases, attention appears to serve as a generic term for selection, as in accounts of WM maintenance depending on a unitary “focus of attention”.

With respect to theories implicating attention in SpWM maintenance (Awh & Jonides, 2001), the present results are largely orthogonal, as we did not test memory for object location per se. It is certainly possible that sustained visual attention is necessary for maintenance of spatial locations but not for maintenance of the visual features of objects, such as shape and color. However, we think it is unlikely that SpWM maintenance and attention are co-extensive. First, computing spatial relationships will often require shifts of attention between objects in precisely the same manner necessary to compare the visual features of objects (Franconeri, Scimeca, Roth, Helseth, & Kahn, 2012), such as determining which golf ball is closer to the cup or which glass is closer to your plate. A model in which SpWM is coextensive with visual attention (e.g., Theeuwes et al., 2009) would have difficulty accounting for such sequential comparison operations. Further, at least some existing evidence suggests that memory for the locations of objects in WM can be dissociated from visual attention. In Hollingworth (2007) and in Hollingworth & Rasmussen (2010), participants’ memory for the features of objects was bound to particular spatial locations, with superior memory performance if the tested object was presented at its original location within the array. Critically, these position memory effects survived spatial transformations of the array, such as whole-array translation and systematic expansion (see also Olson & Marshuetz, 2005). With such transformation, visual attention could not have been directed continuously to a particular target location or locations, because the absolute locations changed unpredictably from study to test. Thus, there appears to be multiple means of coding spatial location in VWM (Hollingworth & Rasmussen, 2010) at least one of which cannot be explained by sustained visual attention.

In more general theories of working memory, the role of attention is less clearly defined. One prominent theory holds that WM involves an internal “focus of attention” that can select one or several items at a time and that, at the broadest level, attention and WM should be considered a single entity (Cowan, 1995, 2001). This account runs into problems of circularity, however. All working memory systems have a limited capacity, and working memory content is directly relevant to ongoing task demands. These properties—limited capacity, selectivity—are the properties that define an attentional system; any system that instantiates these processes can be validly called attention. Without specifying a particular mechanism of selection, however, invoking the concept of attention does no explanatory work; it just restates the fact that VWM is selective. One can escape this circularity only by specifying a particular, well-defined form of attentional selection as operating over VWM representations. Here, we have demonstrated that the specific system of visual attention can be dissociated from selection in VWM. This disconfirms any broad account of VWM and attention in which attention is conceived as a unitary operation that is equivalent to selection within VWM. One might argue that there may be other forms of attention that operate over VWM representations to strategically maintain task-relevant information. However, the burden is upon researchers to specify which particular attentional systems are in involved and the mechanisms by which they implement prioritization.

Rehearsal in VWM

To what extent can selective maintenance in VWM be attributed to explicit rehearsal of a cued object? Rehearsal is observed in the literature on spatial working memory (for a review, see Awh & Jonides, 2001), with attention shifted to particular remembered locations (Awh, Jonides, & Reuter-Lorenz, 1998). Because locations in the world are referenced by motor systems that operate over space (Logie, 1995), spatial memory rehearsal can be supported by motor simulation, implemented by covert shifts of attention (i.e., a motor plan for an eye movement Awh, Armstrong, & Moore, 2006; Rizzolatti, Riggio, Dascola, & Umilta, 1987), overt shifts of the eyes (Tremblay, Saint-Aubin, & Jalbert, 2006), or even pointing. Similar motor simulation is available for the rehearsal of verbal information via articulatory mechanisms (Baddeley, 1966).

However, there is no such motor simulation available for many of the visual features that comprise object representations in VWM. There is no motor system that can simulate or generate color (without adding an artificial form of external representation, such a set of paints). There is no motor system that can easily simulate or generate a complex, 3-D shape. Thus, rehearsal, in the sense of sequential activation of motor representations corresponding to the content of remembered stimuli, is not generally available for object representations in VWM. The absence of a rehearsal mechanism for visual stimuli is consistent with evidence from studies manipulating serial position. Although recency effects are observed consistently, primacy effects, which would suggest a rehearsal benefit for early items, are rarely observed (Broadbent & Broadbent, 1981; Hollingworth, 2004; Phillips, 1983; Phillips & Christie, 1977; Potter & Levy, 1969).

If the objects in the present task had been visible throughout the trial and eye movements had not been controlled, participants might have refixated the cued object during the trial so as to strengthen or refresh their memory for that object, constituting a form of rehearsal (Zelinsky, Loschky, & Dickinson, 2011). Similarly, in real-world tasks such as choosing the best flower for a bouquet, participants might periodically direct gaze back to the current best candidate flower in order to refresh the representation of that flower and facilitate comparison with subsequent objects. Such refixations would not constitute a rehearsal mechanism within memory; the process would depend on resampling the original stimulus rather than reactivating an existing memory representation (akin to asking someone to repeat a phone number). And refixation of an object would most likely reflect failure to retain that object robustly in VWM, generating the need to resample the original stimulus. However, such a process could certainly operate in a manner that was functionally similar to a rehearsal mechanism, enabling the sustained, active maintenance of task relevant objects (Zelinsky et al., 2011). Perceptual resampling is particularly applicable within vision, because the visual scene often remains available throughout an extended event (Ballard et al., 1995; O’Regan, 1992), in contrast with auditory-linguistic stimuli that are typically available for only a short duration. Although perceptual resampling could not have contributed to the present prioritization effects (because each object was removed from the scene after presentation), future research will address the relative contributions of and coordination between perceptual resampling and the type of memorial prioritization observed here.

Conclusion

The strategic management of VWM is critical to support everyday visual tasks. Real-world visual comparisons of objects at different locations, which involve sequential shifts of attention and the eyes, would be difficult to perform, if not impossible to perform, without some degree of top-down control of VWM to ensure that the object relevant to the comparison is strategically retained. Here, we illuminated several key properties of strategic control over VWM content during real-world scene viewing. First, task-relevant objects are prioritized for retention in VWM with minimal interference from subsequent perceptual information. Second, this prioritization can be dissociated from the current locus of visual attention. Third, prioritization is not implemented by blocking consolidation of subsequently attended objects; fixated objects were consolidated into VWM regardless of task relevance. Fourth, selective retention is implemented via protection of memory representations from interference generated by subsequent sensory input. Fifth, prioritization can be effectively reallocated to newly relevant items and removed from previously cued items.