Flexibility in Visual Working Memory: Accurate Change Detection in the Face of Irrelevant Variations in Position

Abstract

Many recent studies of visual working memory have used change-detection tasks in which subjects view sequential displays and are asked to report whether they are identical or if one object has changed. A key question is whether the memory system used to perform this task is sufficiently flexible to detect changes in object identity independent of spatial transformations, but previous research has yielded contradictory results. To address this issue, the present study compared standard change-detection tasks with tasks in which the objects varied in size or position between successive arrays. Performance was nearly identical across the standard and transformed tasks unless the task implicitly encouraged spatial encoding. These results resolve the discrepancies in prior studies and demonstrate that the visual working memory system can detect changes in object identity across spatial transformations.

By definition, a working memory system must be able to flexibly manipulate and work with stored information (Baddeley, 1986; 1996; 2007; Engle, Kane, & Tuholski, 1999; Just & Carpenter, 1992). Without flexibility, a short-term memory system would be of limited value in real-world cognitive tasks. In the verbal domain, the manipulative aspect of working memory has been stressed in studies that assess the role of verbal short-term memory in tasks such as language comprehension and problem solving, which clearly require information manipulation (Baddeley & Hitch, 1974; Baddeley, Logie, Nimmo-Smith, & Brereton, 1985; Daneman & Carpenter, 1980; Hitch & Baddeley, 1976). In contrast, many studies of visual working memory have used change-detection tasks that do not explicitly require any abstraction or manipulation of information (e.g., Luck & Vogel, 1997; Pashler, 1988; Phillips, 1974; Rensink, O’Regan, & Clark, 1997; Simons & Levin, 1997; Vogel, Woodman, & Luck, 2001).

In our own previous studies, for example, subjects were presented with a brief sample array of several colored rectangles that was followed by a short delay and then a test array that was either identical to the sample array or differed in terms of one feature of one of the objects (e.g., its color -- see Luck & Vogel, 1997; Schmidt, Vogel, Woodman, & Luck, 2002; Vogel & Machizawa, 2004; Vogel et al., 2001; Vogel, Woodman, & Luck, 2006). Subjects were simply required to report whether there was a change between the sample and test arrays. Other laboratories have used more naturalistic stimuli, such as photographs of real-world scenes, and have asked subjects to detect changes such as the addition or deletion of an object (Levin & Simons, 1997; Rensink et al., 1997; Simons & Levin, 1997; Simons & Rensink, 2005). To be useful in the natural environment, however, visual working memory must be sufficiently flexible to deal with spatial transformations of the objects in the environment, such as the in-plane translations that arise from eye movements, the expansion and contraction that arise from changes in viewer distance, and the unpredictable shifts in location that arise from movement of the actual objects.

The goal of the present study was to determine whether performance in change-detection tasks reflects a flexible and manipulable working memory representation. An affirmative answer to this question would have two important implications. First, it would indicate that the many interesting results that have been obtained from change-detection experiments are not simply laboratory curiosities, but instead reflect the characteristics of a fundamentally important cognitive system. Second, it would indicate that humans possess a flexible, abstract, and manipulable visual working memory system that can hold multiple objects simultaneously and compare them with incoming sensory information.

Previous studies of the effects of spatial transformations on change-detection performance have yielded contradictory results. Jiang, Olson, & Chun (2000) had subjects perform changed-detection tasks in which the spatial locations of the to-be-remembered objects were scrambled between the sample and test arrays. They reported that these task-irrelevant changes in spatial location interfered with the accurate change detection of the task relevant features (i.e., color). These findings have recently been drawn into question by a study of Logie, Brockmole, & Jaswal (2011), who also had subjects perform an object change-detection task in the face of possible task-irrelevant location changes. Logie and colleagues (2011) found that although the irrelevant location changes resulted in reduced change-detection accuracy when the test array was presented within 1000 ms of the sample array, longer retention intervals exhibited no such interference. These findings appear to be consistent with classic work (Phillips, 1974) as well as more recent studies demonstrating the flexibility of memory after the icon has faded (e.g., Brockmole & Wang, 2003). The present study sought to reconcile these discrepant findings and determine whether visual working memory is sufficiently flexible to allow successful change detection performance in the face of spatial transformations.

To accomplish this, we conducted change-detection experiments in which the sizes and spatial locations of the objects varied between the sample and test displays. Specifically, subjects viewed a sample array consisting of colored squares, and this was followed by a test array in which the sizes or locations of the squares might be the same or different. The subjects were required to ignore any variations in size or position and to indicate whether one of the squares changed in color between the sample and test arrays. If a flexible visual working memory system is used to detect color changes, then it should be possible for subjects to perform this task accurately even when the objects have changed in size or position. In contrast, if change detection is based on a low-level comparison between the sample and test arrays, then performance should be severely disrupted by changes in size and position.

Previous studies of visual working memory have examined the ability of subjects to compare memory representations with new stimuli that differ in location or 3-dimensional orientation (e.g., Carlson-Radvansky, 1999; Carlson-Radvansky & Irwin, 1995; Farah, Rochlin, & Klein, 1994; Irwin, Zacks, & Brown, 1990; Larsen & Bundesen, 1978; Lueschow, Miller, & Desimone, 1994; Phillips, 1974). For example, Lueschow, Miller and Desimone (1994) found that monkeys were able to accurately match a sample stimulus stored in working memory with a test stimulus that was presented at a different location or in a different size; moreover, this behavioral performance was paralleled by memory-related activity in inferotemporal cortex neurons. Similarly, behavioral studies with humans have demonstrated that subjects can accurately detect a change in the identity of a single complex object even when the object moves or undergoes a 3-dimensional rotation between viewings (Carlson-Radvansky, 1999; Carlson-Radvansky & Irwin, 1995; Farah et al., 1994; Irwin et al., 1990). However, these previous studies have involved remembering only one object (or a dot pattern that may be treated as a single object), and the results of these experiments may not generalize to change-detection tasks with multiple objects. The present study, in contrast, involves arrays containing multiple distinct objects.

Experiment 1

Experiment 1 examined performance in a change-detection task in which subjects were instructed to detect changes in the colors of objects and to ignore changes in object size. As illustrated in Figure 1A, the sizes of the objects were identical in the sample and test arrays on half of the trials, but on the other half the entire test array was decreased in size by 50%. Note that the absolute positions of the objects changed as well as their sizes, but this is exactly what happens in the real world when the distance between the viewer and the scene is changed.¹

Figure 1.

Stimuli and results of Experiment 1. (A) Examples of the sample array and the two types of test arrays (normal and halfsize) used in Experiment 1. Fill patterns are used to represent different colors. (B) Results from Experiment 1 (error bars show the 95% within subjects confidence intervals, as described by Lofus and Loftus, 1988, in this and subsequent figures).

Method

Participants

The subjects were 10 undergraduate students of the University of Iowa between the ages of 18 and 35 who received course credit for participating. All reported having normal or corrected-to-normal visual acuity and normal color vision.

Stimuli

The stimuli were colored squares (0.65° × 0.65°) presented on a video monitor at a viewing distance of 70 cm. The color of each square was selected at random (with one replacement) from a set of 7 highly discriminable colors (red, blue, violet, green, yellow, black, and white; see Vogel, Woodman, & Luck, 2001, for the precise color coordinates). At most there could be only two items of the same color in any array and the program insured that when a color change occurred on a trial the square always changed to a different color. The squares were randomly distributed within a 9.8° × 7.3° region with a minimum inter-object distance of 2° (center-to-center). The background color of the video monitor was gray (8.2 cd/m²).

Procedure

Each trial began with a 100-ms presentation of a sample array, which contained 2, 4, or 8 colored squares; we refer to the number of squares in the array as the set size. The sample array was followed by a blank 900-ms retention interval and then a 2000-ms test array. On half of the trials, the sizes and locations of the squares were identical in the sample and test arrays; these are called baseline trials. On the remaining trials, the entire array was reduced in size by 50% (i.e., both the individual items and the inter-item distances were reduced in size by 50%); these are called half-size trials. The baseline and half-size trials were subdivided further into no-change trials, in which the colors remained the same between the sample and test arrays, and change trials, in which the color of one of the items changed to a different, randomly selected color. Change and no-change trials were equiprobable. Subjects were instructed to press one of two keys on a game pad to indicate whether they detected a color change. Accuracy of the discrimination between change and no-change trials was stressed.

To minimize contributions from verbal working memory, subjects performed an articulatory suppression task. Specifically, two randomly selected digits were presented on the video monitor at the beginning of each trial, 1500 ms before the onset of the sample array, and the subjects were instructed to repeat these numbers aloud at a rate of 3–4 digits per second until the end of the trial. Previous research has shown that this task prevents subjects from recoding and storing visual information verbally (Baddeley, 1986; Besner, Davies, & Daniels, 1981; Murray, 1968).

Subjects were given a short block of 10–15 practice trials at the beginning of the session. The main experiment consisted of 30 baseline and 30 half-size trials for each of the 3 set sizes, and these trials were subdivided into 3 blocks separated by short rest periods. All trial types were randomly intermixed within each trial block.

Data Analysis

The primary dependent variable was percent correct, collapsed across change and no-change trials. We also computed A′, a measure of signal detection sensitivity (Macmillan & Creelman, 1991). The pattern of findings was the same for percent correct and A′ in this and all subsequent experiments, and the A′ results are therefore not provided. Although previous studies of change-detection performance have tended to only report response accuracy because observers responses were not speeded by instruction (e.g., Vogel et al., 2001), we also analyzed and report the mean reaction time for correct same and different responses across each condition. The instructions to participants in each experiment of this study stressed accuracy only, however, they were allowed to stop the verbal articulation suppression task after making their button-press response on each trial so motivation for responding as quickly as possible did exist. The data were analyzed using within-subjects analyses of variance (ANOVA) and confidence-interval analyses.

Results and Discussion

Accuracy

The accuracy results of Experiment 1 are shown in Figure 1B. Accuracy (percent correct over both change and no-change trials) decreased as set size increased for both baseline and half-size trials, and accuracy was approximately equal for these two trial types. These observations were supported by an ANOVA with factors of trial type (baseline versus half-size) and set size (2, 4, or 8 objects). This analysis yielded a significant main effect of set size (F(2,18) = 107.78, p < .001). Neither the main effect of trial type nor the Trial Type X Set Size interaction was significant (Fs < 1).

Reaction Time

The mean reaction times (RTs) presented in Table 1 provide two notable findings that support and extend the conclusions drawn from the accuracy data. First, the mean RTs for different responses were highly similar across trial types suggesting that participants could efficiently detect changes in color regardless of irrelevant scaling of the group of items in the test array. This mirrors the pattern observed in the accuracy data. However, the RTs of responses were elevated by irrelevant size changes in test array particularly at the largest set size. Specifically, responses were faster on baseline trials only for different responses at the largest set size. These observations were supported by the statistical analysis of the RTs that yielded a significant interaction of trial type (baseline versus half size) x set size (2 versus 4 versus 8) x response (same versus different), F (1,9) = 4.71, p < .05. With the exception that the main effect of set size was significant, F (2,18) = 26.55, p = .0001, there were no other significant main effects or interactions (all ps > .15). Thus, the RT results are consistent with the accuracy data in suggesting that irrelevant size changes result in minimal interference with the detection of changes in color, with the exception of slowing a subset of the responses at the largest set sizes.

Table 1.

Mean RT (ms +/− SEM) from Experiment 1

	Set Size	Same Response Mean	Different Response Mean
Baseline	2	761.9 +/− 64.3	788.4 +/− 46.7
Baseline	4	825.1 +/− 69.2	897.3 +/− 54.4
Baseline	8	984.7 +/− 114.7	1005.8 +/− 73.2
Half size	2	796.8 +/− 52.7	796.4 +/− 45.5
Half size	4	879.7 +/− 64.8	899.5 +/− 51.7
Half size	8	970.1 +/− 100.9	1153.4 +/− 60.4

These results demonstrate that change-detection performance relies on a memory system that can disregard task-irrelevant variations in the size of the stimulus array from one viewing to the next. This is exactly the sort of memory system that would be useful in natural settings, in which the size of objects will vary as the distance between observer and objects changes. Moreover, these results demonstrate that size-invariant memory performance is possible even when multiple items must be remembered (e.g., mean accuracy was greater than 90% for a set size of 4 objects).

Experiment 2

Translational (picture-plane) movements of objects are also very common in natural settings, and Experiment 2 was designed to determine whether subjects can accurately detect changes in object colors when the objects undergo translational shifts between the sample and test arrays. In this experiment, each array contained 4 squares, and they were confined to either the left or right hemifield. Four types of trials were tested, as illustrated in Figure 2A. On baseline trials, the positions of the objects were identical for the sample and test arrays. On shift trials, the sample and test arrays were presented in different hemifields, but the relative positions of the items remained constant; this condition simulates the effect of an eye movement, which causes an equal displacement of all items across the retina. On scramble trials, the sample and test arrays were both presented in the same hemifield, but the items were presented in different sets of positions for the sample and test arrays; this condition simulates movements of the individual objects. On shift-and-scramble trials, the items were both shifted into the other hemifield and randomized within this hemifield; this condition simply completes the 2 X 2 (shift X scramble) factorial design.

Figure 2.

Stimuli and results of Experiment 2. (A) Examples of the sample array and the four types of test arrays from Experiment 2. Fill patterns are used to represent different colors as in Figure 1. (B) Accuracy results from Experiment 2. (C) RT results from Experiment 2.

Previous studies by Irwin, Carlson-Radvansky, and colleagues have already demonstrated that change-detection performance for individual objects is not impaired by translational movements such as those tested on the shift trials of the present experiment (Carlson-Radvansky, 1999; Carlson-Radvansky & Irwin, 1995; Irwin et al., 1990). The most interesting conditions are therefore the scramble and shift-and-scramble trials. These trials provide a more demanding test of the flexibility of visual working memory, because they test the observers’ ability to keep track of independently moving objects. In addition, these trials also test the hypothesis that the items in the arrays are treated as individual object representations that are at least partially independent of both absolute and relative spatial location (as proposed by Vogel et al., 2001).

Jiang, Olson, & Chun (2000) conducted a similar experiment and found that scrambling the locations led to impairment in change-detection performance. We conducted the present experiment before that study was published, but obtained a very different pattern of results. The reasons for the discrepancy will be explored in Experiment 3.

Method

The method of Experiment 2 was identical except as noted below. A new group of 10 undergraduate students from the same pool participated after providing informed consent. The sample arrays consisted of 4 squares randomly distributed within a 3.25° × 7.28° region centered 3.25° to the left or right of the center of the monitor (see Figure 2). The minimum inter-object distance was 2° (center-to-center).

Each trial began with a 100-ms presentation of a sample array. The sample array was followed by a blank 900-ms retention interval and then a 2000-ms test array. On baseline trials, the spatial positions of the squares were identical in the sample and test arrays. On shift trials, the test-array items appeared on the opposite side of the monitor screen from the sample-array items, but appeared in the same locations relative to one another. On scramble trials, the test-array items appeared at a new set of randomly selected locations in the same hemifield as the sample array. On shift-and-scramble trials, the test-array items appeared at randomly selected locations on the opposite side of the monitor from the sample-array items. Change and no-change trials occurred with equal probability for all 4 of these trial types. The stimulus arrays in Experiment 2 never contained repetitions of a color (i.e., the colors were selected without replacement), and on change trials, the changed item was presented in a color that was absent from the sample array. This was necessary for the scramble and shift-and-scramble trials because it would not be possible to know which item in the test array corresponded to a given item in the sample array if color repetitions were allowed.

Participants were instructed to press one of two keys on a game pad at the end of each trial to indicate whether they detected a color change. Accuracy was stressed, although response times were also collected and analyzed. Each of the 4 conditions was tested in a separate block of 40 trials, and the order of blocks was randomized across participants. Participants received 10–15 practice trials at the beginning of each condition. As in Experiment 1, participants performed an articulatory suppression task during each trial.

Results and Discussion

Accuracy

As illustrated in Figure 2B, performance was approximately the same across all four conditions, with mean accuracy ranging between 86% and 90% correct. Average accuracy was very slightly lower in the two conditions that involved scrambling. An ANOVA with factors of shifting (same-versus opposite-hemifield) and scrambling (same versus different relative locations) yielded no significant main effects or interactions (all ps > .10). Thus, we found no evidence that scrambling the locations led to a reliable impairment in change-detection performance.

The logic of null hypothesis significance testing does not make it possible to conclude that there was no impairment of performance in the scrambled conditions relative to the baseline condition. However, it is possible to show that any differences among these conditions were small. To do this, we computed the difference in accuracy between the baseline condition and each of the two scramble conditions for each participant. Using these differences, we computed 95% confidence intervals for the scramble and shift-and-scramble difference scores. The mean difference for the scramble condition was 3.5% with a 95% confidence interval of +/− 3.1%. In other words, we have 95% confidence that the effect of scrambling was between 0.4% and 6.6%. The mean difference in the shift-and-scramble condition was 2.0 +/− 3.2%, or a range of −1.2% −5.2%. These results indicate that any effects of scrambling are quite small (much smaller than the differences reported by Jiang et al., 2000).

Reaction Time

The mean RTs are shown in Figure 2C and several aspects of these findings are particularly notable. First, the mean RTs for different responses were virtually identical across conditions suggesting that participants could efficiently detect changes in color regardless of irrelevant location changes. This pattern mirrors the pattern observed in the accuracy data. However, correct same responses were elevated by irrelevant changes in location. Specifically, same responses were faster than different responses in the baseline condition but slower than different responses when irrelevant location changes occurred. These observations were supported by the statistical analysis of the RTs that yielded a significant interaction of response (same versus different) x scrambling (unscrambled versus scrambled), F (1,9) = 18.58, p < .01. With the exception that the main effect of scrambling approached significance, F (1,9) = 4.87, p = .055, there were no other significant main effects or interactions (all ps > .15). Thus, the RT results are consistent with the accuracy data in suggesting that irrelevant location changes do not interfere with the detection of changes in color. Moreover, the finding that subjects’ responded same more slowly in the face of an irrelevant spatial change indicates that the observers’ detected the location changes although they could effectively ignore them so as to not disrupt the speed or accuracy of color change detection.

These results demonstrate that subjects can accurately detect changes in object properties regardless of the absolute and relative spatial locations of the objects from one view to the next. We therefore conclude that change-detection performance relies on a flexible visual working memory representation. These results also support the proposal that the representations of objects in visual working memory are relatively independent of each other, because accuracy was essentially the same whether the objects moved as a group or independently. Moreover, these results suggest that representations in visual working memory are not tightly bound to either absolute or relative locations, but are instead relatively abstract. Alternatively, it is possible to explain our findings by proposing a location-based representational format, but with a comparison mechanism that can efficiently match or transform the memory representations at specific locations to the new locations of the objects in the test array. We had expected that if a comparison mechanism was operating by transforming the remembered array to match with the spatially scrambled test array, that this would have resulted in greater slowing of the responses on the scrambled trials relative to the baseline trials. This would be akin to the manipulation process hypothesized to operate during mental rotation to spatial transform object representations (e.g., Shepard & Metzler, 1971). We want to emphasize that the absence of such an RT effect in Experiment 2 is not sufficient to rule out this process explanation and we will return to this issue in Experiment 3.

As described above, the present findings are consistent with the recent study of Logie and colleagues (Logie et al., 2011) that used different manipulations, but also showed that changes in spatial locations do not necessary disturb object representations in visual working memory with retention intervals greater than 1000 ms. However, the findings in Experiment 2 do appear to contradict those of Jiang, Olson, and Chun (2000) in which it was shown that scrambling the locations of objects did result in reductions in change detection performance. In Experiment 3, we sought to reconcile the findings of Jiang et al. with those of the present study.

Experiment 3

The findings of Experiment 2 conflict with results reported by Jiang et al. (2000), who found that participants were significantly impaired at detecting changes in the colors of items when the relative locations of those items changed during the retention interval. There were several differences between the experiments of Jiang and colleagues and Experiment 2 of the present study, but we propose that the key difference is that the Jiang et al. experiments implicitly encouraged the participants to encode the locations of the objects. For example, in the critical experiments of Jiang et al., a cue box surrounded one item in the test array, indicating the one item that might have changed. Although a cue box such as this is useful for eliminating the need for participants to make same-different judgments for each item (see Vogel et al., 2001, for an extensive discussion), it requires that they make use of spatial information. It should not be surprising that disrupting the spatial organization of the stimuli would impair performance if the participants were implicitly encouraged to encode the locations of the objects.

Experiment 3 directly tested this proposal. Specifically, we compared the baseline and scramble conditions of Experiment 2 with baseline and scramble conditions in which a box surrounded the location of one item in the test array. When a box was present in the array, participants were informed that it surrounded the one item that might have changed color, and they were instructed to limit their decisions to this one item. We call this the spatial-cue test condition, and we call the condition without the box the whole-report test condition. We predicted that performance would be equally accurate for baseline and scrambled arrays in the whole-report test condition, replicating the results of Experiment 1, and that performance would be impaired for the scrambled arrays in the spatial-cue condition, due to the implicit spatial demands of this condition.

Method

The stimuli and procedure were identical to those of the baseline and scramble conditions of Experiment 2 with the following exceptions. A new group of ten participants from the same participant pool participated in this experiment. As illustrated in Figure 3B, each subject participated in two different cuing conditions, whole-report test and spatial-cue test, which were tested in separate blocks of trials. The whole-report test conditions were identical to the baseline and scramble conditions of Experiment 1, except that the stimuli were presented within a 7.28° × 7.28° region in the center of the monitor rather than being restricted to one hemifield. The spatial-cue conditions were identical to the whole-report conditions, except that a spatial cue was present in the test array and indicated the one item that might have changed in color on that trial. This cue was a brown outlined square (x = 0.571, y = 0.385, luminance = 13.38 cd/m²) that surrounded the cued item. Each participant performed one block of 48 trials in each condition, and the order of conditions was counterbalanced across participants.

Figure 3.

Stimuli and results of Experiment 3. (A) Example of the sample arrays used and the two types of test arrays used in the different blocks of trials. (B) Accuracy results from Experiment 3. (C) The RT results from Experiment 3 by condition (spatial cue versus whole report) array type (baseline versus scrambled locations) and whether the test array and response was same or different.

Results and Discussion

Accuracy

As illustrated in Figure 3B, scrambling the locations led to a 13% decrease in accuracy relative to baseline in the spatial-cue test condition, whereas scrambling led to only a 4% decrease in the whole-report condition. An ANOVA with factors of test condition (spatial cue versus whole report) and scrambling (baseline versus scrambled) yielded significant main effects of test condition, F(1,9) = 17.15, p < .001, and scrambling, F(1,9) = 23.46, p < .001. Both of these main effects were primarily a result of the lower accuracy for scrambled trials in the spatial-cue test condition, as indicated by a significant interaction between test condition and scrambling, F(1,9) = 8.20, p < .05. Planned comparisons supported the observation that there was a highly significant effect of scrambling (same locations versus scrambled locations) in the spatial-cue condition, F(1,9) = 18.87, p < .001. The small effect of scrambling in the whole-report condition also reached significance, F(1,9) = 8.55, p < .05.

These results are consistent with the hypothesis that the use of a spatial cue in the test array implicitly encourages the use of location information during the color change-detection task, leading to impaired performance when the location information is then disrupted. In contrast, participants can largely ignore information about the spatial locations provided that the change-detection task does not explicitly or implicitly emphasize the use of spatial information. It must be acknowledged that scrambling the locations led to a small decline in performance even when no spatial cue was used (a similar but nonsignificant effect was observed in Experiment 2). However, it is still remarkable that observers are only slightly disrupted by a large change in the absolute and relative locations of the objects.

Reaction Time

The mean RTs from Experiment 3 are shown in Figure 3C. As in Experiment 2, the effect of scrambling was qualitatively different upon same and different responses. As before, same responses were slowed by an average of 213.6 ms when locations were scrambled (collapsed across spatial-cue and whole-report conditions) as compared with a mean 65.9 ms slowing of the different responses. This large effect of spatial scrambling is notable because this is precisely what we would expect if a spatial transformation process resembling mental rotation were being used to compare the scrambled array to the representations held in visual working memory. Consistent with this observation, the interaction of response (same versus different) and scrambling (same locations versus different) neared significance, F (1,9) = 4.89, p = .054. In addition, responses were faster in the whole-report condition (M = 1079.9 ms) than in the spatial-cue condition (M = 1250.9 ms) leading to a significant main effect of test condition, F (1,9) = 15.28, p < .01, and indicating that participants responded less accurately and more slowly in the spatial-cue condition. Finally, due to slower mean RTs in the scrambled (M = 1235.3 ms) compared to the baseline condition (M = 1095.5 ms), the main effect of scrambling was significant, F (1,9) = 28.47, p < .001. Thus, the RT data support the accuracy data in indicating that changes of color are more efficiently detected in the whole-report than the spatial-cue condition with the effect of spatial scrambling being exaggerated by these spatial cues at test.

In summary, both the accuracy and RT findings indicate that detecting color changes is made much more difficult by testing observers’ memory using a spatial cue suggesting that observers are strategically binding object identity and location information even though this proves costly on trials in which the locations change. The use of spatial cues at test in Experiment 3 also resulted in RT effects that provide positive support for an account in which the effects of changes in spatial location are overcome by a comparison mechanism which can spatially manipulate the information in visual working memory to compare with items in a test array that appear at new locations. This finding suggests that the flexibility exhibited by visual working memory may be a feature of the mechanisms that utilize the stored information.

Experiment 4a

In Experiments 1 through 3 we demonstrated that observers can detect changes in the colors of multiple objects accurately even when the locations of the objects change drastically between the sample and test displays. This is one form of evidence supporting the hypothesis that object representations in visual working memory can be handled flexibly and are not fixed to specific spatial locations. In Experiment 4, we used a different approach to provide converging evidence for this hypothesis. Specifically, we presented the to-be-remembered objects sequentially at a single location to remove spatial differences from the to-be-remembered objects.

If color memory is truly flexible and independent of irrelevant features, like location (e.g., Woodman & Vogel, 2008), then it should not matter whether the items are presented at the same location or different locations. If, in contrast, color memories are tied to locations, then they should interfere with each other when all of the items share the same location. This interference could take two forms. First, if the objects are stored in a single array-format representation, like those that have been hypothesized to underlie visual imagery (Farah, 1988; Kosslyn, Ball, & Reiser, 1978), then each sample item should simply overwrite the representation of the previously presented sample item. Second, even if the representations are more abstract, presenting the items at a common location should increase the similarity of the representations, leading to increased interference.

In Experiment 4a, we compared three conditions in which the spatial and temporal attributes of the sample stimuli were varied (see Figure 4). In the simultaneous-peripheral condition, we presented participants with sample and test arrays identical to those used in the baseline condition of Experiment 1. In the sequential-peripheral condition, we presented the to-be-remembered sample objects one at a time in the locations that they would appear at in the test arrays. That is, each object was shown in a randomly selected location, followed by a blank interval and then the next object in the sequence. A test array was then presented, containing all of the objects in a single, simultaneous display. In the sequential-central condition, the objects were presented sequentially in the middle of the screen. After all the items were presented centrally, the test array was presented just as in the other conditions; that is, the test objects were presented simultaneously at randomly chosen peripheral locations. In all three of these conditions the set size on each trial varied randomly between 2, 4, or 6 objects. We note that our sequential versus simultaneous presentation manipulation necessarily results in retention intervals with different lengths. We empirically address the effect of this confound in Experiment 4b.

Figure 4.

Stimuli and results of Experiment 4a. (A) Example of a sample array in the simultaneous presentation condition. Fill patterns are used to represent different colors. (B) Example of the stimulus sequence in the condition using sequential presentation at the test locations. (C) Example of stimulus presentation sequence in the condition using sequential presentation at fixation condition (i.e., a common central location). (D) The accuracy results from Experiment 4a.

By comparing the sequential-central condition with the sequential-peripheral condition, it will be possible to determine whether individuating spatial information is important for maintaining object identity information in working memory. In addition, by comparing the sequential-peripheral condition with the simultaneous-peripheral condition, it will be possible to test the more specific hypothesis of Jiang et al. (2000) that change-detection performance is facilitated when the objects are presented in the same spatial configuration at sample and test. Specifically, it should be more difficult to form a configural representation of the sample stimuli when they are presented sequentially than when they are presented simultaneously, which should lead to impaired performance for the sequential-peripheral condition if the Jiang et al. hypothesis is correct.

This experiment also provides an additional test of the ability of individuals to match a to-be-remembered object presented at one location with a test object presented at a different location (as in the scrambled-location conditions of Experiments 2 and 3). That is, the sequential-central condition uses completely different spatial locations for the sample stimuli and the test stimuli. If performance in this condition is unimpaired, then this will provide a replication of the main finding of Experiments 2 and 3.

The sequential presentation method of the present study also has the advantage of being more similar to previous studies of verbal memory, which almost always involve sequential presentation of the to-be-remembered information. In particular, sequential presentation makes it possible to test for the existence of primacy and recency effects. However, because this issue is tangential to the hypothesis being tested in this study, the results of the experiment that looked at serial position effects will be reported in an Appendix.

Method

The stimuli and procedure were identical to those of the scramble condition of Experiment 2 with the following exceptions. A new group of 15 participants from the same participant pool participated in this experiment (more participants were necessary for this experiment because the trials were longer and fewer trials could be tested for each participant). Each participant completed three blocks of 24 trials, one for each experimental condition, in a randomized order. In all conditions, the objects in the test arrays were randomly distributed within a 9.8° × 7.3° region with a minimum inter-object distance of 2° (center-to-center). The trials in the simultaneous-peripheral condition began with a 100-ms presentation of a sample array, which contained a set size of 2, 4, or 6 colored squares in the same spatial arrangement used for the test arrays. The sample array was followed by a blank 900-ms retention interval and then a 2000-ms test array. The sequential-peripheral condition was identical to the simultaneous-peripheral condition, except that the sample items were presented one at a time; each was presented for 100 ms followed by a 900-ms blank interval (see Figure 4B). The sequential-central condition was identical to the sequential-peripheral condition, except that the sample items were presented in the center of the monitor (see Figure 4C). The task of the participants in all conditions was to determine whether the set of colors presented in the test array was identical to the set of colors presented in the sample phase.

Results and Discussion

Accuracy

The accuracy data from Experiment 4a are shown in Figure 3D. Accuracy decreased as a function of set size in all three conditions, F(2,28) = 41.53, p < .001. Accuracy was also slightly lower (an average of 5.3%) in the sequential-peripheral and sequential-central conditions relative to the simultaneous-peripheral condition, leading to a significant effect of condition, F(2,28) = 6.75, p < .005. Finally, although the set size effect was somewhat greater in the sequential presentation conditions, the Condition x Set Size interaction was not significant (p > .35). Performance was approximately equal for the two sequential conditions; a separate ANOVA with only these two conditions yielded neither a significant main effect of condition (p > .35) nor a significant Condition x Set Size interaction (p > .80). Thus, sequential presentation of the sample array led to a small decline in performance compared to simultaneous presentation. However, for sequential presentation, performance was not impaired when the items were presented at a single location rather than being presented at different, peripheral locations that matched the test locations.

Reaction Time

The RT findings of Experiment 4a are presented in Table 2. Mean RTs in the two sequential conditions were slower than responses in the simultaneous baseline condition, leading to a significant effect of condition, F(2,28) = 74.73, p < .001. Preplanned tests confirmed observations that both of the sequential presentation conditions differed from the simultaneous condition (both ps < .001), but the sequential conditions did not differ from one another, F(1,14) = 0.79, p > .35. The other significant main effect was due to slower responses at larger set sizes than small set sizes, F(2,28) = 21.46, p < .001. Finally, there were significant two-way interactions of condition by set size due to larger set size effects in the sequential presentation conditions, F(4,56) = 3.23, p < .05, and set size (2, 4, or 6) by response (same versus different) due increased slowing of different responses with larger set sizes, F(2,28) = 4.03, p < .05.

Table 2.

Mean RT (ms +/− SEM) from Experiment 4a and 4b

	Set Size	Same Response Mean	Different Response Mean
Simultaneous-Peripheral	2	822.0 +/− 55.9	821.0 +/− 38.9
Simultaneous-Peripheral	4	867.7 +/− 44.8	926.6 +/− 46.9
Simultaneous-Peripheral	6	973.5 +/− 52.2	975.3 +/− 45.8
Sequential-Peripheral	2	1075.5 +/− 60.2	1055.7 +/− 51.2
Sequential-Peripheral	4	1202.4 +/− 61.3	1200.5 +/− 60.9
Sequential-Peripheral	6	1353.1 +/− 91.8	1266.2 +/− 82.3
Sequential-Central	2	1059.8 +/− 57.6	999.3 +/− 47.7
Sequential-Central	4	1236.6 +/− 74.9	1269.4 +/− 61.2
Sequential-Central	6	1434.6 +/− 106.9	1287.2 +/− 83.1
Experiment 4b	2	920.2 +/− 53.9	966.3 +/− 52.9
Experiment 4b	4	1196.5 +/− 52.8	1236.2 +/− 45.3
Experiment 4b	6	1313.5 +/− 90.3	1406.4 +/− 84.0

The small but significant impairment observed for the two sequential conditions compared to the simultaneous condition is consistent with the proposal of Jiang et al. (2000) that configural information is stored in working memory and facilitates change-detection performance. However, there is an obvious alternative explanation for this effect. Specifically, the amount of time that elapsed between the first sample item and the test array on sequential trials was much longer than the amount of time that elapsed between the sample and test arrays in the simultaneous condition. Thus, the greater average retention interval for the sequential conditions may have been responsible for the modest decrement in accuracy observed in these conditions. Experiment 4b was conducted to assess the plausibility of this explanation.

Experiment 4b

There is no perfect way to equate the retention intervals for simultaneous and sequential presentations of the sample items. Consequently, we did not try to equate the retention intervals, but we instead varied the retention interval for the simultaneous condition so that it bracketed the range of retention intervals for the sequential conditions. Specifically, whereas the retention interval for the simultaneous condition in Experiment 4a was equivalent to the retention interval for the final item in the sequential conditions, the retention interval used for simultaneous trials in Experiment 4b was equivalent to the retention interval for the first item in the sequential conditions. That is, the retention interval for simultaneous trials in Experiment 4b was 1900 ms for set size 2, 3900 ms for set size 4, and 5900 ms for set size 6. These data were then compared with the sequential data from Experiment 4a. We predicted that performance from the sequential conditions of Experiment 4a should fall somewhere between performance for the simultaneous condition of Experiment 4a and the new simultaneous condition of the present experiment.

Method

The methods used in Experiment 4b were identical to those used in the simultaneous-peripheral condition Experiment 4a except as follows. In Experiment 4b, a new group of the 15 participants completed one block of 72 trials. The blank period between the offset of the sample array and the onset of the test array was 1900 ms for set size 2, 3900 ms for set size 4, and 5900 ms for set size 6.

Results and Discussion

Accuracy

The results from Experiment 4b are shown projected on the results from Experiment 4a in Figure 5. Accuracy on the simultaneous-peripheral trials of Experiment 4b was approximately equal to accuracy on the sequential-central and sequential-peripheral trials of Experiment 4a (e.g., the 95% confidence intervals shown in Figure 5 are overlapping). To compare the results of Experiments 4a and 4b, we conducted 3 mixed-model ANOVAs, each comparing the simultaneous-peripheral data from Experiment 4b with one of the three conditions from Experiment 4a. Performance for the simultaneous-peripheral condition of Experiment 4b was found to be significantly impaired compared to the simultaneous-peripheral condition of Experiment 4a, F(1,28) = 6.74, p < .05, indicating that increases in retention interval do indeed lead to impaired performance. Moreover, there was a significant Condition x Set Size interaction, F(2,56) = 3.83, p < .05, due to performance decreasing more rapidly as set size increased in the present experiment relative to the simultaneous condition of Experiment 4a. However, performance from the simultaneous-peripheral condition of Experiment 4b was not significantly different from performance in either of the sequential conditions of Experiment 4a (both Fs < 0.7).

Figure 5.

Results from Experiment 4b are shown in black. The results from Experiment 4a are shown in gray for comparison.

Reaction Time

The mean RTs from Experiment 4a and 4b are presented in Table 2. The mean RTs from Experiment 4b were very similar to the mean RTs from the sequential conditions of Experiment 4a. To compare the RT results of Experiments 4a and 4b quantitatively, we conducted additional mixed-model ANOVAs just as we did with the accuracy data. Identical to the pattern of accuracy results, the RTs from Experiment 4b did not significantly differ from those of either sequential presentation condition of Experiment 4a (both Fs < 0.2) but did significantly differ from the simultaneous-peripheral condition of Experiment 4a, F(1,28) = 17.53, p < .001.

The findings of Experiment 4b suggest that a significant portion of the difference between the simultaneous condition and sequential conditions of Experiment 4a can be explained by the longer periods of time that participants were required to retain the larger numbers of object representations in the sequential conditions. Differences in retention interval may not, however, explain all of the difference, because this should have led to worse performance for the simultaneous-peripheral condition from Experiment 4b than for the sequential conditions of Experiment 4a. Thus, we cannot rule out the possibility that performance was improved by the use of configural information on simultaneous trials. However, the present results clearly demonstrate that a large part of the benefit of simultaneous presentation in Experiment 4a was caused by the difference in retention intervals and that any remaining effects of configural information were quite small. Taken together, the findings from Experiment 4a and 4b suggest that the visual system is nearly as good at storing sequentially presented objects devoid of unique location information as it is at storing objects with stable absolute and relative spatial locations.

General Discussion

These experiments demonstrate that change detection accuracy is impaired remarkably little when the to-be-remembered objects change in size or position between the sample display and the test display. These findings indicate that change-detection tasks rely on a flexible, abstract, and manipulable memory system; that is, a working memory system. Consequently, the results from previous and future change-detection experiments can be used to constrain theories of visual working memory.

Although an abstract memory system is implicated by these results, we do not yet know whether the abstractness occurs in the memory representations or in the processes used to compare the memory representations with incoming stimuli. That is, although it is tempting to conclude from these results that visual working memory representations are abstract rather than being retinotopically organized and picture-like, it is possible that picture-like representations could lead to accurate performance in these tasks. Specifically, accurate performance would be obtained if the perceptual representations of the test stimuli can be scaled and shifted flexibly so that they can be compared with picture-like memory representations of the sample stimuli. Indeed, the RT effects observed in Experiment 3 suggest that the source of the impressive flexibility may be a comparison mechanism that can transform mental representations at the time of test. It is also possible that people simply have the ability to ignore spatial information and not encode it into memory when they know it is irrelevant and potentially misleading; however, it is less clear how this type of selective encoding would account for the RT findings. Nonetheless, the present results indicate that the visual working memory system as a whole – including encoding, maintenance, storage, and comparison – functions in a flexible and abstract manner. Distinguishing between a flexible storage system and a flexible comparison mechanism may require neurophysiological experiments in which the representations can be measured directly (Lueschow et al., 1994; Rainer, Asaad, & Miller, 1998; Rao, Rainer, & Miller, 1997; Tanaka, 1996).

In addition to demonstrating that change-detection performance relies on a flexible memory system, the present results also speak to the nature of visual working memory in general. Previous studies have speculated that only a single object can be stored in visual working memory and that multi-element patterns are stored as a single object (Phillips & Christie, 1977a; Phillips & Christie, 1977b). The present study indicates that 3–4 objects can be stored in visual working memory and that they are stored at least somewhat independently. More specifically, the findings of Experiment 4 show that capacity is not greater when the to-be-remembered objects are presented one at a time. One might argue that capacity estimates during change-detection tasks are limited to 3–4 objects because observers cannot simultaneously select all of the objects with limited-capacity attentional mechanisms (e.g., Xu & Chun, 2006). In this regard, the present findings provide useful empirical evidence that capacity estimates are similar whether items are presented one-at-a-time in isolation or in an array of highly discriminable elements.

Viewed in the context of the broader literature, it is clear that there are boundary conditions for our proposal of flexibility in visual working memory that need to be better understood. For example, given evidence that we can select color information in a spatially global manner (e.g., Andersen, Muller, & Hillyard, 2009; Andersen, Hillyard, & Muller, 2008) we might be led to conclude that the flexibility we observed here is restricted to color features. However, the study of Logie, Brockmole & Jaswal (2011) found that the scrambling of spatial locations during retention intervals longer than 1000 ms had little or no effect when subjects were remembering either the color or the geometric shape of the objects. This suggests that the present findings generalize to other object features. As discussed below, we believe that there are certain to be limitations to this when array elements can be grouped or chunked into higher-order configurations, as seems likely to occur in more naturalistic real-world scenes (like those in Henderson & Hollingworth, 1999).

Finally, we are not claiming that the objects are stored completely independently, with no configural information about the entire array; rather, the present results indicate that some significant information about the individual objects is present. Indeed, the small (and non-significant) drop in performance in the two scramble conditions of Experiment 2 may reflect the storage of configuration information; when the configuration is disrupted by a change in the relative locations of the objects, a slight impairment in performance may result. Moreover, people may store some types of information primarily as configurations. For example, the dot matrices used by Phillips (1974) and others may be stored as global, configural shapes. In addition, pilot data from our laboratories suggests that performance in the scramble condition is significantly disrupted when subjects must remember line orientations rather than colors, which may indicate that subjects store sets of line segments as the edges of a single shape rather than as individual objects. This could also underlie the slow encoding and heavier neurophysiological load exerted when oriented lines are the task relevant features that need to be stored in visual working memory (Woodman & Vogel, 2008). Additional research will be useful in determining the conditions under which objects are stored independently or as groups (e.g., Alvarez, 2011).

Appendix

Although this is not germane to the issue of whether the visual memory system for object identity is sufficient flexible to tolerate changes in location location, it is natural to ask whether accuracy in Experiments 4a and 4b varied across sequential positions. The purpose of the experiment reported here was to answer this question. In the sequential conditions of Experiments 4a and 4b, there were not enough trials to examine performance separately for each serial position. To make such an analysis possible, we used only set sizes of 4 and 6 in the sequential-central condition, making it possible to present a large number of trials in which the changed item occurred at a specific serial position (36 changes at set size 4 and 24 changes at set size 6).

At set size 4, we predicted that performance would be slightly worse at earlier serial positions due to the greater delay between sample and test. This same effect of retention interval was expected at set size 6. However, because this set size also exceeded the storage capacity of visual working memory, we also expected that more recent items would tend to displace older items, leading to a substantial performance decrement for earlier serial positions similar to previous reports (Broadbent & Broadbent, 1981; Phillips & Christie, 1977a, 1977b).

Methods

The methods used in this experiment were identical to those of the sequential-central condition of Experiment 4a except that only set sizes 4 and 6 were used and each participant received three blocks of 96 trials. A new group of 15 participants was tested.

Results and Discussion

Figure A1 shows the accuracy (percent correct) for change trials as a function of the sequential position of the item in the sample sequence that was changed in the test display. Although we included both change and no-change trials when computing percent correct in the previous experiments, no-change trials could not be included in the analyses of the present experiment because it is impossible to assign false alarms to specific serial positions. The overall false alarm rate was low (5.6% for set size 4 and 7.5% for set size 6). Also, because the two set sizes contained different numbers of serial positions, it was not possible to analyze both set sizes in a single ANOVA and separate one-way ANOVAs were therefore used to analyze each set size, with serial position as the independent variable.

Figure A1.

Results from the serial position follow-up to Experiment 4 reported in the Appendix (error bars show the 95% within subjects confidence intervals). Percent correct indicates percentage of changes detected at each serial position (i.e., 1–4 or 1–6).

When the sample sequence consisted of four items, performance increased very slightly across serial positions. The effect of serial position was not significant in the ANOVA, however, F (3,42) = 1.42, p > .25. Planned comparisons were conducted to compare each consecutive pair of sequential positions, but none of these comparisons was significant (all ps > .12).

When the sample sequence consisted of six items, accuracy was approximately equivalent across the first four serial positions and then increased at the fifth and sixth positions. The effect of serial position was significant in the omnibus ANOVA, F (5,70) = 4.99, p < .001. Planned comparisons between successive serial positions yielded no significant differences between serial positions 1 and 4 or between 5 and 6 (all ps > .35), but the difference between positions 4 and 5 was significant (F (1,14) = 5.60, p < .05). Thus, there was a substantial recency effect at set size six but not at set size four.

Could the effect of serial position at set size 6 have been due to the mere passage of time? If so, then the effects at positions 3–6 at set size 6 should also have been observed at positions 1–4 at set size 4, because the timing was identical for these sets of positions. To demonstrate that these patterns were significantly different, we conducted a 2-way ANOVA with factors of set size (4 versus 6) and serial position (the last 4 serial positions at each set size). This ANOVA yielded a significant interaction between set size and serial position, F (3,42) = 3.38, p < .05, indicating that the effect of serial position differed across set sizes even though the timing was equivalent for these sets of serial positions.

These results indicate that the memory system tapped by this procedure can be updated in a flexible manner. When capacity is not greatly exceeded (e.g., at set size four), storing a new item in memory does not interfere substantially with the representations of the previously viewed items, leading to a flat serial position curve. In contrast, new information does displace old information when capacity is exceeded (e.g., at set size six); however, it is not strictly a first-in/first-out system, because performance did not decline much beyond the last two serial positions. Thus, this memory system is capable of “appending” new information in an intelligent manner (e.g., Vogel, McCollough, & Machizawa, 2005). These data might seem to imply that objects are stored in visual working memory in a discrete, all-or-none manner, and that an older item may be completely displaced by a newer item. However, it is also possible that partial information about objects is stored in visual working memory and that some information about older items remains when a newer item is presented.