Detecting changes in real-world objects

Abstract

A large body of literature has shown that observers often fail to notice significant changes in visual scenes, even when these changes happen right in front of their eyes. For instance, people often fail to notice if their conversation partner is switched to another person, or if large background objects suddenly disappear.^1,2 These ‘change blindness’ studies have led to the inference that the amount of information we remember about each item in a visual scene may be quite low.¹ However, in recent work we have demonstrated that long-term memory is capable of storing a massive number of visual objects with significant detail about each item.³ In the present paper we attempt to reconcile these findings by demonstrating that observers do not experience ‘change blindness’ with the real world objects used in our previous experiment if they are given sufficient time to encode each item. The results reported here suggest that one of the major causes of change blindness for real-world objects is a lack of encoding time or attention to each object (see also refs. ⁴ and ⁵).

Introduction

One of the most well known phenomena in the study of visual memory is the remarkable failure of observers to detect what should be salient changes in visual scenes if detection of those changes depends on visual memory (change blindness: reviewed in ref. 6). Studies have shown that even after very brief storage intervals, large changes to images can go undetected, and that this change blindness can occur even in real life social situations.^1,2 These results have sometimes been taken to suggest that we maintain only a sparse representation of the world in visual memory.^7–9

However, in a recently published experiment we showed that visual long-term memory is capable of storing a massive number of items with a large amount of detail per item;³ (for other work showing storage of a large number of items in long-term memory (see refs. ^10–12). In this experiment, participants viewed pictures of 2,500 categorically distinct objects one at a time for 3 seconds each, over the course of 5.5 hours. Afterwards, they were shown pairs of images, and indicated which of the two they had seen. The previously viewed item could be paired with either an object from a novel category (a cup you saw versus a clock you never saw), an object of the same basic level category (a cup you saw versus a similar but different cup), or the same object in a different state (the cup you saw empty versus the same cup with juice in it). Performance in each of these conditions was remarkably high (92%, 88% and 87%, respectively), suggesting participants successfully maintained detailed representations of thousands of images.

There are many possible reasons why our recent long-term memory experiment found evidence for detailed visual memory where other researchers have not. One counter intuitive possibility is that long-term memory represents items in greater detail than short-term memory. A more likely possibility is that differences between our long-term memory paradigm and a typical change blindness paradigm led to the difference in results. Here we sought to test this directly, examining the effect of encoding time on the ability to detect changes to real-world objects.

Method and Results

We presented observers (N = 6) with six real-world objects arrayed in two rows of three (Fig. 1). The objects were taken from the test pairs used in our previous study of long-term visual memory³ (the images can be downloaded at http://cvcl.mit.edu/MM/). On each trial, an onscreen message informed viewers of how long the objects would appear on that trial (1.2, 6 or 18 seconds). Observers then pressed a key and the six objects appeared for the specified amount of time. Then the objects disappeared for 1 second, after which a single object reappeared and observers had to indicate whether it was the same exact object that had previously occupied that location. On half of the trials the object was exactly the same, and on the other half of the trials the object changed. We manipulated the similarity of the object that reappeared to the original object in the same manner as in our long-term memory experiment (using the stimuli from³): the image that reappeared could either be an entirely different object (from a novel category), a different exemplar of the same category, or the same exact object in a different state or pose.

Figure 1.

A screen initially displayed the duration the objects would be present for. This was followed by a brief blank, and then the six objects were presented. Afterwards, there was another brief blank and then a test object. Observers had to say whether the test object was the same or different than the object previously presented at that location.

The three duration conditions correspond to 200 ms/item, 1 sec/item and 3 sec/item. The final condition matches the amount of time each object was presented in the long-term memory study.³ In the present study we did not control how long people looked at each item in the display; observers could only use the duration preview at the beginning of each trial in order to allocate their encoding time for each object.

The results are plotted in Figure 2, showing percent correct for objects in the three change-type conditions (novel, exemplar, state change) at all three encoding durations (1.2 s, 6 s and 18 s). At the shortest duration, observers were able to detect changes of an object's category (novel condition) with almost 90% accuracy, significantly better than both exemplar-level changes at 73% (t(5) = 4.84, p < 0.05) and state-level changes at 63% (t(5) = 8.33, p < 0.001). With increasing presentation time, the performance for state and exemplar-level change detection also increases. At the longest presentation time, the novel changes are detected 96% of the time, with state and exemplar changes detected 89% and 91% of the time, respectively. (ANOVA: Main effect of condition: F(2,10) = 44.77, p < 0.001; Main effect of Encoding Time: F(2,10) = 18.02, p < 0.001; Condition × Encoding Time interaction F(4,20) = 3.46, p < 0.05). These accuracy levels match fairly well with the long-term memory study, in which the same novel, exemplar and state pairs were presented in a two-alternative forced choice (novel: 92%, exemplar and state: 87%). These results show that observers can successfully detect large changes (at the basic-level category) with only brief time to encode the display, but encoding all of the objects with sufficient detail to allow change detection at the level of exemplars or states took a considerable amount of time (as much as 3 seconds/item). Interestingly, these rates are much slower than typically found for simpler stimuli, where 100 ms of encoding time is often sufficient.¹³ This suggests that the formation of detailed visual memories of real-world objects takes seconds to form, and factors other than encoding time limit change-detection for simple shapes (reviewed in ref. 14).

Figure 2.

Percent correct for objects in the three change-type conditions (novel, exemplar-level change, state-level change) as a function of encoding durations (1.2 s, 6 s and 18 s). There were six objects presented, so the encoding durations correspond to 200 ms/object, 1 sec/object and 3 sec/object, respectively.

Discussion

Our data suggest that having sufficient time to encode information from each object is crucial to the formation of the detailed visual memories needed to support change detection. Thus, change blindness for real-world objects may sometimes occur because of a failure to sufficiently encode the details of objects in visual memory, preventing the later comparison of those objects when they reappear.

Many studies that have demonstrated poor memory for image details in the form of change blindness have either tested memory for non-salient background regions, or have tested memory for central objects when they were task irrelevant.^1,2 Thus, in line with the current results, it is possible that poor performance on such tasks does not reflect memory limitations, but the failure of observers to encode the relevant details in the first place. In addition, even if items are successfully encoded there may still be errors of comparison or persistence of the representation that cause change blindness (reviewed in ref. 6).

In further support of this encoding failure interpretation of change blindness, studies that have demonstrated better memory for image details have tested task-relevant and attended foreground objects and also have informed observers of exactly what details are relevant for the task (reviewed in refs. ^{5, 15} and ¹⁶). Hollingworth¹⁶ demonstrated that when observers are required to attend to all of the items that might potentially change with the intention of remembering them, they could successfully detect changes to a large number of items from an image (i.e., change-blindness is attenuated; reviewed in ref. 4). The current results support this interpretation that change blindness may result from a failure to encode the items into visual memory in the first place.

These results also highlight an important aspect of change detection paradigms: memory performance will depend not only on the duration of encoding, but also on the difficulty of the subsequent test. For example, the memory performance data in the present experiment could be converted into capacity estimates (e.g., using Cowan's K¹⁷) but this analysis would give three different answers to how many objects observers could hold in memory depending on whether the novel, exemplar, or state change performance was used. The variability in K as a function of the test type suggests that the right units for quantifying the capacity of visual short term memory (or visual memory on any time scale), are not in number of objects, but in an alternate measure that takes into account both the number of items represented and also the precision of each item's representation.^18–20

Addendum to: Brady TF, Konkle T, Alvarez GA, Oliva A. Visual long-term memory has a massive storage capacity for object details. Proc Natl Acad Sci USA. 2008;105:14325–14329. doi: 10.1073/pnas.0803390105.