The role of visual working memory in attentive tracking of unique objects

Abstract

When tracking moving objects in space humans usually attend to the objects’ spatial locations and update this information over time. To what extent do surface features assist attentive tracking? In this study we asked participants to track identical or uniquely colored objects. Tracking was enhanced when objects were unique in color. The benefit was greater when the distance between distractors and targets was smaller, but was eliminated when the objects changed colors 1 to 4 times per second, even though at any instant they were always uniquely colored. Additionally, tracking uniquely colored objects impaired a secondary color-memory task more than tracking identical objects, and holding several colors in working memory eliminated the advantage of tracking uniquely colored objects. Contrary to previous studies showing that feature information is poorly retained during tracking, these findings indicate that surface properties are stored in visual working memory to facilitate tracking performance.

Introduction

Tracking moving objects with attention and remembering their identities in visual working memory are two mechanisms that allow people to maintain temporal continuity in a constantly changing environment. Although both attentive tracking and visual working memory can be used to serve similar functions, they are rarely studied together (Song & Jiang, 2006). Researchers interested in working memory usually investigate memory of static displays (Luck & Vogel, 1997), whereas researchers interested in attentive tracking typically examine tracking of identical objects (Pylyshyn & Storm, 1988). In this study we explore how individuals track moving items that have unique identities. Specifically, we examine the extent to which surface features, such as unique colors, contribute to attentive tracking. This is an important yet largely neglected question in tracking studies. Previous studies have shown that observers usually track objects without remembering their identities (Pylyshyn, 2004). Contrary to this conclusion, here we show that unique features can facilitate tracking. We present four experiments to elucidate how visual working memory interacts with tracking of uniquely colored objects.

Standard multiple-object tracking (MOT) research typically overlooks contributions from surface features. In the MOT task, observers track a subset of pre-specified items among identical distractors. Because the objects are not unique in identity, object memory plays no role in this task. The standard MOT research has left out the component of surface properties for several reasons. First, there is evidence that object identities are poorly retained in attentive tracking. For example, when observers are occasionally probed during tracking of unique objects, they can usually report the targets’ location and motion direction, but not their shape or color (Scholl, Pylyshyn, & Franconeri, 1999). This finding suggests that tracking is achieved primarily by updating an object’s spatiotemporal history, rather than by tagging the object’s identity. Second, evidence from developmental psychology has contributed to the idea that spatiotemporal properties, rather than surface features, are crucial for maintaining object continuity. For instance, Xu and Carey (1996) showed that ten-month-old infants are not surprised when a toy duck disappears behind an occluder and reemerges as a toy truck, even though they are highly sensitive to disruptions in motion continuity. Similarly, visually deprived adults who recently recovered vision rely primarily on movement information to individuate and segregate visual displays. They are initially impaired at using discontinuities in surface features, such as the presence of ‘T’-junctions, in segmenting a static display (Sinha, 2007).

However, when requested, observers are able to remember surface features of moving items. For example, when two unique identities (e.g., ‘A’ and ‘B’) are assigned to two placeholders and then removed, observers readily update the identities inside the placeholders after the placeholders rotate 90° (Kahneman, Treisman, & Gibbs, 1992). Consequently, observers are faster at recalling the letters when they reappear at their original placeholders than when they switch placeholders. Although it is possible to update object identities after simple rotational motion, such updating is more challenging in standard MOT tasks where many objects undergo complex motion. For example, Pylyshyn (2004) placed four digits, one in each of four target circles, before the circles started moving among four distractor circles. The digits were then removed and the target and distractor circles began moving freely on the display. When the motion stopped, participants were able to select the target circles from distractors, but unable to report the associated digit in each target circle. These results suggest that humans often track objects without remembering what they are.

Similar conclusions have been reached using the multiple-identity tracking task. In this task, objects with unique shape or color maintain their unique identities while moving. Observers were asked to remember which objects moved where. At the end of the motion sequence, the objects disappeared and observers are asked to report the identity of a probed target (Oksama & Hyönä, 2004, 2008). Performance in this task was quite poor, supporting the idea that object identities are not well remembered during tracking. For example, Horowitz et al. (2007) asked participants to track cartoon animals with unique identities. When the animals stopped moving and hid behind cactuses, participants were asked to report either the locations of the target animals (standard question) or the location of a specific animal (specific question). The specific question requires participants to remember the individual identities of the target animals. Horowitz et al. estimated that the capacity for recalling the correct location of a specific animal was between 1 and 2, much lower than the number of locations participants knew concealed targets (Horowitz et al., 2007). In Horowitz et al.’s study the animals moved in straight lines, so the motion was less complex than that used in standard MOT. The low capacity estimated for the specific question is perhaps a generous estimation of object memory in MOT.

The literature reviewed above may imply that surface features are rarely, if ever, used in attentive tracking. However, it is also possible that previous research has underestimated the importance of tracking objects with unique identities. MOT studies often use identical objects, an approach that precludes the role of surface features. On the other hand, multiple-identity-tracking studies are often too complex for revealing an effect of unique features. The multiple-identity-tracking task requires participants to simultaneously track the targets and remember their features. The dual-task requirements may induce interference between tracking the targets and remembering the targets’ features (Fougnie & Marois, 2006). In addition, the multiple-identity tracking task usually involves a small number of objects, almost all of which are targets. This task minimizes the demand for individuating targets from distractors, but maximizes the demand for object memory. While this task is useful for characterizing feature memory of moving objects, it is not well suited for examining attentive tracking of unique objects. If one wishes to test how tracking is affected by the uniqueness of objects, it is necessary to combine elements of the multiple-object-tracking and multiple-identity-tracking tasks. Specifically, one needs a task where participants are not required to remember object properties. Instead, observers simply track targets in displays that involve objects with unique identities, similar to real-life experiences.

We conducted four experiments using this paradigm in order to address two questions. First, to what extent can surface properties facilitate attentive tracking of moving objects? Experiments 1 and 2 were designed to establish the basic advantage of tracking unique objects to tracking identical objects. Second, what accounts for the effect of unique features on attentive tracking? Experiments 3 and 4 examine the contribution of visual working memory to tracking.

Experiment 1

Experiment 1 aimed to test whether unique identities affect attentive tracking. On each trial participants were cued to track 4 targets among a total of 8 objects. The 8 objects either had the same color (homogeneous condition), 8 different colors (all-unique), or intermediate levels of color heterogeneity (e.g., 2 or 4 total colors for the 8 objects). Importantly, participants were informed that color was irrelevant and that their memory for colors would not be tested. Sample displays from all experiments can be viewed at: http://jianglab.psych.umn.edu/MOTunique/MOTunique.htm. If surface properties are usually disregarded in an intrinsically spatial task (e.g., Jiang, Olson, & Chun, 2000), and if tracking exclusively relies on spatiotemporal properties, then performance should be equivalent between the homogeneous and heterogeneous colors conditions. Alternatively, heterogeneity among tracked targets may disrupt the grouping of targets (Makovski & Jiang, submitted; Yantis, 1992), lowering performance in the heterogeneous colors condition. A third possibility is that tracking may be enhanced by uniqueness in object identity, either because tracking ability is improved by feature differences between targets and distractors, or because unique identities can be retained in visual working memory. The intermediate levels of heterogeneity (e.g., 4 colors for the 8 objects) allow us to test whether performance change monotonically as a function of object heterogeneity.

Method

Participants

Participants in all experiments were students from the University of Minnesota. They were 18 to 35 years old and had normal color vision and normal or corrected-to-normal visual acuity. The experiments were conducted with the participants’ written consent. Participants received $10/hour or course credit for their time.

There were 20 participants in Experiment 1. Half of them (mean age 24.3 years) completed Experiment 1a and the other half (mean age 22.5 years) completed Experiment 1b.

Equipment

Participants were tested individually in a room with normal interior lighting. They sat approximately 57 cm away from a 19” computer monitor. The experiment was programmed with the Psychophysical Toolbox (Brainard, 1997; Pelli, 1997) implemented in MATLAB (http://www.mathworks.com).

Stimuli

The moving objects were circles (diameter = 0.6°) presented against a gray background. There were eight colors (red, green, blue, yellow, orange, azure, brown, and pink) sampled randomly according to a trial’s condition.

Procedure

Participants completed 125 trials, divided randomly and evenly into five conditions. On each trial, participants pressed the spacebar to initiate the cue period, which brought up 8 stationary objects presented at randomly selected locations within an imaginary square (21° × 21°). The four targets were cued by an outline white square (1.0° × 1.0°). The cue lasted for 1330 ms, after which the white squares disappeared and the objects moved at a constant speed of 17.5 deg/s. Participants were asked to track the cued objects and were encouraged to maintain fixation at the center of the display during tracking. The objects bounced off the edge of the imaginary square or repelled one another at a minimal center-to-center distance of 1.2°. After a few seconds of motion (see Design for specific motion duration), the objects turned black and stopped moving. Participants responded by clicking on four items, after which the correctly selected targets turned green and the missed targets turned red for 1 s to provide feedback.

Design

The five conditions differed in the heterogeneity of item colors used on a tracking trial. Table 1 illustrates the different conditions. In the homogeneous condition, all eight objects were identical in color; the exact color was randomly selected on each trial. In the all-unique condition, the eight objects had 8 different colors. In the paired-four condition, the four targets were unique in color and the four distractors were unique in color, but each target shared its color with one of the distractors. In the paired-two condition, two targets and two distractors were in one color while the other items were in another color. Finally, in the four-unique condition, two targets were one color while two other targets were another color, and two distractors were a third color while two other distractors were a fourth color. All trials were presented in a randomly intermixed order.

Table 1. A schematic illustration of the five conditions tested in Experiment 1.

Condition	# of overall colors	# of target colors	targets	distractors
homogeneous	1	1
All-unique	8	4
four-unique	4	2
Paired-four	4	4
Paired-two	2	2

Experiment versions

The two versions of the experiment (each involving half of the participants) were identical except for the duration of a trial’s motion period and the use of articulatory suppression. In Experiment 1a, the objects moved for 10 s on each trial. In Experiment 1b, the objects moved for an unpredictable amount of time, randomly selected between 4 to 8 s. The randomized trial duration used in Experiment 1b served to discourage participants from anticipating the trial ending. Participants in Experiment 1b also engaged in articulatory suppression, where they repeated a three-letter word as quickly as they could throughout a trial. Articulatory suppression minimized the possibility that participants would verbally recode the color of objects.

Results

Despite differences in motion duration (fixed or random) and articulatory suppression, the two versions of the experiment produced remarkably similar results (Figure 1). An ANOVA on stimulus type and experimental version produced no effect of experimental version, F(1, 18) = 1.60, p > .22, and no interaction, F(4, 72) = 1.20, p > .33. For the rest of the analyses data were collapsed across the two versions of the experiment.

Figure 1.

Tracking accuracy in Experiment 1a (left) and 1b (right). Error bars show ±1 S.E. Trial motion duration was fixed in Experiment 1a and random in Experiment 1b, which employed articulatory suppression.

Tracking accuracy was significantly affected by the heterogeneity of tracked objects, F(4, 72) = 104.80, p < .01, η_p² = .85. Post-hoc contrasts using Bonferonni corrections for multiple comparisons showed that color distinction between targets and distractors enhanced performance. Accuracy was significantly higher in the all-unique condition than the homogeneous condition (p < .01, η_p² = .59), and higher in the four-unique condition than the homogeneous condition, p < .01, η_p² = .89. The four-unique condition was better than the all-unique condition, p < .01, η_p² = .88, possibly because it was easier to group the targets when they were comprised of two rather than four colors. In contrast, when targets were distinctive from one another but not distinctive from distractors, no enhancement was found. The paired-four and paired-two conditions were comparable in accuracy, p > .90, neither of which was better than the homogeneous condition, ps > .35.

Discussion

Experiment 1 established the finding that surface features can be used to enhance attentive tracking. The critical factor for obtaining an advantage is not the heterogeneity of targets among themselves, but the distinction between targets and distractors (Horowitz et al., 2007; Makovski & Jiang, in press). In the paired-four condition, the targets were distinctive from one another, but each target shared the same color as one of the distractors. This condition did not yield any advantage compared with the homogeneous condition. In contrast, in the all-unique condition, the targets and distractors were all different from one another, and a clear advantage for tracking was observed. This advantage should not be treated as a grouping effect (Yantis, 1992), as the targets were distinctive from one another and thus cannot be easily grouped. Nonetheless, grouping clearly interacts with tracking of unique objects, as the uniqueness advantage was eliminated when targets were similar to the distractors (the paired-four condition).

Thus, contrary to the idea that surface features are rarely used during tracking, we have shown that tracking is enhanced when the tracked objects are unique rather than homogeneous. This advantage raises the question as to whether motion tracking was used at all when the tracked objects were all unique. For example, could participants rely on a strategy of “remembering and re-identifying”, where the target colors were remembered in the cue period, and re-identified at the end of a trial’s motion? As we will show in Experiments 3 and 4, visual memory is clearly involved in tracking of unique identities. However, performance in the all-unique condition cannot be supported solely by the remembering and re-identifying strategy. Because of random trial duration (Experiment 1b), participants could not precisely identify the moment at which the colors should be re-identified (note that all items turned black at the end of a trial). This means that tracking is necessary to establish the basis for re-identification or to establish motion continuity. To further strengthen the idea that participants tracked the moving items in our experiments, we conducted a control experiment where color information was available only during the cue phase and the second half of the motion period. In the critical condition, motion sequence was shuffled to produce incoherent jumps. Participants thus had to rely on remembering and re-identifying strategy to establish the correspondence between the cued colors and the colors on the final displays. Yet, performance in this condition dropped to 58%, much lower than its smooth-motion counterpart (73%, p < .01, η_p² = .55), suggesting that participants cannot exclusively rely on color memory to re-identify targets in the all-unique condition.

An interesting finding in Experiment 1a was the lower accuracy in the paired-four and paired-two conditions in comparison to the homogeneous condition, ps < .06, η_p² > .34. This trend toward lower performance involving identity-paired targets and distractors was first observed in Horowitz et al. (2007), but the trend was unstable (Horowitz et al., 2007; Makovski & Jiang, in press). Similarly in our study, the trend was not observed in Experiment 1b, and it was eliminated in a follow-up experiment where different conditions were tested in different blocks. Due to the unreliable nature of this finding, we will not focus on the comparison between the paired and homogeneous conditions in subsequent experiments.

Experiment 2

Psychologists often conceptualize attention as not only limited in the number of foci, but also in the spatial resolution of each focus (He, Cavanagh, & Intrilligator, 1996). Attention is said to have a finite resolution: it is not restricted to the precise location of a target, but can spread to its neighboring space. When the distance between a distractor and a target is smaller than the spatial resolution of attention, the distractor receives attention and may intrude into target perception. The view of attention as limited in spatial resolution has been instrumental in explaining attentive tracking errors (Intrilligator & Cavanagh, 2001) and visual crowding (Chakravarthi & Cavanagh, 2007). In attentive tracking, as distractors get closer to the targets it becomes harder to select just the targets. Consequently, tracking accuracy declines as the minimal target-distractor distance decreases (Shim, Alvarez, & Jiang, 2008).

The main purpose of Experiment 2 is to characterize the interaction between attentional resolution and object uniqueness. We tested whether the advantage afforded by tracking unique objects is constant at different target-distractor distances, or whether it is greater at closer target-distractor distances.

Method

Participants

Nine participants completed Experiment 2. They were all 18-35 years old but exact ages were lost in data collection.

Stimuli, procedure, & design

We manipulated object uniqueness and the minimally allowed target-distractor distance (T-D spacing). The eight objects could be homogeneous, all-unique, or paired-four, just like those used in Experiment 1. The minimally allowed distance between targets and distractors could be 0.6°, 1.5°, or 3.9°. In all conditions, the target-to-target minimal distance was 1.5° and the distractor-to-distractor minimal distance was 0.6°. All objects moved at a constant speed of 19.1 deg/s for 5 s during a tracking trial. Participants completed 180 trials, divided randomly and evenly into 9 conditions (3 T-D spacing x 3 object uniqueness). All other aspects of the experiment were the same as in Experiment 1a.

Results

Results showed that tracking accuracy declined when the minimal T-D spacing was smaller (Figure 2), F(2, 16) = 21.60, p < .01, η_p² = .73. In addition, accuracy was affected by object uniqueness, F(2, 16) = 17.90, p < .01, η_p² = .69, as performance was higher in the all-unique condition than the other two conditions (ps < .01, η_p² > .74), which did not differ significantly from each other, p > .45. Notably, the two factors interacted significantly, F(4, 32) = 3.70, p < .05, η_p² = .32. At the smallest T-D spacing the all-unique condition was reliably more accurate than the homogeneous condition, F(1, 8) = 50.16, p < .01, η_p² > .86, but this difference was largely eliminated at the largest T-D spacing, F(1, 8) = 2.41, p > .15. This interaction indicates that a closer target-distractor distance can be tolerated when tracking unique objects than when tracking homogeneous objects. The pattern of results reported here was unchanged when the dependent variable was log-transformed accuracy (Schweickert, 1985) rather than accuracy. In addition, the same results were obtained when we removed trials in which the ending display contained two items closer than 1.5° (there were 6.8% of trials), which might cue participants that these were likely distractors, as targets were not allowed to get closer to each other than that.

Figure 2.

Tracking accuracy in Experiment 2. Error bars show ±1 S.E.

Discussion

Experiment 2 replicated previous findings in showing that attentive tracking was impaired when the minimal target-distractor distance was reduced (Shim et al., 2008), supporting the idea that attention has limited spatial resolution (Intrilligator & Cavanagh, 2001). The results further showed that the impact of target-distractor spacing on tracking was reduced when objects were uniquely colored. If attentional resolution is operationally defined as the minimal target-distractor distance for performance to reach a certain threshold (e.g., 75% accurate), then it is fair to say that attentional resolution is functionally enhanced by object uniqueness. A similar conclusion has been reached in studies of visual crowding, which found that crowding of flankers on targets is reduced when a salient features differentiates them (Chakravarthi & Cavanagh, 2007; Pelli, Palomares, & Majaj, 2004). However, these results should not be taken as evidence for a change in the structural resolution of attention. Attention may have a constant structural resolution of, say, x° at an eccentricity of y. Any objects falling within x° will not be resolved by attention, whether they are similar or dissimilar. However, any confusion between those objects may be disambiguated on the basis of surface features. For example, when a blue circle target and a green circle distractor move within x°, participants can no longer attach their attentional focus just to the blue circle. When later the blue and green circles move apart, participants could recover a lost target, by remembering that the target was blue. Experiment 2 could not distinguish whether uniqueness in features changed the structural resolution of attention, or whether it only changed its functional resolution by permitting additional target recovery. Whether the structural resolution of attention can ever be changed is beyond the scope of this study. However, as we will see in the following experiments, a mechanism for target recovery can be readily supplied by visual working memory.

When discussing results from Experiment 1 we had provided reasons why motion tracking was actively used even when the objects to be tracked had unique colors. Was tracking one of the components used in the all-unique condition of Experiment 2? Our data suggested that it was. If participants exclusively relied on the “remembering and re-identifying” strategy, then their performance should have been equivalent at all target-distractor distances. Our results showed, however, that performance declined at closer target-distractor distances, even when all tracked objects were unique from one another, p < .05, η_p² = .39. Nonetheless, the idea that color memory was used to assist tracking of unique objects is supported by the following experiments.

Experiment 3

At least two accounts can explain the benefit of tracking unique objects over tracking homogeneous objects. On the one hand, the (structural) resolution of attention may be finer when two distinctive objects rather than two identical objects must be differentiated. Alternatively, target identities may have been encoded and stored in visual working memory, which enhances overall performance by recovering targets lost during motion tracking (Horowitz et al., 2007). To distinguish between these two accounts, we assessed the uniqueness advantage under two conditions: fixed-color and changing-color. In the fixed-color condition, once a color was assigned to an object, it was unvarying throughout the tracking trial. In the changing-color condition, the color of each object was changed periodically. The same color change (e.g., red changing into yellow) occurred to all objects in the homogeneous condition, while all objects changed to other unique colors in the all-unique condition. At any instant the objects were either homogeneous or all unique. If the distinctiveness in colors enhanced attentional resolution, then a benefit should be observed in the all-unique than homogeneous condition regardless of whether object colors were fixed or changing. On the other hand, if the unique colors were stored in visual working memory and the updating of memory is slow or effortful, then the uniqueness advantage should be reduced in the changing-color condition.

Here we reported data from two versions of Experiment 3 that differed primarily in the rate of color change. Experiment 3a changed the colors once every 260 ms, and Experiment 3b changed the colors once every 500 ms or 1000 ms.

Experiment 3a

Method

Participants

Twelve participants (mean age 19.3 years) completed Experiment 3a.

Stimuli, procedure, & design

We orthogonally manipulated object uniqueness (homogeneous, all-unique, or paired-four) and color consistency in a trial (fixed-color or changing-color). In the fixed-color condition, object colors were preserved throughout the trial. In the changing-color condition, object colors were changed once every 260ms. The color arrangement among the three uniqueness conditions was preserved after the color change, such that at any moment there were always 8 different colors in the all-unique condition, 1 color in the homogeneous condition, and 4 paired colors in the paired-four condition. To minimize color repetitions after the change, we used 16 possible colors in this experiment. The added colors were white, black, purple, light gray, light blue, light green, light purple, and light pink. On each trial objects moved at a constant speed of 17.5 deg/s for 7 s. Participants completed 72 trials, divided randomly and evenly into the six conditions (3 object uniqueness x 2 color changes). Other aspects of the experiment were the same as Experiment 1a.

Results

Results showed that when object colors were fixed throughout a trial, tracking accuracy (Figure 3) was higher in the all-unique condition than the other conditions (e.g., all-unique vs. homogeneous, F (1, 11) = 31.76, p < .01, η_p² = .74). However, when the colors were changing once every 260 ms, the reverse was observed, with marginally lower performance in the all-unique condition than the homogeneous condition, F (1, 11) = 3.91, p = .07. This difference resulted in a significant interaction between color uniqueness and color change, F(1, 11) = 27.50, p < .01, η_p² = .71. The interaction was driven primarily by a reduction in accuracy in the all-unique condition when the colors were changing, p < .01, η_p² = .64. Changing colors itself did not impair performance in the homogeneous condition, F < 1, suggesting that the change was not disruptive on its own. Pooled across different conditions, the main effect of color uniqueness was significant, F(1, 11) = 6.40, p < .05, η_p² = .37, as was the main effect of color change. F(1, 11) = 6.7, p < .05, η_p² = .38.

Figure 3.

Tracking accuracy in Experiment 3a. Error bars show ±1 S.E.

Experiment 3b

This experiment was designed to rule out a theoretically uninteresting interpretation of Experiment 3a’s changing-color results. Specifically, because colors changed once every 260 ms, which might have produced flickering, the lack of an advantage in the unique-color condition may be explained by the deficit in perceiving the colors due to the flicker. Experiment 3b used much slower change rates (500 ms in one condition and 1000 ms in another), which should minimize any low-level color flickering. The slow rate ensured that participants could adequately perceive the colors of the objects. However, because the change required that participants update their working memory frequently, it may reduce an advantage in the all-unique condition if the updating of working memory is slow or effortful.

We also employed articulatory suppression and randomized trial duration, similar to those used in Experiment 1b.

Method

Participants

Twelve participants (mean age 20.7 years) completed Experiment 3b.

Stimuli, procedure, & design

Experiment 3b was identical to Experiment 3a except for the following changes. First, we removed the paired-four condition, leaving only the homogeneous and all-unique conditions. This factor was orthogonally manipulated with three color-change conditions: fixed-color, or changing-color once every 500 ms, or changing-color once every 1000 ms. Second, we randomized the motion duration of each trial, terminating the motion at a random moment between 5-9 s. Finally, articulatory suppression was employed to reduce the use of verbal recoding. Participants were asked to repeat a three-letter word as quickly as possible throughout a trial. There were a total 120 trials, divided randomly and evenly into six conditions (2 object uniqueness x 3 color change).

Results

The basic pattern of results from Experiment 3a was replicated in Experiment 3b (Figure 4), which used a slower rate of color change. We found significant effects of color uniqueness, F(1, 11) = 5.70, p < .05, η_p² = .35, color change, F(2, 22) = 7.00, p < .01, η_p² = .39, and their interaction, F(2, 22) = 7.20, p < .01, η_p² = .40. Here too, an advantage for tracking unique objects was restricted to the fixed-color condition, F (1, 11) = 15.33, p < .01, η_p² = .58, but was abolished in the changing-color conditions (F’s < 1 for the 500 ms and 1000 ms change conditions). There was no difference between the 500 ms change and 1000 ms change conditions (Fs < 1). Both rates appeared to be too fast for effective use of color memory.

Figure 4.

Tracking accuracy in Experiment 3b. Error bars show ±1 S.E.

Discussion

Experiment 3 showed that object uniqueness at any moment was insufficient to produce an advantage for tracking unique objects. The idea that the resolution of attention is enhanced when the tracked items are distinctive rather than similar is not supported. Instead, the identity of any object must remain continuous over time. In the changing-color conditions, the new colors of the targets must be constantly updated in visual working memory. Yet, it seems like the slowest rate used in our experiments - 1000 ms per change - should have been sufficient to consolidate the colors into working memory, as consolidation takes as little as 50 ms/item (Vogel, Woodman, & Luck, 2006). However, actively encoding and storing information in visual working memory is attentionally demanding, and can compete with attentive tracking (Fougnie & Marois, 2006; Makovski, Shim, & Jiang, 2006). If participants sometimes fail to update their working memory, then relying on memory of object colors from a preceding moment would lead them to recover the wrong targets. These factors may lead participants to abandon the use of color information, or use this strategy with mixed results.

Experiment 4a

Up to this point, our results have implicated the use of visual working memory in attentive tracking of unique objects. Specifically, we have suggested that the unique colors are stored in visual working memory, which operate in parallel to tracking and can help recover lost targets. However, direct evidence for the use of visual working memory in tracking is lacking. Experiment 4a used a dual-task paradigm to provide direct support for this idea. Participants carried out a concurrent color-memory task during attentive tracking. While they tracked moving objects in the background (similar to Experiment 1), they also engaged in a one-back memory task at the center of fixation. In the one-back memory task participants were asked to remember 0, 1, 2 or 4 colors of a central circle. Using this design, we tested whether color working memory and attentive tracking of unique objects interfered with each other. If visual working memory is actively used to maintain unique color identities for the tracking task, then the uniqueness-advantage in the tracking task should be reduced at high color-memory load. Similarly, performance in the concurrent color-memory task should be impaired when participants tracked uniquely colored objects than when they tracked homogeneously colored objects.

Method

Participants

Twelve participants (mean age 22.4 years) completed Experiment 4a.

Tracking task

The two uniqueness conditions (homogeneous or all-unique) were tested concurrently with a one-back color memory task (Figure 5). The tracking task was similar to that used in Experiment 1, except that the moving objects were prohibited from entering the center of the screen with a diameter of 4.2°. The colors of the tracked objects were fixed throughout a trial. Objects moved at a constant speed of 17.5 deg/s for a randomly selected duration between 7.5 and 8.8 s.

Figure 5.

A schematic illustration of a trial used in Experiments 4a and 4b. Participants tracked 4 targets at the periphery while monitored color changes at the center.

We used 16 possible colors in the experiment, 8 of which were randomly assigned to the tracking task and 8 others were assigned to the one-back color-memory task.

One-back color-memory task

Participants were asked to remember the colors presented inside a central circle (diameter = 3.9°). The to-be-remembered colors appeared every 1500 ms for 495 ms, with the first presentation starting immediately after the tracking cue period. Each trial ended in a random moment (7.5 to 8.8 s, see tracking task) after a total of 5 presentations. The 5 presentations of each trial contained the same number of colors per presentation. However, across different trials, there could be 0, 1, 2, or 4 colors for participants to monitor at each presentation. Participants were told to monitor all colors on each presentation and determine whether they were the same as those presented on the preceding presentation, or whether one of the colors had changed. When a change occurred it always changed to a new color not presented on the preceding presentation. During the 5 presentations of a trial, there could be a total of 0 to 4 changes with equal probability. The one-back color-memory task was memory demanding throughout the trial due to the presence of multiple presentations and continuous monitoring. Figure 5 illustrates a trial used in this study; demos can be found online (http://jianglab.psych.umn.edu/MOTunique/MOTunique.htm). We used this task rather than the standard, one-shot change detection task because memory in the latter task was easily disrupted by an intermediate task such as tracking (Fougnie & Marois, 2006; Makovski & Jiang, 2007; Makovski et al., 2006).

Procedure and design

To minimize verbal recoding of the colors, participants were asked to rapidly count aloud the number of changes throughout a trial, starting from zero at the first presentation (e.g., “zero, zero, one, two, two”). At the end of object motion, participants clicked on the four tracked targets. After a 0.5 s feedback, they were prompted to enter the number of color changes they detected for the one-back color-memory task, after which they received accuracy feedback. A pilot study showed that participants usually prioritized the tracking task over the memory task, resulting in a unidirectional interference on the color-memory task but not on the tracking task. To examine how concurrent color-memory affected tracking, in this experiment we instructed participants to treat the one-back color-memory task as the primary task. Participants were motivated by their intrinsic competitiveness, as we showed their cumulative score after each trial and the average cumulative score from the other participants. Each correct response in the one-back color-memory task was given 2 points and each incorrect response led to a minus 1 point. No scores were given to the tracking performance. No monetary reward was used.

Each participant completed 240 trials, divided randomly and evenly into 8 conditions (2 object uniqueness x 4 memory loads). All other aspects of the experiment were the same as Experiment 1a.

Results

1. Tracking task

Tracking was significantly more accurate in the all-unique condition than the homogeneous condition (Figure 6 left), F(1, 11) = 15.40, p < .01, η_p² = .58. In addition, tracking was more accurate when the concurrent color-memory load was lower, F(3, 33) = 45.0, p < .01, η_p² = .80, showing general interference from a concurrent working memory task (Fougnie & Marois, 2006). Importantly, these two factors interacted, F(3, 33) = 4.10, p < .05, η_p² = .27. The benefit of tracking unique objects was found when the concurrent memory load was low (ps < .01, η_p² > .41 at load 0 and load 1), but was largely eliminated when the concurrent memory load was high (ps > .25 at load 2 and load 4). Furthermore, tracking accuracy was reduced by 8% in the all-unique condition when color-memory load increased from 0 to 1, which was marginally larger than the 4% reduction in the homogeneous condition, F (1, 11) = 3.34, p = .095, η_p² = .23. Thus, tying up visual working memory with a concurrent color-memory task significantly reduced the advantage of tracking unique objects.

Figure 6.

Results from Experiment 4a: tracking accuracy (left) and one-back color-memory accuracy (right). Note that load 0 produced no data for the one-back color memory task. Error bars show ±1 S.E.

2. One-back color-memory task

Performance in the one-back color-memory task was reduced as memory load increased (Figure 6-right; note that no data were obtained at load 0), F(2, 22) = 68.6, p < .01, η_p² = .86. In addition, performance in this task was affected by whether participants tracked homogeneous or unique objects. The one-back memory performance was significantly poorer when participants tracked unique objects rather than homogeneous objects, F(1, 11) = 44.80, p < .01, η_p² = .80. This interference was observed at all loads (all ps < .05, η_p² > .31), although it was numerically smaller at load 4, possibly because participants could spare less memory for the tracking task. The interaction between one-back memory load and tracking condition was marginally significant, F(2, 22) = 2.60, p < .10, η_p² = .19.

Experiment 4b

Experiment 4a revealed that tracking of unique objects was impaired more than tracking of homogeneous objects when color-memory load increased. In addition, color memory declined when subjects tracked uniquely colored objects. These results support the idea that common mechanisms were engaged by color memory and tracking of uniquely colored objects. However, performance in the tracking task was lower than 60% at high color-memory loads. This level of performance might imply participants tracked no more than one item at high memory load (Hulleman, 2005), in which case the manipulation of whether the other items had the same or different colors would cease to matter. A convincing demonstration of the shared mechanism between color memory and tracking of uniquely colored objects entails a replication of Experiment 4a’s results under conditions of high tracking performance. Experiment 4b provided such a replication except that items moved at slower speeds than those of Experiment 4a. The exact speed used here was still within the typical range of motion speed used in previous attentive-tracking studies (Horowitz et al., 2007; Yantis, 1992).

Method

Eight participants (mean age 19.6 years) completed Experiment 4b, which was identical to Experiment 4a except that the objects moved at a slower speed. The speed was 13.16 deg/s for half of the participants who performed well in practice trials and 8.77 deg/s for the others.

Results

1. Tracking task

Overall tracking accuracy did not differ between in the all-unique condition and the homogeneous condition (Figure 7 left), F<1. Tracking was more accurate when the concurrent color-memory load was lower, F(3, 21) = 10.02, p < .01, η_p² = .59. Most importantly and similar to Experiment 4a, these two factors interacted, F(3, 21) = 6.63, p < .01, η_p² = .49. That is, the benefit of tracking unique objects was found only when the concurrent memory load was 0 (p < .05, η_p² > .48). When memory load increased, this benefit was eliminated (load 2, p>.28), and even reversed (load 1 & 4, p’s < .01, η_p² > .7). Thus, even when overall tracking performance was high (above 75% in all conditions), increased color memory load impair tracking more when the tracked items were unique than when they were homogeneous.

Figure 7.

Results from Experiment 4b: tracking accuracy (left) and one-back color-memory accuracy (right). Note that load 0 produced no data for the one-back color memory task. Error bars show ±1 S.E.

2. One-back color-memory task

Performance in the one-back color-memory task was reduced as memory load increased (Figure 7-right), F(2, 14) = 39.12, p < .01, η_p² = .85. Similar to Experiment 4a, we found poorer accuracy when participants tracked unique objects rather than homogeneous objects, F(1, 7) = 35.72, p < .01, η_p² = .84. This interference was observed at all loads (all ps < .055, η_p² > .43), with no interaction between one-back memory load and tracking condition, F(2, 14) = 1.85, p >.19.

Discussion

Taken together, the results of Experiment 4a and 4b clearly showed that concurrent performance of a color working memory task and attentive tracking produced mutual interference. Holding several colors in working memory reduced tracking accuracy, both when the tracked items were homogeneous and when they were unique in colors. This task-general interference suggests that common attentional processes are tapped by these two tasks (Fougnie & Marois, 2006). Notably, a concurrent working memory task produced task-specific interference, leading to a greater interference on tracking of uniquely colored objects than homogeneous objects. The specific interference does not reflect exacerbated general interference: the all-unique condition was no more difficult than the homogeneous condition in tracking, yet it was interfered more by the color memory task. Moreover, this specific interference was found even when overall tracking accuracy was high (Experiment 4b).

Similarly, when tracking uniquely colored objects, performance in the color working memory task was impaired compared with tracking homogeneously colored objects. Why didn’t we find larger interference from tracking unique objects as load increased? One possibility is that this overall impairment reflects a larger perceptual noise in the all-unique condition, resulting with a relatively constant interference across memory loads. Alternatively, it is possible that as memory load increased, observers were less likely to rely on color memory for tracking and thus cancelling out possible increment in interference as load increases. When both the tracking results and the color-memory results are considered together, Experiment 4 provide compelling evidence in support of the idea that common processes are engaged in color working memory and in tracking of uniquely colored objects.

General Discussion

Attentive tracking and visual working memory have been conceptualized as mechanisms people use to maintain spatiotemporal continuity. Previous research has shown that attentive tracking and visual working memory share common cognitive resources (Fougnie & Marois, 2006) and neural substrates (Shim, Alvarez, & Jiang, 2005). However, the extent to which these two mechanisms interact remains unclear (Song & Jiang, 2006). The current study shows that attentive tracking is sensitive to the uniqueness of object identities, even in a task where identity memory is never tested. The identity information is largely retained in visual working memory, which allows participants to recover errors during tracking and enhance the functional resolution of attention.

The idea that surface features can be used to facilitate motion tracking, however, need not imply that “what” and “where” are perfectly integrated in attentive tracking. Our experiments provide little evidence that the “red” color of a target circle is perfectly bound to the motion trajectory of that object. To the contrary, the advantage for tracking unique objects could derive from feature memory and motion tracking working independently, in parallel to each other. Evidence that surface features are poorly integrated with their (moving) locations is found in a recent study (Makovski & Jiang, in press), where participants tracked objects that were unique in a combination of features. Specifically, participants were shown 8 colored digits made of 4 colors and 4 digits. The colors and digits were combined in a way that each object was unique in the combination of color and digit, but a target (e.g., a red 3) always shared its color with one distractor (e.g., a red 4) and its digit with another distractor (e.g., a green 3). If unique features were properly integrated with their moving locations, then performance in the conjunction-unique condition, where two features are bound to the same location, should also be superior to that in the homogeneous condition. However, performance in the conjunction-unique condition was similar to or worse than the homogeneous condition, making it unlikely that unique features are integrated with their moving locations.

If identity memory is not fully integrated with motion tracking (Saiki, 2003), one may expect that a similar uniqueness-benefit would be found when objects undergo smooth motion as when their movement is jumbled. This prediction was partially supported by a follow-up experiment, where the objects either moved with good spatiotemporal continuity (similar to Experiment 1a), or their motion sequence was shuffled to produce incoherent jumps. In this setup, “tracking” cued targets could rely on attentive tracking and identity memory in the smooth motion condition, but it could only rely on identity memory in the jumbled motion condition. Our results showed that unique identities enhanced performance in both the smooth and jumbled motion conditions, although the effect was significantly larger in the former. The presence of a uniqueness benefit in the jumbled condition suggests that identity memory alone could be used to establish correspondence across time. The effect, however, was smaller than the smooth motion condition, perhaps because identity memory was disrupted by jumbled motion (Song & Jiang, 2006).

In light of the current findings, how can we explain Scholl et al. (1999) and Pylyshyn (2004)’s data showing a near absence of identity memory during tracking? Although our results appear to be inconsistent with these earlier studies, at a theoretical level they are quite consistent. Empirically we have shown an influence of identity information on motion tracking, but we do not propose that identity memory is integrated with motion tracking. Pylyshyn (2004) did not find evidence for identity memory in his attentive tracking task possibly because he did not present identity information throughout the trial. Specifically, Pylyshyn presented digit identity inside tracking targets only during the cue period. The digits were then taken away and the targets moved among nontargets. Because the digits were absent during tracking, identity memory could have been easily disrupted by the tracking task (Fougnie & Marois, 2006; Makovski & Jiang, 2007; Makovski et al., 2006). In our study, identity information was present throughout the tracking trial, leading to better retention of target colors. Events such as the disappearance of objects behind cactuses, vanishing behind virtual occluders, or turning into neutral colors or shapes, may significantly interfere with identity memory, explaining why memory barely survives explicit probing of such manipulations (Horowitz et al., 2007; Oksama & Hyönä, 2008; Scholl et al., 1999).

To summarize, our study has shown that surface features can be used to enhance attentive tracking, but the enhancement is largely attributed to the parallel operation of visual working memory for identities and motion tracking. Our suggestion that identity memory works in parallel to attentive tracking is most consistent with the current set of data. Future studies should be conducted to explore the extent to which identity memory and motion tracking are integrated.