Memory for the search path: Evidence for a high-capacity representation of search history

Abstract

Using a gaze-contingent paradigm, we directly measured observers’ memory capacity for fixated distractor locations during search. After approximately half of the search objects had been fixated, they were masked and a spatial probe appeared at either a previously fixated location (varied n-back) or a non-fixated location; observers then rated their confidence that the target had appeared at the probed location. Observers were able to differentiate the 12 most recently fixated distractor locations from non-fixated locations, but analyses revealed that these locations were represented fairly coarsely. We conclude that there exists a high-capacity, but low-resolution, memory for a search path.

What is our memory capacity for where we have looked during search? Search efficiency depends critically on our capacity to remember previously searched locations. If you are searching for your car in a crowded mall parking lot, having a high-capacity memory for where you have already searched will speed up your car search task. Moreover, the rate of this speed-up will depend on the size of the search space; redundantly searching the same locations will be more costly in a large parking lot compared to a small parking lot. There is now good evidence, obtained from a variety of search tasks, that people do have some degree of memory for where they have looked during search (Aks, Zelinsky, & Sprott, 2002; Beck, Peterson, Boot, Vomela, & Kramer, 2006; Boot, McCarley, Kramer, & Peterson, 2004; Dickinson & Zelinsky, 2005; Gilchrist & Harvey, 2000; Peterson, Kramer, Wang, Irwin, & McCarley, 2001; however, cf. Horowitz & Wolfe, 1998, 2001, 2003; Woodman, Vogel, & Luck, 2001). However, most of this work focused on whether memory is used (Kristjansson, 2000; Müller & von Mühlenen, 2000; Shore & Klein, 2000) or not used (Horowitz & Wolfe, 1998, 2001, 2003; Woodman et al., 2001); the question of how much information is represented in memory remains relatively unexplored.

There have been many studies addressing the capacity of VSTM (Alvarez & Cavanagh, 2004; Irwin, 1992, 1996; Irwin & Andrews, 1996; Irwin & Zelinsky, 2002; Luck & Vogel, 1997; Zelinsky & Loschky, 2005), but task differences prevent generalizations from these studies to search. For example, in most VSTM tasks all objects in the display are potential targets, and an object’s location and identity must be retained in memory to make a correct response; in most search tasks there is only one target, and the identities of individual distractors are not task relevant—their identities can be represented simply as “not the target.” Those studies that have quantified memory capacity in the context of search have generally argued for a very limited memory capacity. Klein and MacInnes (1999) found evidence for inhibition of return (IOR) at the two most recently fixated locations during search of Where’s Waldo scenes, suggesting that IOR contributes to search efficiency by preventing reinspections of (at least two) previously searched locations. More recently, McCarley, Wang, Kramer, Irwin, and Peterson (2003) found evidence for memory for the last three to four fixated distractors during a novel search task designed specifically to quantify how many objects (i.e., location + identity) were retained in visual short-term memory (VSTM) during search. In their study, search objects were shown sequentially. After observers had made two fixations, objects were shown three at a time: the currently fixated object, a yet-to-be fixated object, and a previously fixated object. The probability of refixating an object by chance was 50% throughout a trial; thus, a refixation rate below 50% was taken as evidence for distractor memory. Refixations in this task were below chance levels for the last three to four objects that had been fixated during search. Using a very similar paradigm, Boot, McCarley, Kramer, and Peterson (2004), and Beck, Peterson, Boot, Vomela, and Kramer (2006) recently reported similar capacity estimates for rejected distractors—people appear able to remember having searched the three to four most recently fixated locations.

Although all of these studies are valuable in piecing together the relationship between memory and search, they are each limited with regard to specifying a memory capacity for search history. For example, Klein and MacInnes (1999) only tested the last two fixated locations in the search path, making it unlikely that they tapped capacity limits. In contrast, McCarley et al. (2003), Boot et al. (2004), and Beck et al. (2006) tested much further back into the observers’ search histories, thereby eliminating this concern, but the paradigm they used limited the number of objects that were visible to observers at any one time. Given that previously inspected objects typically remain visible in a search task and may enhance memory by serving as an external memory cue (Dickinson & Zelinsky, 2005), it is conceivable that these studies may have also underestimated memory capacity. It is also difficult to infer memory capacity from those studies that have argued for distractor memory based on a below chance rate of distractor refixations (Dickinson & Zelinsky, 2005; Gilchrist & Harvey, 2000; Peterson et al., 2001). Because distractor refixations are subject to non-memory related influences, such as the “pull” of a search guidance signal (Wolfe, 1994; Wolfe, Cave, & Franzel, 1989) and “double-checking” that might occur following an initial search of a display (see McCarley et al., 2003, for similar ideas), estimates of memory obtained from these studies may not accurately reflect our capacity to remember a search history.

The current experiments attempt to fill these gaps in the memory and search literature by assessing memory capacity in the context of a standard search task, one in which the display objects remain visible to the observer throughout his or her search. To do this, we explicitly tested the observer’s memory for distractors using a gaze-contingent spatial probe technique. We systematically probed display locations that were either on or off the observers’ search path, asking observers to rate after each trial how confident they were that the target had appeared at the probed location. This study therefore complements the work of Klein and MacInnes (1999) and McCarley et al. (2003) by directly testing observers’ memory for fixated locations in a search task, as opposed to indirectly estimating memory using a refixation-based measure. Moreover, we quantified memory capacity by varying how far back into observers’ fixation histories we tested, and we did this over a fairly large range so as to increase the likelihood that we would tap capacity limits. Our study also builds on the work by McCarley et al. (2003) in that we used a standard free-viewing search task in which all of the display objects were presented simultaneously and were continuously visually available to the observer. In addition to object-based or inhibition-based distractor representations, our paradigm therefore enables us to assess the contributions from other forms of memory, such as a memory for the path that was followed during search.

If there exists a high-capacity memory for the path followed during search, observers should be able to discriminate fixated locations from non-fixated locations, which would be expressed by high confidence ratings that targets did not appear in fixated locations in our task. In its extreme, meaning a complete representation of a search history, we would expect these high confidence ratings for all fixated locations, regardless of how far back in the search path we tested. However, if this memory were limited, we would expect that at some point in observers’ fixation histories they would not be able to make this discrimination. Moreover, if memory for search history consists of discrete spatial tags, then evidence for memory should appear in only the 3 to 5 most recently searched locations, consistent with the VSTM literature. We will operationally define a capacity limit as the point in an observer’s fixation history at which their confidence ratings for fixated locations do not differ from their ratings for non-fixated locations. We will also address the spatial resolution of this form of memory by examining how observer ratings for non-fixated objects are influenced by their proximity to objects fixated during search.

Experiment 1

Methods

Participants

Ten Stony Brook University students were paid $8/hour for their participation. All had normal or corrected-to-normal vision.

Apparatus and Stimuli

Eye movement and manual response data were collected using the EyeLink II video-based eye tracking system (SR Research Ltd.). Eye position was sampled at a rate of 500 Hz, the system’s spatial resolution was 0.2°, and changes in gaze position were available to the computer running the display program within 8 ms. A Pentium IV PC running Microsoft Windows XP was used to control the experiment. Search displays were presented at a screen resolution of 800 × 600 pixels using a 19” ViewSonic SVGA monitor operating at a refresh rate of 100 Hz. Observers’ head position and viewing distance of 78 cm was fixed with a chinrest, and all responses were made with a standard gamepad controller (Microsoft Sidewinder 1.0). Search judgments were made with the left and right index-finger triggers, trials were initiated with the button operated by the left thumb, and confidence ratings were made using the keyboard’s numbered keypad.

The stimuli were low-contrast gray square frames with a small gap on one side (Figure 1). The target had a gap either on the left or the right; the distractors had a gap either on the top or the bottom. The objects subtended 0.61° × 0.61°, the lines were 0.07° thick, and the gaps were 0.07° wide. Objects were randomly positioned in the display, with the exception of the following constraints. There was a 2.87° center-to-center minimum distance between objects, a 2.84° minimum distance from the center of any object to the center of the display, and a 1.5° minimum distance from the edge of the display to the center of an object, making the maximum display size 24° × 17.3°. Object masks were composed of the same line segments and subtended the same visual angle as the search objects.

Figure 1.

A schematic representation of the search and memory-probe tasks. a. The search stimuli used in both experiments, with the arrows showing a representative sequence of fixations. b. The accompanying placeholder display (Experiment 1 only) showing the intervening object values corresponding to the sequence of fixations in Figure 1a, as well as an example of an on-probe (left; 3-intervening object condition) and an off-probe (right). Note that the intervening object values did not actually appear in the placeholder display, and that only one location was probed per trial; two probes are shown in the figure so as to illustrate both on-probe and off-probe conditions.

Procedure and Design

A 35-object search display was presented on each trial. The observer’s task was to indicate whether the target gap appeared on the left or right by pressing either the left or right index-finger trigger on the gamepad. As they searched, the number, sequence, and locations of fixated objects were recorded, as is illustrated in Figure 1a. An object was tagged as fixated provided that gaze remained within 1.4° of the object’s center for at least 100 ms. If the observer failed to find the target after fixating a prespecified number of objects (ranging from 15 to 19), the search display was replaced by a placeholder display. The placeholder display consisted of object masks at all of the search object locations, and a spatial probe (a bright red box) surrounding one of these locations (Figure 1b). The specific location of the probe depended on whether the trial was on-probe or off-probe. In an off-probe trial, the probed mask corresponded to an object that was not fixated during search. In an on-probe trial, the probe appeared around an object mask corresponding to one of the fixated distractors in the observer’s search path. To explore the question of memory capacity using this gaze-contingent paradigm, for on-probe trials we varied, from 0 to 13, the number of intervening objects that were fixated between the probed object and the last object fixated during search. An intervening object value of 0 would mean that the last object fixated was probed, thereby requiring only 1 object to be held in memory; an intervening object value of 9 would mean that the 10^th object fixated in the search history was probed, thereby requiring 10 objects to be held in memory (for an example of a similar procedure used to explore VSTM capacity, see Zelinsky & Loschky, 2005). If the observer found the target, as indicated by a trigger press response, before the prespecified number of objects had been fixated, the placeholder display would be presented immediately. In this eventuality, the probe for on-probe trials would be randomly selected from previously fixated locations; off-probe trials would not be affected.

In response to the placeholder display, the observer was asked to rate his or her level of confidence as to whether the object that had appeared at the probed location was the target. This rating scale ranged from “1” (high confidence that the target did not appear at the probed location) to “9” (high confidence that the target did appear there), with a midpoint rating of “5” indicating “not sure”. Observers were instructed that the purpose of the experiment was to study how quickly people could find targets, and that we were limiting the amount of time they were given to search the displays so as to make the task challenging. No emphasis was placed on the confidence-judgment task so as to reduce the likelihood that they would intentionally encode the locations of fixated objects. Note also that observers were led to believe, through explicit instruction, that a target was present on every trial, when in actuality this was not the case. This ruse proved convincing in spite of the fact that the display program consistently terminated the search display (replacing it with the placeholder display) after approximately half of the display objects had been searched. Observers attributed their frequent failures to find the target to a brief and time-limited search display; no observer reported becoming aware that the duration of the search display depended on their own eye movement behavior. Moreover, had observers been allowed to fixate more than half of the display objects, they might have become biased towards making a “target was not there” response based on the duration of their search (Chun & Wolfe, 1996).

We manipulated target presence or absence, probe location (on-probe or off-probe), and the number of intervening objects (0–13). There were a total of 810 trials (plus 27 practice) presented in 15 blocks of 54 trials each. Of these 810 trials, 630 were on-probe and 180 were off-probe. The on-probe trials were divided into 210 target-present trials and 420 target-absent trials. There were more target-absent and on-probe trials compared to target-present and off-probe trials so as to accommodate the intervening object manipulation. Both target-present and target-absent on-probe trials were equally divided among the 14 intervening-object levels, yielding 15 target-present trials and 30 target-absent trials per intervening-object condition. The 180 off-probe trials were divided into 60 target-present trials and 120 target-absent trials. The entire experiment lasted 3 hours and was completed in 2 sessions conducted on separate days.

Results and Discussion

Memory Capacity Data

Trials on which search-judgment errors were made were excluded from all analyses. These errors consisted of two types: correctly responding to the presence of a target with the wrong response button (e.g., responding to a right-gapped target with a left trigger press; 1.64%), and false alarms (0.02%). We also limited our analyses of search history memory capacity to the 420 target-absent trials per observer. Observers searching a target-present display would often find the target and terminate the trial after fixating relatively few objects. Consequently, there might be many cases in the 0 intervening object condition, but relatively few in the 13 intervening object condition. However, the termination of target-absent trials required the fixation of roughly half of the display objects, meaning that we could assemble intervening object conditions consisting of roughly the same number of cases. Given the relative stability of the target-absent data across the intervening object manipulation, we therefore use these data to derive our estimate of memory capacity during search. Finally, we confined our analyses to the objects in the “memory set” (i.e., objects fixated between the probed object and the last fixated object, inclusive) that were not refixated during search. Again, we did this in order to maintain the integrity of our intervening object manipulation, as object refixation would require restarting the intervening object counter (see also, Zelinsky & Loschky, 2005).

Observers’ mean on-probe judgments, mean off-probe judgments, and on-probe judgments for each level of intervening object are shown in Figure 2a. Returning to our predictions, memory for search history should be expressed as higher “target was not there” confidence judgments (ratings 1–4) for on-probes than for off-probes, with off-probe ratings expected to hover around “not sure” (rating 5). Moreover, if this memory were complete then this difference should exist for all intervening-object values tested. However, if memory were limited then at some intervening-object value there should be no difference between on-probe judgments and off-probe judgments. To examine first whether observers were able to broadly discriminate fixated object locations from non-fixated ones, we compared observers’ mean on-probe judgments (averaged across all intervening object values) to their mean off-probe judgments. Here, memory for fixated locations would be reflected as higher confidence that the target did not appear at on-probe locations than at off-probe locations. A dependent-means t test revealed that observers were more confident that targets did not appear in previously fixated locations compared to non-fixated ones (2.3 vs. 3.3), t(9) = −3.50, p < .05, suggesting that they were able to discriminate fixated locations from non-fixated ones in memory. Clearly, observers were more confident in their decisions if they fixated the probed location than if they did not. It is also clear that mean off-probe ratings were significantly different from the scale’s midpoint (as revealed by a one-sample t test, t(9) = 4.74, p < .05).

Figure 2.

Mean ratings indicating confidence that the target appeared (ratings 6–9, with 9 being the highest confidence), or did not appear (ratings 1–4, with 4 being the highest confidence), at a probed location. All error bars represent the standard error of the mean. a. Mean confidence ratings for target-absent trials, including mean on-probe (i.e., a probe corresponding to the location of a fixated distractor; diamond marker) ratings, mean off-probe (i.e., a probe corresponding to the location of a non-fixated distractor; square marker) ratings, and on-probe ratings as a function of the number of fixated intervening objects (circle markers). b. Off-probe ratings (target-absent trials) as a function of mean distance from the nearest fixated object location. c. Mean confidence ratings for target-present trials on which the target was found, including mean on-probe ratings for target locations (diamond marker), mean off-probe ratings for distractor locations (square marker), and on-probe ratings for distractor locations as a function of the number of fixated intervening objects (circle markers). d. Off-probe ratings (target-present trials) as a function of mean distance from the nearest fixated object location for trials on which the target was found.

To look for evidence of memory limitations, we analyzed the mean on-probe ratings as a function of the number of intervening objects using a repeated-measures one-way ANOVA. This analysis revealed that confidence judgments did decline over the range of intervening objects tested, F(13,104) = 7.61, p < .05. To determine the specific point at which on-probe ratings differed from off-probe ratings, we conducted post-hoc comparisons using dependent-means t tests and found significant differences (p < .05) for all levels of intervening objects except for the 12 intervening object condition, t(9) = 2.02, p = .075. If we define the point at which on-probe and off-probe differences vanish as the capacity limit, this finding suggests that observers were able to maintain the locations of 12 rejected distractors in memory during this search task.

Reinspection Data

Observers were clearly able to remember a large portion of their search history, but did they use this memory to improve the efficiency of their search? To address this question, we examined the pattern of reinspections in this search task. If memory use is limited to only a handful of discrete spatial tags, observers should tend to avoid reinspecting an object after inspecting up to 4–5 other objects, but distractors visited farther back in the fixation sequence would not be tagged and should therefore be reinspected with greater frequency. However, if the memory used by observers to make their confidence judgments was also used to guide search away from previously visited distractors, observers should avoid reinspecting objects even after 10 intervening object inspections. Figure 3 shows the proportion of target-absent distractor reinspections as a function of the number of intervening objects inspected during search. Object reinspections were rare overall. Observers reinspected one or more of the last 12 fixated objects on 35% of the target-absent trials, and on these trials observers made an average of only 1.3 reinspections. Of these reinspections, the majority (64%) occurred following either one or two intervening distractor inspections, a pattern that likely indicates an incomplete processing of the most recently visited objects (Hooge & Erkelens, 1998). Moreover, the rate of reinspections across the 4–10 intervening object range was uniformly low and certainly no greater than the reinspection rate across the 1–4 intervening object range. This pattern suggests that the memory measured by our confidence-judgment task was at least as effective in preventing distractor reinspections during search as the discrete spatial tags previously implicated in search memory.

Figure 3.

The proportion of object reinspections made during the Experiment 1 target-absent trials as a function of the number of intervening objects inspected. An intervening object is defined as an object visited by gaze following the inspection of an anchor object; multiple successive fixations on a given object did not count toward this intervening object measure. We limited this analysis to the last 12 distractors fixated during search, the range over which observers were able to discriminate fixated from non-fixated locations. Note that this 12 distractor range corresponds to an intervening inspection range of only 10, which would result if a distractor was inspected, then reinspected after the inspection of 10 other distractors.

Memory Resolution Data

The results of the previous analyses suggest that observers retained and used a large amount of information regarding where they had searched; however, these data don’t speak to the resolution of this representation. This could vary from being very high (e.g., the exact pixel coordinates searched) to very low (e.g., the upper-left quadrant). To examine this question, we tested whether off-probe ratings varied as a function of the distance between an off-probe and a fixated object. A high-resolution memory predicts no effect of this distance on off-probe ratings; a lower resolution memory does predict a distance effect as off-probes might become confused with nearby distractors that were fixated during search. For each off-probe trial we computed the distance from the probed object to the nearest fixated object, then accumulated these distances in 1° bins. Figure 2b plots these mean off-probe ratings as a function of distance from a fixated distractor over the 3° to 10° range. Ratings for off-probes with proximities greater than 10.5° were collapsed into a separate 11+ degree bin. A one-way repeated-measures ANOVA for the first five bins¹ (3° through 7°) revealed that ratings differed as a function of distance from the nearest fixated distractor, F(4,36) = 12.14, p < .05. Dependent-means t tests confirmed that this tendency to confuse non-fixated objects with fixated ones decreased reliably in a graded manner as the distance to the nearest fixated object increased from 3° to 6° (all ps < .05), a pattern consistent with findings showing a graded decrease in IOR with increasing distance from an attended location (Bennett & Pratt, 2001; Maylor & Hockey, 1985; Pratt, Adam, & McAuliffe, 1998; Pratt, Spalek, & Bradshaw, 1999; Samuel & Weiner, 2001; Tassinari, Aglioti, Chelazzi, Marzi, & Berlucchi, 1987). Taken together, these data suggest that observers maintained a fairly low-resolution representation of where they searched. Moreover, they provide converging evidence for the validity of our dependent measure; had the ratings in Figure 2a been due to a bias to respond “target was not there”, they would not be expected to vary as a function of the off-probe’s distance from a fixated distractor.

Target-Present Data

To further determine whether observers were using the confidence scale correctly, we examined their ratings for target-present trials on which the target was found (as indicated by buttonpress). On these trials, if the target’s location was probed, observers should be highly confident that the target did appear at the probed location. Similarly, if any other location were probed, they should be equally confident that the target did not appear at that location, regardless of where in the observer’s search history the probed object was fixated (for on-probes), or of its proximity to a previously fixated object (for off-probes). The patterns illustrated in Figures 2c and 2d clearly support these predictions. Consistent with the correct use of the confidence scales, observers were highly confident that a probed distractor location did not correspond to the target when the target was actually found. Likewise, observers’ were equally confident in their judgments when the target was found and the target location was probed.

Experiment 2

The data from Experiment 1 suggest that observers maintained a high-capacity, low-resolution memory for where they had searched, but were they focused on the search task (as instructed) or were they also attempting to remember as many fixated locations as possible in anticipation of the memory test? If our Experiment 1 observers were attempting to encode the locations of rejected distractors into memory, we might expect to find one or more of the following changes to eye movement behavior relative to observers who completed the same search task without an accompanying memory test: (1) longer fixations on individual objects (reflecting a deliberate attempt to encode fixated locations), (2) an increase in fixation durations over the progression of a trial (possibly reflecting memory rehearsal during search), or (3) more distractor refixations (reflecting an attempt to refresh a fading working memory representation). Importantly, the presence of any of these differences might suggest that our estimate of search memory capacity from Experiment 1 was artificially inflated. To investigate whether the presence of a memory test influenced how observers searched, we conducted a second experiment in which a new group of observers completed the Experiment 1 search task without a subsequent memory test. In addition to the aforementioned measures, differences in search behavior might also be revealed in mean reaction time (RT), error rates, or number of fixations.

Methods

Participants

Ten students at Stony Brook University participated in this experiment. All were paid $8/hour and had normal or corrected-to-normal vision, and none participated in Experiment 1. All aspects of the stimuli, design, and procedure were identical to Experiment 1, with the exception that there was no placeholder display presented after the search display, nor was there a memory-probe task. The task was now standard visual search in which observers had to indicate whether a left-gapped or right-gapped target was present in the display.

Results and Discussion

Oculomotor Data

We first compared mean search fixation durations (initial and final fixations were not included) for the two groups to determine if observers fixated individual objects longer when memory was being tested. A 2 × 2 (group x target presence) mixed factorial ANOVA revealed no main effect of group (214 ms vs. 203 ms; Experiment 1 vs. Experiment 2) or target presence (208 ms vs. 209 ms; present vs. absent) and no significant interaction between the two (all ps > .05). We next compared fixation durations for the two groups as a function of fixation number to determine if their durations increased as a trial progressed.² These data are shown in Figure 4. A 2 × 16 (group × fixation number) mixed factorial ANOVA comparing observers’ fourth through nineteenth fixation durations revealed a significant effect of fixation order, F(15, 270) = 3.20, p < .05, but no significant interaction with group, F(15, 270) = 0.98, p > .05. The results of this analysis suggest that observers in Experiment 1 were not taking significantly more time to encode the locations of rejected distractors relative to observers who completed only the search task.

Figure 4.

Mean fixation duration as a function of fixation number in Experiments 1 and 2.

We next compared distractor reinspection rates for the two groups to further examine whether observers in Experiment 1 might frequently refixate objects in an attempt to maintain their locations in working memory. We again limited this analysis to target-absent trials, and collapsed multiple sequential fixations on the same object (i.e., inside the same invisible bounding circle) into a single inspection, as in the previous reinspection analysis. Immediate reinspections (i.e., those with only one intervening fixation) were excluded from this analysis because they might reflect incomplete processing of search objects rather than memory failure (Hooge & Erkelens, 1998). An independent-samples t test revealed no differences in distractor refixation rates between Experiment 1 (.139) and Experiment 2 (.142), t(9) = −0.17, p > .05, suggesting that observers in Experiment 1 were not using eye movements to rehearse previously visited locations during search.

Finally, we examined the distributions of saccadic amplitudes generated during search for the two groups as a simple way to characterize whether observers were searching the displays differently. We included data from only target-absent trials because there were more saccades on these trials compared to target-present trials, providing a more representative sample of gaze behavior. Saccade amplitudes were collapsed into 1° bins with means ranging from 3° to 10°; saccades with amplitudes greater than 10.5° were collapsed into a single bin. As is shown in Figure 5, observers in both experiments showed a fairly strong bias to shift gaze to nearby objects. However, with regard to differences between the Experiment 1 and 2 tasks, these data, along with the previous analyses, suggest that the use of a memory test in Experiment 1 was not meaningfully affecting observers’ oculomotor behavior during search.

Figure 5.

Distributions of saccadic amplitudes for target-absent trials for observers in Experiments 1 and 2.

Manual Data

We also compared observers’ mean reaction times (RTs), error rates, and proportion of targets found for the two experiments to examine whether there were any general differences in search performance that might suggest an effect of memory test in Experiment 1. Independent-samples t tests comparing observers’ RTs in the two experiments revealed no significant differences for either target-present trials (4,282 ms vs. 4,489 ms; Experiment 1 vs. Experiment 2), t(18) = −0.80, p > .05, or target-absent trials (5,689 ms vs. 5,721 ms; Experiment 1 vs. Experiment 2), t(18) = −0.11, p > .05. An analysis of observers’ errors in the two experiments also revealed no significant differences for either responding to the presence of a target with the wrong button (1.64% vs. 0.61%), t(18) = 1.63, p > .05, or false alarms (0.02% vs. 0.3%), t(18) = −1.52, p > .05, in Experiments 1 and 2, respectively. Finally, an analysis of the percentage of targets found also revealed no significant differences (61% vs. 53%; Experiment 1 vs. Experiment 2), t(18) = 1.44, p > .05. As in the case of the eye movement analyses, these analyses suggest that the presence of the memory test did not meaningfully influence how observers searched.

General Discussion

With the debate over whether memory is used during search reaching an end, the question now turns to how much memory exists during search? We found that a substantial proportion of a search history is represented during the course of a search. Observers were able to discriminate fixated locations from non-fixated locations, with this ability extending back to the last 12 locations fixated during search—far more than would be predicted by estimates of VSTM capacity (Alvarez & Cavanagh, 2004; Irwin, 1992, 1996; Irwin & Andrews, 1996; Irwin & Zelinsky, 2002; Luck & Vogel, 1997; Zelinsky & Loschky, 2005). Our estimate of memory capacity for searched locations is also higher than estimates of distractor memory reported in the memory in search literature (e.g., Klein & MacInnes, 1999; McCarley et al., 2003), and even inconsistent with a recent study by Peterson and colleagues (Peterson, Beck, & Vomela, in press) that also showed evidence for a high-capacity memory during search. Their task was designed to examine the relative contributions of prospective memory (i.e., a search plan) and retrospective memory (i.e., a search history) to search behavior. They concluded that both forms of memory contribute to search efficiency, and that the prospective form of memory had a high capacity. However, and in contrast to our results, they also concluded that the retrospective memory component had a fairly low capacity, limited to the 4 to 5 most recently fixated objects.

How do we reconcile our evidence for a high-capacity memory for search history with the many studies showing an extremely limited search memory? Although our conclusion for a high-capacity memory is based on positive and direct evidence and in some sense trumps arguments against its existence, we do not believe that the search literature has been overly hasty in concluding for memory limitations. Rather, we adopt a position similar to the one put forth by Peterson et al. (in press) and suggest that the source of this discrepancy lies in the type of memory being investigated. The concept of memory as a spatially discrete inhibitory tag has dominated the memory in search literature (e.g., Klein, 1988), and the studies designed to reveal these tags have consistently produced capacity estimates in the 3–5 object range, similar to estimates of VSTM capacity. Based on our data, we believe that there exists another, higher capacity, form of retrospective memory in addition to these inhibition-based representations—a memory for the search process itself, not the things that were searched.

Memory for a search path

Having demonstrated that people retain a large amount of information about where they have searched, and assuming that capacity limitations on VSTM prevent the representation of this information as spatially discrete inhibitory tags, it follows from our predictions that observers in Experiment 1 were remembering their search paths.

The importance of a search path is being rapidly realized in the eye movement and search community. In a recent study by Findlay and Brown (2006), observers had to scan the objects in a display, but the sequence in which these objects could be fixated was not constrained by the task (only the first and last objects to be fixated were constrained). Despite the free-viewing nature of the task, these authors reported a high degree of systematicity in observers’ scanning behavior. Moreover, this systematicity took one of two forms: observers either adopted a stereotypical scanning strategy (e.g., scanning left-to-right, top-to-bottom), or they tended to follow a pattern suggested by the layout of the display objects. For example, if display objects were arranged in an “S” type configuration, observers’ gaze might follow the path described by the perceptual grouping of the objects. In a more traditional search experiment, Gilchrist and Harvey (2006) varied the orderliness of objects in a 25-object display. Objects were arranged either in a 5 × 5 grid (yielding a highly organized and symmetrical display), or in 25 randomly selected locations from a 6 × 6 grid (yielding a less organized display) or a 7 × 7 grid (further increasing the degree of disorganization). They found that the majority of saccades in this task were horizontal regardless of display organization, but that this horizontal bias decreased (but did not disappear) as the regularity of the displays decreased.

The Findlay and Brown (2006) and Gilchrist and Harvey (2006) studies demonstrated considerable organization in the path followed by gaze during a variety of challenging search and scanning tasks; we build on this work by suggesting that our observers were representing their search path and using it to make their confidence judgments.

Search paths might be represented in one of two ways. First, and most intuitively, observers may represent the actual trajectory followed during search. As shown in Figure 4, gaze in our task tended to move between neighboring objects. If the path connecting these successively searched locations was then represented (either implicitly or explicitly) in a holistic or configural form, this grouped representation might be used to overcome the object-based capacity limits imposed by VSTM. Peterson et al. (2001) proposed a similar form of grouped representation to explain why their observers refixated distractors only rarely during their search of 12-item displays. Second, a search path might be represented algorithmically. Observers might represent where they searched by representing how they searched—that is, the algorithm used to produce the search path. The influence of this form of path memory would likely increase with the observers’ formation and use of a consistent search strategy. For example, if observers consistently searched left-to-right and top-to-bottom, they could then use their memory for this simple search algorithm, their starting search position, and their current search position to infer whether the probed location fell on this “reconstructed” search path. Note that this differs from a trajectory-based representation in that actual searched locations (other than the starting point and current location) need not be included in the path representation.

Both of these path memory representations can vary in their level of resolution, thereby enabling either to explain our evidence for a fairly low-resolution spatial representation of distractor locations. In the case of a trajectory representation, the precise sequence of vectors connecting each fixated distractor might be grouped to create a high-resolution representation of the search path. Alternatively, the path representation might be sparser, perhaps connecting clusters of objects or gross regions of the display (e.g., upper-left → upper-right → middle, etc.). In the case of an algorithmic representation, algorithms can be constructed with varying degrees of specificity. A highly specific algorithm would have contingencies in place to capture the idiosyncrasies of irregularly spaced distractors (e.g., “look right along the top row, drop down to inspect the oddball item, shift over to the group of three on the left, etc.”), thereby yielding a high-resolution representation of the search path. Simpler algorithms may not be able to capture such display irregularities. To the extent that observers used low-resolution trajectory-based or algorithm-based representations of space in their path descriptions, confusions should arise between fixated object locations and nearby non-fixated object locations, as we observed (see Cohen and Ivry, 1989, for a similar idea). In future work we will attempt to distinguish between these two forms of path representation, as well as to describe how perceptually separable groups, or even groups defined by scene semantics (e.g., the spatial layout of furniture in a scene) might also facilitate memory for a search path and improve search efficiency.

Although a search path representation is very different from a discrete inhibitory tag, both representations code information about distractor locations and are therefore alternative forms of retrospective search memory. As for why search might benefit from a memory of a search path, the answer to this question follows the same logic developed for inhibitory spatial tags (Klein & MacInnes, 1999). By helping to segregate inspected from uninspected display objects, memory for a search path would improve search efficiency by reducing the likelihood of revisiting previously rejected distractors. However, and unlike an inhibitory tag, the usefulness of a path-based search memory may depend on a number of factors, such as the symmetry or regularity of the displayed search items, the complexity of the path description (e.g., the number of direction changes, or the frequency with which the path intersects itself), and the level of resolution that was used to code the search path. It is also likely that the information content of a path representation would be subject to VSTM capacity limitations, meaning that complex paths might sometimes be coded coarsely to maximize the amount of search history that they represent. To the extent that an observer codes a complex search path using a coarse spatial representation, the retained information will be unreliable. Given that less complex paths are easier to remember (Parmentier, Elford, & Maybery, 2005), simple or highly regular search paths might also be more available in memory. Paradoxically, the usefulness of a search path may therefore increase with its complexity (depending on the stimulus), but one’s ability to create and retain an accurate representation of the search path may decrease with path complexity. The advantage of a path-based memory over inhibitory tags, however, is clear: a path memory can represent, albeit sometimes coarsely, the locations of many more distractors inspected during search.

There appears to be no one form of memory available for use during visual search, and the type of memory ultimately used may depend on the specific demands of the search task. When the task makes minimal demands on memory, a small number of inhibitory tags, whose primary use is to keep search moving toward new display objects, may suffice. When the task is more memory intensive, as was the case in our Experiment 1, observers may benefit by supplementing their inhibition-based memory with a memory for the search path so as to reach farther back into their search history. However, a path-based memory is probably most useful (and easiest to use) when the visual search environment remains relatively stable. When studies incorporate display changes into their design (thereby undermining the value of encoding a path; e.g., Peterson, Beck, & Vomela, in press) or present small subsets of the search stimuli sequentially over time (thereby making path encoding more difficult; e.g., Beck et al., 2006; Boot et al., 2004; McCarley et al., 2003), they may be thwarting the use of a path memory. Given that search in the real world often takes place under relatively stable conditions, it is possible that search makes greater use of a path-based memory than what was revealed by these studies.³ Moreover, to the extent that studies have created conditions favorable to a path memory, it is also possible that path representations might have contributed, at least in part, to the evidence for memory reported in the search literature (Aks et al., 2002; Dickinson & Zelinsky, 2005; Gilchrist & Harvey, 2000; Kristjansson, 2000; Müller & von Mühlenen, 2000; Peterson et al., 2001).

In conclusion, we envision a very fluid relationship between visual search and memory, one in which the search process exploits whatever memory representations are available to best meet the demands of the immediate search task. The long-term memory literature tells us that half the battle in finding evidence for memory involves finding the right memory test, and that care should be taken to match encoding and testing conditions before concluding for a memory limitation (Tulving, 1983). When we provided observers with a stable and accurate cue to the locations of the search objects, they were able to discriminate inspected from uninspected objects at points in their search histories that far exceeded traditional estimates of memory capacity during search.