You look familiar, but I don’t care: Lure rejection in hybrid visual and memory search is not based on familiarity

Abstract

In “hybrid” search tasks, observers hold multiple possible targets in memory while searching for those targets amongst distractor items in visual displays. Wolfe (2012) found that, if the target set is held constant over a block of trials, RTs in such tasks were a linear function of the number of items in the visual display and a linear function of the log of the number of items held in memory. However, in such tasks, the targets can become far more familiar than the distractors. Does this “familiarity” – operationalized here as the frequency and recency with which an item has appeared – influence performance in hybrid tasks In Experiment 1, we compared searches where distractors appeared with the same frequency as the targets to searches where all distractors were novel. Distractor familiarity did not have any reliable effect on search. In Experiment 2, most distractors were novel but some critical distractors were as common as the targets while others were 4× more common. Familiar distractors did not produce false alarm errors, though they did slightly increase response times (RTs). In Experiment 3, observers successfully searched for the new, unfamiliar item among distractors that, in many cases, had been seen only once before. We conclude that when the memory set is held constant for many trials, item familiarity alone does not cause observers to mistakenly confuse target with distractors.

Introduction

Consider two of the basic paradigms of cognitive research. In a standard visual search task, observers look for a single target item among a number of distractor items in a visual display. In a standard memory search task, observers are asked if a single probe item is a member of a set of items being held in memory. In their 1977 papers, Shiffrin and Schneider introduced the term “hybrid search” to describe the combination of visual search and memory search in a single task (Schneider & Shiffrin, 1977; Shiffrin & Schneider, 1977). In such a task, observers are given a set of items to hold in memory and subsequently asked if any of those items are present amidst an array of distractor items in a visual display. Hybrid search tasks allow us to investigate the interactions of visual and memory search.

In the same body of work, Shiffrin and Schneider introduced the more famous distinction between “consistent mapping” (CM) and “variable mapping” (VM) tasks in memory search. In CM tasks, targets are always drawn from one set of items and distractors (or “foils”) are always drawn from a second, fixed set. In VM tasks, targets and distractors are drawn from the same set. Therefore a target item on one trial can be a distractor item on another.

In classic hybrid search experiments and in much of the memory search literature, a new memory set is presented on each trial. Thus, even in a CM condition, if the target set for the block was {1,2,6,8,9}, the memory set might be {2,6,9} for one trial, {1,6} for the next and so on. Wolfe (2012) introduced a somewhat different version of the hybrid search task in which observers searched for exactly the same set of target items during an entire block of trials. For example, in five separate blocks, observers memorized sets of 1, 2, 4, 8, or 16 isolated photographs of objects. Within a block, observers performed visual searches for any one of the items in the current memory set, in visual displays of 1, 2, 4, 8, or 16 objects. The use of object photographs allowed memory set sizes much larger than those in traditional hybrid search tasks since observers are able to remember hundreds of images with high fidelity (Brady, Konkle, Alvarez, & Oliva 2008; Konkle, Brady, Alvarez, & Oliva 2010; Shepard, 1967; Standing, 1973; Standing, Conezio, & Haber, 1970). Evidence for this large capacity was also demonstrated in Experiment 2 of Wolfe (2012), where observers searched visual displays for any of 100 items held in memory.

Because object photographs are an essentially unbounded set, it was also possible for every distractor item in an experiment to be novel (or almost so, in Wolfe, 2012). Thus, the Wolfe (2012) experiments can be described as having consistently mapped targets (perhaps, unusually consistently mapped) and “All New” (AN) distractors. This condition could be labeled CM+/AN− following the notation used by Nosofsky and colleagues, in which the “+” term refers to the type of targets and the “−” term refers to the type of distractors (Nosofsky, Cao, Cox, & Shiffrin, 2014; Nosofsky, Cox, Cao, & Shiffrin, 2014).

Hybrid search in Wolfe (2012) produced low error rates and RTs that were a linear function of the visual set size. Such functions are typical in visual searches for specific objects (Vickery, King, & Jiang, 2005; Wolfe, Alvarez, Rosenholtz, Kuzmova, & Sherman, 2011). However, RTs were not a linear function of the memory set size. Rather, they were curvilinear. Curvilinear (or bilinear) effects of memory set size had been reported before (Burrows & Okada, 1975). Most of the classic work in this area was done using alphanumeric characters but this finding has been recently replicated by Nosofsky et al. (2014a; 2014b) using objects, in a classic memory search paradigm that used different memory probes on each trial. Various accounts of the non-linear function have been proposed. The topic is reviewed by Shiffrin (1988) and by Van Zandt and Townsend (1993). A log function has been suggested by some researchers (Briggs, Johnsen, & Shinar, 1974). Shiffrin (1988) argued for a form of a speed-accuracy trade-off as an explanation of the departures from linearity. Theios (1975) suggested a two-part function with small memory set sizes governed by retrieval from short term memory and larger set sizes making use of retrieval from long term memory. Wolfe (2012) claimed that the memory set size functions were quite precisely logarithmic.

The pattern of a linear effect of visual set size combined with a curvilinear (logarithmic) effect of memory set size is not restricted to memory for specific images of specific objects. Specifically, Cunningham et al. (2014) demonstrated that it holds for categories of objects. For example, a search for any instances of “animals”, “coins”, or “clothes” in an array of objects would be a hybrid search with a memory set of 3 categories, not a memory set of the infinite number of instances from each of those categories. In the Cunningham et al. experiment, the task can be considered a CM+/CM− at the category level, since the sets of target categories and distractor categories remain constant over a block of trials. At the same time, the task is AN+/AN− at the object level, since new instances of each category were used on each trial. A similar pattern of results can be found with memorized lists of arbitrary words or even for the words from well memorized passages (song lyrics, poetry, etc.; Boettcher & Wolfe, 2014).

Cunningham et al. (2014) propose a three-step account for how observers carry out these hybrid search tasks:

1. An object in the visual display is selected. This is a ‘guided’ visual search (Wolfe, 1994, 2007; Wolfe, Cave, & Franzel, 1989) where only items that might be plausible members of the memory set, based on physical features, are selected. Therefore, if the memory set consists entirely of photorealistic objects, an alphanumeric symbol would not be selected during this first stage.

2. The object is recognized and categorized by whatever mechanisms perform object recognition.

3. If and only if the recognized object could be a member of the memory set, then a logarithmic search through the memory set is executed. That is, if the memory set is known to be composed of animals, there is no need to do a memory search if the selected item is recognized and categorized as a coffee maker. These steps are repeated until a target is found or the search is otherwise terminated.

Cunningham et al (2014) offer a diffusion account of the logarithmic form of the memory search functions from step 3. They propose that each member of the memory set accumulates information over time from the selected items. This contributes to N diffusion processes where N is the size of the memory set. If the information reaches a decision boundary for one of the items, the observer identifies the selected visual item as that member of the memory set. The decision bound – the amount of evidence required to make a positive response – needs to be set at a level that prevents excessive false alarm errors. Set too low, and a random diffuser could cross the decision bound by chance. The probability of a false alarm would increase with the size of the memory set and the resulting number of simultaneous diffusion processes. To hold the false alarm rate constant, the decision boundary must rise with the memory set size. This will increase the RT as memory set size rises. Under a reasonable set of assumptions, the form of that rise turns out to be logarithmic (Leite & Ratcliff, 2010).

How does the observer know that a target in the visual display is a member of the memory set in a hybrid search task? Understanding how we know that an item is or is not part of a set held in memory has been central concern of memory theorists for decades. Many models fall into the general category of “dual-process” models that distinguish between recollection and familiarity (Mandler, 1980; Yonelinas, 2002; Eichenbaum, Yonelinas, & Ranganath, 2007; Wixted & Mickes, 2010). To quote Yonelinas (2002), “the distinction is illustrated by the common experience of recognizing a person as familiar but not being able to recollect who the person is or where they were previously encountered” (p441, emphasis added). Although Cunningham et al. (2014) were not explicit on the topic, their model suggests that observers give a positive response when one diffuser hits its decision bound and the observer recollects that a specific object is a member of the memory set. Recollection, however, is not the only option. Given the CM+/AN− design of the Wolfe (2012) experiments, positive responses could be based on a simple familiarity signal. The targets are all familiar items. The distractors, being new objects on each trial, are not familiar.

Moving beyond the merely colloquial use of the term “familiar”, familiarity is a central concept in models like Nosofsky “exemplar-based random-walk model” (EBRW). In EBRW, exemplars of items are stored when they are presented. More frequently presented items will be more familiar. There will also be effects of position in a study list, though those will be of secondary importance in the present work. Items are deemed to be “old” based on their similarity to stored exemplars (Nosofsky, Little, Donkin, & Fific, 2011). In a hybrid search task like those of Wolfe (2012) or Cunningham et al. (2014), items in the memory set will be studied initially while distractor items will not be. Moreover, target items will appear more often during the search task, rendering them more familiar. When an observer searches through a visual display, the summed similarity of each item to the stored exemplars could be the basis for determining the presence or absence of a target item.

The exact mechanisms used in hybrid search tasks are not yet established. In their study of the speed of visual recognition memory, Besson et al. (2012) included conditions similar to those of Wolfe (2012) with a visual set size of one item. Because Wolfe (2012) had no time limit on responses, Besson et al. argue that “It is unclear in the findings reported by Wolfe whether it is recollection or familiarity that shows this logarithmic relationship.…This hypothesis clearly merits further study.” (p1145, Besson, et al., 2012). Guild et al. (2014) argue for a contribution from recollection. They reduced the effects of familiarity by reducing the prevalence of targets, making them less familiar. This manipulation preserved the curvilinear relationship between RT and memory set size. However, it also dramatically raised error rates. This leaves open the possibility of a substantial role for familiarity in the Wolfe (2012) results, where error rates were low. Nosofsky et al. (2014a) used the same object stimuli used in Wolfe (2012) in a series of memory search experiments. The use of objects allowed them to use memory set sizes larger than those usually used in the memory search. In CM conditions, they found evidence for contributions of both familiarity and a “categorization” process. In classic CM experiments, the memory set on the current trial is always a subset of a larger set of “positive” items – items that could be part of the memory set on a specific trial. Any members of this larger set can be categorized as “old” or “target” so the observers in a CM+ experiment do not need to pay any attention to the current memory subset. The category of the item suffices to give the response. Nosofsky et al (2014) show that that a similar process is at work in CM- experiments where all the non-target or “new” items are drawn from a fixed set of items (the “negative” category). Here too, one can ignore the specific memory set for the trial because, if the item is in the negative category, the answer is “no” or “new”. This work is highly relevant to our hybrid search work, though in our tasks the memory set for the trial is always the same as the memory set for the task (rather than a subset). If you are holding 8 items in memory, the target on any specific visual search trial can always be any of those 8.

While the literature suggest that familiarity plays an important role in memory search when presented with a single item and when the targets in the current trial may be a subset of the larger trial, it is currently unclear what role familiarity plays in hybrid search tasks of the Wolfe (2012) variety, where the memory set is the same on every trial. To anticipate the results, if we define familiarity as a function of how often an item has been seen and, to a lesser extent, how recently it has been seen, we find that familiarity plays a small role in hybrid search tasks. When targets and distractors appear equally often in the visual display (a version of a CM+/CM− task), the results are the same as when distractors are novel on each trial (CM+/AN−) (Experiment 1), suggesting a strong role for categorization, as defined by Nosofsky et al (2014a). When individual distractors are as common or more common than the targets, observers are not attracted to these lures; at least, not to the point of committing false alarm errors (Experiment 2). Finally, when the target is new and unknown on each trial, observers are still able to perform a hybrid search for the new item, even among old items that are, themselves, quite unfamiliar (Experiment 3). Taken together, these experiments suggest that modulation of familiarity does not disturb search in hybrid search tasks similar to those in Wolfe (2012).

Experiment 1: Balanced Familiarity

Observers

Twelve observers (based on the sample sizes in Wolfe, 2012) were tested. All observers were paid volunteers ($10/hr) who had given informed consent. Each had at least 20/25 visual acuity and normal color vision as assessed by the Ishihara Colour-Blindness test.

Methods

Experiment 1 is similar in design to the Wolfe (2012) hybrid search experiments. Observers memorized N number of targets before searching for those targets among K number of distractors. The critical variable in this experiment was the status of the distractors. In the AllNew condition, distractors were sampled from a large set of items, so that no distractor was ever seen more than once in the experiment (CM+/AN−). In the FrequencyBalanced condition, distractors were sampled from subsets designed so that the average number of appearances of each distractor was the same as the average number of appearances of each target (CM+/CM−). Consider, for example, a memory set of 8 objects. Targets will show up on 200 of the 400 trials. Since there are 8 different targets, each will make 25 appearances. Given visual set sizes 2, 4, 8, & 16, there will be 2800 slots for distractor items. Thus, if we use 112 distinct distractors (2800/25), each distractor will appear 25 times, matching the targets.

Figure 1 illustrates the components of Experiment 1. In Figure 1a, observers were exposed to 2, 4, 8, or 16 objects that would serve as their memory set for the entire block of trials. The order of memory set sizes was randomized across observers. To help them memorize the target items, observers were presented with each item in isolation for 3 seconds at a time. Next, as shown in Figure 1b, observers were tested in an old/new discrimination paradigm to make sure that they had, in fact, memorized the items. Observers needed to exceed 90% correct on two memory tests. Each memory test consisted of 2*N items where N was the memory set size. Half of the items were old (target items from their memory set). In the FrequencyBalanced condition, the other half were drawn from the set of items that would be used for distractors in the visual search task, as described above. In practice, observers were perfect on 66% of memory tests. The minimum of two memory tests was sufficient 75% of the time. At most, five memory tests were required to pass in the case of one observer. Observers received feedback after each trial.

Figure 1.

A. Observers are exposed to the N items that will serve as the memory set for a block of trials. B. Observers are tested to confirm that the set is memorized. C. Observers perform 400 target-present/target-absent visual search trials. This is an example of a trial sequence from the FrequencyBalanced condition where distractors appear just as often as the targets.

Having passed the memory test, observers performed 16 practice and 400 experimental trials of visual search. Targets were present on 50% of trials and the visual set sizes were evenly divided among set sizes 2, 4, 8, and 16. Stimuli were visible until observers responded. Response time (RT) and accuracy were recorded. In this experiment, as in Wolfe (2012), the target could be any member of the memory set and this memory set remained constant over the 416 trials. Distractors were drawn from a set of objects that did not include the target objects. In the FrequencyBalanced condition, the number of distinct distractors was chosen so that distractor appearances would match the frequency of target appearances. Thus, using the example of a memory set size of 8, discussed above, 112 distinct objects would be selected to make up the distractor set. Objects were chosen from this set at random for each trial to complete the visual set size.

Observers responded with a target-present or target-absent key press. Observers received feedback after each trial specifying the status of the trial (Hit, false alarm, miss, or true negative) but the correct target was not highlighted on miss trials.

Apparatus

Experimental sessions were carried out on a Macintosh G4 computer running Mac OS 10.5. Experiments were written in MATLAB 7.5 with the Psychophysics Toolbox (Version 3; Brainard, 1997; Pelli, 1997). Stimuli were presented on a 20-in. CRT monitor (Mitsubishi Diamond Pro 91TXM) with resolution set to 1280 × 960 pixels and an 85-Hz refresh rate. Observers were placed so that their eyes were approximately 60 cm from the monitor. At this viewing distance, 1 cm subtends 1° of visual angle.

Stimuli

Stimuli were color photographs of objects on a white background. On the search trials, objects were presented in random cells of a slightly irregular 5×5 array. The array subtended 28.5 × 28.5 deg at the approximately 60 cm viewing distance. Each object, therefore, fit into a 5.7 × 5.7 deg cell.

Results

RTs were filtered to remove times below 200 msec and greater than 5000 msec.¹ This removed 0.6% of total trials. The main result is clearly seen in Figure 2 where the average RTs are plotted. There is no effect of the relative familiarity of the targets and distractors.

Figure 2.

Average reaction time for 12 observers as a function of memory set size in Experiment One. Larger, light green, outlined symbols show data from the FrequencyBalanced condition. Smaller, darker purple symbols show the AllNew condition. Error bars (where visible) are +/− 1 within-observer S.E.M.; using the corrected method of Cousineau (Cousineau, 2005; Morey, 2008).

This impression is borne out by separate 3-way ANOVAs for the target-present and target absent data. The main effects of visual set size and memory set size are strongly significant (all F(3,33) > 49, all p<0.0001, all generalize eta-square (ges) > 0.22). The effects of condition do not approach significance (both F(1,11)=0.2, p > 0.6, ges = 0.0005). In both cases, Bayes Factor (Masson, 2011) finds 3× more support for the null hypothesis than for an effect of distractor familiarity. Only the visual set size × memory set size interactions were significant (both F(9.99) > 30, p<0.0001. No interactions with condition were significant.

Figure 3 shows error rates as a function of memory set size. Error rates are quite high for the combination of the largest memory and visual set sizes. However, there is no evidence that the FrequencyBalanced condition produces larger rates of error than the AllNew condition. Statistical analyses were performed on arcSin transformed error data (Hogg & Craig, 1995). Three-way ANOVAs on miss and false alarm errors separately show main effects of memory and visual set size for each kind of error (all F(3,33) > 24, all p<0.0001, all ges > 0.14). There is a modest main effect of condition for false alarm errors (F(1,11)= 6.8, p=0.025, ges=0.03). Note, however, that it goes the “wrong way”. If hybrid search were based on familiarity, we would expect more errors in the FrequencyBalanced condition. In fact, there are more false alarm errors in the AllNew condition. The Bayes Factor of 0.196 is equivocal, supporting neither hypothesis. Three observers have quite large error rates. If they are excluded from the analysis, the overall pattern of results remains the same. There is no difference between FrequencyBalanced and AllNew conditions.

Figure 3.

Error rates for 12 observers as a function of memory set size in Experiment 1. Errors are averaged over visual set size. Pale green outline symbols show data from the FrequencyBalanced condition. Darker purple solid symbols show the AllNew condition. Error bars (where visible) are +/− 1 within-observer S.E.M. using the corrected method of Cousineau (Cousineau, 2005; Morey, 2008).

Lag Analysis

In their examination of memory search, Nosofsky et al (2014b) find important effects of “lag”. In the Nosofsky experiments, observers get a new memory set for each trial. In their work, “lag” refers to the serial position of an item in the study set for that trial relative to the probe item for that set. Thus, using letters for convenience, suppose that, on one trial, the memory set that was presented was {A B C D E F}, in that order. A test probe of “E” on that trial would have a lag of 2 while “A” would have a lag of 6. On the short time scale of a single trial, familiarity is greater if lag is shorter.

In the hybrid search paradigm used in the present work, that definition of lag is meaningless since the memory list was memorized at the start of a block and held for the entire block. However, it is interesting to examine another, related lag. How many trials intervene between instances of the same target item? If a chicken appeared as the target on trial 10, does the number of trials between the current and the previous appearances systematically affect the RT for the current occurrence of the chicken? That is, are observers faster to respond to a chicken if it appears on trial 12, rather than on trial 22? The interest in this form of lag is based on the notion that finding a target would serve to give that target a boost in familiarity or, at least, ‘priming’ that target. In visual search for simple stimuli, finding a target on one trial speeds response on subsequent trials (Hillstrom, 2000; Kristjánsson & Campana, 2010; Maljkovic & Nakayama, 1994; Wolfe, Butcher, Lee, & Hyle, 2003). See Huang et al. (2004) for an explicit tie between priming in search and episodic retrieval in memory.

Because targets for each trial are selected at random from the memory set, lags are distributed differently for memory sets of different sizes and are always skewed toward short lags (e.g. with a memory set of 2, it is hard to get a lag of 10 as that requires that 10 successive trials have the same target, the other item in the set). In order to achieve a reasonable number of trials per data point, lags were divided into 5 separate bins (0, 1, 2, 3, 4) defined by log2 of the actual lag, rounded to the nearest integer. For example, a trial with lag 7 would be averaged into bin #3 since the log2(7) = 2.8 which we round to 3. The average lags in the five bins are 1, 2, 4, 8, & 16 respectively. Thus, the trial with a lag of 7, is averaged into a bin with an average lag of 8.

Figure 4 shows that there is a clear effect of lag on RT. The figure shows lag effects for each memory set size, averaging across visual set sizes for the FrequencyBalanced and All New conditions. The patterns are similar in the two conditions. Clearly, observers are primed by the targets on recent trials to respond more quickly to repetitions of those targets. The effect plateaus or grows more slowly at longer lags. A two-way ANOVA (removing memory set size 2 because of low trial counts at high lags) shows significant main effects of lag (F(4,44=14.3, p<0.001, ges=0.087 and memory set size (F(2,22)=7.75, p=0.003, ges=0.068). The interaction is also significant (F(8,88) = 2.91, p=0.006, ges=0.040). The pattern is the same for the AllNew condition: significant main effects of lag (F(4,44=6.72, p<0.001, ges=0.039) and memory set size (F(2,22)=4.35, p=0.026, ges=0.036). The interaction is also significant (F(8,88) = 2.46, p=0.019, ges=0.029). The shapes and magnitudes of these lag effects are quite similar to those shown by Nosofsky et al (Figure 3 of 2014a) even though lag in our experiments refers to the time since the last testing of the specific target. In Nosofsky et al, it refers to the serial position in the memory set for the current trial.

Figure 4.

The effects of lag: number of trials between the current trial and the last appearance of the same target item. RTs are binned by log2(lag). Error bars show +/− 1 SEM (within-observers by the Cousineau/Morey method)

Discussion

The central result of Experiment 1 is that the Frequency Balanced and All New conditions produce essentially the same results, even though the distractors in the Frequency Balanced condition repeat many times during a block of trials while the distractors in the All New condition are unique every time. Given the usual difficulties with null effects, this finding does not rule out familiarity as the primary signal, supporting responses in hybrid search, though the Bayes factor calculation indicates that the absence of a familiarity effect is 3× more likely than the presence of a familiarity effect. Observers do not appear to rely upon the relative familiarity of targets as compared to distractors in order to perform the hybrid search task.

On the other hand, the lag data point to a role of familiarity; at least if we assume that the appearance of a target on one trial makes it more ‘familiar’ for some trials thereafter. The lag effect is large, on the order of 400 msec for a memory set size of 16. Note that this effect of target familiarity would be identical in Frequency Balanced and All New conditions and so would not appear in the average RT data of Figure 2. We can infer from this result that the priming/familiarity effect of encountering a distractor again is negligible. In the CM+/CM− Frequency Balanced condition, distractors are encountered repeatedly. In the CM+/AN− All New condition, this never happens. As we have seen, this difference does not affect performance. Nosofsky et al. (2104a) come to a very similar conclusion. Their Experiment 2 included CM+/CM− and CM+/AN− conditions and found no differences in performance. Note an important difference between these experiments: in Nosofsky et al. (2104a), the memory set on each trial is a subset of the overall memory set while the present experiment has a fixed memory set for all trials.

Perhaps observers in the Frequency Balanced condition learn to rely on something other than familiarity because familiarity was so evidently not useful under the conditions in Experiment 1. In Experiment 2, we tried a different approach. Rather than making all distractors familiar, we made one distractor very familiar in an effort to lure observers into making false alarm errors based on this familiarity signal.

Experiment 2: Familiar Lures

Observers

Fifteen observers were enrolled in the experiment. One observer was removed from analysis due to excessive errors. Note, these were “miss” errors, not false alarms to the lures. All observers were paid volunteers ($10/hr) who had given informed consent. Each had at least 20/25 visual acuity and normal color vision as assessed by the Ishihara Color-Blindness test.

Methods

Experiment 2 was similar to Experiment 1. The same stimuli were used. First, the observers memorized a set of 4 or 16 random objects as the memory set. In the initial presentation, the objects were presented alone on screen for 3 seconds. The memory test for these targets was somewhat different than that in Experiment 1. As before, memory was tested by asking observers to give an old/new response to objects presented, one at a time in the center of the screen. Participants had to perform at better than 90% on two blocks of the memory test before moving on to the search condition. Each memory test had one target present trial for each target in the memory set and three types of non-target trials. Distractors during the memory test appeared with different probabilities. There were distractors that only appeared once during the entire memory test (as in the AllNew condition of Experiment 1). There were lure items that appeared as frequently as the true target items (as in the FrequencyBalanced condition). These two types of distractor were identical in the initial memory test but, as we will see, they differ in the hybrid search task. Finally, there were strong lures that appeared 4 times as often as any true target item. The total number of trials in each memory test was 4 times the memory set size. In the memory test section of the experiment, the intent was to repeat lure objects so that participants would become familiar with them. Observers were given feedback about the accuracy of their responses but were not alerted to the structure of the distractor probabilities.

Once the participant passed the memory test, they moved to a search phase with visual set size of 6. Observers completed 150 visual search trials in each of two memory set sizes: 4 and 16. Memory set size order was counterbalanced across participants. Observers were asked to find the target object in the visual display that was a member of their memory set and click on it with the mouse. For target-absent trials, they were instructed to click in an “absent” box on the left of the screen. In one third of the trials, all distractors were new on each trial. In the other two thirds of the trials, there were lures among the distractor objects. On half of these trials, there was a lure that had appeared in the memory test with the frequency of the target items, the other half of the lure-present trials contained a lure that had been seen 4 times as often as any target. Note that a 4× lure also appeared 4× more frequently during the hybrid search block. Lure presence or absence was independent of target presence or absence. Observers were given feedback about the correctness of their responses.

Once they completed the search phase, observers repeated two blocks of the memory test to determine whether they were mistaking the frequent lures for targets at the end of the experiment. Again, in addition to the targets, some lure distractors appeared with the same frequency as the members of the target set while others appear with 4× of any memory set item. There were 64 total trials at memory set size 4 and 256 at memory set size 16.

Results

Memory Test Results

Most observers performed at better than 90% accuracy on the memory tests and, thus, performed only 2 initial blocks of the memory test at each memory set size. Two subjects required 3 initial memory tests for memory set size 16. One required 4.

There were very few false alarm errors; none, at memory set size 4. At memory set size 16, there were 2.5% false alarms for 1× lures and 3.6% for 4× lures, and observers false alarmed for 1.8% of novel distractors. Clearly, there is not a strong tendency for the lures to be mistaken for targets on the basis of their familiarity. There is a hint of an effect of lures in the RT data (though we were stressing accuracy and not RT in the instructions for this memory task). An ANOVA reveals a main effect of lure type (F(3,33)=5.8, p=0.003, ges=.08) and an interaction of lure type and memory set size (F(3,33)=5.9, p=0.002, ges=.08). There is no main effect of memory set size (F(1,11)=0.002). These effects are driven by longer RTs for the very common, 4× lures, only for memory set size 4. Perhaps the common lures, literally, gave observers pause. Observers were slowed, but they did not mistake those lures for targets.

For the memory tests after the search trials, observers made essentially no false alarm errors to lures (a total of 9 out of 1916 lure trials over all observers in all conditions). The RT data reveal a main effect of Lure Type (F(3,33)=3.5, p=0.025, ges=0.06) but this is mostly driven by slightly slower target present RTs. That effect is most likely due to the fact that target absent responses are more common in this design (75% of trials). More prevalent responses tend to be faster (Wolfe & VanWert, 2010). Overall, the memory test data from before the search trials show an effect of lures on RT that does not translate into a significant effect on errors. The memory test data from after the search trials show no reliable evidence that familiar distractors were mistaken for targets.

Visual Search results

RTs were filtered to remove trials with RTs less than 200 msec or greater than 6000 msec. This removed 3.2% of trials, most of them RTs recorded as 0 msec due to a coding problem. Figure 5 shows average RT as a function of memory set size for the three lure conditions of Experiment 2.

Figure 5.

RT as a function of memory set size for each of the lure conditions. Error bars a +/− 1 S.E.M. (within-observers with the Cousineau / Morey method)

There is, of course, a sturdy effect of memory set size (F(1,13)=7.59,p=0.016, ges=0.088). There is a modest effect of Lure condition on target present trials (F(2,26)=3.68,p=0.039, ges=0.011). The Bayes Factor for this effect is equivocal (0.86). The interaction is not significant (F(2,26)=0.30,p=0.742, ges=0.001). For the absent trials, only the main effect of memory set size is significant (F(1,13)=17.88,p=0.001, ges=0.081). The effect of Lure condition (F(2,26)=0.18, p=0.835, ges=0.000) and the interaction are not significant (F(2,26)=2.54, p=0.10, ges=0.004). The Bayes Factor indicates that the null hypothesis is 26× more likely than an effect of Lure.

Turning to the errors, the critical question is whether observers produce an elevated rate of false alarm errors. The false alarm rates were low and averaged 2.6%, 1.6%, and 1.3% for the 1× lures, the 4× lures, and the No Lure trials. There were no significant effects of memory set size or Lure status on false alarm rates Specifically, the effect of Lure status produces no effect, F(2, 26)=1.7, p=0.20, ges=0.036. Similarly, for miss errors, the rates were 3.1%, 3.7%, and 3.0% for the 1× lures, the 4× lures, and the No Lure trials, respectively. Again, there were no significant effects of memory set size or Lure status on errors (F(2, 26)=0.8, p=0.46, ges=0.011.) Bayes Factor favors the null hypothesis by 12×

Discussion

Experiment 2 makes the same point as Experiment 1 in a slightly different manner. The familiarity status of items in a hybrid search task exerts very little influence on performance. The design of Experiment 2 was intended to tempt observers into errors by mixing in lure items that were familiar but were not members of the target set. Lures produced a measurable effect on RTs for target present trials, as if observers sometimes attended to a lure and took a bit longer to reject it as a non-target. This effect was not seen on absent trials. If observers were making decisions on the basis of a familiarity signal, the lures that appeared as often, or even more often than the targets should have been chosen, resulting in more false alarms. However, whether or not RTs were somewhat slowed, observers successfully rejected the lures. Error rates were low and did not depend on the lure status of a trial. The speed and accuracy with which this hybrid search is performed supports the hypothesis that the target/distractor decision is made on the basis of the item’s categorical status as a member of the memory set, or not. The signal for category membership is not a simple function of the frequency with which items were presented in training or during the search experiment. The existence of small effects on RT indicates that observers were sensitive to the familiarity manipulation but it did not govern their responses.

The results of this experiment are reminiscent of the results in the more classic memory search version found in Experiment 1 of Nosofsky et al (2014a). Their lures were items that had been presented in the memory set of the previous trial. These were items in the overall memory set but not in the memory set for the current trial. The ability to avoid false alarms to those items was taken by Nosofsky et al. as evidence that something more than familiarity, specifically, categorization, must be supporting memory search behavior.

Experiment 3: Unfamiliarity

Experiment 3 made use of a different method to undermine the use of a simple familiarity signal in hybrid search. Specifically, observers were told to find the novel item on each trial. Seen as a familiarity task, observers were being asked to look for the least familiar item in the display but, as will be seen, observers were looking for a novel target among distractors that had only been seen once, hundreds of trials earlier. While targets and distractors in Experiment 1 were matched at a relatively high level of familiarity, the targets and distractors in Experiment 3 are nearly matched in low familiarity,

Observers

Twelve observers were tested. All gave informed consent and were paid for their time. All had 20/25 or better visual acuity, with correction, and normal color vision as assessed by the Ishihara Colour-Blindness test.

Methods

In this experiment, observers clicked on the new item in each search display. All distractors were items that had been the targets on a previous trial. Thus, the memory set size increased from trial to trial. Initially, the visual set size also grew from trial to trial as the number of available targets grew. After the visual set size reached 12, however, all subsequent trials had a visual set size of 6 or 12, randomly assigned. Observers completed 600 trials. Since their memory set grew by one item on each trial, this means that the memory set rose to 600 over the course of the experiment. This is far larger than the 100-item maximum used in Wolfe (2012). Moreover, there are no memorization trials here. Each item becomes part of the memory set simply by having been identified as the target on a single trial. This is analogous to coming to recognize a face or a scene after a single, real-world encounter. Additionally, if observers clicked on the wrong item, they were informed that the item was not the target and were instructed to keep clicking until they found the novel target item. These trials were recorded as incorrect trials.

Stimuli were drawn at random from the 2400+ objects in the image set used in Experiment 1 and Experiment 2. Since distractors were drawn from targets from previous trials, in later trials, a target, appearing for the first time could be seen amidst distractors, which had been seen only once before; perhaps, several hundred trials previously.

Methods were otherwise similar to previous experiments. Observers sat about 57 cm from a 20” CRT. Objects fit within a 2.5° square.

Results and Discussion

For analysis of the RTs, RTs greater than 10 seconds were removed. 75% of these were errors and for the error analysis, these trials are included. Given that each memory set size appears only once for each observer and Wolfe (2012) showed that the effects of memory set size are logarithmic, RTs are binned by log2(memory set size). The results, shown in Figure 6, are the average RTs in those bins. As a consequence, the data points for higher memory set sizes include many more trials than those for lower memory set sizes.

Figure 6.

RT as a function of memory set size (log scale) for the two visual set sizes in Experiment Three. Error bars a +/− 1 S.E.M. (within-observers with the Cousineau / Morey method)

As in other hybrid search experiments (Wolfe, Drew, & Boettcher, 2014), RT is a linear function of the log of the memory set size. The lines on the figure are best-fit regressions, excluding memory set sizes 16 (relatively few trials) and 512 (see below). The correlations (r²) are over 0.94 for both visual set sizes. Correlations would be over 0.92 with all plotted data included. Regression lines were calculated excluding the largest, 512 set size bin in order to illustrate that RTs fall below the regression line when the set size is large. This appears to be a speed-accuracy trade-off (though it could also be a practice effect, given that these trials, necessarily occur later in the block). As shown in Figure 7, errors rise as memory set size rises. In this experiment, that is entirely unsurprising. If a novel target is not successfully encoded into the memory set at the moment of its first appearance, it will be incorrectly judged to be novel on later appearance. Such errors accumulate over the course of the experiment.

Figure 7.

Error rate as a function of memory set size (log scale) in Experiment Three. Error bars a +/− 1 S.E.M. (within-observers with the Cousineau / Morey method)

The effect of memory set size on error rate is highly significant (F(5,55)=41.0, p<0.001, ges=0.65). The effect of visual set size (more errors at set size 12) is not shown here but is also highly reliable (F(1,11)=51.8, p<0.001, ges=0.13). The interaction is not significant.

Returning to Figure 6, the slope of the RT × log(memory set size) function gives a measure of the cost of adding an item to the memory set size (on a log scale). Thus, for a visual set size of 6, the cost of adding a log item to the memory set is 191 msec. For set size 12, the cost is 291 msec. This cost scales with the visual set size because observers need to perform memory searches on twice as many items at visual set size 12 than at visual set size 6. In Figure 8, we plot the slopes of these RT × log(memory set size) functions slopes as a function of visual set size for Experiment 3 and we plot the comparable data from Experiment 2 of Wolfe (2012). In the older experiment, observers localized the old target among new distractors. In the current experiment, observers localize the new target among old distractors. In other ways, the two experiments are methodologically very similar.

Figure 8.

Top: Slope of the RT × log(memory set size) function as a function of the visual set size. Squares show data from Exp 3. Circles show data from Exp 2 of Wolfe (2012). Bottom: Intercepts of the same functions.

Notice that the slopes for Experiment 3 lie on the same function as the slopes from Experiment 2 in Wolfe (2012). This suggests that the same type of memory search is occurring in the “find-the-old item” and the “find-the-new item” tasks or, at the very least, the efficiency of the two memory searches is very similar. In contrast, the intercepts of those functions are markedly longer for the find-the-new item task (Exp 3), suggesting that the decision to call something novel is considerably slower than the decision to call something old. This, in turn, points to a role for familiarity in hybrid search. If the decision were simply made on the basis of a categorical tag (memory set or not) attached to the item, it is hard to see why there should be a difference in the intercepts. Perhaps observers are somewhat faster to decide in the find-the-old task because the old item looks familiar. The right category label plus a feeling of familiarity might be faster than the right category and unfamiliarity. This could be thought of as analogous to a visual search asymmetry (Treisman & Gormican, 1988; Treisman & Souther, 1985) where the presence of a property (in this case, familiarity) is easier to respond to than the absence (Wolfe, 2001).

It is striking how well people perform in this task. Recall that there is no rehearsal or memory test here. Once the target is found, it must be immediately integrated into the memory set and the next trial appears immediately thereafter. In addition, observers are finding a novel target among distractors that are not very familiar. Since distractors are drawn at random from the set of all previous targets, none of them are very common. No distractor had a prevalence of more than 6.9% (the most common being the first target item, which has the most opportunities to appear again as a distractor). Fully, 50% of distractors are items that appear less than 2% of the time. The lag between the time that an item first appears and the time it reappears gets longer and longer as the block of 600 trials goes on. As an example, the average ‘age’ of distractors on trial 300 are 168, yet no errors were made on this trial. Observers are able to find a novel target among distractors that are members of the “old” or “previous target” category even though they have not been seen for many trials. In Experiments 1 and 2, observers were not confused by common lures, in Experiment 3, observers were not confused by rare lures while search for novel targets. Again, this argues either that observers are sensitive to very subtle differences in familiarity or that each novel item is added to the “old” or “previous target” category after it plays its role as the novel target and that it is the membership in that particular category that is crucial for identifying an item as a distractor in this task.

General Discussion

Hybrid search is an interesting task, in part, because it captures some aspects of the everyday task of searching for more than one thing at a time. The results of the present experiments and of other hybrid search studies suggest that we are capable of holding a labeled list in some form of long-term memory. The storage is long-term because it is clearly not fading in the mere seconds that would be available to a short-term store. Consider that, in Experiment 3, a novel item might enter the memory set on one trial and not reappear as a distractor until hundreds of trials and many minutes had gone by. In the absence of rehearsal or any form of retraining, the memory for that item must be assumed to be long-term. In addition, short-term stores like visual short-term memory (VSTM) are small in size, holding the equivalent of about 4 items. However one wishes to conceive the limits on VSTM (Suchow, Fougnie, Brady, & Alvarez, 2014), no one argues that it can hold the hundreds of items that can constitute a hybrid search memory set.

In the version of the hybrid task used here, we argue that hybrid search is likely accomplished by determining if an item is in the “old” or “target” category. Membership in that category is not based on how often an item has been encountered and, thus, observers are not misled by lures that have been seen frequently but do not bear the category label. In Experiment 1, we manipulated the familiarity of the distractors, comparing a condition where the distractors were as familiar as the targets to a condition where all distractors were novel. In the former, observers could not depend on a familiarity signal, yet we did not see a drop in performance relative to the AllNovel distractors condition. In Experiment 2, we tried to lure observers into picking distractor items, some of which were much more common than the targets. Observers were not fooled. It seems very likely that they were aware they had seen these distractors in a higher proportion, but, while these distractors modestly slowed RTs they were not mistaken for targets. Finally, Experiment 3 shows a rare lure does not confuse observers as they search for a novel target. Apparently, a single exposure to an item as a target is enough to enter it into the “old” category and this encoding is sufficient to allow observers to discriminate a novel item from an item seen once, several hundred trials earlier.

This is not to say that familiarity has no role in hybrid search. Such a conclusion would be foolish given the large body of work on the role of familiarity in memory search in general (Nosofsky, Cao, et al., 2014a; Yonelinas, 2002). Mandler (1980) introduced the example of the butcher on the bus where you encounter someone (in this case, the butcher) under circumstances (the bus) under which you fail to recover the relevant label in memory. You are left with the strong feeling that this person looks familiar (Wixted & Mickes, 2010). It is entirely possible that in some trials of our hybrid experiments, the sense of familiarity drove the observers’ response.

In other closely related tasks, familiarity is important. This is clearly seen in the recent memory search work of Nosofsky et al. (2014a) which has many similarities to the present studies. They also conclude that there are conditions in which categorization of items as “old” or “new” allows observers to reject familiar lures. In their experiments, observers get a distinct set of target items on each trial. In their consistent mapping versions, the set on the present trial is a subset of a consistent set that can be held in LTM, but observers are focused on the subset that is valid for this trial. Under these circumstances, familiarity-based lag effects dominate the pattern of responses. In our experiments, we do not have the same effects of lag on a trial-by-trial basis. However, analogous lag effects modulate the pattern of response in Experiment 1 (see Fig 4). An item that has been seen more recently receives a substantial RT advantage. However, while familiarity contributes to RT patterns, we agree with Nosofsky et al. (2014a) that categorization prevents these familiarity effects from persuading observers that a recently seen lure is a target.

The differences between our hybrid tasks and those of Nosofsky et al. point to the range of hybrid-style tasks in the real world. For a home repair project, you may have a set of tools and supplies. At this moment, you need a screwdriver. On the next trial, you need a wrench, a bolt, and a nut. Then you need the screwdriver again. This is close to a Nosofsky-style consistent mapping hybrid search. At a used book sale, you might go from table to table looking for any works by your 8 favorite authors. This hybrid search, also consistent mapping, is in the style of Experiments 1 and 2 of this paper. The volume of T.S. Eliot may attract your attention but familiarity alone will not cause you to select it if you are really looking for the works of George Eliot. There are variants of real-world hybrid search that would be worth investigation. Going into a store with a shopping list in mind is a hybrid search, but one where the memory set size, in principle, decreases each time a target is found. Mental checklists for a radiologist examining an abdomen or a pilot examining the instruments in her cockpit have the quality that different parts of the list are relevant in different places and/or at different times. Each of these hybrid searches is a complex interaction between the contents of the mind and the contents of the world. To the extent that we want to understand and improve performance of pilots, radiologists, and even supermarket shoppers, we need to work out the details of that interaction. The work presented here illuminates one piece of the hybrid search puzzle. In tasks based on consistent use of the same memory set over many visual searches, items that happen to be as familiar or more familiar than the targets do not mislead observers.