Rare targets are often missed in visual search

Abstract

Our society relies on accurate performance of visual screening tasks (e.g. for knives in luggage or tumors in mammograms). These are visual searches for rare targets. We report that target rarity leads to disturbingly inaccurate performance.

Visual search is the subject of a voluminous laboratory literature ¹. Typically, observers perform several hundred searches and targets are presented on 50% of trials. Target prevalence in baggage screening or cancer screening is much lower (~0.3% in routine mammography ²). We compared performance on high and low prevalence versions of an artificial baggage-screening task. Observers looked for “tools” among objects drawn from other categories. Semi-transparent objects were presented on noisy backgrounds and could overlap (Fig 1). The number of objects in a display was 3, 6, 12, or 18. Target prevalence was 1%, 10% or 50%. At 1% prevalence, 12 paid volunteer observers had to be tested for 2000 trials each (broken into 250 trial blocks) to obtain a mere 20 target-present trials each, Each observer was tested for 200 trials in the 10% and 50% conditions. Observers were given feedback on their performance, including a point system designed to emphasize the importance of finding the target (see supplementary methods). Low prevalence search has some similarity to vigilance tasks where observers wait for fleeting signals ^3,4. However our search stimuli are continuously visible until observers choose to respond.

Figure 1.

Stimuli: Observers searched for tools in displays with semi-transparent objects placed randomly on a noisy background.

Figure 2a shows error rates as a function of number of objects. 50% prevalence produced 7% miss errors, typical for laboratory search tasks of this sort. However, errors increased dramatically (and reliably) as prevalence decreased. 10% prevalence produced 16% errors, while at 1% prevalence errors soared to 30%. Errors were primarily “misses” (failing to notice a target). “False alarms” (saying “yes” when targets are absent) were vanishingly rare (0.03%), despite incentives to produce the opposite behavior (see supplementary methods). Simply changing prevalence produced a fourfold increase in error rate. If similar effects occur in socially important searches, this could have significant consequences.

Figure 2.

Figure Two: the effects of target prevalence on search performance.

2a: Error data: When targets were rare (1% prevalence - black bars) observers made more than 4X times the errors made when targets were common (50% prevalence - white bars). Data are averages of 12 observers. Error bars show +/− 1 s.e. for those 12 error rates. Gray bars show 10% prevalence results.

2b: Left: Reaction Times for 50% prevalence. Typical reaction times are longer when the target is absent (open symbols) than when targets a present (closed) Miss error RTs are shown by diamonds.

2c: Right: Reaction Times for 1% prevalence. However, when prevalence is low, observers make “absent” responses that are faster than the “present”. This leads to elevated error rates. For 1c&d, error bars (s.e.) fall within data points.

Why does this happen? The reaction time (RT) data (Fig 2b&c) provide some clues. Observers require a threshold for quitting when no target has been found. This threshold is constantly adjusted; observers slow down after mistakes and speed up after successes⁵. When targets are frequent, fast “no” responses will often lead to mistakes. As a result, “no” RTs are slower than “yes” RTs in high prevalence search (2b). With infrequent targets, observers can successfully say “no” almost all of the time, driving down the quitting threshold. As seen in Fig 2c, the result is too rapid target-absent search. Observers abandon search in less than the average time required to find a target.

The problem cannot be solved simply by adding pseudotargets to increase prevalence (e.g. search baggage for iPods and weapons) In a second experiment, we mixed common (44% prevalence), rarer (10%), and very rare (1%) targets such that some target was present on 50% of trials. Here, observers missed just 11% of common targets but 25% of rarer targets and 52% of very rare targets (see supplementary methods & results). Is the prevalence effect just a by-product of naive Os unfamiliarity with the targets? In a separate investigation, we compared the miss error rate for 4000 trials at 1% prevalence (40 targets, 41% miss errors) to the miss error rate for the first 100 trials at 34% prevalence (34 targets, 11% miss errors). It appears to be prevalence, not just number of targets presented that is critical (See Supplementary Methods).

Visual search is a ubiquitous human signal detection task ⁶. Heuristics that produce acceptable performance over a wide range of target prevalence may betray us at low prevalence. Because the experiments are burdensome, we do not have a clear idea whether these effects occur in the field ^7,8. A scoring system in the lab cannot duplicate the motivation to find a gun or a tumor nor the motivation to move the check-in line along. The training of lab volunteers differs from that of professionals. Nevertheless, there are sufficient similarities between lab and field to strongly suggest that we should find out if the massive increases in error shown here occur in socially important search tasks.