Visual Search: How do we find what we are looking for?

Abstract

In visual search tasks, observers look for targets among distractors. In the lab, this often takes the form of multiple searches for a simple shape that may or may not be present among other items, scattered at random on a computer screen (e.g. Find a red T among other letters that can be black or red.). In the real world, observers may search for multiple classes of target in complex scenes that occur only once (e.g. As I emerge from the subway, can I find lunch, my friend, and a street sign in the scene before me?). This chapter reviews work on how search is guided intelligently. We ask how serial and parallel processes collaborate in visual search. We describe the distinction between search templates in working memory and target templates in long term memory and we consider how searches are terminated.

1. INTRODUCTION

The crowd at Reykjavik airport was impressive and somewhere in the view in front of me might be my wife. Determining whether or not she was in my current field of view was a visual search task. If we had an “all-seeing eye”, search would be unnecessary. The image contained the information and either my wife was present or she was not. However, while we can see something at all locations in the visual field, it should be obvious that we cannot immediately identify everything in a scene. Our capacity is limited in a manner that makes search necessary. Some of those limits have to do with the structure of the visual system. For instance, the retina and visual nervous system are structured so that we have high-resolution vision only in the small foveal region at the point of fixation. Beyond those basic visual limitations, there are capacity limits that are computational. Even a limited version of an all-seeing eye that could process everything, everywhere in the visual field, would require an implausibly large brain (Tsotsos, 2011; improving on Neisser, 1967).

A substantial body of current research strives to understand how human searchers cope with these limits on the ability to perceive the world and to find what we are looking for. Classic laboratory tasks have asked observers to search for a target in an array of distractors, scattered at random on an otherwise blank computer screen. Those tasks have taught us a great deal, though one theme in this chapter will be the need to move to tasks that are closer to those in real-world search.

This chapter will concentrate on four topics:

1. How is visual search guided?

2. How do serial and parallel processes collaborate in visual search?

3. Search Templates and Working Memory

4. Search termination and tasks beyond laboratory search

2. HOW IS VISUAL SEARCH GUIDED?

2.1. Classic visual search

Classic search for a target among some number of distractors has been popular since the 1970s (Egeth, 1977; Wolfe, 1998a). These tasks eliminate the complications produced by real scenes. Much of classic visual search research has been done with stimuli that are deliberately made large enough and spaced widely enough that acuity and crowding limits do not constrain the task. However, these tasks can still show clear evidence for capacity limits and the need to search. An example, taken from Wolfe and Myers (2010), is shown in Figure 1a,b. Observers (Os) are looking for plastic among stone surfaces. Display “set sizes” of only 1, 2, 3, or 4 large patches were presented to Os, each in a different quadrant of the visual field. These are easily recognized. Material properties turn out to be identifiable in a short fraction of a second (Sharan et al., 2014). Nevertheless, the response time (or “reaction time” – RT) increases roughly linearly with the number of patches shown. Moreover, the slope of the RT x set size function is about twice as great when the target is absent than when it is present. If capacity was not limited, then the time required to make a response should be nearly independent of the number of items in the search array. That is what happens for simple “feature searches” as is shown in Figure 1c, d (Wolfe, Palmer, & Horowitz, 2010b). Note that the set sizes are much larger here. Nevertheless, the ability to find a red target among green distractors does not depend on the number of green items (but see Lleras et al., 2020).

Figure 1:

Classic visual search experiments.

The slope of the RT x set size function is a measure of the rate of processing of items. Consider the search shown in 1a, b in which RT increases by 35 msec for each additional item on target-present trials and by 81 msec/item on target-absent trials. One way to think about such searches is to conceive of them as “serial, self-terminating searches” (Sternberg 1966). Each item is examined in turn until the target is found or the search is abandoned. On average, such a search will end after (N+1)/2 items have been examined on target-present trials. If the target-absent trials were truly exhaustive, all N items would need to be examined exactly once (Search termination is more complex than that and is discussed later; Chun & Wolfe, 1996; Cousineau & Shiffrin, 2004; Moran, Zehetleitner, Mueller, & Usher, 2013; Wolfe, 2012). In this view, the target-absent slopes will be about twice the target-present slopes. The true cost per item would be twice the target-present slope. For the data shown here, a simple serial model would argue that Os were processing items at a rate of 1/(0.035*2)) = ~14 items per second. The same analysis would propose that all of the colored squares were being processed simultaneously (given slopes near 0 msec/item). Thus, that processing could be said to occur “in parallel”. Note that slopes would be much steeper if it were necessary to fixate on each item in turn. Voluntary saccadic eye movements occur at a rate of about 3–4 per second, translating to RT x set size slopes of 250–333 msec/item for target-absent trials and about half that for target-present.

In most real-world search tasks, initial parallel processing of the search display can be used to guide subsequent deployment of attention. For example, in Figure 1e, if you search for the letter T, you will need to attend to each item until you stumble upon the target. However, if you know that the T is green, it should be intuitively clear that you will only (or, at least, preferentially) attend to the green items (Egeth et al., 1984). Your search for a letter will be ‘guided’ by the orthogonal information about its color. As a consequence, though there are 21 items in the display, the “effective set size” (Neider & Zelinsky, 2008) will be 7, the number of green items. The slope of the RT x set size function for a search for the green T will be ~1/3^rd of the slope of the function when color is unknown (Figure 1f). The idea of guidance was present in the attention literature from at least the 1950s, (Egeth et al., 1984; Green & Anderson, 1956; Hoffman, 1979). It was at the heart of the “Guided Search” model of Wolfe et al. (1989) that argued that differences in the efficiency of search tasks could be explained by differences in the amount of guidance; more guidance would lead to more efficient search. Guided Search (GS) has been through multiple versions (Wolfe, 1994; Wolfe, 2007; Wolfe et al., 2015; Wolfe et al., 1989; Wolfe & Gancarz, 1996) with Guided Search 2.0 (GS2) version being the best known. The idea of guidance is now generally accepted and is a part of virtually all models of search. Thus, taking the example of Figure 1e, it is hard to imagine a model that would not devote more processing resources to green items if the target was known to be green.

2.2. Guidance and the nature of guiding features

So, what guides attention in visual search? Our understanding of how attention is guided has evolved. Classic GS and other models concerned themselves with guidance by basic features like color and motion. Attention could be guided to features of the target and, to a lesser extent, away from features of the distractors (Gaspelin & Vecera, 2019). Other forms of guidance will be discussed below. First, we briefly review feature guidance.

A limited set of features are available to guide attention. These include uncontroversial properties like color, size, and motion as well as less obvious properties like lighting direction (Enns & Rensink, 1990; Adams, 2008) or axis of rotation (Schill et al., 2019). The list of the one to two dozen guiding features is reviewed elsewhere (Wolfe, 2014; Wolfe & Horowitz, 2004; Wolfe & Horowitz, 2017). Here we will focus on what we have learned about feature guidance in recent years. Feature guidance is derived from the basic sensory input that gives rise to the perception of attributes like color or orientation. However, guidance appears to be a specific abstraction from that input (Below, we will argue that this abstraction is the “search template” Olivers et al., 2011). Thus, for example, small differences in orientation (e.g. 0 from 5 deg) may be easily discriminated if items are scrutinized, but a 0-degree target will not ‘pop-out’ in a display of 5 deg distractors. The orientation difference would need to be larger before it would summon attention (Foster & Ward, 1991). Moreover, guiding signals appear to be categorical. For example, it is easier to search for oriented lines if they are uniquely “steep” or “shallow” (Wolfe et al., 1992). More recently, other aspects of feature guidance have been described. In a series of experiments, Stefanie Becker demonstrated that guidance is often usefully described as a relationship between targets and distractors, rather than as an absolute target value (Becker et al., 2019, Becker, 2010). Thus, observers might be more likely to define a target as the reddest item in a display, rather than a specific red color. The orange target that could be found as the reddest item in one context, might be searched for as the yellowest item in another context. Another way to think about this is to imagine a boundary in feature space that puts targets on one side and distractors on the other. When targets and distractors are “linearly separable” in this way, search is efficient. When targets are flanked in the feature space by distractors (e.g. 0 deg among +/− 20 deg distractors) search is inefficient (Bauer, Jolicœur, & Cowan, 1996; Wolfe et al., 1992). When a target is close to distractors in feature space (e.g. a bluish green among greenish blue distractors), attention will be guided by an internal representation or “target template” that is not centered on the actual target feature. Instead, the template will be centered a bit further from the distractors to avoid mistakenly categorizing a distractor to be a target (Yu & Geng, 2019). In signal detection terms, this is roughly like moving the decision criterion to a more conservative position in order to avoid false alarm (false positive) errors. The precise nature of the feature guidance is probably hidden from the searcher. They look for ripe blueberries without necessarily realizing that this involves setting criterion in color space for items that will attract attention and then using another criterion to decide about the precise status of an attended berry.

Guidance by shape is, perhaps, the greatest puzzle in feature guidance. There are several “basic features” that are related to the shape of an object. These include closure, line termination, curvature, topology (holes), and aspect ratio (reviewed in Wolfe, 2018). Unlike features like color & motion, it is not clear if it makes sense to talk about these aspects of shape as independent features. A new way forward might arise out of the remarkable progress in training deep learning algorithms to identify objects (e.g. Kriegeskorte, 2015). At a high level in a deep learning network, an object or category (e.g. cat) might be defined by a high dimensional vector. While that vector can be used to identify a cat, it probably does not guide attention toward candidate cats (but see Miconi et al., 2016)). It could be that some other, coarser representation, derived from the drawn from the same network might be the basis for guidance by shape (analogous to coarse guidance by orientation, described above), but this hypothesis awaits experimental test.

2.3. Scene Guidance

Classic models of search like Feature Integration Theory and Guided Search focused on the features of the target. Work over the last 25 years has made it clear that feature guidance is only part of the story. Indeed, for many tasks, it may not be the most important force that constrains the deployment of attention. That role would go to “scene guidance”. Classic feature guidance is guidance to locations/objects that appear to have target features. Scene guidance is guidance to locations that are more likely to contain a target, regardless of whether that target and/or its features are present. That is, search for a pillow will be directed to the top end of the bed, whether or not there is any preattentive feature evidence for the presence of a pillow

Scene guidance is not a new idea (Kingsley, 1932; Biederman et al., 1973) but it has taken on greater prominence; in part, because it has become practical to use scenes as stimuli, rather than only random arrays of items.

Working out how scene content and structure guide attention is a project in its fairly early stages. Analogous to an Itti et al. (1998) salience map, Henderson et al. (2017; 2018) have developed “meaning maps” that show how the meaningfulness of a scene varies over the scene. Vo and her colleagues argue that scenes, like language, have a grammar (Henderson & Ferreira, 2004; Vo & Wolfe, 2013). That grammar is concerned with scene semantics - Is an object meaningful in this location in this scene? – and scene syntax – Where is it structurally plausible for an object to be? For instance, toasters do not float (Vo & Henderson, 2009). In a toaster search, attention would be guided toward places where toasters are physically plausible (e.g. horizontal surfaces) and away from places where toasters, while possible, are semantically improbable (e.g. the floor, under the table). Given the relative inefficiency of search for arbitrary, realistic objects in arrays (Vickery et al., 2005) and scenes (Wolfe et al., 2011a), it seems likely that scene guidance is doing the largest portion of attentional guidance in many real-world searches.

Scene guidance and feature guidance work together. In Figure 3 if you look for people, you will quickly find the man in the middle of the image. You may be slower to see the exact copy of his image, pasted on the path at the bottom. You guided your attention to objects/features that could plausibly be human and the man at the bottom is the wrong size. He is only the wrong size, however, because the structure of the scene makes it so (Eckstein et al., 2017, Wolfe, 2017). Guidance mechanisms also know something about the structure of objects. For example, it is easier to find a red object with a yellow part than a red and yellow object, even if both items have the same number of red and yellow pixels (Wolfe, Friedman-Hill, & Bilsky, 1994; Vul et al., 2019).

Figure 3:

The interaction of scene and feature – Look for people.

In any scene search lasting for more than a fraction of a second, it seems evident that the nature of scene guidance will evolve during search. When a scene is first encountered, scene guidance (like feature guidance) appears coarse. Work on search for humans shows rapid, broad guidance toward the ground plane, for example (Ehinger et al., 2009). This was probably evident in Figure 3. Castelhano finds that a very brief preview can bias subsequent search to relatively coarse regions of a scene (Castelhano & Heaven, 2011; Castelhano & Henderson, 2007; Pereira & Castelhano, 2014). Moreover, this scene guidance only pertains to objects in scenes, not to objects pasted on top of scenes (Pereira & Castelhano, 2019). This coarse initial stage of guidance is consistent with evidence that extremely brief exposure to a scene is enough to extract its ‘gist’, giving access to its rough structure and content (Greene & Oliva, 2009; Oliva, 2005; Oliva & Torralba, 2001). As time progresses, more detailed information about a scene and its contents will develop and search can then be guided by that information. Thus, in search for the pillow in Figure 2, attention is guided by the location of the bed (an ‘anchor object’, Boettcher et al., 2018). That guidance presumably requires the identification of the bed which, in turn, probably required deployment of attention to that bed. Even feature guidance takes some time to fully develop (Palmer, Van Wert, Horowitz, & Wolfe, 2019; Lleras et al., 2020), though it remains reasonable to think of feature guidance as being based on the output of a fast, preattentive process. Scene guidance, by contrast, seems to be more of a process that evolves, from initial information about the “spatial envelope” of the scene (Oliva & Torralba, 2001) to guidance based on richer scene understanding and informed by attention to objects in the scene.

Figure 2:

Scene Guidance: Where observers look when told to look for a pillow. (figure courtesy of Sage Boettcher and Melissa Vo)

Another important difference between scene guidance and feature guidance is that much if not all scene guidance must be learned. The genes may code for a set of basic guiding features like color and size, but they are not coding for the pillow/bed relationship. “Contextual cueing” (Chun & Jiang, 1998) can be thought of as a very basic version of scene guidance in which repeated exposure to the same random arrays of stimuli leads to learning of the association of a random array with a target location. This type of guidance appears to be largely implicit, even using real world stimuli (Jiang et al., 2014). Experts in specialized search tasks like those in radiology speak of a “gestalt” stage in processing when an initial assessment of the image guides attention to likely locations of a clinical finding (Nodine & Mello-Thoms, 2019). This form of guidance is clearly learned. Moreover, it is reasonable to consider this a form of scene search since radiologists certainly make use of the learned structure of the image as a whole and do not rely exclusively on their identification of specific features or objects.

2.4. Guidance by search history

In a standard search experiment and, one may suppose, in a string of searches in the world, performance on one search is influenced by performance on previous searches. The “priming of pop-out” phenomenon, described by Maljkovic and Nakayama (1994) is an important early example. Os saw displays of red among green or green among red items. They needed to report the precise shape of the color singleton in the display. Maljkovic and Nakayama found that, even though the search for the color target is trivially easy, Os were faster if the preceding target had been the same color as the current target. The effect was quite automatic. For instance, it did not matter if you knew the next color. This priming was based on the color of the previous target. The result has been replicated and extended to multiple dimensions (Hillstrom, 2000; Kristjánsson, 2006). Priming is not a simple effect. For instance, priming effects for conjunction stimuli can be longer lasting than single feature, priming of pop-out (Kruijne & Meeter, 2015). It may be tied to episodic memory representations (Huang, Holcombe, & Pashler, 2004; see also Brascamp et al., 2011). Distractor properties can also be primed (Lamy et al., 2013) though the effects are smaller than those for targets (Wolfe et al., 2003).

For present purposes, the important point is that finding something on one trial appears to guide attention toward subsequent instances of that target. Originally, Wolfe et al. (2003) argued that priming was a form of top-down guidance. The logic was that there were two forms of guidance. Bottom-up came from the stimulus and top-down came from the observer. Since priming was a form of memory, it was internal to the observer and, hence, top-down. More recently, Awh and colleagues (2012) have argued that this confuses two forms of guidance that should be considered to be distinct. There is top-down, volitional guidance to what you are looking for. This is distinct from the automatic effects of history that occur whether you are looking or not. Theeuwes (2018) argues that much of what passes for top-down guidance is driven by priming by targets. Even if one might not go that far (Ásgeirsson & Kristjánsson, 2019), it seems correct that the recent history of search guides subsequent search.

2.5. Guidance by value

A fifth form of guidance is related to, but probably distinct from guidance by history. Attention is guided to features that have been associated with reward (Anderson et al., 2011). Like priming, effects of value can be quite automatic and can act against the interests of the observer (Hickey et al., 2010). Effects of value can be seen in monkeys (Ghazizadeh et al., 2016). The effects can be distinguished from the effects of just seeing and responding to a feature repeatedly, as in priming studies (Kim & Anderson, 2018) so it probably makes sense to see value as a separate modulator of attentional priority. The role of reward and value has been extensively reviewed by Failing and Theeuwes (2017).

2.6. Building a priority map

To briefly summarize, attention appears to be guided by, at least, five factors: Bottom-up salience, top-down feature guidance, scene guidance, history, and value (Wolfe & Horowitz, 2017). While all those forces are attempting to guide attention at any moment, selective attention generally has a single focus (for the debate on splitting attention see Cave et al., 2010; Jans et al., 2010). Accordingly, the multiple sources of guidance need to be combined with some final common path. This is often imagined as a “priority map” consisting of some weighted average of all forms of guidance (Fecteau & Munoz, 2006; Serences & Yantis, 2006). The term “salience” or “saliency” map should be reserved for bottom-up, stimulus-driven guidance (Koch & Ullman, 1985; Li, 2002) though this terminological distinction is not always clear.

The basic ideas behind the operation of a priority map were established in the early discussions of saliency maps (Itti & Koch, 2001). Information is pooled and, when it is time to deploy attention, a winner-take-all process directs attention to the point of highest activation in the map. This is not the only possibility. For instance, one could propose that attention is directed by one feature map without pooling information across maps. Chan and Hayward (2009) argue that it is possible to respond based on activity in an orientation map or a color map without the need to combine those signals into a priority map. Their “dual-route” account (Chan & Hayward, 2014) acknowledges the combination of signals but does not require that combination. Buetti et al. (2019) provides evidence that the combination of signals does occur. They performed simple “pop-out” searches for color and shape targets. When the target was defined by both color and shape, RTs were faster than responses to either feature alone. This indicates that combining features makes a bigger signal, driving faster responding.

The idea of a weighted average is particularly clearly articulated in the Dimensional Weighting Theory of Mueller and his group (Liesefeld & Müller, 2019). As the name suggests, it puts particular emphasis on modulating the contribution of different dimensions like color, orientation, and so forth, though, within a dimension like one can guide toward a specific feature like red. There appears to be a weaker but real ability to unweight or inhibit distractor features (Cunningham & Egeth, 2016; Lamy et al., 2008; Neumann & DeSchepper, 1991). It is likely that most of the weight-setting is implicit, based on priming/selection history effects. For a particularly fervid argument on this point, see Theeuwes (2013). This may be more relevant in a long series of trials in the lab than it is during a one-off search in the world. There needs to be some way to guide your search for bananas at the store without having to be primed by immediately preceding banana searches.

Attentional “capture” (Yantis, 1993), for example by abrupt onset stimuli (Jonides & Yantis, 1988), refers to situations where attention is summoned to a stimulus or location in spite of and, sometimes, in opposition to the observer’s intentions. The phenomenon suggests that activity in the priority map is unlabeled, in the sense that attention can be “captured” by the wrong signal if that signal is incorporated into the priority map. For instance, in “singleton mode”, attention gets directed to any salient item, regardless of what makes it salient (Bacon & Egeth, 1994) though this capture of attention by any oddball depends on the nature of the search task (Jung et al., 2019). Similarly, while the weighting system is flexible, it is probably impossible to set the bottom-up, stimulus-driven salience weight to zero. The classic statement of this principle comes from Sully (1892) who said, “One would like to know the fortunate (or unfortunate) man who could receive a box on the ear and not attend to it” (p146), meaning that any irrelevant stimulus, if strong enough, will attract attention. Many of the more modern debates over the conditions of attentional capture can be seen as arguments over the strength of the blow to the metaphorical ear.

3. HOW DO SERIAL AND PARALLEL PROCESSES COLLABORATE IN VISUAL SEARCH?

3.1. Overt and covert attention: Eye movements, eccentricity, and the functional visual field

Attention and eye movements are closely related. However, they can be readily dissociated. It is possible to look at one location while paying attention to another (Helmholtz, 1924). It is often useful to refer to “overt” deployments of the eyes and “covert” deployments of attention. (Posner, 1980). Under most circumstances, covert attention is deployed to the target of the next saccade before the saccade arrives at that location (Kowler et al., 1995). Perhaps the strongest effort to link attention to movements of the eyes is Rizzolatti’s “Premotor Theory” which holds that deployment of attention is just a weaker manifestation of the signals that direct action toward a location (Rizzolatti et al., 1987). Premotor Theory is probably too strong (Belopolsky & Theeuwes, 2012) but there is no doubt that eye movements and attention are closely linked in visual search (Hoffman, 1996) even if search can be accomplished similarly with and without eye movements as long as the items are big enough to avoid eccentricity and crowding effects (Klein & Farrell, 1989; Zelinsky & Sheinberg, 1997).

Under most real-world conditions, the effects of eccentricity and crowding cannot be avoided (Rosenholtz et al., 2012; Rosenholtz, 2011) and, as a result, overt eye movements and covert deployments of attention should be seen as participating in a complex, interactive dance during search. The precise details are unclear because we cannot track covert attention as effectively as we can track the eyes. Under normal conditions, observers will fixate on 3–4 items every second. However, in a search like the search for a T among L’s in Figure 2e-f, the RT x set size functions indicate that those items are being processed at a rate of 20–40 items/second (details of mapping from slopes to rate depend on assumptions about memory in search, see below).

These numbers show that observers are processing ~5–10 items per fixation. How one understands the difference between the rate of eye movements and the rate of item processing in search is at the heart of most theoretical arguments in visual search today. At one extreme is the view that the eyes move to a new fixation every few hundred msec and that covert attention is then serially deployed to 5–10 items before the eyes move elsewhere. This was the original Guided Search position. Alternatively, attention might move to a location, permitting parallel processing of all items within a function visual field (FVF) or useful field of view (UFOV) surrounding that point of fixation (There are many versions of this view, but they are well captured by Hulleman & Olivers, 2017). In the strongest version of this FVF view, the difference in efficiency of, say, the search for a T and the search for a green T in Figure 2e, f is not a difference in guidance but a difference in the size of the FVF.

These models sound dramatically different, but they actually may be two ways of looking at the same dance of overt eye movements and covert attention. First, once we leave the realm of large, widely spaced items on a computer screen, any GS-style model needs to incorporate the idea of an FVF. This is illustrated in Figure 4.

Figure 4:

The dance of overt eye movements and covert attention (see text for details).

Upon fixation at some point (#1 in the figure), covert deployments of attention are going to be directed to recognizable items, given the current fixation. Those items will fall within some FVF (2). Outside of the FVF acuity and/or crowding will render the target unrecognizable. Each covert deployment (3), will start a decision process (Is this a “T” or and “L”?) that can be modeled as diffusion of information toward a decision bound (4 - Ratcliff, 1978; Ratcliff & McKoon, 2008). The covert deployments will occur every 50 msec or so. However, even the most dramatically fast object recognition processes take longer than that (Thorpe et al., 1996, VanRullen & Thorpe, 2001). It follows that individual diffusion processes would overlap in time, making the recognition aspect of search (4) into an asynchronous diffusion process. After 200–300 msec, the eyes are redeployed (5) and covert selection continues within a new FVF. In this case, the target is identified (6) and search ends.

Wolfe et al. (2003) uses the analogy of a carwash to describe this asynchronous diffusion process. Cars go in one at a time, in series, but they are being processed in parallel in the sense that several cars are being washed at the same time. An experiment that looked at the contents of the diffuser at one instant in time would see the parallel processing of items within an FVF. The only real difference between this serial, covert attention account and a parallel FVF account is whether items within the FVF begin their diffusions synchronously or asynchronously. Given the lamentable lack of a covert attention tracker, mentioned above, and a welter of free parameters (e.g. diffusion rates for items as a function of eccentricity), these models are probably empirically indistinguishable.

3.2. Varieties of models of visual search

Even if it is hard to distinguish between different models of search, this does not mean that there is just one model of search operating under a range of names. Numerous approaches put different emphasis on factors like the role of eye movements or peripheral visual processing. Each model raises issues that can inform the development of the next generation of models.

Anne Treisman’s seminal Feature Integration Model (FIT; Treisman & Gelade, 1980) was built on the two stage, preattentive-attentive architecture of Neisser (1967). Responding to the neurophysiology of the day, Treisman envisioned the preattentive processing of simple features in parallel across the visual field. “Binding” those features into a recognizable object required serial deployment of attention (Treisman, 1996; von der Malsburg, 1981). Kristjánsson and Egeth have recently reviewed the work that provided the grounding for FIT. GS, in its original formulation, simply added the idea that basic, parallel feature information could be used to help deploy serial attention in an intelligent manner (Fig 2e, f). Other models can also be seen as variants of FIT. The Dimensional Weighting Model of Hermann Müllerand colleagues is quite like GS in giving an important role to feature guidance (Liesefeld & Müller, 2019; Müller et al., 1995; Rangelov et al., 2011). Different features might have different potential to guide attention in a given task. In a search for a red, vertical target it would not make much sense to guide attention based on irrelevant motion or size information. The Dimensional Weighting Model emphasizes the need to weight the output of feature processors to get an intelligent guiding signal. Moreover, the model emphasizes the differences between switching from one feature to another within a dimension (red to green) compared to between dimensions (red/color to big/size). The same group has also developed the “Competitive Guided Search” (Moran et al., 2015; Moran et al., 2013) which improves upon the mechanics of earlier GS models, especially by improving on the treatment of quitting rules; when do you stop searching (discussed below)? Several other more quantitative versions of GS-style models have been developed (Hubner, 2001) with the Integrative Model of Visual Search of Schwarz and Miller (2016) being a recent, notable example.

Rensink’s (2000) “triadic” architecture ties FIT-style search architecture more closely to conscious visual experience, adding “What do we see?” to the problem of “How do we find what we are looking for?” His parallel front-end stage carves the scene into “proto-objects” that can be selected by his version of a second, limited capacity stage. Importantly, he adds a separate “non-selective” pathway, not subject to attentional limits, that produces some visual experience at all locations in the visual field. These ideas about non-selective ‘gist’ processing are important in subsequent models like more recent versions of GS (Wolfe et al., 2011b).

Computational “salience” models, with their roots in the path-breaking work of Koch and Ullman (1985) and Itti and Koch (2000) can be quite similar in spirit to GS, albeit with stronger ties to the underlying neurophysiology. A set of early-vision features, often modeled on neurophysiological data, are used to derive a saliency map that, in turn, directs attention and/or the eyes. Traditionally, such models have typically been more concerned with the bottom-up, stimulus-driven aspects of attention than with top-down, user-driven aspects. That is, they were modeling what aspects of the world captured attention with less emphasis on how observers got their attention to targets of interest to them. Computationally modelers weren’t denying the existence of top-down factors. Initially, their models were just solving other problems. John Tsotsos (2011, 1995) was relatively early in incorporating top-down processing into his computational models and such factors are now part of standard models in the computational saliency map tradition (Hamker, 2004; Navalpakkam & Itti, 2006). Moreover, researchers like Tsotsos moved away from the FIT-style model to models much more concerned with how attention could modulate a network of neural connections in order to have the right answer emerge at the top of a pyramid of processing (Tsotsos, 2011). For more on this topic, see Frintrop’s (2010) excellent review and see the 2015 special issue of Vision Research (Tsotsos et al., 2015).

Some models, like Zelinsky’s (2008) Target Acquisition Model (TAM), place much more emphasis on modeling the overt deployments of the eyes than the more elusive covert deployments of attention. The data that lead to the conclusion that multiple items can be processed with each fixation leads to models that propose that search progresses by the sequential analysis of clumps of items within an FVF, centered around the current point of fixation. Often, the assumption is that this analysis is parallel over the FVF. A notable example of such a model is that of Hulleman and Olivers (2017). They proclaim, “The impending demise of the item in visual search” because they are replacing search by item with search by FVF clump. More recently, Liesefeld & Müller (2019) have argued that both processes – item by item and FVF clump processing - can be used to find targets in search.

The nature of the FVF that can be processed with each fixation takes a central role in models of search like those of Ruth Rosenholtz and her colleagues (Rosenholtz et al., 2012). Rosenholtz argues that the inefficiency of searches like classic searches for a T among Ls are driven less by attentional capacity limits and more by limits imposed by crowding and image degradation in peripheral vision. These visual factors mean that a “T” that is present in the image, may not be present in the representation in the early visual system or may be distorted to resemble a distracting “L”. Other aspects of the statistics of the scene can remain intact and can support good scene perception even if search for specific items in the scene is inefficient (Zhang et al., 2015).

From models that propose parallel processing of clumps of items it is a conceptually short step to fully parallel models of search (Palmer, 1995; Palmer & McLean, 1995; Palmer et al., 2000; Verghese, 2001). In these models, all items are processed by either a limited or unlimited-capacity parallel processor and much of the empirical work is devoted to determining the rule that allows the observer to determine if there is a target-present and where it might be (e.g. Baldassi & Verghese, 2002). In practice, the fully parallel models have usually been applied to simple displays of a few items. If such a model were to be scaled up to deal with complex scenes, it seems likely that it would become a model of the FVF, clump-processing variety because no one would propose that a fully parallel model could work over the whole visual field without eye movements. For example, a model like that of Najemnik & Geisler (2009) might combine parallel processing with the constraints imposed by eccentricity.

This section has merely outlined the classes of attention models and noted the theoretical considerations they raise. For a more extended discussion, the reader might consult the relevant chapters in the Oxford Handbook of Attention (Nobre & Kastner, 2014).

4. SEARCH TEMPLATES AND WORKING MEMORY

A visual search presupposes that something is being searched for. The internal representation of the goal of search is often known as the “search template” (Gunseli et al., 2014) though everyone understands that this is a metaphor and that a search template does not look like the target in any literal sense. Several terms get used interchangeably, often in the same paper. E.g. Search template (Rajsic et al., 2017), target template (Bravo & Farid, 2016), memory template (Kristjánsson et al., 2018) and attentional template (Yu & Geng, 2019). The jargon confuses two types of template that should be kept distinct. First, “visual search is thought to be guided by an active visual working memory (VWM) representation of the task-relevant features, referred to as the search template” (van Loon et al., 2017). Second, “during visual search, observers hold in mind a search template, which they match against the stimulus” (Bravo & Farid, 2014). These are not the same role and, in all probability, not the same internal representation. The first template is involved in guiding attention to likely targets and is only fairly coarsely mapped to the target (Anderson, 2014; Kerzel, 2019). Once the attention is directed to an item, it can be precisely matched to a template (let’s call that one the “target template”). A comparison to the target template, not guidance by the features of the guiding template, makes it possible to know that this gray tabby cat is similar to but not the same as your gray tabby cat.

The guiding and target templates are not the same template. The guiding template is thought to reside in working memory (“WM” - Olivers et al., 2011). Many experiments show that changing the contents of WM can bias the guidance of attention (Soto, Heinke, Humphreys, & Blanco, 2005; Beck, Hollingworth, & Luck, 2011; Dowd & Mitroff 2012). The idea that the templates are maintained in WM goes back at least to neurophysiological work in the 1990s (Desimone & Duncan, 1995). However, while it seems clear that working memory serves an important role in search, it is reasonably clear that the target template resides elsewhere. The clearest evidence on this point comes from work on “hybrid search” (Wolfe, 2012). In hybrid search, Os look for any of several targets. In experiments using photographs of objects as targets, Os have no difficulty searching for as many as 100 distinct targets at the same time (or even more, Cunningham & Wolfe, 2014). Since the capacity of WM is around 4 items (Cowan, 2001), no model of WM would propose that 100 target templates could be in residence in WM at the same time. It could be that target templates shuttle in and out of WM. If that were the case, we might expect to see qualitatively different performance for hybrid searches with fewer than 4 items in the memory set and those with the larger memory sets that would require this shuttling. Moreover, we might expect response time to be a linear function of the memory set size. Instead, response time in hybrid search seems to be a smooth, logarithmic function of the memory set size (Wolfe, 2012). Moreover, Drew et al. (2015) found that loading WM did not impact hybrid search performance, though a hybrid search did seem to reduce WM capacity by about one item.

It seems more plausible to propose that target templates reside in long term memory (LTM); specifically, in “activated long term memory” (ALTM: Cowan, 1995), a partition of LTM that is relevant to the current task. After all, the typical block of search trials and real-world search operate over longer times than those proposed for WM representations, so it is reasonable to assume that some representation of the goal of the search resides in some aspect of LTM. The guiding templates in WM are “representations of the features that distinguish targets from nontarget objects” (Grubert & Eimer, 2018). It might be better to call this a “guiding representation” (p100 of Wolfe, 2007) but we are probably stuck with “template” so perhaps the field can be persuaded to use “guiding template” or “search template” for a coarse WM representation that guides search and “target template” for the LTM representation of the precise goal of the search. The interaction of these types of templates can be seen in categorical hybrid search. Cunningham and Wolfe (2014) had Os memorize several animal images. They then had Os search for those targets amongst distractors that could be other animals or letters. By manipulating the numbers of animals and letters in the visual display, it could be shown that the Os guided attention to the animals – using some set of animal features in a WM guiding template and then matched those animals against the specific “target templates” in LTM.

5. SEARCH TERMINATION AND TASKS BEYOND LABORATORY SEARCH

One of the more vexing problems in search is the question of search termination. When is it time to quit the current search task? The original, feature integration account, assumed that parallel “pop-out” tasks could be thought of as a single, 2AFC decision. Either the target was present or it wasn’t. For search tasks that FIT considered to be serial and self-terminating, search ended when the target was found or when all distractors were examined and rejected. Thus, absent trials were imagined to be serial exhaustive searches (or almost exhaustive, given that some miss errors would be made when Os quit before finding the target). GS complicated this picture by having the parallel guidance mechanism eliminate some distractors without serial examination, so GS2 proposed that search ended when the target was found or when all distractors having activation/priority above some threshold had been examined. That “activation threshold” was set adaptively. It became higher when an observer missed a target and lower after a correct response (Chun & Wolfe, 1996). GS2 also proposed that “some very long searches are terminated when the ...observer... concludes that it rarely takes this long to find a target.” (p210 of Wolfe, 1994)

A key assumption of this class of model is that rejected distractors or regions are marked in some fashion so that they are not attended again. Inhibition of return (IOR) has been a popular proposed mechanism for this marking (Klein, 1988) even though IOR is merely a bias against returning to an attended location, not an absolute prohibition (Klein & MacInnes, 1999). However, Horowitz and Wolfe (1998, 2001, 2003) challenged this idea in a series of experiments from which they concluded that “visual search has no memory” (Horowitz & Wolfe, 1998) – meaning, no memory for rejected distractors. In one set of experiments, they used a dynamic search display where they randomly replotted all the items every 110 msec. They found that the slope of the RT x set size function was the same in this dynamic search than it was in an equivalent static search (Horowitz & Wolfe, 1998). Perfect memory for rejected distractors would have predicted twice the slope in the dynamic case where no memory can be used. Subsequent research has pointed out that dynamic search conditions are not identical to static search conditions. Os could adopt different strategies in the two situations. If items are truly reappearing in random locations, Os could just sit at one location and wait for the target to appear (von Muhlenen et al., 2003). Though Horowitz and Wolfe (1998) tried to thwart a sit-and-wait strategy, Os might still be moving attention around the display less widely in dynamic search. Shi et al. (2019) replicated the dynamic search results but focused on the relationship between target-present and absent slopes. In static search, the absent: present ratio is typically near (or a bit more than) 2:1 (see Fig 2 of Wolfe, 1998b). Shi et al. found that this ratio was consistently near 1:1 for dynamic search, arguing for different strategies in the two conditions. It may well be that observers are answering different questions in the two cases. In a static search, the question might be, “Shall I quit, because I have looked at more or less everything that needs to be looked at?”. In dynamic search, that question does not make sense and the right question might be, “Have I searched long enough that it is unlikely I would have missed the target?”.

Horowitz and Wolfe (2001) used methods other than dynamic search to argue for a failure to keep track of rejected distractors, but others have found evidence for some, albeit, fairly limited memory (Dickinson & Zelinsky, 2005; McCarley et al., 2003). Of course, in extended search tasks over longer time scales (e.g. search the house for your keys), there must be some memory for where you have looked, but it seems safe to argue that search is not terminated by checking when all relevant items have been marked as rejected because observers do not keep reliable track of what distractors have been rejected.

The search termination question becomes more complicated when the family of search tasks expands beyond the search for a single target among discrete distractors. These laboratory tasks do not capture the richness of real-world searches, including socially important artificial search tasks like those in radiology or security screening. For instance, what happens if the number of targets is unknown? In radiology, the phenomenon of “satisfaction of search” (SoS) describes a situation in which finding one target makes it less likely that an observer will find a second target (Nodine et al., 1992; Tuddenham, 1962). The original thought was that SoS was caused by early search termination but that turns out to be incorrect (Berbaum et al., 1991). A body of research shows that there are multiple causes (Berbaum et al., 2019; Cain et al., 2013) including early termination as a possibility.

More generally, search tasks in areas such as medical image perception, airport security (Mitroff et al., 2014; Wolfe et al., 2013) and driving (Robbins & Chapman, 2018) are different from laboratory studies of search for one target that is present on 50% of trials (for recent reviews of Medical Image Perception, see Williams & Drew, 2019; Wolfe, 2016; Wolfe, Evans, & Drew, 2018; Wu & Wolfe, 2019; and most notably Samei and Krupinski’s handbook, 2018).

When the number of targets becomes large, search tasks become foraging tasks as, for example, when you are picking berries. You are performing a search task for each berry, but the more interesting question in foraging becomes, “When is it time to move to the next berry bush?” The animal literature on foraging provides rules for “patch leaving”, like the “marginal value theorem” (MVT: Charnov, 1976). These can be seen as variants of search termination rules. They predict that foragers should leave some “targets” behind as the rate of collection drops and the next patch/bush/display seems more promising. Humans roughly follow these optimal foraging predictions (Wolfe, 2013) though one can imagine times when it would be important to find every target (e.g. cancer metastases) and where an MVT-style strategy would not be good.

Tasks like cancer and airport screening raise a different issue: target prevalence. Laboratory search tasks tend to have targets present on 50% of trials (sometimes 100% if the task is to localize the target). Screening tasks, by contrast, have very low prevalence (e.g. 3–5 cancers per 1000 breast cancer screening cases, Jackson et al., 2012). Low prevalence causes Os to miss targets that they would have found in a higher prevalence context (Wolfe et al. 2005) even in true clinical settings (Evans et al., 2013). Os do tend to quit more quickly in low prevalence search but the main driver of the prevalence effect is a “criterion shift”. Os become more reluctant to identify an ambiguous item as a target (reviewed in Horowitz, 2017).

Can a generalized quitting model apply across these different types of search tasks? Perhaps, we can formulate quitting across tasks as a negative answer to the question, “Is continuing to search this stimulus a good use of my time?”. Thus, a basic quitting model could be a quitting signal that rises toward a threshold as a function of time or of successive deployments of attention. This is roughly equivalent to a collapsing decision bound that terminates the search early, typically with a target-absent response. In the quitting signal version, search ends when the target is found or enough targets are found or when the quitting signal reaches a quitting threshold. This is the rule that was implemented in Wolfe and VanWert (2010). But how is that quitting threshold set? There are some complications. Chun and Wolfe (1996) proposed an adaptive rule where observers search longer after making an error and for a shorter period after correctly quitting and versions of this idea have been part of GS ever since (c.f. Shi et al., 2019). Simply searching for about N msec or searching through about N items will not work, even in a standard search experiment, if the set size is varied. The quitting time on a trial must vary with the set size on that trial. Interestingly, RT x set size functions and error rates look substantially the same whether different set sizes are mixed or run as separate blocks (Wolfe et al., 2010a). Thus, it seems that the quitting threshold on a given trial is based on a rapid assessment of the stimuli present on that trial. You can’t search through some percentage of the items if you do not have some notion of how many items are present. Or, roughly equivalently, you can’t know how long to search, if you do not have some ideas about the stimulus that is being searched.

This initial assessment of the nature of the search array must be done very quickly given that RTs for simple target-absent trials can be just a few hundred msec long, including the motor response time. Thus, the first step in setting a quitting threshold probably involves assessing the gist of the scene via the non-selective pathway, mentioned earlier. A glance will give an observer some idea of how long to search before giving up on the hunt for granola at the hotel breakfast buffet. Importantly, this assessment and the setting of that initial quitting threshold does not require experience with a block of practice trials. After all, though lab experiments may involve hundreds of essentially similar trials, many real-world searches are unique, one-time events. If there are multiple searches, Os can adaptively refine quitting thresholds based on feedback about errors (Chun & Wolfe, 1996), target prevalence (Wolfe & Van Wert, 2010), and developing expertise (Brams et al., 2019). Moreover, Os probably adjust quitting thresholds within a single search based on an evolving assessment of the current stimulus. This is nicely captured in the “Competitive Guided Search” model of Moran et al. (2013). They model the probability of quitting at each step in a search as P(quit) = W_quit/( W_quit+Σ(stimuli)), where W_quit is a quitting signal that grows on each step and Σ(stimuli) is some assessment of the likelihood that a target could be present (Moran’s et al.’s definition is more precise.) Early in a search, P(quit) will be dominated by stimulus. As time goes on, W_quit will push P(quit) toward 1.0.

If we assume that the stimulus signal is bigger when a target is present (even if that target has not been found, see, for example, Evans et al., 2013), then this formulation can help explain a curious mystery of search termination. If we look at RT distributions, we can compare the mean quitting time to the target present distribution. One would expect that the percentage of target-present RTs falling above that target-absent mean would be roughly the percentage of miss errors. In fact, the percentage of target-present RTs above the target-absent mean is much greater than the miss error rate (Fig 5 of Wolfe et al., 2010b). One possibility, captured by the Moran et al. model, is that a larger stimulus signal on target-present trials keeps Os searching longer when there is, in fact, a target to be found. This topic deserves further research.

Conclusions

Writing a review of visual search is daunting. For an apparently narrow topic, the literature is very large and the demands of space have resulted in the omission of many topics. Whole reviews could be and have been written about the underlying neurophysiology (e.g. Treue, 2014) and much more could be written about topics that were only glancingly discussed, like attentional capture. This review has tried to emphasize areas with significant potential for future progress. A non-comprehensive list of research topics would include:

1. Guidance; notably, scene guidance. Scene guidance does a great deal of work in the real world. It is not entirely clear how that is accomplished.

2. Search templates and the internal workings of search. There is work to be done if we want to understand how an intention to find the cat is translated into target templates and guiding templates that allow you to perform the task.

3. Search termination: Do different tasks use versions of the same termination strategy or do we need to figure out different rules for each task.

4. Use-inspired search tasks: We are only at the beginning of drawing research projects from real-world search problems. Most of what we have studied to date has involved real world tasks that conveniently mimic the repetitive structure of trial-based lab paradigms. For practical reasons, we know very little about search tasks that evolve over minutes, hours, or days. We also know little about one-shot search tasks. Searching for a place to eat in Campo de Fiore in Rome. You are not going to do that hundreds of times. How do you get it right (or right enough) the first time? (The answer, by the way, is Roscioli’s.)

The questions will not run out before it is time for the next review.