Visual search in scenes involves selective and non-selective pathways

Abstract

How do we find objects in scenes? For decades, visual search models have been built on experiments in which observers search for targets, presented among distractor items, isolated and randomly arranged on blank backgrounds. Are these models relevant to search in continuous scenes? This paper argues that the mechanisms that govern artificial, laboratory search tasks do play a role in visual search in scenes. However, scene-based information is used to guide search in ways that had no place in earlier models. Search in scenes may be best explained by a dual-path model: A “selective” path in which candidate objects must be individually selected for recognition and a “non-selective” path in which information can be extracted from global / statistical information.

Searching and experiencing a scene

It is an interesting aspect of visual experience that we can look for an object that is, literally, right in front of our eyes, yet not find it for an appreciable period of time. It is clear that we are seeing something at the object’s location before we find it. What is that something and how do we go about finding that desired object? These questions have occupied visual search researchers for decades. While visual search papers have conventionally described search as an important real-world task, the bulk of research had observers looking for targets among some number of distractor items, all presented in random configurations on otherwise blank backgrounds. In the last decade, there has been a surge of work using more naturalistic scenes as stimuli and this has raised the issue of the relationship of the search to the structure of the scene. This paper will briefly summarize some of the models and solutions developed with artificial stimuli and then describe what happens when these ideas confront search in real-world scenes. We will argue that the process of object recognition, required for most search tasks, involves the selection of individual candidate objects because all objects cannot be recognized at once. At the same time, the experience of a continuous visual field tell us that some aspects of a scene reach awareness without being limited by the selection bottleneck in object recognition. Work in the past decade has revealed how this non-selective processing is put to use when we search in real scenes.

Classic Guided Search

One approach to search, developed from studies of simple stimuli randomly placed on blank backgrounds can be called “classic Guided Search” [1]. It has roots in Treisman’s Feature Integration Theory [2]. As will be briefly reviewed below, it holds that search is necessary because object recognition processes are limited to one or, perhaps, a very few objects at one time. The selection of candidate objects for subsequent recognition is guided by preattentively acquired information about a limited set of attributes like color, orientation, and size.

Object recognition is capacity limited

We need to search because, while we are very good at recognizing objects, we cannot recognize multiple objects at the same time. For example, all of the objects in Figure 1 are simple in construction, but if you are asked to find “T”s that are purple and green, you will find that you need to scrutinize each item until you stumble upon the targets (there are four). It is introspectively obvious that you can see a set of items and could give reasonable estimates for their number, color, and so forth. However, recognition of a specific type of item requires another step of binding the visual features together [3]. That step is capacity-limited and very often, attention demanding [4] (however see [5]) .

Figure 1.

Find the four purple and green Ts. Even though it is easy to identify such targets, this task requires search.

In the case of Figure 1, the ability to recognize one object is also going to be limited by the proximity of other, similar items. These “crowding” phenomena have attracted increasing interest in the past few years ([6] [7]). However, though it would be a less compelling demonstration, it would still be necessary to attend to item after item in order to bind their features and recognize them even if there were very few items and even if those items were widely spaced [8].

The selection mechanism is a serial / parallel hybrid

While it is clear that object recognition is capacity-limited, the nature of that limitation has been less clear (for an earlier discussion of this issue, see [9]). The classic debate has been between “serial” models that propose that items are processed one after the other [2] and “parallel” models that hold that multiple objects, perhaps all objects, are processed at the same time but that the efficiency of processing of any one item decreases as the number of items increases [10] [11]. The debate has been complicated by the fact that the classic reaction time data, used in many experiments, is ambiguous in the sense that variants of serial and parallel models can produce the same patterns of data [12]. Neural evidence has been found in support of both types of processes (see Box 1).

Box 1: Neural signatures of parallel and serial processing.

What would parallel and serial processing look like at a neuronal level? One type of parallel processing in visual search is the simultaneous enhancement of all items with a preferred feature (e.g. all the red items). A number of studies have shown that for cells demonstrating a preference for a specific feature, the preference is stronger when the task is to find items with that feature [13]. For serial processing, one would like to see the “spotlight” of attention moving around from location to location. Buschman and Miller [14] saw something like this when it turned out that monkeys in their experiment liked to search a circular array of items in the same sequence on every trial. As a result, with multiple electrodes in place, they could see an attentional enhancement rise at the 3 o’clock position, then fall at 3 and rise at 6, as attention swept around in a serial manner to find a target that might be at the 9 o’clock position this time. Similar shifts of attention can be seen in human evoked potential recordings [15]. Bichot et al. [16] produced an attractive illustration of oth processes at work in visual area, V4. When the monkey was searching for “red”, a cell that liked red would be more active, no matter where the monkey was looking and/or attending. If the next eye movement was going to take the target item into the cell’s receptive field, the cell showed another burst of activity as serial attention got there in advance of the eyes.

Like many cognitive science debates, the correct answer to the serial/parallel debate is probably “both”. Consider the timing parameters of search. One can estimate the rate at which items are processed from the slopes of the reaction time (RT) by set size functions. Although the estimate depends on assumptions about factors like memory for rejected distractors (Box 2), it is in the range of 20-50 msec/item for easily identified objects that do not need to be individually fixated [26]. This estimate is significantly faster than any estimate of the total amount of time required to actually recognize an object [27]. Even on the short end, object recognition seems to require more than 100 msec/item (<10 items/second). Note that we are speaking about the time required to identify an object, not the minimum time that an observer must be exposed to an object, which can be very short, indeed [28] .

Box 2: Memory in Visual Search.

There is a body of seemingly contradictory findings about the role of memory in search. First, there is the question of memory during a search. Do observers keep track of where they have been by, for example, inhibiting rejected distractors? There is some evidence for inhibition of return in visual search [17, 18] though it seems clear that observers cannot use inhibition to mark every rejected distractor [19, 20]. Plausibly, memory during search serves to prevent perseveration on single salient items [18, 21].

What about memory for completed searches? If you find a target once, are you more efficient when you search for it again? A body of work on “repeated search” finds that search efficiency does not improve even over hundreds of trials of repetition [22, 23]. On the other hand, observers can remember objects that have been seen during search [24] and implicit memory for the arbitrary layout of displays can speed response [25]. How can all of these facts be true? Of course, observers remember some results of search. (Where did I find those scissors last time?). The degree to which these memories aid subsequent search depends on whether it is faster to retrieve the relevant memory or to repeat the visual search. In many simple tasks (e.g. with arrays of letters; [23]), memory access is slower than visual search [22]. In many more commonplace searches (those scissors), memory will serve to speed the search.

As a solution to this mismatch of times, Moore and Wolfe [20] proposed a metaphorical “carwash” (also called “pipeline” in computer science). Items might enter the binding and recognition carwash one after another every 50 msec or so. Each item might remain in the process of recognition for several hundred milliseconds. As a consequence, if an experimenter looked at the metaphorical front or the back of the carwash, serial processing would dominate but if one looked at the carwash as a whole, one would see multiple items in the process of recognition in parallel.

Other recent models also have a serial / parallel hybrid aspect though often quite different from the carwash in detail [29, 30]. Consider, for example, models of search with a primary focus on eye movements [31-33]. Here, the repeated fixations impose a form of serial selection every 250 msec or so. If one proposes that 5 or 6 items are processed in parallel at each fixation, one can produce the throughput of 20-30 items/second items found in search experiments. Interestingly, with large stimuli that can be resolved in the periphery, the pattern of response time data is similar with and without eye movements [34]. Given the close relationship of eye movements and attention [35],it could be proposed that search is accomplished by selecting successive small groups of items, whether the eyes move or not. Note that all of these versions are hybrids of some serial selection and parallel processing.

A set of basic stimulus attributes guide search

Object recognition may require attention to an object [36], but not every search requires individual scrutiny of random items before the target is attended. For example, in Figure 1, it is trivial to find the one tilted “T”. Orientation is one of the basic attributes that can guide the deployment of attention. A limited set of attributes can be used to reduce the number of possible target items in a display. If you are looking for the big, red, moving vertical line, you can guide your attention toward the target size, color, motion, and orientation. We label the idea of guidance by a limited set of basic attributes as “Classic Guided Search” [37]. The set of basic attributes is not perfectly defined but there are probably between one and two dozen [38]. In the search for the green and purple Ts of Figure 1, guidance fails. T’s and L’s both contain a vertical and a horizontal line, so orientation information is not useful. The nature of the T or L intersection is not helpful [39], nor can guidance help by narrowing the search to the items that are both green AND purple. When you specify two features (here two colors) of the same attribute, attention is guided to the set of items that contain either purple OR green. In Figure 1, this is the set of all items [40] so no useful guidance is possible.

The internal representation of guiding attributes is different from the perceptual representation of the same attributes. What you see is not necessarily what guides your search. Consider color as an example. An item of unique color “pops out”. You would have no problem finding the one red thing among yellow things [41]. The red thing looks salient and it attracts attention. It is natural to assume that the ability to guide attention is basically the same as the perceived salience of the item [42, 43]. However, look for the desaturated, pale targets in Figure 2 (There are two in each panel). In each case, the target lies halfway between the saturated and white distractors in a perceptual color space. In the lab, though not in this figure, the colors can be precisely controlled so that the perceived difference between red and pale red is the same as the difference between pale green and green or pale blue and blue. Nevertheless, the desaturated red target will be found far more quickly [44], a clear dissociation between guidance and perception. Similar effects occur for other guiding attributes such as orientation [45]. The representation guiding attention should be seen as a control device, managing access to the binding and recognition bottleneck. It does not reveal itself directly in conscious perception.

Figure 2.

Find the desaturated color dots. Colors are only an approximation of the colors that would be used in a carefully calibrated experiment. The empirical result is that it is much easier to find the pale red (pink) targets than to find pale green or blue.

Visual Search in Natural(istic) Scenes

The failure of classic guided search

To this point, this article has described what could be called “classic Guided Search” [1, 37]. Now, suppose that we wanted to apply this classic Guided Search theory to the real world. Find the bread in Figure 3a. Guided Search and similar models would say that the 1-2 dozen guiding attributes define a high-dimensional space in which objects would be quite sparsely represented. That is, “bread” would be defined by some set of features [33]. If attention were guided to objects lying in the portion of the high-dimensional feature space specified by those features, few other objects would be found in the neighborhood [46]. Using a picture of the actual bread would produce better guidance than its abstract label (“bread”) because more features of the specific target would be precisely described [47]. So in the real world, attention would be efficiently guided to the few bread-like objects. Guidance would massively reduce the “functional set size” [48] .

Figure 3.

Find the loaf of bread in each panel.

It is a good story, but it is wrong or, at least, incomplete. The story should be just as applicable to search for the loaf of bread in Figure 3b; maybe more applicable as these objects are clearly defined on a blank background. However, searches for isolated objects are quite inefficient [49] while searches like the kitchen search are very efficient (given some estimate of “set size” in real scenes) [50]. Models like Guided Search, based on bottom-up and top-down processing of a set of “preattentive” attributes seem to fail when it comes to explaining the apparent efficiency of search in the real world. Guiding attributes do some work [33, 51], but not enough.

The way forward: Expanding the concept of guidance for search in scenes

Part of the answer is that real scenes are complex, but never random. Elements are arranged in a rule-governed manner: People generally appear on horizontal surfaces [52, 53], chimneys appear on roofs [54], and pots on stoves [55]. Those and other regularities of scenes can provide scene-based guidance. Borrowing from the memory literature, we will refer to “semantic” and “episodic” guidance. Semantic guidance includes knowledge of the probability of the presence an object in a scene [55] and of its probable location in that scene given the layout of the space [52, 56], as well as inter-object relations (e.g. knives tend to be near forks, [57]). Violations of these expectations impede object recognition [58] and increase allocation of attention [55]. It can take longer to find a target that is semantically misplaced, (e.g., searching for the bread in the sink [59]). Episodic guidance, which we will merely mention here, refers to memory for a specific, previously encountered scene that comprises information about specific locations of specific objects [60]. Having looked several times, you know that the bread is on the counter to the left – not in all scenes, but in this one. The role of memory in search is complex (Box 2), but it is certainly the case that you will be faster, on average, to find bread in your kitchen than bread in another’s kitchen.

When searching for objects in scenes, classical sources of guidance combine with episodic and semantic sources of guidance to efficiently direct our attention to those parts of the scene that have the highest probability of containing targets [52, 61-63]. In naturalistic scenes, guidance of eye movements by bottom-up salience seems to play a minor role compared to guidance by more knowledge-based factors [63, 64]. A short glimpse of a scene is sufficient to narrow down search space and efficiently guide gaze [65] as long as enough time is available to apply semantic knowledge to the initial scene representation [58]. However, semantic guidance cannot be too generic. Presenting a word prime (e.g. “kitchen”) instead of a preview of the scene does not produce much guidance [47]. Rather, the combination of semantic scene knowledge (kitchens) with information about the structure of the specific scene (this kitchen) seems to be crucial for effective guidance of search in real-world scenes [58, 63].

A problem: Where is information about the scene coming from?

It seems reasonable to propose that semantic and episodic information about a scene guides search for objects in the scene, but where does that information come from? In order for scene information to guide attention to likely locations of “bread” in Figure 3a, you must know that the figure shows something like a kitchen. One might propose that the information about the scene develops as object after object is identified. A “kitchen” hypothesis might emerge quickly if you were lucky enough to attend first to the microwave and then to the stove, but if you were less fortunate and attended to a lamp and a window, your kitchen hypothesis might come too late to be useful.

A non-selective pathway to gist processing

Fortunately, there is another route to semantic scene information. We are able to categorize a scene as a forest without selecting individual trees for recognition [66]. A single, very brief fixation on the kitchen of Figure 3a would be enough to get the “gist” of that scene. “Gist” is an imperfectly defined term but, in this context, it includes the scene’s basic-level category, an estimate of the distributions of basic attributes like color and texture [67], and the spatial layout [66, 68-70]. These statistical and structural cues allow very brief exposures to support above chance categorization of scenes into, for example, natural or urban [66, 71, 72] or containing an animal [28, 73]. Within a single fixation, Os would know that Figure 3a was a kitchen without the need to segment and identify its component objects. At 20-50 objects/second, the observer will have collected a few object identities as well but, on average, these would not be sufficient to produce categorization [66, 74].

How is this possible? The answer appears to be a two-pathway architecture somewhat different from but perhaps related to previous two-pathway proposals [75, 76], and somewhat different from classic two-stage, preattentive-attentive models (see Box 3). The basic idea is cartooned in Figure 4. Visual input feeds a capacity-limited “selective pathway”. As described earlier, selection into the bottleneck is mediated by classic guidance and, when possible, by semantic and episodic guidance. In this two-pathway view, the raw material for semantic guidance could be generated in a non-selective pathway that is not subject to the same capacity limits. Episodic guidance would be based on the results of selective and non-selective processing.

Box 3: Old and new dichotomies in theories of visual search.

The dichotomy between selective and non-selective pathways, proposed here, is part of a long tradition of proposing dichotomies between processes with strong capacity limits that restrict their work to one or a few objects or locations and processes that are able to operate across the entire image. It is worth briefly noting the similarities and differences with some earlier formulations.

Preattentive and Attentive processing

Preattentive processing is parallel processing over the entire image. Like non-selective processing, it is limited in its capabilities. In older formulations like Feature Integration Theory [2], it handled only basic features like color and orientation but it could be expanded to include the gist and statistical processing abilities of a non-selective pathway. The crucial difference is embodied in the term ‘preattentive’. In its usual sense, preattentive processing refers to processing that occurs before the arrival in time or space of attentive processing [77]. Non-selective processing, in contrast, is proposed to occur in parallel with selective processing with the outputs of both giving rise to visual experience.

Early and Late Selection

The non-selective pathway could be seen as a form of late selection in which processing proceeds to an advanced state prior to any bottleneck in processing [78]. The selective pathway embodies early selection with only minimal processing prior to the bottleneck. Traditionally, these have been seen as competing alternatives that here coexist. Note, however, traditional late selection would permit object recognition (e.g. word recognition) prior to a bottleneck. The non-selective pathway, while able to extract some semantic information from scenes is not proposed to have the ability to recognize objects or letters.

Figure 4.

A two-pathway architecture for visual processing. A selective pathway can bind features and recognize objects, but it is severely capacity-limited. The limit is shown as a “bottleneck” in the pathway. Access to the bottleneck is controlled by guidance mechanisms that allow items that are more likely to be targets preferential access to feature binding and object recognition. Classic guidance, cartooned in the box above the bottleneck, gives preference to items with basic target features (e.g. color). This paper posits scene guidance (semantic and episodic), with semantic guidance derived from a non-selective pathway. This non-selective pathway can extract statistics from the entire scene, enabling a certain amount of semantic processing but not precise object recognition.

What is a “non-selective pathway”? It is important not to invest a non-selective pathway with too many capabilities. If all processing could be done without selection and fewer capacity limits, we would not need a selective pathway. Global non-selective image processing allows observers to rapidly extract statistical information from the entire image. Observers can assess the mean and distribution of a variety of basic visual feature dimensions: size [79], orientation [80], some contrast texture descriptors [81], velocity and direction of motion [82], magnitude estimation [83], center of mass for a set of objects [84], and center of area [85]. Furthermore, summary statistics can be calculated for more complex attributes like emotion [86] or the presence of classes of objects (e.g. animal) in a scene [87].

Using these image statistics, models and (presumably) humans, can categorize scenes [66, 68, 69] and extract basic spatial structure [66, 71]. This non-selective information could, then, provide the basis for scene-based guidance of search. Thus, non-selective categorical information, perhaps combined with the selective pathway’s identification of an object or two, could strongly and rapidly suggest that Figure 3a depicts a kitchen. Non-selective structural information could give the rough layout of surfaces in the space. In principle, these sources of information could be used to intelligently direct the resources of the selective pathway so that attention and the eyes can be deployed to likely locations of bread.

Our conscious experience of the visual world is comprised of the products of both pathways. Returning to the example at the outset of this article, when we have not yet found the object that is “right in front of our eyes”, our visual experience at that location must be derived primarily from the non-selective pathway. We cannot choose to see a non-selective representation in isolation but we can gain some insight into the contributions of the two pathways from Figure 5. The non-selective pathway would ‘see’ the forest [66] and could provide some information about the flock of odd birds moving through it. However, identification of a tree with both green and brown boughs or of a bird heading to the right would require the work of the selective path [73].

Figure 5.

What do you see? And how does that change when you are asked to look for an untilted bird or trees with brown trunks and green boughs? It is proposed that a non-selective pathway would ‘see’ image statistics like average color or orientation in a region. It could get the ‘gist’ of forest and, perhaps, the presence of animals. It would not know which trees had brown trunks or which birds were tilted.

Expert searchers like radiologists hunting for signs of cancer or airport security officers searching for threats may have learned to make specific use of non-selective signals. With some regularity, such experts will tell you that they sometimes sense the presence of a target before actually finding it. Indeed, this “Gestalt process” is a component of a leading theory of search in radiology [88]. Doctors and technicians screening for cancer can detect abnormal cases at above chance levels in a single fixation [89]. The abilities of a non-selective pathway may underpin this experience. Understanding how non-selective processing guides capacity-limited visual search could lead to improvements in search tasks that are, literally, a matter of life and death.

Concluding remarks

What is next in the study of search in scenes? We do not understand how scenes are divided up into searchable objects or proto-objects [90]. There is much work to be done to fully describe the capabilities of non-selective processing and even more to document its impact on selective processes. Finally, we would like to know if there is a neurophysiological reality to the two pathways proposed here. Suppose one “lesioned” the hypothetical selective pathway. The result might be an agnosic who could see something throughout the visual field but could not identify objects. A lesion of the non-selective pathway might produce a simultagnosic or Balint’s patient, able to identify the current object of attention but otherwise unable to see. This sounds similar to the consequences of lesioning the ventral and dorsal streams, respectively [76] but more research will be required before “selective” and “non-selective” can be properly related to “what” and “where”.