Template-to-distractor distinctiveness regulates visual search efficiency

Abstract

All models of attention include the concept of an attentional template (or a target or search template). The template is conceptualized as target information held in memory that is used for prioritizing sensory processing and determining if an object matches the target. It is frequently assumed that the template contains a veridical copy of the target. However, we review recent evidence showing that the template encodes a version of the target that is adapted to the current context (e.g., distractors, task, etc.); information held within the template may include only a subset of target features, real world knowledge, pre-existing perceptual biases, or even be a distorted version of the veridical target. We argue that the template contents are customized in order to maximize the ability to prioritize information that distinguishes targets from distractors. We refer to this as template-to-distractor distinctiveness and hypothesize that it contributes to visual search efficiency by exaggerating target-to-distractor dissimilarity.

A. Introduction

All models of attention include the concept of the attentional template (or equivalently, a target or search template or attentional set). It refers to the representation of target information held in memory during visual search [1–5]. Template information is used to locate targets by selectively prioritizing relevant sensory information and by serving as the pattern for determining target-matches or non-matches. Importantly, the template reflects expectations of what information will be useful in locating the target. The idea of the template is central to models of attention because it reflects the source of information to constrain search processes and determine the relevance of information for current behavior. However, despite its ubiquitous presence in the study of attention, there are still many open questions regarding what information is stored in the template and how its contents affect visual search efficiency. Here, we review studies that demonstrate the flexibility of the template contents and propose that the template-to-distractor distinctiveness is an underappreciated factor in predicting visual search efficiency

B. Historical perspective

The concept of the attentional template predates the actual term. In some of the earliest formal studies of visual search, it was discovered that changing what subjects know about the target can radically change visual search efficiency. For example, Green and Anderson (1956) asked subjects to find a colored two-digit number among colored distractor digits, some of which shared the target color [7]. They found that search times were proportional to the number of distractor digits that shared the target color when observers knew the target digit’s color in advance, but search times depended on the total number of digits when the target’s color was not known in advance. These early demonstrations [8, 9] showed that visual search times are determined by the type of information observers have about the target and how well that information distinguishes targets from distractors.

Despite similar ideas in other theories [10–18], Duncan and Humphreys (1989) appear to be the first to use the term “template” in the human literature to describe the hypothesized internal representation of target information that modulates sensory gain and is used to determine if visual inputs match the target. The idea that the template contributes to two processes - proactive setting of sensory priority and reactive decisions about the target match - is consistent with brain imaging studies that find separate “source” and “site” regions engaged during attentionally demanding tasks [19–32], and behavioral studies that find early and late effects of target-to-distractor and distractor-to-distractor similarity in visual search [33–35].

The template is the stored memory representation of the target, but it is frequently assumed to be an exact replica of the target. In more recent years, however, it has become increasingly clear that the template contents are not always a veridical copy of the target. Sometimes, the template may be poorly specified (e.g., when looking for a dog, without knowing its specific features), contain extra-sensory knowledge (e.g., using contextual predictors of the target), or may even be strategically shifted to exaggerate the dissimilarities between targets and distractors. This has led us to the view that the template is a highly dynamic and flexible representation that is customized with all the available information about the target given the observer and context. We propose that the customization of the target template is an overlooked source of visual search efficiency that is described by the template-to-distractor similarity, which is separate from the target-to-distractor, distractor-to-distractor similarity, or even template-to-target relationship (Figure 1).

Figure 1:

A) A cartoon illustration of how template information is shaped by learned expectations about the search context and impacts sensory processing. The target is the pink circle within the search displays. Multiple search displays over time contribute to the learned expectations of distractor properties. Because the target appears with “bluer” distractors, the template encodes a “redder” target than veridical. This produces larger “top-down” gain enhancements and sensory responses that prioritize the off-veridical “reddish” color over the true target “pinkish” color. Despite being slightly off-veridical, the shift in the template is advantageous because it prioritizes target features that are more distinct from distractors. B) Illustration of the representational geometry of the target template and different objects in the visual display. All four relationships (labeled 1–4) are important for predicting visual search efficiency. The specific geometry here is just an illustrative case of the more general thesis that the target template is rarely an exact replica of the searched-for target, and instead, an adapted version that accentuates information that best differentiates the target from distractors. The specific geometry (illustrated distances 1–4) are expected to change according to stimulus context and task goals.

C. Content Specificity

If the attentional template serves to filter stimuli entering the visual system and working memory, it follows that more precise templates should produce faster attentional guidance to the correct target object(s), and shorter target-match decisions. For example, Hout and Goldringer (2015) provided observers with target cues that were either an exact visual match, the same object from a different viewpoint, or a different exemplar from the same category. They found that less specific cues resulted in longer saccade scan paths as well as longer fixation durations on the target after it was fixated [36–40]. More specific templates also produce larger N2pc ERPs, an index of visual attentional selection, than more general ones, suggesting precise templates lead to more effective guidance [41].

These studies argue for the advantage of template specificity, but search in all these cases involved a specific object whose true features could be known in advance. Real world situations often require tolerance for uncertainty in visual properties of the target either because the target itself is variable (e.g., the clothes on a friend) or the target comes from a category (e.g., cars on the road). Bravo and Faird (2012, 2016) asked observers to find a tropical fish (of any kind) in naturalistic undersea scenes [42, 43]. Critically, during training, one group saw targets that were identical to the cue; another group saw targets that were from the same species but possibly a different exemplar. A day later, the observers that only saw exact cue-target pairs showed poorer performance than the group that underwent cue-category training when a new target fish was introduced. The enhanced generalization of the template to the new exemplar was attributed to the fact that the subjects in the cue-category group learned to extract target templates that contained information common across a species but that subjects in the cue-image group learned to store specific visual features that were most useful when cues were exact. This suggests the utility of precise details in the template is task-dependent and sparser representations may be more optimal when generalizing over variable targets.

In addition to being more generalizable, categorical templates have the advantage of being less demanding to encode and maintain in memory as a verbal heuristic. It is therefore judicious to trade precision for category information whenever broad templates are sufficient for distinguishing targets from distractors [44, 45]. In an elegant example of this, Schmidt and Zelinsky (2017) asked subjects to find a teddy bear target image embedded within non-teddy bear objects (easy search) or other teddy bears (difficult search). The contralateral delay activity, an ERP index of the number of items held in working memory during a delay [46, 47], had a larger amplitude during blocks of difficult search [48]. This suggested that more target details were held in the target template when target selection was expected to be difficult. The template contents depended on the expected number of visual details necessary for the task, not the target stimulus, per se. The ability to hold only necessary information within the template to discriminate the target from distractors maximizes the efficiency of target selection while minimizing memory demands [41, 49].

In real-world scenes, there is evidence that prior knowledge about object regularities and likely actions are also brought into the template to aid search [36, 37, 50–52]. Thus, search templates may have more information than just the visual properties of the target. For example, Malcolm and Henderson (2009) found that prior knowledge, such as that clocks are more likely on walls than floors, produced additive advantages during search with cue specificity (i.e., cued with the exact visual image vs. a verbal label). The advantages were seen in both the guidance of attention (saccade scan paths) and decision times to verify the target once fixated. That target templates in the real world contain learned associations demonstrates the importance of accounting for prior knowledge [50, 51, 53]and individual differences on visual search [25, 54].

Together these studies suggest that the most optimal level of specificity in a target template depends on the environmental context. More precise templates are better for targets in complex scenes with many similar distractors, but efficiency can be increased even for precise templates by including extra-sensory information such as learned probabilities and semantic relatedness. However, only subset of (or coarser) information may be sufficient when the distractor context is dissimilar or when targets have uncertain or variable features.

D. Optimal Feature Tuning

Perhaps some of the most compelling evidence for the idea that the template contents are based on template-to-distractor distinctiveness, rather than a veridical feature value, comes from studies in which the target representation shifts “off-veridical” to optimize behavior. Even when the target features are available to the observer, the target representation may be shifted in order to optimize the ability to distinguish targets from distractors, sacrificing template-to-target accuracy (Figure 1B). Navalpakkam and Itti (2007) gave the intuition of looking for a tiger in the grasslands: if the tiger is orange-yellow, but the grasslands are generally more yellow than the tiger, it may be more optimal to only bias top-down sensory enhancement of neurons encoding “orange-reddish” features [14]. This is because those features are unique to the tiger whereas “yellowish” elements are shared with the grassland. Shifting of target features has been found for varied stimulus dimensions including orientation, size, and color and have been measured using a number of methods including: set size estimates of distractor costs, irrelevant cues to probe for attentional capture, working memory estimation probes, and 2-alternative forced choice identification tasks [14, 55–60]. These studies suggest that linearly separable distractors (i.e., distractor sets that can be distinguished from targets by a single line within feature space) produce efficient visual search because the template can be shifted away from distractors, which increases the template-to-distractor distinctiveness [61].

Although it is clear that this behavior is efficient when searching for a target among linearly separable distractors, different underlying mechanisms have been proposed. Earlier models of selective attention hypothesized this effect was due to weighting parallel features that are less likely to select distractors [2, 18, 45]. Navalpakkam and Itti (2007) used signal detection theory and empirical data to demonstrate that the shift in target representation was due to “optimal gains” modulation of neurons that encode the most distinctive target features [62, 63]. Becker and colleagues have proposed the “relational theory of visual attention” that hypothesizes the target is encoded relationally (e.g., as the “reddest” or “shortest” object) rather than a specific value [64, 65]. The relational theory does not posit changes in sensory gain for specific feature values in advance of the search display; instead the stimulus that best fits the relational rule, e.g., “reddest” is automatically selected, even if it is distant from the actual target. Furthermore, others have found that the similarity of the expected distractor set to the target may produce asymmetrical sharpening of the template in addition to overall shifting [55, 58]. It is interesting to consider that these strategies may not be mutually exclusive but depend upon task goals. While the exact mechanisms are still to be determined, these studies emphasize that the goal of the attentional template is to prioritize information that distinguishes targets from distractors, i.e., the template-to-distractor distinctiveness rather than represent the veridical target directly.

In addition to shifting target representations away from specific features within a stimulus dimension, it is possible to selectively enhance or suppress feature dimensions in sensory cortex analogous to other models of feature-based attentional selection [66] [67]. Reeder et al. (2017) used a conjunction Gabor target composed of two feature dimensions (spatial frequency, orientation) but only one feature dimension distinguished targets from distractors (i.e., informative dimension) [68]. They found that the neural similarity of expected targets during the post-cue delay period reflected the informative dimension. Biasing template representations in memory towards feature dimensions that are expected to be informative has also been found using retro-cues [60]. Together these studies show that optimal template representations encode information that are expected to maximally differentiate targets from distractors and that this information may involve shifting, supplementing, or even reducing information about the veridical target object. However, the use of optimal strategies may depend on individual differences in perceived effort [69, 70].

E. Definitions of Similarity

The concept of similarity underlies all attentional theories [5] and is a core concept in understanding the template contents, but similarity can be defined based on physical, categorical, semantic, contextual, or even idiosyncratic characteristics. Even stimuli from single continuous feature dimensions, such as color and orientation, have strong nonlinear categorical boundaries that vary across observers and tasks [71–73]. One challenge for studies of attention is that the architecture of similarity must be measured outside of concurrent visual search tasks if they are used to predict visual search efficiency. Estimating similarity within visual search (or any attentionally demanding task) risks circularity by theorizing that target-similar stimuli interfere with target selection, and concluding that a distractor is target-similar because it interferes with attentional selection.

Some attention researchers have begun to use independent measurements of similarity using stimulus pair-wise similarity ratings, multidimensional scaling, and computational models [74–79]. Studies employing these techniques have found visual search dynamics depend on visual similarity, categorical membership, conceptual relatedness, and stored world knowledge about scenes [25, 33, 40, 52, 76, 80–85]. For example, search performance for a target category (e.g., teddy bears) depended on the rank ordered similarity of distractor objects obtained in an independent task (e.g., toys, clothing, tools, etc.) [86]. The similarity task provided an objective prediction of what objects would interfere with target selection without relying on preconceived notions of visual or semantic structure. Such independent ratings can reveal surprising consistency between the organization of knowledge and visual search performance. For example, we found that idiosyncratic differences in the perceptual categorization of facial identities predicted visual search efficiency when those faces were targets and distractors in visual search [25]. There is no doubt that target-similar distractors are more likely to capture attention and are harder to reject, but our understanding of how similarity is defined is still emerging. The challenge now is to understand how stable visual, semantic, and episodic representational structures are transformed into attentional templates during visual search that maximize the ability to distinguish targets from distractors [21–23, 27]

F. Conclusions

The idea of templates is ubiquitous in theories of attention. Most hypothesize that template information is used to set sensory gain in order to enhance processing of task-relevant information, and to compute if an object matches the target (Figure 1A). Much of the literature has assumed that the best target template is one that perfectly matches the veridical target, but encoding a perfectly veridical target might be the exception instead of the rule in real-world situations. We have argued here that the template should be understood as a custom set of information that will best differentiate the target from non-targets in the current environment. Sometimes this might mean holding only subset of target features, including extra-sensory knowledge about the target, or even shifting the target representation. A better understanding of what optimal templates contain will require a more systematic mapping of similarity between stimuli within a context. The concept of the template-to-distractor distinctiveness captures the relationship between these ideas and describes how the internal representation of the target is customized to contain the best available information that distinguishes the target from distractors in the current moment of time.

Highlights.

• The attentional template encodes information about the target and is held in working or long-term memory.

• The attentional template does not contain a veridical copy of the target but a version that maximizes the ability to distinguish targets from distractors.

• Template-to-distractor distinctiveness describes the relationship between the version of the target held in memory and the current environment.