Neural mechanism of priming in visual search

Abstract

Selective attention affords scrutinizing items in our environment. However, attentional selection changes over time and across space. Empirically, repetition of visual search conditions changes attentional processing. Priming of pop-out is a vivid example. Repeatedly searching for the same pop-out search feature is accomplished with faster response times and fewer errors. We will review the psychophysical background of priming of pop-out, focusing on the hypothesis that it arises through changes in visual selective attention. We then describe research done with macaque monkeys to understand the neural mechanisms supporting visual selective attention and priming of pop-out. Next, we will survey research on priming of pop-out using noninvasive brain measures with humans. We will conclude by hypothesizing three alternative neural mechanisms and highlighting open questions.

Introduction: Attentional Selection, Visual Search, and Priming

Our environment is filled with information, some is useful to our goals, and some of it is not. Selective attention is the process through which we scrutinize items in our environment (Pashler 1997). Visual search is an especially effective task in investigating how items are selected for directed attention, a process known as attentional selection. Experimental work on selective attention began in the mid-twentieth century (Cherry 1953; Broadbent 1958) as did research on visual search (Ellis 1947; Carpenter 1948; Mackworth 1948, 1952). However, not until later were they discussed together. Some of the earliest influential work integrating visual search and selective attention was performed by Charles W. Eriksen, whom this special issue memorializes.

Pop-out visual search, perhaps the simplest form of visual search and what we will focus on in this review, is characterized by identification of a conspicuous target. That is, pop-out refers to search conditions where an object is noticeably different in one feature relative to the objects surrounding it, as for example is a red berry among green leaves. In the early 1950’s, Eriksen performed the first study investigating the speed at which items in visual search displays are selected through careful manipulation of the features making up the objects participants searched through (Eriksen 1952). He found that targets distinguished by single features (i.e., pop-outs) were identified faster than conjunctions of features. In arrays of items comprised of multiple features, a conjunction of unique features (i.e. pop-out in more than one feature) made search faster (Eriksen 1953) and more accurate (Eriksen & Hake 1955). Furthermore, in these heterogenous displays, search time increased with the number of distractors (Eriksen 1955). This early work laid the foundation for further investigation into the factors influencing human visual search performance and the role of attention in searching these arrays.

The observed behavioral differences between single-feature pop-outs and conjunction search, originally reported in the work of Eriksen and colleagues, gave rise to the idea of two interactive processes for visual search. Those processes being characterized by being ‘bottom-up’, where features are processed in parallel, and ‘top-down’, where features are integrated. Under this framework, pop-out only requires the bottom-up process as features don’t need to be bound for target identification to occur. If single features are processed in parallel across space and pop-out is defined by a difference in a single feature, then surely pop- out search is already as fast as possible. However, further investigation would prove otherwise.

Investigators showed that when participants repeatedly perform the pop-out search with the same stimuli, search times become faster. The task took the form of repetitive pop-out visual search where the search contingencies varied throughout the entire experiment (e.g., multiple features distinguished target and distractors), but remained consistent in blocks of trials (e.g., the features distinguishing target and distractor did not change for several trials) (Maljkovic & Nakayama 1994, 1996, 2000). Under these conditions, although the visual search task was as efficient as it could be (e.g., single-feature, homogenous distractors), repetitive performance in fact speeded behavioral response times (Maljkovic & Nakayama 1994, 1996, 2000). This speeding has been replicated across multiple research groups (Huang, Holocombe, & Pashler 2004; Kristjánsson 2006; Lamy, Antebi, Aviani, & Carmel 2008, for example) and species (Bichot & Schall 1999, McPeek & Keller 2001) (Figure 1). This finding suggests that pop-out search, in isolation, is not always as fast as it can be. Changes can be implemented that make pop-out search faster, search history permitting.

Figure 1.

Priming of pop-out task and response time profiles. Top: Ten example trials organized in a row showing each of the trial epochs (fixation, array onset, target identification/saccade, and reward/inter-trial interval [ITI]). Priming occurs when search conditions remain consistent (e.g., first five and last five example trials). Bottom: Response time profiles adapted from the original Maljkovic and Nakayama study (1994) (left) as well as examples from early monkey studies (McPeek & Keller 2001; Bichot & Schall 2002) (right). Response time is plotted against trial since the pop-out feature change. As search conditions repeat, response times speed

Subsequently, the mechanism underlying the behavioral improvements associated with repetition of performance in pop-out in visual search has been a matter of some debate in psychology (Kristjánsson & Campana 2010; Kristjánsson & Asgeirsson 2018). While it was initially hypothesized that priming of pop-out is mediated by memory mechanisms (Maljkovic & Nakayama 1994, 1996, 2000) and there is some psychophysical evidence suggesting this (Hillstrom 2000; Huang, Holocombe, & Pashler 2004; Huang & Pashler 2005; but see Asgeirsson & Kristjánsson 2011), other psychophysical evidence largely supports the idea that it is mediated through attentional priming. That is, speeding of behavioral responses observed in priming of pop-out is accomplished through speeding the process of attentional selection. In interleaving choice trials following priming of pop-out sequences, participants are more likely to freely choose, and are faster to attend to stimuli that were the targets of the preceding priming block (Brascamp, Blake, & Kristjánsson 2011). Furthermore, evidence for priming of pop-out as priming of attentional selection can be found beyond the psychophysics literature.

Electrophysiological (Eimer, Kiss, & Cheung 2010) and, as will be detailed below, neurophysiological (Bichot & Schall 2002; Westerberg, Maier, & Schall 2020) evidence also exists to support the priming of attentional selection hypothesis. In each of these studies, signals in or originating from brain areas associated with attentional selection during visual search, rather than areas associated with memory reactivation, were the ones to change. While other cognitive phenomena may modulate as a function of priming in this task, such as performance monitoring, which will be explored later in this review (Westerberg, Maier, Woodman, & Schall 2020), they are likely secondary to the changes in attentional selection (Kristjánsson & Asgeirsson 2018).

While a substantial contribution to the psychophysical understanding of priming of pop-out has been made over the years, a coherent picture of the neural mechanism supporting these changes is incomplete. That is not to say that the pieces do not exist. Several studies have identified brain areas contributing to, or at least affected by, priming in visual search, but these results have not been synthesized. Hence, the purpose of this review is to introduce and discuss the brain areas involved in priming of pop-out to elucidate a more integrated conception of the neural mechanism and formulate testable hypotheses for future investigation. More generally, in understanding the neural mechanism of priming of pop-out, we will gain a greater understanding of how brain areas coordinate to select behaviorally relevant stimuli in the visual environment. To achieve these objectives, we will begin by introducing how attentional selection is measured in the brain, specifically in brain areas where priming has been investigated. Next, we will survey those neurophysiological studies of priming in visual search and bridge them to noninvasive studies in humans. We will conclude the review by generating and providing support for three rival hypotheses for the neural mechanism of priming during visual search.

Attentional Selection during Visual Search at the Neural Level

Behavioral work of early pioneers in attention research, like Eriksen, has been ongoing for decades. This work has attracted neuroscientists to investigate the neural underpinnings of the behavioral phenomenon of these seminal behavioral studies. Since the genesis of neurophysiological investigation into attention began in the late 20^th century, study into visual selective attention at the neural level has blossomed and produced more work on the neural basis of selective attention than can be reasonably reviewed here (e.g., Carrasco 2011; Moore & Zirnsak 2017). Instead, we will focus on introducing attentional selection during visual search in the brain areas implicated in priming of pop-out that will be discussed later. Frontal and visual cortex are perhaps two of the most well-studied regions with respect to visual search. Specifically, the frontal eye field (FEF) in frontal cortex and area V4 in visual cortex.

FEF is a distinct frontal cortical area found in all primates (Schall, Zinke, Cosman, Schall, Pare, & Pouget 2017). It contributes to the representation of visual information and the coordination of eye movements including saccades and smooth pursuit eye movements (Schall 2015). Early work also identified several populations of functional cell types in FEF involved in the representation of visual information, fixation, and motor signaling. More recent reports suggest FEF functional populations might lie on a spectrum of more visual to more motor responses (Lowe & Schall 2018) and there might even be more complex functional types such as those that track previously viewed stimuli (Mirpour, Bolandnazar, & Bisley 2019). The responses of visual cells in FEF modulate during visual search (Schall & Hanes 1993) which suggests implications for attentional selection. Specifically, FEF neurons discriminate between targets and distractors in search arrays. Subsequent work has investigated whether this modulation is related to the motor preparation in FEF or if it is the process of attentional selection. Even though investigation in FEF had been ongoing for over a century, it was not considered an ‘attention area’ until the mid-1990’s. Earlier work suggested that modulation of responses to stimuli in FEF only occurred when gaze was to be directed to the stimulus (Goldberg & Bushnell 1981; Bruce & Goldberg 1985). Later work showed that FEF activity during the selection epoch indicated an attentional selection process, as the motor process could be distinguished from the selection process at the single-unit level (Kodaka, Mikami, & Kubota 1997; Juan, Shorter-Jacobi, & Schall 2004; Thompson, Bichot, & Sato 2005; Monosov, Trageser, & Thompson 2008). For example, in an antisaccade task where the saccadic endpoint can be dissociated from the location of the attentional target, selectivity in FEF neurons representing the space where the target is can still be observed when the saccade must be made elsewhere (Juan, Shorter-Jacobi, & Schall 2004).

Investigation into selectivity of FEF neurons in visual search tasks has uncovered additional insights important in understanding the neural underpinnings of behavioral changes associated with attentional selection. While FEF neurons predominantly do not show selectivity for features of visual stimuli, under certain circumstances, visual cells in FEF can develop feature selectivity. By repeatedly performing a visual search task where the target feature of the search array remains consistent across days, FEF neurons can develop selectivity for the behaviorally relevant feature (Bichot, Schall, & Thompson 1996). This has been observed for colors (Bichot, Schall, & Thompson 1996) as well as for shapes (Lowe & Schall 2019). This experience-driven change in the function of FEF neurons demonstrates their plasticity in the representation of features. Furthermore, FEF neurons appear to be sensitive to salient colors present in a search array, even when they are not the target, perhaps through the initial feedforward sweep of visual information from V4 (Cosman, Lowe, Zinke, Woodman, & Schall 2018). As priming of pop-out seems to be driven through the potentiation of features, this might be important to this process, albeit these reports show this effect at a much longer timescale than in priming of pop-out.

Area V4 is a mid-level visual cortical area in the ventral visual processing stream that was originally thought to be the ‘color area’ (Zeki 1973, 1978). It has since been shown to show selectivity for other features such as shapes and contours among others (Roe, Chelazzi, Connor, Conway, Fujita, Gallant, Lu, & Vanduffel 2012). It also shows robust attentional modulation (Moran & Desimone 1985), to a greater degree than earlier visual cortical areas (Luck, Chelazzi, Hillyard, & Desimone 1997; Ghose & Maunsell 2002; Buffalo, Fries, Landman, Liang, & Desimone 2010). Removal of area V4 through lesions impacts performance in attention tasks (Schiller & Lee 1991; De Weerd, Desimone, & Ungerleider 1996, 1999, 2003). Given V4’s selectivity for shapes, colors, contours, and other features as well as its robust attentional modulation, it has become a highly relevant area in the study of neural processing during visual search. The earliest work investigated the modulation of V4 neurons during tasks where spatial selective attention was directed towards or away from an item in a search array colocalized with the neuronal response field (Motter 1993, 1994). In arrays containing competing stimuli, the spiking responses of V4 neurons modulated as a function of attentional state. Specifically, attention directed to the response field of the neurons enhanced their responses. This modulation of activity was also found during memory-guided visual search where the timing of the target cue and the array onset was interrupted by a delay epoch (Chelazzi, Miller, Duncan, & Desimone 2001). Further work has since delved into the nuances of this modulation.

Ogawa and Komatsu (2004) had monkeys perform a multidimensional feature search task to investigate the role of V4 feature selectivity when the feature may or may not be relevant to the search. In this task, monkeys were shown search arrays with feature conjunctions (e.g. shape x color) and identified the oddball along one of those feature dimensions (e.g. red circle or square among green circles and squares). They found that when V4 responses were aligned to the presentation of the array, individual neurons did not reliably indicate the presence of a target in their response field. However, a shift in the population response was observed when a target was present. This suggests that while individual V4 neurons can’t discriminate behavioral targets in the initial visual response, at the population level, this can be evaluated. A similar study investigating the feature selectivity during visual search in V4 replicated this result (Mirabella, Bertini, Samengo, Kilavik, Frilli, Della Libera, & Chelazzi 2007). However, they found that individual neurons in V4 could distinguish behaviorally relevant stimuli. This discrepancy might be explained by a difference in the time epochs where selection was measured (Ogawa & Komatsu 2006). Measuring the selectivity nearer to the time of saccade show selection in individual V4 neurons in the multidimensional search task. Nonetheless, it was known from work prior to these studies that V4 modulates with feature-based attention (McClurkin & Optican 1996) and both of these studies indicate feature-based attentional selectivity in V4 during visual search. However, evidence exists this feature-based attentional selectivity is limited to goal-directed behaviors (Hayden & Gallant 2005) such as visual search (Ipata, Gee, & Goldberg 2012).

From the studies detailed above we know that FEF and V4, independently, are important for attentional selection enabling visual search. Notably, they show similar modulation of visual responses to target stimuli in visual search whereby following the initial visual response and their activity distinguishes between targets and distractors in their visual response fields (Schall & Hanes 1993; Ipata, Gee, & Goldberg 2012). Coupled with previous work demonstrating these areas are interconnected (Schall, Morel, King, & Bullier 1995; Stanton, Bruce, & Goldberg 1995; Ungerleider, Galkin, Desimone, & Gattass 2008; Ninomiya, Sawamura, Inoue, & Takada 2012), investigation has been conducted into the relationship between these areas during attention tasks. Simultaneous recordings in area V4 and FEF during an attentional task has shown that they are indeed functionally coupled, and that coupling changes with attentional state (Gregoriou, Gotts, Zhou, & Desimone 2009; Zhou & Desimone 2011; Gregoriou, Gotts, & Desimone 2012). While we do not know the impact of V4 lesioning of FEF responses during an attention task, lesioning FEF limits attention-driven modulation in V4 (Gregoriou, Rossi, Ungerleider, & Desimone 2014). Together, these studies imply a role for the FEF-V4 circuit in attention modulation and attentional selection during visual search.

To summarize, FEF and V4 have been implicated in the process of selecting behaviorally relevant stimuli during visual search. Anatomically, they have been shown to be interconnected and functionally, the activity of each of these areas affects the other. Additionally, their processing during search is not identical, indicating there are neural computations being performed between these two areas that might be important in the selection process. This circuit gives us a target to investigate how attentional selection might change through priming and as will be detailed next, we know that these areas are implicit in priming of pop-out.

Brain Areas Contributing to Priming of Pop-out: Frontal and Visual Cortex

Investigators have begun to causally affect neural populations and record activity during priming of pop-out tasks. Priming of pop-out is hypothesized to speed neural processes underpinning attentional selection of targets during visual search. Researchers have investigated this hypothesis by probing the areas of the brain highlighted above that are involved in visual selective attention.

Visual cortical areas were the first to be implicated in priming of pop-out. Walsh and colleagues investigated the roles of areas V4 and TEO in priming through a lesion study (Walsh, Le Mare, Blaimire, & Cowey 2000). To do so, they trained control monkeys and monkeys with bilateral V4 or TEO lesions to a priming of pop-out visual search task. Ultimately, they found that although baseline search performance was not impacted significantly, the effects of priming were diminished in the lesion monkeys. However, the priming effect was diminished rather than eliminated, suggesting there is not a single source for priming. No monkeys included had bilateral V4 and TEO lesions. Alternatively, priming effects could have been recovered partially following the lesion as neural plasticity can lead to recovery from deficits following non-reversible lesioning (Finger & Stein 1982; Newsome & Pare 1988; Raisman 1969; Yamasaki & Wurtz 1991), leaving it possible that V4 and TEO are the source of priming in the normal brain. Regardless, this finding suggests a role for visual cortex in priming of pop-out. Further work has since delved into that role.

We have begun to investigate the underlying change in neural processing that is occurring in area V4 related to the diminished priming of the lesion study (Westerberg, Maier, & Schall 2020). Monkeys trained to a pop-out task showed speeded attentional selection in area V4 with priming (Figure 2). This was mediated by greater enhancement in the neural responses to targets and suppression of distractors following priming as compared to the unprimed condition. This suggests the mechanism is not solely mediated by changes in target enhancement or distractor suppression, but rather a combination. Area V4 is not the only brain region investigated with respect to the attentional selection changes associated with priming of pop-out. Area FEF has also been shown to reflect changes in a very similar, yet subtly different manner. Bichot and Schall (2002) trained monkeys to perform a priming of pop-out search task and recorded the activity of neurons in FEF.

Figure 2.

Changes in neural responses associated with priming of pop-out. Neural responses from areas FEF (left) and V4 (right) when the target (black) of a search array is in the response field and when a distractor (gray) is in the response field (RF) when the search is unprimed (top) relative to primed (bottom). Data are adapted from Bichot & Schall 2002 and Westerberg, Maier, & Schall 2020. Neurons discriminate between targets and distractors at some point in time (target selection time, TST, red arrows). The target selection time occurs earlier in primed trials as compared to unprimed. Additionally, the distinguishability between targets and distractors is enhanced as indicated above by a greater difference between the target and distractor responses with priming (blue fills).

Much like the results of the V4 study, they found that visuomovement neurons in FEF showed speeded target selection with priming of pop-out mediated by target enhancement and distractor suppression (Figure 2). However, the relationship between the activity in FEF and the observed behavior was distinctly different than that same comparison between V4 and behavior. This provides some insight into the mechanism of priming of pop-out.

In comparing the change in response time associated with priming of pop-out to the change in neural target selection times, we drew inferences about the roles of FEF and V4 in priming of pop-out. In area V4, the neural target selection times speeded more than the associated behavior. This contrasts the one-to-one relationship between FEF and behavior. If anything, this highlights the role of FEF in motor output as behavior more closely follows changes in this area than changes in visual cortex (Bruce & Goldberg 1985, 1990; Bruce, Goldberg, Bushnell, & Stanton 1985; Lowe & Schall 2018).

In addition to the work done in FEF, another frontal area has been investigated with respect to its role in priming of pop-out. Investigation has been done into the role of supplementary eye field (SEF), a medial frontal cortical area involved in sensory and internally guided eye movements (Schlag & Schlag-Rey 1987, Schall 1991). Purcell and colleagues (2012) investigated whether area SEF contributes to saccade target selection during visual search and associated changes with priming of pop-out. They found no target selectivity in the visual response across their sample of SEF neurons and no change in target selectivity with priming of pop-out. This indicates no role for SEF in the selection of targets or the speeding of attentional selection during visual search. However, SEF not only has visual responses, but is implicated in other cognitive processes that can be examined with respect to priming of pop-out.

One additional study has been performed in SEF investigating a potential neural process to instantiate the change in neural selection seen in FEF. SEF has been implicated in the process of performance monitoring (Stuphorn, Taylor, & Schall 2000; Sajad, Godlove, & Schall 2019). Performance monitoring is the process by which outcomes during behavioral tasks are overseen by a “supervisory attentional system” to alter behavior on subsequent trials (Norman & Shallice 1986). Neurons in SEF signal errors during visual search (Purcell, Weigand, & Schall 2012). We hypothesized that this supervisory control could initiate the changes observed in attentional selection (Westerberg, Maier, Woodman, & Schall 2020). SEF is interconnected with area FEF (Schall, Morel, & Kaas 1993), and thus could enact changes in activity that could be related to the changes in attentional selection associated with priming of pop-out (Bichot & Schall 2002). To investigate the potential role of performance monitoring in SEF during priming of pop-out, we measured the activity of SEF neurons which showed differential activity when a correct response versus an incorrect response was made relative to the position in the priming sequence. We found that the neural activity of this performance monitoring population does not modulate with priming of pop-out.

To summarize, physiological and lesion studies of priming of pop-out have been ongoing for the last couple of decades. These studies have identified three areas, and rejected one, in contributing to priming of pop-out (Figure 3). Areas FEF (Bichot & Schall 2002), V4, and TEO (Walsh, Le Mare, Blaimire, & Cowey 2000; Westerberg, Maier, & Schall 2020) all contribute to priming of pop-out. The physiology in V4 (Westerberg, Maier, & Schall 2020) and FEF (Bichot & Schall 2002) indicates changes in the timing of attentional selection with priming of pop-out. Area SEF did not show any changes with priming of pop-out, neither in its visual processing (Purcell, Weigand, & Schall 2012) nor its performance monitoring (Westerberg, Maier, Woodman, & Schall 2020).

Figure 3.

Brain areas involved in priming of pop-out (PoP). Neurophysiological recordings in the frontal eye field (FEF) (Bichot & Schall 2002) in frontal cortex and area V4 (Westerberg, Maier, & Schall 2020) in visual cortex have shown modulation of neural activity in these areas with priming of pop-out. No modulation was found in the supplementary eye fields (SEF) during priming of pop-out (Purcell, Weigand, & Schall 2012; Westerberg, Maier, Woodman, & Schall 2020). Lesions of areas TEO and V4 reduce the magnitude of the priming effects with repetitive pop-out search (Walsh, Le Mare, Blaimire, & Cowey 2000), indicating a causal role for these areas.

Noninvasive Indicators for Neural Changes with Priming of Pop-out

The first evidence for changes in neural activation found in humans associated with priming of pop- out was through neuroimaging (Kristjánsson, Vuilleumier, Schwartz, Macaluso, & Driver 2007). The blood oxygen level dependent (BOLD) signal was measured while participants performed a color-based priming of pop-out visual search task. Changes associated with priming were found in both frontal and occipito-parietal regions of the brain, mirroring the findings described in the neurophysiology literature. Specifically, investigators observed repetition suppression of the BOLD response with priming. Notably, FEF showed changes. Changes with priming of pop-out were also observed in the area surrounding the fusiform gyrus (FG), nearby to the location where the human homologue of macaque area V4 (human V4, hV4) resides (Lueck, Zeki, Friston, Deiber, Cope, Cunningham 1989; McKeefry & Zeki 1997). Additionally, changes were observed in the intraparietal sulcus (IPS), an area where neurophysiology has not been performed.

What is repetition suppression and how does it tell us anything about the neural mechanism of priming of pop-out? Repetition suppression is the diminishing response that is often associated with repetition of visual stimuli (Grill-Spector & Malach 2001). Not limited to BOLD responses, it can also be observed in EEG (Sambeth, Maes, Quian Quiroga, & Coenen 2004) and electrocorticography (ECOG) (Puce, Allison, & McCarthy 1999) as well as responses at the single neuron (Baylis & Rolls 1987) and neural population levels (Brunet, Bosman, Vinck, Roberts, Oostenveld, Desimone, De Weerd, & Fries 2014). Originally described in the inferotemporal cortex when complex (multidimensional) stimuli were presented repeatedly (Baylis & Rolls 1987; see also Gross, Schiller, Wells, & Gerstein 1967), repetition suppression has since been described across many visual cortical areas. For example, V4 shows repetition suppression of gamma responses (Brunet, Bosman, Vinck, Roberts, Oostenveld, Desimone, De Weerd, & Fries 2014). Repetition suppression can be observed also at the earliest stage of visual cortical processing in V1 and is hypothesized to come about due to changes in feedback activation (Westerberg, Cox, Dougherty, & Maier 2019). Furthermore, repetition suppression is hypothesized to come about through more efficient encoding of visual stimuli, perhaps through predictive coding (Summerfield, Trittschuh, Monti, Musalem, & Egner 2008; Aukztulewicz & Friston 2016). However, recent work suggests changes in encoding efficiency and the repetition suppression reflect different processes (Tang, Smout, Arabzadeh, & Mattingley 2018). Regardless, repetition suppression is associated with changes in the processing of visual stimuli, seemingly in a top-down fashion and in order to more efficiently process stimuli. In relating these properties to repetition suppression in priming of pop-out, this would suggest that priming results in more efficient processing of visual stimuli perhaps through top-down modulation of visual processing.

Following the work in humans (Kristjánsson, Vuilleumier, Schwartz, Macaluso, & Driver 2007) and macaques (Bichot & Schall 2002) identifying FEF as a relevant area in priming of pop-out, investigators sought to understand whether TMS of FEF in humans altered behavioral performance in the task (O’Shea, Muggleton, Cowey, & Walsh 2007). While stimulation of FEF during the intertrial interval had no effect on behavioral performance (Taylor, Muggleton, Kalla, Walsh, & Eimer 2011), stimulation during the period of array presentation had mixed effects. Stimulation during feature (color) priming blocks had no effect on behavioral performance. However, stimulation during spatial priming blocks did affect behavioral performance, but only stimulation of left FEF. These results suggest that FEF is not the source of priming for features but is important in the priming of spatial location. Coupled with the lesion studies of attention in macaques (Rossi, Bichot, Desimone, & Ungerleider 2007; Gregoriou, Rossi, Ungerleider, Desimone 2014), including priming of pop-out (Walsh, Le Mare, Blaimire, & Cowey 2000), these findings might suggest a source for feature priming in visual cortex and a source for spatial priming in frontal cortex.

EEG in humans has also been a powerful noninvasive tool in investigating priming of pop-out. Visual selective attention has a robust electrophysiological index known as the N2pc (Eimer 1996; Luck & Hillyard 1990, 1994; Woodman & Luck 1999). The N2pc is an event-related potential (ERP) measured in the EEG that indexes whether selective spatial attention has been allocated contra- or ipsilaterally. Part of the history of the N2pc involved investigating how it changes with temporal manipulation of attentional selection. Cuing the location of an attentional target elicits an earlier N2pc than when the target position remains non-cued (Foster, Bsales, & Awh 2020). Priming of pop-out also affects the timing of the N2pc. Eimer and colleagues found the N2pc speeds with priming of pop-out (Eimer, Kiss, & Cheung 2010). This was the first noninvasive electrophysiological evidence for a temporal change in attentional selection associated with priming of pop-out. More recent work has investigated the timing of attentional selection in visual search by decoding the raw EEG (Ort, Fahrenfort, ten Cate, Eimer, & Olivers 2019). In a visual search task where the target remains the same trial-to-trial versus trials where the targets switches, they found that evidence accumulation for the target was equivalent, but the onset of that accumulation was delayed for switches. This finding complements that of the earlier N2pc findings by showing changes in temporal processing of visual search with priming through noninvasive, high temporal resolution EEG. However, how these changes map to those observed through invasive electrophysiology remains an open question.

Other non-attention-related EEG ERPs have since been shown to change with priming of pop-out. Investigators have found event-related potentials in the EEG associated with performance monitoring (error- related negativity [ERN], Gehring, Goss, Coles, Meyer, & Donchin 1993; error positivity [Pe], Falkenstein, Hohnsbein, Hoormann, & Blanke 1991) did modulate with priming of pop-out (Westerberg, Maier, Woodman, & Schall 2020). While this may suggest changes associated with priming of pop-out manifest as a result of performance monitoring, alternatively, the observed changes may be related to a surprise signal. That is, a change in the target feature during priming of pop-out could generate a greater surprise signal in the EEG. Recall, neurophysiological recordings in area SEF, a known contributor to the ERN (Sajad, Godlove, & Schall 2019), did not show any changes at the neural level (Westerberg, Maier, Woodman, & Schall 2020).

Reconciling the fMRI Repetition Suppression with the Neurophysiology

fMRI findings indicate that the BOLD response in brain areas implicated in priming of pop-out (FEF, FG, IPS) show repetition suppression (Kristjánsson, Vuilleumier, Schwartz, Macaluso, & Driver 2007). However, the neurophysiology results indicate a mixture of enhanced (when the target is in the response field) and suppressed (when a distractor is in the response field) responses (Bichot & Schall 2002; Westerberg, Maier, & Schall 2020). Why might there be this dissociation between the measures?

One hypothesis suggests that the changes observed during priming of pop-out in the neurophysiology, namely the enhanced processing of targets and suppression of distractors, lead to overall enhanced processing efficiency at the mesoscopic scale. That is, while the processing of targets is indeed enhanced, seemingly opposite to the fMRI finding, that only applies to the retinotopic region of cortex representing the target stimulus. It should follow that the remainder of the same retinotopic map (i.e. the remainder of the V4 or FEF map of visual space) has overall less neural activation. Therefore, on the whole, the activation across one retinotopic map will be less for the efficient primed case as opposed to the relatively inefficient unprimed case. At this scale (swaths of cortex rather than individual neurons) the neural activation would be less with priming. This change in population dynamics would be congruent with the fMRI finding.

An alternative hypothesis explains the dissociation through distinction between the signal generators. It may be that the BOLD measure is not a good indicator of activity at the neural level. BOLD is a measure of change in blood flow rather than the electrical activity of neural populations. Additionally, the time course of the BOLD signal is an order of magnitude slower than neurophysiological responses. Therefore, BOLD may not be an accurate indicator of activity at the single-unit level. Indeed, this may be the case as fMRI seems to be more tightly correlated to fluctuation in the local field potentials (LFP) (Logothetis & Wandell 2004). This would suggest that fMRI and single-unit results need not parallel each other. However, it would suggest that changes in the LFP might manifest as diminished responses with priming. It is important to note that this may not be the case as again, LFP does not cause the BOLD, it merely correlates better than other neural measures. All in all, there are several alternative hypotheses as to why we see the dissociation between the neurophysiology and the functional imaging. Therefore, these results need not be seen as opposing, just as different indices that provide distinct insight into the neural mechanism.

Relating N2pc and Neurophysiology

The speeding of attentional selection in V4 (Westerberg, Maier, & Schall 2020) parallels the finding of the speeding of the N2pc (Eimer, Kiss, & Cheung 2010). Since its discovery, the N2pc has been described as having greatest magnitude over the posterior brain (Luck & Hillyard 1994). It was therefore hypothesized that its neural source lies in occipital or parietal cortex. Evidence from magnetoencephalography (MEG) supports this hypothesis (Hopf et al. 2000). Given that the N2pc is largest over occipital and parietal cortex, V4 lies in occipital cortex, and both V4 and the N2pc show evidence of speeding of attentional selection with priming of pop-out, the logical hypothesis is that V4 contributes to the generation of the N2pc. This hypothesis can be tested in the macaque monkey.

Woodman and colleagues (2007) demonstrated that the monkeys performing visual search show a homologue of the human N2pc (see Woodman 2012 for review). The N2pc has been found across several search tasks and across monkeys (Cohen, Heitz, Schall, & Woodman 2009; Cosman, Lowe, Woodman, & Schall 2018; Heitz, Cohen, Woodman, & Schall 2010; Purcell, Schall, & Woodman 2013). The N2pc homologue allows for the investigation of the neural origins for the N2pc through concurrent EEG and invasive neurophysiological recordings that would otherwise be off-limits in human participants. Examining the relationship between V4 and the N2pc during a priming of pop-out task would provide further evidence for or against V4 being a neural contributor to the N2pc. If V4 contributes to the N2pc, the changes in attentional selection observed in V4 during priming should be consistent with the changes in the N2pc.

How does FEF contribute to the generation of the N2pc? FEF activity changes with priming of pop-out (Bichot & Schall 2002). Additionally, the time course of attentional selection measured in FEF and the N2pc correlates well and perhaps suggests a causal relationship (Cohen, Heitz, Schall, & Woodman 2009; Purcell, Schall, & Woodman 2013). Are the changes in FEF secondary to those in V4? Or, are FEF projections to V4 driving this change? The latter would suggest that although V4 is generating the N2pc signal, FEF may actual be the signal source. Further investigation through concurrent EEG and neurophysiological recordings of FEF and V4 is necessary to answer these questions. This too may provide insight into the source of the priming changes and how those changes flow through attention networks to produce the speeded neural and behavioral selection measures.

Neural Mechanism for Priming of Pop-out: Three Hypotheses

Physiological, imaging, and lesion studies have identified areas in the brain which contribute to or are affected by priming of pop-out. However, those identifications are not enough to generate a complete hypothesis regarding the neural mechanism. It is also important to know the primary source of change and what is secondary. For example, changes may come about in visual cortex which are inherited in a feedforward, or bottom-up, manner by frontal cortex, or the changes arise in frontal cortex and are fed back to visual cortex in a top-down manner (Kinchla & Wolfe 1979). Here we propose three hypotheses to explain the temporal relationships measured between neural selection and response times across studies of priming of pop-out (Figure 4).

Figure 4.

Predictions regarding the relationship between neural and behavioral selection times for each mechanistic hypothesis. A. Relationship between the behavioral response time and neural target selection time found for FEF (top, Bichot & Schall 2002) and V4 (bottom, Westerberg, Maier, & Schall 2020). B. Bottleneck hypothesis: V4 identifies the target which is fed forward to FEF. Visual cortex is the source of priming. FEF accumulates the information from V4 but is not as efficient as the V4 priming. C. Feedback hypothesis: FEF identifies the target location and feeds back that information to V4. Frontal cortex is the source of priming. FEF primes and the magnitude of the priming effect is amplified when fed back to V4. D. Selection tradeoff hypothesis: FEF identifies the target position in unprimed trials and V4 on primed. The source of priming is not localized to one area and arises in the interaction between areas.

Bottom-up Mechanism

The automatic nature of priming of pop-out suggests a bottom-up mechanism (Maljkovic & Nakayama 1994), consistent with the automaticity of feedforward visual processing (Kinchla & Wolfe 1979). Psychophysical models assert lower-order feature extraction is integral to pop-out visual search (Treisman & Gelade 1980; Wolfe, Cave, & Franzel 1989; Cave & Wolfe 1990; Cave 1999). Feature contrasts across space can generate salience maps (Koch & Ullman 1985; Itti & Koch 2000, 2001) which are biologically feasible (Tsotsos, Culhane, Wai, Davis, & Nuflo 1995; Mazer & Gallant 2003; Thompson & Bichot 2005; Hopf, Boehler, Luck, Tsotsos, Heinze, & Schoenfeld 2006; Boehler, Tsotsos, Schoenfeld, Heinze, & Hopf 2009, 2011; Hopf, Boehler, Schoenfeld, Heinze, & Tsotsos 2010; Bruce, Wloka, Frosst, Rahman, & Tsotsos 2015). Priming of bottom-up feature extraction or salience map generation would manifest as V4 selecting the search target and also the source of priming. These would be fed forward to FEF. This seems plausible as the axons of many V4 neurons terminate in FEF (Schall, Morel, King, Bullier 1995; Stanton, Bruce, & Goldberg 1995; Ungerleider, Galkin, Desimone, & Gattass 2008) and physiology during attentional processing supports this (Gregoriou, Gotts, & Desimone 2012; Gregoriou, Gotts, Zhou, & Desimone 2009), importantly during visual search (Zhou & Desimone 2011). V4 has also been shown to be sensitive to pop-out stimuli in the absence of controlled attention (Burrows & Moore 2009). It is worth noting that sensitivity disappeared when selective attention was directed elsewhere, suggesting the salience of the pop-out item alone is not sufficient. Even so, V4 seems sensitive to pop-out stimuli, diminishes priming when lesioned, and is affected by priming as measured through the changes in neural activation.

However, the change in neural selection time as a function of priming in V4 is about twice the magnitude of the change in RT. As noted above, FEF is closer to the behavioral output through connections to the brainstem saccade generator. Thus, the incongruencies in timing measures can arise through FEF acting as a bottleneck to the overall priming effect through its accumulation of information from various visual inputs to generate responses (Purcell, Heitz, Cohen, Schall, Logan, & Palmeri 2010; Purcell, Schall, Logan, & Palmeri 2012). Eriksen and colleagues also presented evidence investigating the relationship of evidence accumulation and response preparation (Coles, Gratton, Bashore, Eriksen, & Donchin 1985; Gratton, Coles, Sirevaag, Eriksen, & Donchin 1988). Priming of pop-out has not been investigated from the perspective of evidence-accumulation and perceptual decision-making, nonetheless it seems a fruitful avenue to evaluate the validity of the ‘bottom-up’ hypothesis. Still, it is not the only framework in which to consider this hypothesis. The bottleneck could also be a result of the stimulus-response mapping necessary for accurate behavior. That is, while the speeding of selection might be greatest in earlier sensory areas, the translation of the visual information into the motor selection signal can be a distinct process that speeds at a different rate. Human psychophysical evidence shows that response mapping contributes significantly to priming of pop-out (Meeter & Olivers 2006; Olivers & Meeter 2006). Stimulus-response mapping and continuous evidence accumulation need not be considered mutually exclusive as models of perceptual decision-making are compatible with both (Purcell, Heitz, Cohen, Schall, Logan, & Palmeri 2010; Purcell, Schall, Logan, & Palmeri 2012) and neurophysiology supports such compatibility (Bichot, Rao, & Schall 2001). Regardless of the relationship between stimulus-response mapping and evidence accumulation, evidence supports a bottom-up mechanism, albeit further neurophysiological research should investigate the biologically feasibility of such bottlenecks.

When considering a bottom-up mechanism, it is also interesting to speculate about functional localization of feature processing in visual cortex. That is, different brain areas in visual cortex tend to represent different visual features such as color, shape, motion, or orientation, among others. From the human psychophysical work, we know that pop-out occurs with features other than shape and color and judging the presence of multiple features occurs more quickly than judging the presence of a single feature (Fournier, Eriksen, & Bowd 1998). While an area like V4 is ideal for the investigation of priming with respect to colors or shapes (Roe, Chelazzi, Connor, Conway, Fujita, Gallant, Lu, & Vanduffel 2012), it might not be ideal for motion, for example. Perhaps, priming of spatial frequency could occur in V1 or V2 (Foster, Gaska, Nagler, & Pollen 1985), priming of color or shape in V4 or TEO (Zeki 1973; Tootell, Nelissen, Vanduffel, & Orban 2004; Conway, Moeller, & Tsao 2007; Conway & Tsao 2009), and priming of motion in MT (Mikami, Newsome & Wurtz 1986a, 1986b; Newsome & Pare 1988). This might suggest that the process of priming, if instantiated early in visual processing through a bottom-up mechanism, might be distributed across areas. It would potentially explain the compounded priming effect found in the human psychophysics where changes in RT are greater when multiple features are primed (Kristjánsson 2006, 2009). To investigate this possibility, one could employ the same techniques of prior studies of priming of pop-out on data collected from V4 using a feature it does not strongly represent, like motion. Complementarily, recording from area MT during a motion priming search task (Kristjánsson 2009) would also shed light on this question. If V4 does not show changes associated with motion priming of pop-out while MT does, this might suggest a common bottom-up cortical mechanism for changes in attentional selection with priming. However, it could suggest that the feedback from frontal cortex is highly specific. We know that feedback pathways from FEF to V4 and MT are distinct (Ninomiya, Sawamura, Inoue, & Takada 2012). Should both V4 and MT show changes associated with priming, it would be indicative of a diffuse top-down mechanism, which would be evidence for our next hypothesis.

Top-down Mechanism

An alternative hypothesis predicts FEF neural selection of the target always precedes V4 neural selection. This comes about if FEF selects the target and delivers that information to V4. Therefore, a change in selection time in FEF, elicited through priming or any other similar manipulation, would lead to a change in the timing of selection in V4. FEF is considered an attentional control structure, thereby coordinating attentional selection across brain areas. Anatomical work has demonstrated dedicated projection pathways from FEF to V4, as projections from FEF to area MT were distinct from those to V4 (Ninomiya, Sawamura, Inoue, & Takada 2012) and different neurons projecting to V4 and to superior colliculus (Pouget et al. 2009). Additionally, previous work has demonstrated that electrical stimulation FEF can exert an attention-like modulation on V4 (Moore & Armstrong 2003; Armstrong, Fitzgerald, & Moore 2006; Armstrong & Moore 2007) and physiological work reaffirms this (Gregoriou, Gotts, Zhou, & Desimone 2009; Zhou & Desimone 2011; Gregoriou, Gotts, & Desimone 2012). Furthermore, lesions to prefrontal cortex diminish attentional modulation in V4 (Rossi, Bichot, Desimone, & Ungerleider 2007; Gregoriou, Rossi, Ungerleider, & Desimone 2014). Together, these findings demonstrate the capacity for a top-down mechanism which lends support for a feedback hypothesis for priming of pop-out.

Further evidence for a top-down mechanism can perhaps be drawn from the processing of distractors in frontal cortex. Previous work has identified frontal cortex, specifically FEF, as a source for the processing of distractor stimuli in the brain (Cosman, Lowe, Woodman, & Schall 2018). We know that a primed target feature becoming the distractor on subsequent trials will lead to slowed responses on those trials (Kristjánsson & Driver 2008). This might result from the now distractor feature being facilitated (e.g. negative priming). That is, the distractor feature that has previously been ignored has become the target, leading to slowed responses to that target (Dalrymple-Alford & Budayr 1966; Greenwald 1972; see Milliken, Joordens, Merikle, & Seiffert 1998 for review). With FEF as the source for distractor processing, it is likely that changes in that processing also come about in FEF which could be taken as evidence for FEF as the source thereby implying it as the source of changes associated with priming of pop-out. However, this is speculative. In the neurophysiological studies of FEF and V4, target and distractor features were not independent. When a target feature became a distractor, the distractor became the target. There were instances where both target and distractor changed in the study of FEF (Bichot & Schall 2002), however comparisons were not made between those switches and the target-distractor swaps. This makes direct evaluation of the independence target and distractor processing changes impossible with current neurophysiological reports. It is worth mentioning, both enhancement of neural responses to targets and suppression of neural responses to distractors were observed in V4 and FEF (Bichot & Schall 2002; Westerberg, Maier & Schall 2020) hinting at perhaps the existence of multiple mechanisms. However, psychophysical evidence exists indicating priming affects processing of targets and distractors independently (Kristjánsson & Driver 2008). To summarize, psychophysics indicates distinct processes instantiating changes in target and distractor processing and neurophysiology indicates distractor processing is instantiated in frontal cortex. Together this suggests changes in distractor processing with priming of pop-out might originate in frontal cortex and the findings in visual cortex are secondary.

Selection Tradeoff Hypothesis

The final hypothesis to consider is a switch in the brain area that first selects the target as a function of priming. In the context of FEF and V4, the could manifest as frontal cortex selecting the target first in trials early in the priming sequence, when attentional demand is perhaps greatest and visual cortex selecting first once the target feature has been established (e.g. later in the priming sequence). The rationale behind the switch could also be that frontal cortex must override a sort of attentional selection template that is driving selection in visual cortex when the search conditions change, thus necessitating earlier selection in frontal cortex earlier in the priming sequence which could then switch once the template is established. This would also potentially explain the higher error rate early in the priming sequence (Bichot & Schall 2002; Westerberg, Maier, & Schall 2020) as perhaps frontal cortex does not inhibit the selection of visual cortex for the incorrect item quickly enough. Regardless, the most general assertion of this hypothesis is that attentional selection of the target first occurs in different brain areas depending on the priming state. Neurophysiologically, this would be observed as a crossover in selection times between FEF and V4 as a function of priming. The selection times for area V4 seem to speed roughly twice as much as the selection times in FEF when comparing across studies (Bichot & Schall 2002; Westerberg, Maier, & Schall 2020). Therefore, it is plausible that the selection first occurs in FEF in the unprimed state but occurs first in V4 following priming. While perhaps a bit more abstract than the other hypotheses, this could be interpreted as persistent top-down modulation of visual processing. In other words, frontal cortex identifies novel features in the visual environment which are useful for directing behavior. Once that feature is identified (e.g. the first trial in the priming sequence), frontal cortex imposes modulatory activity on visual cortex to change processing in such a way that it promotes the activation in response to that feature (e.g. trials later in the priming sequence). While such a change did not manifest as modulation of baseline activity in V4 (Westerberg, Maier, & Schall 2020), there are a number of other mechanisms through which top-down control could affect visual processing. Oscillatory activity could change with priming through modifications in the power of certain frequency bands related to attention, such as alpha (see Klimesch 2012 for review), in cortical regions representing the feature of interest. An alternative might be through phase-resetting of neural oscillations which have shown the capacity to coordinate large-scale networks during attentional processing (Voloh & Womelsdorf 2016). These are only a couple of possibilities for the adaptations that might exist to generate the selection tradeoff. This hypothesis is perhaps the least likely given the complexity of the changes that would be necessary to elicit the priming effect. Nonetheless, it is important to recognize its feasibility.

Which of these hypotheses is correct? When considering the evidence from the lesion study where priming diminished with lesions of visual cortical areas (Walsh, Le Mare, Blaimire, & Cowey 2000), it seems the source of priming lies prior to FEF. FEF cannot be the sole source of priming as it is nearest to the production of the behavior and lesioning areas V4 or TEO attenuate priming. This would suggest that the feedforward bottleneck or selection tradeoff is more likely the neural mechanism than feedback amplification. The sensitivity of area V4 to pop-out stimuli even in the absence of selective attention might provide further evidence for the feedforward bottleneck hypothesis (Burrows & Moore 2009). Furthermore, the evidence from human TMS studies suggests FEF is not the source for these feature-based priming effects (Taylor, Muggleton, Kalla, Walsh, & Eimer 2011). It may also be the case that the underlying mechanism is even something different than what is proposed here as the evidence is limited to only several studies. Additionally, we are only considering the effect of priming of spike-rate changes. It may also be the case that changes associated with priming of pop-out are driven through changes in multivariate representations of stimulus features beyond simple spike-rates (Goddard, Solomon, & Carlson 2017; Tovar, Westerberg, Cox, Dougherty, Carlson, Wallace, & Maier 2019) or through changes in synchrony between frontal and visual cortical areas (Gregoriou, Gotts, & Desimone 2012; Gregoriou, Gotts, Zhou, & Desimone 2009).

Conclusions

We set out to review the neural mechanism at the heart of priming of pop-out and provide insight into how the brain automatically adjusts visual selective attention, primarily with respect to the timing of selection. The dynamics of timing in attentional selection during visual search stems from the work of Eriksen who first reported how differences in the composition of visual arrays influences the speed to which we can search. These fundamental observations led to studies of how search times change with search history. We introduced the psychophysics of priming of pop-out and the changes that have been observed in neural activity associated with priming of pop-out to capture a perspective on where visual search investigation has gone in the time since Eriksen’s early work. This included review of studies in functional imaging, human electrophysiology, and macaque neurophysiology, among others. We discussed the regions of the brain implicated in the production of the priming effect including areas FEF and V4. While work up to this point has identified potential sources for priming of selective attention, further work is needed for a complete understanding of the neural mechanism. For one, a gap exists in the breadth of brain areas investigated with respect to attentional priming. Areas like parietal cortex, superior colliculus, and visual pulvinar are highly involved in attention and have not been investigated at the neural level, just to name a few. Perhaps the most notable gap is how priming flows through the attentional processing brain structures. To that end, we proposed three rival hypotheses regarding the neural pathway supporting changes in attentional selection. Specifically, these hypotheses suggest changes in attentional selection are driven: (1) bottom-up and therefor instantiated in the early visual processing pathway and inherited by frontal attentional control structures, (2) top-down from attentional control structures to earlier visual areas, or (3) interactively whereby attentional control structures recognize switches in targets and adjust attentional selection before earlier visual areas take control once a pattern is established. These represent potential avenues for future investigation into the neural basis of priming of pop-out which in turn will inform visual selective attention more generally.