Cortical Mechanisms of Prioritizing Selection for Rejection in Visual Search

Abstract

In visual search, the more one knows about a target, the faster one can find it. Surprisingly, target identification is also faster with knowledge about distractor-features. The latter is paradoxical, as it implies that to avoid the selection of an item, the item must somehow be selected to some degree. This conundrum has been termed the “ignoring paradox”, and, to date, little is known about how the brain resolves it. Here, in data from four experiments using neuromagnetic brain recordings in male and female humans, we provide evidence that this paradox is resolved by giving distracting information priority in cortical processing. This attentional priority to distractors manifests as an enhanced early neuromagnetic index, which occurs before target-related processing, and regardless of distractor predictability. It is most pronounced on trials for which a response rapidly occurred, and is followed by a suppression of the distracting information. These observations together suggest that in visual search items cannot be ignored without first being selected.

SIGNIFICANCE STATEMENT How can we ignore distracting stimuli in our environment? To do this successfully, a logical hypothesis is that as few neural resources as possible should be devoted to distractor processing. Yet, to avoid devoting resources to a distractor, the brain must somehow mark what to avoid; this is a philosophical problem, which has been termed the “ignoring paradox” or “white bear phenomenon”. Here, we use MEG recordings to determine how the human brain resolves this paradox. Our data show that distractors are not only processed, they are given temporal priority, with the brain building a robust representation of the to-be-ignored items. Thus, successful suppression of distractors can only be achieved if distractors are first strongly neurally represented.

Introduction

The frequently-performed task of visual search requires us to localize an item (the target) based on knowledge about some of its properties. The more we know about these properties, the faster we are to identify the target (e.g., searching for a friend in a crowd is made easier if we know that this friend is wearing a red hat). Interestingly, preknowledge about irrelevant items (distractors) can also significantly facilitate target selection (Müller et al., 1995; Found and Müller, 1996; Watson and Humphreys, 1997, 2000; Olivers and Humphreys, 2003; Braithwaite et al., 2005; Arita et al., 2012). How knowing what to ignore can help with target identification is, however, paradoxical and has been described in other contexts as the “white bear phenomenon” (Wegner et al., 1987; Wegner, 1994). Specifically, the attempt to not think about something necessarily implies the act of thinking about it. Put in the context of visual search: to avoid being distracted by irrelevant items, it is beneficial to select them in some manner, as it is impossible to ignore something without knowing what to ignore (i.e., the “ignoring paradox”; Moher and Egeth, 2012). Yet the degree to which this selection occurs neurally, and the ultimate resolution of this paradox in the brain, are unknown. Theoretically, there are at least two principal solutions to this problem. One possibility is that preknowledge about the distractor may serve to reduce the cortical response to features defining the distractor, which, in turn, instantiates a preset bias for the target, giving the target features a competitive edge. Alternatively, the brain may rapidly build a representation of the distractor for optimizing the neural processes that eliminate its negative influence on target identification. Both solutions make opposite predictions regarding measurable indices of attentional selection in visual cortex. The former would lead to a largely reduced or eliminated response to the distractor, whereas the latter would facilitate the cortical response indexing distractor selection.

Here, we investigate the dynamics of cortical processes underlying target and distractor selection in visual search with the highest possible spatiotemporal resolution using event-related magnetic field (ERMF) recordings. Subjects search for a color-defined pop-out (target) among task-irrelevant distractors, one of which is also a color pop-out (Fig. 1a). Adding a distractor singleton in the same feature dimension (color) as the target allows us to assess and compare brain activity reflecting the selection of the target and the distractor while matching their salience (color contrast). Specifically, we examined the N2pc and Pd: two well-characterized lateralized electrophysiological components, indexing target and distractor processing, respectively (for review, see Luck, 2011). In a series of experiments we manipulate the prominence and predictability of the color of the distractor while keeping target-defining experimental parameters constant. We find that the selection of the distractor, indexed by an early lateralized component resembling the N2pc (termed here the N1pc), has temporal priority relative to the selection of the target. This priority, which is independent of whether the distractor color is predictable from trial to trial, speeds distractor rejection and subsequent target selection. The present observations together indicate that the brain resolves the ignoring paradox in visual search by a temporal prioritization of distractor selection, which facilitates the rejection of the distractor thereby facilitating overall search performance.

Figure 1.

Stimulus setup and structure of trial-block sequence. a, Example search frames in Experiment 1. Each search frame contained four neutral distractors (blue Ts), one target (red T) and one distractor item (green T). To isolate the target-related magnetic response (top row), the distractor was placed at the vertical meridian position with the target appearing at one of the two positions in the left (left) or the right visual field (right). The distractor-related response (bottom row) was isolated analogously way by placing the distractor at lateral positions, with the target appearing on the vertical meridian. The tERL or dERL was derived by subtracting the response to right visual field items from left visual field items; a subtraction that nulls the magnetic response elicited by the item placed on the vertical meridian. b, Illustration of trial-block structure and color assignment to the search items in Experiment 1–4. Each of the colored bars indicates the color assigned to the target (T) and distractor (D) on a given trial. The four neutral distractors were always blue in all experiments, and, except for Experiment 3, the task was always to determine the orientation of the target “T”.

Materials and Methods

Subjects

Twenty-four healthy volunteers participated in Experiment 1 (1 left-handed, 10 male, age range 22–34 years, mean = 25.4 years), 24 volunteers participated in Experiment 2 (1 left-handed, 16 male, age range 22–36 years, mean = 27.6 years), 20 volunteers participated in Experiment 3 (1 left-handed, 9 male, age range 21–36 years, mean = 27.3 years), and 20 volunteers participated in Experiment 4 (1 left-handed, 13 male, age range 20–35 years, mean = 27.9 years). Five (Experiment 1), eight (Experiment 2), seven (Experiment 3), and five (Experiment 4) additional participants were excluded due to excessive eye movements/physiological noise in their electrophysiological data, poor accuracy, or excessive head movement. Participants were financially compensated for their time and all gave written, informed consent before participation. All methods and procedures were approved by the ethics committee at the Otto-von-Guericke University of Magdeburg.

Experimental design and statistical analysis

General experimental design

A challenge of identifying neural activity underlying target and distractor selection in visual search is that both processes run in parallel, which results in substantial overlap of the cortical activity modulations. To be able to separate target- and distractor-related activity, the series of experiments reported here exploit advantages of asymmetric search displays previously used to differentiate between hemisphere-lateralized ERP components indexing target and distractor selection in visual search (Woodman and Luck, 2003; Hickey et al., 2009; Hilimire et al., 2012). In asymmetric search arrays, as illustrated in Figure 1a, either the distractor (green T, top row) or the target (red T, top row) is placed on the vertical meridian, whereas the target or the distractor simultaneously appears at a lateralized position, respectively. This stimulus configuration allows for the isolation of the event-related response exclusively reflecting the lateralized item (event-related lateralization [ERL]). For instance, subtracting the brain response to search frames with the distractor in the right hemifield from the response to search frames with the distractor in the left hemifield (Fig. 1a, bottom row), isolates the distractor-related brain response elicited in the hemisphere contralateral to the distractor (distractor-related ERL [dERL]), but nulls the response elicited by the target because the target appears in both cases at the vertical meridian position, where it does not elicit a hemisphere-lateralized brain response. The same logic applies when the setup is reversed, which isolates the target-related brain response (Fig. 1a, top row). That is, the target is laterally positioned and the distractor is placed on the vertical meridian [target-related ERL (tERL)]. Of note, due to the nature of the subtraction, the hemisphere-lateralized response elicited by left and right visual field items is combined. To simplify data presentation and analysis, we will always collapse left and right VF responses, which represent the average response contralateral to the item. Using this subtraction logic we are able to derive ERL indexing target and distractor selection in isolation. One such component is the N2pc: a hemisphere-lateralized modulation indexing target selection in visual search (Luck and Hillyard, 1994a; Eimer, 1996; Luck et al., 1997; Woodman and Luck, 1999; Hopf et al., 2000, 2006; Kiss et al., 2008a; Woodman et al., 2009; for review, see Luck, 2011). Another modulation is the Pd: a hemisphere-lateralized response contralateral to conspicuous (pop-out) distractors, which is known to reflect the suppression of distractor information (Hickey et al., 2009; Sawaki and Luck, 2010, 2011; Hilimire et al., 2012; Kiss et al., 2012; Sawaki et al., 2012; Jannati et al., 2013; McDonald et al., 2013; Gaspar and McDonald, 2014). A crucial feature of the present experimental design is that both the target and the distractor pop-out in the same feature dimension (color).

Stimuli and task

Experiment 1

Each trial consisted of a display of six letters (all T's at various orientations), presented on a gray background (Fig. 1a). Within each display, four of the letters were blue, one letter was red, and one letter was green. The stimuli were arranged in a circle, such that the top letter and bottom letter were positioned along the vertical meridian. The stimuli were equidistant from central fixation at a visual angle of 7.3°, and each “T” subtended 3.4° of visual angle. The stimuli were also equidistant from each other, occupying six places in the circle. In a given trial, the search display was presented for 300 ms, followed by 1100–1600 ms of jittered fixation. In total, 2400 trials were presented across the experiment.

In each search array, the red and green letters served as color pop-outs, and they were equated for luminance before the start of the experiment. In a given block, participants were instructed to attend to a particular color (e.g., red) and to determine whether the T presented in that color was pointing to the left or to the right. The attended color was switched between red and green on every block, and the order of blocks was randomized and counterbalanced across participants. Thus, in every block, one color served as the target (e.g., red), and the other pop-out color (e.g., green) served as a distractor (Fig. 1b, top row). The target and distractor stimuli were arranged in the search display such that on a given trial, the target could be at a vertical meridian and the distractor could be at one of the lateral positions, or the distractor could be at the vertical meridian and the target could be at one of the lateral positions, or both target and distractor could be at one of the lateral positions (always in opposite visual fields, in that case). The final distribution of stimulus positions across trials was as follows: 40% of the trials contained a target centrally presented and a distractor laterally presented, 40% of trials contained a distractor centrally presented and a target laterally presented, and 20% of trials contained a target laterally presented and a distractor laterally presented. Within these limitations, the positions varied randomly, with equal number of trials at each possible configuration. Of note, only those trials in which either the target or distractor appeared at the meridian were considered for further analysis.

To ensure that they remained in position and maintained fixation, participants were monitored through the use of a closed-circuit video camera and given verbal feedback on both after each block. Additionally, participants practiced the task before the start of the experiment to ensure they were familiar with the stimulus configurations and response mapping. Accuracy and response time (RT) were recorded, and trials were only considered for behavioral and neural analysis if they contained a response that fell between 200 and 1000 ms after the onset of the search display.

Experiment 2

The stimuli in Experiment 2 were the same as used in Experiment 1, except this time the pop-out distractor was a color that never served as target color (Fig. 1b, middle row). The distractor colors were magenta, gray, brown, and cyan, while the target colors were still green and red. The experiment had eight blocks (e.g., 1 block would be: red target, gray distractor). The order of the blocks was randomized and counterbalanced across subjects. All of the other parameters were the same as in the first experiment.

Experiment 3

Stimuli, the stimulus presentation protocol, and the blockwise (per run) assignment of color to the target and distractor were the same as in Experiment 2 (Fig. 1b, middle row), with one exception. Temporally unrelated to the trial-by-trial presentation of the search arrays, a rapid serial stream of characters (white in color) was presented in the center of the screen (at fixation). The duration of each character was 100 ms. The characters presented were predominantly letters; however, on occasion, a target character (#) was presented. The participants' task was to count how many times the target character appeared, and at the end of a short block (7 blocks per run, each block lasting ∼1 min), the participants were asked, via multiple choice, how many #'s were presented in that block. The participants were given four response options, including one “none of the above” choice, and had to respond, via button press, as to which option was correct. Participants were instructed throughout the duration of the experiment to ignore the array of T's and to only pay attention to the letter stream.

Experiment 4

Stimuli and Task.

The stimuli in Experiment 4 used the same target colors of red and green as in Experiments 1 and 2, but this time the distractor color varied randomly. Specifically, the pop-out distractor could either be yellow, brown, light gray, white, turquoise, or purple on any given trial (Fig. 1b, bottom row). Aside from the randomly-chosen distractor color, all of the parameters used were the same as in Experiment 1.

Data acquisition

For all experiments, the EEG and MEG data were simultaneously recorded while participants were seated in an electrically-shielded chamber. EEG data were continuously recorded using a Neuroscan system (RRID:SCR_015818) and a 32-channel cap (Easycap) referenced online to the right mastoid. One electrode was placed below the right eye, and one to the right of the right eye and another to the left of the left eye, serving as the VEOG and HEOG channels, respectively. All impedances were kept <5 kΩ during recording. The MEG data of Experiments 1, 2, and 4 were recorded with 248 magnetometers using a BTI Magnes 3600 whole-head system (Neuroimaging). The data of Experiment 3 were recorded with an Elekta Neuromag TRIUX triple-sensor system (102 magnetometers). On the BTI system the MEG and EEG data were sampled at 508 Hz, and were bandpass filtered online from DC to 50 Hz, and on the Elekta system the data were sampled at 1000 Hz. The environmental noise in the MEG data (BTI Magnes 3600 system) was canceled online via reference coils (Robinson, 1989). Noise cancelling on the Elekta system used online signal space projection and MaxFilter technology for off-line suppression of spatial interferences. Using a Polhemus system (Polhemus 3Space Fastrak system), the position of the electrodes was digitized, as were the nasion and bilateral preauricular points, and the position of five spatially distributed coils that were attached to the cap.

Data analysis and statistical validation.

Artifact rejection was performed off-line for both the MEG and EEG data using MSI software for Experiments 1, 2, and 4, and Fieldtrip (Oostenveld et al., 2011; RRID:SCR_004849) for Experiment 3. A rejection threshold was determined for each subject, separately, in a manner that was blind to the trial type. The thresholds were determined separately for EEG and MEG data, and they were done so in an iterative manner by determining whether a given threshold needed to be raised or lowered, to get rid of the physiological artifacts while keeping the artifact-free trials in the data. The final thresholds for the MEG across participants were as follows: Experiment 1: 2.2–3.5 pT (14% of trials rejected); Experiment 2: 2.3–4.0 pT (11% of trials rejected); Experiment 4: 2.5–3.8 pT (13% of trials rejected). For Experiment 3, the thresholds were done using Fieldtrip's z-transformation method, which resulted in an average of 16% of trials rejected. After the artifact rejection, selective averages were obtained for each condition from 200 ms prestimulus and 800 ms poststimulus. The EEG data were re-referenced to the average of the left and right mastoids.

To account for the differences in the head positions of the participants relative to the MEG sensors, we repositioned the data recorded in the BTI system in sensor space. This was done by taking the data from sensor space and first computing a lead field using the minimum norm least-squares method using an MNI brain as the template. The data were then back-transformed into sensor space, with a reference sensor set (also based on the MNI brain) used to register each participant's data into the reference sensor space. The newly coregistered data were then averaged across participants to obtain a grand average.

To determine the onsets of these relative components, moving window t-tests (10 ms, moving in 1 ms steps) were conducted on the difference-waves from 0 to 500 ms for each component, respectively, using a baseline period of −200 to 0 ms (−100 to 0 ms for Experiment 3, shortened to mitigate the high level of alpha activity present due the fundamental differences in the task). To control for false-positive results, effects were only considered to be significant if more than five t-tests in a row had a p value < 0.05 (Guthrie and Buchwald, 1991). To determine whether the onsets of the N1pc component differed across conditions, a jackknife-based method (Miller, 1974) was adopted, with the fractional peak latency (as implemented in ERPSS Software by J. C. Henson, ERPL, UCSD) at 25% of the peak amplitude between 100 and 200 ms serving as a measure. This technique is a robust way to determine differences in onsets of effects across conditions (Miller et al., 1998; Kiesel et al., 2008).

Results

Experiment 1

Behavior

Participants were highly accurate (M = 90.53%, SD = 4.32%) and needed on average 660 ms (SD = 44 ms) to respond to the target. Response time and accuracy not did not significantly differ as a function of whether the target color was red or green (all p-values >0.1).

ERMF data

Figure 2a displays the magnetic waveforms elicited by the target (blue) and the distractor (pink) when presented in the contralateral (solid) and ipsilateral VF (dashed) relative to the sensor hemisphere the signal was recorded from. Figure 2b shows corresponding difference waves (contralateral VF-ipsilateral VF responses) representing tERLs (black) and dERLs (red). The black and red horizontal bars indicate time ranges of significant response differences (sliding t-tests) of the tERL and dERL, respectively. Both response differences show the first significant activation in the form of a negative polarity modulation peaking at ∼150 ms, with the dERL effect becoming significant slightly earlier (∼5 ms) than the tERL effect. The field distribution (Fig. 2c) at the peak in b is comparable for both modulations. This effect is characterized by an efflux-influx (red-blue) transition zone over lateral occipitotemporal cortex (black ellipses), suggesting a cortical response maximum in lateral and ventral extrastriate visual cortex. Of note, for the magnetometer data reported here, the maximum of the cortical response corresponds with the transition zone between efflux and influx field maxima (asterisks). Following this initial modulation, the tERL and dERL evolve into different response patterns. The tERL displays a broad negative enhancement, with an initial submaximum at ∼220 ms that ramps up to a global maximum at 280 ms. The underlying field shows a lateral occipitotemporal topography, with a more anterior-lateral distribution at 220 ms (early N2pc), which transitions to a more posterior distribution at 280 ms (late N2pc). The time course, the overall field topography, and the transition of the field distribution to more posterior regions, indicate that the tERL modulation is indeed the magnetic analog of the N2pc response (Hopf et al., 2000, 2006; Boehler et al., 2011). The dERL, in contrast, evolves into a positive modulation peaking at ∼250 ms. The effect, which shows opposite field polarity relative to the N2pc, represents the magnetic analog of the distractor positivity (Pd). As this is the first time that the Pd has been observed with MEG, we present the simultaneously-recorded ERP response (Fig. 3), which can be compared with the ERMF to confirm that the response we see is, indeed, a magnetic analog of the Pd. Figure 3 shows the dERL (red) and tERL (black) response of the ERP (a) together with azimuthal topographical maps of the electric field distribution at representative time points after search frame onset (b). The N2pc is visible as prominent negative modulation between 200 and 300 ms peaking at ∼270 ms. In contrast to the ERMF response, the N2pc of the ERP shows only a single maximum between 200 and 300 ms. To allow for comparability, the topographical maps of the N2pc show the electric field distribution at the early and late maximum of the ERMF response (compare Fig. 2b). The Pd appears in the same time range but peaks at ∼240 ms, that is, before the N2pc maximum. As for the ERMF data, the Pd and the N2pc show very similar field distributions, differing only in relative polarity, consistent with the proposal that the Pd represents a “reversed” N2pc process (Sawaki et al., 2012).

Figure 2.

ERMF results of Experiment 1. a, Waveforms elicited by the lateralized target (blue) and the lateralized distractor (pink) when presented in the contralateral (solid) and ipsilateral visual field (dashed) relative to the sensor hemisphere the ERMF response was recorded from. b, ERL difference waveforms (solid-minus-dashed in a) of the lateralized target (black) and the lateralized distractor (red). The black and red horizontal bars highlight time ranges of significant ERLs (sliding-window t test) for the target and distractor, respectively. c, Magnetic field distribution maps of the tERL and dERL at selected time points corresponding to the temporal maxima of the ERL responses (small vertical lines in b). The maps show the left-right hemisphere collapsed field distributions. The black ellipses encompass efflux-influx field components reflecting the N1pc, Pd, and N2pc. The asterisks mark the location of the efflux-influx field transition, where the underlying current source would approximately be located. Note, the scale varies across the different topographical maps, as demarcated.

Figure 3.

ERP results of Experiment 1. a, ERL difference waveforms elicited by the lateralized target (black) and lateralized distractor (red) at electrode sites PO7/8. The black and red horizontal bars highlight time ranges of significant ERLs for the target and distractor, respectively. b, Topographical maps of the tERL and dERL are depicted at the marked points (small vertical lines in a). The maps of the N2pc and Pd show the electric field distribution at the latency of the field maxima of the corresponding ERMF response (compare with Fig. 2). Note, the tERL shows no separate maximum corresponding with the tN1pc, presumably due to overlap with the following N2pc. The tN1pc map is therefore displayed at the latency of the dN1pc maximum. All topographical maps show left-right hemisphere collapsed field distributions, and the scales (variable) are demarcated in the figure.

The most intriguing observation here is that both the target and the distractor are associated with an early modulation of the ERMF response whose topography and relative polarity resembles that of the late N2pc (Fig. 2). An inspection of the dERL in Figure 3 reveals that this modulation, indeed, corresponds with a negative ERP deflection, appearing slightly later than in the ERMF response, but in the time range of the N1 component. Note, in contrast to the tERL of the ERMF, this modulation is not visible as a separate maximum in the tERL of the ERP, presumably due to overlap with the subsequent N2pc. Also, the size of the N1 modulation appears to be overall smaller in the ERP than in the ERMF response. Differences of the magnetic and electric field projections with a less optimal coverage of the electric field may be responsible for this. Although we have no definitive explanation for this discrepancy, it emphasizes the importance of considering complementary measures of brain activity for characterizing stimulus-evoked response components.

Previous ERP studies have reported early lateralized responses in the N1 time range, referred to as N1pc, which were suggested to reflect asymmetries of the stimulation setup (Wascher et al., 2009; Wascher and Beste, 2010; Verleger et al., 2012; Schettino et al., 2016). As will be shown below (Experiment 3), the tERL and dERL in the N1 time-range do not arise from such asymmetry of stimulation, but reflect truly attention-driven modulations analogous to the N2pc. We therefore adopt the term N1pc for its magnetic analog. It should be noted that the N1pc has been previously suspected to at least partially reflect the influence of endogenous factors also defining the N2pc (Verleger et al., 2012). The close similarity in terms of field polarity and distribution with the late magnetic N2pc clearly supports this interpretation. The very presence of an N1pc to the target (tN1pc) and the to the distractor (dN1pc) suggests that both are selected by an attentional process that is again involved later during the ultimate selection of the target. The N1pc appeared slightly earlier for the distractor than for the target. A jackknife latency analysis (see Materials and Methods), however, yielded no significant difference (F_(1,23) = 1.18, p = 0.29). In sum, the N1pc represents the earliest measurable lateralized response to the singleton items in the present search arrays. Given its similarity to the N2pc, we suppose that it reflects an attention process that marks significant (singleton) items for further processing, i.e., the distractor for later rejection (Pd) and the target for further selection (N2pc).

Experiment 2

That the N1pc indexes a representation for selection (tN1pc) or rejection (dN1pc) would be an important finding. There are alternative explanations, however, which need to be addressed. One possibility is that the N1pc reflects some form of attention driven color priming across experimental blocks (akin to priming of pop-out; Maljkovic and Nakayama, 1994, 2000; Fecteau, 2007; Kristjánsson and Campana, 2010). The target color in a given block of trials in Experiment 1 was always the distractor color in the next trial block (Fig. 1b, first row). It is possible that the N1pc reflects a persisting priority bias for the target colors (red, green), even in blocks where one or the other defines the distractor. In Experiment 2, we address this possibility in the following way: the color defining the distractor will change from block to block (constant within blocks) as in Experiment 1, but it will never serve to define the target (Fig. 1b, second row). If the dN1pc reflects a target-related color bias persisting throughout trial blocks, then it should not be elicited by the distractor in Experiment 2.

Behavior

Participants were highly accurate (Mean = 93.0% correct, SD = 6.7%) and responded rapidly to both red and green targets (Mean = 481 ms, SD = 53 ms). Response times and accuracy did not significantly differ as a function of target color (p values >0.05).

ERMF data

Figure 4a shows waveforms of the tERL (black) and dERL (red) difference obtained in Experiment 2. As in Experiment 1, prominent tN1pc and dN1pc responses are elicited, which rules out the idea that the N1pc merely represents a persisting feature bias for the target color. Importantly, the dN1pc appears ∼20 ms earlier than the tN1pc (jackknife analysis: F_(1,23) = 5.94, p < 0.05), suggesting that the initial selection of the distractor gained temporal priority over target selection. Finally, Experiment 2 replicates the modulation pattern seen between 200 and 300 ms in Experiment 1. That is, clear N2pc and Pd responses are elicited by the target and the distractor, respectively. The N2pc displays two phases with a local maximum at 210 ms and a later global maximum at 270 ms. The Pd arises before the maximum of the N2pc and shows a response with reversed polarity peaking at 240 ms.

Figure 4.

ERMF results of Experiment 2. a, ERL waveforms elicited by the lateralized target (black) and distractor (red). b, RT median-split analysis. ERL waveforms of fast (thick dashed) and slow RTs (thin dashed) are separately shown for the target (tERL; black) and the distractor (dERL; red). The gray and pink horizontal bars highlight time-ranges of significant ERLs for the tERL and dERL, respectively. The filled and open bars refer to the fast and slow RTs, respectively.

It is quite striking that we observe a temporal prioritization of the dN1pc, despite the fact that the distractor color was never a target-defining color. That is, although the overall relevance of the distractor was reduced by eliminating potential effects of target-color priming, distractor selection was present and even gained temporal priority. This observation speaks against target selection being facilitated by reducing selectivity for the distractor. Instead, we find a clear temporal prioritization of the distractor, rather suggesting that target selection is aided by expediting distractor selection and subsequent rejection. If this is the case, performance facilitation should be associated with an earlier and/or more pronounced dN1pc, Pd, and ultimately an earlier N2pc. To address this possibility we separately analyzed trials in which subjects were fast versus slow in identifying the target (RT median split; fast RTs: mean = 439.3 ms, slow RTs: mean = 537.4 ms). Figure 4b shows the respective waveforms of fast (solid traces) and slow RTs (dashed traces). As can be seen, the dN1pc of fast responses is more pronounced than for slow responses (significant amplitude difference between 130 and 160 ms; F_(1,23) = 4.62, p < 0.05), with slow RTs being associated with a smeared out and less consistent amplitude enhancement. The dN1pc for fast RTs also shows an earlier onset than the dN1pc for slow RTs. The latency difference, however, did not reach significance. Importantly, the Pd for slow RTs follows with a significant delay relative to fast RTs (∼10 ms, jackknife analysis: F_(1,23) = 4.37, p < 0.05), indicating that the distractor attenuation is expedited for fast RTs. Whereas the tN1pc of fast RTs is visibly larger than that of slow RTs, a sliding-window t test yielded no significant difference. The tN1pc is equally consistent for fast and slow RTs and shows no difference of onset-latency (jackknife analysis: F_(1,23) = 0.33). In contrast, the subsequent N2pc appears substantially earlier for fast versus than slow RTs (jackknife analysis: F_(1,23) = 5.47, p < 0.05), indicating that the ultimate selection of the target is speeded as a consequence. Hence, a comparison of trials with fast and slow responses confirms that the facilitation of target selection is due to a more pronounced and consistent selection of the distractor that speeds distractor rejection and subsequent target identification.

Note, the RT-dependent difference of the latency and amplitude of the dN1pc speaks against the possibility that the N1pc merely reflects a lateralized response due to the low-level feature contrast between the color singletons and the neutral distractors. That is, if this difference were an effect of low-level physical contrast, then it should be the same across all trials, regardless of whether or not participants were faster or slower to respond to the targets; however, these data are not as clean as the primary data, so this result should be interpreted with caution. Nonetheless, from block-to-block in Experiment 2, the target color alternated between two colors, while the distractor color varied among four different colors. One may ask whether this difference in color variation potentially gave rise to a differential color-contrast driven response appearing earlier for the distractor than the target singleton. The following Experiment (3) directly addresses this possibility.

Experiment 3

Experiment 3 aims at assessing whether the N1pc reflects an asymmetry of stimulation causing a lateralized color-contrast driven response instead of being an index of attentional selection of the color singletons. To address the issue, we tested whether the target and distractor singletons of Experiment 2 would elicit an N1pc response when color is entirely task irrelevant and spatial attention is drawn away from the search items. The stimulation protocol (the search arrays, the color assignment to the search items; Fig. 1b, middle row) was identical to Experiment 2, except that an additional RSVP stream of characters was presented at fixation (see Materials and Methods), which subjects had to continuously attend and perform a detection task (detecting the occasional “#” among letters). The search items in the periphery were irrelevant, and subjects were instructed to ignore them. To derive ERL responses, the data were analyzed as a function of target and distractor color assignment as well as lateralization as in Experiment 2. Note, for simplicity we will refer to the colors defining the color singletons as target and distractor color analogous to Experiment 2, although those colors were irrelevant and did not define the RSVP target or distractors in Experiment 3.

Behavior

The accuracy for the detection of the number of target characters present in block in the RSVP was 70.1%. Of note, because participants were given four possible response options as to the number of targets presented, chance performance would be 25%. The accuracy did not vary as a function of the target color in the search array (red T vs green T; p = 0.9). Response times were not analyzed, as participants were given as long as they needed to make their choice.

ERMF data

Figure 5 shows the waveforms of the tERL (black) and dERL difference (red) of Experiment 3 together with the corresponding ERLs of Experiment 2 (dashed gray/light-red, replotted from Fig. 4a) as a comparison. As visible, the dERL shows a minimal modulation in the N1pc time range, which appears smaller than the dN1pc in Experiment 2, and only marginally significant (0.05 < p < 0.10; between 126 and 137 ms). The minimal modulation of the tERL in this time range is not significant (all p > 0.3 in the N1pc time range), and, in fact, of opposite polarity relative to the N1pc. Hence, both the dERL and tERL show some minimal modulation in the N1pc time range, which may reflect imbalances of stimulus lateralization due to the asymmetric search arrays. These modulations, however, are very minimal relative to dN1pc and tN1pc seen in Experiment 2. Finally, the tERL shows a negative modulation in the early N2pc time range, which is significant, but, again, very small relative to the N2pc in Experiment 2. It potentially reflects the higher frequency of singletons appearing in the target color (red/green) versus the distractor color (magenta/gray/brown/cyan). In sum, Experiment 3 demonstrates that no significant N1pc response is elicited when attention is drawn away form the search items, and when color is irrelevant for the experimental task. This observation rules out the possibility that imbalances of low-level color contrast account for the modulation, thereby emphasizing that the N1pc reflects a truly attention-driven selection of the color singletons.

Figure 5.

ERMF results of Experiment 3. ERL waveforms elicited by the task-irrelevant color singletons whose color defined the target (black) or the distractor (red) in Experiment 2. The gray and light-red waveforms replot the N2pc and Pd of Experiment 2 (Fig. 4a), respectively, for comparison. The horizontal black bar highlights the time range of a significant tERL difference.

Experiment 4

Experiment 2 showed that the N1pc does not merely reflect a persisting color bias for the target color. Furthermore, rendering the distractor color globally irrelevant caused a significantly earlier onset of the dN1pc relative to the tN1pc, suggesting that selection for rejection gains temporal priority over target selection, even when the competitive advantage for the target color is high. An obvious question is whether the priority selection for the distractor depends on the predictability of the distractor color. It has been shown that observers can strategically avoid searching for irrelevant items matching a working-memory representation (Woodman and Luck, 2007), and that cuing the color of non-target items can facilitate search (Arita et al., 2012). Those observations led to the proposal that observers may build some form of feature template that aids rejection of the distractor (template for rejection), and this template thereby facilitates search. Such a template for rejection account may apply to Experiments 1 and 2, in which the color of the distractor was constant within trial blocks. In Experiment 4, therefore, we test whether the dN1pc is an index of a color template for rejection. To this end, we minimize the predictability of the distractor color within a trial block. As in Experiments 1 and 2, the target color will either be red or green on a given block, but the distractor color will randomly vary from trial to trial among six possible non-target colors (Fig. 1b, bottom row). If the earlier onset of the dN1pc reflects the priority selection of the distractor based on its predictable color (i.e., a feature-template), the latency advantage of the dN1pc should disappear or even be delayed relative to the tN1pc. Alternatively, if the earlier onset of the dN1pc reflects priority selection of the distractor in a more general way, it should be elicited here with an earlier latency, even though the distractor color on a given trial is not known.

Behavior

Participants were again highly accurate, with the mean accuracy for the red targets being slightly higher than to the green (red Mean = 95.4% SD = 2%; green Mean = 94.6%, SD = 3%; t₍₁₉₎ = 2.67, p = 0.02). Participants were also slightly faster at responding to the red targets (red Mean = 511 ms SD = 32 ms; green Mean = 527 ms SD = 31 ms; t₍₁₉₎ = 8.11, p < 0.001).

ERMF data

Figure 6a displays the results of Experiment 4. Both the tERL and the dERL difference waves show a prominent N1pc response. Importantly, the dN1pc arises earlier than the tN1pc, indicating that the temporal priority of distractor selection does not reflect the operation of a feature template for rejection. A jackknife latency analysis confirms that the latency difference is significant (∼15 ms; F_(1,19) = 5.77, p < 0.05). Note, the present data cannot determine whether the onset latency difference reflects a speeded onset of the dN1pc relative to the tN1pc, or a delay of the tN1pc relative to the dN1pc. In any case, the earlier onset of the dN1pc indicates that distractor selection gained temporal priority independent of preknowledge about the distractor defining color. Finally, as in Experiments 1 and 2, a prominent N2pc and Pd response is visible, with the Pd peaking (260 ms) before the global maximum of the N2pc (290 ms), indicating that ultimate target selection follows after distractor rejection.

Figure 6.

ERMF results of Experiment 4. a, ERL waveforms elicited by the lateralized target (black) and distractor (red). b, Fractional peak latency analysis the tN1pc (black) and dN1pc (red) in Experiments 1, 2, and 4. c, Peak latency analysis the N2pc (black) and Pd (red) in Experiments 1, 2, and 4.

Discussion

In four experiments, we investigate the neural processes underlying target and distractor selection in visual search. Specifically, we sought to determine the degree to which a distractor must be selected before it can be rejected, how this varies as a function of the properties of the distractor, and how this interacts with target selection. To this end, we implement asymmetric search displays, allowing us to separate underlying neural activity (Woodman and Luck, 2003; Hickey et al., 2009). Replicating previous observations, we find that target and distractor selection are associated with distinct N2pc and Pd responses, respectively. Importantly, we find that both the target and the distractor elicit a distinct early response before target selection and distractor rejection, which we refer to as N1pc. The N1pc has been previously described as an ERP component (Wascher et al., 2009; Wascher and Beste, 2010), and was attributed to a sensory imbalance of left and right hemisphere stimulation. As shown in Experiment 3, such sensory imbalance does not account for the N1pc response observed here. While a sensory imbalance in the form of a single lateralized search item was previously found to influence the P1/N1 components (Luck and Hillyard, 1994b; Hickey et al., 2009), no modulation of those components was seen for a lateralized color pop-out among multiple items (Luck and Hillyard, 1994b). Finally, as discussed below, many ERP studies investigating the role of lateralized singleton distractors with asymmetric search arrays did not find N1pc responses, consistent with this component reflecting a more specific correlate of active item selection in visual search.

The N1pc resembles the N2pc in polarity and field topography, suggesting that target and distractor are initially selected by an attentional mechanism that reappears during the N2pc phase of target selection (Verleger et al., 2012). Most notably, and contrary to the intuition that the selection of target information should have priority, we find that not only is the distractor robustly represented early on, distractor selection gains temporal priority over target selection during the very initial stage of item processing, as well as during the later phase, where distractor attenuation (Pd) appears before ultimate target selection (late N2pc; Fig. 6c).

Paradoxically, as visible in Figure 6b, comparing the fractional peak latency of the N1pc across Experiments 1, 2, and 4, this priority selection is particularly apparent when the distractor is optimally suited for being ignored, that is, when its defining color never serves as a target (Experiments 2 and 4), and when its color is constant and therefore always predictable (Experiment 2). First, when the target color also served as a distractor color (Experiment 1), we observe that the relative latencies of the dN1pc and tN1pc show only a minimal advantage for the distractor, whereas when the distractor color was optimally ignored (Experiment 2), we observe that the temporal advantage of the dN1pc over the tN1pc is biggest. In both of these experiments the distractor color is equally predictable, but in Experiment 1 the distractor color gains global prominence due to being the target color on half of the trial blocks. This suggests that some form of long-term color priming (Kristjánsson and Campana, 2010) can alter the temporal prioritization process. Additional evidence for the temporal priority selection of the distractor singleton can be seen in the comparison of the fractional peak latency of the N1pcs between Experiments 2 and 4 (Fig. 6b). Although the color assignment and predictability of the target are identical in both experiments, the latency of the tN1pc in Experiment 4 is delayed (∼25 ms) relative to Experiment 2. The delay is in the order of the delay of the dN1pc (∼30 ms), suggesting that both components are roughly shifted in time. Because distractor selection has temporal priority, target selection is postponed by the delay that distractor selection incurs. Of course, caution has to be taken when comparing between experiments, and we therefore take these observations as tentative evidence, with further research needed.

The present finding of a temporal prioritization of distractor selection seems to be counterintuitive at a first glance. Nevertheless, it suggests a possible mechanism by which the brain resolves the ignoring paradox (Moher and Egeth, 2012; Cunningham and Egeth, 2016). The solution is to grant priority to the selection of salient, but to-be-ignored, stimuli. This allows for the optimization of the distractor rejection (Pd) and speeds target selection (N2pc). Thus, feature-based target identification is facilitated in visual search. Consistent with this interpretation, we find that faster target identification in Experiment 2 is associated with a more pronounced and consistent dN1pc, followed by speeded Pd and N2pc responses. We conclude that the N1pc indexes the initial representation of salient items, which sets the stage for subsequent selection and rejection processes, with the latter critically determining search performance. The more consistent and earlier the distractor is marked for later rejection, the faster its rejection (Pd) will be, which, in turn, will “free the way” for faster target selection (N2pc). Finally, as illustrated in Figure 6c showing the peak latency of the late N2pc and the Pd in Experiments 1, 2, and 4, the Pd arose in the time range of the N2pc but always before its absolute maximum, indicating that distractor rejection appeared before ultimate target selection. The temporal relationship between the Pd and the N2pc can be variable, however, with the Pd being reported to appear before (Sawaki and Luck, 2010) and after the N2pc (McDonald et al., 2013; Kiss et al., 2012). Importantly, the temporal priority of the Pd was suggested to reflect the higher salience of distractors (Sawaki and Luck, 2010) consistent with the present observations, and further supporting the notion that the brain biases target selection by effectively prioritizing distractor rejection.

The functional role of the N1pc suggested here is conceptually related to the recently-proposed “attend-to-me” signal: a postulated signal indicating the presence of an item with competitive advantage in visual search (Sawaki and Luck, 2010, 2011). Although representing a logical necessity, the attend-to-me signal has thus far not been linked to any measurable neural process. The N1pc would be a plausible correlate of this signal. Sawaki and Luck (2010) proposed that salient singletons trigger an attend-to-me signal automatically, which informs subsequent processing stages. Whether attentional capture will be prevented depends on the degree to which top-down control processes can subsequently suppress distractor selection (signal suppression hypothesis of controlled attention capture). The present observations dovetail with this proposal, and they add that it may be the relative timing of the attend-to-me signals (as indexed by the N1pc) that plays a critical role for biasing competition between salient items. Note, the temporal prioritization of the dN1pc is in the order of ∼15–20 ms in Experiments 2 and 4, which is, by any measure of the speed of spatial attention (Duncan et al., 1994; Wolfe, 1994; Moore et al., 1996; Ward et al., 1996), too short to reflect a sequential shift of attention from the distractor to the target singleton. This suggests that the attentional process underlying the N1pc essentially operates in parallel across the search array. We presume that it reflects some form of global marking of salient item locations based on feature contrast which “sets the pointer” for further selection and rejection. Such parallel selection process may be akin to what has been proposed to underlie visual marking for accounting for the beneficial effect of previewing distractors on search performance (Olivers et al., 2002; Watson et al., 2003; Humphreys et al., 2004).

Temporal priority selection of distractors has been previously inferred from the observation that distractor singletons elicit an N2pc with an earlier onset compared with the target (Hickey et al., 2006, 2010). Although this observation is conceptually in line with the present interpretations (for a critical re-evaluation of the evidence, see McDonald et al., 2013), Hickey et al. (2006, 2010) did not find a response to the distractor before the N2pc. Additionally, other studies investigating the processing of salient irrelevant distractors with ERPs failed to find an N1pc response as seen here (Eimer and Kiss, 2008; Kiss et al., 2008b; Töllner et al., 2012; Jannati et al., 2013; McDonald et al., 2013). An important difference to the present experiments is that in all those studies the salient distractor was a pop-out in another feature dimension than the target (e.g., a target shape singleton combined with a distractor color singleton). According to the dimension weighting account (Müller et al., 1995, 2010; Found and Müller, 1996; Töllner et al., 2012), subjects could give less weight to the irrelevant dimension and only search for the target using the relevant dimension. This would then preempt prioritized distractor selection as indexed by the dN1pc. In the present experiments, subjects could not adopt such singleton search mode (Bacon and Egeth, 1994), as the target and the distractor were both color pop-outs. Theoretically, an analogous down-weighting could be applied to irrelevant feature-values within the same dimension (Braithwaite and Humphreys, 2003; Olivers and Humphreys, 2003), instantiating a template for rejecting the color-value of the distractor (Arita et al., 2012). The results of Experiments 2 and 4 speak against such preset weighting mechanism. A temporally prioritized dN1pc was seen independent from whether the color of the pop-out distractor was predictable. That is, feature-value selective weighting was either not possible in the reported experiments, or equally applied to all values of the task-relevant dimension.

The fact that down-weighting the distractor color is not involved in the present experiments has a notable implication. It potentially suggests that top-down driven feature-value-specific biasing can be instantiated only after values of the relevant dimension are physically presented and have triggered at least an initial response in feature-selective neurons. Such a notion has been put forth based on the nature of ascending and descending connectivity in visual cortex (Shipp, 2007) and neurophysiological data in the monkey support such interpretation. For example, color selective neurons in monkey V4 were found to increase firing when a preferred color presented in the receptive field (RF) matches a target defining color-cue (Motter, 1994), but not when the preferred color is cued while the RF of the neuron is not stimulated. Likewise, feature-selective units in IT increase baseline firing upon the presentation of an effective feature cue falling into the unit's large RF (Chelazzi et al., 1998). In contrast, when the effective feature cue was presented outside the RF, like in V4 (where RFs are smaller), no such baseline increase was seen (Chelazzi et al., 2001). Together with our data, this implies that in the implementation of the selection for a certain color, any presented color-value must be represented first, to subsequently let top-down modulations take effect.