Neural Basis of the Set-Size Effect in Frontal Eye Field: Timing of Attention During Visual Search

Abstract

Visual search for a target object among distractors often takes longer when more distractors are present. To understand the neural basis of this capacity limitation, we recorded activity from visually responsive neurons in the frontal eye field (FEF) of macaque monkeys searching for a target among distractors defined by form (randomly oriented T or L). To test the hypothesis that the delay of response time with increasing number of distractors originates in the delay of attention allocation by FEF neurons, we manipulated the number of distractors presented with the search target. When monkeys were presented with more distractors, visual target selection was delayed and neuronal activity was reduced in proportion to longer response time. These findings indicate that the time taken by FEF neurons to select the target contributes to the variation in visual search efficiency.

INTRODUCTION

Because only some of all the information entering the visual system is relevant for acting in a given environment, attention must be allocated to selectively process one among many alternatives, especially when those alternatives are difficult to distinguish. This has been investigated using visual search for a target stimulus in an array of distractor stimuli (Bergen and Julesz 1983; Carrasco and Yeshurun 1998; Duncan and Humphreys 1989; Palmer 1994; Treisman and Gelade 1980; Wolfe 2007). The observation that visual search response time (RT) often increases with the number of distractors has been central in theories of attention (Bundesen 1990; Bundesen et al. 2005; Desimone and Duncan 1995; Duncan and Humphreys 1989; Treisman and Gelade 1980; Wolfe 2007). Previous work has shown that brain areas in the macaque visuomotor system have distinct classes of neurons involved in different stages of visual search. Visual and visuomovement neurons in the frontal eye field (FEF) (Schall and Hanes 1993; Thompson et al. 1996), lateral intraparietal area (LIP) (Bisley and Goldberg 2003; Ipata et al. 2006; Thomas and Paré 2007), and superior colliculus (SC) (McPeek and Keller 2002) participate in the visual selection of target objects by increasing activity when the target is inside the neuron's receptive field (RF) relative to when a distractor is in the RF. Other neurons in these areas participate in preparing eye movement responses (Bruce and Goldberg 1985; Hanes and Schall 1996; Segraves and Goldberg 1987; Sommer and Wurtz 1998; Sparks et al. 1976; Wurtz and Goldberg 1971). Neurophysiological recordings from nonhuman primates can distinguish between competing accounts of the locus of increases in RT with increased set size because the activity of different types of neurons reveals the timing of different stages of processing (Sato et al. 2001; Sternberg 2001; Woodman et al. 2008).

To determine the neural locus of processing delays with increasing search set sizes, we measured the activity of visually responsive neurons in FEF during a demanding visual search task. The activity of these neurons discriminates between target and distractor, thus measuring the time of target selection, which corresponds to attention allocation (Bichot and Schall 1999; Juan et al. 2004; Murthy et al. 2001; Schall 2004; Thompson et al. 1997, 2005). We trained two monkeys to search for a target stimulus among one, three, or seven distractor stimuli (set sizes 2, 4, and 8) in a T/L search display (Fig. 1A). Monkeys fixated a spot at the center of the display and received a liquid reward for producing a single saccadic eye movement to the target (in Fig. 1A, the L). Across experimental sessions, the monkeys alternated between searching for Ts or Ls among randomly rotated distractors. This visual search task results in longer RT with larger set size in human observers (Bergen and Julesz 1983; Carrasco and Yeshurun 1998; Duncan and Humphreys 1989; Palmer 1994; Treisman and Gelade 1980; Wolfe 2007). We found that neurons selected search targets later and discharged fewer spikes when presented with more distractors.

FIG. 1.

Visual search task. A: monkeys made saccades to a target (here, an upright L; not to scale) presented with 1, 3, or 7 distractors. The monkey's eye position is represented by the dashed circle, which is invisible to the monkey. B: response time (RT, circles) and percentage error (squares) vs. set size for each monkey. Error bars represent SE around the mean of the session means. Filled symbols: monkey Q. Open symbols: monkey S. Error bars for percentage error are smaller than plotted points.

METHODS

Behavioral task and recording

Activity of FEF neurons was recorded in both hemispheres of two male macaques (Macaca radiata) performing memory-guided saccade and visual search tasks. Recordings were acquired from the rostral bank of the arcuate sulcus using tungsten microelectrodes (FHC). In the search task, monkeys searched for a target (T or L) among distractors (L or T) (Woodman et al. 2007). Distractors could be homogeneous (e.g., upright Ls) or heterogeneous (e.g., Ls oriented differently). Each trial began with monkeys fixating a central spot for about 600 ms. A target was then presented at one of eight isoeccentric locations equally spaced around the fixation spot (Fig. 1A). The other seven locations contained seven distractor stimuli, three distractors, or one distractor. During each recording session, the target was a T or an L rotated by 0, 90, 180, or 270°. Monkeys were given a liquid reward for making a saccade to the target location and fixating it for 1,000 ms.

Activity from each neuron was recorded during a memory-guided saccade task to distinguish visual- from movement-related activity (Bruce and Goldberg 1985; Hikosaka and Wurtz 1983). The target was flashed alone for 80 ms. Monkeys were required to maintain fixation for 500–1,000 ms after the target onset. When the fixation spot disappeared, the monkey was rewarded for a saccade to the remembered location of the target.

Our data set consists of 57 neurons from monkey Q and 26 neurons from monkey S. Spikes were sorted on-line and off-line using principal components analysis and template matching (Plexon, Dallas, TX).

Monkeys were surgically implanted with a head post, a subconjunctival eye coil, and recording chambers. Surgery was conducted under aseptic conditions with animals under isoflurane anesthesia. Antibiotics and analgesics were administered postoperatively. All surgical and experimental procedures were in accordance with the National Institutes of Health Guide for the Care and Use of Laboratory Animals and approved by the Vanderbilt Institutional Animal Care and Use Committee.

Data analysis

To measure the firing rate of each neuron, we used a spike density function, convolving each spike with a kernel resembling a postsynaptic potential (Thompson et al. 1996).

We used a memory-guided saccade task to classify neurons. Visual neurons had significantly greater activity in the 100 ms after the target flash than in the 100 ms before the target flash. Movement neurons had greater responses in the 100 ms leading up to the saccade than in the 100 ms before the target flash. Visuomovement neurons had greater responses in the 100 ms after the target flash and in the 100 ms leading up to the saccade than in the 100 ms before the target flash. Neurons analyzed in this study were visual and visuomovement and had significant above-baseline activity during the memory delay.

To measure the time of target selection we used millisecond-by-millisecond Wilcoxon rank-sum tests. Selection time is defined as the time at which the distribution of activity when the search target was inside a neuron's RF was significantly greater than the distribution of activity when the target was opposite the RF for ten consecutive milliseconds with P < 0.01. This “neuron–antineuron” approach presumes that a population of neurons in the brain representing the location of the target competes with a population of neurons representing the location of distractors opposite the target. Measuring selection time with a receiver operating characteristic analysis (Thompson et al. 1996) yielded similar results. There was no difference in target selection time between visual and visuomovement neurons.

All statistical tests were done with Bonferroni corrections for multiple comparisons. All analyses were done with R (http://www.r-project.org/).

RESULTS

Behavior

Both monkeys exhibited a consistent and strong effect of set size on RT (Fig. 1B). Across sessions RT was significantly longer for set size 8 compared with set size 4 and for set size 4 compared with set size 2 and, within sessions, RT increased significantly with set size in 100% of sessions (Table 1). The slopes of RT by set size corresponded to an increase of 22 ms per additional distractor for monkey Q and 10 ms per additional distractor for monkey S. Percentage error (moving the eyes to a distractor) was low for both monkeys and increased with set size only for monkey S (Fig. 1B; Wilcoxon signed-rank test, P < 0.001). These results indicate that the monkeys consistently emphasized accuracy over speed.

TABLE 1.

Comparisons of response time (RT), target selection time, and firing rate across search array size

Values are means±SE. Pairwise comparisons are Wilcoxon signed-rank tests with significant differences highlighted:

**for P < 0.001,

*for P < 0.05, and N.S. for not significant. The sample sizes are 57 for monkey Q and 26 for monkey S.

Set-size effects on target selection time

We recorded the activity of 57 visually responsive FEF neurons from monkey Q and 26 from monkey S that selected the target in the visual search task. This selection has been identified with the allocation of attention (Bichot et al. 2001; Schall 2004; Thompson and Bichot 2005; Thompson et al. 2005). Each neuron exhibited visual responses and sustained activity during the delay period of a memory-guided saccade task. Target selection time measures the time at which a neuron's activity on trials with the search target inside its RF exceeded activity on trials with a distractor in its RF (and the target opposite its RF). To determine how variation of target selection time related to variation of RT, we aligned neuronal responses to the time of presentation of the search array and measured target selection time for each set size.

Figure 2 shows the average response of a neuron when the target versus a distractor appeared in the RF for each set size, aligned to the onset of the search array. The neuron selected the target after 160 ms when one distractor was present (Fig. 2A, vertical line), after 177 ms when three distractors were present (Fig. 2B), and after 230 ms when seven distractors were present (Fig. 2C). The delay of the target selection corresponded with the delay of the RT (set size 2 mean RT ± SE, 219 ± 4.2 ms; set size 4, 265 ± 7.8 ms; set size 8, 386 ± 12.3 ms). Median RT for each set size is indicated by the black arrowhead in each panel.

FIG. 2.

Target selection time by set size for an example neuron. A: average firing rate when the target was inside the neuron's receptive field (RF; dark band around black curve, 102 trials) and opposite the RF (light band around gray curve, 113 trials) for set size 2. Median RT is denoted by the black arrowhead. Filled bands indicate SE around the mean firing rates. The solid vertical line indicates target selection time. The dashed horizontal line indicates the peak firing rate. The inset shows an example search array of set size 2. B: the same as in A, but for set size 4 (dark, 108 trials; light, 105 trials), with the dashed vertical line indicating target selection time. The dashed horizontal line indicates the peak firing rate in A. C: the same as in A, but for set size 8 (dark, 102 trials; light, 92 trials), with the solid gray line indicating selection time and the dashed horizontal line indicating peak firing rate in A.

Neurons selected the target later when confronted with larger set sizes (Fig. 3, Table 1). Comparing across neurons, target selection time was significantly later for set size 8 than for set sizes 2 or 4 (Kruskal–Wallis rank-sum test, P < 0.01). Comparing within neurons, target selection time was significantly later for set size 4 compared with set size 2 for monkey S and was significantly later for set size 8 than for set size 4 and for set size 8 than for set size 2 for both monkeys.

FIG. 3.

Target selection time by set size. A: cumulative distributions of selection time across neurons for set sizes 2 (black), 4 (dashed), and 8 (gray). B: stacked histogram of differences in selection time between set sizes 4 and 2 (dark gray), 8 and 2 (gray), and 4 and 2 (light gray), for monkey Q. Most of the histogram falls to the right of zero, indicated by the dashed vertical line. C: the same as in B, but for monkey S.

We measured the relationship between RT and selection time by computing a linear regression for each set size (Table 1). The regression for monkey Q was significant for set size 8 (solid gray line in Fig. 4C). For monkey S, regression was significant for set size 4 (solid line in Fig. 4B) and set size 8 (solid black line in Fig. 4C). Combined across monkeys, regression was significant for set size 8, with a slope of 0.79 ± 0.16 (P < 0.001) and intercept of 189 ± 31 ms (P < 0.001). That is, for set size 8, a 10 ms increase in selection time corresponded to an approximately 8 ms increase in RT. Thus target selection time of FEF neurons accounts for a significant amount of variance in RT for large set sizes.

FIG. 4.

RT vs. selection time for set sizes 2 (A), 4 (B), and 8 (C). The solid black lines indicate significant regression lines for monkey S, the gray line for monkey Q. Dashed gray lines indicate unity.

Set-size effects on firing rate

We measured peak mean firing rate for each neuron from search array onset until saccade onset when the search target was inside the neuron's RF. For the example neuron in Fig. 2, the dashed horizontal lines indicate peak mean firing rate for set size 2. Peak firing rate decreased from 72 to 64 to 56 spikes/s for set sizes 2, 4, and 8, respectively. Across the population of neurons, peak firing rate decreased with increasing set size (Fig. 5, Table 1). Firing rate was significantly higher for set size 2 than that for set size 4 for monkey Q. Firing rate was significantly higher for set size 2 than that for set size 8 and for set size 4 than that for set size 8 for both monkeys. Thus increasing set size decreases peak firing rate in FEF.

FIG. 5.

Firing rate by set size. A: cumulative distributions of firing rate across neurons for set sizes 2 (black), 4 (dashed), and 8 (gray). B: stacked histogram of differences in firing rate between set sizes 2 and 4 (dark gray), 2 and 8 (gray), and 4 and 8 (light gray), for monkey Q. Most of the histogram falls to the right of zero, indicated by the dashed vertical line. C: the same for monkey S.

DISCUSSION

We discovered that neurons in FEF that select the location of the target in a visual search array do so later with fewer spikes when presented with progressively more, complex distractors. This result complements earlier observations that the time course of target selection by FEF neurons is related to the visual similarity between targets and distractors in singleton and conjunction search (Bichot et al. 1999, 2001; Sato et al. 2001; Thompson et al. 1996). The target selection process first described in FEF (Schall and Hanes 1993) has now been observed in posterior parietal cortex (Constantinidis and Steinmetz 2001; Ipata et al. 2006; Thomas and Paré 2007) and SC (McPeek and Keller 2002). These results are consistent with the hypothesis that a population of neurons in FEF, LIP, and SC can be identified with the salience map in models of visual search and attention (Bundesen 2005; Itti and Koch 2000; Thompson and Bichot 2005; Wolfe 2007). These neurons represent the extent to which a stimulus in their RF is the focus of attention. The target selection process marks the outcome of visual processing and the covert allocation of attention to the target that precedes overt responses (Schall 2004; Thompson and Bichot 2005). We found that RT varied with selection time as if the saccade could not be produced until the target was located. This occurs because the onset of activation of presaccadic movement neurons in FEF is delayed when visual search is less efficient, yielding longer RT (Woodman et al. 2008). In other words, under the conditions of our experiments, preparation of the eye-movement response does not begin until perceptual processing of the visual array is completed. Note that although target selection time in FEF accounted for a large amount of variation of RT, other sources also contributed to the variation in RT; the major source is the postselection stochastic process of saccade preparation (Woodman et al. 2008).

Theories of visual attention propose that the cause of longer RT with increased set size arises because of the limited capacity of the visual system to simultaneously analyze all elements in a visual search array (Bundesen 1990; Bundesen et al. 2005; Desimone and Duncan 1995; Duncan and Humphreys 1989; Treisman and Gelade 1980; Wolfe 2007). Such theories predict that a neural index of the allocation of attention to the target is delayed when more nontarget stimuli must be processed. The time at which visually responsive neurons in FEF signal the location of the target provides a neurochronometric measure of the conclusion of perceptual analysis (Juan et al. 2004; Murthy et al. 2001; Schall 2004; Thompson et al. 1997, 2005). According to some theories of attention, such as Treisman's Feature Integration Theory (Treisman and Gelade 1980) and Wolfe's Guided Search (Wolfe 2007), target selection maps onto a mechanism directing attention in the visual field. Our measure of selection time indicates when, on average, attention is reliably focused on the target. This time is later with larger set sizes because attention shifts more often to one or more distractors before focusing on the target. According to other theories of attention, such as Bundesen's Theory of Visual Attention (TVA) (Bundesen 1990; Bundesen et al. 2005), attention is deployed to all of the possible targets simultaneously and eventually selects an object when it enters short-term memory. According to TVA, the neurochronometric measure of target selection would mark the time at which perceptual processing is completed and the target representation is transferred into memory. Because FEF activity has been associated with attention and working memory (Funahashi et al. 1989) both of these theoretical explanations of our findings are tenable and distinguishing between them is a goal for future research.

The reduced discharge rate with larger stimulus arrays has been observed previously (Balan et al. 2008) and described in terms of increased uncertainty (Basso and Wurtz 1998), number of alternatives (Lee and Keller 2008), or RF surround suppression (Schall et al. 1995a). This may underlie the capacity limitation of visual search with increasing number of distractors (Desimone and Duncan 1995; Palmer 1994; Treisman and Gelade 1990). Lower discharge rates among the neurons representing the target and distractors result in less dynamic range for discriminating between the alternative representations. Thus more time is taken to achieve a given signal-to-noise ratio (Bichot et al. 2001; Shadlen et al. 1996). This is consistent with accounts of variation of search efficiency in terms of signal detection theory (Palmer et al. 2000). Confronted with stimulus ambiguity and armed with a target template, a top-down influence can bias processing of stimuli that resemble the target (Bundesen 2005; Desimone and Duncan 1995; Sato et al. 2003). The dense reciprocity of connectivity between FEF and extrastriate visual cortex (Schall et al. 1995b) makes it difficult to distinguish the respective contributions of activity in areas like V4 and MT signaling the features of search items (Ogawa and Komatsu 2004) and activation in FEF feeding back to influence extrastriate areas (Hamker 2005; Moore and Armstrong 2003). This basic question may be resolved in future studies that directly compare the timing of signals across areas.

In contrast to what we found in FEF, a recent study of area LIP in macaques performing a different visual search task reported that the time for target selection did not vary with set size (Balan et al. 2008). We believe that these contradictory findings are unlikely to be due to differences between LIP and FEF and instead can be attributed to differences in performance. The effect of set size on RT was larger in our study of FEF than it was in the study of LIP. For example, we obtained a significant increase of RT with set size in 100% of the experimental sessions, compared with roughly 70% of the sessions reported in the LIP study. Also, we obtained a variation of RT relative to the overall mean RT from the largest (8) to the smallest (2) set size of 25–43%; the study of LIP found a ratio from the largest (6) to the smallest (2) set size of around 10%. Finally, the error rates in our experiment did not vary substantially with set size; the error rates in the LIP study increased with set size. Consequently, it seems most plausible that we observed the variation of RT with neural selection time because we obtained a larger variation of RT against which to perform the regression.

In summary, when presented with complex search arrays containing more distractors, FEF neurons, as part of a distributed network, take longer to locate the target because the representations of both target and distractors are weaker. This provides a mechanistic basis for the capacity limitation that is expressed when visual inputs compete for representation.

GRANTS

This work was supported by National Institutes of Health Grants T32-MH-064913, R01-EY-08890, P30-EY-08126, and P30-HD-015052, and Robin and Richard Patton through the E. Bronson Ingram Chair in Neuroscience.