Neurons in the lateral intraparietal area create a priority map by the combination of disparate signals

Abstract

Primates search for objects in the visual field with eye movements. We recorded the activity of neurons in the lateral intraparietal area (LIP) in animals performing a visual search task in which they were free to move their eyes, and reported the results of the search with a hand movement. We distinguished three independent signals: (1) a visual signal describing the abrupt onset of a visual stimulus in the receptive field; (2) a saccadic signal predicting the monkey's saccadic reaction time independently of the nature of the stimulus; (3) a cognitive signal distinguishing between the search target and a distractor independently of the direction of the impending saccade. The cognitive signal became significant on average 27 ms after the saccadic signal but before the saccade was made. The three signals summed in a manner discernable at the level of the single neuron.

Introduction

Despite a plethora of studies, there is no strong consensus on the role of the lateral intraparietal area (LIP) in the organization of behavior. LIP has been studied with different paradigms, which have resulted in different conclusions about what the area contributes to behavior. Thus, activity in LIP has been shown to correlate with attention (Colby et al. 1996; Bisley and Goldberg 2003, 2006), saccades (Colby et al. 1996; Barash et al. 1991; Snyder et al. 1997; Ipata et al. 2006a, b), salience (Bisley and Goldberg 2003; Gottlieb et al. 1998), expected value (Sugrue et al. 2004), perceptual (Shadlen and Newsome 2001) or economic (Platt and Glimcher 1999) decisionmaking, perceived motion (Eskandar and Assad 1999; Assad and Maunsell 1995), categorization of the direction of movement (Freedman and Assad 2006), stimulus shape (Sereno and Maunsell 1998) and location of the active limb (Oristaglio et al. 2006). Conversely, other studies have shown that LIP activity seems not to be associated with attention (Platt and Glimcher 1997; Snyder et al. 1997) or saccadic eye movements (Bisley and Goldberg 2003, 2006; Powell and Goldberg 2000). These results arose from experiments in which neurons were recorded in monkeys trained to make or not make specific saccades, and the conclusions may have reflected the constraints of the experimental design. In the real world, however, primates make saccades to facilitate vision, and there is no such thing as a wrong saccade.

We examined the responses of single LIP neurons while the animals performed a free viewing visual search task (Ipata et al. 2006a, b) in which they reported what they saw by making a hand movement. After an initial fixation they were free to move their eyes. Because their eye movements were neither punished nor rewarded, the monkeys adopted a strategy in which they made short latency saccades which acquired the target on the first saccade about half the time. We analyzed activity between the array onset and the saccade by sorting the data into four different types of trials according to the direction of the first saccade and the object in the receptive field—saccades to the target or distractor in the receptive field, and saccades away from the target or distractor in the receptive field. With these trials, we distinguished three signals: an undifferentiated bottom-up visual signal, not affected by saccade direction or the nature of the stimulus in the receptive field; a saccadic signal, described elsewhere (Ipata et al. 2006a, b); and a cognitive signal, which identified the target in the receptive field even when the monkey made a saccade to a distractor outside the receptive field. Because the cognitive signal was manifest on average about 29 ms after the choice of the saccade goal, these results demonstrate that the monkey often made saccades to objects in the array without having certain knowledge of the nature of the target. The three signals sum in a consistent way, and enable us to construct accurately, on a cell-by-cell basis, the temporal waveform of the response from a particular trial type from the temporal waveforms of the responses in the three other trial types.

We propose that LIP creates a priority map (Serences and Yantis 2006) by summing disparate signals which can be used by the oculomotor system to determine the goal and reaction time of the saccade, should a saccade be appropriate, and by the visual system to determine the loci of attention. We suggest that this view of LIP enables us to resolve the seeming contradictions in the catalog of results described above.

Materials and methods

Two male rhesus monkeys (Macaca mulatta) weighing 8–12 kg had scleral coils, head restraining devices and recording chambers implanted during sterile surgery under ketamine and isoflurane anesthesia as described previously (Bisley et al. 2004). Chambers were positioned using magnetic resonance images and neurons were identified as being in LIP by their consistent visual, delay-period and saccade related response in memory guided saccade task. All experimental protocols were approved by the Animal Care and Use Committees at Columbia University and the New York State Psychiatric Institute as complying with the guidelines established in the Public Health Service Guide for the Care and Use of Laboratory Animals.

A Hitachi CPX275 LCD projector running the VEX open GL-based graphics system (available by download from http://www.lsr-web.net) rear-projected stimuli onto a screen. We measured image luminance using a Minolta photometer. We recorded neurons using glass-coated tungsten electrodes (Alpha-Omega), and commercially available amplification (FHC or Alpha Omega) and filtering (Krohn-Hite) equipment. We measured eye position using a two channel Northmore Phase Detector. We used the MEX system (available by download from http://www.lsr-web.net) to sort and digitize action potentials online.

Behavioral task

Behavioral control data collection was done on personal computers using REX system (Hays et al. 1982). Monkeys were trained to perform a free-viewing visual search task in which they were required to report the orientation of a target among seven distractors (Fig. 1). The target consisted of a capital T that could be presented in an upright or an inverted position. Monkeys had to release the left bar when the upright oriented target was present, or the right bar when the inverted T was present. The distractors were composed of the same vertical and horizontal components, and differed only by where the horizontal line crossed the vertical line. One of the distractors, the popout, was green and brighter than the target and the other distractors. The neural responses when the popout distractor appeared in the receptive field have been described previously (Ipata et al. 2006b), so in this study, we have excluded trials in which it was present in the receptive field or in which the monkey made the first saccade towards it. The target and the distractors were positioned around an imaginary circle whose eccentricity was chosen so that at least one of the objects appeared in the center of the receptive field. Each trial started when the monkey fixated a central fixation point. If the animal maintained fixation for about 1–1.75 ms within a 3° window, the fixation point extinguished, and the array appeared. They were given 3 s to report the orientation of the target and they did not have any constraints on the direction and number of ocular movements. The position of the target and of the distractors varied randomly from trial to trial. Correct responses were rewarded with a drop of water.

Fig. 1.

The free-viewing visual search task. After an initial fixation period (1–1.75 s), the central fixation spot extinguished and the search array appeared without any delay. One of the stimuli, either the target (an upright or inverted T) or a distractor, appeared in the center of the neuron's receptive field (dashed line). Monkeys had 3 s to report the orientation of the target by releasing one of two bars. During the presentation of the search array, no constraints were imposed on the monkey's eye movements and they were not required to fixate the target before giving the response

Data analysis

We wrote all data analysis programs in Matlab (Mathworks Inc). We calculated a spike-density function for each trial by convolving the spike train, sampled at 1 kHz, with a Gaussian of sigma 10 ms (Richmond et al. 1987). We calculated the neuronal response over an interval of interest as the average of the number of spikes during that period. To determine the timing of the cognitive signal we used the same sliding-window technique that we had previously used to calculate the timing of the saccadic signal. For each millisecond, we calculated the activity in a 50 ms bin centered at that time for each class of response. We then compared the activity in each pair of bins using a two-tailed t test. We defined the time of separation as the first bin of 20 consecutive bins with P < 0.05 (Ipata et al. 2006a).

Results

Behavior

The monkeys initiated each trial by grabbing two bars, one with each hand. Then a spot appeared in the center of the screen. The monkey had to look at this spot for 1–1.75 s after which a search array appeared and the monkey was free to move its eyes (Fig. 1). The array consisted of a target (an upright or an inverted capital T) and seven distractors (four different stimuli with horizontal and vertical components of the same width and height as the target, but which intercepted each other at different heights), arranged in a radially symmetric circular pattern so that one stimulus appeared in the center of the receptive field (dashed circle, Fig. 1). On each trial the position of the target varied randomly. Monkeys were required to signal the orientation of the target by releasing one of the two bars. Eye movements were neither rewarded nor punished, so the animals' oculomotor behavior was totally unconstrained. The monkeys had 3 s to respond and the responses were classified as incorrect only if monkeys released the wrong bar or ran out of time. The average reaction times for the eight-stimulus arrays were 545 and 555 ms for monkey R and Z, respectively. On most trials the monkey fixated the target before responding, the exception to this being that monkey Z fixated the upright T on every trial in which it was presented, but did not always fixate the downward T. Close examination of this monkey's behavior indicated that he may have treated down-T trials as absent up-T trials (Krishna et al. submitted). Because of this, we have only included up T trials in our analysis for monkey Z. Both monkeys performed the task correctly on 95–99% of the trials.

In agreement with a previous visual search experiment in which monkeys were not punished for making a saccade to a distractor (Motter and Belky 1998), the monkeys’ mean first saccadic latencies were quite short (151 ± 23 ms for monkey R, 146 ± 22 ms for monkey Z), and both monkeys made their first saccades to a distractor on roughly half of the trials.

We found that visual search behavior in monkeys resembled visual search in humans (Treisman and Gelade 1980): their reaction times exhibited a set size effect for difficult, but not for easy targets. We collected behavioral data while monkeys performed the search task in which the number of distractors varied randomly trial by trial (7, 11 or 15 distractors). Both manual reaction time (calculated from the onset of the array to when monkey released the bar) and the number of saccades executed on each trial showed a set size effect (manual reaction time slope = 16.8 and 9.6 ms/item in monkeys R and Z, respectively; number of saccades slope 0.46 saccades/item and 0.26 saccades/item in monkeys R and Z, respectively) (Bisley et al. 2008).

Neuronal dataset

We recorded a total of 73 LIP neurons from four hemispheres of two monkeys (42 from monkey R and 31 from monkey Z). The saccadic properties of these neurons have been reported elsewhere (Ipata et al. 2006a, b). Every neuron had visual activity as determined by a short-latency response to the abrupt onset of a visual stimulus. We ascertained their visual receptive fields by studying their activity during a memory-guided saccade task (Hikosaka and Wurtz 1983); 71% of the neurons responded significantly during the delay period of the memory-guided saccade task (P < 0.05, t tests compared to background activity). We arranged the search array for each cell so that one array object lay at the center of the receptive field as determined in the delayed saccade task. For the majority of results shown here, we describe only those trials in which the monkeys actually fixated the target after one or two saccades (this made up 76 and 73% of the trials from the monkeys R and Z, respectively). We divided the trials into 4 classes according to the direction of the first saccade (to orawayfrom the receptivefield) and accordingto the object in the receptive field (target or distractor). Some neurons responded to stimuli placed in more than one location in the array. For purposes of analysis we defined the receptive field as the optimum object location, and we excluded from analysis trials in which the monkey made a saccade to one of the active flanking locations, or when the popout or target appeared at those locations. We also excluded two-saccade trials in which, after the first saccade, the target or the goal of the second saccade still lay in the receptive field of the neuron. We performed each analysis on only those cells in which we recorded two or more trials from the relevant classes. We performed some analyses only on cells with more than 5 or 10 trials, and specify this when describing the results.

Three disparate signals

Under conditions of free visual search, LIP exhibited three different signals: an undifferentiated visual response, a saccadic response, and a stimulus identity related signal, which we will call the cognitive response.

The undifferentiated visual response was the earliest signal. This was a response to the appearance of a stimulus in the receptive field. It did not discriminate between the target and the distractor (Fig. 2a, b, P = 0.9 for monkey R and P = 0.5 for monkey Z, Wilcoxon Sign Rank test) and nor was it affected by the direction of the upcoming saccade (Fig. 2c, d, P = 0.2 for monkey R and P = 0.9 for monkey Z). The two-way ANOVA (direction of the saccade to or away from the receptive field and target position in or outside the receptive field) did not result in any significant (P < 0.05) main effect or interaction (Thomas and Pare 2007).

Fig. 2.

The undifferentiated visual response. a, b The mean activity of each cell in the interval from 40 to 70 ms after array onset in which the target appeared in the receptive field (ordinate) is plotted against the mean activity in the same interval for that cell when the distractor appeared in the receptive field (abscissa) for each monkey. Data were taken from 1- and 2- saccade trials, independent of the direction of the first saccade. c, d The mean activity in the interval from 40 to 70 ms after array onset for each cell in trials in which the monkey made the first saccade to the receptive field (ordinate) plotted against the mean activity in the same interval from trials in which the monkey made the first saccade away from the receptive field (abscissa), independent of the stimulus in the receptive field

The saccadic signal appeared next. A previous analysis of the activity of the neurons in this data set showed that LIP predicted when the upcoming saccade would occur (Ipata et al. 2006a). The choice of saccade goal became manifest in LIP an average of 90 ms after the appearance of the search array.

The cognitive signal was the last to appear. It reflected the stimulus identity of the array object in the receptive field regardless of the direction of the impending saccade (Fig. 3). When the target was in the receptive field, the activity was significantly greater than when there was a distractor in the receptive field, even when the monkey made a saccade to a distractor outside the receptive field (P = 0.003 for Monkey Z, P = 1.76 × 10⁶ for Monkey R, Wilcoxon Sign Rank test). Figure 3a, c show individual cell examples from each monkey, and Fig. 3b, d show the activity of every cell in the sample. We classified the difference in activity between these two trial types as the cognitive signal since it was neither a pure visual or motor response. Because negative spike counts are impossible, we half-wave rectified the cognitive signal.

Fig. 3.

The cognitive signal when the monkey made a saccade away from the receptive field. a Monkey R: single neuron spike density function from trials in which the first saccade was made away from the receptive field. Solid blue trace: target in the receptive field. Dotted blue trace: distractor in the receptive field. Dotted black trace: the half-wave rectified difference between the two blue traces is the cognitive signal, the difference in the response to the target and distractor. The spike-density functions are aligned with the onset of the array. b Mean activity of two or more trials of each neuron in the interval from 150 to 250 ms after array onset when monkey R made a saccade away from a target in the receptive field (ordinate) plotted against the activity in the same interval when the monkey made a saccade away from a distractor in the receptive field (abscissa). c, d Same as in a, b for monkey Z. e Spike density function average across the population from trials in which the first saccade was made away from a target in the receptive field and monkey acquired the target on the second saccade (blue trace) and when the monkey acquired the target on the third or later saccade (black trace). Dotted blue trace: distractor in the receptive field. Dotted black trace: half-wave rectified difference between the black line and dotted blue trace is the cognitive signal. Data are from monkey R. f Same conventions as in b, except plotted in the ordinate is the mean activity of each neurons when monkey R made a saccade away from the target in the receptive field and did not acquire the target on the second saccade

The cognitive signal is unlikely to be a pattern-specific signal describing the low-level visual properties of the object in the receptive field. Had it been, we would have expected to see some neurons selective for a distractor pattern. We never saw one. The cognitive signal was also not merely related to the next saccade. We were careful to eliminate from analysis all trials in which the target remained in the neuron's receptive field after the first saccade, so the response could not be a simple presaccadic enhancement. One monkey made a reasonable number trials with more than two saccades, and for that monkey the activity evoked by the target in the receptive field when the monkey acquired the target on the second saccade was not different from the activity evoked by the target in the receptive field when the monkey acquired the target on the third or later saccade (Fig. 3e), so the activity could not have been an enhanced signal for the next saccade. Indeed, the mean activity evoked by the target in the receptive field when the monkey made a saccade away from it in trials with three or more saccades was greater, on a cell-by-cell basis, than the activity evoked by a distractor in the receptive field when the monkey made a saccade away from it (Fig. 3f, P = 0.001, Wilcoxon Sign Rank test).

Finally, there was also a cognitive signal even when the monkey made a saccade to the receptive field, as is shown by the significant difference (P = 3.2 × 10⁻⁶ for Monkey R, P = 0.03 for Monkey Z, Wilcoxon Sign Rank tests) between trials in which the monkey made a saccade to a target in the receptive field and a saccade to a distractor in the receptive field (Fig. 4a, c for individual cell examples, Fig. 4b, d for the population). The mean spike density functions from the populations of the two types of trials were different in both monkeys (P = 4.6054e–008 for Monkey R, P = 0.04 for Monkeys Z, Kolmogorov–Smirnov tests). Thus, we see the cognitive signal as a top down indication of how likely the stimulus is to be a target, since one would not expect the nature of the target to cause a difference in response if LIP had only visual and saccadic signals.

Fig. 4.

The cognitive signal when the monkey made a saccade to the receptive field. a Spike density functions for a single cell from monkey R. The single cell example is from the same cell shown in Fig. 3a. Red trace: target in the receptive field. Dotted red trace: distractor in the receptive field. Black trace: cognitive signal, the difference in the response to the target and distractor. The spike-density functions are aligned with the onset of the array. b Mean activity of each neuron in the interval from 150 to 250 ms after array onset when monkey R made a saccade to a target in the receptive field (ordinate) plotted against the activity in the same interval when the monkey made a saccade to a distractor in the receptive field (abscissa). c, d Same as in a, b for monkey Z. The single cell example is from the same cell shown in Fig. 3c

The two cognitive signals are not identical: the cognitive signal calculated when the monkey made a saccade to the receptive field was less than the cognitive signal calculated when the monkey made a saccade away from the receptive field (compare the black dotted lines in Fig. 3 to the black solid lines in Fig. 4). However, the waveforms were strikingly similar, and the two cognitive signals were correlated. Figure 5 plots the cognitive signal for each cell during a perisaccadic time-epoch when the saccade was made to the receptive field against the cognitive signal calculated when the saccade was made away from the receptive field. To minimize the noise of the signal we only performed this analysis on cells in which there were at least 10, in monkey R, or 5, in monkey Z, trials of each trial type, which, perforce, resulted in a diminution of the number of cells we could use (12 in monkey R and 6 in monkey Z). We used an epoch of 50 ms, starting 25 ms before the saccade for monkey R, and a 100 ms epoch starting 50 ms before the saccade for monkey Z. The smaller number of saccade-away trials in monkey Z necessitated using a larger window. While the two signals are clearly different in both monkeys (P > 0.03, Wilcoxon Sign Rank tests), they are significantly positively correlated (Pearson correlation coefficients: 0.5 for monkey R and 0.9 for monkey Z), indicating that the difference between the response to the target and the response to the distractor incorporates a cognitive component, albeit one that may be diminished when the monkey plans a saccade to the receptive field.

Fig. 5.

Correlation of the cognitive signals measured when the monkey made a saccade to or away from the receptive field. a The mean of the perisaccadic activity of the cognitive signal from trials in which the saccade is made to (ordinate) or away (abscissa) from the receptive field. Each dot represents the mean of the number of spikes in an interval between 25 ms before and 25 ms after the saccade, for monkey R. Least-squares correlation line is shown in black. b Same data for Monkey Z, except the window used was 50 ms before and 50 ms after the onset of the saccade

The cognitive signal became significant later when the monkey made a saccade away from the receptive field than it did when the monkey made a saccade to the receptive field, becoming significant at 117 ms in the saccade-away case, and 133 ms in the saccade-to case.

LIP combines the three signals

The neural signals exhibited by LIP in our search task can be constructed from the three disparate signals: the visual signal, the saccadic signal and the cognitive signal. To demonstrate this, we defined the response waveform in the trials in which the monkey made a saccade away from a distractor in the receptive field as the undifferentiated visual signal; the response waveform when the monkey made a saccade away from the target which was in the receptive field as the sum of the undifferentiated visual signal and the cognitive signal; the response waveform when the monkey made a saccade to a distractor in the receptive field as the sum of the undifferentiated visual and saccadic waveforms; and the response waveform when the monkey made a saccade to the target in the receptive field as the sum of all three signals. We then constructed the waveform in trials when the monkey made a saccade to the target in the receptive field (ie. that with all three components) by adding the components calculated using waveforms from the three other trial types, each of which had only one or two of the components. Because the two cognitive signals were different but correlated, we used the slope and intercept of a regression line similar to that of Fig. 5 to calculate the cognitive signal when the monkey made a saccade to the receptive field from the cognitive signal when the monkey made a saccade away from the receptive field, such that:

$$C_{\text{s}}\left(t\right)=m{C}_{\text{a}}\left(t\right)+b\text{for}{C}_{\text{a}}\left(t\right)>0$$ (1)

where C_s(t) is the cognitive signal in the saccade-to-the-receptive field case at time t, C_a(t) is the half-wave-rectified cognitive signal in the saccade-away-from-the-receptive field case at time t, m is the slope and b is the intercept of the regression line. Because Eq. 1 allows for the possibility of a negative spike count, we added the constraint that

$$C_{\text{s}}\left(t\right)=0\text{for}{C}_{\text{s}}\left(t\right)<0$$ (2)

Note that we calculated the waveform of C_s(t) for each individual cell using the waveform C_a(t) of that cell, but using the values of m and b calculated from the regression line fitted to the data of all of the neurons except the one being analyzed, to avoid any circularity in the analysis. We then constructed the waveform for the saccade-to-the-target-in-the-RF case S_t(t), which contains all three components, by summing the waveforms of the measured saccade-to-the-distractor-in-the-RF case S_d(t), which contains the uncomplicated visual signal and the saccadic signal, and the cognitive signal for the saccade-to-the-target case, C_s (t), as calculated above:

$$S_{\text{t}}\left(t\right)=C_{\text{s}}\left(t\right)+S_{\text{d}}\left(t\right)$$ (3)

The calculated waveforms for both single cells (Fig. 6a, b) and for the population (Fig. 6c, d) are extraordinarily close to the activity recorded in the saccade-to-the-target-in-the-RF trials. Indeed, the population mean spike density functions were not significantly different in either monkey (P = 0.09 for Monkey R, P = 0.1 for Monkeys Z, Kolmogorov–Smirnov tests). Thus, we can generate the waveform for a given trial type for a given cell from the waveforms of the cell's activity in the three other trial types and the slope and intercept of the regression line calculated from the entire population except for the current cell in a given monkey.

Fig. 6.

Calculation of the waveform when the monkey made a saccade to the target in the receptive field from the other three trial types. a Single cell responses in trials in which the saccade was made to the target inside the receptive field (red traces, the thickness of the line is the standard error of the mean) are compared to the calculated signal obtained by summing the three components obtained in different trial types (green traces). The spike density functions are aligned with the onset of the array. Data from the same cell shown in Figs. 3a, 4a from monkey R. b Single cell data from monkey Z, the same cell shown in Figs. 3c, 4c, d): spike density functions averaged across the population in monkey R and Z, respectively. Conventions as in a, b

To quantify the correlation between the measured and constructed signals, we divided each trace into 20 ms bins from 80 to 240 ms after array onset, calculated the mean of each bin, and plotted the means from the measured signal against the means from the calculated signal (S_t(t)) for each cell. The two signals were not different at the single cell level (single cell examples in Fig. 7a, b). More importantly, the slopes of the individual regression lines, converted into degrees, clustered around 45°, with means ± SEM of 42.5 ±1.2 and 40 ± 1.4° for Monkeys R and Z, respectively (Figs. 7c, d). This provides further evidence that the constructed signal gives an excellent approximation of the actual signal.

Fig. 7.

Comparison of calculated and measured activity for saccades to the target in the receptive field. a Each point plots the mean activity in a 20 ms time interval from 80 to 240 ms after array onset measured in saccade-to-target-in-the-receptive-field trials (abscissa) against the activity in the same intervals calculated from the other three trial types (ordinate), for a single neuron from Monkey R. Least-squares correlation line is shown in black. b Same data for a single neuron from Monkey Z. c Bar plot of the distribution of the slopes, in degrees, from the regression analyses for each cell in Monkey R. d Same data for Monkey Z

Discussion

These experiments show that neurons in association cortex actually combine disparate signals in a predictable fashion to create their output. The three signals, which are combined by LIP in our visual search task, include an undifferentiated visual signal, a saccadic signal (Ipata et al. 2006a; Thomas and Pare 2007), and a cognitive signal. The undifferentiated visual signal reports the abrupt onset of a visual stimulus in the receptive field (Bisley et al. 2004), but is unaffected by the direction of the monkey's impending saccade or the nature of the stimulus in the receptive field. Previous work suggests that the undifferentiated signal is a response to the appearance of a new object in the environment and is associated with the attention evoked by abrupt onset (Bisley et al. 2004), because a stable object in the environment does not evoke such an exuberant response when a saccade brings it into the receptive field of an LIP neuron (Gottlieb et al. 1998). The saccadic signal specifies the goal and latency of the impending saccade, but is independent of the task-relevance of the object in the receptive field (Ipata et al. 2006a; Thomas and Pare 2007). The cognitive signal distinguishes between the search target and the distractors, independent of the direction of the impending saccade.

The cognitive signal did not have a trivial, saccade-related explanation; it was independent of the direction of the next saccade. In one monkey who made enough three-saccade trials to have them analyzed, the cognitive signal appeared in trials in which the second saccade was also made to a distractor. It could not have been related to a perisaccadic receptive field shift (Duhamel et al. 1992; Kusunoki and Goldberg 2003) because we excluded trials in which the target remained in the receptive field after the first saccade. Because every cell we studied had a visual response, we cannot exclude the possibility that the cognitive signal and even the saccadic signal were behavioral modulations of visual responses (Goldberg and Wurtz 1972), and with this caveat in mind we use the terms as shorthand (Shen and Pare 2007) have recently described a similar signal in the intermediate layers of the monkey superior colliculus while monkeys perform an oculomotor conjunction search.

Depending on the monkey's behavior in the task (whether it made a saccade to a distractor or to the target; or whether it made a saccade toward or away from the receptive field of the neuron under study), LIP neurons express one, two, or all three of these signals, and the activity in a given trial type can be constructed from the activity in the other trial types. Although the sources of these signals are unknown, there are good candidates for each of them. The undifferentiated visual signal could easily come from visual areas MT, V3 and V3a, all of which have short latency visual responses (Schmolesky et al. 1998; Bair et al. 2002) and monosynaptic connections to LIP (Baizer et al. 1991; Lewis and Van Essen 2000). The saccadic signal could easily come from the frontal eye field, which has neurons which discharge before purposive saccades (Bruce and Goldberg 1985) and which has a monosynaptic connection to LIP (Blatt et al. 1990). The source of the cognitive signal is more complex. It could reflect the value of the signal (Sugrue et al. 2004; Platt and Glimcher 1999), in which case it could arise from the anterior cingulate (Blatt et al. 1990), or indirectly from the amygdala (Paton et al. 2006). Alternatively, it could reflect the pattern identification of the signal, in which case it could arise from V4 or inferior temporal cortex, both of which have monosynaptic projections to LIP (Baizer et al. 1991; Distler et al. 1993). Finally, it could represent an attentional signal once some other area has found the capital T. A similar cognitive signal may underlie the increased responsiveness of frontal eye field neurons to distractors that resemble the search target (Sato et al. 2003; Thompson et al. 1996; McPeek and Keller 2002).

The activity of neurons in LIP in our visual search task reflects the monkeys' behavior in the task. In paradigms in which the monkey is either rewarded for making its first saccade to the target, and therefore punished with lack of reward for making the wrong saccade (Thompson et al. 1996), the saccadic reaction times are on the average 75 ms longer than in our paradigm. In our task, clearly the monkeys choose to make the saccade before they know with absolute certainty the location of the target. This bias of moving the eyes without really knowing what object they are going to foveate is reflected in the fact that the monkeys make a high number of saccades to distractors and is also manifested in the activity of LIP: the saccade choice is made on average 90 ms from the onset of the array (Ipata et al. 2006a) which is (on average) 27 ms before the cognitive signal reliably identifies the target on average at 117 ms from the onset of the array. This in turn suggests that in relatively difficult visual search monkeys adopt a strategy of making saccades as rapidly as they can, tolerating an increase in the time needed to select the target, when such saccades are not punished. It is also important to note that once the saccadic choice is made, there is a consistent delay before the saccade is made (Ipata et al. 2006a, b; Thomas and Pare 2007). In popout search paradigms where the search goal is a saccade target, the interval between saccade target identification and saccadic initiation correlates with saccadic reaction time (Thompson et al. 1996). The difference between this result and our finding in free visual search may result from the monkey's extending saccadic reaction time to confirm the choice of saccade goal, thus avoiding the punishment of missing a reward because of a wrong saccade.

Most models of higher cortical function include a stage where neurons in association cortex sum disparate signals (Metcalfe 1993). Linear summation has been demonstrated at the synaptic level (Jagadeesh et al. 1993; Cash and Yuste 1999) and Avillac et al. (2007) showed that some neurons in the ventral intraparietal area (VIP) sum visual and tactile signals linearly. Yang and Shadlen (2007) recently showed that the log likelihoods contributed by different factors establishing the reward for a saccade add and subtract in a linear fashion in LIP. A more common mathematical operation performed by cortex is multiplication or division as a gain control (Haider et al. 2007). In our model the saccadic signal acts as a divisive gain control on the cognitive signal, and the product of this operation adds linearly with the visual and saccadic signals.

Finally, our results suggest a new interpretation of the role of LIP in neural processes. We have shown that LIP sums sensory, motor, and cognitive signals in a simple fashion. The result of such summation predicts the direction of the impending saccade and the identity of the stimulus in the receptive field. Rather than claiming a specific role for the output, such as the allocation of attention (Bisley and Goldberg 2003), perceptual decision making (Shadlen and Newsome 2001), or saccade planning (Snyder et al. 1997), we suggest that LIP specifies the priority of the different parts of the visual field in a general manner. This priority map can then be used by areas to which LIP projects in a manner specific to those recipient areas. Thus the oculomotor system uses the priority map to choose the goal of saccadic eye movements when saccades are appropriate, and the visual system simultaneously uses it to determine the locus of visual attention. The priority map may be built up not only from the signals that we have demonstrated here, but also from the expected reward value of the object in the receptive field (Sugrue et al. 2004), the arousal associated with the forced cancellation of a planned saccade (Bisley and Goldberg 2006), perceptual decision-making (Shadlen and Newsome 2001), perceived motion (Assad and Maunsell 1995; Freedman and Assad 2006), stimulus shape (Sereno and Maunsell 1998), top-down control of stimulus importance (Ipata et al. 2006b) and even by the choice of effector of the behavior that the stimulus induces (Dickinson et al. 2003; Oristaglio et al. 2006). We suggest that any analysis of an object in the visual field will increase the salience of the spatial location of that object, and that increased salience will be reflected in the increased activity in the representation of that location in LIP. The various cognitive exercises shown to increase activity in LIP may therefore reflect top-down inputs to the priority map, rather than participation by LIP in the decisions underlying those exercises.

We suggest that the use to which the brain puts the signal constructed by LIP depends not so much on the signal itself, but to the area receiving the signal from LIP and generating the function in question. Thus the visual system can use the LIP signal to pin attention at the peak of the priority map even when the signal arises from a saccade plan (Bisley and Goldberg 2003), and the oculomotor system can make a saccade to the peak of the priority map even when it arises predominantly from the exuberant response to the abrupt onset of a visual object in the visual field (Ipata et al. 2006a, b; Bisley et al. 2004). In other words, the whole question of whether an LIP response is ‘visual’ or ‘motor’ is irrelevant. Keeping track of whether a given spike from LIP is ‘visual’ or ‘motor’ is a difficult and unnecessary computational problem. We suggest instead that the recipient areas, visual or oculomotor, merely count all the spikes they receive from LIP, which are not tagged as to their genesis.