Predictive Activity in Macaque Frontal Eye Field Neurons During Natural Scene Searching

Abstract

Generating sequences of multiple saccadic eye movements allows us to search our environment quickly and efficiently. Although the frontal eye field cortex (FEF) has been linked to target selection and making saccades, little is known about its role in the control and performance of the sequences of saccades made during self-guided visual search. We recorded from FEF cells while monkeys searched for a target embedded in natural scenes and examined the degree to which cells with visual and visuo-movement activity showed evidence of target selection for future saccades. We found that for about half of these cells, activity during the fixation period between saccades predicted the next saccade in a sequence at an early time that precluded selection based on current visual input to a cell's response field. In addition to predicting the next saccade, activity during the fixation prior to two successive saccades also predicted the direction and goal of the second saccade in the sequence. We refer to this as advanced predictive activity. Unlike activity indicating the upcoming saccade, advanced predictive activity occurred later in the fixation period, mirroring the order of the saccade sequence itself. The remaining cells without advanced predictive activity did not predict future saccades but reintroduced the signal for the upcoming saccade at an intermediate time in the fixation period. Together these findings suggest that during natural visual search the timing of FEF cell activity is consistent with a role in specifying targets for one or more future saccades in a search sequence.

INTRODUCTION

Searching our visual environment is an essential skill and is most effective when the targets for successive saccades are not chosen at random but follow an internally generated plan (Aivar et al. 2005; Findlay and Brown 2006; Zingale and Kowler 1987). Although it is established that the frontal eye field (FEF) contributes to the control of voluntary saccadic eye movements in both humans and monkeys (for reviews, see Goldberg and Segraves 1989; Schall 1997), little is known about the FEF's role in the control of the series of multiple saccades made during visual search. Bruce and Goldberg (1985) demonstrated that the activity of about one-third of FEF cells is closely tied to saccadic eye movements, leaving a majority of cells that do not play a direct role in saccade production. Many studies support a role for these cells in visual selection to guide both covert and overt orienting responses (Sato et al. 2001, 2003; Schall 2001; Schall and Hanes 1993, 1998; Schall et al. 1995; Thompson et al. 1996). In addition, there is increasing evidence that the FEF plays a role in the top-down control of visual attention (Buschman and Miller 2007; Moore and Armstrong 2003; Moore and Fallah 2001, 2004; Wardak et al. 2006; but see also Khan and colleagues 2009)). Early human and monkey behavioral studies suggested that the FEF is involved in the generation of sequences of saccades (Collin et al. 1982; Luria et al. 1966). However, with a few notable exceptions (e.g., Balan and Ferrera 2003; Murthy et al. 2007; Tian et al. 2000; Umeno and Goldberg 1997), the single saccade trial structure of most FEF neurophysiological studies was not intended to test the FEF's role in generating multiple saccades.

In this study, we looked for evidence of FEF cell involvement in selecting future targets for the sequences of saccades made during self-guided search of two-dimensional images. Previously we have shown that while freely viewing natural scenes, FEF visual cell activity was modulated by the target of the upcoming saccade (Burman and Segraves 1994b), and preliminary work done at that time suggested that the FEF was involved in selecting targets for future saccades (Burman and Segraves 1994a). Here we recorded FEF cell activity while monkeys searched scenes for an embedded target. This task allowed monkeys the freedom to direct saccades at will but also forced them to assess the content of the scenes, thus providing a more realistic environment in which both top-down and bottom-up forces were at work (Chen and Zelinsky 2006; Itti and Koch 2000; Pomplun 2006). Preliminary reports of these experiments have been published in abstract form (Phillips and Segraves 2007, 2008).

METHODS

Animals and surgery

Two female adult rhesus monkeys (Macaca mulatta) were used for these experiments and are identified in this report as MAS14 and MAS15. Northwestern University's Animal Care and Use Committee approved all procedures for training, surgery, and experiments performed. Each monkey received preoperative training followed by an aseptic surgery to implant a subconjunctival wire search coil, a Cilux plastic recording cylinder aimed at the FEF, and a titanium receptacle to allow the head to be held stationary during behavioral and neuronal recordings. All of these methods have been described in detail elsewhere (Dias and Segraves 1999; Helminski and Segraves 2003). Surgical anesthesia was induced with the short-acting barbiturate thiopental (5–7 mg/kg iv) and maintained using isoflurane (1.0–2.5%) inhaled through an endotracheal tube. The FEF cylinder was centered at stereotaxic coordinates anterior 25 mm and lateral 20 mm. The location of the arcuate sulcus was then visualized through the exposed dura and the orientation of the cylinder adjusted to allow penetrations that were roughly parallel to the bank of the arcuate sulcus. Both monkeys had an initial cylinder placed over the left FEF. Monkey MAS14 later had a second cylinder place over the right FEF.

Behavioral paradigms

We used the REX system (Hays et al. 1982) based on a PC computer running QNX (QNX Software Systems, Ottawa, Ontario, Canada), a real-time UNIX operating system, for behavioral control and eye position monitoring. Visual stimuli were generated by a second independent graphics process (QNX – Photon) running on the same PC and rear-projected onto a tangent screen in front of the monkey by a CRT video projector (Sony VPH-D50, 75-Hz noninterlaced vertical scan rate, 1,024 × 768 resolution).

Visually guided and memory-guided delayed saccade tasks

Monkeys fixated a central red dot for a period of 500–1,000 ms. At the end of this period, a target stimulus appeared at a peripheral location. On visually guided trials, the target remained visible for the duration of the trial. On memory-guided trials, the target disappeared after 350 ms. After the onset of the target, monkeys were required to maintain central fixation for an additional 700–1,000 ms until the central red dot disappeared, signaling the monkey to make a single saccade to the target (visually guided) or the location at which the target had appeared (memory-guided). The delay period refers to the period of time between the target onset and the disappearance of the fixation spot. These two tasks were used to characterize the FEF cells by comparing neural activity during four critical epochs. An FEF cell could be categorized by any combination of visual, delay, or premotor activity (see Data analysis). Typically, trials of these types were interleaved with each other and with the scene search tasks described in the following text. However, in some cases, there was only enough data for statistical analysis from one of the delayed saccade tasks. The visually guided task was also used initially to determine the response field of the cell.

Scene search task

This task was designed to generate large numbers of purposeful, self-guided, saccades. Monkeys were trained to find a picture of a small fly embedded in photographs of natural scenes (Fig. 1A). After monkeys learned the standard visually guided and memory-guided search tasks, the target spot was replaced with the image of the fly. After 30 min, the scene task was introduced. Both monkeys used in this experiment immediately and successfully sought out the fly. The photographs were taken using a digital camera and included scenes with engaging objects such as animals, people, plants, or food. After a few sessions performing this task, it became obvious that monkeys were finding the target after only one or two saccades. We therefore used a standard alpha blending technique to superimpose the target onto the scene. This method allows for varying the proportions of the source (target) and destination (the background scene) for each pixel and was used to create a semi-transparent target. Even after extensive training, we found that the task was reasonably difficult with a 65% transparent target, requiring the production of multiple saccades while the monkeys searched for the target. Monkeys began each trial by fixating a central red dot for 500–1,000 ms, then the scene and embedded target appeared simultaneously with the disappearance of the fixation spot, allowing monkeys to begin searching immediately. The fly was placed pseudorandomly such that its appearance in one of eight 45° sectors of the screen was balanced. Within each sector its placement was random between 3 and 30° of visual angle from the center of the screen. Trials ended when the monkeys fixated the target for 300 ms or failed to find the target after 25 saccades. Images of natural scenes were pseudo-randomly chosen from a library of >500 images, such that individual images were repeated only after all images were displayed. An essential feature of this task is that although they searched for a predefined target, the monkeys themselves decided where to look. The location where the target was placed on the image did not predict the amplitudes and directions of the saccades that would be made while searching for it nor the vector of the final saccade that captured it.

Fig. 1.

Scene search task. A: sample scene with embedded target fly. Monkey's eye traces during the trial appear in yellow. Bottom right: zoom in on target for better visibility. B: extrsaction process for saccades of this trial. Blue, response-field; red, anti-response-field; green, neutral fields; gray, excluded border zones. C: polar plot of vector endpoints for all saccades made while recording activity from the neuron with response field depicted in B.

Neuronal recordings

The recording of single-neuron activity was done with tungsten microelectrodes (A-M Systems, Carlsborg, WA). Electrode penetrations were made through stainless steel guide tubes that just pierced the dura. Guide tubes were positioned using a Crist grid system (Crist et al. 1988) (Crist Instrument, Hagerstown, MD). Recordings were made using a single electrode advanced by a hydraulic microdrive (Narashige Scientific Instrument Lab, Tokyo, Japan). On-line spike discrimination and the generation of pulses marking action potentials were accomplished using a multichannel spike-acquisition system (Plexon, Dallas, TX). This system isolated a maximum of two neuron waveforms from a single FEF electrode. Pulses marking the time of isolated spikes were transferred to and stored by the REX system. During the experiment, a real-time display generated by the REX system showed the timing of spike pulses in relationship to selected behavioral events.

The location of the FEF was confirmed by our ability to evoke low-threshold saccades from the recording sites with current intensities of ≤50 μA and the match of recorded activity to established cell activity types (Bruce and Goldberg 1985). To stimulate electrically, we generated 70-ms trains of biphasic pulses, negative first, 0.2-ms width per pulse phase delivered at a frequency of 330 Hz.

Data analysis

FEF CELL CHARACTERIZATION.

We examined average cell activity during four critical epochs while the monkey performed the memory-guided delayed saccade task to determine if the cell displayed visual, delay, or premotor activity. If not enough data were available from this task, data from the visually guided delayed saccade task were used. The baseline epoch was the 200 ms preceding target onset, the visual epoch was 50–200 ms after target onset, the delay epoch was the 150 ms preceding the disappearance of the fixation spot, and the presaccade epoch was the 50 ms preceding the saccade onset. FEF cells were characterized by comparing epochs in the following manner using the Wilcoxon sign-rank test. If average firing rates during the visual or delay epochs were significantly higher than the baseline rate, the cell was considered to have visual or delay activity, respectively. If the activity during the presaccade epoch was significantly greater than the delay epoch, the cell was considered to have premotor activity. We found that FEF cells could exhibit the entire range of these activities, from having no significant levels of visual, delay, or motor activity to having significant levels of all three. These criteria are similar to those used by Sommer and Wurtz (2000).

FEF CELL RESPONSE LATENCY.

To determine the response latency of each FEF cell to a visual stimulus, we combined data from the visually and memory-guided saccade tasks. We calculated a threshold level as 2 SD above the mean firing rate during the baseline epoch. Then mean firing rates were calculated by using a sliding 50-ms window incremented in 1-ms steps starting from target onset. The midpoint of the 50-ms epoch in which the mean firing rate reached threshold was determined to be the response latency of the cell. Similar methods have been used to determine response latencies of neurons in other brain regions such as area MT (Bisley et al. 2004).

DETERMINING THE RESPONSE-FIELD SIZE.

The initial response field (RF) for a cell was determined using a joystick to position the target on the screen as the monkeys performed the delayed saccade tasks. As locations were sampled, a combination of real-time rasters and spike-density functions, accompanied with audio monitoring of multiunit activity, allowed us to find a good approximation of the center of the RF. This location and its 180° opposite were typically used to collect data for the cell-characterization analysis described in the preceding text. For the scene search tasks, however, it was essential to define the RF more rigorously to group the wide ranging saccade vectors obtained while monkeys were searching freely for the target. First, we took all saccades made during the scene search tasks and removed the first saccade of each trial as well as the last saccade made to the target. This was to eliminate any interference from the onset of the scene or the effect of the alpha-blended target on the cell's activity. The remaining saccades were grouped by saccade angle into 18 groups, each comprising a range of 20°. Average spike rates were calculated for each group from a period of 50–200 ms following the beginning of the fixation before the saccade. The average spike rate of each group was then compared with the group 180° away. If the difference between these two spike rates was >2.5 times the SD of the activity obtained from all 18 groups, then the group with the higher rate was considered part of the cell's RF. In this manner, we found cells with RF sizes with directions ranging from 20 to 60° across. For no cell did we find an RF composed of multiple groups that were not spatially continuous. For the sequence analysis (see following text), we also designated exclusion zones for the 20° sector bordering both the RF and the anti RF, the remaining areas are referred to as neutral zones (Fig. 1, B and C).

For our analyses, we did not take into consideration the amplitude of the saccades although we did exclude saccades with amplitudes <2° of visual angle and >40°. There were several reasons for this. First, the response-fields of FEF cells are not simply round with a hot spot in the center (Gaussian). Most FEF cells have RFs that are log-Gaussian, meaning that after a certain amplitude the response of the cell does not change appreciably (Bruce and Goldberg 1985). Second, taking amplitude into consideration unnecessarily reduces the data set of saccades available for analysis. A subset of data for several cells was analyzed taking amplitude into account, and the results were not noticeably different.

RECEIVER OPERATOR CHARACTERISTIC DISCRIMINATION TIME.

Receiver operator characteristic (ROC) analyses are often used in decision-making and target-selection studies to determine the time at which a neuron's activity differentiates to reflect a decision, or the presence of a target (Horwitz and Newsome 2001; Kim and Shadlen 1999; McPeek and Keller 2002a; Sato et al. 2001; Thompson et al. 1996). We generated ROC curves from spike trains produced during the fixation period before saccades made into and away from the cell's RF. Data were excluded from analysis if the previous saccade was made within 20° of the RF or its opposite (Fig. 1C). Figure 2A shows a saccade that was excluded for this reason.

Fig. 2.

Saccades excluded from analysis. Activity recorded at the colored fixation spots were excluded for varying reasons. A: although saccade A is toward the response field, activity recorded while fixating at the gray spot is excluded because the vector of the preceding saccade also was directed toward the response field. B: saccade B was excluded in our 2nd analysis because due to the size of frontal eye field (FEF) response fields, the portion of the scene located around the gray spot was in the cell's response field for 2 successive fixation periods (blue and green spots). Therefore early increases in activity while fixating at the green spot could have been due to prior activation during the previous fixation period. C: 2nd goal activity recorded while fixating at the location marked by the blue spot was excluded because although the 2nd goal (the endpoint of vector C) was toward the response field, the 2nd saccade (saccade B) was as well.

The area under the ROC curve (AUC) is a measure of the degree to which the spike rates at a given time differ depending on the direction of the upcoming saccade. To determine the earliest time at which this differentiation occurred, the AUC was obtained for every 5 ms starting from 15 ms before the beginning of a fixation period until the onset of the saccade. We then used a bootstrap analysis similar to Horwitz and Newsome (2001) to evaluate the significance of the AUC values. The preceding analysis was repeated 2,000 times with random assignment of each saccade to one of two saccade direction groups before each repetition. Next, at each time point, we compared the “true” AUC to the 2,000 AUCs obtained from shuffling saccades between the groups. If the true AUC was >1,900 (95% confidence level) of the “false” AUCs for 10 consecutive intervals (50 ms), we assigned the time of the first of those 10 AUCs as the ROC prediction time (PT).

RESULTS

FEF cell-predictive activity for the upcoming saccade during a scene search

We recorded from 52 FEF visual (n = 37) and visuomovement (n = 15) neurons from two rhesus monkeys (M14, n = 31; M15, n = 21) as they searched natural scenes for an embedded target. ROC analysis determined that the vast majority of cells (49/52, 94%) strongly modulated their activity during the scene search task depending on the direction of the upcoming saccade. The remaining analyses consider these, or a subset of these 49 cells. The ROC analysis allowed us to determine when the cell's activity predicted the direction of the upcoming saccade. Figure 3 shows results from representative FEF visual and visuomovement cells. Both cells predicted the direction of the upcoming saccade at the beginning of the fixation period as indicated by the vertical green line. Overall the ROC prediction time for both visual (40 ± 49 ms after the beginning of fixation, mean ± SD) and visuomovement cells (33 ± 43 ms) occurred early in the fixation period. A t-test revealed no significant difference in these discrimination times (P value = 0.62), and as a result, unless otherwise noted, subsequent analyses combine data from both visual and visuomovement cells. The mean prediction time for all 49 cells was 38 ± 47 ms after the start of fixation. For the ROC analysis of individual neurons with predictive activity, the minimum number of combined on- and off-direction saccades was 27, the maximum was 532.

Fig. 3.

Early prediction times during the scene search task. A: representative visual cell. Rows 1 and 2 show spike rasters and spike density curves for mean firing rates during the fixation period prior to saccades made into the response field (blue), and into the anti response field (red). Row 3 compares the firing rates between the 2 conditions and indicates the receiver operator characteristic (ROC) prediction time (green line). Black vertical lines indicate the beginning of the fixation period before the saccade (red dots in rasters). Activity occurring after the mean saccade latency shaded in gray. Row 4 displays the same cell's activity during the memory-guided saccade task. Activity is aligned by target onset (left) and saccade onset (right). Onset of visual response indicated by vertical green line. The cell fires strongly after target onset but not before the saccade. B: representative visuomovement cell. C: comparison between the mean visual response latency and the ROC prediction times for visual and visuomovement cell. Mean visual response latency was significantly greater than the ROC prediction times for either type of FEF cell.

The mean prediction time was earlier than expected. In fact, it was less than most reported visual latencies for FEF activity (92 ms: Bruce and Goldberg 1985; 40–80 ms: Schall 2001; Thompson et al. 1996; 75 ms: Schmolesky et al. 1998). To make our own direct comparison between saccade prediction times in the scene search task and the visual latencies for the same FEF neurons, we employed a sliding 50-ms window on activity obtained from both visually and memory-guided delayed saccade tasks and compared mean firing rates during successive periods to the baseline firing rate before target onset (see methods). The results can be seen in Fig. 3C. The mean response latency was significantly longer than the timing of the predictive activity reported in the preceding text (mean = 58 ms; t-test, P value = 0.0141) but within the range of previously reported FEF cell visual latencies. In fact, during the scene search task, nearly a quarter of cells (12/49) discriminated the direction of the upcoming saccade before the beginning of the fixation period that preceded it. Our ROC analysis began 15 ms before the start of fixation periods because we didn't want to include activity generated when the eye was at a prior fixation location. The outcome of this was that the earliest statistically detectable prediction time was −15 ms. However, it was clear from looking at the spike density plots similar to Fig. 3 that many of the 12 cells with prefixation prediction times began their discrimination much earlier than 15 ms prior to fixation. Thus our calculated mean prediction time might in fact be later than it actually is. We consider predictive activity during prior fixation periods separately in the following text.

One explanation for this finding is that owing to the large size of FEF receptive fields, objects may stay in a cell's receptive field for two successive fixation periods. If, in this situation, a saccade is made into the response-field after the second fixation period, early increases in activity could be due to visual responses to the content of the response-field during the first fixation period. To avoid this, we performed the same ROC analysis after removing all eye movement sequences that included saccades into the RF where the endpoint of the saccade initiated from the previous fixation location also fell within the cell's RF. An example of a saccade removed for this reason can be seen in Fig. 2B. With these saccades removed, the mean prediction time for all cells increased to 56 ± 53 ms and was significantly greater than the original prediction time determined without this control (paired t-test, P < 0.001) but did not differ from the visual response latency of the cells (t-test, P value = 0.935). For the visual and visuomovement cell types, the mean prediction times were 56 ± 56 and 57 ± 50 ms. Thus despite removing the contaminating factor, activity predicting the vector of the next saccade exists coincident with the earliest FEF visual responses. These results strongly suggest that activity of these FEF cells is driven by extra-retinal components that begin to differentiate before information in the cells' RFs reaches the FEF, and precludes a selection process based solely on that visual information.

Advanced FEF cell predictive activity before two successive saccades

Perhaps of equal importance to the first finding of early predictive activity for the upcoming saccade was that the extremely early prediction times initially observed were in part driven by activity during a prior fixation (note that the preceding saccade was not toward the RF, see methods and Fig. 2A). This led us to examine the ways in which activity during fixations might predict the outcome of future saccades. We looked for two types of predictive activity during the fixation period prior to two successive saccades during the scene search task. First we determined if the activity during fixation could predict the vector of the second saccade of a pair of successive saccades. This is referred to as second saccade predictive activity. Gray circles in Fig. 4A depict fixation periods preceding pairs of successive saccades used in this analysis. We compared cases in which the second saccade of a pair was either into or away from the cell's RF. Next we determined if the activity during fixation could predict the spatial location or goal of the second saccade. The position of the endpoint of the second saccade is referred to as the goal of the two-saccade sequence. Cells that predicted the goal of the sequence were said to have second goal predictive activity. Gray circles in Fig. 4B identify fixation periods preceding second goals into and away from the cell's RF. Together, second saccade and second goal activity are referred to as advanced predictive activity. In many cases, both the second saccade and the second goal had similar vectors, and those pairs of saccades were not included in the analysis. Only sequences in which both the first saccade in the sequence and the goal of the sequence fell in neutral fields (green areas in Fig. 1, B and C) were included in the second saccade analysis (Fig. 4A, solid blue and red arrows), whereas only sequences in which both saccades landed in neutral fields were included in the second goal analysis (Fig. 4B, dashed blue and red arrows). An excluded saccade pair that did not meet these criteria can be seen in Fig. 2C. Also excluded from the analysis were sequences that included the first or last saccade of a trial. For the ROC analysis of cells with second saccade activity, the minimum number of combined sequences with on- or off-direction second saccades was 12, the maximum was 131. For the analysis of cells with second goal activity, the range of sequences used was 10–72.

Fig. 4.

Second saccade and second goal determinations. Solid arrows indicate examples of 2 successive saccades. Often a single trial yielded multiple saccade pairs for analysis. Activity obtained during the fixation period preceding the saccade pairs was analyzed for predictive activity (gray circle). A: 2nd saccade analysis. Instances in which the 2nd saccade was directed into the response field (top, solid blue arrow) were compared with cases in which the 2nd saccade was directed away from the response field (bottom, solid red arrow). These sequences were included because vectors of both the 1st saccade (solid green arrows) and the 2nd goal (dotted green arrows) fell in neutral areas far from the response field or its opposite direction. B: 2nd goal analysis. Instances in which the 2nd goal was within the response field (top, dotted blue arrow) were compared with cases in which the 2nd goal was located in a direction opposite to that of the response field (bottom, dotted red arrow). These sequences were included because vectors of both the 1st saccade and 2nd saccades (solid green arrows) fell in neutral areas far from the response field or its opposite. The circular inset in the lower right corner depicts the cell's response field (RF) in a manner identical to Fig. 1B.

Figure 5 shows representatives of four types of cells we encountered each displaying a different pattern of predictive activity. The top row depicts a cell that exhibited second saccade but not second goal predictive activity. Overall 12% (6/49) of cells followed this pattern. The second row shows a cell that predicted the spatial location of second goals but not the vector of second saccades. This type of cell comprised 22% (11/49) of the cells we tested. Another 20% (10/49) of cells were similar to the profile of the cell shown in row three and modulated their activity to indicate the direction of both second saccades and second goals. The remaining cells (22/49, 45%) did not have activity predictive of the second saccade or goal (Fig. 5, 4th row). It is clear from these data that at least two subpopulations of cells exist, those that predict the future second goal and/or second saccade (27/49, 55%), and those without any type of advanced predictive ability (22/49, 45%). Considering all cells that included one or both types of advanced predictive activity, we found a slightly higher prevalence of second goal over second saccade activity (21/49, 43% vs. 16/49, 33%).

Fig. 5.

Types of 2nd saccade and 2nd goal predictive activity. We found cells that displayed 2nd saccade and/or ^2nd goal predictive activity as well as cells that did neither. Left. firing rates during fixation periods in which the 2nd saccade was directed toward (blue) and away from (red) the response-field. Right. firing rates during fixation periods in which the 2nd goal was located either within (blue) or at a location opposite to the response-field (red). Black vertical line indicates the beginning of the fixation period preceding the pair of saccades. Vertical green line mark the time at which ROC analysis indicated that advanced predictive activity occurred. Time after the mean latency of the 1st saccade shaded in gray. Row 1: a cell that could predict only the 2nd saccade of a sequence but not the goal. Row 2: a cell that could only predict the 2nd goal of a sequence but not the 2nd saccade. Row 3: a cell that could predict both the 2nd saccade and the 2nd goal of a sequential pair of saccades. Row 4: a cell that did not display any advanced predictive activity.

While these results are intriguing, they pose an interesting problem. If activity during a fixation period evolves to predict the next saccade as well as the saccade vector or goal that will follow the next saccade, how does the system “know” which saccade to make? To address this issue, for those cells with second saccade and/or second goal activity, we examined the time at which advanced predictive activity occurred during the fixation period and compared it to the prediction time observed before the upcoming saccade. As noted in the preceding text, when all 49 cells were included in the analysis, the mean prediction time for the upcoming saccade was 56 ms into the fixation period. However, when we only include the subpopulation of cells with advanced predictive activity for second saccade and/or goal, the mean prediction time before the upcoming saccade drops to 34 ms. On average, the activity of this same subpopulation of cells predicted the second saccade or goal later in the fixation period at 85 ms for second saccade and 86 ms for second goal (Fig. 6A). A one-way ANOVA revealed a significant difference between these three means (F = 6.09, P = 0.004). Post hoc analysis revealed that both second saccade and second goal activity occurred significantly later in the fixation period than activity predicting the upcoming saccade but were not different from each other. These results indicate that FEF vector and goal-related activity is modulated sequentially during fixation periods. Early during the fixation period, activity of advanced predictive cells reflects the vector and spatial goal of the upcoming saccade (for the upcoming saccade, these are the same), while later in the fixation period, activity evolves to indicate the vector and/or spatial goal for the second saccade in the sequence. Thus the timing of differential activity might be used to determine the order of successive saccade vectors and goals.

Fig. 6.

Timing of predictive activity. A: comparison between prediction times for upcoming saccades and future saccades. Activity that predicts the upcoming (1st) saccade occurs significantly earlier in the fixation period than that of the 2nd goal or 2nd saccade (*). B: prediction times before the upcoming saccade. When FEF cells are divided into those with advanced predictive activity and those without, a clear distinction can be seen. FEF cells that combined both types of advanced predictive activity indicated the direction of the upcoming saccade significantly earlier than FEF cells that did not display advanced predictive activity (*). Cells with only 2nd goal activity also showed earlier prediction times, but this did not reach significance.

The lower mean prediction time of 34 ms for advanced predictive cells suggests that the remaining cells that did not have advanced predictive activity signal the target for the upcoming saccade later in the fixation period. To confirm this, we compared the prediction time before upcoming saccades for FEF cells with and without advanced predictive activity. FEF cells with advanced predictive activity did indeed differentiate activity much earlier than other FEF cells (means = 34 ms and 69 ms, respectively, t-test P value = 0.013). Thus the original prediction time of 56 ms was actually an average derived from two subpopulations of cells that increase their activity at different times during the fixation period to indicate the direction of the upcoming saccade. When the prediction times from the different cell types depicted in Fig. 5 were compared separately with a one-way ANOVA, post hoc analysis showed cells that exhibited both types of predictive activity discriminated the upcoming saccade much earlier than cells without advanced predictive activity (Fig. 6B; mean = 23 ms; F = 4.29, P value = 0.02). Cells showing only second goal activity tended to indicate the upcoming saccade earlier (mean = 40 ms), but this difference was not significant. Although the second saccade group also showed an early mean prediction time (mean = 46 ms), it was not included in the analysis due to the small sample size. These results indicate that cells that predict the outcome of two successive saccades begin to indicate the outcome of the first saccade earlier than those cells that can only predict the next saccade.

Behavioral evidence for search strategy

To better understand the underlying function of the neuron activities we observed, it is necessary to evaluate the strategies the monkeys used to perform the scene search task. The design of the task ensured that the monkeys' saccades were self-guided, but this did not guarantee that the movements were part of an active visual search versus being made to locations chosen at random. In addition, we could not assume that the monkey identified the target when it appeared in the peripheral field of vision or if the target needed to be foveated to be identified. To distinguish between these possibilities, we examined the latency distributions of saccades made during this task. The purpose was to look for evidence suggesting that the monkey identified the target before a saccade was made to it and to look for variations in saccade latency during the trial that would be consistent with latency patterns seen in active visual search.

We looked first at the latencies of the final saccades of each sequence that landed on the target. These saccades consistently occurred at shorter latencies than those made while the monkey was searching the scene before the final saccade was made. This difference was statistically significant with saccades toward the target having an overall mean saccade latency of 207 ms, while other saccades had a mean latency of 241 ms. (P value < 0.001). Although saccade latencies tended to vary slightly day by day depending on the monkeys' motivation, we observed only one instance in which the mean latencies of a given session did not follow this pattern. Figure 7 shows the mean latencies calculated for each recording session. Regression analysis shows a clear linear relationship such that as saccades toward the target increase in latency, so do those landing on other of portions of the scene (R² = 0.84). The slope of the regression line was 0.61, and all but 1 point lies below the dotted x = y line, indicating that saccades to the target fly had shorter latencies. This finding suggests that the monkeys identified the location of the target before initiating the final saccade to fixate it.

Fig. 7.

Shorter saccade latencies to target. Saccade latencies toward the target are plotted against saccade latencies toward other portions of the scene during the search. Each black dot represents mean latency data from 1 recording session. Regression line in solid black. Dotted line indicates expected values if there were no differences in latency between the 2 conditions.

This overall latency trend does not preclude the possibility that some additional factor such as the ordinal number within a trial sequence or the amplitude of the final saccade to the target is responsible for the shortened latency of saccades to the target. Therefore we compared target-saccade latencies to scene-saccade latencies according to the saccade number within a trial (Fig. 8A). We found that while saccade latencies to the scene increased throughout the trial, those to the target remained relatively the same and after the initial first five saccades were consistently significantly shorter that those saccades made to portions of the scene without the target (t-test, alpha = 0.01). While the monkeys may have increased the amount of time fixating between saccades in an effort to examine the scene more carefully when they could not quickly find the target, it is clear that when they did find the target, saccades were made rapidly. The increase in latency for scene-directed saccades as the trial progresses could represent a gradual change in strategy to increase time spent inspecting portions of the image as well as an increase in the number of re-fixations of locations that had been fixated earlier in the trial. For human subjects viewing natural images, these re-fixations have been shown to have longer durations (Hooge et al. 2005).

Fig. 8.

Latency and amplitude for saccades to scene and target. A: comparison of latency of saccades to the target vs. saccades to nontarget portions of the image as a function of saccade order in the trial. *, number of saccade in trial where latency of saccade to target was significantly less than saccade to a nontarget part of the scene. B: comparison of saccade latencies and amplitudes for target and nontarget saccades. *, saccade amplitudes where latency of saccades to nontarget parts of scene were significantly longer than saccades to the target. Vertical bar indicates SE.

A comparison of saccade amplitudes revealed that short-latency saccades to the target were not simply due to a limited distribution of amplitudes for saccades to target versus saccades to the scene. Figure 8B shows that while saccades with amplitudes of 3–5° have shorter latencies when directed toward the target (t-test, alpha = 0.01), this was also the case for much larger saccades between 17 and 33° (significance was only reached up to saccades 25° in amplitude). Interestingly, middle ranged saccades between 7 and 17° appeared to have a fairly constant latency irrespective of amplitude or target of the saccade. Larger saccades between 17 and 31° appear to get longer in latency if directed toward the scene and shorter in latency when directed toward the target. The number of saccades >31°, both to the scene and to the target, was significantly much less, accounting for greater variability, and statistical analysis was unable to determine any trends. It is clear from these results that the reduced latency of saccades directed toward the target was not simply the result of a limited range of saccade amplitudes or chance landings near the target. We also looked for a possible gradation of saccade amplitude across the duration of the trial but did not find any correlation between amplitude and ordinal number in the trial. The tendency toward shorter latency for saccades made to the target may be analogous to the findings of Harwood and colleagues (2008), who found that human saccade latencies were shorter when attention was directed to a smaller stimulus feature regardless of the distance of the feature from the fovea. This may be similar to the search behavior in our paradigm where the final saccade is made to a relatively small target versus earlier saccades that are directed to larger portions of the scene so that they may be examined. Together these analyses indicate that the monkeys identified the search target before a final saccade was made to foveate it and that the distributions of latencies observed were consistent with those seen in human subjects during active visual search.

DISCUSSION

We examined the changes in activity in FEF visual and visuomovement cells during a scene search task that embedded a target in a natural image. Virtually all of these cells modulated their activity during scene search to predict the direction of upcoming saccades at latencies equal to or less than visual latencies determined in visually and memory-guided saccade tasks. In addition, the activities of a subpopulation of slightly more than half of these cells predicted the saccade vector or spatial goal of the saccade that would follow the upcoming saccade. A unique aspect of these findings is that they were observed while monkeys made self-guided eye movements during the search of a natural image. Earlier studies, where one or more saccades were directed to target light spots or simple geometric shapes, established the involvement of FEF cells in predictive remapping of visual stimuli, the maintenance of a map of target salience or saccade probability and the rapid early selection of saccade targets for corrective saccades (Balan and Ferrera 2003; Goldberg and Bruce 1990; Murthy et al. 2007; Thompson and Bichot 2005; Thompson et al. 2005a; Tian et al. 2000; Umeno and Goldberg 1997). This report extends these findings to a more natural behavior where choice of saccade targets is directly motivated and controlled by the subject. Our findings will be discussed in light of these earlier reports.

Predictive remapping of visual activity

Only a limited number of studies have examined monkey oculomotor system activity during the performance of tasks where multiple saccades are made. One of the classic examples of these tasks is the double-step task where, while a monkey maintains fixation, two target lights are flashed in quick succession. Both target lights are extinguished before the monkey can make a saccade, and the monkey is rewarded for making a pair of accurate saccades to the target locations in the order they were presented. Hallett and Lightstone (1976) demonstrated that human subjects are able to make a sequence of spatially accurate saccades to briefly flashed targets, and monkeys are also able to correctly perform the double-jump task (Mays and Sparks 1980). When the subject completes the first saccade, the location where the second target landed on the retina is no longer sufficient to make an accurate saccade to the target. The oculomotor system must also take into account the eye movement made to the first target. By subtracting the vector of the first saccade from the retinotopic location of the second target, the system can map the true spatial location of the second target. In a remarkable discovery, Mays and Sparks (1980) described a class of cells in the SC they named quasi-visual (QV) cells. Although it's unlikely that these cells performed the vector subtraction themselves, their activity represented the outcome of this process and provided a signal that coded the spatially correct location for the second target. The name quasi-visual reflects the combination of both sensory visual and extra-retinal efference copy (corollary discharge) input required to form the signal carried by these neurons. Goldberg and Bruce (1990) demonstrated that FEF cells with visual activity exhibited properties similar to the QV cells of the superior colliculus by signaling the correct spatial location of the second saccade target in the double-jump task. Using a task similar to the double-jump task with the main difference that it did not require that a saccade be made to the second stimulus light, Goldberg and colleagues found that the lateral intraparietal cortex (LIP), the FEF, and the SC all show evidence for the remapping of retinotopic location of the stimulus to produce a spatially accurate map of stimulus location (Duhamel et al. 1992; Umeno and Goldberg 1997; Walker et al. 1995). These results indicate that LIP, FEF, and SC are all capable of contributing to a process that is essential to control a sequence of saccades where future target positions must be updated after each movement in the sequence. Tian and colleagues (2000) looked at the process of updating target position in the FEF with a triple-step task where three target lights were flashed during the initial fixation period and the monkey made a sequence of three saccades to the remembered locations of the target flashes. This allowed them to test whether FEF QV cells coded exclusively for the spatial location of the next saccade in the sequence or whether separate populations of QV cells coded for the locations of all of the targets remaining in the sequence—a map of target positions that would require updating after each saccade. Their results supported the latter possibility, suggesting that when the monkey makes a sequence of saccades, distinct populations of FEF QV cells code for the targets of each saccade in the sequence. The corollary of this is that for each saccade in the sequence, there must be a remapping to account for the movement and an activation of new populations of QV cells that code for the remaining targets.

The experiments we describe in this report have extended the investigation of FEF activity during generation of multiple saccades to a natural-image search task where the selection of targets for a series of saccades is under the volitional control of the monkey. All but a few of the visual and visuo-movement cells that we studied predicted the target of the next saccade before new sensory visual input from the point of fixation could be processed. About 25% of these cells predicted the target for the next saccade before the end of the prior eye movement. Within our population of cells that predicted the target of the next saccade, we found a subpopulation of cells that display two forms of advanced predictive activity for the saccade that will follow the upcoming saccade. Activity during the fixation period before two successive saccades indicated the vector and/or spatial goal of the second saccade. The goal-related activity is similar to that reported when monkeys performed a triple-saccade task (Tian et al. 2000). A model for the generation of saccade sequences predicts that within the FEF there are neurons that encode for target locations in sequence, storing them in memory similar to the cells with second goal activity that we found (Mitchell and Zipser 2003). FEF activity related to the vector or goal of the second saccade of a double-saccade task has been reported to begin immediately after the first saccade (Goldberg and Bruce 1990), but during our scene search task, we found many cells actually began such activity before the beginning of the first saccade in the sequence. The FEF has also been shown to predict the future presence of a spot of light in a neuron's RF (Umeno and Goldberg 1997) or the memory trace of a prior cue that will be the target for a future saccade (Balan and Ferrera 2003). In our paradigm, every saccade brings a new visual stimulus into the receptive field of every visual and visuomovement neuron in the FEF. Because all of the cells that showed predictive activity had visual responses, it is reasonable to interpret this activity as the product of a shifting receptive field effect. It's important to emphasize that in our experiments, the shifting receptive fields are linked to making a saccade to the contents of the receptive field. Although each saccade provided new visual input to a neuron's RF, the increases in activity were predictive of future saccade vectors and spatial goals and thus were a part of a saccade planning process.

The subpopulation of cells with advanced predictive activity differed from other FEF cells not only in their predictive ability but also in the timing in which they indicated the upcoming saccade. This difference, and the existence of the two subpopulations, may account for some of the FEF's involvement in the control of both upcoming saccades and future ones. Cells without advanced predictive activity modulated their activity to indicate the upcoming saccade later during the fixation period than those cells with advanced predictive activity (69 vs. 34 ms after beginning of fixation). This reveals an organization in which advanced predictive cells specify the target of the upcoming saccade early during fixation. Later these cells begin to specify the goal or vector for the saccade that will follow the upcoming saccade while cells without advanced predictive activity begin to indicate the direction of the upcoming saccade. This re-introduction of a signal for the upcoming saccade may be another way the system reinforces the proper order of saccades (Fig. 9). It is also possible that cells without advanced-predictive activity are more closely linked to movement cells involved in the actual saccade generation process, although we found the proportion of visual and visuo-movement cells to be roughly equal between the two subpopulations with and without advanced predictive activity (Fig. 10). Support for the late specification by advanced predictive cells of the spatial goal of the second saccade comes from a study in which the left and right FEF were electrically stimulated with a delay of 30–250 ms between stimulus trains (Fujii et al. 1998). The result was a sequence of two saccades in which the first went to the movement field of the first stimulated site, and the second went to a location within the movement field of the second site referenced to the eye position during stimulation. That is to say, the resulting sequence of saccades indicated that the second stimulation acted as an artificial second goal activity not second saccade activity.

Fig. 9.

Relative timing of FEF visual and visuomovement cell activity for the generation of a sequence of 2 saccades. After the start of fixation, at relative time point A, cells with advanced predictive activity are the first to signal the direction of the upcoming saccade (S1). Later at time point B, cells without advanced predictive activity also signal the direction for S1. Later in the fixation period at time point C, advanced predictive cells signal the spatial goal (G2) and saccade vector (S2) for the eye movement that will follow the upcoming saccade. The relative times of these activities are based on the values illustrated in Fig. 6 and discussed in the text.

Fig. 10.

Distribution of FEF cells with and without advanced predictive activity. This diagram shows the relative numbers of visual neurons with no motor activity and visuomovement neurons with motor activity. The distribution of these 2 cell types across the groupings of cells with and without advanced predictive activity was roughly the same.

The next problem to resolve in this process is how the second saccade goal and vector signals are interpreted to indicate the direction of the upcoming saccade in the sequence. As Fig. 4 demonstrates, depending on the direction of the first saccade, the directions of the second saccade vector versus goal can be very different. This means that during any given fixation period, there could be at least three different focuses of activity within the FEF's saccade representation. The highest level of activity would be at the site representing (in an oculocentric reference frame) the target of the next saccade to be made. All of the visual and visuomovement neurons examined in this study demonstrated they would contribute to this activity when the target for the saccade fell within their RF (see also: Burman and Segraves 1994b). In addition, there could be as many as two additional loci of activity at sites representing the saccade vector and spatial goal for the saccade that will follow the upcoming saccade. Our results suggest that one site would consist of cells with second saccade vector as well as cells with combined second saccade vector and goal activity signaling the vector of the second saccade in the sequence, the other site would consist of cells with second goal activity and cells with combined activity signaling the vector and spatial goal of the second saccade. Despite these separate loci of activity, this does not mean there is an ambiguity in the signals representing the target for the second saccade, rather the multiple sites are a consequence of the second saccade target being represented in different reference frames. We think it is most likely that around the time of the first saccade, the predictive remapping process results in the cessation of activity at the second goal locus and the validation and strengthening of activity at the second vector locus. This strengthened locus of activity would then be in register with appropriate movement cells to generate the next saccade in the sequence. It is entirely possible that cells with second saccade vector activity that we observed did not comprise a fundamentally different class of neurons separate from those with second saccade goal activity. In fact, a number of cells modulated their activity to indicate the direction of both second saccades and second goals. Instead second saccade vector cells may be part of the subpopulation of cells with advanced predictive activity that show the effects of predictive remapping at an earlier time than do the cells identified with second saccade goal activity alone.

Salience and saccade probability

There are many factors working together to direct our gaze when we scan or search a natural scene. Models that rely on salience maps to predict eye movements do well when subjects freely view images and appear to be relevant for both humans and rhesus monkeys (Berg et al. 2009; de Brecht and Saiki 2006; Itti and Koch 2000, 2001; Peters et al. 2005). However, it has been known for some time that bottom-up influences cannot entirely account for scan paths, especially when people are not freely viewing a scene. Asking subjects to evaluate a scene in different ways, or to memorize its content, results in scan paths that focus on specific elements of the scene and ignore others (Hayhoe and Ballard 2005; Yarbus 1967). In effect, cognitive control overrides the automatic bottom-up saliency of objects and makes objects that match an internal representation of the target more salient (Pomplun 2006). Our search task elicited this form of top-down control as monkeys searched scenes for the embedded target. Our results show that changes in a FEF cell's activity that predict future saccades are likely to be based on internal plans to make saccades or shift attention to particular locations. As mentioned in the preceding text, in our paradigm, every saccade brings a new visual stimulus into the receptive field of every visual and visuomovement neuron. Under these conditions, visual elements in the image with a high level of saliency may increase a cell's activity; possibly even before the eye movement that places the salient stimulus in the receptive field. We are currently investigating this possibility (Fernandes et al. 2009). For this report, however, our findings depend entirely on where the monkey moved its eyes.

For the oculomotor field, the term salience carries more than a pure bottom-up sensory meaning to include top-down influences important for guiding eye movements under task conditions (Thompson et al. 2005a). Even though it has been shown that the representation of salience or saccade probability in FEF can be dissociated from actual saccade production, we cannot make that separation in our experiments (Bichot et al. 2001; Thompson et al. 1997, 2005b). We have no independent measure of the monkey's intent. We can examine the data only with respect to where eye movements are made. Nevertheless our results are entirely consistent with and lend support to the idea of a target salience or saccade probability map in the FEF where during each fixation, the locus of highest activity specifies the vector of the upcoming saccade. This locus of highest activity would be accompanied by other less robust loci of activity arising from cells with advanced predictive activity representing the goal of the second saccade as well as a remapped spatial goal signal in the form of an oculocentric second saccade vector signal.

Rapid target selection

Becker and Jürgens (1979) demonstrated that under conditions where the delay between first and second target light is sufficiently short, saccades can be programmed in parallel in the double step task. Murthy and colleagues (Murthy et al. 2001, 2007) have demonstrated that a similar process takes place in a search-step task where the search target is moved to a new location at a variable delay before the beginning of the saccade to the original target location. As the delay between target appearance at its original location and its step to a new location increased from 30 to 140 ms, there was increasing probability that a saccade would be made to the first target location followed by a corrective saccade to the new location of the target. Under these conditions, FEF visual, visuomovement, and movement neurons all showed increases in activity that were preparatory for the corrective saccade at or even before the end of the first saccade that was made in error to the original location of the target. This activity is analogous to what we observe in the scene search task in that the changes in activity of visual and visuomovement neurons occur before new visual input at the end of the error saccade is available. In the search-step task, this provides a rapid mechanism for generating corrective saccades. Murthy and colleagues (Murthy et al. 2007) report mean ROC discrimination times of 40 ms after the end of the first saccade for visual neurons and 60 ms for visuomovement cells. This is comparable to the discrimination times we found of 56 ms for visual and 57 ms for visuomovement cells. Similar activities have been observed in the monkey SC by McPeek and Keller (2002b), who observed increases in activity of visuomovement neurons analogous to the second goal activity seen in our experiments. In the scene search task of our experiments, we have not developed a way to distinguish when the monkey is making a corrective saccade or an abrupt change in plans regarding where to make the next saccade. The prevalence of early predictive activity that we see suggests that it is part of the normal saccade generation process and is not present only when abrupt changes in saccade target are introduced.

Planning saccade sequences during natural image search

The processes of predictive visual remapping, maintenance of salience and saccade probability maps, and the rapid correction of error saccades are all components of a saccade planning process. Our analysis of saccade latencies during scene search indicate that the monkey identified the target before it was foveated and revealed distributions of latencies that were similar to those generated by humans engaged in active visual search (Harwood et al. 2008; Hooge et al. 2005). These findings infer the presence of a plan for future movements beyond the next movement in the sequence. Whether or not the monkey makes a plan for multiple saccades at a conscious level is unknown. Nevertheless our results along with those described in the preceding text demonstrate FEF activities that comprise a movement plan that includes the next saccade as well as the one that will follow it. It is unknown whether or not this plan extends further into the future. Clearly the FEF does not function alone in this process. The supplementary eye field, for example, has been implicated in saccade ordering in learned sequences of saccades (Histed and Miller 2006; Isoda and Tanji 2003; Lu et al. 2002).

There is a rich history of studies to reveal if and how sequences of multiple movements are planned. Early studies of rapid movement sequences focused on behavioral evidence for planning, arguing that increases in reaction time for longer sequence lengths in speech and typing experiments were due to advanced planning (Rosenbaum et al. 1983, 1984; Sternberg et al. 1978). Advanced planning theories argue that motor programs for movement sequences are constructed and stored before motor execution begins and that the latency for the first movement reflects the time to retrieve information from a stored plan (Henry and Rogers 1960).

Studies of sequences of saccades in humans have also shown increases in latency with sequence length. In a study by Inhoff (1986); human subjects were required to make one to three saccades after the appearance of a visual cue. The paradigm was run under two different conditions. In the parafoveal cue condition, asterisks on the screen after the go signal served as targets for the saccades, and saccades could be programmed and generated serially. In the no cue condition, subjects were told the number of saccades to make before a block of trials began and had to maintain an internal representation of the motor program in memory. Saccade latency increased only in the no-cue condition, suggesting that saccade sequences can be programmed and executed by different mechanisms. Shortly after the Inhoff study, Zingale and Kowler (1987) reported a linear increase in first saccade latency as the number of saccades in the sequence increased. In contrast, other studies of human saccades have failed to show a response complexity effect between sequences of single and multiple saccades (Pratt et al. 2004; van Donkelaar et al. 2007), most likely due to differences in tasks used versus those used by Inhoff, Zingale, and Kowler. These differences emphasize that different tasks may recruit different motor sequence planning mechanisms.

The structure of the scene-searching task attempted to approximate real-world conditions. The design of pop-out oddball discrimination tasks forces the choice of next saccade to take place after the search array appears and the target has been identified. No plan can exist before fixation starts or even while fixating before array onset. Saccades in the real world, however, are not made in isolation. During natural visual search, visual processing is continuous, and what lands in a cell's receptive field may have already been identified during a previous fixation. Under these conditions, plans for future eye movements may develop continuously within subpopulations of FEF neurons with the timing and strength of activity modulation playing a crucial role in determining the order and direction of future eye movements.

GRANTS

This work was supported by the National Eye Institute Grants EY-08212 and EY-07128.