A Neural Representation of Sequential States Within an Instructed Task

Abstract

In the study of the neural basis of sensorimotor transformations, it has become clear that the brain does not always wait to sense external events and afterward select the appropriate responses. If there are predictable regularities in the environment, the brain begins to anticipate the timing of instructional cues and the signals to execute a response, revealing an internal representation of the sequential behavioral states of the task being performed. To investigate neural mechanisms that could represent the sequential states of a task, we recorded neural activity from two oculomotor structures implicated in behavioral timing—the supplementary eye fields (SEF) and the lateral intraparietal area (LIP)—while rhesus monkeys performed a memory-guided saccade task. The neurons of the SEF were found to collectively encode the progression of the task with individual neurons predicting and/or detecting states or transitions between states. LIP neurons, while also encoding information about the current temporal interval, were limited with respect to SEF neurons in two ways. First, LIP neurons tended to be active when the monkey was planning a saccade but not in the precue or intertrial intervals, whereas SEF neurons tended to have activity modulation in all intervals. Second, the LIP neurons were more likely to be spatially tuned than SEF neurons. SEF neurons also show anticipatory activity. The state-selective and anticipatory responses of SEF neurons support two complementary models of behavioral timing, state dependent and accumulator models, and suggest that each model describes a contribution SEF makes to timing at different temporal resolutions.

INTRODUCTION

In a wide range of experimental paradigms, performance improves if a subject can anticipate when an instructional cue will become available, and expert performance is frequently accompanied by anticipatory or short-latency movements (Miyashita et al. 1996) which optimize the rate of reward (Glimcher 2004). The two cortical oculomotor areas of interest in this report, lateral intraparietal area (LIP) and supplementary eye fields (SEF), have both been implicated in behavioral timing (Campos et al. 2009; Leon and Shadlen 2003; Ohmae et al. 2008). Anatomical connections suggest that SEF could be directly involved in specifying when a saccade should occur (Shook et al. 1990), and SEF microstimulation can trigger an already planned movement (Fujii et al. 1995; Missal and Heinen 2004). The timing of SEF microstimulation, however, has to be in the appropriate interval, otherwise it will delay the reaction time and instead facilitate fixation (Isoda 2005), cautioning that the temporal responsibilities of SEF are not limited to contributing to saccade production but may also enhance fixation behavior when that is appropriate (Bon and Lucchetti 1990). SEF is also involved in the correct ordering of multiple saccades (Gaymard et al. 1990; Histed and Miller 2006; Isoda and Tanji 2002, 2003; Lu et al. 2002).

LIP neurons are generally thought to be involved in specifying the spatial location of eye movement targets and salient cues (Andersen 1995; Andersen et al. 1985, 1997; Gottlieb et al. 1998). In contrast to the behavioral ordering disruptions resulting from an SEF lesion (Gaymard et al. 1990) or reversible inactivation (Histed and Miller 2006), temporary LIP inactivation results in deficits of saccade metrics (Li and Andersen 2001). LIP activity, however, correlates with eye movement start times (Ipata et al. 2006) and exhibits a slight anticipatory rise before predictable cue presentations (Colby et al. 1996), revealing access to timing information. Recent studies have further proposed that LIP neurons might themselves represent the passage of time during motor planning (Janssen and Shadlen 2005; Leon and Shadlen 2003). It remains unclear, however, if the representation of timing information in LIP is independent of eye movement planning.

In this study, we characterize the responses of LIP and SEF neurons while monkeys performed a memory-guided saccade task. The task featured variable time intervals between events and a dense sampling of visual and motor space, which allowed us to separately assess spatial and temporal components of neural signals. First, the variable intervals between sensory events allowed us to assess neural responses with respect to elapsed time in a manner that was orthogonal to movement planning responses. Second, the dense sampling of visual and motor space allowed us to confidently identify neurons with robust responses that were not spatially tuned. Finally, the use of a standard oculomotor task allowed us to compare our findings directly with a large body of literature regarding cortical oculomotor control. Responses were characterized with respect to five externally defined events and in the intervals between them (states). These responses were further categorized as spatially tuned when possible, or modulated but not spatially tuned, because this distinction had important consequences for a neuron's ability to contribute to a representation of the sequential states of the task. Recording from these two oculomotor structures during the performance of the same task, we are better able to understand how frontal and parietal oculomotor areas represent events and states in an oculomotor task, and consequently how they make specialized contributions to behavioral timing.

METHODS

Studies were performed with two behaving male rhesus monkeys (Macaca mulatta). Each was chronically fitted with a stainless steel head post for head immobilization and two recording chambers over small craniotomies for electrode insertions. Experimental procedures were in accordance with the California Institute of Technology Institutional Animal Care and Use Committee.

Monkeys were seated in a dimly lit room, 34 cm from a tangent LCD monitor. Stimuli were presented with 800 × 600 resolution and a refresh rate of 75 Hz using a custom built software display client with OpenGL. Task logic was controlled by National Instruments real-time LabView software.

Eye movements were monitored with an infrared oculometer (ISCAN). A high-speed camera (temporal resolution: 240 Hz, spatial resolution: 0.06°) was mounted on a wooden frame above the monkey's head and, along with an infrared source, directed into an infrared reflective hot-mirror held fixed at a 45° angle just in front of the monkey's eyes. A trapezoidal notch was cut from the mirror so that it could be placed close around the monkey's nose. For both monkeys, the left eye was monitored.

Recording procedure

Neurons were accessed on vertical penetrations with glass coated platinum-iridium electrodes (Fred Haer) or Thomas Recording electrodes (ThomasRecording Gmbh). The electrodes were advanced with a Narishige or Thomas microdrive system through a blunt stainless steel guide tube pressed against the dura for SEF recordings or a sharp stainless steel guide tube puncturing the dura and driven down 1 mm for LIP recordings. Neurons were generally found 1–3 mm beneath the exterior of the dura for SEF recordings and 5–9 mm beneath the level of the dura for LIP recordings. Both areas were identified based on a combination of anatomical localization using an MRI scanned after the chamber placement surgery and then observation of saccade related activity at each recording location. We also performed low-threshold microstimulation (<50 μA) to evoke saccades with monkey M to confirm the location of SEF.

Waveforms were amplified and isolated on-line with a commercial hardware and software package (Plexon). Cell activity was monitored with custom built on-line data visualization software written in Matlab.

Behavioral task

A memory-guided saccade task was used. The monkey was instructed to perform a saccade from a central fixation point to one of 43 target locations placed at regular intervals to cover the entire visual field out to 15° of visual angle in every direction from central fixation. In a typical recording session, the monkey performed three trials to each target location, for a total of 129 trials. It is atypical to use such a large number of target locations in an oculomotor task (but see Platt and Glimcher 1997). For our study, we used this dense sampling of saccade target locations not only to characterize the spatial tuning exhibited by individual neurons, but also so that we could confidently establish the lack of spatial tuning of an otherwise robust neural response. In addition, considering evidence that some SEF neurons encode eye movements in extra-retinal coordinates, such as head-centered coordinates, but that LIP neurons utilize a gain modulated retinal coordinate frame (Andersen et al. 1985; Constantin et al. 2007; Martinez-Trujillo et al. 2004; Park et al. 2006), we were careful to use a straight-ahead fixation point at the start of each trial. This consideration ensured that eye- and head-centered coordinate frames were identical and that spatial tuning in either coordinate frame would be expressed in our paradigm.

Each trial featured five external events that governed the progression of the task. These events are highlighted in Fig. 1A where the temporal progression of a trial is shown. After the intertrial interval, a fixation point appeared (event 1: fixation point on), that the monkey was required to fixate within 1,000 ms. Following a variable delay of 300–600 ms (for monkey M; 800–1,100 ms for monkey L), a cue was briefly (250 ms) flashed (event 2: cue on) at 1 of 43 targets in the periphery. Following a variable interval of 500–800 ms (for monkey M; 750–1,050 ms for monkey L), the fixation point was extinguished (event 3: fixation point off), and the monkey was required to make a saccade to the remembered target location and fixate there. After a final variable interval of 250–550 ms the target reappeared (event 4: visual feedback), and then following an additional 450 ms fixation at the target, the monkey was rewarded with a drop of juice (event 5: reward).

Fig. 1.

Sequence of events and states in the oculomotor task. A: timeline of the task highlighting the 5 events that governed the task progression. B: representation of the task as a sequence of states with external events serving as transitions between states (see methods).

Cell categorization analysis

MODULATIONS IN EVENT AND STATE INTERVALS.

Responses were assessed with respect to the five externally defined events highlighted in Fig. 1A and in the intervals between them, highlighted in B. We refer to the intervals immediately following an event as “event” intervals, and we refer to the intervals between events as “states.” We carefully assessed firing activity in all state and event intervals because we had observed that neural modulations could occur at times far removed from the eye movement planning and execution intervals that are typically analyzed in oculomotor tasks. For instance, it is often assumed that the neurons are “resting” between trials, and likewise, previous oculomotor studies have considered the firing activity in the intertrial interval to be the baseline firing rate of the neuron (Isoda and Tanji 2003). We observed, however, that several SEF neurons were especially active in these resting intervals, and so we were careful to perform our cell categorization analysis in a way that would correctly identify, for example, that a cell was active in the intertrial interval or when the monkey was waiting for the presentation of the spatial cue. To achieve this goal, we defined the baseline activity differently than previous studies. The advantages and disadvantages of this technique are described in the following text and in the discussion. The exact definitions of all event intervals incorporated the known latencies of responses in LIP and SEF, which are both between 70 and 100 ms (Bisley et al. 2004; Schall 1991b). The event intervals used for neuron classification analysis began 75 ms after events and lasted for 150 ms. This interval was meant to capture any initial response to the event.

Because we used variable duration intervals between events, the duration of a particular state varied from trial to trial. Previous researchers have used variable intervals between events to identify “preparatory set” activity—namely activation between cue presentation and saccade onset—in the SEF (Hanes et al. 1995). The durations of each state varied by ≤300 ms, and the distribution of the durations was uniform. For cell categorization analysis, we segregated the trials into two groups—trials in which the state was short versus long. Specifically, short-duration trials contained states between the shortest possible duration and 150 ms longer. Long duration trials contained states between 150 ms shorter than the longest possible duration and the longest possible duration. The segregation of trials by state durations is demonstrated in Fig. 2, where all trials are sorted according to the duration of the precue state. The firing rates were then collected from the last 150 ms prior to the shortest duration trial of each of these two groups. The start and end of these intervals is marked by vertical blue lines in the spike train rasters in Fig. 2. Firing data from the first group (between vertical blue lines in the green rasters) therefore was collected from the 150 ms prior to the shortest possible state duration. Data from the second group (between vertical blue lines in the red rasters) included the subsequent 150 ms interval. These groups also correspond to intervals in which there is decreasing uncertainty about the remaining duration of the state. The same spike trains are shown aligned to the subsequent event in the right panels. The tick marks in the bottom panels represent the mean firing rate of each group (green = short, red = long, black = baseline), with the height of the tick mark indicating the standard error. Note that Fig. 2A is similar to Fig. 1D from a recent paper by Ohame and colleagues (2008), although the data presented here are taken from the precue interval and not the delay interval. Importantly, we applied the same analysis to all of the variable intervals in the task. The duration of the intertrial interval was not randomized. We performed similar analysis on this interval, but instead of segregating the trials into the groups, we collected data from each trial in the two 150 ms time bins immediately preceding the fixation point on event.

Fig. 2.

Detail of state interval activity types. A: neuron exhibiting anticipatory activity. B: neuron exhibiting state encoding activity in the precue state. The precue state extended from the time that the fixation point was acquired until the cue was presented. The time between these events was 300–600 ms. Rasters are aligned to the time that the fixation point was acquired (left) or the cue presentation (right). The spike trains are sorted from top to bottom according to the duration of the precue state in the trial. The black dots in the left rasters indicate the time that the cue was presented in each trial. The rasters at the top (green) are the trials in which the precue state duration was 300–450 ms, and the rasters on the bottom (red) are the trials in which the duration was 450–600 ms. Firing rates in the 150 ms intervals immediately preceding the shortest duration of each of these groups (marked by the vertical black bars in the rasters) were used to distinguish anticipatory and state encoding activities. The average firing rates for the analysis intervals ± the SE are superimposed over the smoothed firing rates (Gaussian kernel, SD = 50 ms) in the bottom panels along with the average baseline firing rate ± SE shown in black.

Neural activity was tested in two ways in each analysis interval. First, we tested whether the neural response in the interval averaged over all trials was significantly different from the baseline firing rate exhibited by the neuron (modulated). Second, we tested whether the activity associated with trials in which a saccade was planned toward the neuron's response field was significantly different from activity in trials in which the saccade was planned in the opposite direction (spatially tuned). Neurons that were spatially tuned did not always pass the test for modulation with respect to baseline depending on the size of the response field and the variability of the responses. In this section, we describe the first test for modulations of spiking activity with respect to baseline.

The baseline firing rate was defined as the average firing rates observed over the entire trial, for each correctly performed trial. There were, therefore, as many observations of the baseline firing rate as there were successfully performed trials. The exact interval used for the calculation of the baseline firing rate extended from 500 ms prior to the fixation point on event until 1,500 ms after the delivery of the reward. Importantly, this calculation of the baseline firing rate does not assume that any particular interval is a common resting interval for all of the neurons. As noted in the preceding text, previous researchers have used neural activity from the intertrial interval as the baseline firing rate for the purpose of statistical comparisons with firing activity in other task-related intervals (Isoda and Tanji 2003). One of the important technical points that we hope to make in this report is that this practice for defining the baseline firing rate can be misleading when trying to categorize activity in the supplementary motor cortex.

The firing rates observed in the five event detecting intervals (fixation point on, cue on, fixation point off, visual feedback, and reward) and four state intervals (pretrial, precue, memory, and postsaccade), with two groups of firing rates for each state interval, as well as the group of baseline firing rates were compared using ANOVA with Bonferroni correction for multiple comparisons. If the neural response in an event interval was significantly different from baseline (P < 10^-5), the response was classified as event detecting. Neural responses in the state intervals were classified as either state encoding or anticipatory because rising or decreasing activity between events could serve an anticipatory function, and a tonic response between externally defined events that is not rising or falling could serve a state encoding function. Figure 2 shows a detailed demonstration of one anticipatory and one state encoding neuron. If the two groups of firing rates from the intervals within a state were statistically indistinguishable and were different from baseline, the neural response in that interval was classified as state encoding. If the intervals had significantly different firing rates, with at least one significantly different from baseline, the neural response was classified as anticipatory.

SPATIALLY TUNED RESPONSES.

The firing activity of individual neurons was tested for spatial tuning in the six event or state intervals that followed the cue presentation. In the three intervals prior to the cue presentation, spatial tuning was not assessed because the monkey had no spatial information at those times. Trials were selected in which the cue appeared inside the neuron's putative response field or in the opposite direction and compared for significant differences. The direction of the response field was calculated separately in each event or state interval using population vectors. The firing rates observed in the interval were multiplied by the vector of the cue from the same trial. These vectors were then summed, and the direction of the resultant vector was taken as the preferred direction. After the preferred direction was identified in this way, all locations within the same quadrant (± 45°) as the preferred direction were defined to be in the response field of the neuron. All locations in the opposite quadrant were likewise defined to be away from the response field of the neuron. The grouping of trials into quadrants was done to increase statistical power and therefore the confidence with which we could categorize a neuron as exhibiting spatially tuned activity. The use of many targets covering the visual field in every direction let us conclude with confidence that a given neuron was not spatially tuned even though it might be modulated in a particular interval. This was important because the response fields of SEF neurons can be very large. An SEF neuron with a large response field, for example, might respond similarly for two targets that are spaced far apart from each other and might appear to be nonspatial if only tested with a small number of saccade targets.

To assess the significance of spatial tuning, average neural firing activity for all six intervals was compared using ANOVA with Bonferroni correction (P < 10^-5) for trials in two stimulus conditions—trials in which the cue appeared in the response field versus away. A neuron exhibited spatial tuning if there was a significant difference between the firing rates observed in trials in which the cue was in the response field versus away. This method was intentionally coarse-grained so that spatial tuning could be established or, alternatively, so that we could conclude that a neuron was not spatially tuned.

Analyses of combined spike trains

Three additional types of analysis were performed to reinforce and amplify the main findings that resulted from the cell categorization analyses just described. Unlike the cell categorization analysis, for which our task was designed, these follow-up analyses required a pooling of responses from different trials to be treated as if they were recorded simultaneously. Because the individual trials were of variable duration, it was necessary to “standardize” the spike trains so they could be treated as if they were the same length.

To standardize the spike trains for each trial, spike trains were aligned to four different events that occurred at varying intervals. The four spike trains surrounding those events were then connected into one spike train of 3 s duration. Some portions of the spike trains from trials with longer intervals were cut out, and in some of the shorter trials, portions of the spike train were duplicated. Possible adverse effects of this standardization procedure are considered in Consideration of task design. For the purposes of these analyses, it would have been preferable either to have used fixed duration intervals for all of the recordings or to have recorded the entire data set in a single experimental session as may be possible in the future with chronic array implants. We had chosen to use variable intervals, however, because this allowed us to properly identify what the neurons were encoding—as analyzed with the cell categorization method described in the preceding text. For spike train standardization, the alignment events were fixation point on, cue on, fixation point off (go signal), and visual feedback. The visual feedback event led directly to the reward event after a fixed interval. The standardized duration spike trains extended 400 ms before the fixation point on event until 350 ms afterward, thus including the final portion of the intertrial interval and any transient neural response to the presentation of the fixation point. This was followed by the spike train data from the period 225 ms before the cue on event until 575 ms afterward, thus including responses leading up to the presentation of the cue as well as any responses to the start or the end of the cue presentation. This was connected to spike train data from 350 ms before the fixation point off event until 400 afterward, which included responses leading up to the go signal as well as activity associated with the execution of the saccade. The last portion of the standardized duration spike train included data from 200 ms before the visual feedback until 500 ms afterward, which included responses leading up to the visual feedback event as well as responses to the presentation of visual feedback and the subsequent reward delivery.

While this spike train standardization procedure was not used for the cell categorization described in Figs. 1–5 of the manuscript, the intervals used for spike train standardization were also used to plot the four spike trains histograms that were combined for each example neuron in Fig. 3. As illustrated in Fig. 3, summed firing rates for the example neurons tended to match up well with the previous and subsequent spike train segments to which they were connected in the spike standardization procedure. These standardized spike trains were used in the continuous measurement analysis illustrated in Fig. 6, the temporal decode analysis illustrated in Figs. 7 and 8, and the information theoretic analysis illustrated in Fig. 9. In the following subsections, these combined spike train analyses are described in detail.

Fig. 3.

Examples of activity types that support a sequential state representation. Each row consists of spike train rasters and smoothed (Gaussian kernel, SD = 50 ms) average firing rates for 1 example neuron in each of the 3 categories. All neurons are from the supplementary eye field (SEF). Numbers on the ordinate are the scale for the firing rate for each neuron. The columns include neurons with conspicuous event detection (A), anticipation (B), and state encoding (C) activities. See results for further description.

Fig. 4.

Prevalence of activity types in neuronal populations. Prevalence of activity types in the SEF database (A), data from both monkeys combined. The labels on the horizontal axis are the 5 events and 4 states during which each neuron was tested for modulated (significantly active or suppressed; ■) or spatially tuned (▧) neural activity. Activity within each state was further classified as state encoding (state) or anticipatory (ant). B: prevalence of activity types in the lateral intraparietal area (LIP) databases, data from both monkeys combined.

Fig. 5.

Combinations of activity types in single neurons. A: state and event intervals for which each task related neuron (each row = 1 neuron) was significantly modulated (■) or spatially tuned (▧). B: number of neurons exhibiting modulated activity only, spatially tuned activity only, or both for each population. C: numbers of neurons exhibiting event detecting, state encoding and/or anticipatory activations that are both modulated (left number) or spatially tuned (right number).

Fig. 6.

Continuous measures of activation, suppression, and spatial tuning. A: continuous measures of firing rate increases and decreases for all recorded neurons in the SEF databases, data from both monkeys combined. Top: normalized firing rates, each row corresponding to 1 neuron. The color scale ranges from no firing activity (blue) to 5 times the baseline firing rate (red). Neurons are sorted according to the occurrence of their maximum firing rates. The time progression corresponds to merging the four variable intervals among the fixation point on, cue on, fixation point off, and visual feedback events. The black lines indicate the timing of these events. Subtle discontinuities can be observed between these events where the spike trains were merged. Bottom: total number of significantly modulated (see methods) neurons at all points in time, shown as the percentage of the total number of recorded neurons in the sample. B: r² values (goodness-of-fit) for nonlinear regressions on a 2-dimensional Gaussian, using smoothed firing rates at every 10 ms and the positions of the saccade targets. Each row corresponds to the neuron from the same row in the left column. Color scale ranges from 0 (blue) to 1 (red). Bottom: percentage of neurons that were spatially tuned of the neurons that were responsive at that point in time (spatially tuned and/or significantly active or suppressed). C and D: continuous measures of spatial tuning and firing rate increases and decreases for all recorded neurons in the LIP databases, data from both monkeys combined.

Fig. 7.

Temporal decode performance. Temporal decode results from monkey M, SEF database. A: confusion matrix for the 40 actual temporal intervals (horizontal axis—each interval is 75 ms) and 40 decoded temporal intervals (vertical axis). Intensity indicates the percentage of trials decoded as each time interval, at each point in time (range: 0–50%). Span of the different event periods and states are boxed. B: temporal decode performance across the trial calculated as the percentage of trials correctly decoded in the exact same temporal interval ( · · · ) or a temporal interval within the same event period or state (—). C: confusion matrix for the temporal decode by state and event intervals. Intensity indicates the percentage of trials within each event period or state (horizontal axis) decoded as a temporal interval in each of the 9 possible state and event intervals (vertical axis). Range is 0–100%.

Fig. 8.

Decode performance by population size. A: monkey M. Temporal decode results using different neuron population sizes. Mean temporal decode performance for correct state classifications (vertical axis) using random samplings (without replacement) of different neuron population sizes (horizontal axis) with a 75 ms time window for spike train inputs. Error bars are standard error of the mean for the different samples. Black lines are SEF results. Gray lines are LIP results. B: monkey L, same conventions as in A. C: data from both monkeys combined, same conventions as in A.

Fig. 9.

Information theoretic analysis. A: firing rates of an example SEF neuron (same shown in 4th row of Fig. 3A) divided into 25 ms nonoverlapping time bins (x axis) for each trial (y axis). Time bins were assigned to 1 of 16 firing-rate groups, indicated by different colors (dark blue = group 1, dark red = group 16). All firing-rate groups contained the same number of members, so that time bins with zero spikes occupied the 1st 12 groups and part of the 13th. B: example LIP neuron rasters and smoothed firing rates. C: uncertainty about firing rates given the time interval for SEF example neuron (black line), and the similarly responsive but spatially tuned LIP neuron (blue line). D: spatially organized cue period activity for LIP neuron shown in B with white intensity corresponding to higher firing rates. E: spike train rasters for trials to preferred (red) and anti-preferred (blue) target locations around the time of cue presentation. F: uncertainty of firing rates for restricted subset of trials shown in E.

Continuous measures of firing rate modulations and spatial tuning

In addition to testing for modulations in entire event or state intervals, as described in the cell categorization subsection in the preceding text, we tested for significant modulations in a continuous fashion (every 10 ms), as shown in Fig. 6. The instantaneous firing rate was first estimated by summing the standardized spike trains over all trials, then smoothing with a Gaussian kernel (SD = 50 ms). Assuming a Poisson distribution of the instantaneous firing rate with the mean equal to the baseline firing rate, a significant modulation was identified when the likelihood of the instantaneous firing rate fell below P < 0.05. The number of neurons that met this criterion was counted at each 10 ms interval.

In addition to testing for spatial tuning in entire event or state intervals, we tested for spatial tuning in a continuous fashion (every 10 ms), as shown in Fig. 6. Standard quantitative models were implemented with MATLAB to evaluate the dependence of firing rate on cue position (Campos et al. 2006; Draper and Smith 1981; Press et al. 2002; Zar 1974). An expanded description of the technique can be found in Campos et al. (2006) and is summarized briefly here. The standardized spikes trains were smoothed by convolving with a Gaussian kernel (SD = 50 ms) to estimate the instantaneous firing rate for individual trials. An initial estimate of the center of the response field (b₃, b₄) was calculated as the vector average of all of trials with associated firing rates that were >50% of the maximum firing rate. Initial estimates of the remaining parameters were chosen arbitrarily: b₀ = 100, b₁ = 4, b₂ = 3. The firing rates were then regressed on a two-dimensional Gaussian using the following equation

$$F=b_0+b_1\cdot \exp \left(\frac{{\left(x-b_3\right)}^2+{\left(y-b_4\right)}^2}{b_2^2}\right)$$ (1)

where (x,y) is the position of the target. The regression was computed (Campos et al. 2006; Draper and Smith 1981; Press et al. 2002; Zar 1974), and the values for the parameters were stored along with the P and r² values describing the goodness-of-fit for the regressions.

Temporal interval decode

We used the spiking activity of several neurons from a given cortical area to “decode” the temporal interval information contained in the activity of the ensemble of neurons. While the population of neurons was recorded over several distinct sessions spanning a few months, for the purposes of the temporal interval decode, all of the neurons from a given cortical area were considered to have been recorded simultaneously. We grouped spike trains from trials to the same saccade target from different recording sessions and then standardized the spike train durations, as described in the preceding text, because in the originally recorded trials, the variable intervals between events lead to variable trial durations.

The following method is adapted from a recent study, and the following description closely resembles the method previously published (Quian Quiroga et al. 2006). Each data set contained ≥129 trials (43 targets × 3 repetitions). From the standardized spike trains of 3,000 ms duration, varying window sizes were used to count spiking events to then be used as input into the decoding algorithm. The 75 ms time window produced a higher percentage of correctly decoded temporal intervals than window sizes that were slightly larger or smaller, and so 75 ms window sizes are used in all of the temporal decode results presented.

Cells were considered simultaneously recorded in the sense that for each of three trials to each of the 43 target positions, the responses of all cells were grouped together as a single trial with m values (in which m is the number of cells). Trials were considered as points in an m-dimensional space, each coordinate representing the mean firing rate for each of the m cells. One at a time, each of these 129 trials was decoded based on the distribution of all other trials (leave-one-out decoding) and was assigned to the class of its nearest neighbor in the m-dimensional space using Euclidean distance (Duda et al. 2001; Quian Quiroga et al. 2006). In the event that there were two trials recorded for a given direction, the third trial was randomly chosen from the other two recorded trials. When only one trial was recorded, the neuron contributed to the decode of other directions but not to decodes of the directions with only one recorded trial.

For the temporal decode, each trial was decoded at every 75 ms from the beginning to the end of the trial. Firing rates from the left out trial were compared with mean firing rates collapsed across all directions in the same interval and all other intervals. The time interval of the nearest neighbor was assigned as the decoded time interval of the left-out trial.

The simple model-free nearest-neighbor decode usually leads to error rates greater than the minimum possible, the Bayesian rate (Duda et al. 2001). Other decode algorithms (e.g., support vector machine) may outperform the results presented here. This approach makes no assumptions about the structure of the data. Cells from different functional categories, for example, were not treated differently. A clear illustration of the decode technique can be found in Quian Quiroga et al. (2006).

Information theoretic analysis

To better understand the temporal decode results, a related information theoretic analysis was performed (Buschman and Miller 2007; Shannon and Weaver 1949). Mutual information between firing rates and the current temporal position within the trial were computed (Fig. 9). The combined spike trains were first divided into 25 ms nonoverlapping time bins. The firing rates observed in those temporal bins were then sorted and organized into 16 equally sized firing rate group bins. In this procedure, several of the firing rate group bins contained temporal bins in which no spikes were observed. The bins were sorted so that zero-spike containing bins were spread evenly across the temporal extent of the trials, and this sorting avoided bias effects.

At each point in time in the trial, the likelihood that a firing rate belonging to each of the 16 firing rate bins would be observed at that point was computed by adding the number of members of each firing rate bin at that time interval and dividing by the total number of firing rate bins. This measure yielded the uncertainty about firing rates given the time interval. To decode the behavioral state, it is the uncertainty about the time interval given the firing rate that is of interest. However, these two values are interconvertible because the mean value of either of these measures gives the mutual information (because all time intervals and all firing rate bins are equally likely). More details of the information theoretic analysis technique for spiking data can be found in the supplementary methods of Bushman and Miller (2007).

RESULTS

Neurons were recorded from the SEF and LIP of two monkeys during the performance of a memory-guided saccade task. All neurons encountered in these anatomical regions that could be reliably isolated were recorded. Neural recordings were only excluded from the population if the isolation was not held for a sufficient period of time (∼80 trials) or if the waveform and spiking properties indicated cell body damage. There were 149 task-related neurons found in SEF (monkey M: 108, L: 41) from a sample of 285 recorded SEF neurons (monkey M: 175, L: 110). There were 53 task-related neurons found in LIP (monkey M: 21, L: 32) from a sample of 151 recorded LIP neurons (monkey M: 57, L: 94).

Response intervals and activity types

Based on the temporal profile of the neural responses, we defined three functional activity types relating to the progression of the task. We propose that a phasic response immediately following an external event could serve an event detecting function. Second, rising or decreasing activity between events could serve an anticipatory function. Finally, a tonic response between externally defined events that is not rising or falling could serve a state encoding function. While these response profiles have been observed previously in SEF and adjacent areas (Akkal 2004; Brody et al. 2003; Genovesio et al. 2006; Hanes et al. 1995; Isoda and Tanji 2002, 2003; Lebedev et al. 2008; Matsuzaka et al. 1992; Ohmae et al. 2008; Schall 1991b), they have not been observed in SEF in all intervals of an eye movement task, such as the precue or intertrial intervals. Furthermore, these various functional types have never been presented together or suggested to perform a single coherent functional role. We will show examples of these activity types and explain how a diverse collection of these responses can encode the progression of the task. With our task design, it is difficult to know if the responses actually carried meaning for the task or if they were simply stimulus driven. Regardless of whether these neural responses served specific functions, and perhaps in addition to the functions they served, we can show that the collective responses of the population of neurons were capable of carrying information about the current event or state of the task.

We observed that many of the task-related neurons in our database of SEF neurons exhibited robust responses that were not spatially tuned for any particular range of directions or amplitudes. These neural responses were similar to omni-directional neuron responses (Chen and Wise 1996) but were observed at many different phases of the trial, including times removed from the saccade execution itself. The nonspatially tuned responses are also similar to the bidirectional set-related and buildup activation patterns previously observed in the presupplementary motor area (Matsuzaka et al. 1992), which is just medial to the SEF, but again we did not limit the analysis to the interval following the cue presentation as in this previous report. In the next three sections, we focus on nonspatially tuned neural responses in SEF at all phases of the instructed trials. In Fig. 3, there are examples of SEF neurons that exhibited nonspatial neuronal activity that relate to the progression of the task. We will use these examples to introduce the response intervals and functional neuronal response types. For clarity, a brief review of previous published observations of each of the three functional subtypes will be included at the end of the next three subsections.

Event detection

A group of neurons responding to different events could, in principle, indicate which event the monkey is experiencing at any point in the task. Several examples of event detection activity are shown in Fig. 3A. All of the neurons shown in Fig. 3 are examples that did not exhibit spatial tuning in the intervals that were tested (following the cue presentation). The first three example neurons responded to one event much more so than any other event. The first responded to the fixation point on event with an activation that was much larger than the cue on and visual feedback responses also exhibited by the same neuron. The second neuron responded to the cue on event with a brief depression, and the third neuron responded to the fixation point off event with an activation. The third neuron also showed an extended suppression during the memory period that could be described as state encoding activity. As detailed in the following text, these activity type categories could be expressed in single neurons at different times. By interpreting the event detecting activity of these three neurons, the most recent event can be known by noting which neuron was recently modulated. For example, on a given trial, if the neuron illustrated in the first row of Fig. 3A was active but the neuron illustrated in the second row had not yet been depressed, then an observer could deduce that the fixation point on event had just occurred. The remaining examples shown in Fig. 3A respond to multiple events with similar firing rates. These neurons do not themselves definitively tell us which event has just occurred. For example, if the neuron illustrated in the fourth row is active, we can only deduce that either the cue on or the visual feedback event has just occurred. However, if we combine this information with the information from the neuron illustrated in the second row, which tells us that the cue on event has just occurred or that it has not, then we can reliably state with basic logic that either the cue is on (4th row neuron active, 2nd row neuron depressed) or the visual feedback is on (4th row neuron active, 2nd row neuron at baseline). Combinations of additional neurons can encode the most recent event with increasing precision and coverage of the temporal intervals in the trial.

Neural activity patterns that resembled event detection activity have been observed previously in the SEF and neighboring areas. Robust responses were observed in dorsolateral prefrontal neurons during the cue and saccade periods of a delayed-saccade task (Fuster and Alexander 1971; Niki and Watanabe 1979). Nonspatially tuned event detecting activity in the saccade interval (e.g., Fig. 3A, 3rd row) also resembles rank-selective activity, which was observed during the first, second, or third saccade of a sequence regardless of the direction of the saccade (Isoda and Tanji 2003). Neurons in SEF that were active during the delivery of juice reward were previously described as reward detecting neurons (Amador et al. 2000; Stuphorn et al. 2000). To our knowledge, there have been no reports of neurons that were activated after the presentation of the fixation point, such as the neuron shown in Fig. 3A, first row.

Anticipation

While it may be satisfactory to know which event happened most recently, it may also be helpful to know how much time has elapsed since the previous event. For this function, neurons with anticipatory activity are very well suited (Brody et al. 2003). For example, a neuron exhibiting anticipatory activity might start with a baseline firing rate of 10 spikes/s (1 spike every 100 ms) and then after a triggering event fire one additional spike in every subsequent 100 ms period. Within the first 100 ms, the neuron would discharge two spikes, yielding an instantaneous firing rate of 20 spikes/s. Within the next 100 ms the neuron would discharge a total of 3 spikes and so on. If the neuron is known to steadily increase in this manner, and later on we measure the firing rate at 100 spikes/s (10 spikes/s in a 100 ms period), we can then infer that 900 ms have passed since the triggering event. The equation that yields the current time would be: (current firing rate – baseline)/(increase in firing rate per second). The example neurons shown in Fig. 3B rise in anticipation of certain events and then reset once those events occur. As shown in that figure, a single neuron can rise before just one event in the trial or several. In the case that the neuron exhibits anticipatory activity prior to multiple events, knowledge of the current state of the monkey is required to make sense of each anticipatory buildup.

The plots in Fig. 3 are aligned to four different events for each neuron, and smoothed mean firing rates are shown with respect to these alignment times. Anticipatory activity is defined by how the firing rate of a neuron evolves over the course of the state interval with reference to the time at which the interval began. The analytical treatment of the variable intervals between states is shown in Fig. 2, where the difference between anticipatory and state encoding activity patterns is illustrated. The plots shown in Fig. 3, however, qualitatively convey the same firing rate patterns in a compact format.

Neural activity patterns that resembled anticipatory activity have been observed previously in the SEF and neighboring areas. Anticipatory responses were observed in dorsolateral prefrontal neurons prior to cue presentation or saccade onset (Fuster and Alexander 1971; Niki and Watanabe 1979), resembling the second and third rows of Fig. 3B. Coe and colleagues more recently reported a similar anticipatory rise of activity prior to the onset of saccade targets in SEF (Coe et al. 2002), and a significant rise in premovement activity was reported in SMA (Romo and Schultz 1992) and SEF (Ohmae et al. 2008). Neurons in SEF that were increasingly active prior to the delivery of juice reward were previously described as reward predicting neurons (Amador et al. 2000; Stuphorn et al. 2000). To our knowledge, there have been no reports of anticipatory activity prior to the fixation point onset, such as the neuron shown in the first row of Fig. 3B. Isoda and Tanji, however, described SEF neurons that exhibited “first-trial selectivity.” These neurons were active when the monkey transitioned from one task set to another with one example increasingly activated just before the first trial of a new set of trials (Isoda and Tanji 2003). This activation profile was arguably similar to the neuron shown in the first row of Fig. 3A, which exhibited anticipatory activity before the start of each trial instead of each set of trials.

Sequential states

The final piece of the event-based representation of sequential states in SEF is the population of neurons that represent different states. As the monkey moves through the task, he goes from one state to another (Fig. 1B), first waiting for the fixation point to appear, then fixating that point while awaiting the movement instruction, then continuing to fixate while planning the instructed eye movement, then re-fixating on the saccade target while awaiting visual feedback and reward. The example neurons in Fig. 3C show a variety of neurons that are tonically active during one or more of these states. None of these responses were spatially tuned where assessed, and like event-detection activity, can be used together to decode the state in which the monkey is currently engaged.

Neural activity patterns that resembled state encoding activity have been observed previously in the SEF and neighboring areas. Schall observed preparatory set cells in SEF (Schall 1991a) that resembled the example neuron shown in Fig. 3C, third row, active in the interval in which the monkey could prepare a saccade. Romo and Shultz made a similar observation in the SMA (Romo and Schultz 1987, 1992). Schall made note of pause-rebound cells in SEF and SMA that were probably similar to the neurons shown in Fig. 3c, second or fourth rows (Schall 1991b). The neuron in Fig. 3C, fourth row, also resembled our own previous characterization of reward expectancy activity in the SMA (Campos et al. 2005). Last, Schall counted several modulated but unclear cells in FEF and SEF, which showed “some apparently systematic modulation during the trial,” but these units were not analyzed further. To our knowledge, there have been no reports of SEF neurons selective to the precue or intertrial intervals of an eye movement task, such as the neurons shown in Fig. 3C, first and second rows. Intertrial selectivity has been reported in the temporal lobe (Yakovlev et al. 1998) and the dorsolateral prefrontal cortex (Mansouri et al. 2006). Precue modulations have also been identified in the cingulate motor area (Hoshi et al. 2005) and the dorsal prefrontal cortex (Hasegawa et al. 2004).

The preceding description segregates a set of neurons into three distinct categories even though individual neurons lie on a gradient between these categories. For example, sequential state activity and anticipatory activity differ from each other only by the slope of the average firing rates over time. Figure 2 details two neurons that are modulated in the precue state, the first exhibiting anticipatory activity and the second exhibiting state encoding activity. The firing activity of the anticipatory neuron continues to ramp up as the duration stretches longer and longer. This firing rate profile can encode how much time has passed since the previous event or, alternatively, how likely it is for the next event to happen—which is directly related to how much time has elapsed (see methods). The state encoding neuron, however, continues firing at the same rate. It only encodes that the monkey is in the precue state. Neurons in the population modulated in the state intervals exhibited a variety of slopes, putting them at various points on a gradient between these two categories. The different slopes are functionally significant. Anticipatory activity with a high slope can provide highly resolved information about the time elapsed since the last event or the estimated time remaining before the next event. Anticipatory activity with a lower slope can provide less well resolved information about elapsed time. Finally, sequential state activity with a flat slope can only provide information that the monkey is in the time period following a certain event, without providing any information about how much time has elapsed, or how much time remains, in that state. Similarly the definitions of sequential state activity and event detecting activity differ in the duration of their responses, and in this case too individual neurons with different response durations lie on a functional gradient between these defined categories. Understanding these limitations, for the sake of characterizing responses in each population, we assigned individual neurons to one of these three groups in each analysis interval (see methods). In the following section, we quantify the percentages of neurons across the two populations that exhibited event detecting, anticipatory, or sequential state encoding activity, further subdividing each group according to whether it exhibited spatial tuning. We also detail the extent to which individual neurons exhibited different types of activity at different intervals.

Population characteristics

The prevalence of all types of activity is shown graphically in Fig. 4A for the SEF population and B for the LIP population. For each interval, the number of neurons that could be classified as modulated, meaning they were significantly modulated (active or depressed) but not classifiable as spatially tuned (see methods), is shown in black. Prior to the cue-presentation, spatial tuning was not tested, and in these intervals a neuron's activity could be either categorized as modulated or not task related.

Spatial tuning was defined as an increased firing rate associated with a cluster of target positions relative to the fixation point at the center of the stimulus display monitor (see methods). In the postsaccadic, visual feedback, and reward intervals, the spatial tuning was assessed in the same way although the monkey was fixating the target location at these times. Spatial tuning in these intervals could therefore reflect remaining postsaccadic responses, eye position sensitivity, or early preparations for a re-centering saccade to start the next trial. The number of neurons with spatially tuned activity is indicated by the height of the gray bars for each tested interval (cue on event and following). The number of neurons exhibiting event detecting activity is indicated by the height of the black bars above the event interval labels (fp on, cue, go, feedback, rew). The number of neurons exhibiting state encoding activity or anticipatory activity is indicated by the height of the black bars above the state interval labels. As can be seen in these panels, the neurons in LIP were more likely to be spatially tuned during the trial compared with modulated but nonspatial as in SEF. The spatially tuned activity exhibited by SEF neurons tended to occur early in the trial and then taper off, whereas in LIP more neurons were spatially tuned throughout the trial with larger numbers reaching significance around the time of the movement or just after the movement and the cue presentation. Note that the SEF population contained more significantly modulated neurons that were not spatially tuned at all points in time within the trial after cue onset compared with LIP. Finally, the LIP population showed little modulation before cue onset, whereas the SEF population showed greater modulation before the cue.

The preceding activity type analysis considered each event or state interval independently, but as can be seen in the example neurons of Fig. 3, single neurons frequently exhibited multiple types of activity at different points in the trial. For instance, the event detecting neuron shown in Fig. 3A, third row, also exhibited a consistent suppression of activity during the memory state and therefore expressed state encoding activity as well. Similarly, the anticipatory neuron shown in Fig. 3B, second row, also showed a tonic elevated discharge during the memory period as another example of state encoding. In general, these three types of activity were not segregated into distinct populations of neurons, but instead, multiple activity types could be expressed in single neurons at different times in the trial. Individual cells were also modulated in multiple consecutive intervals.

In Fig. 5, we show the extent to which single neurons expressed multiple activity types at different times in the trial. Event detecting neurons and state encoding neurons, for instance, do not appear to belong to segregated neuronal populations but instead refer to phasic and tonic discharge patterns that can be expressed in one neuron at different times. Figure 5A shows the state and event intervals for which each task related neuron (each row = 1 neuron) was significantly modulated (black) or spatially tuned (gray). Clearly many neurons are modulated in more than one interval. Figure 5B shows Venn diagrams for the numbers of neurons that were either spatially tuned, modulated but untuned, or exhibiting both tuned and untuned activity at different intervals. As described in methods, we defined spatially tuned and modulated responses to be mutually exclusive (i.e., a modulated response was significantly elevated or depressed with respect to baseline and not classifiable as spatially tuned). Because these tests were made in multiple intervals, it was possible for a neuron to exhibit modulated activity in one interval and spatially tuned activity in another interval. The Venn diagrams refer to the numbers of neurons that only exhibited spatially tuned activity, only exhibited modulated activity, or exhibited both at different intervals. The population sizes recorded in each area were unequal with about twice as many neurons recorded in SEF compared with LIP, and a higher proportion of neurons in SEF were found to be task related. Both populations contained similar numbers of cells that were spatially tuned only (and not modulated but untuned in other intervals), indicating that the proportion of spatial neurons to all task related neurons in SEF is about half that in LIP. There were, however, about five times as many neurons in the SEF population that were modulated and not spatially tuned in other intervals, making up ∼2/3 of the task-related population in SEF and one-third of the task-related population in LIP. Figure 5C shows the number of neurons exhibiting each of the three activation types, alone or in combination, separately for modulated and spatially tuned activations. Each segment of the diagram contains two numbers, the first counting combinations of modulated activations, and the second combinations of spatially tuned activations.

Because any description of firing rates in particular state or event intervals will be influenced by the exact choice of start and end time of these intervals, we present analysis in Fig. 6 of continuous measures of firing rate modulations and spatial tuning without regard to our interval definitions. In Fig. 6, A and C, the smoothed (Gaussian kernel, SD = 50 ms) normalized firing rates are shown for SEF and LIP populations, respectively. The normalized firing rates are averaged over all trials, and consequently, all saccade target directions. In Fig. 6, B and D, the goodness-of-fit for a spatial tuning model of smoothed firing rates versus target location (2D Gaussian) is shown as calculated at each 10 ms. The neurons are sorted according to the time of occurrence of their maximum, normalized firing rates in both panels. All of the recorded neurons are included in these intensity plots. The results shown by these continuous methods are qualitatively similar to those found using predefined intervals. That is, ∼40% of neurons are significantly modulated in the SEF population during the cue presentation, and between 10 and 20% at all other intervals in the trial (Fig. 6A, bottom). In the intervals in which spatial tuning was assessed (during and after cue presentation), only about two-fifths of the responsive (modulated or spatially tuned at each point in time) SEF neurons were spatially tuned (Fig. 6B, bottom). In contrast, about three-quarters of responsive LIP neurons were spatially tuned from the cue onset through the saccade period, and a much higher percentage of responsive LIP neurons were spatially tuned around the time of the saccade (Fig. 6D, bottom). Prior to the cue presentation, the LIP neurons showed little modulation, whereas the SEF population exhibited greater modulation at that time.

Decoding the current temporal interval

Based on the observation that the neurons that were not spatially tuned were consistently responding at specific intervals in the trials, we next tested whether the responses of a population of neurons could be used to decode the current temporal interval on a trial-by-trial basis. We adapted a model-free decode method previously applied to the spatial decode of eye and arm movements (Duda 2001; Quian Quiroga et al. 2006) and quantified its performance in decoding the specific time interval within the task to which a set of firing rates belonged (see methods). In Fig. 7 are the results of the method using the full database of SEF neurons collected in monkey M (because the SEF neurons were more likely than LIP neurons to be modulated but not spatially tuned). The performance of the monkey M SEF database was higher than that of monkey L, mainly on account of a larger sample size, although the results were qualitatively similar. In the summary subsection in the following text, we illustrate the decode performance as a function of sample size via bootstrapping.

Figure 7A shows the confusion matrix for the temporal decode for the SEF database. The trials were divided into 40 nonoverlapping 75 ms intervals for the decode (see methods). High intensity along the diagonal indicates that time intervals were decoded correctly. The decode algorithm frequently decoded the exact temporal interval. Figure 7B shows the percentage of exactly correct decodes (dotted line) for all points in time in the trial. The algorithm tended to have peaks of performance around the fixation point on, cue on, fixation point off, and reward events. These points in time correspond to the externally generated stimuli that lead to robust and unique event detecting responses.

Although the algorithm was less able to decode the exact temporal interval at times removed from the four events listed in the preceding text, it frequently did succeed in decoding a temporal interval that was nearby. That is, between events the decoded temporal interval tended to be another temporal interval within the same state (e.g., precue). This ability to decode the exact temporal interval within a state depends on the neurons expressing anticipatory activity because these are modulated across the time within the state. The state-encoding neurons, in contrast, are similarly responsive during the entire state and therefore will help to limit the decode to the correct state interval but not more precisely within the state. The boxes in the confusion matrix in Fig. 7A outline the temporal extent of the different states and event periods in the trial. The cue event and the memory state boxes are labeled in Fig. 7A, and the sequence of boxes from the top left to the bottom right follow the same order as the labels on the axes of C. Figure 7C averages over these periods and shows the confusion matrix of the temporal decode for each state and event. Intensity along the diagonal indicates that firing rates for each temporal interval were almost always decoded as the same or another temporal interval within the same state or event. The performance of the temporal decode within the same state or event is shown as a solid line in Fig. 7B. Notice that while the dotted line (exact temporal interval decode performance) dips between events, the solid line (temporal interval decode within the correct state or event) maintains high performance.

The database from area LIP of the same monkey did not perform well when using the same temporal decode algorithm. There were substantially fewer LIP neurons recorded, but when we dealt with the issue of sample size, we found that for similar numbers of cells, the LIP population performs considerably poorer in decoding temporal intervals. The temporal decode performance of each area is shown in Fig. 8, which compares performances using randomly chosen subsets of neurons so that data from the two areas can be directly compared even though the original numbers of recorded neurons were unequal. SEF performed better in decoding the current state or event for equally sized and randomly chosen subsets of the population. The temporal decode results in Fig. 8 show the correct state or event decode performance for increasing population sizes. The mean of the solid line in Fig. 7B would be plotted as a single point at the far right side of Fig. 8, left, because the results shown in Fig. 7 are from the entire SEF dataset of monkey M. It is clear that SEF achieves better performance even when the neuron sample sizes are comparable. The mean values correspond to the performance that one could hope to achieve with a random sampling of neurons in each area, such as would be encountered with an unmovable electrode array.

Information theoretic analysis

The LIP population as a whole is less suited for the temporal decode because of the spatial tuning exhibited by the vast majority of neurons found there. Spatially tuned neurons will be inactive either because of the current behavioral state (e.g., pretrial) or because the target appeared at a nonpreferred location. Therefore if the neuron is not firing, there is no information about the temporal interval, and these neurons will generally not fire in the majority of trials. During intervals that the neurons might fire, the spatial tuning also leads to higher variance than if the neurons were not spatially tuned.

Using information theoretic analysis we quantified the relative information about the current temporal position available from two example neurons. The first neuron is an untuned SEF neuron that responds to two separate events, cue on and visual feedback, and the second, from LIP, also responds to two separate events, cue on and fixation point off (go) although in a spatially tuned manner. In Fig. 9A we show the firing rates of the example SEF neuron (the same as the neuron in Fig. 3A, 4th row) divided into 25 ms nonoverlapping time bins (x axis) for each trial (y axis)—see methods. As can be seen, there are two time intervals with consistently high firing. The raster for the example LIP neuron is given in Fig. 9B. and the cue period activity is organized according to the location of the cue in D with white intensity corresponding to higher firing rates. In contrast to the amount of information in time given by the example SEF neuron, shown in Fig. 9C as the black line, the same measure of information applied to a similarly responsive but spatially tuned LIP neuron, shown in the same panel with a blue line, is clearly substantially lower.

When only targets that fall within a neuron's response field are considered, the variance decreases, and we can recover behavioral state information from spatially tuned neurons. In the spatial intensity plots (Fig. 9D), the center of mass of the firing rates from the time of the cue presentation is shown as a yellow star, and the 10 closest targets to this response field center are labeled in red. The 10 targets that are farthest away are labeled yellow. In Fig. 9E, we show the rasters of these two groups of trials, the preferred vector trials on the top row and the anti-preferred direction on the bottom row. Finally when we apply the same information analysis to this restricted subset of trials, we get the results presented in Fig. 9F, which shows that for the preferred direction trials (red), there is higher information at the cue and saccade periods because the neuron consistently fires at a higher rate. In the anti-preferred trials (blue), the firing rate is consistently suppressed throughout the entire trial, and there is overall no mutual information available between the firing rates and temporal interval. Furthermore, in comparison to the smaller peaks of information seen when all trials are taken together (blue line in Fig. 9C), when the highly informative preferred vector trials are separated from the uninformative remaining trials, the information level for these trials is much higher.

The same procedure was applied to all spatially tuned neurons to assess the effect of limiting the information theoretic analysis to trials in which eye movements were directed into the neuron's response field. We tested neurons in both LIP and SEF populations that were spatially tuned during the cue presentation (the time period that evoked substantial responses in both areas) and observed a consistent increase in information when we compared trials in which the target was in the response field versus a location opposite. In the LIP population, we observed an average 50%, increase in firing rate information in the period extending from the cue presentation to the saccade when we calculated the information content on the restricted set of trials in which the target was in the response field. In the SEF population of spatially tuned neurons, we observed an 18% increase in firing rate information in the same interval. In the postsaccadic and intertrial intervals, there was no advantage associated with restricting the calculation of information content to trials in which the target had been in the response field for both the LIP and SEF populations.

DISCUSSION

In this report we have shown that SEF contains a collection of neuronal response profiles that can collectively support the representation of sequential states within the context of an oculomotor task. This representation depends on robust activations occurring at particular intervals of the task and is especially reliable if the same neurons do not exhibit spatial tuning for the direction of the saccade instructed in the current trial. While a subset of these response profiles have been observed previously in SEF and adjacent areas (Akkal 2004; Brody et al. 2003; Genovesio et al. 2006; Hanes et al. 1995; Isoda and Tanji 2002, 2003; Lebedev et al. 2008; Matsuzaka et al. 1992; Ohmae et al. 2008; Schall 1991b), our report is the first to propose a single coherent functional role for modulations that occurred at all intervals of a recorded trial. To our knowledge, this report also contains the first demonstration of robust modulations in SEF in the intertrial interval and in the interval just prior to the cue presentation of an eye movement task. Nonspatially tuned activations can serve different functions if they occur at different times as has been proposed previously and as we describe in the following text, but in this report, we make the argument that collectively they can represent the sequential states of the task, which in turn can assist in behavioral timing. This report is similar to a recent report on the role of SEF in smooth pursuit (Shichinohe et al. 2009) in that it describes a set of single unit responses, most of which have been described previously, that when taken together can perform an aggregate function (Schlag and Schlag-Rey 2009).

LIP neurons tended to respond spatially and were not found to assist as well in the sequential state representation. It has been previously reported that elapsed time influences LIP activity when it is integral to the eye movement instruction (Janssen and Shadlen 2005; Leon and Shadlen 2003). These previous reports considered eye movements either directly into or away from the neurons' preferred directions, and the neural activity was found to correlate with elapsed time. Furthermore the anticipatory modulation that correlated with elapsed time was found to be strongest in LIP neurons that represent the locus of the intended eye movement (Janssen and Shadlen 2005) and therefore was correlated with the likelihood of the eye movement. The results in the current report help contextualize these previous findings. We found that when only preferred vector saccades were considered, the LIP neurons did carry significant information about the temporal interval. When considering all possible saccade directions, however, SEF provided a better read-out of the current temporal interval—the current external event or state—within the context of the task and for the entire duration of the trial. LIP, with more spatially tuned activations, was less able to provide an accurate read-out of the current state or event. Overall the prevalence of SEF neurons with activity that was not spatially tuned made it a better predictor of the current behavioral state in all trials, while LIP neurons generally contained state information about saccades directed to their preferred locations. The current results, therefore argue that SEF contains a better all purpose code for time than LIP (Janssen and Shadlen 2005; Leon and Shadlen 2003) because the code in SEF is independent of saccade vector specification.

Decoding of movement intentions

The spatial tuning for saccade targets found in abundance in area LIP provides a basis from which to decode eye movement vectors. To properly decode the movement vectors from parietal neurons, it is necessary to know if the current representation formed by the firing activity of the neurons indeed represents the upcoming motor intention and, if so, when it is meant to be executed. There have been some efforts to selectively decode parietal activity from the appropriate interval, including a state machine decoding design in which the movement encoding state necessarily follows the visual onset state (Shenoy 2003) or through the use of spectral analysis of local field potentials to identify state transitions (Bokil et al. 2006). It may instead be simpler to look to SEF for state sequence information because the SEF population has abundant information regarding the sequential progression of the task. Importantly, the SEF population contains many neurons modulated outside of the sensorimotor transformation interval (cue presentation to saccade execution). The population of SEF responses can therefore provide ongoing detailed information of the current behavioral state of the monkey, for instance by detecting the transition between the intertrial interval and precue interval at which times the LIP population is largely unmodulated. With access to SEF and LIP neural activity, one could decode the current state from SEF and use that information to collect firing activity from LIP spanning an entire relevant state to achieve the most accurate spatial decode possible.

Consideration of task design

Two of the novel observations presented in this report are state encoding activity in the SEF during the precue and intertrial intervals of an eye movement task. It is difficult to know with certainly why these results have not been observed before. We suspect that a major reason is that SEF has been studied in the same way as other oculomotor areas, which excluded the possibility of observing these responses. First, several studies have focused exclusively on neurons that exhibited spatial tuning (Roesch and Olson 2005), which is a hallmark of neurons in the oculomotor system. Second, the analysis of the recorded data has usually focused on neural responses starting with the presentation of the cue and ending with the completion of the instructed saccade—the sensorimotor interval. Previous researchers have used firing activity from the intertrial interval as a baseline period (Isoda and Tanji 2003), indicating an underlying assumption that all of the SEF neurons are in a resting state at that time. Likewise, in 1992, Matzuaka and colleagues used the precue interval as the baseline period (Matsuzaka et al. 1992), which may have prevented them from observing precue anticipatory activity that was reported a decade later (Coe et al. 2002) or the precue state encoding activity reported here. Neural activity has been observed during active fixation in the SEF in tasks in which monkeys were required to maintain fixation but not generate saccades (Bon and Lucchetti 1990; Schlag and Schlag-Rey 1987; Schlag et al. 1992). This active fixation signal may have been the same as precue, memory, or visual feedback state encoding activity because the monkey was required to maintain fixation at these times in the task employed here.

To identify modulations in all of the intervals in the task, such as the precue and intertrial intervals, it was necessary to define the baseline firing rate as the average activity observed for each neuron over the entire recording and not in any one preselected interval (see methods). This definition of the baseline appears to be a novel analytical method of our study and required some caution so as to avoid false positives. For instance, one might suspect that a large activation during the trial might artificially increase the baseline firing rate, and result in the false identification of a modulation (specifically a suppression of activity) in other intervals. For this reason, a very conservative significance threshold (P < 10^-5) was used that also had to survive correction for multiple comparisons. This method was also verified by visual inspection and was observed to work properly. We observed, for example, that neurons with a strong modulation in one interval were not mistakenly classified as having a significantly suppressed activation in all other intervals. Importantly, even if there was a problem of false positives, there is no reason to believe that our definition of baseline firing activity would lead to falsely detected modulations in any one interval (such as the intertrial interval) more so than any other interval. In some cases, neurons were active for multiple consecutive intervals. The analysis presented in Figs. 5 and 6 show the activation patterns of each task related neuron.

We used a memory-guided saccade task to assess firing rate modulations that were related to elapsed time, but in a way that was orthogonal to eye movement planning (see methods). Because events in the task were separated by variable intervals, the monkeys could not exactly predict the occurrence of each subsequent event. This aspect of the task design may have diminished the utility of extremely accurate time interval estimation and consequently affected the percentages of neuron response types that we encountered in our neuronal populations—especially anticipatory activity that could contribute to a high-resolution time interval estimate. While the use of variable intervals was ideal for the cell categorization analysis presented in Figs. 1–5, it posed a challenge to the analyses presented in Figs. 6–9 (see methods). To deal with these challenges, we introduced a spike train standardization procedure. A possible unintended effect of this procedure could have been a better decode between events, but this is where the algorithm did not perform as well as described in results. In the future, it might be fruitful to vary the temporal predictability of targets (Akkal 2004), for instance, by comparing blocks of trials with variable intervals with blocks of trials with fixed intervals.

Alternative interpretations

Individual neurons could serve other functions in addition to their contribution to a sequential state representation. Event detecting neurons, for example responding to the cue presentation, might serve to inhibit a reflexive saccade to the cue stimulus. Neurons responding after the delivery of reward have been proposed to serve a reward detection function (Amador et al. 2000; Stuphorn et al. 2000). We do not see these as conflicting interpretations of our results but rather complementary. Consider that an alarm clock that plays radio news as the alarm signal could serve two functions, one, to wake up the owner and two, to provide him with the morning's news. The variety of circumstances under which event detecting neurons fired, however, implies that none of the previously proposed functional explanations for event detecting activity could be applied to the variety of responses we observed. Note that the example event detecting neuron in Fig. 3A, last row, responded to visual stimuli onsets or offsets in central or peripheral locations. The data presented here suggest that a diverse set of neurons in SEF collectively represent every functional state of behavior and that this could facilitate the expression of behavior in the correct sequence.

Anticipatory activity has been described previously in SEF neurons and was considered as a bias for a particular movement (Coe et al. 2002). Such a bias does not seem a reasonable explanation for the precue anticipatory activity reported in our study because there were 43 potential targets, chosen randomly, and it would be a highly inefficient strategy if the monkey tried to predict the upcoming saccade target. For the neurons exhibiting presaccadic climbing activations that were not spatially tuned, these could not contribute to the motor preparation for the particular saccade that would be executed on any given trial. Instead such neurons may have contributed to the timing of the executed saccade on every trial, and the buildup of activity at this final stage may have contributed to a growing readiness of move the eyes. Similarly, the anticipatory activity in the precue period may have contributed to a readiness to perceive relevant information. Anticipatory activity near the end of the trial may signal how close the monkey was to reward.

State encoding neurons, in addition to representing the current state of the trial, may also serve behavioral purposes such as facilitating fixation (Bon and Lucchetti 1990; Schlag and Schlag-Rey 1987; Schlag et al. 1992) in the presaccadic intervals via inhibitory connections to omnipause neurons (Shook et al. 1990); withholding reflexive movements (Everling and Munoz 2000); preparing for any movement in general, termed “preparatory set” (Everling and Munoz 2000; Hoshi et al. 2005; Shima et al. 1996); integrating task rules for the selection of task-appropriate motor plans (Miller et al. 2005); and stimulus encoding in a manner that is dissociated from the spatial location of the stimuli. State encoding activations in the period between visual feedback and reward delivery have been observed in the nearby supplementary motor area and proposed to signal reward expectation (Campos et al. 2005).

One might suspect that the state encoding activations observed in the precue period could be attributable to microsaccades during central fixation or postsaccadic visual responses following the acquisition of the central fixation point. As detailed in Fig. 2B, however, not only do the precue state-selective neurons begin firing after the fixation point has been acquired, they also stop firing only after the cue was presented. As Schall pointed out with regard to preparatory set activity, it was necessary to use variable intervals to observe this type of activation pattern (Schall 1991b). The time at which the neurons stop firing cannot be attributed to a postsaccadic visual response or microsaccades during fixation.

Neural representation of behavioral timing

The three general nonspatial cell types identified in SEF support both the accumulator (Brody et al. 2003; Macar et al. 1999; Reutimann et al. 2004) and state-dependent (Lucchetti et al. 2005; Mauk and Buonomano 2004) models of the neural representation of behavioral timing. The state-dependent network model is similar to a three-colored traffic light, while the accumulator model is like a pedestrian countdown signal. Anticipatory neural activity, characterized by increases or decreases in firing rate activity prior to external events, which can support precise behavioral timing, has been found in a variety of brain structures including the striatum (Apicella et al. 1992), caudate (Hikosaka et al. 1989; Lauwereyns et al. 2002a,b; Watanabe and Hikosaka 2005), frontal eye fields (Bruce and Goldberg 1985; Coe et al. 2002), dorsal premotor cortex (Mauritz and Wise 1986; Vaadia et al. 1988), cingulate motor area (Niki and Watanabe 1979), supplementary motor area, and presupplementary motor area (Akkal 2004) as well the two areas under study in this report, the supplementary eye fields (Coe et al. 2002; Ohmae et al. 2008) and the lateral intraparietal area (Coe et al. 2002; Colby et al. 1996; Maimon and Assad 2006). Anticipatory neural activity has been supposed to support the expectation of predictable sensory events (Apicella et al. 1992) that are linked to reward (Lauwereyns et al. 2002a), expectation of reward itself (Amador et al. 2000; Campos et al. 2005; Stuphorn et al. 2000), behavioral biases for a particular movement (Coe et al. 2002; Hikosaka et al. 1989; Mauritz and Wise 1986; Takikawa et al. 2002; Watanabe and Hikosaka 2005) to attenuate reaction-time delays (MacKay and Crammond 1987), the readiness to produce or cancel a movement (Libet et al. 1983) that has similarly been described as the beginnings of a proactive triggering process (Maimon and Assad 2006), the inhibition of reflexive movements (Guitton et al. 1985), and tracking behavior, as in smooth pursuit (Heinen and Liu 1997; Shichinohe et al. 2009). Ohmae and colleagues reported climbing activity in the SEF during the delay period of a delayed-saccade task that was not correlated with reaction time. The authors suggested that this activation profile reflected anticipated and elapsed time during this phase of the task (Ohmae et al. 2008).

Anticipatory, or climbing, neural activity is the hallmark of one of the two prevailing views on how behavioral timing is accomplished in the brain (Brody et al. 2003; Macar et al. 1999; Reutimann et al. 2004). The competing model posits that timing is encoded as particular groupings of activated neurons in a network (Lucchetti et al. 2005; Mauk and Buonomano 2004), such as the event detecting and state encoding neurons in our dataset. The two models are not mutually exclusive. The anticipation of events posited by accumulator models and the detection of those events required for network models are complementary processes and have long been observed within the firing of single neurons (Fuster and Alexander 1971). In support of both models, Niki and Watanabe first discussed timing units, which included neurons with anticipatory rises before perceptual events and during motor preparation as well as tonic activations extending through the delay period (Niki and Watanabe 1979). More recently, there have been studies presenting neural activations not related to spatial parameters that provide further evidence for the network-based representation of behavioral timing, including state-encoding responses in the rostral cingulate motor area before, during, and after every event in a trial, thereby reflecting each step of the behavioral task (Hoshi et al. 2005) and event detecting activity in the claustrum preceding all movements in a nonselective manner (Shima et al. 1996). Results such as these have been the basis of the hypothesis that the prefrontal cortex is primarily engaged in the temporal structuring of behavior (Fuster 1989).

Our results support both of these behavioral timing models. Event detecting neurons could serve to transition the local network into a new state, which would be in turn stably maintained by the activity of state-encoding neurons. Anticipatory neurons could also accumulate inputs from the tonic state encoding neurons, providing high-resolution estimates of the elapsed time since the previous event or the time remaining before the next event. These three types of units can therefore work together to provide temporal information on multiple time scales, suggesting that state-dependent network and accumulator models describe the representation of time within SEF at different levels of temporal resolution.

Flexible representation of sequential states?

The task employed in this study could not determine whether individual neurons encoded the ordinal position of each behavioral state or the behavioral states themselves independent of the order (Berdyyeva and Olson 2009) because we could not permute the various events. For instance, we could not instruct the monkey to wait for the spatial cue after the cue had already been presented. We discussed this issue in a recent report of an experiment featuring two instructional cues (Campos et al. 2009). Suppose that there are two cues presented asynchronously in a task, and an SEF neuron is found to be tonically active between the presentations of the cues. Such a neuron might fire because the first event just occurred or the second event is about to occur or because the monkey is currently waiting between events one and two. If the presentation order of the cues is reversed, the activation patterns of individual neurons might change as a consequence depending on the “meaning” of the activation pattern. We hypothesize that the nature of the change, if any, would depend on how the monkey has learned the task, how often the temporal order of the cues was reversed, how predictable was the reversal, and how those cues were utilized in the task. We further hypothesize that several neurons would fire at a different task interval but that several would remain unchanged (Berdyyeva and Olson 2009). This issue, however, does not affect the ability of the population of SEF neurons to encode the sequential states of a task. Even if several neurons fired at a different interval, the sequential state representation could still be decoded based on the set of neurons active in each interval. Our paper suggests that, while many individual activation profiles can be found in SEF that might reflect individual behavioral states or the ordinal position of states, when an aggregate view is taken, the resulting population characteristics can serve the function of piecing together the various temporal epochs, and their related task demands, to form a coherent string of behaviors required to perform even the most basic task.

Is timing a general function of the medial frontal cortex?

Supplementary motor cortex is composed of three subregions—the SEF, which is specialized for movements of the eyes, the supplementary motor area (SMA), and the presupplementary motor area (pre-SMA), which are primarily concerned with movements of the body and limbs. It is not clear from the results presented here if timing is a general function of supplementary motor cortex or if the representation of sequential states in SEF is particular to eye movement tasks. Would SEF neurons respond in a similar fashion during an arm-movement task? Previously, we reported evidence for a complementary question, whether timing information concerning an eye movement task is present in the SMA. We found a neural activation in the SMA that signaled the expectation of reward at the end of an eye movement task (Campos et al. 2005) that could be useful in predicting the onset of a reward delivery event as has been found previously in SEF (Amador et al. 2000; Stuphorn et al. 2000). We hypothesized that it would be useful to the monkey to broadcast reward information broadly to support learning and to acquire additional rewards more efficiently in the future. For example, if there was some combination of limb or body movements that influenced the amount of reward that monkey received, the reward expectancy signal in SMA and SEF could reinforce whatever high-level motor plans were spontaneously active prior to a large reward delivery.

We suspect that while all of the supplementary motor cortex contains reward related information, it is also an area where signals that will be used to control arm versus eye movement behaviors are to some degree segregated. We would hypothesize that SMA or pre-SMA neurons would collectively represent the sequential states of a delayed or memory-guided arm movement task. That is, there would be event detecting, anticipatory, and state encoding activity present for each state and event for a task in which the monkey was required to control the movements of his arms. We hypothesize that SEF sequential state activations in such a task would be limited by the demands placed on eye movement behavior. If the monkey was required to fixate for the entire trial, there might be neurons that represent the oculomotor state of fixation (from fixation acquire until the end of the trial) with very few or no neurons responding to other events in the trial. If the monkey was not required to fixate, we would hypothesize that SEF would only become active around the time of reward delivery because the reward related signals are broadcast broadly.

Using a button-hold task, Mita and colleagues found “time-graded” neurons in the SMA and pre-SMA that exhibited changes in elapsed time similar to the neurons that we described as “anticipatory.” In addition, many neurons in the pre-SMA, but rarely in the SMA, were “time-selective,” meaning that they encoded the instruction given in the task (Mita et al. 2009). The authors suggested that pre-SMA was therefore involved in retrieving information to structure the forthcoming motor behavior. This time-selective activity could therefore be similar to what we described here as “state encoding” because the monkeys learned that each behavioral state had its own range of possible durations. Our sample size was too small and our recording locations were not done in such a systematic manner as to judge whether there was a rostral-caudal segregation of state encoding versus anticipatory neurons. If such segregation does exist, it could form the basis for the identification of a pre-SEF in monkeys.

Hierarchy of sequence information

Our study has shown that a population of neurons in the SEF reflects every behavioral state related to an instructed eye movement behavior. Previous work has shown that SEF neurons can also represent individual eye movements within an instructed sequence (Lu et al. 2002), mirroring findings about sequential limb movements in the supplementary motor area (Tanji and Shima 1994). Based on these results and others, supplementary motor cortex has long been thought to play a role in programming sequential movements of the eyes and limbs. In contrast, neurons downstream from the SEF, in primary motor cortex, do not have cellular activity that suggests a role in the programming of sequential behavior (Tanji and Shima 1994). Our results show that the representation in SEF is not limited to eye movement encoding but also extends to all aspects of an instructed eye movement task, including fixation and intertrial intervals.

An instructed eye movement or sequence of eye movements, however, is only a small portion of the behavioral programs that must be operating within our subjects. Each trial is situated within a recording session, during which there were individual periods of focus or disengagement from the task. There were then multiple recording sessions during an experimental session, which in turn only comprised a fraction of the monkey's day. It is likely that neurons in other areas might encode sequences of behavior on these longer time scales. Fujii and Graybiel found neurons in an area lateral and maybe slightly anterior to the SEF that were selectively active at the end of saccade sequences (Fujii and Graybiel 2003). This suggests that sequences of actions are represented at multiple time scales with some regions encoding all relevant behavioral states in service to a particular action (such as reported here in SEF) and others representing the start and end of coordinated groups of actions. A related theory has been proposed for cognitive control more generally (Koechlin et al. 2003). Mansouri and colleagues likewise found neurons in the same region as Fujii and Graybiel that were selective to a given rule across multiple trials, or transitions between blocks of trials (Mansouri et al. 2006), paralleling the state encoding activity described here on a longer time scale.

GRANTS

This work was supported by the National Eye Institute and the James G. Boswell Foundation.

DISCLOSURES

No conflicts of interest, financial or otherwise, are declared by the author(s).