Timescales of Sensory- and Decision-Related Activity in the Middle Temporal and Medial Superior Temporal Areas

Abstract

The contribution of sensory neurons to perceptual decisions about external stimulus events has received much attention, but it is less clear how sensory responses are integrated over time to produce decisions that are both rapid and reliable. To address this issue, we recorded from middle temporal area and medial superior temporal area neurons in rhesus macaques performing a task requiring the detection and discrimination of unpredictable speed changes. We examined how neuronal activity encoded the sign of the speed change and predicted the animals' behavioral judgments and reaction times, with a focus on the timescales over which neuronal activity is informative. False detection trials, on which animals reported a speed change even though none had occurred, were grouped according to the animals' discrimination judgment. By comparing the neuronal responses between the two groups of false detection trials, we were able to predict the animals' choices from the sensory activity of single neurons at levels significantly better than chance. These choice probability measurements were strongest using spike counts in an 80 ms window ending 150 ms before a choice saccade began, but significant choice probabilities were observed in windows as short as 10 ms. While the maximum deviation in spiking rate following a speed change is evident in the transient response, averaging neuronal activity in longer time windows can be more informative about both the stimulus and the animals' behavioral judgments. Thus the timescales found in this study represent a trade-off between producing rapid reactions and overcoming the noise inherent in short time windows.

Introduction

How does the brain balance the need for fast judgments with the desire to make accurate decisions? A person driving a car through dense fog may barely discern an oncoming vehicle, but still needs to estimate its speed and make rapid adjustments that depend on this estimate. Under such circumstances there is a trade-off: longer viewing times potentially improve accuracy, but delay reaction. One limit on the speed of sensory decisions is the rapidity with which neurons can modulate their firing rates in response to environmental changes. Sensory neurons in the primate visual system carry significant information on timescales of tens of milliseconds. For example, single neurons in the middle temporal area (MT) convey reliable information about direction of motion within 75 ms of motion onset (Pack and Born, 2001) and this activity supports eye movements at very short latencies (Kawano and Miles, 1986; Ibbotson et al., 2007). However, simply because stimulus information is rapidly available does not mean that perceptual decisions are based on activity over the same timescales. By averaging neuronal activity across time, it should be possible to reduce noise and improve behavioral performance.

Early studies linking the responses of sensory neurons to behavioral choices used stimuli of fixed, 1–2 s durations (Britten et al., 1996; Uka and DeAngelis, 2004; Liu and Newsome, 2005). While much was learned from these paradigms, they are not as informative about the timescales relevant for sensory integration because animals do not necessarily use the entire trial to formulate their decisions (Uka and DeAngelis, 2003; Huk and Shadlen, 2005), and sensory neurons do not integrate information uniformly over time (Buracas et al., 1998; Luna et al., 2005). Recent studies addressed these shortcomings using reaction time tasks, in which both the animal and the neurons were forced to integrate sensory information over behaviorally relevant timescales (Cook and Maunsell, 2002; Thiele and Hoffmann, 2008; Cohen and Newsome, 2009). However, these studies did not explore the metrics of the integration windows used to guide perceptual choices. One study that did explore this dimension used very brief stimulus perturbations (<85 ms), which by their very nature produce only transient neuronal responses (Ghose and Harrison, 2009). This design would predetermine the observation of fast timescales, since longer integration times could only impair performance, whether neuronal or behavioral.

We thus investigated the timescales of neuronal processing contributing to the detection of unpredictable, but sustained, stimulus changes. Macaque monkeys were trained to detect changes in the speed of a moving stimulus, and to also discriminate whether the speed increased or decreased. Even though MT and medial superior temporal area (MST) neurons recorded during the task continued to respond to the sustained change, which lasted up to 750 ms, the most informative integration windows were surprisingly short, averaging just a few tens of milliseconds. A measure of rapid adaptation (the transient-sustained index) was negatively correlated with a neuron's optimal integration window and its discrimination performance, suggesting that while adaptation may limit the advantages of longer viewing times, it is the neurons with sustained responses that are most informative.

Materials and Methods

Two adult male monkeys (Macaca mulatta, 9–11 kg) were trained to perform a speed change detection and discrimination task, requiring them to detect a step change in speed of a visual stimulus and indicate with a saccadic eye movement if the change was an increase or decrease in speed. The initial training process in which the animals associated the sign of a speed change with a specific saccade target took 1–2 weeks. Electrophysiological recordings commenced after ∼6 months of additional training, which was necessary for each animal to have stable psychophysical performance across different stimulus locations, directions, speeds and durations. Before electrophysiological recordings, each animal was implanted with a custom titanium head-post, a scleral search coil for monitoring eye position and a vertically aligned Cilux recording cylinder (Crist Instrument Co.) to protect a craniotomy centered posterior 3 mm, lateral 15 mm relative to ear bar zero. All animal procedures complied with the National Institutes of Health Guide for Care and Use of Laboratory Animals, and were approved by the Harvard Medical Area Standing Committee on Animals.

Stimuli and task.

Monkeys were trained to perform a speed change detection-discrimination task while fixating a central target. The trial sequence is shown in Figure 1A. On each trial, 200 ms after the animal acquired fixation, two saccade targets were presented for 450 ms, followed by a random duration of reference motion (8 or 12°/s) and up to 750 ms of test motion (4–18°/s). The saccade targets were a green triangle and magenta cross used to indicate speed increases and decreases, respectively. The targets were typically placed 6° above and below the fixation point and shifted laterally so that targets were at least 2° away from the receptive field boundaries. The duration of the reference period was between 500 and 5000 ms. Durations were randomly chosen from a truncated exponential distribution with time constant τ = 550–1000 ms, chosen to maintain the percentage of rewarded trials at ∼70–85%. Trials with each reference speed and all speed ratios were randomly interleaved.

Figure 1.

Stimulus sequence and behavioral performance. A, After the animal acquired fixation, two saccade targets appeared, followed by a random duration of reference motion (8 or 12°/s) and up to 750 ms of test motion (4–18°/s). At any time, a saccade could be made to the green triangle or magenta cross to indicate that the stimulus speed became faster, or slower, respectively. B, Behavioral responses on trials with reference speed 8°/s and 12°/s are shown separately for monkey N and P. For each speed ratio, the percentage of faster, slower and missed responses sums to 1. C, Median reaction times (±quartiles) on correct discrimination trials are shown separately for each reference speed. Only responses in the window 150–750 ms after the speed change were considered correct discriminations. D, Percentage of started trials that ended as false detections with faster or slower choices. False detections occurred on 15% (monkey N) and 25% of trials (monkey P).

The animal was rewarded for correctly indicating when the stimulus increased or decreased in speed by making a saccadic eye movement to the appropriate choice target within 150–750 ms of the change. During recording, trials were terminated without reward if the eye position prematurely left a square window of 2–3° side length centered on the fixation target. In addition, a trial was only included in offline analysis if the total eye displacement before the response saccade was <1–1.5°. Reaction times were calculated for correct discrimination trials in a two step process. First, the time was found when the animal's eye position first left the fixation window and moved toward the choice target. Working backwards in time from this marker, the time of saccade onset was taken as the earliest time when the eye speed first exceeded 100°/s.

Stimuli were white dots presented on a black screen placed 410 mm from the animal, with resolution 1024 × 768 pixels (17.8 pixels/degree) and refresh rate 100.1 Hz. Dots had spatial density 1.5 dots/deg², 150 ms lifetime and had 100% coherence (i.e., there were no noise dots, but the dots flickered because of the limited lifetime). Dots were restricted to either a stationary circular aperture that matched the excitatory receptive field center of the neuron being studied (MT neurons) or a circular aperture with a diameter equal to the eccentricity of the stimulus center (MST neurons). For each neuron, direction tuning was initially characterized while the animals were rewarded for passively maintaining fixation on a central target. Subsequent stimuli were presented moving in the direction that evoked the largest response. For most neurons we also characterized speed tuning using dot speeds of 0.5–128°/s.

Neuronal dataset.

We recorded single-unit and multiunit data from MT and MST in visual cerebral cortex using methods described previously (Born et al., 2000). We focused on these areas because their input and output connectivity are well understood, the visual motion stimuli that modulate neuronal activity are well characterized, and a range of perturbations such as lesions and microstimulation have shown them to be critical for motion perception (Maunsell and Van Essen, 1983a,b; Newsome and Paré, 1988; Salzman et al., 1990; Celebrini and Newsome, 1995).

Neurons were assigned to MT or MST based on a range of criteria including the following: position of the electrode in the chamber judged relative to postsurgical magnetic resonance imaging; electrode depth relative to the dura; size of the spatial receptive field relative to the eccentricity; overlap of the receptive field into the ipsilateral hemifield; and transitions between gray and white matter as the electrode was advanced (MT was typically encountered at depths beyond MST). Only neurons with at least 32 completed discrimination trials were included in analysis. In monkey N we recorded 132 neurons (MT: 62 single, 21 multiunit; MST: 36 single, 13 multiunit). In monkey P we recorded 71 neurons (MT: 56 single, 13 multiunit; MST: 2 single unit). Across the population of neurons, the median number of completed discrimination trials was 166 (quartiles 113 and 294). No qualitative differences were observed between single- and multiunit recordings, nor between brain regions, thus data were combined across these categories. Significant differences between monkeys are highlighted in the text.

Three metrics have become common in analyses linking a neuron's responses to stimulus and behavior on single trials. The first, neurometric tuning, quantifies the ability of an ideal observer to perform the task using only a single neuron's firing rate on each trial (Britten et al., 1992). Second, choice probability (CP) assesses how reliably a behavioral choice can be predicted from a neuron's firing rate (Britten et al., 1996). Finally, the correlation between firing rate and behavioral reaction time indicates what aspects of a single neuron's response might influence the speed of an animal's decision (Roitman and Shadlen, 2002). While widely applied, these metrics give little insight into when or how long sensory activity provides information about stimulus changes or behavioral choices because they use fixed time windows for calculating spike rates. To address this, we calculated the three metrics using spike counts in a range of time windows relative to stimulus changes and behavioral choices.

Neurometric performance.

Neurometric performance (NP) is the proportion of trials in which the sign of the change in stimulus speed (i.e., faster versus slower) can be predicted from the sign of the change in a single neuron's firing rate. All trials in which a speed change occurred were analyzed, regardless of whether the animal actually detected the change (i.e., faster choices, slower choices and missed changes). Across the population of neurons, there were no consistent changes in firing rate 500–5000 ms after the start of the reference period, thus trials were aligned on the time of the speed change, regardless of the reference duration. Part of our rationale for only examining speed changes >500 ms after motion onset was to ensure that neuronal responses were in “steady-state,” and therefore responses to the speed change would not be conflated with transient responses to motion onset.

For single trials, a difference rate was calculated using spiking rates in a test and reference time window (R_Diff = R_Test − R_Ref). Test windows had variable width (10, 20, 40, 80, 120, 160, 200, 240, 280 ms) and end times of −200 to 300 ms relative to the time of the speed change, calculated with 10 ms resolution. Responses are always aligned on the end of the window, since this is the earliest time at which information from the full window is available. Reference windows were the 300 ms period immediately before the start of the test window, regardless of the test window width. Time windows that included periods after the choice saccade began were truncated to the time of saccade onset. Identical trends were observed with reference window widths of 100–500 ms or if the test and reference window had matching widths. However, there was greater variability in performance with shorter reference windows. More significantly, with short reference windows there was a larger decline, or even reversal in NP <0.5, for short test windows that ended >150 ms after the speed change (see Fig. 3B, lower right regions).

Figure 3.

Temporal evolution of neurometric performance for each animal. A, Median (±quartiles) neurometric performance in a sliding time window of 120 ms width. The vertical line at 180 ms indicates the end time of the window used to calculate neurometric performance in Figure 4A. B, Median (±quartiles) neurometric performance (black) in a time window of variable width, that starts 50 ms after the speed change. The gray line shows the peak value of the median neurometric performance for all windows of a given width, but with the window start time chosen to maximize performance. C, Median neurometric performance in sliding time windows of variable width (10–280 ms) aligned on the time of the speed change. D, Distribution of the ends (x-axis) and widths (y-axis) of the spike counting window that gives the largest neurometric performance for each neuron, shown as a pseudo-color two-dimensional histogram. The colorbar indicates the number of reference conditions associated with each time bin. Each neuron contributes twice because there are two reference speeds. Optimal window widths measured with each reference speed were correlated across neurons (monkey N: r = 0.39; p < 0.01. monkey P: r = 0.42; p < 0.05).

The sign of R_Diff was mapped to a vote for faster or slower depending on its relationship with the sign of the slope of the speed tuning curve at the reference speed, measured during fixation. For the majority of neurons, the reference speed falls on the positive-slope flank of the speed tuning curve, thus increases in firing rate were mapped to an increase in speed while decreases in firing rate and 0 changes were mapped to decreases in speed. This relationship was reversed for neurons in which the reference speed fell on the negative-slope flank of the speed tuning curve. Neurometric performance then represents the overall percentage of correct predictions, averaged across speed ratios. Thus, NP > 0.5 for all neurons with clear speed tuning.

Choice probability.

We calculated choice probabilities to quantify the correlation between a neuron's firing rate and the perceived speed change on single trials. Trials in which animals falsely reported a speed change were aligned on the start time of the choice saccade and grouped according to whether a faster or slower choice was made. The two distributions of firing rates were then compared by calculating the area under a receiver-operating characteristic curve, a technique derived from signal detection theory (Green and Swets, 1966). This area is termed choice probability (CP) and is a value between 0 and 1 representing the ability of an ideal observer to predict the animal's choice from the neuron's firing rate on a single trial (Celebrini and Newsome, 1994; Britten et al., 1996). Firing rates were measured in fixed time windows with width 5–300 ms and end points from 0 to 500 ms before saccade onset (5 ms resolution).

CP was calculated independently for each reference speed, including a neuron's data only if responses were recorded during at least 6 faster and 6 slower false detections. Across the population, the median number of false detections for each reference speed was 41 (quartiles 24 and 67). To test significance of individual CP measurements, we used a permutation test (n = 2000), in which CPs were calculated from the same overall distribution of spiking rates, but with trials randomly assigned to the faster or slower choice categories. Only CP values outside the central 95% confidence interval of this shuffled distribution were deemed significant.

As in the calculation of neurometric performance, the calculation of CP takes into account the slope of the neuron's speed tuning curve at the reference. Thus CP > 0.5 indicates that the mean difference between responses on faster and slower false detection trials has the same sign as the difference between test responses to faster and slower speed ratios. For example, if a cell has a preferred speed >12°/s, then speed increases generate higher firing rates than speed decreases. If, on average, the neuron's firing rate is higher when the animal makes a faster than a slower false detection, the, CP will be >0.5.

Reaction time–spiking rate correlation.

The correlation between reaction time (RT) and neuronal responses (R) was calculated for all correct discrimination trials with the same test and reference speeds. Only neurons with at least 16 correct discrimination trials of the fastest test-reference speed ratio were included. However, trials in which the spiking rate was >4 SDs from the mean were discarded as outliers, to avoiding skewing the correlation values.

For the fixed time window analysis (see Fig. 5), 3 different spike rates were measured: a reference response (R_Ref, 0–300 ms before the speed change); a test response (R_Test, 60–180 ms after the change); and a difference response (R_Diff = R_Test − R_Ref). For the variable time window analysis (see Fig. 6), spike rates were counted in sliding time windows of width 10, 20, 40, 80, 120, 160 and 200, with end times of −200 to 300 ms relative to the time of the speed change, in 10 ms increments.

Figure 5.

Correlation between reaction time and spiking rate. A, Large colored spots show the median reaction time and the mean spiking rate of a single neuron for 8 test speed ratios, with reference 12°/s. Smaller blue and red points show reaction times and spiking rates from individual trials with speed ratios 0.5 (58 trials, blue) and 1.5 (59 trials, red). Spiking rates are quantized because spikes were counted in fixed windows 60–180 ms after the speed change. Solid blue and red lines shows linear regression for the data with speed ratios 0.5 and 1.5, respectively. Vertical black line indicates the mean reference firing rate. B, Population summary of the correlation between the difference firing rate and reaction time on single trials— r(R_Diff,RT). Trials with speed ratio 1.5 and reference speed 8°/s (x-axis) and 12°/s (y-axis) are shown. Solid red and blue lines in the scatter plot indicate the mean correlation. Black line indicates regression (r = 0.29, p < 0.01).

Figure 6.

Correlation between reaction time and spiking rate, in sliding time windows of variable width. A, Mean ± SEM correlation, for windows of width 120 ms, ending from −100 to 300 ms relative to the time of the speed change. Horizontal lines indicate the time windows in which the mean correlation was significantly <0. B, C, Mean correlations for monkey N (B) and monkey P (C) are shown for window widths 10–200 ms. White outlined regions indicate time windows in which the mean correlation was significantly <0. Solid white regions in the lower right of C indicate time windows in which there were no neurons with sufficient trials to calculate the correlation. This occurred because some trials had reaction times <300 ms.

Results

Psychophysical performance

Monkeys performed a reaction time task in which they made a saccade to one of two targets when they detected a randomly timed change in the speed of a visual stimulus. Their choice of saccade target was used to discriminate increases from decreases in speed (Fig. 1A). The initial, reference speed of 8 or 12°/s lasted 500–5000 ms (τ = 550–1000 ms) and was followed by a step change to a test speed of 4–18°/s. Behavioral responses were assigned to one of five categories: (1) faster or (2) slower, if a saccade was made to the corresponding choice target 150–750 ms after the speed change; (3) miss, if fixation was maintained for >750 ms after the change; (4) false detection, if a choice saccade was made during the reference period or in the first 150 ms of the test period; (5) fixation break, if the eye position left a 2–3° fixation window, but no saccade was made to either choice target within 100 ms. Only correct detection and discrimination was rewarded.

Behavioral performance depended on the test-reference speed ratio. Speed ratios closer to 1 were associated with more frequent missed changes (Fig. 1B, blue curves) and a higher median and broader distribution of reaction times (Fig. 1C). For all speed ratios, the primary errors were in detection, not discrimination, of the speed change; i.e., if a change was correctly detected then it was typically discriminated correctly (99% correct). This suggests that the animals only responded when they were certain that they had detected the speed change. The alternative—discrimination without detection—is quite an un-natural decision, and typically only arises when subjects are engaged in forced choice discrimination tasks, in which they must make some judgment about a stimulus even if they did not consciously detect it.

The reference speed biased the distribution of false and failed detections. With a reference of 8°/s, false detections consisted predominantly of slower choices, and failed detections were more common for 20% increases in speed than for 20% decreases. With a reference speed of 12°/s, the opposite pattern of responses was observed (Fig. 1B,D). This suggests that animals were not making a purely relative comparison but may have also incorporated the average speed over many trials into their judgments. The extreme version of this tendency would be absolute speed judgments in which the animal compared the current speed to a fixed internal reference consisting of the average speed across all trials. It is clear, however, that neither animal used such an absolute strategy—a test speed of 9.6°/s could occur as either a 20% increase from a reference of 8°/s or a 20% decrease from a reference of 12°/s, but the sign of this speed change was reliably reported when it was detected (Fig. 1B,C, arrows).

Single-neuron discrimination performance

Responses of a single neuron are shown aligned on the time of the speed change during the behavioral task (Fig. 2A). The responses reflect the neuron's tuning and preferred speed of 32°/s during passive fixation: speed increments increased the firing rate and speed decrements reduced firing rate. The responses of most neurons to the speed change had a prominent transient component, thus the tuning curve measured during the task depends critically on the time window used to average the firing rate (Fig. 2B, average rate 60–120 and 160–220 ms after the change). Tuning curves were steeper for the earlier time window, suggesting that the period immediately after the speed change is more informative about the sign of the speed change.

Figure 2.

Single-neuron responses during the task. A, Responses to 8 test speeds following a 12°/s reference, aligned on the time of the speed change. B, Speed tuning of the same neuron based on the averaging spike rate 60–120 ms and 160–220 ms after the change. SEs were smaller than the data markers. Black data point at speed ratio 1 shows the mean ± SD of the reference response (0–300 ms before speed change). Inset shows the speed tuning measured during passive fixation, 60–120 ms after motion onset. C, Distribution of single-trial test and reference responses, for trials with the speed ratio 1.2 (filled points in B). Diagonal lines indicate rate differences of −40,0 and 40 spikes/s. Histogram shows the distribution of difference firing rates (R_Test − R_Ref). Overall, test firing rate is greater than reference on 81% (blue) and 53% (red) of trials. For clarity, points have been randomly jittered by Gaussian noise, μ = 3 spikes/s. D, Neurometric tuning, given by the percentage of trials in which the test firing rate was greater than the reference firing rate.

A single neuron's ability to discriminate speed changes, or “neurometric performance” was determined from the proportion of trials in which the sign of the change in firing rate between the reference and test period correctly predicted the sign of the change in stimulus speed. Traditional approaches to NP use signal detection theory to compare the response distributions of a neuron and its anti-neuron, which is defined to have the opposite tuning preference to the neuron under consideration. Rather than comparing two spiking rate distributions, our approach has the advantage that that it only uses the single-trial response of a single neuron, but at two times. In calculating NP, we take the sign of the tuning curve slope at the reference into account, so that for neurons with positive and negative slope, increases in firing rate vote for speed increases and decreases, respectively.

Figure 2C compares a single neuron's test and reference firing rates from trials with reference 12°/s and test 14.4°/s (Fig. 2B, filled markers). If the test firing rate is greater than the reference firing rate, the neuron is assumed to vote for a speed increase on that trial. This occurred on 81% of trials for the 60–120 ms test window and 53% of trials for the 160–220 ms window for this speed ratio, and is shown for all speed ratios in Figure 2D. The overall NP is the average percentage of correctly discriminated trials across all speed ratios, and was 92% and 76% for the early and late time windows, respectively. We collapse NP across all speed ratios because we want a single metric for quantifying performance, and all neurons were tested with the same set of stimulus conditions.

Neurometric performance was measured in 180 MT and MST neurons recorded in 2 animals. Trends observed with MT and MST neurons were similar, thus neurons from the two areas are combined here, and in later population analysis. There was no simple relationship between a neuron's preferred speed and NP. Neurons with preferred speeds close to the reference had NP near chance (supplemental Fig. 1, available at www.jneurosci.org as supplemental material). However, neurons with high preferred speeds did not automatically have strong NP, because their firing rates may have only been weakly modulated in the tested range of 4–18°/s. More importantly, NP was strongly correlated with the speed tuning curve slope at the reference speed (r = 0.56, p < 0.01; supplemental Fig. 1, available at www.jneurosci.org as supplemental material). While comparable studies assessing neurometric tuning placed the reference speed near the center of a flank of each neuron's tuning curve, we did not optimize the stimulus speeds for each neuron to make them maximally informative during the task, accounting for the higher proportion of highly sensitive neurons in other studies (Liu and Newsome, 2005; Krekelberg et al., 2006).

Figure 2 demonstrates that the choice of spike counting window critically affects the ability of single neurons to discriminate speed changes; however, it is not clear how to define the optimal time window. Trivially, it makes sense that if the spiking rate is compared between two time windows, they should straddle the time of the change. Since the stimulus speed is constant after the change, we predicted that increasing the duration of the spike counting window should increase the information about the stimulus and give higher accuracy. To characterize the temporal evolution of neurometric performance we compared spike rates in a test window (end points −100 to 300 ms relative to the speed change and widths 10–280 ms) with rates in an immediately preceding 300 ms reference window. Median NP calculated in a sliding 120 ms test window is shown separately for each animal in Figure 3A. In both animals, performance first deviates significantly from chance (p < 0.05) in the test window ending 60 ms after the speed change and peaks 180 ms after the change. This reflects the 50–60 ms latency of most neurons and the observation that the transient response following a speed change provides a steeper speed tuning curve than later responses. These trends were observed if we examined neuronal data from area MT or MST independently, and if we examined single-unit or multiunit data independently (Figs. 2, 3).

Importantly, NP does not continue to increase as window width is increased. In Figure 3B, median NP is shown for windows of variable width that start 60 ms after the speed change. In both animals, NP reaches a plateau for window widths of 80–120 ms. This plateau is also evident in Figure 3C, which shows median NP across the full range of window end times and widths: performance first deviates from chance after 50–60 ms and peaks at the latest time window that includes the period 50–60 ms after the speed change. For all window widths, periods that include the transient response after the speed change were more informative than later periods that exclude the transient response. It is also clear that population performance saturates for window widths of 120–160 ms; median NP did not increase with wider test windows.

While monitoring activity in wider test windows increases the overall spike count, it has the deleterious effect of reducing spike rate for neurons with transient responses following the speed change. Since neurons show different levels of transient activity, we found for each neuron the single test window that maximized NP (Fig. 3D). There was large variability in optimal window widths across the population, but their end times typically fell along a diagonal such that almost all windows included the transient response ∼60 ms after the speed change. The optimal window width was negatively correlated with a measure of rapid adaptation, the transient sustained index (TSI) (r = −0.15, p < 0.01). The TSI was calculated from the transient (R_Trans, 50–150 ms) and sustained (R_Sust, 150–400 ms) components of the response to the reference motion: TSI = (R_Trans − R_Sust)/(R_Trans + R_Sust). Thus neurons with more transient responses to reference motion onset typically had shorter optimal window widths for decoding the stimulus change.

Two observations demonstrate that, the more transient neurons were overall less informative about the stimulus. First, TSI was negatively correlated with the actual value of NP in the optimal window (r = −0.20, p < 0.01). Second, NP was also negatively correlated with the Fano factor (FF) of the spike count across trials (r = −0.20, p < 0.01). Fano factor was calculated using spike counts in a window 500–600 ms after the start of the reference period, by taking the ratio of the variance and mean of the spike count. Further analysis demonstrated that the precise spike counting window did not affect the nature of these correlations. This suggests that the most informative neurons are those with the lowest variability in their spiking rate, both across and within trials.

The prevalence of short optimal window widths indicates that even when a stimulus was presented for an extended period of time (up to 750 ms), many neurons would be most useful for decoding the stimulus immediately after the speed change. This goes against the intuition that longer spike counting windows should improve accuracy. Despite this, the more informative neurons were actually those with more sustained responses (low or negative TSI) and less variability across trials (low FF). The large variability in optimal window widths may provide a balance between noise susceptibility (with short integration times) and sensitivity to small changes (with long integration times). Alternatively, they may reflect the unintended consequences of a circuit mechanism such as rapid adaptation that serves some other purpose, such as adjusting dynamic range to the statistics of a rapidly changing environment.

Correlations between spiking rate and behavioral choice

On many trials, animals made a saccade to a choice target during the reference period—a false detection of a speed change. CP was calculated from false detection trials by comparing neuronal responses on trials in which the animal indicated it perceived speed increases or decreases (see Materials and Methods). CP is analogous to neurometric performance, but it quantifies how well a neuron's spiking rate correlates with the animal's decisions, rather than the stimulus change. As with NP, the spike counting time window is critical—firing rates averaged over many seconds are unlikely to predict behavior, but at the other extreme, the correlation between behavioral choice and spiking rates in brief time windows is susceptible to noise.

Across the population of neurons, CP and NP were weakly correlated such that neurons more informative about the stimulus were also more informative about the animal's choices (monkey N: r = 0.19, p < 0.05; monkey P: r = 0.31, p < 0.01) (Fig. 4A). For this comparison, we used spike rates from 150 to 230 ms before saccades on false detection trials (CP) and 60–180 ms after the speed change (NP). One concern with this analysis is that the correlation between NP and CP may only be significant for this specific combination of time windows. To address this, we calculated the correlation between NP and CP in 6 combinations of windows, using spike counts from 2 nonoverlapping CP windows and 3 nonoverlapping NP windows (150–190 and 190–230 ms before the change for CP; 60–100, 100–140 and 140–180 ms after the change for NP). For the CP window 150–190 ms before the speed change, r(NP,CP) varied from 0.15 to 0.22 and was significantly positive (p < 0.01) for all three NP windows. For the CP values calculated 190–230 ms before the change, r(NP,CP) varied from 0.08 to 0.14, and was significant (p < 0.05) in 2 of the 3 NP windows. Clearly, r(NP,CP) falls to 0 when the spikes are unrelated to the stimulus or the decision (e.g., an NP window >500 ms after the speed change), but this demonstrates that the correlation between NP and CP is not simply due to injudicious choice of spike counting windows.

Figure 4.

Temporal evolution of choice probability. A, Population comparison of CP and NP. Solid lines shows linear regression (r_N = 0.17, p < 0.05; r_P = 0.31, p < 0.01), demonstrating that neurons more sensitive to the stimulus were also more informative about the animal's choices during the task. Trials with the 8 and 12°/s reference speeds were analyzed independently, thus each neuron contributes twice to this summary if sufficient trials were recorded. This was the case in 172 neurons with 295 reference conditions. B, Distribution of choice probabilities from cells with NP > 0.75 (72 neurons, 111 reference conditions). Mean CPs calculated for individual animals were significantly greater than chance (p < 0.01). C, CP calculated in a sliding 80 ms window for neurons with NP > 0.75. Traces are aligned on the time of the choice saccade and show mean ± SEM. Spots at bottom of panel indicate time windows in which CP was significantly different from 0.5 (p < 0.05). D, Mean CP across neurons with NP > 0.75 for window widths 5–300 ms. The region between the two black lines demarcates windows that include spikes in the period 180–200 ms before saccade initiation. White boxes indicate time windows in which the mean CP is significantly different from chance (t test, p < 0.05).

Across all neurons, the mean CP was weakly, but significantly greater than chance (For both animals, μ = 0.52, p < 0.01). This was also true if we examined neuronal data from area MT or MST independently (supplemental Fig. 3, available at www.jneurosci.org as supplemental material). In previous studies, it has been common to optimize stimulus parameters to match each neuron's preferred direction and speed. In our experiments, we optimized stimulus direction, but not speed, meaning that many neurons were unresponsive to the range of stimulus speeds, and thus had neurometric performance close to chance. For comparison with earlier studies in which stimulus parameters were optimized for each neuron, we also considered a subset of the 40% of neurons that that were most informative about the stimulus (NP > 0.75). For this subset, the mean CP for each animal was higher than the overall population mean and was significantly greater than chance (monkey N, 0.553; monkey P, 0.545) (Fig. 4B). Further, NP and CP were significantly correlated in one animal (monkey N: r = −0.11, p = 0.42; monkey P: r = 0.31, p < 0.05). This is consistent with downstream neurons involved in the decision making process preferentially monitoring the sensory activity of MT/MST neurons that are most informative about the stimulus.

It remains unclear whether CP in sensory areas reflects variability and noise in the feedforward sensory inputs that subsequently influences decisions, or whether it also captures attention or postdecision feedback from higher cortical regions (Herrington and Assad, 2009; Nienborg and Cumming, 2009). Some insight into this can be gained from examining CP in sliding time windows of different durations. The feedforward hypothesis predicts that CP should be transiently elevated above chance shortly before a saccade and return to chance at the time of the saccade, since sensory information in the ∼50 ms before a saccade cannot affect the planned movement. A feed-back signal associated with attention or decision need not abate at the time of the saccade, predicting a more sustained increase in CP (Nienborg and Cumming, 2009). It is also possible that a postdecision, feedback signal could briefly synchronize MT activity, but it is unclear what purpose this would serve.

Figure 4C shows the mean CP for each animal, calculated in a sliding 80 ms window for the period 0–500 ms before a saccade was initiated on false detection trials. The mean CP was significantly greater than chance for a contiguous set of window end times from 180 to 100 ms (monkey P) and 185–140 ms (monkey N) before the saccade. Here we only show data for neurons with NP > 0.75, with similar but weaker effects observed if we included all neurons regardless of their neurometric performance (supplemental Fig. 4, available at www.jneurosci.org as supplemental material). Since single sensory neurons are unlikely to be driving decisions, this result suggests that false detections may be driven by transient changes in spike rate that are synchronized across neurons. Note that we cannot address whether the brief period of elevated CP on false detection trials reflects the influence of a synchronizing signal from elsewhere in the brain or if false detections simply occur when a small population of neurons randomly change their firing rate at the same time.

Brief elevations in CP were seen across all examined window widths, with mean CPs significantly greater than chance predominantly for window end times <200 ms before saccade initiation (Fig. 4D). The common element in almost all time windows in which significant CPs were observed is that they included the period ∼180–200 ms before the saccade. This is true even when spikes were counted in windows of only 10–20 ms width. Across all window widths, the highest mean value of CP was 0.548 in an 80 ms window ending 150 ms before the saccade. In 20 and 40 ms windows, the mean CP peaked at 0.533 and 0.545, respectively. The observation that CP in the 20–40 ms windows is almost as large as in the optimal 80 ms window provides further support for the hypothesis that a brief period of synchronized activity drives perception.

CP measurements in windows dominated by spikes ∼50 ms before the saccade return to chance performance, or even fall significantly <0.5 (Fig. 4D, dark blue region). Clearly, sensory spikes in this period cannot contribute to the animal's decision, because it takes time to integrate the sensory information and program the eye movement. However, if CP reflects a sustained feedback signal then we may expect it to remain elevated until the time of the saccade. Choice probabilities <0.5 imply that the neuron's instantaneous firing rate is opposite to that expected given the impending choice saccade, suggesting that CP is not simply reflecting a feedback signal. Rather, the drop in CP before the saccade may be associated with the transient nature of motion responses in area MT and MST and a neural decoding strategy that detects the biphasic response profile associated with true speed changes (Fig. 2).

Combining the NP and CP analysis, MT/MST activity is informative as early as 60 ms after the speed change and is correlated with decisions ∼200 ms before the saccade. This would predict that the shortest reaction times during the task should be ∼260 ms, matching those observed for the largest speed changes (Fig. 1). Longer reaction times would then be associated with the read-out of neurons with longer integration windows. Unlike the optimum window width for NP, the adaptation properties of the neurons (TSI) and intertrial variability (FF) were not significantly related to either the CP values or the width of the optimum CP window (no significant correlation).

One potential concern with CP analysis is that it could be corrupted if the animal perceives the retinal slip introduced by microsaccades as changes in the stimulus speed. Two lines of evidence argue against this possibility. First, in both animals, the microsaccade rate was significantly higher when changes were missed versus correctly detected (supplemental Fig. 5, available at www.jneurosci.org as supplemental material), suggesting that motion perception and/or behavioral responses are suppressed around the time of a saccade (Herrington et al., 2009; Rolfs, 2009). Second, the microsaccade rate on correct and false detection trials was similar, dropping to <0.1 saccade/s in the 200–400 ms before choice saccades. This does not imply that animals anticipated the speed change or their impending saccade and reduced the variability in eye position. Rather, on trials in which microsaccades occurred, the change was more likely to be missed.

Correlations between spiking rate and reaction time

The significant choice probabilities measured on false detection trials demonstrate that MT/MST responses are correlated with qualitative reports of the sign of speed changes. Here we address a more quantitative aspect of decisions, by comparing reaction times (RT) and spiking rate (R) on correct discrimination trials. Averaged across trials, larger changes in speed were associated with shorter median reaction times and larger changes in mean firing rate from the reference response (Fig. 5A). This suggests that reaction time and spiking rate may also be correlated across trials with the same speed ratio. This is clearly demonstrated in the responses of an example neuron, in which the sign of the correlation depends on the speed ratio (Fig. 5A: r_0.5 = 0.46, p < 0.01; r_1.5 = −0.57, p < 0.01). It is clear that the response distribution is compressed for speed decreases, because on many trials the firing rate drops to zero. In the subsequent population analysis we avoid this floor effect by only assessing trials with speed ratio 1.5 from neurons with the reference speed lying on the positive-slope flank of a neuron's tuning curve. Thus, we predominantly expect firing rates greater than zero spikes/s.

Initially, we examined three spiking rates: R_Ref—the rate 0–300 ms before the speed change; R_Test—the rate 60–180 ms after the change; R_Diff—the difference between these test and reference firing rates. Correlations for the difference firing rate—r(R_Diff,RT)—are shown separately for each reference speed and neurons from each animal (Fig. 5B). The correlations measured with each reference speed were themselves positively correlated (r = 0.29, p < 0.01) and mean correlations were significantly less than zero (t test, p < 0.05) for monkey N (−0.113 for reference 8°/s; −0.110 for 12°/s) and one condition for monkey P (−0.082 for reference 8°/s; 0.004 for 12°/s). The weaker effects in monkey P may reflect the fact that fewer neurons were recorded in this animal and fewer trials per condition (mean number of correct discrimination trials/condition: 38.7, monkey N; 24.3, monkey P).

Averaged across animals and reference speeds, mean correlation coefficients for the three firing rate measures were as follows: r_Ref = 0.017 (p = 0.18); r_Test = −0.101 (p < 0.01); r_Diff = −0.101 (p < 0.01). The finding that the mean r_Ref was not significantly different from zero is important because it suggests that the significant correlations associated with the test spiking rate (r_Test and r_Diff) were not simply produced by global changes in activity attributable to trial-to-trial differences in attention or alertness. Due to the similarity of r_Test and r_Diff distributions, for simplicity we will only examine firing rates in a single time window, rather than examining difference firing rates.

As in the previous analyses of NP and CP, we used a range of spike counting window widths and end times to examine the timescales of correlations between spiking rate and reaction times, and focus on neurons with NP > 0.75. The mean value of r(R_Test,RT), calculated in a sliding window of width 120 ms is shown for each animal in Figure 6A. The mean correlation became significantly negative for spikes counted in the period after the speed change (p < 0.05 from 70 to 280 ms, monkey N; 130–230 ms, monkey P). The full distribution of mean r(R_Test,RT) is shown for all window widths and end times in Figure 6B (monkey N) and 6C (monkey P).

When measured in short time windows (10–80 ms), reaction time and spiking rate decorrelate ∼200 ms after the speed change, even though the majority of reaction times were longer than 250 ms. This suggests that the sustained sensory activity evoked >200 ms following the speed change does not drive the animal's behavioral responses. This is expected, because information about the stimulus, quantified by neurometric performance, also returns to chance levels with a similar time course. This demonstrates that counting spikes in short time windows can be extremely informative about both a visual stimulus and the nature and timing of a behavioral outcome based on that stimulus.

Discussion

We examined the behaviorally relevant timescales of sensory processing in MT and MST using a task that required the detection and discrimination of speed changes. Single-neuron activity during the task was analyzed using three metrics that quantify the relationship between neuronal spiking and the stimulus change (neurometric performance), the animal's decision (choice probability) and reaction times (spike rate–reaction time correlations). A key feature of our experimental design and subsequent analysis is that it allowed us to examine not just if but when a neuron's firing rate is informative by varying the duration and time of spike counting windows relative to the behaviorally relevant speed change or the initiation of choice saccades. For example, while choice probability measurements were strongest using spike counts in an 80 ms window ending 150 ms before a choice saccade began, the activity of single neurons in time windows as short as 10 ms was reliably informative about the stimulus change, the animal's decisions and reaction times. Importantly, choice probabilities were only transiently elevated above chance levels well before the execution of decision saccades, indicating that brief periods of sensory activity may drive behavioral judgments.

Why are these timescales important? Intuitively, it is trivial that there must be sensory- and decision-related information present in the brain on short timescales—driving a car or catching a ball require constant motion analysis and responding to stimuli lasting only tens of milliseconds. Further, sensory neuronal responses adapt, meaning that the earliest portion of the response to any new stimulus produces the largest change in spiking rate. Despite this, many previous studies have found reaction times of many hundreds of milliseconds in tasks using stimuli near behavioral threshold, supporting the idea that visual sensory information is integrated for relatively long periods of time (Cook and Maunsell, 2002; Roitman and Shadlen, 2002; Thiele and Hoffmann, 2008; Cohen and Newsome, 2009).

While the only strategy to filter out stimulus noise is a longer viewing time, noise inherent in neuronal encoding presents different challenges since it can be overcome by averaging over time, averaging across neurons, or by selectively monitoring the most informative neurons (Ludwig et al., 2005; Ghose and Harrison, 2009). Recent work has suggested that animals rely primarily on the initial 250–400 ms of stimulus presentation, either integrating stimulus information to a decision threshold and disregarding subsequent stimulus information or placing a stronger weight on early relative to late information (Luna et al., 2005; Kiani et al., 2008). However, in many situations, even these integration times may be too long to be behaviorally relevant. While our results are compatible with an integrate-to-bound solution to stimulus detection or discrimination (Gold and Shadlen, 2007), the different timescales suggest significant differences in implementation to account for the differences in task and stimuli. For example, to avoid spurious responses due to the fluctuations in spike count occurring on short timescales, the integrator may need to be leaky so that only strong, short-duration signals are integrated past the boundary. Alternatively, the integrator may use the spike-rate derivative as input rather than spike-rate, which would help detect stimulus changes. We are currently exploring the suitability of specific decoding algorithms in this context.

A recent study in which detection of a brief stimulus change was limited by noise in neuronal encoding demonstrated that the activity of MT neurons over tens of milliseconds was strongly correlated with both sensory events and animals' judgments (Ghose and Harrison, 2009). From these results, it was proposed that sensory activity in MT may predominantly inform behavioral decisions on timescales of tens of milliseconds. However, the stimulus used in that study was only behaviorally relevant for a short duration (60–83 ms). For tasks with brief stimulus presentations, sensory integration times longer than the stimulus duration are likely to degrade behavioral judgments, because they include neuronal responses unrelated to the stimulus to be judged. This makes such tasks poor tests of the timing trade-offs that we wished to study.

Our task design circumvented this problem because, while the timing of changes in the visual stimulus was unpredictable, the speed after the change remained constant for up to 750 ms. Thus a sensible strategy may have been to compare the spiking rates of speed-tuned neurons before and after the speed change in time windows several hundreds of milliseconds long. An alternative strategy is to detect the transient change in firing rate associated with the speed change, but reliably detecting changes in firing rate may still require comparing spiking rates in long time windows.

Despite a task design potentially favoring neuronal integration over long periods of time, we found that in the subpopulation of neurons with high NP, the mean CP was highest in an 80 ms window (0.55), with CP values significantly above chance in time windows as short as 10 ms. This is larger than CPs based on “stimulus-independent decisions” measured in a similar task, but using a 200 ms window (Thiele and Hoffmann, 2008) and also comparable to CPs measured in tasks with stimulus parameters optimized for the recorded neurons and spikes counted in much longer time windows (Britten et al., 1996; Uka and DeAngelis, 2004; Liu and Newsome, 2005; Cohen and Newsome, 2009). The magnitude of the CP we observed is surprising since at the timescale of 10–80 ms, individual neurons cannot produce many action potentials. Further, CP is normally predicted to increase with the duration of the integration window, since this increases mean spike count. Our failure to observe this scaling of CP with window duration suggests that spike rates are not stationary over the timescales we observed. Rather, weakly synchronous spikes across a small population of neurons may be necessary and sufficient to produce a percept (Huber et al., 2008).

We hypothesize that the CP measured in our task reflects variability in feedforward sensory inputs. The brief elevation of CP before false detections is consistent with decisions being based only on transient activity, even though monitoring a longer, sustained period of activity could improve stimulus characterization. This does not remove the possibility that a feedback signal—related to attention or decision—could also influence CP (Nienborg and Cumming, 2009) but it is unlikely to be the primary source of the CP that we measured. An explicit prediction of such a feedback signal would be that the elevated CP should be sustained, perhaps persisting up until the time of the decision saccade. Rather, we saw that CP returned to, or dropped below, chance performance ∼50 ms before saccades. This has been observed in previous studies (Thiele and Hoffmann, 2008; Cohen and Newsome, 2009) and probably reflects either that the processing time in which new sensory information cannot affect an impending, programmed saccade (Findlay and Harris, 1984) or that animals had learned that detecting the transient, biphasic changes in neuronal activity associated with speed changes was a good strategy for solving the task.

A trivial explanation for the high values of NP, CP, and r(R,RT) observed in short time windows is that the task was too easy, so long viewing times were unnecessary. This is unlikely to be the case, because the miss rate and reaction times increased as speed changes became more subtle. False detections were also common even though they decreased the reward rate, indicating that the animals were frequently working near detection threshold. It is more likely that the brevity of the time windows that maximized the three metrics stems from response adaptation: following a speed change, a neuron's firing rate tended to increase or decrease rapidly, and then return toward the firing rate before the change (Fig. 2A). This is borne out by the negative correlations between the TSI, which quantifies response adaptation, and the optimal window length for maximizing NP as well the actual NP value. Adaptation has been observed previously in MT (Petersen et al., 1985; Lisberger and Movshon, 1999; Priebe et al., 2002) and is associated with a saturation of Shannon information measured for a sustained stimulus after 250–500 ms (Buracas et al., 1998). It is a widespread feature of sensory processing thought to improve stimulus encoding by allowing neurons to adjust their limited dynamic range to better match the statistics of the environment (Wark et al., 2009). Thus, while beneficial for encoding dynamic stimuli, adaptation appears to produce a “law of diminishing returns” when it comes to integration of spikes produced by a sustained stimulus.

It has been common to relate stimulus properties and decisions to periods of neuronal activity measured over hundreds of milliseconds. Our results suggest that for some tasks, activity on shorter timescales may be more behaviorally relevant. Due to the variability of spike count measurements in short time windows, the activity of single neurons is unlikely to significantly influence perception. Rather, detecting synchronized activity across neuronal populations is likely to be critical for perception. This is especially the case in tasks in which noise intrinsic to the neuronal responses limits behavioral performance and can be overcome by averaging across neurons (Ghose and Harrison, 2009), as opposed to stimulus noise, which must be overcome by averaging across time (Britten et al., 1996; Cook and Maunsell, 2002; Cohen and Newsome, 2009). Here we have demonstrated that the spike rates of single neurons are informative about stimulus and behavior on timescales of just tens of milliseconds. It remains to be seen if decision making processes downstream from MT/MST can make use of such short periods of activity, or if the short timescales we observed are simply the smallest in which it was possible to measure significant correlations with behavior.