Decision-related activity in sensory neurons reflects more than a neuron’s causal effect

Abstract

During perceptual decisions, the activity of sensory neurons correlates with a subject’s percept, even when the physical stimulus is identical^1–9. The origin of this correlation is unknown. Current theory proposes a causal effect of noise in sensory neurons on perceptual decisions^10–12, but it could result from different brain-states associated with the perceptual choice¹³ (top-down). These two schemes have very different implications for the role played by sensory neurons in forming decisions¹⁴. Here, we used white-noise analysis¹⁵ to measure tuning-functions of V2 neurons associated with choice and simultaneously measure how the variation in the stimulus affects subjects’ (two macaques) perceptual decisions^16–18. In causal models stronger effects of the stimulus upon decisions, mediated by sensory neurons, are associated with stronger choice-related activity. However, we find that over the timecourse of the trial, these measures change in different directions—at odds with causal models. An analysis of effect of reward size supports the same conclusion. Finally, choice was associated with changes in neuronal gain that are incompatible with causal models. All three results are readily explained if choice is associated with changes in neuronal gain caused by top-down phenomena that closely resemble attention¹⁹. We conclude that top-down processes contribute to choice-related activity. Thus even forming simple sensory decisions involves complex interactions between cognitive processes and sensory neurons.

Considerable progress has been made towards explaining the neuronal mechanisms underlying decision making¹²-a major goal in systems neuroscience. For simple perceptual decisions, recent theory proposes that sensorimotor areas accumulate sensory evidence about the physical world, delivered by sensory neurons^{10,11,20–22}. Noise in the sensory neurons causes variability in the behavioral response^10–12, resulting in a co-variation between the neuronal activity and behavior^1–9. (Note that this causal effect of noise in the sensory representation has only been invoked for sensory areas, not for sensorimotor areas.) However, this co-variation could also arise from top-down effects¹³ in which brain states²³ that are associated with one behavioral response, also alter the response of the sensory neurons. A third (bottom-up) possibility is that sensory neurons that themselves have no causal effect on the decision are correlated with sensory neurons that do have a causal effect. These schemes have markedly different implications for the role played by sensory neurons in forming decisions. Sensory neurons either only encode the physical stimulus, or they simultaneously form an integral part of the mechanism used by the brain to decode the sensory information. In order to distinguish these views, we combined the measurement of choice-related activity in disparity selective V2 neurons in a disparity discrimination task, with a stimulus that permitted the use of white-noise analysis. This allowed the simultaneous application of 1) “subspace mapping”¹⁵, to describe how disparity affects the neuronal response (“disparity subspace map”), and 2) “psychophysical reverse correlation”^16–18 to extract a kernel describing how disparity affects the subjects’ (two macaques) perceptual choices. This comprehensive dataset enables us to differentiate these schemes.

Two macaque monkeys performed a coarse disparity discrimination task (Fig. 1a), while we recorded extracellularly from disparity selective neurons in their V2. The stimulus, a circular random dot stereogram, contained a spatially uniform binocular disparity which varied randomly on each video-frame. We exploited this random variation to perform psychophysical reverse correlation^16–18, and to measure simultaneously neuronal subspace maps¹⁵ for disparity.

FIGURE 1.

Methods. a Schematic of the sequence of events during one trial. b Example time series of the stimulus. c Probability mass distributions of the stimuli for one experiment (probability as a function of disparity), with signal disparities: −0.3° and 0.15°. Each panel depicts one signal condition (negative percentages indicate near signal disparities). d The monkey’s performance for this experiment(in percent near choices as a function of % added signal).

First, we examined how the monkeys weight the disparity signal in the stimulus to form their decision¹⁶. We calculated the difference between the average stimulus preceding the monkeys’ near choices and the average stimulus preceding the monkeys’ far choices. This “psychophysical kernel” measures the relative probability with which the disparity on any given frame occurred preceding the monkeys’ near choice. The amplitude of the kernel declines substantially over the course of the trial (Fig. 2b). (The supplementary material discusses the shape of the psychophysical kernel and shows that this linear analysis adequately captures the monkeys’ behavior.) This means that the monkeys rely predominantly on the stimulus disparities in the beginning and progressively less towards the end of the trial. Now consider neurons representing this sensory evidence. Their activity early in the trial should have a stronger effect on the decision than activity late in the trial. Thus, if the choice-related activity reflected only the causal effect of the neuronal firing on the choice, the size of the choice-related activity should also decrease over time. This prediction follows directly from the fact that, in the causal explanation, choice-related activity is the effect of noise in the sensory evidence that is used to make a decision.

FIGURE 2.

Psychophysical kernel and choice-related signal have different time-courses. a Psychophysical kernel (averaged over 76 experiments; n=17200 trials; two monkeys) as a function of disparity and time. Color represents amplitude (in occurrences/frame). b Normalized amplitude of the psychophysical kernels decreases over time. c Averaged choice-related signal over time. b,c Shaded gray areas: ± 1 standard error. d The correlation coefficient (R) over time between CP (for individual neurons) and the amplitude of the mean psychophysical kernel against a neuron’s mean CP. Color-code: temporal integration time (supplementary methods); bold edges: significant R (p<0.05, by resampling); circles, squares data from monkey 1 and 2, respectively.

To evaluate this prediction, we quantified the choice-related signal as ”choice-probability” (CP)³. (CPs were corrected for the stimulus-induced component; see supplementary Material.) The time course of the choice-related signal in our data (Fig. 2c) is quite different from that predicted from the time-course of the psychophysical data in the causal-only scheme. CPs were measured for 76 neurons which had simultaneously been recorded while the data for the psychophysical kernel were gathered. For 57/76 neurons, for which CP was >0.5, we examined the mean CP as a function of time (Fig. 2c). Consistent with previous findings⁴, CP plateaus after about 500ms -quite different from the statistically significant decrease in amplitude of the psychophysical kernel over time (r=−0.81, p<10⁻²³ between amplitude and time, over the second half of the trials). Although CP timecourses for individual neurons are noisy, we addressed the possibility that some neurons behave as if they play a causal role, by computing the correlation coefficient between the timecourse of CP for each individual neuron with the timecourse of the average psychophysical kernel amplitude (R, Fig. 2d). We find a significant negative correlation between these Rs and a neuron’s CP (r=−0.28, p<0.05) indicating that neurons with high CPs tended to show a negative correlation with the timecourse of the psychophysical kernel amplitude, as expected from the average data (Fig. 2b). This and other features of individual timecourses (see supplementatry discussion) are at odds with the causal model.

These results are incompatible with the causal-only account. It suggests that CPs are at least partially of non-causal, possibly top-down, origin. We therefore sought a signature of possible top-down mechanisms at the level of individual neurons. This could employ a mechanism similar to attention which characteristically alters the gain of sensory neurons¹⁹. We designed our disparity-varying stimulus such that it permitted the measurement of subspace maps for disparity (see Methods), in order to test this possibility explicitly.

These subspace maps quantify the effect of each disparity (in the stimulus with 0% added signal) on the neuron. Calculating subspace maps separately for stimuli associated with “near” and “far” choices quantifies any effects of choice upon the neuronal response. Choice-related activity itself implies some difference between these subspace maps. If the difference is caused by a change in neuronal gain, the two subspace maps should be scaled versions of each other. Example subspace maps for one neuron (Fig. 3a) show that the gain of this neuron increased by 84%, while the additive change was close to 0 (−0.032 spikes/frame). A second example shows a more typical gain increase (18%, y-offset: 0.005 spikes/frame).

FIGURE 3.

Choice-dependent gain change. a Sub-space maps for preferred (red), null (blue) choices superimposed (neuron d1456). Dashed lines: 0° disparity, 0 spikes/frame. b Null-choice responses plotted against preferred-choice responses, yielding relative gain (slope, 1.84), additive change (y-offset, −0.032 spikes/frame). Dashed lines: unity, 0 spikes/frame. c,d Same format as a,b for neuron d1394 whose slope (1.18), y-offset (0.005 spikes/frame) resemble the population-mean. e Slope and choice-probability are correlated. Filled, open symbols: cells with, without significant choice-probability. Circles, squares: data for monkey 1, 2. Dashed lines: 0.5 choice-probability, relative gain of 1. f No correlation between y-offset and choice-probability (symbols as e). Dashed lines: 0.5 choice-probability, 0 spikes/frame. Solid lines in e,f: median standard error for slope, y-offset.

The distribution of relative gain change as a function of CP demonstrates that CPs are associated with choice-related changes in neuronal gain (Fig. 3e, n=76, r=0.44, p<10⁻⁴; monkey 1 n=42, r=0.54, p<0.001; monkey 2 n=34, r=0.32, p<0.07). The geometric mean of the relative gains was 1.16 (1.17 and 1.15 for monkey 1 and 2, respectively), which is significantly different from 1 (p<0.001, by resampling). Conversely, there was no systematic relationship between the y-offset and CP (r=0.03, p=0.77; r=−0.18, p=0.25 and r=0.18, p=0.31 for monkey 1 and 2, respectively; mean offset: −0.03 spikes/frame). Thus, it is the choice-dependent change in response gain which explains the difference in mean response rate between preferred and null choices.

A modest gain change could arise, even in the causal account of CP, from the firing properties of cortical neurons (e.g. Poisson spiking²⁴). A shuffling technique showed that this effect was too small to account for the observed gain changes (supplements).

The gain change suggests the operation of a mechanism similar to feature selective attention¹⁹, but which varies from trial-to-trial. This could arise in several ways. First, as the decision is formed, a signal altering the neuronal gain may be sent back to those neurons supporting this decision. Alternatively, this gain change may implement a perceptual working memory²⁵, or a perceptual bias/expectation: attending “near” increases the response gain of “near” neurons and thus makes a “near” response more likely.

An additional feature of our data provides evidence that at least the latter mechanism operates. The reward size depended systematically on the animals’ performance (see Methods). This performance was better on trials for which a large reward was available (Fig. 4a and supplements), indicating that animals used more information about the stimulus when reward size was large. It allows us to further test causal explanations for CP: when the animal uses more stimulus-derived information CP should be larger. Contrary to this prediction, we find that CPs were significantly smaller for trials on which a large reward was available (p<0.006, paired t-test, Figure 4b). This result can be explained if one assumes that the animal has some bias (or expectation) at the start of each trial (regardless of reward size), and this bias engages our proposed top-down mechanism. When the available reward is small and the monkeys make less use of the sensory input (as demonstrated by the psychophysical kernel, Fig. 4a), the bias will have a stronger impact on the behavioral response. Conversely, when a large reward is available, the improved performance implies that any initial bias is more likely to be overridden by the evidence provided by the visual stimulus. Hence any component of CP reflecting a top-down effect of bias will be smaller on large-reward trials when the decision is more strongly driven by the actual stimulus and less by the monkey’s initial bias.

FIGURE 4.

Reward size affects behavior and choice-related signal. a Psychophysical kernel as a function of disparity and available reward (in occurrences/1000ms; n=6886 trials for large reward, red line; n=10314 trials for small reward, blue line; averaged over the first and second 1000ms of each trial in the left and right panel, respectively.). Improved performance is mainly caused by a larger psychophysical kernel in the first (kernel difference p<0.001, by resampling), not second half of the trials (difference n.s.) b Choice-probability computed for the first half of the trials was larger when a smaller reward was available (p<0.006, n=76). Dashed line: unity.

Our results provide three lines of evidence against the view that decision-related activity in sensory reflects only the causal effect of neuronal noise on sensory decisions. First, the time-course of the decision-related signal was incompatible with that predicted from the behavioral data in the causal-only scheme. Second, larger rewards systematically improved the animals’ behavior, but reduced CP, the opposite of the expectation from causal explanations. Finally, CPs were associated with choice-dependent gain changes larger than could be explained in the causal scheme. All three phenomena follow naturally from a top-down scheme in which the animals’ perceptual state alters the response of sensory neurons. An alternative explanation is that neurons which do not contribute to the decision show CP because they are correlated with neurons that do contribute. Such a scheme, if sufficiently rich, might explain the data without invoking a top-down mechanism (see supplementary Discussion), but nonetheless abandons the principle that CP reflect only the causal effect of sensory noise upon decisions. Given that the choice-dependent gain changes we observe are characteristic of top-down mechanisms such as attention, our top-down scheme is the most parsimonious.

Changes in neuronal gain may facilitate the decoding of neuronal populations by appropriately weighting relevant neurons^26–28. Implementing such a decoding mechanism at the level of sensory neurons allows the brain extraordinary flexibility to perform sensory decisions in different circumstances. Here we show that these gains vary with a subject’s choice, within a fixed task. This gain change could implement a perceptual bias or expectation (attending to “near” or “far”), and could also follow the formation of a decision. It may serve to promote perceptual stability in the presence of ambiguous²⁹ or noisy sensory signals. In either case, our data suggest that even simple sensory decisions involve top-down mechanisms that entwine cognitive processes and the sensory representation in previously unreported ways.

Methods summary

All procedures were in agreement with the Public Health Service policy on the humane care and use of laboratory animals and all protocols were approved by the Institute Animal Care and Use Committee. We recorded extracellular activity from disparity selective single V2 units while two monkeys (Macaca mulatta) performed disparity discrimination. Upon fixation, a stimulus was presented for 2sec, followed by two choice targets. After a saccade to the correct target, the monkeys received a liquid reward. Stimuli were dynamic random dot patterns: a disparity-varying center (disparity changed randomly on each frame, 96Hz framerate) and a surrounding annulus at 0°. The center disparity was chosen from a set of evenly spaced disparity values centered around 0° (encompassing the preferred and null disparity of each neuron). We introduced a detectable signal by increasing the probability of occurrence for one disparity within some trials. These signal disparities approximately matched each neuron’s preferred and null disparity. Signal trials served only to control behavior: all analyses were restricted to trials with 0% signal added. Psychophysical kernels were computed as the difference of the mean stimulus matrix preceding near- and far choices, respectively. The average kernel was a weighted average of the kernel for each experiment for which neuronal data were included. Choice-probabilities were obtained as described previously^3,6, but corrected for the stimulus-induced component (Supplements). For the sub-space analysis, the average response of each neuron following one frame of a given disparity (d_i) was computed as a spike density function (S_i(t)). We calculated the total number of spikes elicited by one frame of d_i as the sum of the overall mean number of spikes/frame and the integral of the deviation of S_i(t) around this mean. Separate analyses for all trials preceding a near choice (far choice) yield the subspace maps separated by choice.

Methods

Task and reward-regimen

Two monkeys were trained in a binary forced choice disparity-discrimination task (Fig. 1a). They judged whether the central stimulus-region appeared in front or behind the surrounding annulus. Trials started upon fixation (within 0.5° of a fixation marker), initiating a 2sec stimulus presentation followed by the appearance of two choice targets (3° above and below the fixation marker). If the monkeys indicated their decision by a saccade to the correct choice target within 500ms of the stimulus disappearance, they received liquid rewards. If the monkeys made correct choices on three consecutive trials, the reward on the fourth and on all subsequent correct trials was approximately three times its normal size, until the monkey made an error. After an error, the reward size was set back to its normal size.

Recordings

We recorded extracellular activity from disparity selective single-units in these monkeys’ V2, as described previously^6,30. Both animals were implanted with scleral search coils in both eyes³¹, head fixation posts and a recording chamber under general anesthesia. Positions of both eyes (for 17/58 neurons for monkey 2 signals were available only for one eye) were measured with a magnetic scleral search system (C-N-C Engineering) and digitized at 800Hz. The monkeys viewed the stimuli on EIZO Flexscan F980 monitors in a Wheatstone stereoscope configuration (89cm viewing distance). All procedures were in agreement with the Public Health Service policy on the humane care and use of laboratory animals and all protocols were approved by the Institute Animal Care and Use Committee.

Stimulus

All stimuli were circular dynamic random dot stereograms (RDS; 50% black and 50% white dots of 99% contrast, dot-density generally 40%, dot size 0.09×0.09°). Each RDS had a disparity-varying center (3–5° in diameter) and a 1–2° wide surrounding annulus at 0° disparity (Fig. 1b). On each frame, all center dots had the same disparity, but this disparity value changed randomly from frame to frame (96Hz frame-rate). For the condition with 0% added signal, the disparity on each frame was drawn at random from a uniform distribution of discrete, equally spaced disparities (symmetrical about 0° disparity, center Panel in Fig. 1c; encompassing the preferred and the null disparity of each neuron). Signal disparities (always one near, one far disparity) approximately matched the preferred and null disparity of the neuron. Disparity signal was introduced by increasing the probability of the signal disparity on each frame (Fig. 1c).

Psychophysical reverse correlation

Only 0% added signal trials were included in the analysis. Each stimulus trial was summarized by a two-dimensional matrix in which each row corresponds to one disparity, and each column to one stimulus frame. For each column in this matrix, there is one entry with a 1, corresponding to the disparity at this frame, and all other entries are 0. We then computed the average matrix preceding the monkey’s near and far choice. For each of the 200 stimulus frames, the resulting values correspond to the probability with which each disparity preceded a near choice or far choice, respectively. This yields a two-dimensional (time × disparity) probability-distribution. The difference between the probability distribution preceding near choices and far choices defined the psychophysical kernel for each experiment. (Negative disparities are defined as near.) The kernel-shapes change little between monkeys or as a function of the signal disparities (supplementary Fig. 2). We therefore collapsed all the data into a single psychophysical kernel to maximize the temporal resolution. The average psychophysical kernel (Fig. 2a) was obtained for all experiments for which the simultaneously recorded neurons passed the inclusion criteria. Since the disparity range was adjusted for each neuron, the psychophysical kernel for each experiment was weighted by the number of disparity values included in this experiment (this ranged between 5–13 disparity values) and by the number of trials for this experiment. Only data for disparity values [−0.4°, −0.3°…, 0.4°] were included in this average. As an estimate of the amplitude of the psychophysical kernel we computed the inner product of the time-averaged psychophysical kernel with the psychophysical kernel (temporally smoothed, 10ms boxcar) at each 10ms bin, and normalized this inner product by its overall mean. Confidence intervals for all measures were derived by resampling. All analyses were based on the linear kernel of the psychophysical data. Consistency-analyses (see supplements) show that this linear kernel provides an excellent description of the monkeys’ behavior. Further analyses indicate that second-order interactions were negligible (see supplements).

Sub-space analysis

The analysis is based on all 0% added signal trials. First, the average response of each neuron following one frame of a given disparity (d_i) was computed as a spike density function (S_i(t)), smoothed by a 10ms-wide boxcar (colored solid lines in supplementary Fig. 3a). As an estimate of the impact of one frame of this disparity (d_i) on the firing rate of the neuron, we calculated the total number of spikes (s_i) elicited by one frame of d_i. This metric corresponds to the sum of the mean number of spikes/frame (μ, black line in supplementary Fig. 3a) and the integral of the deviation of S_i(t) around this mean.

$$s_i=\mu +\int (S_i(t)-\mu )dt$$

The disparity sub-space map (s_i) is plotted as a function of d_i (supplementary Fig. 3b). Performing this analysis separately for all trials preceding a null choice (preferred choice) yields the subspace maps separated by choice (Fig. 3a,c). To quantify the choice-dependent modulation in tuning, we plotted the responses on the null choice trials against those on the preferred choice trials (Fig. 3b,d), and estimated (type II regression) the slope (gain-change) and the y-offset (additive change). Note that because the spike density function is a mean rate calculated separately for each choice, variations in the disparity content of the stimulus that are associated with choice will not produce differences in the subspace maps.

Analysis of choice-probabilities

Choice probabilities were obtained for all 0%-added signal trials as described previously ^3,6. As the psychophysical kernel demonstrates, there are systematic differences in the stimuli preceding the monkeys’ choices. CPs were corrected for this stimulus-induced component (see supplementary methods).