A modality-specific feedforward component of choice-related activity in MT

Summary

The activity of individual sensory neurons can be predictive of an animal's choices. These decision signals arise from network properties dependent on feedforward and feedback inputs; however, the relative contributions of these inputs are poorly understood. We determined the role of feedforward pathways to decision signals in MT by recording neuronal activity while monkeys performed motion and depth tasks. During each session, we reversibly inactivated V2 and V3, which provide feedforward input to MT that conveys more information about depth than motion. We thus monitored the choice-related activity of the same neuron both before and during V2/V3 inactivation. During inactivation, MT neurons became less predictive of decisions for the depth task but not the motion task, indicating that a feedforward pathway that gives rise to tuning preferences also contributes to decision signals. We show that our data are consistent with V2/V3 input conferring structured noise correlations onto the MT population.

Introduction

How sensory information is used to guide decisions is a longstanding question in cognitive and systems neuroscience. The well mapped visual response properties of the middle temporal visual area (MT) in the macaque monkey (reviewed in Born and Bradley, 2005) have provided a fertile test bed for linking sensory signals to perceptual decisions (reviewed in Parker and Newsome, 1998). Such a linkage has now been firmly established between MT neurons and tasks involving visual cues for motion and depth using a variety of approaches, ranging from lesions/inactivation (Chowdhury and DeAngelis, 2008; Newsome and Pare, 1988) to microstimulation (DeAngelis et al., 1998; Krug et al., 2013; Salzman et al., 1990) to measuring correlations between the activity of single neurons and both sensory stimuli (Britten et al., 1992; Uka and DeAngelis, 2003) and behavior (Britten et al., 1996; Dodd et al., 2001; Parker et al., 2002; Uka and DeAngelis, 2004). The presence of this latter type of correlation means that an animal's choices during a perceptual task can be predicted, albeit imperfectly, by measuring the activity of single MT neurons, a relationship referred to as either "choice probability" (CP) or "detect probability" (DP), depending on the nature of the task. These signals, subsequently shown to be present in a number of brain areas during a variety of perceptual tasks (see Haefner et al., 2013 and Nienborg et al., 2012 for discussion), have figured prominently in models of sensory decision making (Haefner et al., 2013; Shadlen et al., 1996).

More recently, neurophysiologists have sought to address the question of how and where these decision-related signals arise. Early studies focused on bottom-up sources, such as shared sensory inputs (Shadlen et al. 1996); however, more recent experiments have made it clear that top-down factors, such as attention, also play an important role (Cohen and Newsome, 2009; Dodd et al., 2001; Nienborg and Cumming, 2009, 2010). A top-down contribution has been observed as early in the visual hierarchy as V2 (Nienborg and Cumming, 2009).

On the other hand, many of MT's most salient stimulus-related response properties appear to be directly inherited from its inputs (Movshon & Newsome 1996; Pack et al. 2006; Priebe et al. 2006). Individual MT neurons are tuned to both direction of motion and stereoscopic depth of visual stimuli, and it appears that information about these two features arrives via segregated anatomical pathways: a direct projection from V1 provides predominantly motion information (Movshon & Newsome 1996) while an indirect input through V2 and V3 provides mainly binocular disparity information (Figure 1A; Ponce et al., 2008). In the latter study, it was shown that reversibly inactivating V2 and V3 selectively impaired the tuning of MT neurons for binocular disparity, while leaving tuning for direction of motion largely intact.

Figure 1. Experimental Design.

(A) Schematic of the two major cortical inputs to MT. The cube icon indicates data related to depth and the arrows icon indicates data related to motion throughout this paper.

(B) Behavioral task design. Each panel depicts a phase of the trial. The gray region indicates the inactivation “scotoma”, the dotted circle indicates the edges of a neuron’s receptive field, and the solid circle depicts the extent of the visual stimulus.

(C) Experimental timeline. “RF” indicates receptive field mapping and “tasks” refers to the epoch in which the animal performed the motion and depth tasks.

We exploited our ability to selectively and reversibly inactivate the indirect pathways to MT in order to determine how feedforward input contributes to decision-related activity of individual MT neurons. We hypothesized that in a feedforward framework the same inputs that carry information about a task-relevant stimulus attribute will also give rise to decision-related signals in this task. Insofar as inputs from V2/V3 are important sources of depth, but not motion, signals we should see a reduction in decision-related activity in MT during a perceptual task dependent on depth but not one dependent on motion. To test our hypothesis, we trained two macaque monkeys to perform motion and depth detection tasks while we reversibly inactivated V2/V3. While animals performed the tasks, we recorded the activity of single MT neurons, which allowed us to monitor the changes in choice-related activity of the same neuron both before and during inactivation. We found that V2/V3 inactivation reduced the selectivity of MT neurons for binocular disparity—an important cue for depth—more so than the selectivity for direction of motion, as reported previously (Ponce et al., 2008). In addition, V2/V3 inactivation reduced MT neurons’ correlation with behavioral reports during the depth detection task but not during the motion detection task, indicating that this feedforward input has a significant and modality-specific contribution to choice-related activity in MT. Finally, we discuss our results in the context of a computational feedforward framework and show that they can be explained by assuming that detection decisions are based on the comparison of the activity of two neural pools.

Results

Two macaque monkeys performed reaction-time motion and depth detection tasks in which they were rewarded for detecting the onset of coherent motion or depth in a noisy random dot stimulus (Figure 1B). Signal onset was randomly timed and animals were rewarded for responding within 650 ms; otherwise, the trial was classified as a ‘miss’ and no reward was given. The two tasks were interleaved in blocks of 25 trials and the monkeys had to correctly complete two trials at the easiest signal strength at the start of each block before more difficult trials were introduced. Each monkey was experimentally naïve before these experiments began and thoroughly trained to perform both tasks before we began inactivating V2/V3.

We recorded the activity of 75 well-isolated single neurons in MT both immediately before and during inactivation of V2/V3 (34 in monkey S; 41 in monkey Q). Task stimuli were matched to the receptive fields of each neuron, which were confined to the “scotoma,” the part of the visual field previously shown to be affected by cooling (Figure 2C of Ponce et al. 2008). The experimental timeline is shown in Figure 1C. Each day, we initiated cooling after mapping the neuron’s receptive field properties and collecting neuronal and behavioral data during both tasks at physiological temperature (“pre-cool”; typically requiring one hour). We repeated these measurements during the “cool” phase, which lasted at most one hour, and when possible, again during “recovery” when the temperature returned to within ~5°C of physiological temperature.

Figure 2. Behavioral Performance.

(A) Sample behavioral performance during the depth and motion task is shown on the left and right, respectively. Psychometric thresholds are shown in the upper right corner.

(B) Comparison of pre-cool and cool psychometric thresholds during the depth (left) and motion (right) tasks, color-coded by monkey. Error bars are 95% confidence intervals on the threshold from bootstraps on the function fit, shown for every fifth data point. Note log-log axes.

(C) Psychometric thresholds for performance in the visual hemifield ipsilateral to the cryoloops, where we did not expect effects of cooling. Same conventions as in (B).

(D) Paired comparison of the Effect Indices (EI; see text) in the scotoma.

(E) Histogram of changes in EI in the ipsilateral visual field. See also Table S1, Figure S1.

Effects of V2/V3 inactivation on behavioral performance

In order for us to reliably measure choice-related activity, animals had to be actively engaged in both tasks. Although monkeys’ behavioral performance was impaired during both tasks during inactivation, several lines of evidence indicate that they continued to fully engage in and perform both tasks quite well. In the example session in Figure 2A the percent increase in behavioral thresholds was 56% and 29% during the depth and motion tasks, respectively (depth thresholds: 27% pre-cool, 42% cool; motion thresholds: 24% pre-cool, 31% cool). During this session and others, both monkeys continued to rely on stimulus information during inactivation, evidenced by the sigmoidal relationship between performance and signal strength. More than 90% of psychometric functions were well fit by a sigmoid during both tasks in all conditions (deviance cumulative probability < 0.95, Wichmann and Hill, 2001). Across sessions, behavioral performance was typically impaired somewhat during both tasks (Figure 2B, D). We summarized the effect on behavioral threshold in each task with an “effect index” (EI) of the following form:

$$EI=\frac{cool-pre}{pre}$$ (1)

where pre and cool refer to the behavioral threshold before and during cooling, respectively. One of the two monkeys (S) was consistently more impaired during the depth task (median paired difference in EI, depth − motion = 0.29; p = 0.0005, sign test; see Table S1 for EI values). The other animal (Q) tended to be slightly more affected during the motion task but the paired difference between tasks was not significant (median paired difference in EI = −0.14; p = 0.4, sign test). We discuss the difference between animals in the context of the changes in the neuronal representation of depth and motion in MT in the next section.

Additional evidence suggests no change in motivation or degree of guessing. Fitted lapse rates never exceeded 5%, and we observed only small changes in fixation breaks and false alarms in both monkeys (Table S1). During 22 sessions we included a general control for motivation, in which, on alternating trials, we measured behavioral performance in the visual field ipsilateral to the location of the cryoloops (Figure 2C, E). Median behavioral EIs in this part of the visual field were ≤ 0.05 (Table S1), indicating that behavioral effects of cooling were restricted to the scotoma and therefore not due to a generic reduction in motivation.

Behavioral performance was stable within and across cooling sessions, suggesting no change in strategy. We did not find a significant difference in performance between the first and last third of each cooling session in either animal: the median differences in thresholds were less than 3.5% for both animals during both tasks and these differences were not significantly different from zero (sign test p-values: depth: 0.24 monkey S, 0.49 monkey Q; motion: 0.38 monkey S, 0.54 monkey Q). The behavioral effects were also stable across several months of repeated inactivation (Figure S1). As in the example in Figure 2A, performance returned to pre-cool values following recovery on all days on which it was tested. The median difference in behavioral thresholds during recovery and pre-cool period was less than 1% in both animals during both tasks (n = 12 for each animal, data not shown).

Effects of V2/V3 inactivation on the representation of depth and motion in MT

At the neuronal level, we found that V2/V3 inactivation led to larger impairments of binocular disparity processing than direction processing in MT of both animals. We computed each neuron’s neurometric performance (NP), which describes how well each neuron can perform the signal detection task and quantified the change in NP with an effect index, as described by equation (1). NP values were first converted to distances from 0.5, the chance value (e.g. pre = NP − 0.5). NP was impaired during both tasks but more so during the depth task in both monkeys (Figure 3A, B). The difference between depth and motion EI was significant in the combined data and in monkey S, but not in monkey Q (difference in medians, EI_depth − EI_motion: combined = −0.13, p = 0.03, Wilcoxon rank sum test; Monkey S = −0.16, p = 0.03; Monkey Q = −0.1, p = 0.24; see Table S2 for EI values). These effects are similar to the behavioral impairments of each animal, reinforcing the idea that MT activity is closely linked to performance on these tasks. Specifically, the larger impairment in depth task NP in monkey S is consistent with this animal’s larger behavioral impairment during the depth task and the more similar effects of cooling on monkey Q’s NP in the two tasks is consistent with the similar behavioral impairment in the two tasks, although monkey Q’s behavioral performance was affected slightly more during the motion task while NP was affected slightly more during the depth task. It is important to note, however, that both differences were small and in neither case could we reject the null hypothesis of no difference (at alpha = 0.05). We did not find a significant relationship between the magnitude of the EI and the behavioral impairment across sessions during either task in either animal (Figure S2). Although we believe that neurometric impairments in MT are largely responsible for the animals’ behavioral decrements, the change in a single neuron’s neurometric threshold probably only captures a small fraction of the variability of the global cooling effects on a given day.

Figure 3. Neuronal Effects of Inactivation.

(A) Each neuron’s neurometric performance (NP) during the depth (left) and motion (right) tasks, color-coded by monkey. Error bars are the SEM, shown for every fifth data point.

(B) Histogram of NP effect indices (EI) combined across monkeys (left) and for each monkey individually (right two panels). Depth task effects shown in black and motion task effects shown in grey. Triangles indicate medians.

(C) Tuning discrimination indices (DI) for binocular disparity (left) and direction (right) tuning. Same conventions as in (A).

(D) Paired comparison of the effect of cooling between the direction and binocular disparity tuning DI. Same marker conventions as in (A).

See also Table S2, Figure S2.

Basic tuning for binocular disparity and direction of motion followed a similar pattern, but with a more pronounced difference between the effects on binocular disparity and direction tuning, consistent with a previous report using different animals (Figure 3C,D; Ponce et al., 2008). We compared tuning with a discrimination index (DI; Ponce et al., 2008; Prince et al., 2002). Values near one indicate the neurons are strongly modulated by the stimulus while small values suggest modulations are simply due to noise. The median change in DI (cool − pre-cool) in data combined across monkeys was −0.14 for binocular disparity and −0.03 for direction tuning with a median pairwise difference (depth − motion) = −0.08, which was significantly different from zero (sign test, p = 3.7 × 10⁻⁸; see Table S2). Notably, monkey Q exhibited a larger impairment in direction tuning than monkey S (difference in median direction tuning impairment = 0.05, p = 0.0007, Wilcoxon’s rank sum test), a pattern consistent with the neurometric and behavioral impairment differences between the animals.

Choice related activity in MT is selectively reduced during the depth task

We measured the degree to which neuronal activity in MT was predictive of behavioral reports by computing the "detect probability" (DP), a metric that compares response distributions between trials on which the signal onset was detected and those on which it was missed. We computed the time course of DP from firing rates aligned to the signal onset and combined data across the population of recorded MT neurons in both animals (Figure 4A). The population DP computed in a 375 ms window following signal onset (gray region) decreased by 40% during the depth task between pre-cool and cool conditions from 0.60 to 0.56 (p = 0.001, resampling test for equal pre-cool and cool DPs). The change in motion DP was smaller and not statistically significant (from 0.58 to 0.59, p = 0.8). The DP change was significantly different between the two tasks, as determined by a separate bootstrap procedure that compared the distributions of re-sampled effect magnitudes (p = 0.003). Results were similar for each animal considered individually (Figure 4B; Table S3).

Figure 4. Detect Probability.

(A) Time course of detect probability (DP) ±SEM, aligned to signal onset at t = 0. The gray region indicates the time window used to calculate the population DP reported in the text, the individual neurons’ DP in (C) and the DP in (D). Only neurons that contributed data to both tasks and conditions (pre-cool and cool) are included in this analysis.

(B) DP time course shown separately for each monkey.

(C) DP for each of the 75 neurons, color-coded by monkey. Filled symbols indicate DPs that were statistically significantly different between pre-cool and cool conditions.

(D) Grand DP ±SEM computed in the time window indicated by the gray region in (A). Note only the 9 neurons for which we had data for all three conditions were included here.

See also Table S3, Figure S3.

We observed the same pattern when we computed DP for each neuron (Figure 4C): the median DP during the depth task changed from 0.57 to 0.55 (p = 0.03) with cooling and during the motion task from 0.58 to 0.59 (p = 0.64). On a neuron-by-neuron basis we found no relationship between the changes in DP during the motion and depth tasks, consistent with the interpretation that individual neurons’ DP changed independently for the two tasks (r = 0.07, p = 0.56). Further, we found no significant relationship between the change in DP and changes in behavior or neurometric performance during either task (ΔDP vs Δbehavior: depth r = −0.01, p = 0.92, motion r = 0.08, p = 0.51; ΔDP vs ΔNP: depth r = 0.09, p = 0.48, motion r = −0.10, p = 0.40).

The selective reduction in depth DP was reversible within single sessions. For the 9 units for which we collected recovery data, DP returned close to pre-cool values during the depth task (Figure 4D), demonstrating that the changes in DP during inactivation were reversible on a similar timescale (depth DP: pre-cool = 0.65, cool = 0.57, recovery = 0.63, p = 0.25 for difference in DP between pre-cool and recovery; motion DP pre-cool = 0.59, cool = 0.59, recovery = 0.60, p = 0.61).

V2/V3 inactivation reduces spiking variability in MT

Although MT neurons remained robustly visually responsive during cooling, visually evoked responses decreased by an average of 42% (Figure 5A), consistent with the inactivation of a major excitatory input. Further, although cortical neurons typically exhibit a stereotyped relationship between the mean and variance of the spike count across trials ("Fano factor", Geisler and Albrecht, 1997; Nawrot et al., 2008; Shadlen and Newsome, 1998; Tolhurst et al., 1981), this relationship changed dramatically in MT neurons during inactivation (Figure 5A, B). These data were similar between the two animals and were therefore combined. The raw Fano factor (FF) computed in 50-ms bins is shown aligned to stimulus onset and, separately, signal onset for each task in Figure 5A. The FF was lower during inactivation than during pre-cool throughout the trial during both tasks, including during the stimulus-driven decline in FF (Churchland et al., 2010). We computed the FF for each neuron in a 250 ms time window after signal onset (see Experimental Procedures). Unlike other cooling-induced effects, the decline in FF was not significantly different between tasks (median paired difference = 0.02, p = 0.76, Wilcoxon signed-rank test) so we pooled these data for subsequent analyses. The median FF was 1.0 before cooling and 0.67 during cooling, a reduction that was significantly different from zero in a paired comparison (p = 3.6 × 10⁻¹⁷, sign test). The decrease in FF was independent of differences in the mean spike count; it persisted when we compared the FF for spike counts ranging from 5-12 (gray region in Figure 5B), which contained approximately the same number of data points in the two conditions (pre-cool FF = 1. 0, cool FF = 0.61; p = 1.8 × 10⁻¹⁵, sign test). The implications of reduced variability in MT for decision-related activity are elaborated upon in the Discussion.

Figure 5. Trial-to-Trial Variability.

(A) Average firing rate (top) and Fano factor (FF) ± SEM (bottom) aligned separately to onset of the visual stimulus and the time of the change during the depth (left) and motion (tasks).

(B) Scatter plot of the variance and mean of the spike count collapsed across tasks and time. Each data point corresponds to a unique stimulus presented to one neuron and there are 2-14 data points per neuron.

Possible mechanisms for the changes in choice-related activity in MT

Theoretical work has shown that choice-related activity depends both on the structure of correlated variability among the neurons under study (Shadlen et al., 1996; Nienborg and Cumming, 2010; Haefner et al., 2013) and on the way neurons’ activity is read-out by decision-making areas (Haefner et al., 2013, Newsome & Parker, 1998). Several lines of evidence indicate that our results are most consistent with a change in noise correlations in MT and not a change in read-out weights. Although animals can quickly alternate between strategies—i.e. read-out weights—within a day’s session (e.g. Cohen and Newsome, 2009; Sasaki and Uka, 2009), these strategies must first be learned via a mechanism akin to reinforcement learning over the course of weeks or months of training (Law and Gold, 2008, 2009; Uka et al., 2012). We saw no evidence of the gradual improvements in performance that would accompany such learning either within or across sessions (Figure S1), even though we monitored behavioral performance of both animals from the first cooling day. We also measured DPs as early as the fifth cooling day and in 34 sessions thereafter in one of the animals (Monkey S) and found no trend or significant relationship between cool DPs during either task and session number (Figure S3, regression slope < |0.001|, p > 0.7 in both cases; see legend for values). There was no relationship between DP and session number in the other animal in which we first measured DPs in the fifteenth cooling session and in 42 sessions thereafter (regression slope < |0.001|, p > 0.1 in all cases; see Figure S3 for values). Thus we saw no evidence that behavior or DP changed gradually, as would be expected with a learning process.

Instead, it is more likely that our results can be explained by a change in the correlations among MT neurons resulting from inactivating a proportion of their inputs. Cooling can affect the correlations among MT responses in two ways: 1) by altering the input correlations that MT inherits from V1 and V2/V3, and 2) by reducing the firing rate of MT neurons. We first address why the latter cannot account for our result. De la Rocha et al. (2007) reported that a reduction in the firing rate implies a reduction in the magnitude of the output (or response) correlations even as input correlations are unchanged. The relationship presented by de la Rocha et al. allows us to estimate the overall reduction in the correlations between MT responses as a direct result of the observed reduction in firing rate to be about 7%, which translates into a reduction in DPs of roughly 4% (Supplemental Experimental Procedures; Figure S4). Thus the reduction in MT firing rate can only explain a small part of the 40% reduction in DPs that we observed in the depth task.

Excluding the possibility that the decrease in DP is due to a reduction in MT firing rate leads to the hypothesis that cooling reduces the input correlations conferred by V2/V3. While we acknowledge that cooling V2/V3 may also indirectly affect feedback to MT or V1, we will show that our data can be explained by our manipulation of the direct feedforward pathway alone.

Although the nature of the read-out of sensory neurons in detection tasks is currently unknown, our results are consistent with a model in which the subject’s decision is based on comparing the average response of a population of sensory neurons that increase their firing in response to the target stimulus (called ‘pref pool’) to the average response of neurons that either decrease their response to or are indifferent to the target stimulus (‘null pool’)—in analogy to models of discrimination tasks (Link and Heath, 1975; Shadlen et al., 1996). Such a read-out strategy has the benefit of being able to subtract out common sources of variability and thereby increase performance compared to one-pool models (e.g. Smith et al. 2011). As previously recognized for discrimination tasks (Nienborg & Cumming 2010, Haefner et al. 2013), choice-related activity in the 2-pool model is the result of a difference in the average correlation between neurons in the pref pool, and the average correlation between neurons belonging to different pools:

$$\begin{array}{c}\hfill \langle D{P}_{motion}-\frac{1}{2}\rangle {}_{motion-pref}=\frac{\sqrt{2}}{\pi }\sqrt{c_{motion-same}^{MT}-c_{motion-diff}^{MT}}\text{and}\hfill \\ \hfill \langle D{P}_{disp}-\frac{1}{2}\rangle {}_{disp-pref}=\frac{\sqrt{2}}{\pi }\sqrt{c_{disp-same}^{MT}-c_{disp-diff}^{MT}}\hfill \end{array}$$ (2)

$c_{motion-same}^{MT}$ is the average correlation across pairs of MT neurons in which both neurons belong to the same pool (pref or null) as defined by the motion task, while $c_{motion-diff}^{MT}$ is the average correlation across pairs with neurons belonging to different pools as defined by the motion task. $c_{depth-same}^{MT}$ and $c_{depth-diff}^{MT}$ are the equivalent quantities for the depth task. In the feedforward framework, the observed correlations in MT are primarily inherited from the major input areas V1 and V2. As a first approximation, we assume the inputs to MT neurons to be a linear combination of its V1 and V2 inputs: ${\rho }_i^{MT-input}={\rho }_i^{V1}+\kappa {\rho }_i^{V2}$. We assume cooling scales down the V2 input, quantified by parameter ▱ ranging from 1 (no cooling) to 0 (complete cooling). Then the correlation between the inputs to two neurons ▱ and ▱ in the MT population is given by (Experimental Procedures, assuming equal variances):

$$c_{ij}^{MT-input}=\frac{c_{ij}^{V1}+{\kappa }^2{c}_{ij}^{V2}+\kappa [c_{ij}^{V1-V2}+c_{ji}^{V1-V2}]}{\sqrt{(1+{\kappa }^2+2\kappa c_{ii}^{V1-V2})(1+{\kappa }^2+2\kappa c_{jj}^{V1-V2})}}$$ (3)

Importantly, the input correlations to MT depend in such a way on the correlations between the V1 inputs, $c_{ij}^{V1}$, the correlations between the inputs from V2, $c_{ij}^{V2}$, and the correlations between the inputs from V1 and $c_{ij}^{V1-V2}$, that they can either increase, stay unchanged, or decrease as κ decreases from 1 towards 0 during cooling depending on the specific values for these input correlations. In order to gain an intuition, we can ignore the inter-area correlations (which appear both in numerator and denominator and which are likely smaller than the within-area correlations), and find that as a first approximation, the input correlations to MT are the weighted average of the correlations contributed by V1 and V2, respectively. This means that if the correlation contributed by V1 is greater than that contributed by V2, cooling will lead to an increase in correlations, otherwise a decrease.

We assume that binocular disparity and direction tuning in MT are independent of each other (DeAngelis and Newsome, 1999; Smolyanskaya et al., 2013) and, for simplicity, that V2/V3 input confers only information about binocular disparity and V1 input only information about motion. We further assume that the correlation between two inputs depends primarily on their stimulus tuning, i.e. the correlation of V1 inputs differs systematically depending on the motion preference of the MT neurons they contribute to, while V2 inputs selective to binocular disparity will have different correlations depending on their disparity tuning. This assumption is based on the positive relationship between similarity of tuning and noise correlations observed in a large number of studies (reviewed in Cohen and Kohn, 2011). As a result, on average, model V2 input correlations will be the same to MT neuron pairs belonging to the same or to different motion pools, but will vary with regard to MT neuron’s binocular disparity tuning preferences. The opposite will be true for MT neurons belonging to the same or different binocular disparity pools.

It then directly follows from equation (3) that the correlation difference between motion pools increases with cooling, i.e. with decreasing κ (ignoring V1-V2 correlations for simplicity, full equations in Experimental Procedures):

$$c_{motion-same}^{MT-input}-c_{motion-diff}^{MT-input}=\frac{1}{1+{\kappa }^2}\left(c_{motion-same}^{V1}-c_{motion-diff}^{V1}\right)$$ (4)

Equivalently, it follows for the disparity-task related input correlations:

$$c_{disp-same}^{MT-input}-c_{disp-diff}^{MT-input}=\frac{{\kappa }^2}{1+{\kappa }^2}\left(c_{disp-same}^{V2}-c_{disp-diff}^{V2}\right)$$ (5)

We see that as κ decreases, the binocular-disparity related difference in input correlations decreases and—together with the decrease in firing rates—leads to the decreasing disparity DPs that we empirically observe. Note that while complete cooling reduces the disparity-related structure in the correlations, and hence disparity DP, by 100%, the increase in motion-related structure is only 50%, which would translate into a DP increase of 22% if firing rates did not change with cooling. Since firing rates decrease, the increase in observed DP is predicted to be even smaller. Note also that optimal read-outs are a special case of the 2-pool read-out discussed here, with the main difference being that the above equations systematically overestimate DPs for read-outs close to optimal (Haefner et al. 2013).

For illustration purposes, we show the results of a representative simulation based on homogenous V1 and V2 populations providing input to MT. We assumed limited-range correlations commonly found in cortex and read-out weights that depend systematically on the preferred stimulus of each neuron (Figure 7). As in the data and the analytical prediction, cooling leads to a reduction in DP during the depth task and a slight increase in DP during the motion task. The results are qualitatively the same for realistic simulations with heterogeneous neuronal populations and are robust to the details of the read-out (i.e. whether pooling weights are random or optimal; Figure S5). These simulations demonstrate that a large class of feedforward models, including a wide range of read-outs from uniform to optimal, and realistic correlation structure, is compatible with our empirical observations.

Figure 7. Simulation of the effect of V2 inactivation.

(A) Correlation structure of inputs from V1 to MT. MT neurons sorted by preferred motion direction.

(B) Correlation structure of inputs from V2 to MT. Since MT neurons are sorted by preferred direction, and since we assumed no systematic relationship between direction and disparity tuning in MT, the limited-range correlation structure with respect to disparity (analogous to the one with respect to motion direction in A) is shuffled in this view.

(C) Resulting input correlation structure to MT in the control condition. During cooling, the influence of V2 decreases, and with complete cooling, the total input correlations to MT are identical to those provided by V1 shown in A.

(D) Read-out weights depend on the preferred stimulus in each of the motion and depth tasks (different for each task) such that most informative neurons are preferentially read out. Qualitatively identical results are obtained in panels E and F for random weights, and optimal weights (see also Figure S5).

(E) Motion DP change due to cooling. Blue: simulation, red: analytical approximation.

(F) Depth DP change due to cooling. For simulation parameters see Experimental Procedures. For DP calculations we assume the total input correlations to MT to be equal to the response correlations since the effect of the decrease in firing rate on output correlations is small in our data.

Discussion

We found that the feedforward V2/V3 input to MT, which conveys more information about depth than motion, makes a substantial contribution to choice-related signals in MT during a depth task but not a motion task. Our modeling results suggest that the selective change in decision signals can be explained by a change in noise-correlation structure among MT neurons that would be expected from the selective inactivation of inputs that confer selectivity for binocular disparity.

Based on several lines of evidence, we think it is highly likely that V2 confers noise correlations on pairs of MT neurons according to the similarity of their tuning for binocular disparity. There are two basic, non-exclusive ways in which this can happen: 1) V2 might already contain such correlations and pass them on to MT and 2) they might be created by the convergence of V2 inputs on to MT neurons. With respect to the first possibility, there is already evidence from many studies that similar stimulus selectivity generally entails higher noise correlations (summarized in Cohen & Kohn 2011) and, furthermore, such a correlation structure is required for observing choice-related activity in V2 in a depth discrimination task, as reported by Nienborg and Cumming (2006). The second possibility is likely due to the mechanisms of inherited stimulus selectivity—that is, MT neurons that have similar tuning properties for binocular disparity are more likely to receive convergent input from similarly tuned V2 neurons. Given these reasonable assumptions, it follows that inactivating V2 would reduce the noise correlations among MT neurons selectively according to the similarity of their tuning for binocular disparity.

We did not record from pairs of neurons in our experiment and were therefore not able to measure such noise correlations directly. However, the reduction in spike count variability in our data is highly suggestive of a reduction in correlated input in a classic model of cortical neuron responses relating spike count variability to input correlations (Shadlen and Newsome, 1998). Prevailing models of cortical spiking necessitate the presence of input correlations to account for the typically high levels of spike count variability (e.g. FF ≥ 1) observed in neocortex (Shadlen and Newsome, 1998; Stevens and Zador, 1998). A decline in Fano factor is therefore likely a signature of a decline in correlated input arriving to MT neurons via the V2/V3 pathway. The observation that the Fano factor was reduced during both tasks, both before and during stimulus onset, and during receptive field mapping (data not shown), when the animal’s only task was to maintain fixation, argues against a feedback-mediated change in task strategy. That one of the two animals exhibited a neurometric and behavioral impairment during the motion task provides additional evidence against a top-down change. The impetus for top-down mediated changes—in the form of an increased number of errors during inactivation—was almost as strong in this animal with regard to motion processing as that for depth; however motion task DP did not change.

It is not clear why one animal’s task-related motion processing was impaired even though tuning was substantially more impaired for binocular disparity, as in the three other animals in which it has been tested (this study and Ponce et al. 2008). It is possible that more motion information arrived via V2/V3 in this animal, or that this animal was simply more susceptible to non-specific visual changes that may have been produced by inactivation. Whatever the reason, the data indicates that even if some motion information arrives via V2/V3, this input does not confer the structured correlations that give rise to motion task DPs. Instead, these arrive via the convergence of V1 input, from feedback to MT, or both.

We found that considerable choice related activity remained during the depth task, even though V2 neurons were virtually silenced by inactivation (Ponce et al., 2008). This reveals an oversimplification in our model, in which we assume that the V2/V3 input is the only source of binocular disparity information in MT. In fact, various measures of binocular disparity information in MT—neurometric performance, tuning index—confirm that some information remains during V2/V3 inactivation. These signals could either arrive directly from V1, which also contains neurons tuned for binocular disparity (Cumming and DeAngelis, 2001) or they could arise from neurons in V2 or V3 that are incompletely silenced.

Our analytical modeling results assume that the decision is based on the difference in average activity of two pools of neurons (Shadlen et al. 1996). Using simulations we confirmed that our qualitative conclusions hold for a wide range of read-out schemes. As long as the two pools are approximately balanced in their contribution to the decision, depth-task DPs depend primarily on the binocular disparity-related structure in MT correlations, and motion DPs depend primarily on the motion-related structure, and not on other details such as average correlations or structure with respect to variables that are not related to the task. If, on the other hand, the decision is only based on comparing the activity of a single pool of neurons to an internal threshold, the effect of cooling on DP will depend on these other factors and on the V1-V2 correlations. In this scenario the model requires fine-tuning to reproduce the observed effects, making the 2-pool model the more parsimonious one.

Our work differs from previous studies aimed at dissociating the roles of different sources of choice-related signals, which focused primarily on measuring the relative contributions of causal bottom-up signals and top-down cognitive modulations (Cohen and Newsome, 2009; Nienborg and Cumming, 2009; Smith et al., 2011). It has been reported that a large component of the decision signals in V2 arises from a source resembling top-down attention (Nienborg and Cumming, 2009). As a consequence, the feedforward pathway from V2 to MT is likely to carry a mix of bottom-up and top-down information. Thus inactivating V2 has the effect of removing both a sensory (“bottom-up”) and a task-related (“top-down”) contribution to decision-related signals in MT. Furthermore, it is possible that direct feedback onto MT neurons changed during V2/V3 inactivation simply by virtue of the fact that inactivation likely changes activity in higher level visual areas that feed back onto MT. To account for the disproportionate change in DP during the depth task this top-down input would have to selectively affect neurons based on their depth preferences. Although such selective feedback mechanisms are known to exist (Treue and Martinez-Trujillo, 1999), it is not known whether or how they are affected by inactivation of bottom-up inputs.

There are several trivial explanations for changes in decision signals that we believe do not account for our results. They fall into two categories of factors that may differ with inactivation: changes in neuronal response properties or changes in the animals’ behavioral state. The differential changes in DP during the two tasks, which were interleaved in short blocks and measured for every neuron reported, could not be due to generic changes in the gain of the neuronal response or reduced behavioral motivation. Instead, to account for our results any changes must be specific to the depth task. Since the changes in neuronal responsiveness were similar between the two tasks, this rules out confounds from changes in spiking statistics.

The link between neuronal activity and behavior can be broken if animals simply guessed more often during inactivation. However, our data indicate that motivation was unchanged. Both animals’ performance was unaffected on interleaved trials presented in the ipsilateral visual field, they continued to respond to the stimuli in proportion to the amount of signal in each trial, and we found no changes in false alarms or breaks in fixation that would suggest reduced motivation or a decrease in criterion. We also found no relationship between changes in behavioral performance and changes in DP within sessions. Together, these results demonstrate that our results are not due to changing behavioral strategies.

In summary, we used a causal intervention to selectively probe the role of a feedforward anatomical input to decision signals in MT. The strength of our experiment comes from monitoring the choice-related activity of the same neuron both before and during inactivation, allowing us to make a direct assessment of the contribution of this input. We found that the V2/V3 pathway, which conveys sensory information predominantly about binocular disparity to MT, makes a substantial contribution to choice-related activity during a depth task but not a motion task. These results provide the first direct evidence for a modality-specific role of feedforward inputs to choice-related activity. Further, we propose a mechanism by which this input contributes to choice-related activity in MT by inducing structured correlated noise among MT neurons. Combining these methods with population recoding techniques in the future will allow us to more completely understand how feedforward inputs affect the correlation structure and decision signals at subsequent processing stages.

Experimental Procedures

All animal procedures complied with the National Institutes of Health Guide for Care and Use of Laboratory Animals, and were approved by the Harvard Medical Area Standing Committee on Animals.

Detailed methods regarding the behavioral task, visual stimuli, and electrophysiology can be found in the Supplemental Experimental Procedures. In brief, two experimentally naïve adult male macaques (Macaca mulatta, 10 and 12 kg) performed reaction-time depth and motion detection tasks in which they detected onset of coherent depth—as specified by binocular disparity—or motion in noisy random dot stimuli (Cook and Maunsell, 2002a). After a 500 ms fixation period, a random dot stimulus appeared. Stimuli were noisy with regard to either depth or motion, depending on the task block, and after a random time (0.5-5.5 seconds, exponentially distributed with a mean of 1.4-1.6 seconds) changed to contain signal in the relevant dimension (“signal onset”). Animals were rewarded for making an eye movement toward the stimulus within 200-650 ms after signal onset. Trials were classified as false alarms if the animal responded early or less than 200 ms after signal onset and as misses if the animal did not respond within 650 ms of signal onset. When fixation deviated from the 0.8-1.2° window around the fixation point, the trial was aborted and excluded from analysis.

Activity of single neurons in MT was recorded using standard electrophysiological techniques. Great care was taken to ensure single unit isolation during the entire experiment using online windowing and offline spike sorting.

Inactivation

Cortical tissue was inactivated by cooling loops of metal tubing—“cryoloops”—chronically implanted in the lunate sulcus as described by Ponce et al. (2008). Chilled methanol was pumped through the cryoloops to cool the surrounding brain tissue to 10-15°C, which is sufficient to eliminate visually evoked activity in the immediately surrounding cortex (Lomber et al., 1999). Temperature at each cryoloop was monitored and independently controlled by changing the flow rate of methanol from its dedicated pump.

Experimental Protocol

Figure 1C depicts each day’s experimental timeline. Upon isolation of a single MT neuron at physiological temperature (35-38°C) its preferred location, size, and speed were determined by hand mapping and direction and binocular disparity tuning were measured quantitatively using 100% coherent stimuli presented for 400 ms at least 6 times. Direction tuning was measured with a range of 8 directions spaced 45° apart and disparity tuning with 11 disparities with the following values: ±1.2, ±0.8, ±0.6, ±0.4, ±0.2, 0. If a neuron did not have a clear tuning preference for either direction or binocular disparity, it was not studied further, although such neurons were rare. Otherwise, monkeys performed the detection tasks with visual stimuli tailored to the preferences of the neuron. “Pre-cool” task data were collected as monkeys performed the tasks for approximately one hour. “Cool” data collection began after the temperature at the cryoloops had stabilized at 10-15°C for at least 5 minutes. This was done in the same way as the pre-cool data, with repeated measures of tuning properties followed by about an hour of task performance. Cooling lasted at most one hour and was never initiated more than once per day. Whenever possible, we collected “recovery” data when temperatures returned to at least 30°C at the cryoloops, sufficiently warm to resume normal visually evoked activity (Lomber et al., 1999).

Data Analysis

Behavioral performance

The proportion correct trials (out of correct and missed trials) as a function of signal strength was fitted with a logistic function using the psignifit toolbox version 2.5.6 for Matlab (http://bootstrap-software.org/psignifit/), which implements the maximum-likelihood method described by Wichmann and Hill (2001). To account for any changes in the animal’s guess rate between conditions we fixed the lower saturation to an estimate of the guess rate, which was obtained by convolving the rate of false alarms as a function of trial duration with the probability of the signal onset as a function of trial duration, while accounting for the allowed reaction time window. Typically the animals could make correctly timed guesses with a frequency of 5-15%. Behavioral performance was summarized with the behavioral threshold, the signal strength at which performance was 80% correct.

Task-related neuronal activity

Spiking activity of single neurons was collected as the animals performed the two tasks described above. Neurons were included in this study only if their mean spiking response to low-signal stimuli was greater than 15 spikes/sec in all conditions (pre-cool and cool) to preclude violations of normal assumptions. Results were similar when these neurons were included. Neuronal data were aligned to the onset of the signal stimulus unless stated otherwise. Only correct and missed trials were used.

Neurometric performance

For trials at each signal strength, the distribution of responses in the 500 ms immediately before signal onset was compared to those 50-550 ms after signal onset with the area under the receiver operating characteristic (ROC) curve (e.g. Bosking and Maunsell, 2011). For correct trials, spikes were only included up to 100 ms before the reaction time to exclude post-decision signals (Cook and Maunsell, 2002b; Price and Born, 2010) but results were similar with a variety of windows. To determine the effect of cooling on cell sensitivity we approximated a neurometric threshold by choosing the signal strength at which the neurometric performance was closest to 0.8 in the pre-cool condition and calculating the change in neurometric performance during cooling only at that signal strength. Since most MT neurons are more strongly modulated by direction than binocular disparity (DeAngelis and Uka, 2003) many neurons did not achieve neurometric performance at or above 0.8 at the strongest disparity signal strengths, leading to slightly lower pre-cool thresholds during the depth task.

Detect probability

Detect probability (DP) was calculated for a given signal strength from the area under the ROC curve for spike counts compared between correct and missed trials. The DP time course was computed in a 100-ms time window moved in 20 ms steps (Price and Born, 2010). The results did not vary with reasonable variations of this window. For each neuron, DP was calculated using only stimuli that had at least 5 completed trials and in which one trial outcome (i.e. correct or miss) did not occur more than 75% of the time (Kang and Maunsell, 2012). Responses were z-scored and combined across all signal strengths that met these criteria. For the population comparison, responses were combined across all neurons with both pre-cool and cool data for both tasks; thus responses were combined across neurons that were not recorded simultaneously. The DP at each time point was given by the area under the ROC curve for the combined z-scored responses at that time. The standard error of the mean (SEM) was computed via a bootstrap procedure (Efron and Tibshirani, 1998). At each time point, we sampled, with replacement, unpermuted pairs of z-scored rate and behavioral response to compute a new DP value. The standard deviation of 1,000 samples was taken to be the SEM.

To compare changes in DP between pre-cool and cool conditions, DP was computed from the combined z-scored rates across all neurons in a single time window 100-475 ms following signal onset. We used a similar resampling procedure to determine whether DP values were different between pre-cool and cool conditions, with the null hypothesis being that differences in DP were drawn from a distribution with mean 0. We first computed resampled DPs for each task and condition (cool, pre-cool) by sampling with replacement from the neuronal responses associated with each behavioral outcome. The reported p-value is the probability of observing a difference greater than zero in the distribution of differences between resampled cool and pre-cool DP for each task. Controls for involuntary eye movements are described in the Supplemental Experimental Procedures.

Spike count mean to variance relationship

The Fano factor (FF) time course was computed in a sliding 50-ms window moved every 25 ms using the Variance toolbox (Churchland et al., 2010) without mean-matching. At each time point we computed the mean and variance of the spike count response to each unique stimulus tested with each neuron. Individual neurons’ FF was computed from spike counts collected in a fixed window 50-300 ms after signal onset. The variance-to-mean ratio was computed for each unique stimulus type—each signal strength for each task—and then averaged for each neuron.

Modeling

Assuming a linear read-out of the responses r of a population of sensory neurons by a hypothetical decision neuron, r^D = w^T r (Shadlen et al. 1996, Haefner et al. 2013), which is compared to a fixed threshold to make a binary decision, the detect probability of an individual neuron i, DP_i, is related to the noise covariance matrix for the sensory population, C = cov(r, r), the read-out weights, w, and the fraction of detect trials, p^detect, by the following (approximate) equation (Haefner 2015):

$$D{P}_i\approx \frac{1}{2}+\frac{\sqrt{2}}{\pi }\frac{{\left(\mathit{Cw}\right)}_i}{\sqrt{C_{ii}{\mathit{w}}^{\mathit{T}}\mathit{Cw}}}g\left(p^{detect}\right)\text{with}g\left(p^{detect}\right)=\frac{exp\left\{-\frac{1}{2}{\left[{\Phi }^{-1}\left(p^{detect}\right)\right]}^2\right\}}{4p^{detect}(1-p^{detect})}$$ (8)

As described in Supplemental Experimental Procedures, the range of fraction of detect trials, 0.3 ≤ p^detect ≤ 0.7, is narrow enough in our experiments that the influence of the criterion, g(p^detect), can be ignored, and that we can use equation (2) previously derived for choice probabilities (Haefner et al. 2013). While equation (2) was derived for uniform weights within each pool, it is a good approximation for the average DP for a wide range of read-out weights, e.g. weights proportional to how informative a neuron is about a stimulus (Figure S5). The deviation from the true DPs becomes larger for weights closer to linear optimality but is still largely confined to an overall scaling (Haefner et al. 2013).

For our model we assume the inputs to MT neurons to be a linear combination of the V1 and V2 inputs: ${\rho }_{\text{i}}^{\text{MT}}={\rho }_{\text{i}}^{\text{V}1}+\kappa {\rho }_{\text{i}}^{\text{V}2}$. Based on this assumption we can compute the correlation between the inputs to two MT neurons i and j from the ratio of their covariance and variances:

$$cov({\rho }_i^{V1}+\kappa {\rho }_i^{V2},{\rho }_j^{V1}+\kappa {\rho }_j^{V2})=C_{ij}^{V1}+{\kappa }^2{C}_{ij}^{V2}+\kappa C_{ij}^{V1-V2}+\kappa C_{ij}^{V2-V1}$$ (9)

and

$$var({\rho }_i^{V1}+\kappa {\rho }_i^{V2})=C_{ii}^{V1}+{\kappa }^2{C}_{ii}^{V2}+2\kappa C_{ii}^{V1-V2}$$ (10)

Where $c_{ij}^X$ is the covariance between inputs i and j from area X, and C_ii are the corresponding variances. For homogenous inputs of equal response variance, the covariance C in these equations can be replaced by correlations to directly yield equation (3).

In the remainder of the main text we ignored the correlations between inputs received from V1 and from V2 for simplicity. Including them, equations (4) and (5) read:

$$\begin{array}{cc}\hfill c_{motion-same}^{MT-input}& -c_{motion-diff}^{MT-input}\hfill \\ \hfill & =\langle \frac{c_{ij}^{V1}}{\sqrt{1+{\kappa }^2+2\kappa c_{ij}^{V1-V2}}}\rangle {}_{(i,j)\in motion-same}-\langle \frac{c_{ij}^{V1}}{\sqrt{1+{\kappa }^2+2\kappa c_{ij}^{V1-V2}}}\rangle {}_{(i,j)\in motion-diff}\hfill \end{array}$$ (11)

Since $c_{ij}^{V1-V2}$ will generally be small and positive (Cohen and Kohn, 2011), this expression behaves qualitatively the same as the case of $c_{ij}^{V1-V2}$ = 0 presented in the main text, increasing in value as κ decreases from 1 to 0 with cooling. Equivalently, it follows for the disparity-task related input correlations:

$$\begin{array}{cc}\hfill c_{disp-same}^{MT-input}& -c_{disp-diff}^{MT-input}\hfill \\ \hfill & =\langle \frac{{\kappa }^2{c}_{ij}^{V2}}{\sqrt{1+{\kappa }^2+2\kappa c_{ij}^{V1-V2}}}\rangle {}_{(i,j)\in disp-same}-\langle \frac{{\kappa }^2{c}_{ij}^{V2}}{\sqrt{1+{\kappa }^2+2\kappa c_{ij}^{V1-V2}}}\rangle {}_{(i,j)\in disp-diff}\hfill \end{array}$$ (12)

We assumed a population size of 1000 neurons in our simulations. We drew the covariance matrices for the V1 and V2 inputs from a Wishart distribution with 10⁴ degrees of freedom around a mean defined by limited-range correlations with an exponential decay. A 10:1 ratio between degrees of freedom and number of neurons was chosen to yield an intermediate level of heterogeneity in the resulting covariance structures but our results were not sensitive to this parameter. The maximum value for inputs to neurons with the most similar tuning preferences was 0.2 and it decreased to 0.07 for the least similar neurons. We did not attempt to fit measured DP values exactly since that would require too many assumptions about unconstrained details of the correlation structure, but emphasize the qualitative agreement with our data.

Figure 6. Feedforward model framework.

Correlated inputs from V1 and V2 determine the output (response) correlations in MT. Cooling reduces the firing rate of V2 and MT neurons and thereby the relative weighting of correlated V1 and V2 inputs. Since our data suggest no change in read-out weights with cooling, the observed changes in DP must be due to the reduction in correlated V2 input. Quantities affected by cooling are indicated in blue. See also Figure S4.