Correlations between the activity of sensory neurons and behavior: how much do they tell us about a neuron’s causality?

Abstract

How the activity of sensory neurons elicits perceptions and guides behavior is central to our understanding of the brain and is a subject of intense investigation in neuroscience. Correlations between the activity of sensory neurons and behavior have been widely observed and are sometimes used to infer how neurons are used to guide a certain behavior. This view is challenged by 1) theoretical considerations that these correlations rely on the existence of correlated noise and its structure, and 2) recent empirical observations suggesting that such correlated noise is not a fixed network property but that it depends on various sources, and varies with a subject’s mental state.

Introduction

A core question neuroscientists are venturing to resolve is how the information carried by sensory neurons is used by the brain to create perceptions and guide behavior. A direct approach is to perturb the activity of sensory neurons and quantify the effect on behavior. As such direct measurements of the causal effect of sensory neurons have traditionally lacked single cell resolution (1–6, but see 7), and knowledge of the tuning properties of the individual manipulated neurons, an alternative route has been to observe the activity of individual sensory neurons without perturbing the system, while an animal is performing a sensory task. For such tasks, weak but consistent trial-to-trial correlations between the activity of individual sensory neurons and the animal’s behavior have been found for different sensory modalities, brain areas and perceptual tasks8–20. Frequently, these correlations have been quantified as ‘choice-probabilities’, which measure the probability with which one could predict an animal’s choice on each trial, from the spike counts on a given trial (see box 1). Choice-probabilities have often been –implicitly or explicitly-interpreted as reflecting the causal effect of these sensory neurons in the particular sensory task. In this interpretation, larger CPs in a neuron imply that a neuron is given more weight in the decision-process, and hence CPs can be used to infer how neuronal activity is “read out” 15,21. However, theoretical work has long recognized that this interpretation is complicated by the structure of correlated activity in the population of sensory neurons 22,23. Recent theoretical and physiological work on these correlations, reviewed below, has revealed even greater complexity, so that the extent to which CP reflects the “read out” of neuronal activity is still poorly understood.

Box 1: Choice-probability

Assume a neuron’s firing is correlated trial-by-trial with an animal’s perceptual judgment. A number of ways have been used to quantify this correlation. 1) Measure the neuron’s mean firing separated by the animal’s choice (e.g. choice-contingent post-stimulus-time-histogram, PSTH). The stronger the correlation between the neuron’s firing and the animals judgment, the larger the choice-contingent difference in the neuron’s mean firing. 2) Choice-probability is a robust way to quantify the reliability of this correlation based on signal-detection-theory. The spike counts for each trial are separated according to the animal’s perceptual decision, yielding two distributions of spike counts, one for each decision. From these distributions, receiver-operating-characteristics (ROC) curves are obtained, and a neuron’s choice-probability corresponds to the area under the ROC curve. It corresponds to the probability with which an ideal observer would be able to predict the animal’s choice on a each trial, given one spike count from each pool, and the distributions of spike counts for each choice. Choice-probability is bound between 0 and 1, and is a signed metric, i.e. it takes the neuron’s tuning for the task relevant feature into account, (as opposed to metrics such as, e.g. mutual information):

• CP>0.5: the neuron fires more spikes when the animal chooses the neuron’s preferred feature.

• CP=0.5: the neuron is uninformative about the animal’s choice

• CP<0.5: the neuron fires fewer spikes when the animal chooses the neuron’s preferred feature.

Choice-probabilities depend on the correlations between sensory neurons (‘noise correlations’)

Much of our understanding of how CP might arise was developed in a classic computational study 22. Although this model has been extended e.g.24, its basic structure remains the dominant theoretical framework for perceptual decision-making 25. One key insight was that if sensory decisions are based upon a pool of many independent neurons, then the correlation between the activity of any one neuron and that decision (i.e. the model CP) will be very weak. Only by introducing correlations in activity between neurons was it possible to model CP of the observed magnitude. Because this correlated activity is introduced across presentations of identical stimuli, it is often referred to as “noise correlation”. The importance of the noise correlation is illustrated by how this study explained a common finding. Many studies have found that the size of CPs in a population of neurons is correlated with these neurons’ reliability for the task 9,10,14,21,26–29. This relationship is very natural if CP reflects how a neuron is read out: an optimal decoder should weight those neurons more strongly that are more reliable for the task at hand. However, when Shadlen and colleagues22 implemented such weighting in the simple pooling model, it did not reproduce this relationship. There are two possible solutions. First, if the number of neurons contributing to the pool is small (fewer than 100), CP did reflect the weight given to a neuron. Such a small pool seems biologically implausible. The second possibility, which was proposed by Shadlen et al., was to adjust the structure of the correlations between neurons. Introducing higher correlations between neurons with high reliability than for neurons with lower reliability also explained the observed relationship. This demonstrates that it was not a neuron’s contribution to a decision, but its correlation with other neurons supporting a decision that was the decisive factor for a neuron’s CP. A similar observation was made by a study23 modeling the effect of learning in a direction discrimination task. A recent study showed the consequence of this at its extreme. Cohen and Newsome30 used a very similar model, using two pools of correlated neurons, but split each pool into two groups: one group used for the decision, i.e. it was causal, while the second group was not causal (activity was ignored in the decision rule). When the noise in all neurons was uncorrelated, neurons in the non-causal pool had no CPs. However weak correlated noise produced significant CPs in neurons of the non-causal pool. Indeed, when there were 100 or more neurons in each pool, the CPs for the neurons in the non-causal pool became as large as the CPs in the causal pool. This indicates that at least in a simple pooling model one can construct a network architecture in which neurons that make no contribution to the decision (weight of 0) show equal CPs only by virtue of their correlation with neurons that are causal. Of course it is essential that the model contain a pool of neurons that do contribute to the decision (non-zero weights) – without these no neurons would show CP.

A virtue of these simple pooling models is that causality is clearly defined. In more realistically complex networks, sensory neurons may contribute to a decision via computations that are not captured by simple pooling. How these different computations affect the relationship between neuronal signals, interneuronal correlations, and decisions is an area that needs further exploration. But given current models, it is not clear that CP can be used as an estimate of how neurons are weighted in a decision, and this intuitive-seeming explanation should be used with caution 21,31. For the same reason, the structure of neuronal correlations can explain what appear to be discrepancies between psychophysical measures of sensory weights and neuronal CPs (32, their Fig. 6).

At first sight, explaining the relationship between reliability and CP by postulating the necessary correlation structure may seem artificial, but there are a number of plausible ways this might arise. Neurons with similar stimulus preferences tend to have higher noise correlations. A number of incidental properties of the stimulus (e.g. size, speed) might therefore determine which neurons are most sensitive to the parameter under study in a given stimulus. Those neurons will have higher inteneuronal correlation because of their shared preference for the incidental parameters.

The structure of the correlations, not their absolute size, is relevant for choice-probabilities

The above computational work showed that the size of CPs is determined by the size of the noise correlations in a population. However, as Cohen and Newsome 30 recognized, CPs depend not only on the size of the noise correlation, but also on the structure of the noise correlation. Consider a discrimination task in which the observer’s task is to determine whether the motion in a stimulus is upward or downward. In simple pooling models following Shadlen et al.22 the decision is based on the activity of two pools of sensory neurons, one pool supporting upward motion, the other downward motion. As discussed above, for plausible pool sizes (100 or more), observed CPs only occur if the noise in the neurons within each pool is correlated. But it is equally important that correlations between neurons in different pools are weaker that those between neurons within a pool. This effect is demonstrated in Figure 1, which repeats a simulation from 22 with one modification. In that study, the noise correlation between neurons in different pools was 0. Here, we allow for noise correlation between pools (r_between), and vary this independently of the noise correlation between neurons in the same pool (r_within). It is clear that CP is largely determined by the difference in these noise correlations (r_within-r_between), not by the absolute value of either. This follows mathematically as the decision variable corresponds to the difference of the summed neural activity in each pool. The shared noise between pools is removed by subtracting it from itself. As a consequence, even when the mean correlation across the entire population is zero, CPs still occur when r_within exceeds r_between.

Figure 1. Influence of the structure of noise correlations on choice-probability.

Left panel: Schematic of the pooling model developed by Shadlen et al. (1996) for a direction of motion discrimination task. For a two alternative discrimination task, the decision is based on the activity of two pools of sensory neurons, each of which supports one of the alternative decisions (${{\displaystyle x}}_1^{up}\dots {{\displaystyle x}}_n^{up}$ for upward motion vs ${{\displaystyle x}}_1^{down}\dots {{\displaystyle x}}_n^{down}$ for downward motion in the presented case). The decision is based on the decision-variable, which corresponds to the difference of the summed activity of each pool of sensory neurons. Right panel: Choice probabilities are computed for a simple pooling model (100 neurons/pool), in which the model decision on each fixed duration trial is determined by which pool has the largest summed activity 22. We vary both the extent of correlation between pairs of neurons belonging to the same pool (r_within), and the correlation for pairs where the neurons come from different pools (r_between), over the range of empirically observed values. The results are plotted as a function of the mean correlation, and the difference. Independent of the average value of noise correlation, the size of the observed choice probability depends upon difference in correlation (r_within – r_between) between these two pairings. The two black markers show how these values changed with task instruction in the study by Cohen and Newsome30. To compute these numbers we assumed that all neurons whose preferred direction has a vector component to the right contribute to the rightward pool.

This result only applies to discrimination tasks, where the decision must depend on the difference between two pools of neurons. For detection tasks, where the activity of all neurons might be pooled, there is no need for such structured correlations. The term “Detect probability” 11 is used to describe decision-related neuronal activity in such tasks, emphasizing the different implications.

This need for a structured correlation matrix clarifies that a recent result may still be compatible with our current understanding of CPs. Angelaki et al. recently reported noise correlations near 0 between neurons in area MSTd of monkeys trained on a heading discrimination task (Society for Neuroscience Meeting, 2009, Abstract 558.10). The same authors had previously reported significant CPs in MSTd in the same task 21,so this finding at first sight seems to challenge the notion that CPs require noise correlation. However, a more detailed analysis of the noise correlations in MSTd showed that despite the low overall noise correlation, neurons with similar tuning showed higher noise correlation than neurons with dissimilar tuning, exactly what figure 1 shows is required by the simple pooling model.

Thus, in these pooling models CPs depend on the structure of the noise correlation, not its overall size. In more complex networks this relationship may also depend upon the exact computation that a given neuron performs, but the correlation structure is likely to remain an important factor. This is an important point to recognize, since this structure is harder to measure empirically than the mean correlation. Indeed current measures in awake animals do not constrain this crucial value very tightly. For the widely-used motion task in area MT, the value varies considerably between studies 33,34, as does it for area V135–38. As second important consequence is that even a dramatic reduction of overall noise correlation with spatial attention (see below) would not affect CPs for such pooling models if it occurs uniformly across neurons with similar and dissimilar tuning, as found by 39 (their Fig.2d).

Noise correlations are not a fixed network property

The term “noise correlation” is widely used to describe correlations between neurons measured across repeated presentations of the same stimulus. This distinguishes these from “signal correlations” 40 – correlations measured across presentations of different stimuli that reflect the similarity of tuning properties. But whether “noise correlations” reflect true noise is far from clear. The fact that two neurons show “noise correlation” means that they both receive some common input signal that is not determined by the stimulus alone. This signal may reflect the accumulated effect of noise in divergent ascending afferents. But this is only one possible interpretation, and the observation of “noise correlations” alone is in no way evidence in favor of this particular interpretation. It should also be noted that if the noise correlations were attributed to a completely different source, it would have no impact on CPs in pooling models discussed above.

Many observations are compatible with the view that noise that is common to ascending afferents generates correlations. Noise correlations are stronger between nearby neurons and those that have similar tuning properties 34, which could reflect the fact that these pairs are more likely to share common input. More recently, it has been shown that noise correlations depend on the stimulus presented 36,37,41 in anaesthetized animals. These properties might also reflect underlying connectivity, since different stimuli will activate different populations of afferent neurons. Finally, recent evidence suggest that training on a particular task can also decrease (Society for Neuroscience Meeting 2009, Abstract 558.10) or increase42 noise correlations. This could reflect changes in the connectivity as a result of training.

However, a number of other studies have suggested that correlated fluctuations in activity not caused by external stimuli may reflect complex network states, not simply noise (see review by Ringach43). In particular, two recent studies have shown that noise correlations decrease when monkeys attend to a spatial location compared to when they attend elsewhere39,44. These changes occur on timescales too short to reflect changes in the underlying connectivity, indicating that noise correlations do not only reflect fixed network properties.

Noise correlations and CPs have multiple sources, including feed-back

The most relevant study of changes in noise correlations for CPs 33 used an elegant design in which monkeys performed a direction discrimination task along one of two orthogonal axes, while recording from pairs of direction selective neurons in MT. For example, on some trials animals might be required to discriminate upward from downward motion, while on other trials they performed a left-right discrimination. The axes were carefully chosen such that for one task both neurons belonged to the same decision pool, while the other axis placed them in opposite decision pools. This design allowed an identical stimulus, with no motion signal, to be presented in the context of two different tasks. Interneuronal correlation depended on task the animal had been set: it was systematically higher when the task placed both neurons in the same decision pool.

Cohen and Newsome explained these task dependent changes in noise-correlation by suggesting variations in attention, which in turn affects spike rates 45–47. Importantly, similarly to an earlier suggestion48, they invoked a component of feature selective attention that changed even within the same task (‘alternating attention’): e.g. if the axis of discrimination is up versus down, on some trials, the monkey would attend more to up and on some it would attend more to down. This variation in feature selective attention provides a common signal that adds to interneuronal correlation. The authors used a simulation to show that this effect was not simply a consequence of the CP demonstrated by the neurons. The reverse however is not true: the attentional signal that contributes to correlation could contribute significantly to the measured CP, because it changes the structure of interneuronal correlations (compare Figure 1). For simple pooling models the effect can be estimated from Figure 1. The two crosses mark the mean values of correlation within and across pools, under the two conditions studied. The change in predicted CP is from 0.59 to 0.55. Thus, this study demonstrated a top-down origin of noise correlation and suggests a top-down component to CPs.

Another recent study also suggested a contribution of such top-down signals to CP 49. It used a variant of reverse correlation to compute how monkeys weighted the visual stimulus in a disparity discrimination task, while simultaneously measuring CPs in V2 neurons. They showed that animals gave less weight to the visual stimulus as the trial progressed, but CPs increased and remained high until the end. To explain this in the current theoretical framework it is necessary to invoke substantial changes in structure of interneuronal correlations (increasing for neurons within a pool, but not between pools), during the course of each trial. It is hard to see how noise shared by ascending afferents could produce such effects in current pooling models. Such behavior could arise from an effect of feature selective attention very similar to that proposed by Cohen and Newsome. Rather than a random alternation between trials, these data would result if feature-selective attention were directed to the feature corresponding to the upcoming choice. Such a feedback signal would also explain a feature in a study50 where monkeys were trained to switch between a direction and a disparity discrimination task. As the monkeys’ choice targets were the same for both tasks, the neuronal preferences could map onto the same choice target for both stimulus dimensions (‘congruent neurons’), or onto different choice targets (‘incongruent neurons’). Incongruent neurons only showed CPs in one of the tasks. Indeed, in the task for which the mean CP was not significant, neuronal responses became negatively correlated with behavior starting about 250ms after stimulus onset, delayed compared to the CP for the task showing significant CPs. As it was not a central focus of the study, the statistical significance of this delayed signal was not reported. But a feedback signal with a sign that depended on the location of the upcoming choice target would exactly cause such a delayed negative correlation.

The feature selective attention signal can also be thought of as expectation of the occurrence of a particular feature. One observation that is compatible with such an expectation signal in the study by Nienborg & Cumming was that CPs were already present at the moment of the stimulus onset. In other words, it is possible that at the moment of stimulus presentation the cortex was in a state that influenced the animal’s subsequent decision about the stimulus. Signs for such an expectation signal at stimulus onset, which subsequently influences the perceptual decision, have also been observed for area V1 51, LIP52, and in fMRI studies in humans53,54.

Conclusions

Several pieces of evidence suggest at first sight that CPs are a useful metric to evaluate the contribution of sensory neurons to perceptual decisions. Neurons that are reliable for a task tend to show higher CPs than neurons that are less reliable 9,10,14,21,26,27. Where psychophysical evidence indicates that a particular group of neurons is not used in a task, those neurons did not show a CP 32. Nonetheless, in our current theoretical framework, CP in a given pooling model (where at least some neurons contribute to the decision) is governed by interneuronal correlations in activity, not by the contribution any one neuron makes to the decision. The property that determines CP is the difference between (1) the mean correlation in pairs of neurons from a single decision pool, and (2) the correlation between pairs of neurons that come from different decision pools. Paradoxically, this correlation structure, implies a regime in which noise correlations degrade the population code (as reviewed by 55, but see also 56).

Although there is wide agreement that interneuronal correlations are necessary to explain observed CPs, the origin of these correlations remains unclear. They might reflect the summed effect of noise in shared ascending afferents. But any other common input, including top down signals, is equally possible. Recent demonstrations that interneuronal correlations change with an animal’s task, or with changes in attention, indicate that at least a component of these correlations are produced by top-down signals. An important question for future research is to quantify the relative contributions of these different mechanisms and their possible function, and to explore how correlations and their different sources influence sensory processing and decision making.