Quantifying information conveyed by large neuronal populations

Abstract

Quantifying mutual information between inputs and outputs of a large neural circuit is an important open problem in both machine learning and neuroscience. However, evaluation of the mutual information is known to be generally intractable for large systems due to the exponential growth in the number of terms that need to be evaluated. Here we show how information contained in the responses of large neural populations can be effectively computed provided the input-output functions of individual neurons can be measured and approximated by a logistic function applied to a potentially nonlinear function of the stimulus. Neural responses in this model can remain sensitive to multiple stimulus components. We show that the mutual information in this model can be effectively approximated as a sum of lower-dimensional conditional mutual information terms. The approximations become exact in the limit of large neural populations and for certain conditions on the distribution of receptive fields across the neural population. We empirically find that these approximations continue to work well even when the conditions on the receptive field distributions are not fulfilled. The computing cost for the proposed methods grows linearly in the dimension of the input, and compares favorably with other approximations.

1. Introduction

Information theory has the potential of answering many important questions about how neurons communicate within the brain. In particular, it can help determine whether neural responses provide sufficient amounts of information about certain stimulus features, and in this way determine whether these features could possibly affect the animal’s behavior (Rieke et al., 1997; Bialek, 2012). In addition, a number of previous studies have shown that one can understand many aspects of the neural circuit organization as those that provide maximal amounts of information under metabolic constraints (Laughlin et al., 1998; Bialek, 2012). Key to all of these analyses is the ability to compute the Shannon mutual information (Cover and Thomas, 2012). When estimating the information transmitted by neural populations from experimental recordings, all empirical methods produce biased estimates (Paninski, 2003). There are several approaches to trying to reduce or account for this bias (Nemenman et al., 2004; Strong et al., 1998; Brenner et al., 2000; Treves and Panzeri, 1995), but these approaches do not have finite-sample guarantees and are generally ineffective when the population response is high dimensional. In order to make progress on this problem, we consider the case where the response functions of individual neurons can be measured and where the stimulus-conditional (“noise”) correlations between neural responses can be described by pairwise statistics (Schneidman et al., 2006). Historically, even with these assumptions the mutual information is notoriously difficult to compute in part due to the large number of possible responses that a set of neurons can jointly produce (Nemenman et al., 2004; Strong et al., 1998). The number of patterns grows exponentially with both the number of time points (Strong et al., 1998; Dettner et al., 2016) and the number of neurons.

In this paper we will describe a set of approaches for computing information conveyed by responses of large neural populations. These methods build on recent advances for computing information based on linear combinations of neural responses across time (Dettner et al., 2016; Yu et al., 2010) and/or neurons (Berkowitz and Sharpee, 2018). We will show that when each individual neuron’s firing probability depends monotonically on a (potentially nonlinear) function of the stimulus, the information contained in the full population response can be completely preserved by a linear transformation of the population output. This calculation still involves computing information between high dimensional vector variables. Therefore, we further show how the full information can be effectively approximated using a sum of conditional mutual information values between pairs of low-dimensional variables. The resulting approach makes it possible to avoid the “curse of dimensionality” with respect to the number of neurons when computing the mutual information from large neural populations.

2. Framework setup

Our analysis will target neural responses considered over sufficiently small time windows such that no more than one spike can be produced by any given neuron. We model the neural population as a set of binary neurons with sigmoidal tuning curves with response probability described by:

$$P\left(r_n=1\mid \overrightarrow{s}\right)=\frac{1}{1+e^{2f_n(\overrightarrow{s})}},$$ (1)

where $\overrightarrow{s}\in {ℝ}^D$ is the input, r_n ∈ {−1, 1} is the activity of the n^th neuron, and $f_n(\overrightarrow{s})$ is a scalar function of $\overrightarrow{s}$ representing the activation function of the n^th neuron. The population consists of N such neurons, and the population response is denoted as $\overrightarrow{r}=\left(r_1,\dots ,r_N\right)$. For clarity of the derivation, we will initially assume that neural responses are independent conditioned on $\overrightarrow{s}$:

$$P(\overrightarrow{r}\mid \overrightarrow{s})=\prod _{n}P\left(r_n\mid \overrightarrow{s}\right),$$ (2)

and later discuss under what conditions our results generalize to the case where neural responses are correlated for a given stimulus $\overrightarrow{s}$. A few lines of algebra suffice to show that Eq. (2) can be expressed in the following form:

$$P(\overrightarrow{r}\mid \overrightarrow{s})=\text{exp}\left(\sum _n{r}_n{f}_n(\overrightarrow{s})-A_n(\overrightarrow{s})\right),$$

$$A_n(\overrightarrow{s})=\text{log}\left(2\text{\hspace{0.17em}cosh}\left(f_n(\overrightarrow{s})\right)\right..$$ (3)

This formulation will assist all of the approaches described below for computing the mutual information.

3. An unbiased estimator of information for large neural populations

In order to test the approaches described in subsequent sections, we first developed a Monte-Carlo method for computing the “ground-truth” mutual information that works for large neural populations. The approach relies on the knowledge of neural response parameters $\left\{f_n(\overrightarrow{s})\right\}$ to produce unbiased estimates of mutual information between $\overrightarrow{R}$ and $\overrightarrow{S}$ for different choices of $\left\{f_n(\overrightarrow{s})\right\}$ or $P(\overrightarrow{s})$. Here and in what follows, upper case letters (e.g. $\overrightarrow{S}$) represent random variables, while lower case letters (e.g. $\overrightarrow{s}$) represent specific values of the associated random variables. The input distribution $P(\overrightarrow{s})$ is defined by drawing N_stim samples; we denote this set of samples as $\left\{{\overrightarrow{s}}_{\mu }\right\}$. Because of this approximation however, $I(\overrightarrow{R},\overrightarrow{S})$ will be bounded above by log(N_stim) (as will be any unbiased estimator of mutual information).

Although there are several formulations of the mutual information in terms of the entropies of $\overrightarrow{R}$ and $\overrightarrow{S}$ it serves to examine just one:

$$I(\overrightarrow{R},\overrightarrow{S})=H(\overrightarrow{R})-H(\overrightarrow{R}\mid \overrightarrow{S}).$$ (4)

Here, $H(\overrightarrow{R})$ is the Shannon entropy of the marginal distribution of $\overrightarrow{R}$ and $H(\overrightarrow{R}\mid \overrightarrow{S})$ is the conditional entropy of $\overrightarrow{R}$ given $\overrightarrow{S}$. Because we intend to use this estimator as a way to test the quality of other approximations, we will only consider here the case of conditionally independent neural responses $\overrightarrow{R}$. In this case, the noise entropy $H(\overrightarrow{R}\mid \overrightarrow{S}=\overrightarrow{S})$ decomposes into a sum over neurons:

$$H(\overrightarrow{R}\mid \overrightarrow{S}=\overrightarrow{s})=\sum _n{\overline{r}}_n(\overrightarrow{s})f_n(\overrightarrow{s})-A_n(\overrightarrow{s}),$$ (5)

where ${\overline{r}}_n(\overrightarrow{s})$ is the expected value of R_n given $\overrightarrow{s}$:

$${\overline{r}}_n(\overrightarrow{s})=\text{tanh}\left(f_n(\overrightarrow{s})\right).$$ (6)

We denote $\widehat{H}(\overrightarrow{R}\mid \overrightarrow{S})$ as the finite sample approximation to $H(\overrightarrow{R}\mid \overrightarrow{S})$:

$$\widehat{H}(\overrightarrow{R}\mid \overrightarrow{S})=-\frac{1}{N_{\text{stim }}}\sum _{\mu }\sum _n{\overline{r}}_n\left({\overrightarrow{s}}_{\mu }\right)f_n\left({\overrightarrow{s}}_{\mu }\right)-A_n\left({\overrightarrow{s}}_{\mu }\right).$$ (7)

The conditional entropy $\widehat{H}(\overrightarrow{R}\mid \overrightarrow{S})$ can be evaluated in O(N * N_stim) time, not including the cost of evaluating $f_n(\overrightarrow{s})$. However, the marginal distribution of $\overrightarrow{r}$ will in general not factor. Thus evaluating $H(\overrightarrow{R})$ requires computing the marginal $P(\overrightarrow{r})$ for all $\overrightarrow{r}\in {\left\{-1,1\right\}}^N$. This computation grows like O(N * N_stim *2^N). Thus, evaluation of Eq. (4) is known to become intractable for realistic population sizes. To derive our estimator, we begin by rewriting $H(\overrightarrow{R})$:

$$H(\overrightarrow{r})=-\sum _{\overrightarrow{r}}P(\overrightarrow{r})\text{\hspace{0.17em}log}(P(\overrightarrow{r}))=-{\langle F(\overrightarrow{r})\rangle }_{\overrightarrow{r}},$$

$$F(\overrightarrow{r})=\text{log\hspace{0.17em}}\left({\langle P(\overrightarrow{r}\mid \overrightarrow{s})\rangle }_{\overrightarrow{s}}\right).$$ (8)

We approximate the log-marginal $F(\overrightarrow{r})$ with an empirical average:

$$\widehat{F}(\overrightarrow{r})=\text{log\hspace{0.17em}}\left(\frac{1}{N_{\text{stim}}}\sum _{\mu }P\left(\overrightarrow{r}\mid {\overrightarrow{s}}_{\mu }\right)\right)=\text{log\hspace{0.17em}}\left(\frac{1}{N_{\text{stim }}}\sum _{\mu }\text{exp\hspace{0.17em}}\left(\sum _n{r}_n{f}_n\left({\overrightarrow{s}}_{\mu }\right)-A_n\left({\overrightarrow{s}}_{\mu }\right)\right)\right)$$ (9)

In terms of numerical implementation, $\widehat{F}(\overrightarrow{r})$ can be efficiently and stably evaluated in O(N_stim) time using the logsumexp function that is implemented in many numerical libraries. To approximate the averaging with respect to $P(\overrightarrow{r})$ we draw B samples of $\overrightarrow{r}$ for every ${\overrightarrow{s}}_{\mu }$, which is easily done with (1) and (2), and denote these samples as $\left\{{\overrightarrow{r}}_{\nu }\right\}$. We can thus produce an unbiased estimate of $-{\langle \widehat{F}(\overrightarrow{r})\rangle }_{\overrightarrow{r}}$:

$$\widehat{H}(\overrightarrow{R})=\frac{1}{B{N}_{\text{stim}}}\sum _{\nu }\text{log\hspace{0.17em}}\left(\frac{1}{N_{\text{stim}}}\sum _{\mu }\text{exp\hspace{0.17em}}\left(\sum _n{r}_{n\nu }{f}_n\left({\overrightarrow{s}}_{\mu }\right)-A_n\left({\overrightarrow{s}}_{\mu }\right)\right)\right)$$ (10)

Importantly, the response entropy $\widehat{H}(\overrightarrow{R})$ requires $O\left(N*B*N_{\text{stim}}^2\right)$ operations, a substantial improvement over exact evaluation of $H(\overrightarrow{r})$ when B * N_stin ≪ 2^N. We note that even though we are able to produce unbiased estimates of $\widehat{H}(\overrightarrow{R})$, this estimator systematically underestimates the “infinite sample” entropy computed with respect to $P(\overrightarrow{s})$ explicitly, i.e. not defined by input samples (see Appendix A). Our Monte-Carlo estimator of $I(\overrightarrow{R},\overrightarrow{S})$ is the straightforward combination of $\widehat{H}(\overrightarrow{R})$ and $\widehat{H}(\overrightarrow{R}\mid \overrightarrow{S})$:

$$\widehat{I}(\overrightarrow{R},\overrightarrow{S})=\widehat{H}(\overrightarrow{R})-\widehat{H}(\overrightarrow{R}\mid \overrightarrow{S})$$ (11)

Although $\widehat{I}(\overrightarrow{R},\overrightarrow{S})$ is an unbiased estimator of the mutual information (after accounting for the approximation of $P(\overrightarrow{s})$ by samples $\left\{{\overrightarrow{s}}_{\mu }\right\}$) the variance of $\widehat{H}(\overrightarrow{r})$ and thus of $\widehat{I}(\overrightarrow{R},\overrightarrow{S})$ can be difficult to quantify. However, $F(\overrightarrow{r})$ is a bounded function because $\overrightarrow{r}$ has finite support (or, more generally, $F(\overrightarrow{r})$ can be treated as a continuous function on the compact set [−1, 1]^N). Thus, standard concentration bounds show that $\widehat{H}(\overrightarrow{R})$ is a consistent estimator of $H(\overrightarrow{R})$.

In order to test our derivation that $\widehat{I}(\overrightarrow{R},\overrightarrow{S})$ is an unbiased estimator of $I(\overrightarrow{R},\overrightarrow{S})$, we analyzed the statistics of $\widehat{I}(\overrightarrow{R},\overrightarrow{S})$ on a tractable neural population where $I(\overrightarrow{R},\overrightarrow{S})$ can be computed exactly. We let N = 10 and $f_n(\overrightarrow{s})={\overrightarrow{\varphi }}_n\cdot \overrightarrow{s}$ with ${\overrightarrow{\varphi }}_n$ uniformly distributed along the unit circle. $P(\overrightarrow{s})$ is a spherical, two-dimensional Gaussian distribution and N_stim = 8, 000. We evaluate $I(\overrightarrow{R},\overrightarrow{S})$ exactly, and get that $I(\overrightarrow{R},\overrightarrow{S})=1.3384$ nats. This is well below the upper bound of log(8, 000) ≈ 8.987 nats. We computed $\widehat{I}(\overrightarrow{R},\overrightarrow{S})$ 100 times for B = 1 and B = 3, with {s_μ} fixed. For each repetition we record the residual $\widehat{I}(\overrightarrow{R},\overrightarrow{S})-I(\overrightarrow{R},\overrightarrow{S})$. Distribution plots of the residuals are shown in Figure 1. For both distributions the sample mean is not significantly different from zero with P = 0.848 (B = 1) and P = 0.851 (B = 3) in a two-sided t-test. The simulation results therefore support the derivation of zero-bias in the proposed model-based Monte-Carlo estimator.

Figure 1:

Between exact calculation and the Monte Carlo results for the test neural population described in Section 3. Dashed black line indicates zero, while red marker and error bar are the sample mean and standard deviation.

4. Simplifying the mutual information with sufficient statistics

4.1. A vector-valued sufficient statistic

The method introduced in Section 3 can be applied for very general formulations and parametrizations of the activation functions. However, when we constrain the activation functions to be affine we can show that $P(\overrightarrow{r}\mid \overrightarrow{s})$ has especially useful properties. Specifically, we assume the following parametrization of $f_n(\overrightarrow{s})$:

$$f_n(\overrightarrow{s})={\overrightarrow{w}}_n\cdot \overrightarrow{s}-{\alpha }_n,\forall n$$ (12)

While Eq. (12) implies a strong restriction on how stimuli drive the neural responses, some results of this section can be generalized to other activation functions. The reason for this is that even the general formulation of $P(\overrightarrow{r}\mid \overrightarrow{s})$ given in Eq. (3) can be viewed as an exponential family, with sufficient statistic $\overrightarrow{r}$ and natural parameter $\overrightarrow{f}(\overrightarrow{s})$. In particular, the framework can be extended to quadratic activation functions, which are an important model for describing neurons that are sensitive to multiple stimulus features. See Appendix E for further discussion.

If Eq. (12) holds, then Eq. (2) can be rewritten as follows:

$$P(\overrightarrow{r}\mid \overrightarrow{s})=h(\overrightarrow{r})\text{\hspace{0.17em}exp}(\overrightarrow{s}\cdot \overrightarrow{t}(\overrightarrow{r})-A(\overrightarrow{s})),$$ (13)

where,

$$\overrightarrow{t}(\overrightarrow{r})=W\cdot \overrightarrow{r},W\equiv \left({\overrightarrow{w}}_1^T,\dots ,{\overrightarrow{w}}_N^T\right)$$ (14)

$$h(\overrightarrow{r})=e^{-{\sum }_n{r}_n{\alpha }_n},$$ (15)

$$A(\overrightarrow{s})=\sum _n\text{log\hspace{0.17em}}\left(2\text{\hspace{0.17em}cosh}\left({\overrightarrow{w}}_n\cdot \overrightarrow{s}-{\alpha }_n\right)\right).$$ (16)

Equation (13) is an exponential family with sufficient statistic $\overrightarrow{t}\in {ℝ}^D$, natural parameter $\overrightarrow{s}$, base measure $h(\overrightarrow{r})$, and log-partition function $A(\overrightarrow{s})$ (Wainwright and Jordan, 2008).

The stimulus-conditional probability distribution $P(\overrightarrow{t}\mid \overrightarrow{s})$ can be defined by marginalizing over all $\overrightarrow{r}$ that map to the same $\overrightarrow{t}$:

$$\begin{gathered} P(\overrightarrow{t}\mid \overrightarrow{s})=\sum _{\overrightarrow{r}}\delta (\overrightarrow{t},\overrightarrow{t}(\overrightarrow{r}))h(\overrightarrow{r})\text{\hspace{0.17em}exp}(\overrightarrow{s}\cdot \overrightarrow{t}(\overrightarrow{r})-A(\overrightarrow{s})) \\ =\text{exp}(\overrightarrow{s}\cdot \overrightarrow{t}-A(\overrightarrow{s}))\sum _{\overrightarrow{r}}\delta (\overrightarrow{t},\overrightarrow{t}(\overrightarrow{r}))h(\overrightarrow{r}) \\ =\text{exp}(\overrightarrow{s}\cdot \overrightarrow{t}-A(\overrightarrow{s}))h(\overrightarrow{t}). \end{gathered}$$ (17)

Note that $h\left({\overrightarrow{t}}^{\prime }\right)=0$ if there does not exist an $\overrightarrow{r}$ such that ${\overrightarrow{t}}^{\prime }=\overrightarrow{t}(\overrightarrow{r})$. An important property of sufficient statistics is the conservation of information (Cover and Thomas, 2012):

$$I(\overrightarrow{S},\overrightarrow{R})=I_{\text{vector}}(\overrightarrow{S},\overrightarrow{T})$$ (18)

with $\overrightarrow{T}$ defined by Eq. (14). Although $\overrightarrow{T}$ does not lose information relative to $\overrightarrow{r}$, it is worth making a few comments on $\overrightarrow{T}$ and Eq. (18). Because $\overrightarrow{R}$ is a discrete variable (with cardinality of at most 2^N) and $\overrightarrow{T}$ is a deterministic function of $\overrightarrow{R}$, then $\overrightarrow{T}$ is also a discrete variable with finite cardinality. Indeed, outside of cases of degeneracy between the columns of W, there will generally be a one-to-one mapping between values of $\overrightarrow{R}$ and $\overrightarrow{T}$. Thus, even in cases where $D\ll N$, computing $H(\overrightarrow{T})$ can be just as difficult as computing $H(\overrightarrow{R})$. Furthermore, unlike $\overrightarrow{R}$, the components of $\overrightarrow{T}$ will generally not be conditionally independent. $H(\overrightarrow{T}\mid \overrightarrow{s})$ will thus be similarly intractable. While it may seem that we have not gained any computational advantage by transforming from $\overrightarrow{R}$ to $\overrightarrow{T}$ we will now show that Eq. (18) can be expressed in a convenient form that facilitates several useful approximations.

4.2. Decomposition of mutual information based on sufficient statistics

We start by noting that the ordering of the components of $\overrightarrow{S}$ and $\overrightarrow{T}$ is arbitrary, because applying any matching permutation to the components of $\overrightarrow{S}$ and $\overrightarrow{T}$ does not affect $I(\overrightarrow{S},\overrightarrow{R})$. We will use the following notations for components of vectors $\overrightarrow{s}=\left(s_1,\dots ,s_D\right):s_{\neg d}=\left(s_1,\dots ,s_{d-1},s_{d+1},\dots s_D\right)$, s_<d = (s₁, …, s_d−1), and similarly for s_>d, s_≥d, and s_≤d. Note that S_¬d = (S_<d,S_>d). Additionally, we will at times consider information theoretic quantities involving variables that are the concatenation of two other variables, such as X and Y. Such compound variables will be denoted as {X, Y}. Using these notations and applying the chain rule for mutual information to Eq. (18) yields: (Cover and Thomas, 2012):

$$I_{\text{vector}}(\overrightarrow{S},\overrightarrow{T})=\sum _{d=1}^{D}I\left(S_d,\overrightarrow{T}\mid S_{<d}\right).$$ (19)

In Eq. (19), $I\left(S_d,\overrightarrow{T}\mid S_{<d}\right)$ is the mutual information between $\overrightarrow{T}$ and S_d conditioned on S_<d.

$$\begin{gathered} I\left(S_d,\overrightarrow{T}\mid S_{<d}\right)=H\left(S_d,S_{<d}\right)+H\left(\overrightarrow{T},S_{<d}\right)-H\left(S_d,\overrightarrow{T},S_{<d}\right)-H\left(S_{<d}\right) \\ =\text{TC}\left(S_{<d},S_d,\overrightarrow{T}\right)-I\left(S_{<d},S_d\right)-I\left(S_{<d},\overrightarrow{T}\right) \\ =I\left(\overrightarrow{T},\left\{S_{<d},S_d\right\}\right)-I\left(\overrightarrow{T},S_{<d}\right)=I\left(\overrightarrow{T},S_{\le d}\right)-I\left(\overrightarrow{T},S_{<d}\right) \\ ={\langle I\left(S_d,\overrightarrow{T}\mid s_{<d}\right)\rangle }_{s_{<d}}, \end{gathered}$$ (20)

where $TC\left(S_{<d},S_d,\overrightarrow{T}\right)$ is the total correlation between S_<d, S_d, and $\overrightarrow{T}$. All four formulations of $I\left(S_d,\overrightarrow{T}\mid S_{<d}\right)$ are equivalent provided they all exist. A notable situation is when there is functional dependence between S_d and S_<d, such as when the support of S_≤d lies on a manifold of intrinsic dimension < d. In this case I(S_<d,S_d) diverges and the second line of Eq. (20) is ill-defined. However, it easy to show that $I\left(S_d,\overrightarrow{T}\mid S_{<d}\right)=0$ if such a functional dependency exists using the fourth line of Eq. (20). Formally, let s_d ≡ g(s_<d) where $g\left(s_{<d}\right):{\mathcal{R}}^{d-1}\to \mathcal{R}$ is a function defined explicitly or implicitly. Then we note that the mutual information between two variables is zero if at least one variable is constant:

$$I\left(S_d,\overrightarrow{T}\mid s_{<d}\right)=I\left(g\left(s_{<d}\right),\overrightarrow{T}\mid s_{<d}\right)=0.$$ (21)

Thus, $I\left(S_d,\overrightarrow{T}\mid S_{<d}\right)$ will also be zero:

$$I\left(S_d,\overrightarrow{T}\mid S_{<d}\right)={\langle I\left(S_d,\overrightarrow{T}\mid s_{<d}\right)\rangle }_{s_{<d}}={\langle I\left(g\left(s_{<d}\right),\overrightarrow{T}\mid s_{<d}\right)\rangle }_{s_{<d}}=0.$$ (22)

Computing just $I\left(S_d,\overrightarrow{T}\mid s_{<d}\right)$ remains challenging for the same reasons as computing $I_{\text{vector}}(\overrightarrow{S},\overrightarrow{T})$. However, we can achieve a further reduction by taking advantage of the fact that $P(\overrightarrow{t}\mid \overrightarrow{s})$ is an exponential family. To see this we first express two important marginalized forms of (14):

$$P\left(\overrightarrow{t}\mid s_d,s_{<d}\right)=h(\overrightarrow{t})\text{\hspace{0.17em}exp}\left(s_{<d}\cdot t_{<d}+s_d{t}_d\right){\langle \text{exp}\left(s_{>d}\cdot t_{>d}-A(\overrightarrow{s})\right)\rangle }_{s>d\mid s\le d}.$$ (23)

$$P\left(\overrightarrow{t}\mid s_{<d}\right)=h(\overrightarrow{t})\text{\hspace{0.17em}exp}\left(s_{<d}\cdot t_{<d}\right){\langle \text{exp}\left(s_{\ge d}\cdot t_{\ge d}-A(\overrightarrow{s})\right)\rangle }_{s\ge d}\mid s_{<d}.$$ (24)

The notation ${\langle f(\overrightarrow{s})\rangle }_{s_{>d}\mid s_{\le d}}$ denotes the expectation of $f(\overrightarrow{s})$ with respect to P(s_>d|s_≤d), with analogous meanings for ${\langle f(\overrightarrow{s})\rangle }_{s_{\ge d}\mid s_{<d}}$ and so forth. Marginalization over conditioned variables is expressed implicitly: $P\left(\overrightarrow{t}\mid s_{<d}\right)={\langle P(\overrightarrow{t}\mid \overrightarrow{s})\rangle }_{s_{\ge d}}$. The important consequence of (23) and (24) is that the log-likelihood ratio of $P\left(\overrightarrow{t}\mid s_d,s_{<d}\right)$ and $P\left(\overrightarrow{t}\mid s_{<d}\right)$ is independent of t_<d. From this we can show that $I\left(S_d,\overrightarrow{T}\mid s_{<d}\right)=I\left(S_d,T_{\ge d}\mid s_{<d}\right)$:

$$\begin{gathered} I\left(S_d,\overrightarrow{T}\mid s_{<d}\right)={\langle \sum _{\overrightarrow{t}}P\left(\overrightarrow{t}\mid s_d,s_{<d}\right)\text{\hspace{0.17em}log}\left(\frac{P\left(\overrightarrow{t}\mid s_d,s_{<d}\right)}{P\left(\overrightarrow{t}\mid s_{<d}\right)}\right)\rangle }_{s_d} \\ ={\langle \sum _{t\ge d}P\left(t_{\ge d}\mid s_d,s_{<d}\right)\text{\hspace{0.17em}log}\left(\frac{P\left(t_{\ge d}\mid s_d,s_{<d}\right)}{P\left(t_{\ge d}\mid s_{<d}\right)}\right)\rangle }_{s_d}. \\ =I\left(S_d,T_{\ge d}\mid s_{<d}\right) \end{gathered}$$ (25)

This leads to our next reduction:

$$I_{\text{vector}}(\overrightarrow{S},\overrightarrow{T})=\sum _{d=1}^{D}I\left(S_d,T_{\ge d}\mid S_{<d}\right).$$ (26)

We note that the d^th term in (19) has D+d+1 degrees of freedom, whereas the corresponding term in (26) has D+1 degrees of freedom. This effective dimension reduction has important algorithmic implications for the nonparametric estimators we use to compute the individual terms of (26) (c.f. Section 5). In section 5.1 and 5.2, we use Eq. (26) to evaluate $I_{\text{vector}}(\overrightarrow{S},\overrightarrow{T})$.

4.3. Lower bounds for the mutual information

While Eq. (26) represents a significant improvement in complexity over naive evaluation of $I(\overrightarrow{S},\overrightarrow{T})$, individual terms of $I_{\text{vector}}(\overrightarrow{S},\overrightarrow{T})$ may be still too high dimensional to reliably evaluate. In this section, we will present a series of lower bounds on $I_{\text{vector}}(\overrightarrow{S},\overrightarrow{T})$ that are more easily estimated. In particular we consider bounds that arise by replacing T_≥d in the d^th term of Eq (26) by a lower dimensional, deterministic transformation of T_≥d denoted Z_d. Applying the Data Processing Inequality (DPI) to each term in Eq (26), will yield a lower bound for the mutual information. There are many possible lower bounds to $I_{\text{vector}}(\overrightarrow{S},\overrightarrow{T})$ of this form. We focus on a variable Z_d = {T_d, |T_>d|} where |T_>d| is the L₂-norm of T_>d. This leads to the following lower bound approximation to $I_{\text{vector}}(\overrightarrow{S},\overrightarrow{T})$

$$I_{\text{iso}}(\overrightarrow{S},\overrightarrow{T})=\sum _{d}I\left(S_d,\left\{T_d,\left|T_{>d}\right|\right\}\mid S_{<d}\right),$$ (27)

which we term isotropic. In Appendix B, we show that this approximation becomes exact in the asymptotic limit of large neural populations, meaning that $I_{\text{iso}}(\overrightarrow{S},\overrightarrow{T})=I_{\text{vector}}(\overrightarrow{S},\overrightarrow{T})$, when the stimulus distribution is isotropic and the distribution of receptive fields (RF) $\overrightarrow{w}$ across the population is such that $A(\overrightarrow{s})=A(\left|\overrightarrow{s}\right|)$. Notably, this is achieved when RFs uniform cover the stimulus space, meaning that $P(\overrightarrow{w})$ is described by an uncorrelated Gaussian distribution. For finite number of neurons, $A(\overrightarrow{s})$ will never be perfectly isotropic. However, for large populations (N ≫ 1) where the receptive fields $\overrightarrow{w}$ are drawn from an isotropic distribution and the distribution of α is independent of $\overrightarrow{w}$, $A(\overrightarrow{s})$ will become isotropic asymptotically as N → ∞, cf. appendix B.1 for further details. The analogue of approximation Eq. (27) for the case where RFs and $\overrightarrow{s}$ are described by a matching correlated Gaussian distribution is described in appendix B.2.

The next reduction we consider is to drop |T_>d| from each term of Eq. (27):

$$I_{\text{comp-cond}}(\overrightarrow{S},\overrightarrow{T})=\sum _{d=1}^{D}I\left(S_d,T_d\mid S_{<d}\right).$$ (28)

By the data-processing inequality, it again follows that $I_{\text{iso}}(\overrightarrow{S},\overrightarrow{T})\ge I_{\text{comp-cond}}(\overrightarrow{S},\overrightarrow{T})$. Overall, one obtains a series of bounds:

$$I_{\text{vector}}(\overrightarrow{S},\overrightarrow{T})\ge I_{\text{iso}}(\overrightarrow{S},\overrightarrow{T})\ge I_{\text{comp-cond}}(\overrightarrow{S},\overrightarrow{T}).$$ (29)

Our final, simplest approximation is to drop the conditioning on S_<d in each term of Eq. (28).

$$I_{\text{comp-ind }}(\overrightarrow{S},\overrightarrow{T})=\sum _{d=1}^{D}I\left(S_d,T_d\right).$$ (30)

We show in Appendix C, that this last approximation becomes exact in the case where neural populations split into independent sub-populations with orthogonal RFs between sub-populations. Mathematically, this corresponds to the case where both the stimulus distribution $P(\overrightarrow{s})$ and the function $A(\overrightarrow{s})$ factor in the same basis:

$$P(\overrightarrow{s})=\prod _{k=1}^{D}P\left(s_k^{\prime }\right),A(\overrightarrow{s})=\sum _{k=1}^{D}A\left(s_k^{\prime }\right).$$ (31)

In general, $I_{\text{comp-ind }}(\overrightarrow{S},\overrightarrow{T})$ may be greater or less than $I_{\text{comp-cond}}(\overrightarrow{S},\overrightarrow{T})$ (or $I_{\text{vector}}(\overrightarrow{S},\overrightarrow{T})$) (Renner and Maurer, 2002). However when $P(\overrightarrow{s})={\prod }_{d}P\left(s_d\right)$, the following additional inequality holds:

$$I_{\text{comp-cond}}(\overrightarrow{S},\overrightarrow{T})\ge I_{\text{comp-ind }}(\overrightarrow{S},\overrightarrow{T}).$$ (32)

To derive (32) we first note that we can decompose I(S_d, {T_d, S_<d}) (for d > 1) in two different ways:

$$\begin{gathered} I\left(S_d,\left\{T_d,S_{<d}\right\}\right)=I\left(S_d,T_d\mid S_{<d}\right)+I\left(S_d,S_{<d}\right) \\ =I\left(S_d,S_{<d}\mid T_d\right)+I\left(S_d,T_d\right) \end{gathered}$$ (33)

Equating the first and second lines of (33) we can rewrite the residual I(S_d,T_d|S_<d) − I(S_d,T_d):

$$I\left(S_d,T_d\mid S_{<d}\right)-I\left(S_d,T_d\right)=I\left(S_d,S_{<d}\mid T_d\right)-I\left(S_d,S_{<d}\right)$$ (34)

Though either side of (34) may be positive or negative in general, when we make the assumption that $P(\overrightarrow{s})$ factors across dimension, then I(S_d, S_<d) = 0. Thus (34) is nonnegative and I(S_d, T_d|S_<d) ≥ I(S_d, T_d), implying (32).

In the opposite extreme case where the value of S_d is a deterministic function of S_<d, Eq. (22) can be generalized to show that I(S_d,T_d|S_<d) = 0. Thus, in this case I(S_d,T_d) ≥ I(S_d,T_d|S_<d), which in turn indicates that $I_{\text{comp-ind }}(\overrightarrow{S},\overrightarrow{T})\ge I_{\text{comp-cond}}(\overrightarrow{S},\overrightarrow{T})$. For example, when the support of $P(\overrightarrow{s})$ lies on a one-dimensional curve, e.g. $\overrightarrow{S}$ represents position along a one-dimensional nonlinear track, S_d is fully determined from the values of other variables S_<d ∀d regardless of component ordering. In this case, $I_{\text{comp-ind }}(\overrightarrow{S},\overrightarrow{T})\ge I_{\text{comp-cond}}(\overrightarrow{S},\overrightarrow{T})$.

In the intermediate cases with some statistical dependencies between stimulus components, $I_{\text{comp-ind }}(\overrightarrow{S},\overrightarrow{T})$ is not generally guaranteed to be a lower bound to either $I_{\text{comp-cond}}(\overrightarrow{S},\overrightarrow{T})$ or $I_{\text{vector}}(\overrightarrow{S},\overrightarrow{T})$. Nevertheless, we observed that even for some correlated $P(\overrightarrow{s})I_{\text{comp-ind }}(\overrightarrow{S},\overrightarrow{T})<I_{\text{comp-cond}}(\overrightarrow{S},\overrightarrow{T})$, c.f. section 5.2.

4.4. Alternative Approximations of $I(\overrightarrow{R},\overrightarrow{S})$

Previous authors have proposed other approximations to the mutual information. There exists a non-parametric upper bound to the mutual information computed in terms of pairwise relative entropies between $P(\overrightarrow{r}\mid \overrightarrow{s})$ and $P\left(\overrightarrow{r}\mid {\overrightarrow{s}}^{\prime }\right)$ (Haussler et al., 1997; Kolchinsky et al., 2017):

$$I_{\text{k}-\text{w}}(\overrightarrow{R},\overrightarrow{S})=-\int d\overrightarrow{s}P(\overrightarrow{s})\text{\hspace{0.17em}log}\left(\int d{\overrightarrow{s}}^{\prime }P\left({\overrightarrow{s}}^{\prime }\right)\text{\hspace{0.17em}exp}\left(-D_{KL}\left(P(\overrightarrow{r}\mid \overrightarrow{s})\Vert P\left(\overrightarrow{r}\mid {\overrightarrow{s}}^{\prime }\right)\right)\right)\right.$$ (35)

The model we consider for $P(\overrightarrow{r}\mid \overrightarrow{s})$ is an exponential family and thus has a tractable relative entropy (Banerjee et al., 2005):

$$D_{KL}\left(P(\overrightarrow{r}\mid \overrightarrow{s})\left|\right|P\left(\overrightarrow{r}\mid {\overrightarrow{s}}^{\prime }\right)=\sum _n\left(\text{tanh}\left(f_n(\overrightarrow{s})\right)f_n(\overrightarrow{s})-A_n(\overrightarrow{s})\right)-\left(\text{tanh}\left(f_n(\overrightarrow{s})\right)f_n\left({\overrightarrow{s}}^{\prime }\right)-A_n\left({\overrightarrow{s}}^{\prime }\right)\right)\right.$$ (36)

In Eq. (35) we have used the generalized definitions of Section 3. The evaluation of the upper bound (35) is quadratic in the sample size N_stim as opposed to O(N_stim log² N_stim) for the estimator in section 5. In the limit where N ≫ D, another popular approximation exist based on Fisher information (Brunel and Nadal, 1998); it can be computed with O(N_stim) operations. Recent work has shown that this approximation is valid only for certain classes of input distributions (Huang and Zhang, 2018). In appendix D we discuss the relationship between this approximation and $I_{\text{Fisher }}(\overrightarrow{R},\overrightarrow{S})$. We include numerical comparisons between $I_{\text{k}-\text{w}}(\overrightarrow{R},\overrightarrow{S})$ and the methods proposed in this paper in sections 5.1 and 5.2. However, we found that the Fisher Information approximation drastically overestimated the true mutual information. Therefore to avoid obscuring differences between other results, the approximation based on Fisher Information is not included in Figures 2–3. Full plots including this approximation can be found in Appendix D.

Figure 2:

Information curves for neural populations with uncorrelated RF and stimulus distributions. Lines and error bars are mean and standard deviation over ten repeats of the estimator. Insets show RF distribution for several population sizes.

Figure 3:

Information curves for neural populations with correlated RF distributions, cf. section 5.2. Lines and errorbars are mean and standard deviation over ten repeats of the estimator. In all panels, circles represent information computed in the original basis, while squares and triangles are computations performed in decorrelated basis. I_vector (A) and I_iso (B) recover full information. These two computations do not benefit from working in the decorrelated stimulus basis. Stimulus decorrelation improves the performanceof I_comp-cond (C) and I_comp-ind (D)). In (D), computations are invariant to ordering of components.

We note that there are other variational approximations to mutual information (Belghazi et al., 2018; Barber and Agakov, 2003). However, because comparing the information for different choices of the parameters of $P(\overrightarrow{r}\mid \overrightarrow{s})$ and $P(\overrightarrow{s})$ requires training a different variational approximation each time, direct comparison requires substantial computational resources and we leave them for future work.

5. Numerical Simulations

We now test the performance of the above described bounds under several representative situations that include correlated and uncorrelated stimulus distributions, and isotropic and anisotropic receptive field distributions, including experimentally recorded receptive fields from the primary visual cortex, as well as the case where intrinsic “noise” correlations are present.

To empirically estimate the bounds on mutual information [I(S_d, T_d), I(S_d, T_d|S_<d), I(S_d, {T_d, |T_>d|}|S_<d), and I(S_d, T_≥d|S_<d)], we use the KSG estimator (Kraskov et al., 2004), a non-parametric method based on distributions of K nearest-neighbor distances. We chose to use the KSG estimator because even though we have reduced the mutual information between two high-dimensional variables into a sum over pairs of scalars, computing even just I(S_d, T_d) can still be a daunting task, even more so for terms involving conditional informations. T_d may still have exponentially large cardinality, and complicated interdependencies between components of $\overrightarrow{T}$ present difficulties in forming explicit expressions for P(t_d|s_d), so exact evaluation of H(T_d) and H(T_d|S_d) is not feasible at present. The KSG estimator requires only that we can draw N_stim samples of $\overrightarrow{S}$ and $\overrightarrow{T}$ from $P(\overrightarrow{s},\overrightarrow{t})$, discarding the unused components. Sampling from $P(\overrightarrow{s},\overrightarrow{t})$ is easily done given samples from $P(\overrightarrow{s})$. Given a sample $\overrightarrow{s}$, we draw $\overrightarrow{r}$ from $P(\overrightarrow{r}\mid \overrightarrow{s})$, which is easily done because of Eq. (2), and transform $\overrightarrow{r}$ into $\overrightarrow{t}$ using (14). This estimator has complexity O(N_stim log² N_stim) when implemented with KD-Trees. For the case of two scalar variables the ℓ₂ error of the estimate decreases like $1/\sqrt{N_{\text{stim }}}$ (Gao et al., 2018), though if the true value of the mutual information is very high then the error may still be large (Gao et al., 2015). In order to partially alleviate this error, we use the PCA based Local Nonuniformity Correction of (Gao et al., 2015) (KSG-LNC). We extend this estimator to compute the conditional mutual information terms using a decomposition analogous to the second line of Eq. (20), and set the nonuniformity threshold hyperparameter according to the heuristics suggested in (Gao et al., 2015). Additionally, we assume that the distribution of $\overrightarrow{S}$ (and thus S_≤d, S_<d, and S_d ∀d) is non-atomic. Thus, because $\overrightarrow{T}$ is discrete but real valued, the KSG estimator is applicable as neither $\overrightarrow{T}$ nor $\overrightarrow{S}$ is a mixed continuous-discrete variable (Gao et al., 2017).

5.1. Large populations responding to uncorrelated stimuli

We evaluated the performance of the bounds on information developed in section 4.3 for large populations ranging from N ≈ 100 to 1, 000. Specifically, to test the performance of $I_{\text{iso}}(\overrightarrow{S},\overrightarrow{T})$ we chose a highly isotropic population and stimulus distribution. We set D = 3, M = 8, 000, and let $P(\overrightarrow{s})$ be a zero mean gaussian with unit covariance matrix. For each value of N, the ${\overrightarrow{w}}_n$ were placed uniformly on the surface of the unit sphere, using the regular placement algorithm of (Deserno, 2004). Because N is too large for exact evaluation of $H(\overrightarrow{r})$ ground truth values were estimated using the Monte Carlo estimator $\widehat{I}(\overrightarrow{R},\overrightarrow{S})$ of section 3 with B = 3. Results are plotted in Figure 2. We find that for large N, $I_{\text{iso}}(\overrightarrow{S},\overrightarrow{T})$ tightly approximates $I_{\text{vector}}(\overrightarrow{S},\overrightarrow{T})$ and both are accurate approximations to $\widehat{I}(\overrightarrow{R},\overrightarrow{S})$, strongly outperforming $I_{\text{k}-\text{w}}(\overrightarrow{R},\overrightarrow{S})$. We note that for this case the upper bound of log(N_stim) = log(8, 000) ≈ 9 (nats) is well above all of the curves other than $I_{\text{k}-\text{w}}(\overrightarrow{R},\overrightarrow{S})$, which is already known to be an upper bound to $I(\overrightarrow{R},\overrightarrow{S})$, demonstrating that we are in the well-sampled regime. Once again we see that inequalities (29) and (32) hold.

5.2. Correlated stimulus distributions

We now consider the case of correlated Gaussian stimuli. and model $P(\overrightarrow{s})$ as a zero-mean Gaussian with a full-rank non-diagonal covariance matrix C. To better understand the effects of stimulus correlations we also perform computations in stimulus bases where components are independent. For this, we decompose C as C = VΛV^T where V is an orthogonal matrix whose columns are the eigenvectors of C.

$$\widehat{S}=V^T\overrightarrow{S},\widehat{T}=V^T\overrightarrow{T}.$$ (37)

Note that we have $\widehat{t}\cdot \widehat{s}=\overrightarrow{t}\cdot \overrightarrow{s}$. It is easy to see that $P(\widehat{t}\mid \widehat{s})$ is also an exponential family. Additionally because the mappings from $\overrightarrow{s}$ to $\widehat{s}$ and $\overrightarrow{t}$ to $\widehat{t}$ are diffeomorphisms the information is preserved:

$$I_{\text{vector}}(\overrightarrow{S},\overrightarrow{T})=I_{\text{vector}}(\widehat{S},\widehat{T}).$$ (38)

We note that while Eq. (38) holds in principle, in practice we may see variation as the KSG family of estimators is not invariant to under diffeomorphisms. Importantly we also note that Eq. (25) holds for ($\widehat{S}$, $\widehat{T}$). Given samples from $P(\overrightarrow{s},\overrightarrow{t})$, we automatically have samples from $P(\widehat{s},\widehat{t})$. We can straightforwardly generalize $I_{\text{vector}}(\overrightarrow{S},\overrightarrow{T})$, $I_{\text{comp-cond}}(\overrightarrow{S},\overrightarrow{T})$, $I_{\text{comp-ind }}(\overrightarrow{S},\overrightarrow{T})$, and $I_{\text{iso}}(\overrightarrow{S},\overrightarrow{T})$, to $I(\widehat{S},\widehat{T})$, $I_{\text{comp-cond}}(\widehat{S},\widehat{T})$, $I_{\text{comp-ind}}(\widehat{S},\widehat{T})$, and $I_{\text{iso}}(\widehat{S},\widehat{T})$ respectively. Eq. (38) does not generalize to I_iso, I_comp-cond, or I_comp-ind as they are not expressible as mutual information quantities between two variables. We note that $I(\overrightarrow{R},\overrightarrow{S})=I(\overrightarrow{R},\widehat{S})$ so we do not modify $\widehat{I}(\overrightarrow{R},\overrightarrow{S})$.

Simulations in Figure 3 were done using the following stimulus covariance matrix

$$C=\left(\begin{array}{ccc}1.74716093& 1.3103707& 0.87358046\\ 1.3103707& 1.74716093& 1.3103707\\ 0.87358046& 1.3103707& 1.74716093\end{array}\right).$$ (39)

For this choice of C ρ_1,2 = ρ_2,3 = 0.75, ρ_1,3 = 0.5, and |C| = 1. The covariance matrix of $\widehat{S}$ is diagonal:

$$\widehat{C}=\left(\begin{array}{ccc}0.283& 0& 0\\ 0& 0.868& 0\\ 0& 0& 4.032\end{array}\right).$$ (40)

The receptive field configurations are the same as in Section 5.1. We note that in the $\overrightarrow{s}$ coordinates, all components have the same variance, whereas this symmetry is broken in the decorrelated components $\widehat{s}$. We compared sorting the components of $\widehat{s}$ in increasing and decreasing order of variance (triangles and squares respectively) in Figure 3A–C. Component order does not matter for $I_{\text{comp-ind }}(\widehat{S},\widehat{T})$, Figure 3D. We find that for both I_vector and I_iso it is optimal to perform computation in the original basis, with both quantities accurately matching $\widehat{I}(\overrightarrow{R},\overrightarrow{S})$. For I_comp-cond and I_comp-ind accuracy is increased by using decorrelated components and, for I_comp-cond, sorting components by decreasing variance.

5.3. Highly asymmetric receptive field distributions

Next, we consider a small population (N = 9) with a highly asymmetric distribution of redundant receptive fields in two stimulus dimensions. In particular we are interested in a population where many different configurations of $\overrightarrow{R}$ map to the same configuration of $\overrightarrow{T}$, demonstrating the utility of using $\overrightarrow{T}$ as a non-trivial sufficient statistic of $\overrightarrow{R}$. With this in mind, we chose a heavily redundant configuration of $\left\{{\overrightarrow{w}}_n\right\}:{\overrightarrow{w}}_n=(0,1)(n=1,2,3)$, ${\overrightarrow{w}}_n=(1,0)(n=4,5,6)$, ${\overrightarrow{w}}_n=(1,1)(n=7,8,9)$. The cardinalities of $\overrightarrow{R}$, $\overrightarrow{T}$, T₁ and T₂ are 512, 37, 7, and 7 respectively. Because N is small, ground truth values of $I(\overrightarrow{R},\overrightarrow{S})$ were computed by exactly evaluating $P(\overrightarrow{r}\mid \overrightarrow{s})\forall \overrightarrow{r}\in {\left\{-1,1\right\}}^N$, for every sample of $\overrightarrow{s}$. Given $P(\overrightarrow{r}\mid \overrightarrow{s})$ we average across $\overrightarrow{s}$ to get $P(\overrightarrow{r})$ explicitly, and calculate $H(\overrightarrow{R})$, $H(\overrightarrow{R}\mid \overrightarrow{S})$, and $I(\overrightarrow{R},\overrightarrow{S})$ from these distributions. $P(\overrightarrow{s})$ is gaussianly distributed with diagonal covariance matrix, in accordance with (32). We set N_stim = 10, 000. To test how the relative values of I(S₁,T₁) and I(S₂,T₂) impact the optimal ordering of components in $I_{\text{comp-cond}}(\overrightarrow{S},\overrightarrow{T})$ we fixed σ₂ = 1 and varied σ₁ between 0.5 and 2.5. Results are plotted in Figure 4. As predicted, the hierarchy of bounds (29) and (32) holds for all σ₁. It is also notable that in this case, just like in the case of large neural populations, for computing $I_{\text{comp-cond}}(\overrightarrow{S},\overrightarrow{T})$, it always seems best to start the information computation with the stimulus component that has the largest variance. As expected, the ordering or components does not strongly impact $I_{\text{vector}}(\overrightarrow{S},\overrightarrow{T})$.

Figure 4:

Information curves for the example population with highly redundant RFs from Sec. 5.3. Lines and errorbars are mean and standard deviation over ten repeats of the estimator. Although neither the component-conditional nor the component information are guaranteed to tightly approximate the information, both provide good approximation to the full information (as estimated via unbiased Monte Carlo method), reaching values within ≥ 80% of the maximum. For the vector-sufficient statistic, both component orderings accurately reproduced the full information. The next best approximation to the full information is provided by the component-conditional computation with components added in the order from largest to smallest variance. This approximation reaches accuracy within 95% of the full value over the range of neural population sizes.

5.4. Experimental stimuli and receptive fields

In the previous three experimental sections we considered synthetic distributions of low-dimensional stimuli and artificial configurations of receptive fields. In this section we analyze a population of model neurons with receptive fields and α values that were fit to responses of primary visual cortex neurons (V1) elicited by natural stimuli (Sharpee et al., 2006). We use 147 pairs of $\left({\overrightarrow{w}}_n,{\alpha }_n\right)$ values that were fit using the Maximally Informative Dimension (MID) algorithm as in (Sharpee et al., 2006). Stimuli are 10 pixel by 10 pixel patches extracted from the same set of images used to fit the model parameters. Receptive fields are normalized and centered on the patch, and we chose a 10 × 10 sub-patch of the original 32 × 32 shaped receptive fields so that all dimensions are well sampled by receptive fields. That is, for all pixels of the 10 × 10 patch, at least 115 of the 147 neurons have a nonzero value in the corresponding component of their receptive field. Additionally, the stimuli are z-scored by subtracting the mean and dividing by the standard deviation, with both quantities computed across all samples and pixels collectively.

Because of high stimulus dimensionality we could only compute the $I_{\text{comp-ind }}(\overrightarrow{S},\overrightarrow{T})$ bound (via the KSG estimator) and the ground truth information (via the Monte Carlo method). Because the pixels of natural image patches are clearly not independent, we also computed I_comp-ind in two additional coordinate systems. The first coordinate system is simply the linearly decorrelated components $\widehat{S}$ and $\overrightarrow{T}$ defined in Eq. (37). The second coordinate systems uses independent components derived using independent component analysis on $\overrightarrow{S}$

$$\tilde{S}=U\overrightarrow{S},\tilde{T}={\left(U^{-1}\right)}^T\overrightarrow{T}$$ (41)

Here, U is an unmixing matrix computed using Infomax ICA (Bell and Sejnowski, 1997) on the samples of $\overrightarrow{S}$. As is done in (Bell and Sejnowski, 1997), U includes a linear whitening matrix. We note that in Eq. (41), $\overrightarrow{T}$ is multiplied by (U⁻¹)^T and not U because the ICA unmixing matrix is generally not orthogonal and we require that $\overrightarrow{t}\cdot \overrightarrow{s}=\tilde{t}\cdot \tilde{s}$. As before $I_{\text{vector}}(\overrightarrow{S},\overrightarrow{T})=I_{\text{vector}}(\widehat{S},\widehat{T})=I_{\text{vector}}(\tilde{S},\tilde{T})$. However, the same cannot be said for I_comp-ind expressed in different coordinate systems.

To evaluate the effect of using different coordinate systems to evaluate I_comp-ind for different population sizes we first ranked the 147 neurons in descending order by the information each neuron carried about the stimulus. We computed $I\left(R_n,\overrightarrow{S}\right)\forall n\in \left\{1,\dots ,147\right\}$, which is easily done exactly since R_n is a binary variable, and then sorted neurons so that $I\left(R_n,\overrightarrow{S}\right)>=I\left(R_{n+1},\overrightarrow{S}\right)\forall n$. We considered populations of size N = 60, 70, …, 140, where each population contained the first N neurons under the aforementioned ordering. For each value of N we computed $\widehat{I}(\overrightarrow{R},\overrightarrow{S})(B=3)$, $I_{\text{comp-ind }}(\overrightarrow{S},\overrightarrow{T})$, $I_{\text{comp-ind }}(\widehat{S},\widehat{T})$, and $I_{\text{comp-ind }}(\tilde{S},\tilde{T})$. We note that log(N_stim) = log(49, 152) ≈ 10.8 nats. Results are plotted in Figure 5.

Figure 5:

Information curves for populations based on experimentally recorded RFs and probed with D = 100 natural visual stimuli. The full information (solid black line) is computed via the Monte Carlo method and is compared to I_comp-ind approximations computed in two different bases: the PCA basis (blue dashed line) and the ICA basis (blue dotted line). We do not show the calculation in the original pixel basis, because it $I_{\text{comp-ind}}(\overrightarrow{S},\overrightarrow{T})$ is omitted as it was yielded values ~ 25 nats across the range of population sizes and obscured the other curves. Because of non-Gaussian statistics of natural stimuli, PCA components remain correlated, and as a result the approximation is no longer guaranteed to be a lower bound of the true information. Computation in the ICA basis respects the lower bound requirements, and achieves ≥ 75% of the full information across the range of population sizes.

We observe that both $I_{\text{comp-ind }}(\overrightarrow{S},\overrightarrow{T})$ and $I_{\text{comp-ind }}(\widehat{S},\widehat{T})$ overestimate the true information, especially $I_{\text{comp-ind }}(\overrightarrow{S},\overrightarrow{T})$. This overestimation occurs because stimulus components are not independent. By comparison, computation performed in the ICA basis, $I_{\text{comp-ind }}(\tilde{S},\tilde{T})$, lower bounds the mutual information for all N, achieving ≥ 75% of the full information across the range of population sizes.

5.5. Handling intrinsically correlated neurons

In order to simplify derivations, we assumed that the neural responses were independent after conditioning on $\overrightarrow{s}$. However, all of the analytic results in Section 4.2 can be extended to specific forms of intrinsic interneuronal correlation to allow for the presence of correlations in neural responses for a given stimulus $\overrightarrow{s}$. Formally, we modify the base measure $h(\overrightarrow{r})$ to include a pairwise coupling term:

$$h(\overrightarrow{r},J)=e^{{\sum }_{mn}{J}_{mn}{r}_m{r}_n-{\sum }_n{r}_n{\alpha }_n}.$$ (42)

In Eq. 42, J is a symmetric N × N matrix where J_mn describes the intrinsic coupling between the m^th and n^th neurons. In this case $P(\overrightarrow{r}\mid \overrightarrow{s},J)$ can still be written as an exponential family in a canonical form:

$$P(\overrightarrow{r}\mid \overrightarrow{s})=h(\overrightarrow{r},J)\text{exp}(\overrightarrow{s}\cdot \overrightarrow{t}(\overrightarrow{r})-A(\overrightarrow{s},J)),$$

$$A(\overrightarrow{s},J)=\text{log\hspace{0.17em}}\left(\sum _{\overrightarrow{r}}h(\overrightarrow{r},J)\text{\hspace{0.17em}exp}(\overrightarrow{s}\cdot \overrightarrow{t}(\overrightarrow{r}))\right).$$ (43)

The form of the sufficient statistic remains unchanged, though $A(\overrightarrow{s},J)$ generally lacks a closed form. Nevertheless, all of the decompositions, equalities, and inequalities in Section 4.2 require only for the exponential family to be in canonical form and remain valid.

We tested the accuracy of our approximations on a small (N = 10) population of intrinsically correlated neurons. Receptive fields are uniformly distributed on the unit circle, α_n = 0 ∀n, and $P(\overrightarrow{s})$ is a standard two-dimensional Gaussian (N_stim = 20, 000). Intrinsic coupling is set proportional to the overlap between receptive fields with a coupling strength J₀, the sign of which determines whether the intrinsic couplings perform stimulus decorrelation or error-correction (Tkačik et al., 2010).:

$$J_{mn}=J_0{\overrightarrow{w}}_m\cdot {\overrightarrow{w}}_n.$$ (44)

The algorithms of Sections 5 and 3 all depend on being able to sample easily from $P(\overrightarrow{r}\mid \overrightarrow{s})$. For large N and general J this is usually difficult, particularly for configurations of J that exhibit glassy dynamics. Additionally, evaluating Eq. (9) requires explicit knowledge of $A(\overrightarrow{s})$, though methods such as mean-field theory or the TAP approximation may be used to approximate $A(\overrightarrow{s})$ (Opper et al., 2001). Since this population is small, we evaluate the ground truth information exactly as in Section 5.3. Likewise, we sample $\overrightarrow{r}$ from $P(\overrightarrow{r}\mid \overrightarrow{s})$ exactly by computing all 2^N (1, 024) probabilities for every sample of $\overrightarrow{s}$. Analyzing the error introduced in using approximate sampling strategies such as Markov Chain Monte Carlo is left to future investigation. Results are plotted in Figure 6. As predicted, $I_{\text{vector}}(\overrightarrow{S},\overrightarrow{T})$ matches the ground truth values of the information. Similarly the hierarchy of bounds (29) and (32) is preserved, though for strongly negative couplings $I_{\text{comp-cond}}(\overrightarrow{S},\overrightarrow{T})\approx I_{\text{comp-ind }}(\overrightarrow{S},\overrightarrow{T})$. In sum, the presence of noise correlations does not invalidate the approximations and bounds that are derived above. However, numerical computation can become more difficult in the presence of noise correlations.

Figure 6:

Information curves for populations with intrinsic correlations. Lines and errorbars are mean and standard deviation over ten repeats of the estimator.

6. Conclusions and Future Work

We have presented three approximations that can be used to estimate the information transmitted by large neural populations. Each of these three approximations represents different trade-offs between accuracy and computational ease and feasibility. The best performance in terms of accuracy was provided by the isotropic approximation I_iso. This approximation worked well even in cases where is not guaranteed to become asymptotically exact with increasing population size. For example, the isotropic approximation was derived assuming a matching covariance matrix for both the stimulus and RF distributions, cf. Appendix B. Yet, this approximation matched the full ground-truth information values even for correlated stimuli with RFs remaining uncorrelated across the neural population (Fig. 3.) This approximation provided the best overall performance among the bounds tested, consistent with the theoretically expected inequalities between these bounds, cf. Eq. (29).

The component-conditional information I_comp-cond offered the second-best performance. This approximation performed especially well when computed in the stimulus basis where stimulus components were not correlated. This approximation is less computationally difficult compared to I_iso, because each conditional information is evaluated between just two quantities S_d and T_d compared to S_d and a conjunction of T_d and |T_>d| as in I_iso. For this reason, the finite-sample bias of I_comp-ind can also be less than I_iso, because bias in the evaluation of the mutual information is usually larger for higher-dimensional calculations.

The last approximation I_comp-ind is the least accurate of the three approximation but is computationally the easiest. It is the only approximation among the three we considered here that we were able to implement in conjunction with high dimensional stimuli. This approximation becomes most accurate in the stimulus bases where stimulus components are independent. There is strong evidence that neural receptive fields are organized along the ICA components of natural stimuli (Bell and Sejnowski, 1997; Olshausen and Field, 2004; Smith and Lewicki, 2006). This raises the possibility that the approaches proposed here will fair well when applied to recorded neural responses. Indeed, we found that I_comp-ind ≥ 75% of the full information value for large neural populations constructed using experimentally recorded RFs and probed with natural stimuli.

At present, the main limitation for computing the conditional approximations I_comp-cond and I_comp-iso is not the number of neurons but rather the stimulus dimensionality. For stimulus distributions where P(s_≥d|s_<d) can be easily sampled from, such as Gaussian distributions, we can take advantage of the fourth line of Eq. (20) to compute unbiased estimates of I_comp-cond and I_iso, albeit with possibly high variance. Developing methods that can efficiently approximate these conditional computations represents an important opportunity for future research.

A. Bias of $\widehat{H}(\overrightarrow{R})$

In this section we give a self-contained proof that $\widehat{H}(\overrightarrow{R})$ systematically underestimates the ”true” entropy $H(\overrightarrow{R})$. We first assume that $P\left(\left\{{\overrightarrow{s}}_{\mu }\right\}\right)={\prod }_{\mu }P\left({\overrightarrow{s}}_{\mu }\right)$: The ${\overrightarrow{s}}_{\mu }$ are drawn independently from $P(\overrightarrow{s})$, whether $P(\overrightarrow{s})$ is a smooth density on ${\mathcal{R}}^D$ or some larger set of samples. We define a an empirical version of the marginal distribution on $P(\overrightarrow{r})$:

$$\widehat{P}\left(\overrightarrow{r}\mid \left\{{\overrightarrow{s}}_{\mu }\right\}\right)=\frac{1}{M}\sum _{\mu }P\left(\overrightarrow{r}\mid {\overrightarrow{s}}_{\mu }\right)$$ (45)

The true marginal distribution is the expected value of $\widehat{P}\left(\overrightarrow{r}\mid \left\{{\overrightarrow{s}}_{\mu }\right\}\right):{\langle \widehat{P}\left(\overrightarrow{r};\left\{{\overrightarrow{s}}_{\mu }\right\}\right)\rangle }_{P\left(\left\{{\overrightarrow{s}}_{\mu }\right\}\right)}=P(\overrightarrow{r})$. The Shannon entropy is a concave function of $\widehat{P}\left(\overrightarrow{r};\left\{{\overrightarrow{s}}_{\mu }\right\}\right)$, which can be considered a random vector in the 2^N dimensional probability simplex. Thus, by Jensen’s inequality we have the following:

$${\langle \widehat{H}(\overrightarrow{R})\rangle }_{P\left(\left\{{\overrightarrow{s}}_{\mu }\right\}\right)}\le H(\overrightarrow{R}:P=\langle \widehat{P}\rangle )\equiv H(\overrightarrow{R})$$ (46)

This bias holds even in the case of evaluating $\widehat{H}(\overrightarrow{R})$ through exact enumeration. We note that we are able to produce unbiased estimates of $\widehat{H}(\overrightarrow{R})$ because we have full access to $P\left(\overrightarrow{r}\mid {\overrightarrow{s}}_{\mu }\right)$: We can evaluate $P\left(\overrightarrow{r}\mid {\overrightarrow{s}}_{\mu }\right)$ explicitly and deterministically, and thus $\widehat{F}(\overrightarrow{r})$ as well (up to factors of numerical precision). If we were always constrained to drawing samples from $P\left(\overrightarrow{r}\mid {\overrightarrow{s}}_{\mu }\right)$, then we would once again be limited to making biased estimates of $\widehat{H}(\overrightarrow{R})$ (Paninski, 2003).

B. On the asymptotic tightness $I_{\text{iso}}(\overrightarrow{S},\overrightarrow{T})$

Consider a large population (N ≫ 1) where the distribution of $\overrightarrow{w}$ and α is such that $A(\overrightarrow{s})=A(\left|\overrightarrow{s}\right|)$ (in some sense to be made more precise later). Consider the likelihood ratio in the definition of (25):

$$\begin{gathered} \frac{P\left(t_{\ge d}\mid s_d,s_{<d}\right)}{P\left(t_{\ge d}\mid s_{<d}\right)}=\frac{{\langle \text{exp}\left(s_d{t}_d+s_{>d}\cdot t_{>d}-A(\overrightarrow{s})\right)\rangle }_{s_{>d}\mid s_{\le d}}}{{\langle \text{exp}\left(s_d{t}_d+s_{>d}\cdot t_{>d}-A(\overrightarrow{s})\right)\rangle }_{s_{\ge d}\mid s_{<d}}} \\ =\frac{{\langle \text{exp}\left(s_d{t}_d+s_{>d}\cdot t_{>d}-A(\left|\overrightarrow{s}\right|)\right)\rangle }_{s_{>d}\mid s_{\le d}}}{{\langle \text{exp}\left(s_d{t}_d+s_{>d}\cdot t_{>d}-A(\left|\overrightarrow{s}\right|)\right)\rangle }_{s_{\ge d}\mid s_{<d}}} \end{gathered}$$ (47)

Additionally we consider a stimulus distribution that is similarly isotropic so that the conditional distribution P(s_>d|s_≤d) can be written in a convenient factored form:

$$P(\overrightarrow{s})=P(\left|\overrightarrow{s}\right|)=P\left(\sqrt{s_{<d}^2+s_d^2+s_{>d}^2}\right)$$

$$P\left(s_{>d}\mid s_{\le d}\right)=\frac{P(\overrightarrow{s})}{P\left(s_{\le d}\right)}=\frac{P\left(\sqrt{s_{<d}^2+s_d^2+s_{>d}^2}\right)}{P\left(s_{\le d}\right)}$$

We will show that in this situation, we can replace T_≥d in (25) with the variable that is the concatenation of T_d and |T_>d| without loss of information:

$$I\left(S_d,T_{\ge d}\mid s_{<d}\right)\approx I\left(S_d,\left\{T_d,\left|T_{>d}\right|\right\}\mid s_{<d}\right)$$ (48)

To show this using the Fisher-Neyman factorization theorem, it suffices to show that the numerator and denominator in (47) can be factored as follows:

$$\begin{gathered} {\langle \text{exp}\left(s_d{t}_d+s_{>d}\cdot t_{>d}-A(\left|\overrightarrow{s}\right|)\right)\rangle }_{s_{>d}\mid s_{\le d}}=g_1\left(s_{<d},s_d,t_d,\left|t_{>d}\right|\right)g_2\left(t_{>d}\right) \\ {\langle \text{exp}\left(s_d{t}_d+s_{>d}\cdot t_{>d}-A(\left|\overrightarrow{s}\right|)\right)\rangle }_{s\ge d}\mid s_{<d}=f_1\left(s_{<d},t_d,\left|t_{>d}\right|\right)f_2\left(t_{>d}\right) \end{gathered}$$ (49)

With the requirement that f₂(t_>d) = g₂(t_>d), so that dependence on t_>d cancels out in (47). We note that the first line of (49) implies the second so we examine that term in more detail.

$$\begin{gathered} {\langle \text{exp}\left(s_d{t}_d+s_{>d}\cdot t_{>d}-A(\left|\overrightarrow{s}\right|)\right)\rangle }_{s_{>d}\mid s_{\le d}}=\text{exp}\left(s_d{t}_d\right)\int \text{exp\hspace{0.17em}}\left(s_{>d}\cdot t_{>d}-A\left(\sqrt{s_{<d}^2+s_d^2+s_{>d}^2}\right)\right)P\left(s_{>d}\mid s_{\le d}\right)d{s}_{>d} \\ =\frac{\text{exp}\left(s_d{t}_d\right)}{P\left(s_{\le d}\right)}\int \text{exp\hspace{0.17em}}\left(s_{>d}\cdot t_{>d}-A\left(\sqrt{s_{<d}^2+s_d^2+s_{>d}^2}\right)\right)P\left(\sqrt{s_{<d}^2+s_d^2+s_{>d}^2}\right)d{s}_{>d}. \end{gathered}$$ (50)

We note that s_>d is a K = D − d dimensional vector. We assume that d < D − 1, so that K > 1, otherwise no further reduction of (50) is possible. We convert the integral over ${\mathcal{R}}^K$ in (50) into spherical coordinates and break it into three parts: Integration over ρ ∈ [0, ∞) where |s_>d| = ρ; integration over θ ∈ [0, π] where s_>d ·t_>d = ρ|t_>d|cos(θ), and φ ∈ Ω_K−1 is the set of all directions in ${\mathcal{R}}^K$ with constant θ. The integrand of (50) doesn’t depend on φ so we can integrate over it automatically, yielding a constant B_K that is a function only of K. We can now restate (50) in these coordinates:

$$\begin{gathered} \cdots =\frac{B_K\text{\hspace{0.17em}exp}\left(s_d{t}_d\right)}{P\left(s_{\le d}\right)}{\int }_0^{\infty }{\int }_0^{\pi }d\rho d\theta {\rho }^{K-1}{\text{\hspace{0.17em}sin}}^{K-2}(\theta )P\left(\sqrt{s_{<d}^2+s_d^2+{\rho }^2}\right)\text{\hspace{0.17em}exp\hspace{0.17em}}\left(\rho \left|t_{>d}\right|\text{\hspace{0.17em}cos}(\theta )-A\left(\sqrt{s_{<d}^2+s_d^2+{\rho }^2}\right)\right) \\ =\frac{B_K\text{\hspace{0.17em}exp}\left(s_d{t}_d\right)}{P\left(s_{\le d}\right)}{\int }_0^{\infty }d\rho {\rho }^{K-1}P\left(\sqrt{s_{<d}^2+s_d^2+{\rho }^2}\right)\text{\hspace{0.17em}exp\hspace{0.17em}}\left(-A\left(\sqrt{s_{<d}^2+s_d^2+{\rho }^2}\right)\right){\int }_0^{\pi }d\theta {\text{\hspace{0.17em}sin}}^{K-2}(\theta )\text{\hspace{0.17em}exp\hspace{0.17em}}\left(\rho \left|t_{>d}\right|\text{\hspace{0.17em}cos}(\theta )\right) \end{gathered}$$ (51)

We next evaluate the integral over θ in (51).

$${\int }_0^{\pi }d\theta {\text{\hspace{0.17em}sin}}^{K-2}(\theta )\text{\hspace{0.17em}exp}\left(\rho \left|t_{>d}\right|\text{\hspace{0.17em}cos}(\theta )\right)=\sqrt{\pi }\frac{\Gamma \left(\frac{K}{2}-\frac{1}{2}\right)}{\Gamma \left(\frac{K}{2}\right)}{}_0{F}_1\left(\frac{K}{2},\frac{{\rho }^2{\left|t_{>d}\right|}^2}{4}\right)\equiv F_K\left(\rho \left|t_{>d}\right|\right)$$ (52)

Where Γ(x) is the Gamma function, and ₀F₁(a, z) is the confluent hypergeometric limit function. We have our final expression for the first term in (49):

$$\begin{gathered} \cdots =\frac{B_K\text{\hspace{0.17em}exp}\left(s_d{t}_d\right)}{P(s_{\le d})}{\int }_0^{\infty }d\rho {\rho }^{K-1}P\left(\sqrt{s_{<d}^2+s_d^2+{\rho }^2}\right)\text{\hspace{0.17em}exp\hspace{0.17em}}\left(-A\left(\sqrt{s_{<d}^2+s_d^2+{\rho }^2}\right)\right)F_K\left(\rho \left|t_{>d}\right|\right) \\ =\frac{B_K\text{\hspace{0.17em}exp}\left(s_d{t}_d\right)}{P(s_{\le d})}g\left(s_{<d},s_d,\left|t_{>d}\right|\right) \end{gathered}$$ (53)

By setting g₁ in (49) equal (53), and letting g₂ = f₂ = 1, we have established (48).

B.1. Approximating $A(\overrightarrow{s})$ for Gaussian $P(\overrightarrow{w})$

In the previous section we assumed that the distribution $P(\overrightarrow{w})$ and P(α) are such that $A(\overrightarrow{s})=A(\left|\overrightarrow{s}\right|)$. In the special case when N ≫ 1, P(α) = δ(α) and $\overrightarrow{w}$ are Gaussianly distributed we can approximate $A(\overrightarrow{s})$ in a semi-closed form. Let $P(\overrightarrow{w})$ be a zero-mean Gaussian with with positive-definite covariance matrix C:

$$\begin{gathered} A(\overrightarrow{s})=N\int d\overrightarrow{w}\frac{\text{exp\hspace{0.17em}}\left(-\frac{1}{2}{\overrightarrow{w}}^T{C}^{-1}\overrightarrow{w}\right)}{\sqrt{\text{det}(2\pi C)}}\text{log}(2\text{\hspace{0.17em}cosh}(\overrightarrow{w}\cdot \overrightarrow{s})) \\ =N\int dx\frac{\text{exp\hspace{0.17em}}\left(\frac{-x^2}{2{\sigma }_x^2}\right)}{\sqrt{2\pi {\sigma }_x^2}}\text{log}(2\text{\hspace{0.17em}cosh}(x)) \end{gathered}$$ (54)

$${\sigma }_x=\sqrt{{\overrightarrow{s}}^{T}C\overrightarrow{s}}$$ (55)

Where we have taken advantage of the fact that $\overrightarrow{w}\cdot \overrightarrow{s}$ is a scalar gaussian variable with standard deviation that depends on $\overrightarrow{s}$ and C. We next take an infinite series expansion of log(2cosh(x)).

$$\text{log}(2\text{\hspace{0.17em}cosh}(x))=\left|x\right|+\text{log}(1+\text{exp}(-2\left|x\right|))=\left|x\right|+\sum _{m=1}^{\infty }\frac{{(-1)}^{m+1}}{m}\text{exp}(-2m\left|x\right|)$$ (56)

As an aside, the first equality in (56) is a useful and numerically stable expression for A(x). The ”softplus” function l(y) = log(1 + exp(y)) is implemented in many scientific computing packages, and using this alternate form for A(x) sidesteps computing the hyperbolic cosine. We next take the appropriate Gaussian average of each term in (56):

$$\frac{1}{\sqrt{2\pi {\sigma }_x^2}}{\int }_{-\infty }^{\infty }dx\text{\hspace{0.17em}exp}\left(\frac{-x^2}{2{\sigma }_x^2}\right)\left|x\right|=\sqrt{\frac{2}{\pi }}{\sigma }_x$$ (57)

$$\frac{1}{\sqrt{2\pi {\sigma }_x^2}}{\int }_{-\infty }^{\infty }dx\text{\hspace{0.17em}exp}\left(\frac{-x^2}{2{\sigma }_x^2}-2m\left|x\right|\right)=\text{erfcx}\left(\sqrt{2}m{\sigma }_x\right)$$ (58)

Where erfcx(y) is the scaled complementary error function. Thus we have our final form for $A(\overrightarrow{s})$:

$$A(\overrightarrow{s})=N\sqrt{\frac{2}{\pi }}{\sigma }_x+N\sum _{m=1}^{\infty }\frac{{(-1)}^{m+1}}{m}\text{erfcx}\left(\sqrt{2}m{\sigma }_x\right)$$ (59)

We note that erfcx(y) monotonically decreases to zero, so for large values of $\sqrt{{\overrightarrow{s}}^{T}C\overrightarrow{s}}$, $A(\overrightarrow{s})$ is well approximated by the first term in (59). Regardless we see that $A(\overrightarrow{s})$ depends on $\overrightarrow{s}$ only through $\sqrt{{\overrightarrow{s}}^{T}C\overrightarrow{s}}$:

$$A(\overrightarrow{s})=A\left(\sqrt{{\overrightarrow{s}}^{T}C\overrightarrow{s}}\right)=A(|U\overrightarrow{s}|)$$ (60)

Where $U=C^{\frac{1}{2}}$ is the cholesky decomposition of C.

B.2. Generalizing $I_{\text{iso}}(\overrightarrow{S},\overrightarrow{T})$ for matched anisotropy

In this section we will show that a generalized form of $I_{\text{iso}}(\overrightarrow{S},\overrightarrow{T})$ will also asymptotically converge to $I_{\text{vector}}(\overrightarrow{S},\overrightarrow{T})$ when both $P(\overrightarrow{s})$ and $A(\overrightarrow{s})$ obey a certain form of ”matched” anisotropy. Specifically we assume that $P(\overrightarrow{s})$ and $A(\overrightarrow{s})$ depend on on $\overrightarrow{s}$ through a quadratic function of $\overrightarrow{s}$ with positive-definite kernel C.

$$P(\overrightarrow{s})=P\left(\sqrt{{\overrightarrow{s}}^{T}C\overrightarrow{s}}\right)=P(|U\overrightarrow{s}|)$$ (61)

$$A(\overrightarrow{s})=A\left(\sqrt{{\overrightarrow{s}}^{T}C\overrightarrow{s}}\right)=A(|U\overrightarrow{s}|)$$ (62)

Where, as in Section B.1, $U=C^{\frac{1}{2}}$ is the cholesky decomposition of C. Let us transformed versions of $\overrightarrow{S}$ and $\overrightarrow{T}$:

$$\tilde{S}=U\overrightarrow{S}$$ (63)

$$\tilde{T}=U^{-T}\overrightarrow{T}$$ (64)

Where U^−T is the transpose of the inverse of U, which is well defined since C was positive definite. We note that $\tilde{t}\cdot \tilde{s}=\overrightarrow{t}\cdot \overrightarrow{s}$ and $P(\tilde{t}\mid \tilde{s})$ can once again be written as an exponential family in canonical form:

$$P(\tilde{t}\mid \tilde{s})=\text{exp}(\tilde{s}\cdot \tilde{t}-A(\tilde{s}))h(\tilde{t})$$ (65)

As the mappings from $\overrightarrow{S}$ to $\tilde{S}$ and $\overrightarrow{T}$ to $\tilde{T}$ are diffeomorphisms, we have that $I_{\text{vector}}(\overrightarrow{S},\overrightarrow{T})=I_{\text{vector}}(\tilde{S},\tilde{T})$. Furthermore, equation (65) implies that an analogous form of equation (26) holds for $I_{\text{vector}}(\tilde{S},\tilde{T})$:

$$I\left({\tilde{S}}_d,\tilde{T}\mid {\tilde{S}}_{<d}\right)=I\left({\tilde{S}}_d,{\tilde{T}}_{\ge d}\mid {\tilde{S}}_{<d}\right)$$ (66)

Additionally, $A(\tilde{s})=A(|\tilde{s}|)$ and $P(\tilde{s})=P(|\tilde{s}|)$. Thus, we may reuse the derivation of section 48 to derive the following analogy of equation (48):

$$I\left({\tilde{S}}_d,{\tilde{T}}_{\ge d}\mid {\tilde{s}}_{<d}\right)\approx I\left({\tilde{S}}_d,\left\{{\tilde{T}}_d,\left|{\tilde{T}}_{>d}\right|\right\}\mid {\tilde{s}}_{<d}\right)$$ (67)

Therefore $I_{\text{iso}}(\tilde{S},\tilde{T})$ is asymptotically equal to $I_{\text{vector}}(\overrightarrow{S},\overrightarrow{T})$:

$$I_{\text{iso}}(\tilde{S},\tilde{T})=\sum _{d}I\left({\tilde{S}}_d,\left\{{\tilde{T}}_d,\left|{\tilde{T}}_{>d}\right|\right\}\mid {\tilde{S}}_{<d}\right)\approx I_{\text{vector}}(\overrightarrow{S},\overrightarrow{T})$$ (68)

We note that an example of such a matched isotropy situation would be where both the stimuli and receptive fields (for a large population) are distributed according to a Gaussian distribution with covariance matrix C (c.f. section B.1).

C. Independent Sub-populations

In this section we present an example where one of our proposed approximations, $I_{\text{comp-ind }}(\overrightarrow{S},\overrightarrow{T})$ in this case, is equal to $I_{\text{vector}}(\overrightarrow{S},\overrightarrow{T})$. Let $\left({\widehat{e}}_1^{\prime },\dots ,{\widehat{e}}_D^{\prime }\right)$ be an orthonormal basis for ${\mathcal{R}}^D$. Suppose that the distribution of $\overrightarrow{s}$ and $\overrightarrow{w}$ are such that both $P(\overrightarrow{s})$ and $A(\overrightarrow{s})$ factor when expressed in this basis $\left(s_k^{\prime }=\overrightarrow{s}\cdot {\widehat{e}}_k^{\prime }\right)$:

Similarly defining $t_k^{\prime }=\overrightarrow{t}\cdot {\widehat{e}}_k^{\prime }$ we have that $\overrightarrow{t}\cdot \overrightarrow{s}={\overrightarrow{t}}^{\prime }\cdot {\overrightarrow{s}}^{\prime }$. Because mutual information is invariant under bijective transformations of the variables (e.g. a change of basis) (Cover and Thomas, 2012) we have that $I_{\text{vector}}(\overrightarrow{S},\overrightarrow{T})=I\left({\overrightarrow{S}}^{\prime },{\overrightarrow{T}}^{\prime }\right)$. It is easy to show that $P\left({\overrightarrow{t}}^{\prime }\mid {\overrightarrow{s}}^{\prime }\right)$ can be written as follows:

$$P\left({\overrightarrow{t}}^{\prime }\mid {\overrightarrow{s}}^{\prime }\right)=h\left({\overrightarrow{t}}^{\prime }\right)\prod _k\text{exp}\left(s_k^{\prime }{t}_k^{\prime }-A\left(s_k^{\prime }\right)\right)$$ (69)

Eq (69) implies that the log-likelihood ratio of $P\left({\overrightarrow{t}}^{\prime }\mid {\overrightarrow{s}}^{\prime }\right)$ to $P\left({\overrightarrow{t}}^{\prime }\right)$ decomposes across $s_k^{\prime }$:

$$\text{log\hspace{0.17em}}\left(\frac{P\left({\overrightarrow{t}}^{\prime }\mid {\overrightarrow{s}}^{\prime }\right)}{P\left({\overrightarrow{t}}^{\prime }\right)}\right)=\sum _k\text{log\hspace{0.17em}}\left(\frac{P\left(t_k^{\prime }\mid s_k^{\prime }\right)}{P\left(t_k^{\prime }\right)}\right)$$ (70)

Thus we have the following reduction of $I_{\text{vector}}(\overrightarrow{S},\overrightarrow{T})$:

$$I_{\text{vector}}(\overrightarrow{S},\overrightarrow{T})=I_{\text{vector}}\left({\overrightarrow{S}}^{\prime },{\overrightarrow{T}}^{\prime }\right)=\sum _{k}I\left(S_k^{\prime },T_k^{\prime }\right)=I_{\text{comp-ind }}\left({\overrightarrow{S}}^{\prime },{\overrightarrow{T}}^{\prime }\right)$$ (71)

We note that (31) includes the case where for some k, ${\overrightarrow{w}}_n\cdot {\widehat{e}}_k^{\prime }=0\forall n$. In such a case $A\left(s_k^{\prime }\right)=A(0)=\text{log}(2)$, $t_k^{\prime }=0$ with probability one, and $I\left(S_k^{\prime },T_k^{\prime }\right)=0$. Thus, in the case of independent subpopulations, $I_{\text{vector}}(\overrightarrow{S},\overrightarrow{T})$ can be reduced to computing $I_{\text{comp-ind }}(\overrightarrow{S},\overrightarrow{T})$ following a change of basis.

D. Relationship between $I_{\text{k}-\text{w}}(\overrightarrow{R},\overrightarrow{S})$ and $I_{\text{Fisher}}(\overrightarrow{R},\overrightarrow{S})$

In this appendix we relate $I_{\text{k}-\text{w}}(\overrightarrow{R},\overrightarrow{S})$ to the Fisher Information based approximation of (Brunel and Nadal, 1998):

$$I_{\text{Fisher }}(\overrightarrow{R},\overrightarrow{S})=H(\overrightarrow{S})+\frac{1}{2}\int d\overrightarrow{s}P(\overrightarrow{s})\text{\hspace{0.17em}log\hspace{0.17em}}\left(\frac{|J(\overrightarrow{s})|}{{(2\pi e)}^D}\right)$$ (72)

Where $J(\overrightarrow{s})$ is the Fisher Information Matrix of $P(\overrightarrow{r}\mid \overrightarrow{s})$:

$$J_{ab}(\overrightarrow{s})={〈\frac{{\partial }^2}{{\partial }_a{\partial }_b}\text{log\hspace{0.17em}}P(\overrightarrow{r}\mid \overrightarrow{s})〉}_{P(\overrightarrow{r}\mid \overrightarrow{s})}$$ (73)

We begin by considering the inner expectation over ${\overrightarrow{s}}^{\prime }$ in $I_{\text{k}-\text{w}}(\overrightarrow{R},\overrightarrow{S})$:

$$L(\overrightarrow{s})=\int d{\overrightarrow{s}}^{\prime }P\left({\overrightarrow{s}}^{\prime }\right)\text{\hspace{0.17em}exp\hspace{0.17em}}\left(-D_{KL}\left(P(\overrightarrow{r}\mid \overrightarrow{s})\Vert P\left(\overrightarrow{r}\mid {\overrightarrow{s}}^{\prime }\right)\right)\right.$$ (74)

We next assume that the activation function $f_n(\overrightarrow{s})$ is affine (e.g. (13)), and thus in canonical form. We also assume that, $P(\overrightarrow{r}\mid \overrightarrow{s})$ is identifiable:

$$D_{KL}\left(P(\overrightarrow{r}\mid \overrightarrow{s})\Vert P\left(\overrightarrow{r}\mid {\overrightarrow{s}}^{\prime }\right)=0\iff \overrightarrow{s}={\overrightarrow{s}}^{\prime }\right.$$ (75)

For (13) a necessary and sufficient condition for identifiability is that the matrix W has full rank, a reasonable assumption when N ≫ D. We utilize the following properties of exponential families in canonical form:

1. $\frac{{\partial }^2}{{\partial }_a^{\prime }{\partial }_b^{\prime }}{D}_{KL}\left(P(\overrightarrow{r}\mid \overrightarrow{s})\Vert P\left(\overrightarrow{r}\mid {\overrightarrow{s}}^{\prime }\right)=\frac{{\partial }^2}{{\partial }_a^{\prime }{\partial }_b^{\prime }}A\left({\overrightarrow{s}}^{\prime }\right)=J\left({\overrightarrow{s}}^{\prime }\right)\right.$.

2. $A\left({\overrightarrow{s}}^{\prime }\right)$ is convex and $J\left({\overrightarrow{s}}^{\prime }\right)$ is positive semi-definite. When $P(\overrightarrow{r}\mid \overrightarrow{s})$ is identifiable replace $A\left({\overrightarrow{s}}^{\prime }\right)$ becomes strictly convex, $J\left({\overrightarrow{s}}^{\prime }\right)$ positive definite, and $D_{KL}\left(P(\overrightarrow{r}\mid \overrightarrow{s})\Vert P\left(\overrightarrow{r}\mid {\overrightarrow{s}}^{\prime }\right)\right.$ has a global minimum with respect to ${\overrightarrow{s}}^{\prime }$ of 0 at ${\overrightarrow{s}}^{\prime }=\overrightarrow{s}$.

In the limit N ≫ D we approximate $L(\overrightarrow{s})$ using Laplace’s Method (Bender and Orszag, 1999), expanding around ${\overrightarrow{s}}^{\prime }=\overrightarrow{s}$:

$$L(\overrightarrow{s})\approx P(\overrightarrow{s})\sqrt{\frac{{(2\pi )}^D}{|J(\overrightarrow{s})|}}$$ (76)

Plugging (76) into the definition of $I_{\text{k}-\text{w}}(\overrightarrow{R},\overrightarrow{S})$ we have the following asymptotic expression for $I_{\text{k}-\text{w}}(\overrightarrow{R},\overrightarrow{S})$:

$$\begin{gathered} I_{\text{k}-\text{w}}(\overrightarrow{R},\overrightarrow{S})\approx H(\overrightarrow{s})+\frac{1}{2}\int d\overrightarrow{s}P(\overrightarrow{s})\text{\hspace{0.17em}log\hspace{0.17em}}\left(\frac{|J(\overrightarrow{s})|}{{(2\pi )}^D}\right) \\ =I_{\text{Fisher}}(\overrightarrow{R},\overrightarrow{S})-\frac{D}{2} \end{gathered}$$ (77)

For stimulus distributions where the entropy $H(\overrightarrow{S})$ is known a priori, such as the Gaussian distributions in sections 5.1 and 5.2, $I_{\text{Fisher}}(\overrightarrow{R},\overrightarrow{S})$ can be computed in O(M) time. If not, then $H(\overrightarrow{S})$ must be estimated, a challenging task in high dimensions. In figure 7, we replot the results of sections 5.1 with the inclusion of $I_{\text{Fisher }}(\overrightarrow{R},\overrightarrow{S})$. We see that $I_{\text{Fisher }}(\overrightarrow{R},\overrightarrow{S})$ is a very loose upper bound of $I(\overrightarrow{R},\overrightarrow{S})$ and of $I_{\text{k}-\text{w}}(\overrightarrow{R},\overrightarrow{S})$, indicating that the convergence of Laplace’s Method may be very slow in this situation.

Figure 7:

Information curves for the population of section 5.1 compared to Fisher approximation. Lines and errorbars are mean and standard deviation over ten repeats of the estimator. $\widehat{I}(\overrightarrow{R},\overrightarrow{S})$ (solid black line), $I_{\text{k}-\text{w}}(\overrightarrow{R},\overrightarrow{S})$ (dashed black line), $I_{\text{vector}}(\overrightarrow{S},\overrightarrow{T})$ (solid yellow line), $I_{\text{Fisher }}(\overrightarrow{R},\overrightarrow{S})$ (dotted black line), $I_{\text{comp-cond}}(\overrightarrow{S},\overrightarrow{T})$ (solid red line), $I_{\text{comp-ind }}(\overrightarrow{S},\overrightarrow{T})$ (solid blue line), $I_{\text{iso}}(\overrightarrow{S},\overrightarrow{T})$ (solid green line)

E. Extension to polynomial activation functions

In section 4.1 we assumed that the activation functions were affine functions of the stimulus vector $\overrightarrow{s}$. In this appendix we will show how to generalize some of the results of section 4.1 to polynomial activation functions. For clarity of exposition we will demonstrate this generalization for quadratic functions as the procedure for higher order polynomials follows quickly. To begin, we add a quadratic term to Eq (12):

$$f_n(\overrightarrow{s})={\overrightarrow{s}}^T{\gamma }_n\overrightarrow{s}+{\overrightarrow{w}}_n\cdot \overrightarrow{s}-{\alpha }_n$$ (78)

${\gamma }_n\in {\mathcal{R}}^{D\times D}$ is a symmetric D × D matrix representing the quadratic kernel of the n^th neuron’s activation function. We note that ${\overrightarrow{s}}^T{\gamma }_n\overrightarrow{s}=\overrightarrow{s}{\overrightarrow{s}}^T◦{\gamma }_n$ where A ◦ B is the hadamard product between equally shaped matrices A and B and $\overrightarrow{s}{\overrightarrow{s}}^T\in {\mathcal{R}}^{D\times D}$ is the outer product of $\overrightarrow{s}$ with itself. We define a vector embedding of $\overrightarrow{s}$ into ${\mathcal{R}}^{D+D^2},\overrightarrow{\psi }(\overrightarrow{s})$:

$$\psi {(\overrightarrow{s})}_d=\left\{\begin{array}{ll}{s}_d\hfill & \text{ if }d\le D\hfill \\ s_a*s_b\hfill & \text{ if }d>D\hfill \end{array}\right.$$

Where a = d mod D and b = ⎿d/D⏌ are index mappings that map a D × D matrix into a vector of length D². We define a similar vector embedding of ${\overrightarrow{w}}_n$ and γ_n, ${\overrightarrow{\tau }}_n\left({\overrightarrow{w}}_n,{\gamma }_n\right)$:

$${\tau }_n{\left({\overrightarrow{w}}_n,{\gamma }_n\right)}_d=\left\{\begin{array}{ll}{w}_{n,d}\hfill & \text{ if }d\le D\hfill \\ {\gamma }_{n,ab}\hfill & \text{ if }d>D\hfill \end{array}\right.$$

For the sake of brevity we henceforth omit the dependence of ${\overrightarrow{\tau }}_n$ on ${\overrightarrow{w}}_n$ and γ_n. By construction we have the following equivalence:

$${\overrightarrow{s}}^T{\gamma }_n\overrightarrow{s}+{\overrightarrow{w}}_n\cdot \overrightarrow{s}={\overrightarrow{\tau }}_n\cdot \overrightarrow{\psi }(\overrightarrow{s})$$ (79)

If all neurons have activation functions of the form in (78) then $P(\overrightarrow{r}\mid \overrightarrow{s})$ may once again be written as an exponential family

$$P(\overrightarrow{r}\mid \overrightarrow{s})=h(\overrightarrow{r})\text{\hspace{0.17em}exp}\left({\overrightarrow{t}}^{\text{quad}}(\overrightarrow{r})\cdot \overrightarrow{\psi }(\overrightarrow{s})-A(\overrightarrow{s})\right)$$

$$A(\overrightarrow{s})=\sum _n\text{log}\left(2\text{\hspace{0.17em}cosh}\left({\overrightarrow{\tau }}_n\cdot \overrightarrow{\psi }(\overrightarrow{s})-{\alpha }_n\right)\right)$$

$${\overrightarrow{t}}^{\text{quad}}(\overrightarrow{r})=\sum _n{\overrightarrow{\tau }}_n{r}_n$$ (80)

As ${\overrightarrow{t}}^{\text{quad}}(\overrightarrow{r})\in {\mathcal{R}}^{D+D^2}$ is the sufficient statistic for this family, and $\overrightarrow{\psi }(\overrightarrow{s})$ is the natural parameter. As before, $I(\overrightarrow{R},\overrightarrow{S})=I({\overrightarrow{T}}^{\text{quad}},\overrightarrow{S})$. However, we note that $P(\overrightarrow{r}\mid \overrightarrow{S})$ can be written entirely in terms of $\overrightarrow{\psi }$. Additionally, we note that the support of $\overrightarrow{\psi }$ lies on a D-dimensional manifold in ${\mathcal{R}}^{D+D^2}$ and $\overrightarrow{s}$ maps injectively into this manifold. Thus $I({\overrightarrow{T}}^{\text{quad}},\overrightarrow{S})=I({\overrightarrow{T}}^{\text{quad}},\overrightarrow{\Psi })$.

We note several properties of $I({\overrightarrow{T}}^{\text{quad}},\overrightarrow{\Psi })$. First, we can in principle expand $I({\overrightarrow{T}}^{\text{quad}},\overrightarrow{\Psi })$ like Eq. (19):

$$I\left({\overrightarrow{T}}^{\text{quad}},\overrightarrow{\Psi }\right)=\sum _{d=1}^{d=D+D^2}I\left({\overrightarrow{T}}^{\text{quad}},{\Psi }_d\mid {\Psi }_{<d}\right)$$ (81)

Secondly, the same reduction as Eq. (26) holds for $I\left({\overrightarrow{T}}^{\text{quad}},{\Psi }_d\mid {\Psi }_{<d}\right)$:

$$I\left({\overrightarrow{T}}^{\text{quad}},{\Psi }_d\mid {\Psi }_{<d}\right)=I\left(T_{\ge d}^{\text{quad}},{\Psi }_d\mid {\Psi }_{<d}\right)$$ (82)

Most notably however, is that $I\left({\overrightarrow{T}}^{\text{quad}},{\Psi }_d\mid {\Psi }_{<d}\right)=I\left(T_{\ge d}^{\text{quad}},{\Psi }_d\mid {\Psi }_{<d}\right)=0$ for d > D. This holds because ψ_d = g(ψ_≤D) for d > D, where g(ψ_≤D) is just the product of the two relevant components of ψ_≤D. Because of this functional dependence we can just apply the generalization of Eq. (22). Therefore the expansion of $I({\overrightarrow{T}}^{\text{quad}},\overrightarrow{\Psi })$ can be truncated after D terms.

$$I\left({\overrightarrow{T}}^{\text{quad}},\overrightarrow{\Psi }\right)=\sum _{d=1}^{d=D}I\left(T_{\ge d}^2,{\Psi }_d\mid {\Psi }_{<d}\right)=\sum _{d=1}^{d=D}I\left(T_{\ge d}^{\text{quad}},S_d\mid S_{<d}\right)$$ (83)

In fact, we can make an even stronger reduction by noting that conditioning on components of $\overrightarrow{S}={\psi }_{\le D}$ effectively also conditions on elements of ψ_>D. For clarity of exposition we break down ${\overrightarrow{T}}^{\text{quad}}$ into vector and matrix valued components:

$${\overrightarrow{T}}^{\text{quad}}\equiv \left\{T_{\le D},T_{\le D}^{\le D}\right\}$$

$$T_{\le D}^{\text{quad}}\to T_{\le D}\in {\mathcal{R}}^D$$

$$T_{>D}^{\text{quad}}\to T_{\le D}^{\le D}\in {\mathcal{R}}^{D\times D}$$

We note that conditioning on S_d conditions on the components of $\overrightarrow{\psi }$ corresponding to T_d and $T_d^d$. Additionally conditioning on S_<d conditions on the components of T_<d and on the components of $T_{d_1}^{d_2}$ for all indices d₁ and d₂ such that 1 ≤ d₁,d₂ < d. Thus Eq. (83) can be further generalized:

$$I\left({\overrightarrow{T}}^{\text{quad}},\overrightarrow{S}\right)=\sum _{d=1}^{d=D}I\left(\left\{T_{\ge d},T_{\ge d}^{\ge d}\right\},S_d\mid S_{<d}\right)$$ (84)

The d^th term in Eq (83) has D²+D+1 degrees of freedom while the d^th term in Eq (84) has D²+D+1−(d−1)² degrees of freedom. The above procedure can be generalized to polynomial activation functions of arbitrarily high but finite order, though the dimensionality of the sufficient statistic and natural parameter grow exponentially with the order. However, Eq. (84) holds for any order of polynomial, so that one one needs only compute the first D terms of the expansion of the mutual information between the sufficient statistic and natural parameter.