Information Loss Associated with Imperfect Observation and Mismatched Decoding

Abstract

We consider two types of causes leading to information loss when neural activities are passed and processed in the brain. One is responses of upstream neurons to stimuli being imperfectly observed by downstream neurons. The other is upstream neurons non-optimally decoding stimuli information contained in the activities of the downstream neurons. To investigate the importance of neural correlation in information processing in the brain, we specifically consider two situations. One is when neural responses are not simultaneously observed, i.e., neural correlation data is lost. This situation means that stimuli information is decoded without any specific assumption about neural correlations. The other is when stimuli information is decoded by a wrong statistical model where neural responses are assumed to be independent even when they are not. We provide the information geometric interpretation of these two types of information loss and clarify their relationship. We then concretely evaluate these types of information loss in some simple examples. Finally, we discuss use of these evaluations of information loss to elucidate the importance of correlation in neural information processing.

Introduction

Neurons in early sensory areas represent the information of various stimuli from the external world by their noisy activities. The noise inherent in neural activities needs to be properly handled by the nervous system for the information to be accurately processed. One simple but powerful means for coping with the neural noise is population coding. Neurophysiological experiments have shown that many neurons with different selectivities respond to particular stimuli. These findings suggest that the nervous system represents information through population activities, which would be helpful for accurate information processing. This coding scheme for stimulus information is known as population coding.

An important feature of population coding is that the activity of neurons is correlated (Gray et al., 1989; Gawne and Richmond, 1993; Zohary et al., 1994; Meister et al., 1995; Lee et al., 1998; Ishikane et al., 2005; Averbeck et al., 2006; Ohiorhenuan et al., 2010; but see Ecker et al., 2010). A crucial question is how much information the nervous system can extract from correlated population activities. In general, it is difficult for the nervous system to maximally extract information when its activities are correlated because two conditions must be satisfied. First, downstream neurons must perfectly observe the responses of the upstream neurons (perfect observation of neural responses). Second, downstream neurons must optimally decode the information from the observed neural responses (optimal decoding of stimulus). In other words, if either the observation or the decoding is imperfect or non-optimal, which are both likely situations in the nervous system, stimuli information is inevitably degraded. In this work, we discuss the amount of information loss associated with imperfect observation and non-optimal decoding.

With regard to non-optimal decoding, several researchers including the authors of this paper have investigated how much information would be lost if neural correlation is ignored in decoding (Nirenberg et al., 2001; Wu et al., 2001; Golledge et al., 2003; Averbeck and Lee, 2006; Oizumi et al., 2010). This type of decoding is called mismatched decoding (Merhav et al., 1994; Oizumi et al., 2010) because the decoding does not match the actual neural activities, i.e., the actual neural activities are correlated but the correlations are ignored in decoding. With regard to imperfect observation, this work is the first to address this issue. As an example of imperfect observation, we specifically consider the situation that the activity of neurons is not simultaneously observed by downstream neurons (Figure 1). This is related to whether the coincidence detector plays an important role in information processing in the brain (Abeles, 1982; König et al., 1996). If a large proportion of the total information is lost when the responses of neurons are not simultaneously observed, coincidence detection would be necessary for accurate information processing.

Figure 1.

Schematic of simultaneous and non-simultaneous observation of neural responses.

The framework of mismatched decoding was introduced to quantify the importance of correlated activity because this importance could be quantified by the amount of information loss when neural correlation is ignored in decoding (Nirenberg et al., 2001; Wu et al., 2001; Nirenberg and Latham, 2003). Similarly, the importance of correlated activity could be also measured using the amount of information loss due to non-simultaneous observation. We therefore obtain two different measures for the importance of correlated activity by introducing the concepts of imperfect observation and mismatched decoding. We aim to clarify the relationship between the measures and provide a simple guide on how to use them.

Combining the concepts of imperfect observation and mismatched decoding produces four types of situations where stimulus is inferred: (1) observation is perfect and decoding is optimal; (2) decoding is optimal but observation is imperfect; (3) observation is perfect but decoding is mismatched; and (4) observation is imperfect and decoding is mismatched. We discuss the inferences in these four types of situations from the viewpoint of information geometry (Amari and Nagaoka, 2000) and then clarify their relationships. We also specifically compute the amount of information obtained through these four inference types for neural responses described by the Gaussian model and by a binary probabilistic model.

This paper is organized as follows. First, we introduce the concept of an exponential family and two probability distributions that belong to the exponential family, i.e., the Gaussian distribution and the binary probabilistic model, which have both been intensively investigated as representative models of neural responses (Abbott and Dayan, 1999; Amari, 2001; Nakahara and Amari, 2002). Second, we provide information geometric interpretation of the four types of inference mentioned above and describe how to evaluate the information in each of the four types by using the Fisher information. Third, we compute the amount of information in each of the four types of inference in the Gaussian model and explain the relationship between the inference with imperfect observation and that with mismatched decoding. Fourth, we also compute the amount of information obtained by the four types of inference in simple binary probabilistic models. Finally, we summarize the results and discuss how to use the two measures introduced in this study for quantifying the importance of neural correlation and mention some of the future directions of this work.

Exponential Family of Probability Distributions

Let us denote the conditional probability distribution for a neural response r = (r₁, r₂,…,r_N) over a population of N neurons being evoked by a stimulus s as p(r; s), where s is a continuous variable. We assume that p(r; s) belongs to the exponential family. Probability distributions that belong to an exponential family can be written as

$$p(r;s)=\exp \left({\displaystyle \sum _{I=1}^n{R}_I(r){\theta }_I(s)-\psi (s)}\right),$$ (1)

where R_I(r) is a function of neural responses r, θI(s) is a function of stimulus s, and a normalization constant ψ(s) is also a function of stimulus s. θI(s) is called the natural parameter of the exponential family. Two examples of probability distributions are investigated in this paper.

Example 1. Gaussian distribution

The number of spikes emitted by a neuron over a fixed time period (time-averaged rate) or the total number of spikes emitted by a population of neurons (population-averaged rate), which is denoted by r, may be described by the Gaussian distribution

$$p(r;s)=\frac{1}{\sqrt{{(2\pi )}^N\det [C(s)]}}\exp \left(-\frac{1}{2}{(r-f(s))}^T{C}^{-1}(s)(r-f(s))\right),$$ (2)

where N is the number of neurons, f(s) is the average number of spikes, and C(s) is the covariance matrix. If we rewrite Eq. 2 as follows, we can see that the Gaussian distribution belongs to the exponential family

$$\begin{array}{l}p(r;s)=\exp (C^{-1}(s)f(s)\cdot r-\frac{1}{2}{C}^{-1}(s)\cdot \overline{r}{\overline{r}}^T-\frac{1}{2}\log \det [C(s)]\\ -\frac{1}{2}{f}^T(s)C^{-1}f(s)).\end{array}$$ (3)

In this case,

$$R_1=\left\{r_i\right\}=\left\{r_1,\cdots ,r_N\right\},$$ (4)

$$R_2=\left\{r_i{r}_j\right\}=\left\{r_1{r}_1,r_1{r}_2,\cdots ,r_{N-1}{r}_N,r_N{r}_N\right\},$$ (5)

$${\theta }_1=\left\{{\displaystyle \sum _{j=1}^N{C}_{ij}^{-1}(s)f_j(s)}\right\},$$ (6)

$${\theta }_2=\left\{-\frac{1}{2}{C}_{ij}^{-1}(s)\right\},$$ (7)

$$\psi (s)=\frac{1}{2}\log \det [C(s)]+\frac{1}{2}{f}^T(s)C^{-1}f(s).$$ (8)

Example 2. Log-linear model of binary neural response

When we analyze neural responses within a short time period (∼1–10 ms, typically), the neural responses are considered stochastic binary variables: r_i = 1 when the ith neuron fires within the time bin, and r_i = 0 when it does not. The joint distributions of N random binary variables can be generally written in the following form (Amari, 2001; Nakahara and Amari, 2002):

$$\begin{array}{l}p(r;s)\\ =\exp \left({\displaystyle \sum _{i=1}^N{\theta }_i(s)r_i}+{\displaystyle \sum _{i<j}{\theta }_{ij}(s)r_i{r}_j+\cdots +{\theta }_{12\dots N}(s)r_1{r}_2\cdots r_N-\psi (s)}\right),\end{array}$$ (9)

where the normalization constant ψ(s) is given by

$$\psi (s)=-\log p\left(r_1=r_2=\cdots =r_N=0;s\right).$$ (10)

This probability distribution is clearly in the exponential family form. In this case,

$$\begin{array}{l}{R}_1=\left\{r_i\right\}=\left\{r_1,\cdots ,r_N\right\},\\ R_2=\left\{r_i{r}_j\right\}=\left\{r_1{r}_1,r_1{r}_2,\cdots ,r_{N-1}{r}_N,r_N{r}_N\right\},\\ \cdots ,\\ R_N=\left\{r_1{r}_2\cdots r_N\right\}\end{array}$$ (11)

$$\begin{array}{l}{\theta }_1=\left\{{\theta }_i(s)\right\},\\ {\theta }_2=\left\{{\theta }_{ij}(s)\right\},\\ \cdots ,\\ {\theta }_N=\left\{{\theta }_{12\cdots N}(s)\right\}.\end{array}$$ (12)

Recent investigations have shown that the observed statistics of neural responses can be sufficiently captured by this type of probabilistic model, which contains up to second-order correlation terms (Schneidman et al., 2006; Shlens et al., 2006; Tang et al., 2008), although the importance of higher-order correlations has also been discussed (Amari et al., 2003; Montani et al., 2009; Ohiorhenuan et al., 2010). For simplicity, we consider only the second-order correlations and ignore higher-order correlations in this work, i.e.,

$${\theta }_I=0,(\text{for\hspace{0.17em}}I>2).$$ (13)

Inference of Stimulus and Fisher Information

We consider the inference problem of how accurately the stimulus value s can be estimated when the stochastic neural response r is given. We assume that neural response r is observed many times. The neural response at the tth trial is denoted by r(t) and the number of trials by T. If each neural response is independent and identically distributed, the probability distribution for T observations of neural responses is given by

$$p_T\left(r^T;s\right)={\displaystyle \prod _{t=1}^{T}p(r(t);s)},$$ (14)

$$=\exp \left({\displaystyle \sum _{I=1}^2{\overline{R}}_I(r)\cdot {\theta }_I(s)-T\psi (s)}\right),$$ (15)

where ${\overline{R}}_I$ are sufficient statistics for the probability distribution and are given by

$${\overline{R}}_1=\left\{{\overline{r}}_i\right\}{\overline{r}}_i=\frac{1}{T}{\displaystyle \sum _{t=1}^T{r}_i(t)},$$ (16)

$${\overline{R}}_2=\left\{{\overline{r}}_{ij}\right\}{\overline{r}}_{ij}=\frac{1}{T}{\displaystyle \sum _{t=1}^T{r}_i(t)r_j(t)}.$$ (17)

We evaluate the accuracy of the estimate by using the Fisher information,

$$g(s)=-E_{p_T}\left[\frac{d^2{l}_s}{d{s}^2}\right]=-{\displaystyle \int d{r}^T{p}_T\left(r^T;s\right)\frac{d^2\log p_T(r^T;s)}{d{s}^2}},$$ (18)

where l_s = log p_T(r^T; s) and $E_{p_T}$ denotes the expectation with respect to the distribution p_T(r^T; s). Through the Cramér–Rao bound, the Fisher information bounds the average squared decoding error for an unbiased estimate as follows:

$$E_{p_T}\left[{(s-\stackrel{\wedge }{s})}^2\right]\ge \frac{1}{g(s)},$$ (19)

where s is the true stimulus value and $\widehat{s}$ is the estimate. Since the Fisher information is the lower bound of the mean square error, behavior of the mean square error and the Fisher information could be different in general (Brunel and Nadal, 1998; Yaeli and Meir, 2010). However, the maximum likelihood estimator, which chooses s for an estimate that maximizes likelihood function p_T(r^T; s), achieves the Cramér–Rao bound (Eq. 19) as T → ∞. A Bayesian estimator, which is generally a biased estimator, can also achieve the Cramér–Rao bound as T → ∞ because it becomes equivalent to the maximum likelihood estimator as T → ∞.

We compute the Fisher information with respect to stimulus s from the Fisher information matrix with respect to the natural parameters,

$$g_{IJ}(\theta )=E_{p_T}\left[{\partial }_I{l}_{\theta }{\partial }_J{l}_{\theta }\right],$$ (20)

$$=-E_{p_T}\left[{\partial }_I{\partial }_J{l}_{\theta }\right],$$ (21)

where l_θ = p_T(r^T; θ), and ∂_I = ∂/∂θ_I. From the definition of the Fisher information matrix with respect to the natural parameters (Eqs. 20 and 21), the natural parameters are given by

$$g_{IJ}(\theta )=E_{p_T}\left[\left({\overline{R}}_I-{\eta }_I\right)\left({\overline{R}}_J-{\eta }_J\right)\right],$$ (22)

$$=T{\partial }_I{\partial }_J\psi (\theta ),$$ (23)

where

$${\eta }_I=E_{p_T}\left[{\overline{R}}_I\right]=E_p\left[R_I\right].$$ (24)

By using g_IJ(θ), the Fisher information with respect to the stimulus s can be written as

$$g(s)={\displaystyle \sum _{I,J}{g}_{IJ}(\theta )\frac{\partial {\theta }_I}{\partial s}}\frac{\partial {\theta }_J}{\partial s}.$$ (25)

The Fisher information g(s) described above determines the accuracy of the estimate of s under two conditions: (1) all sufficient statistics ${\overline{R}}_I$ are available, and (2) the likelihood function p(r; s) is exactly known. Regarding the first condition, downstream neurons may not be able to simultaneously access the responses of all upstream neurons. This imperfect observation of neural responses by downstream neurons leads to loss of information. Similarly, regarding the second condition, downstream neurons are unlikely to completely know the likelihood function p(r; s). Downstream neurons are more likely to only partially know p(r; s) and to decode the stimulus based on a decoding model q(r; s), which is not equal to p(r; s) but partially matches p(r; s) (Nirenberg et al., 2001; Wu et al., 2001; Oizumi et al., 2010). This mismatched decoding of stimuli by downstream neurons also results in loss of information. These two types of information loss are evaluated next.

Information Loss Caused by Imperfect Observation of Neural Responses

In this work, we specifically consider the situation that second-order sufficient statistics ${\overline{R}}_2$ are not accessible to downstream neurons and only first-order sufficient statistics ${\overline{R}}_1$ are available to them. This is related to whether coincidence detector neurons are needed to accurately estimate the stimulus. To evaluate the loss of information associated with loss of data, we first marginalize the joint probability distribution $p({\overline{R}}_1,{\overline{R}}_2;s)$ over ${\overline{R}}_2:$

$$p\left({\overline{R}}_1;s\right)={\displaystyle \int d{\overline{R}}_{2}p\left({\overline{R}}_1,{\overline{R}}_2;s\right)}.$$ (26)

When only ${\overline{R}}_1$ is observed, the Fisher information with respect to stimulus s is given by

$$g^L(s)=-{\displaystyle \int d{\overline{R}}_{1}p\left({\overline{R}}_1;s\right)\frac{d^2\log p({\overline{R}}_1;s)}{d{s}^2}}.$$ (27)

The information loss associated with loss of ${\overline{R}}_2$ is

$$\Delta g=g(s)-g^L(s).$$ (28)

Information Loss Caused by Mismatched Decoding of Stimulus

We evaluate the loss of information when downstream neurons infer the stimulus parameter s based on not the correct probability distribution p(r; s) but a mismatched probability distribution q(r; s). We assume that q(r; s) also belongs to the exponential family and that the maximum likelihood estimation based on q(r; s) is consistent, i.e.,

$$E_p\left[\frac{d{l}_q(r;s)}{ds}\right]=0,$$ (29)

where E_p denotes the expectation with respect to p(r; s) and l_q(r; s) = log q(r; s). We evaluate the squared decoding error of the maximum likelihood estimation with the mismatched likelihood function q(r; s) based on T observations of neural responses, r^T. The estimate ${\widehat{s}}_q$ is given by

$${\widehat{s}}_q=\underset{s}{\arg \max }\hspace{0.17em}{l}_{q_T}\left(r^T;s\right),$$ (30)

$$=\underset{s}{\arg \max }{\displaystyle \sum _{t=1}^T{l}_q(r(t);s)}.$$ (31)

By differentiating l_qT(r^T; s) with respect to s at ${\widehat{s}}_q,$ we obtain the quasi-likelihood estimating equation

$${\displaystyle \sum _{t=1}^T\frac{d{l}_q\left(r(t);{\stackrel{\wedge }{s}}_q\right)}{ds}}=0.$$ (32)

When the left-hand side terms of Eq. 32 are expanded at the true value of stimulus parameter s,

$$\begin{array}{l}0={\displaystyle \sum _{t=1}^T\frac{d{l}_q\left(r(t);{\stackrel{\wedge }{s}}_q\right)}{ds}},\\ ={\displaystyle \sum _{t=1}^T\frac{d{l}_q\left(r(t);s\right)}{ds}}+{\displaystyle \sum _{t=1}^T\frac{d^2{l}_q\left(r(t);s\right)}{d{s}^2}}\left(s-{\stackrel{\wedge }{s}}_q\right).\end{array}$$ (33)

According to the central limit theorem, the first term on the right-hand side of Eq. 33 converges to a Gaussian distribution with mean 0 and variance TE_p[(dl_q(r; s)/ds)²] as T → ∞. From the weak law of large numbers, the coefficient of the second term on the right-hand side of Eq. 33 becomes

$${\displaystyle \sum _{t=1}^T\frac{d^2{l}_q(r(t);s)}{d{s}^2}}=T{E}_p\left[\frac{d^2{l}_q(r;s)}{d{s}^2}\right].$$ (34)

Taken together, we have

$$E_p\left[{\left({\widehat{s}}_q-s\right)}^2\right]=\frac{1}{T}\frac{E_p\left[{\left(\frac{d{l}_q(r;s)}{ds}\right)}^2\right]}{{\left(E_p\left[\frac{d^2{l}_q(r;s)}{d{s}^2}\right]\right)}^2}.$$ (35)

If we regard the inverse of the squared decoding error as the Fisher information, the Fisher information for mismatched decoding model q(r; s) is

$$g\ast (s)=T\frac{{\left(E_p\left[\frac{d^2{l}_q(r;s)}{d{s}^2}\right]\right)}^2}{E_p\left[{\left(\frac{d{l}_q(r;s)}{ds}\right)}^2\right]}.$$ (36)

Note that when q(r; s) = p(r; s), g*(s) = g(s).

Information Geometric Interpretations

We discuss the inference of stimulus when neural responses are only partially observed and that when a mismatched probability distribution is used for decoding, from the information geometric viewpoint (Amari and Nagaoka, 2000). We consider the following four types of inference.

Inference 1 (Perfect observation and matched decoding) The complete data $({\overline{R}}_1,{\overline{R}}_2)$ are available and the true probability distribution p(r; s) is used for decoding.

Inference 2 (Imperfect observation) The true probability distribution p(r; s) is used for decoding but only partial data $({\overline{R}}_1)$ are available.

Inference 3 (Mismatched decoding) The complete data $({\overline{R}}_1,{\overline{R}}_2)$ are available but a mismatched probability distribution q(r; s) is used for decoding.

Inference 4 (Imperfect observation and mismatched decoding) A mismatched probability distribution q(r; s) is used for decoding and only partial data $({\overline{R}}_1)$ are available.

We assumed that both the true probability distribution p(r; s) and mismatched probability distributions q(r; s) belong to the exponential family of probability distributions S given in Eq. 1. S is specified by n-dimensional natural parameters θ = (θ_{1, θ2, …, θn}). If we take θ as a coordinate system introduced in set S of probability distributions, we can regard S as an n-dimensional manifold (space). A point in S represents a specific probability distribution determined by the parameters θ. The true statistical model p(r; s) denoted by M and the mismatched statistical model q(r; s) denoted by M*, both of which are parameterized by a single variable s, are considered as curves in the manifold S, i.e., one-dimensional submanifolds having a coordinate s.

Inference 1 First, we describe Inference 1 from the viewpoint of information geometry (Amari, 1982; Amari and Nagaoka, 2000). Let us denote the observed data $({\overline{R}}_1^o,{\overline{R}}_2^o)$ by $\overline{x}.$ $\overline{x}$ can be considered as a point in S, which we call the observed point. The observed point $\overline{x}$ is distributed near the point s that represents the true probability distribution when stimulus s is presented, p(r; s). The deviation of $\overline{x}$ from the point specified by the true stimulus parameter s can be decomposed into the deviation in the parallel direction to M and the deviation in the orthogonal direction to M. The maximum likelihood estimation corresponds to the minimizer of the Kullback–Leibler divergence between the distribution corresponding to the observed point $\overline{x}$ (which is not in M in general) and distributions in M,

$$\widehat{s}=\underset{s}{\arg \min }KL\left[\overline{x}\left({\overline{R}}_1^o,{\overline{R}}_2^o\right)\parallel p(r;s)\right],$$ (37)

where $\widehat{s}$ is the maximum likelihood estimator. The geometric interpretation of the maximum likelihood estimation is the orthogonal projection to M from $\overline{x}$ (Figure 2). The orthogonal projection completely eliminates the deviation of $\overline{x}$ from s in the orthogonal direction to M but the deviation in the parallel direction to M remains. This remaining deviation corresponds to the Fisher information of M (Eq. 18). If we use other estimators that are not orthogonal projections to M (e.g., the moment estimator), the decoding error necessarily becomes larger than the orthogonal projection.

Figure 2.

Information geometric picture of Inference 1 (perfect observation and matched decoding).

Inference 2 In the Inference 2 case (Amari, 1995), only ${\overline{R}}_1$ is observed. Let us define a submanifold D, which is formed by the set of observed points, where ${\overline{R}}_1$ is fixed at the observed value ${\overline{R}}_1^o$ but unobserved ${\overline{R}}_2$ takes arbitrary values. Submanifold D is called the data submanifold. The maximum likelihood estimation based on partial observed data corresponds to searching for the pair of points and that minimizes the Kullback–Leibler divergence between $\widehat{x}\in D$ and $\widehat{s}\in M$ (Figure 3), i.e.,

Figure 3.

Information geometric picture of Inference 2 (imperfect observation).

$$\min KL\left(D\parallel M\right)=\underset{\text{x}\in D,s\in M}{\min }K\left(\widehat{x}\parallel s\right).$$ (38)

The estimated value of s is expressed as

$${\widehat{s}}_d\text{=}\underset{s}{\arg \min }\left[\underset{{\overline{R}}_2}{\min }KL\left[\widehat{x}\left({\overline{R}}_1^o,{\overline{R}}_2\right)\parallel p(r;s)\right]\right].$$ (39)

For a given candidate point in D, $\widehat{x},$ the point in M that minimizes the Kullback–Leibler divergence is given by the orthogonal projection $\widehat{x}$ of to M.

Inference 3 For Inference 3 (Figure 4), we assumed that the maximum likelihood estimation based on a mismatched model q(r; s) is unbiased (Eq. 29). This condition corresponds to the case where the point p(r; s) in M and the point q(r; s) in M*, which both represent a given stimulus parameter s, are the mutually nearest points in S in terms of the Kullback–Leibler divergence, i.e.,

Figure 4.

Information geometric picture of Inference 3 (mismatched decoding).

$$q(r;s)=\underset{q(r;s^{\prime })}{\arg \min }KL\left[p(r;s)\parallel q(r;s^{\prime })\right].$$ (40)

If we differentiate Eq. 40 with respect to s′, we obtain Eq. 29. Similar to in Inference 1, the maximum likelihood estimation based on a mismatched model M* corresponds to the minimization of the Kullback–Leibler divergence between the observed point $\overline{x}$ and points in M*,

$${\widehat{s}}_q=\underset{s}{\arg \min }KL\left[\overline{x}\left({\overline{R}}_1^o,{\overline{R}}_2^o\right)\parallel q(r;s)\right].$$ (41)

This corresponds to the orthogonal projection of the observed point $\overline{x}$ to M* (Figure 4). The orthogonal projection to M* cannot completely eliminate the deviation in the direction perpendicular to M, unless M and M* are in parallel. Thus, information is inevitably lost depending on the angle between M and M*. The Fisher information for the mismatched decoding model is given by Eq. 36.

Inference 4 In Inference 4 (Figure 5), the maximum likelihood estimation with partial observed data ${\overline{R}}_1^o$ and a mismatched probability distribution q(r; s) corresponds to searching for two points in the data submanifold D and the mismatched model M*:

Figure 5.

Information geometric picture of Inference 4 (imperfect observation and mismatched decoding).

$${\widehat{s}}_{qd}=\underset{s}{\arg \min }\left[\underset{{\overline{R}}_2}{\min }KL\left[\overline{x}\left({\overline{R}}_1^o,{\overline{R}}_2\right)\parallel q(r;s)\right]\right].$$ (42)

Relationship between Inference with Partial Observed Data and Inference with Mismatched Decoding Model: Gaussian Case

In this section, we compute the Fisher information obtained by the four types of inference described in the previous section when the probability distributions are Gaussian. We also discuss the relationship between the inferences.

One-dimensional case

Before we deal with the multidimensional Gaussian model, we first consider the one-dimensional case as a toy example. We specifically consider the Gaussian distribution with mean μ(s) = s and variance σ²(s) = s²:

$$p(r)=\frac{1}{\sqrt{2\pi s^2}}\exp \left(-\frac{1}{2s^2}{(r-s)}^2\right).$$ (43)

The statistical model M = {N(s, s²)} is expressed as a curve in the manifold S = {N(μ, σ²)} with coordinates of mean μ and variance σ² (Figure 6). The probability distribution on T observations of r is given by

Figure 6.

Information geometric picture of four types of inference in one-dimensional Gaussian model.

$$p(\overline{r},\overline{R})=\exp \left(T\left(\frac{{\overline{r}}^2}{s}-\frac{\overline{R}}{2s^2}-\log \left(\sqrt{2\pi s^2}\right)-\frac{1}{2}\right)\right),$$ (44)

where $\overline{r}$ and $\overline{R}$ are sufficient statistics

$$\overline{r}=\frac{1}{T}{\displaystyle \sum _{t=1}^{T}r(t)},$$ (45)

$$\overline{R}=\frac{1}{T}{\displaystyle \sum _{t=1}^T{r}^2(t)}.$$ (46)

In this model, we compute the Fisher information and the information loss when the data of variance $\overline{R}$ is lost and those when the decoding model whose variance is mismatched with the actual one is used.

Inference 1 When the data of neural responses $\overline{x}=(\overline{r},\overline{R})$ are completely observed and the actual statistical model M is used in decoding, the maximum likelihood estimation of s corresponds to the orthogonal projection from $\overline{x}$ to M (Figure 6). In this case, we can compute the Fisher information as

$$I_{F1}\left(s;\overline{r},\overline{R}\right)=\frac{3T}{s^2}.$$ (47)

Inference 2 When data of variance $\overline{R}$ is lost, the data manifold D is given by

$$D=\left\{N\left(\mu ,{\sigma }^2\right)|\mu =\overline{r},{\sigma }^2\text{ is arbitrary}\right\}.$$ (48)

In this case, the estimated value of s is the intersection point of D and M (Figure 6). By using Eq. 27, we can obtain the Fisher information,

$$I_{F2}(s;\overline{r})=\frac{T+2}{s^2}\sim \frac{T}{s^2},$$ (49)

where we used the fact that the marginal distribution over $\overline{R}$ can be written as

$$p(\overline{r})=\frac{1}{\sqrt{2\pi {\left(\frac{s}{\sqrt{T}}\right)}^2}}\exp \left(-\frac{1}{2{\left(\frac{s}{\sqrt{T}}\right)}^2}{\left(\overline{r}-s\right)}^2\right).$$ (50)

$p(\overline{r})$ can be derived by considering that $\overline{r}$ also obey a Gaussian distribution and that the mean and the variance of $\overline{r}$ are s and s²/T, respectively.

The information loss is given by

$$\Delta I_{F2}(s)=\frac{2T}{s^2}.$$ (51)

Inference 3 We specifically consider the inference with the following mismatched decoding model to compare it with the inference when $\overline{R}$ is lost:

$$q(r)=\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp \left(-\frac{1}{2{\sigma }^2}{(r-s)}^2\right).$$ (52)

In this model, the mean is equal to the actual one but the variance is mismatched with the actual one. The maximum likelihood estimation based on the mismatched decoding model M* corresponds to the orthogonal projection from the observed point $\overline{x}$ to M* (Figure 6). By using Eq. 36, we obtain the Fisher information

$$I_{F3}(s)=\frac{T}{s^2}.$$ (53)

Inference 4 When the mismatched decoding model q(r), where the variance is independent of s, is used for decoding, the data of variance $\overline{R}$ does not affect the results of the inference. Thus, even if $\overline{R}$ is lost, no information is lost in this mismatched decoding. The Fisher information in the Inference 4 case is the same as that in the Inference 3 case:

$$I_{F4}(s)=I_{F3}(s).$$ (54)

As Eqs. 49 and 53 show, I_F3(s) is equal to I_F2(s). We can also easily show that I_F3(s) is equal to I_F2(s) in one-dimensional cases in general. However, in the multidimensional case, I_F3(s) is not equal to I_F2(s). In the next section, we explain the general relationship between I_F2(s) and I_F3(s) in the multidimensional Gaussian model.

Multidimensional case

We next consider the multidimensional Gaussian distribution shown in Eq. 2. The probability distribution for T observations of neural responses r is given by

$$p\left(\overline{r},\overline{R}\right)=\exp \left(T\left(\theta (s)\cdot \overline{r}+\Theta (s)\cdot \overline{R}-\psi (s)\right)\right),$$ (55)

where the sufficient statistics $\overline{r}$ and $\overline{R}$ are

$$\overline{r}=\frac{1}{T}{\displaystyle \sum _{t=1}^{T}r(t)},$$ (56)

$$\overline{R}=\frac{1}{T}{\displaystyle \sum _{t=1}^{T}r(t)r{(t)}^T},$$ (57)

the natural parameters θ and Θ are

$$\theta ((s)={\text{C}}^{-1}(s)f(s),$$ (58)

$$\Theta (s)=-\frac{1}{2}{C}^{-1}(s),$$ (59)

and the normalization constant ψ(s) is

$$\Psi (s)=\frac{1}{2}f(s)C^{-1}(s)f(s)+\log \left(\sqrt{\det [\text{C}]}\right)+\log \left(\sqrt{{(2\pi )}^N}\right).$$ (60)

Inference 1 First, let us consider the Fisher information in the Inference 1 case. The Fisher information matrix with respect to the natural parameters is given by

$$-\frac{{\partial }^2\log p\left(\overline{r},\overline{R}\right)}{\partial {\theta }_i\partial {\theta }_j}=T{C}_{ij},$$ (61)

$$-\frac{{\partial }^2\log p\left(\overline{r},\overline{R}\right)}{\partial {\theta }_i\partial {\Theta }_{jk}}=T\left(C_{ji}{f}_k+C_{ki}{f}_j\right),$$ (62)

$$-\frac{{\partial }^2\log p(\overline{r},\overline{R})}{\partial {\Theta }_{ij}\partial {\Theta }_{kl}}=T\left(2C_{ik}{f}_l{f}_j+2C_{jk}{f}_l{f}_i+2C_{ik}{C}_{lj}\right).$$ (63)

By using the Fisher information matrix with respect to the natural parameters, we can obtain the Fisher information with respect to stimulus s from Eq. 25:

$$\frac{I_{F1}\left(s;\overline{r},\overline{R}\right)}{T}={f^{\prime }}^T{C}^{-1}{f}^{\prime }+\frac{1}{2}Tr\left[C^{\prime }{C}^{-1}{C}^{\prime }{C}^{-1}\right].$$ (64)

Inference 2 Second, let us consider the Fisher information in the Inference 2 case. We consider the situation that the second-order sufficient statistics $\overline{R}$ in Eq. 55 are lost and only the first-order sufficient statistics $\overline{r}$ are observed. The marginalized distribution over missing data $\overline{R}$ is given by

$$\begin{array}{l}p(\overline{r})\\ =\frac{1}{\sqrt{{(2\pi )}^N\det [\text{C}(s)/T]}}\exp \left(-\frac{T}{2}{(\overline{r}-f(s))}^T{C}^{-1}(s)(\overline{r}-f(s))\right),\end{array}$$ (65)

$$=\exp \left(T\left(\theta (s)\cdot \overline{r}+\Theta (s)\cdot \overline{r}{\overline{r}}^T-\Psi (s)\right)\right),$$ (66)

where the natural parameters θ and θ are

$$\theta (s)=C^{-1}(s)f(s),$$ (67)

$$\Theta (s)=-\frac{1}{2}{C}^{-1}(s),$$ (68)

and the normalization constant ψ(s) is

$$\begin{array}{l}\Psi (s)=\frac{1}{2}f(s)C^{-1}(s)f(s)+\frac{1}{T}\log \left(\sqrt{\det [C]}\right)+\frac{1}{T}\log \sqrt{{\left(\frac{2\pi }{T}\right)}^N},\\ \sim \frac{1}{2}f(s)C^{-1}(s)f(s),\end{array}$$ (69)

where we ignored the terms of the order of 1/T in the limit of T → ∞. In this case, the Fisher information matrix with respect to the natural parameters is computed as follows:

$$-\frac{{\partial }^2\log p(\overline{r})}{\partial {\theta }_i\partial {\theta }_j}=T{C}_{ij},$$ (70)

$$-\frac{{\partial }^2\log p(\overline{r})}{\partial {\theta }_i\partial {\Theta }_{jk}}=T\left(C_{ji}{f}_k+C_{ki}{f}_j\right),$$ (71)

$$-\frac{{\partial }^2\log p(\overline{r})}{\partial {\Theta }_{ij}\partial {\Theta }_{kl}}=T\left(2C_{ik}{f}_l{f}_j+2C_{jk}{f}_l{f}_i\right).$$ (72)

If we compare the components of the Fisher information matrix when $\overline{\mathbf{\text{R}}}$ is missing with those when the data are complete, the information loss due to the missing data is seen to be represented in the components $\frac{{\partial }^2\log p(\overline{r},\overline{R})}{\partial {\Theta }_{ij}\partial {\Theta }_{kl}}$ (Eqs. 63 and 72). By using the Fisher information matrix with respect to the natural parameters, we can compute the Fisher information with respect to stimulus s as

$$\frac{I_{F2}\left(s;\overline{r}\right)}{T}={f^{\prime }}^T{C}^{-1}{f}^{\prime }.$$ (73)

From Eqs. 64 and 73, we find that the information loss due to the missing data $\overline{\mathbf{\text{R}}}$ is

$$\frac{\Delta I_{F2}(s)}{T}=\frac{1}{2}\text{Tr}\left[C^{\prime }{C}^{-1}{C}^{\prime }{C}^{-1}\right].$$ (74)

This information loss solely depends on C′C⁻¹ and is always positive.

Inference 3 Third, let us consider the Fisher information in the Inference 3 case. In the Inference 2 case, we considered that the second-order sufficient statistics $\overline{\mathbf{\text{R}}}$ which are the variance and covariance data of neural responses, are lost. A mismatched probability distribution q(r; s) that is comparable with the inference when $\overline{\mathbf{\text{R}}}$ is lost would be that the mean in q(r; s) is the same as that in p(r; s) but the covariance matrix in q(r; s) does not match the true covariance matrix in p(r; s). As a simple example, we assume that the covariance matrix in the mismatched probability distribution is a constant matrix C_q that is independent of s:

$$q(r;s)=\frac{1}{\sqrt{{(2\pi )}^N\det \left[C_q\right]}}\exp \left(-\frac{1}{2}{(r-f(s))}^T{C}_q^{-1}(r-f(s))\right).$$ (75)

In this case, we can show that the maximum likelihood estimation based on the mismatched probability distribution is consistent, i.e., the condition shown in Eq. 29 is satisfied, as follows:

$$\begin{array}{l}{E}_p[{\partial }_s{l}_q(r;s)]=E_p\left[r^T{C}_q^{-1}\frac{\partial f}{\partial s}-f^T{C}_q^{-1}\frac{\partial f}{\partial s}\right],\\ =0.\end{array}$$ (76)

By using Eq. 36, we obtain the Fisher information for the mismatched model q(r; s):

$$\frac{I_{F3}(s)}{T}=\frac{{\left({f^{\prime }}^T{C}_q^{-1}{f}^{\prime }\right)}^2}{{f^{\prime }}^T{C}_q^{-1}C{C}_q^{-1}{f}^{\prime }}.$$ (77)

Inference 4 and comparison Finally, we consider the Fisher information in the Inference 4 case and compare the four types of inference described above. It is obvious that the Fisher information obtained by Inference 1, I_F1, is the largest and the Fisher information obtained by Inference 4, I_F4, is the smallest. On the other hand, the relationship between the Fisher information obtained by Inference 2, I_F2, and that obtained by Inference 3, I_F3, is not clear, i.e.,

$$I_{F1}>I_{F2},I_{F3}>I_{F4}.$$ (78)

However, the relationship between I_F2 and I_F3 can be clarified by considering the Fisher information obtained by Inference 4, I_F4. We assumed that a mismatched model that is related to the loss of covariance data, $\overline{\mathbf{\text{R}}}$ is the statistical model whose covariance matrix is a constant matrix. In this case, the vector of natural parameters Θ in Eq. 55, which is coupled with $\overline{\mathbf{\text{R}}},$ does not depend on s. Thus, when we use a mismatched model q(r; s) whose covariance matrix is independent of s, the inference does not change even if the data about covariance $\overline{\mathbf{\text{R}}}$ are lost. Thus, Inferences 3 and 4 result in the same estimate of s and the same Fisher information:

$$I_{F4}(s)=I_{F3}(s).$$ (79)

To summarize, the relationship between the Fisher information in each of the four inference cases is

$$I_{F1}>I_{F2}>I_{F3}=I_{F4}.$$ (80)

Information Loss in Log-Linear Model of Binary Neural Response

In this section, we evaluate the information loss associated with loss of data and mismatched decoding in the log-linear model of binary neural response.

Two-neuron model

As the simplest example, we first consider the two-neuron model,

$$p\left(r_1,r_2\right)=\exp ({\theta }_1(s)r_1+{\theta }_2(s)r_2+\Theta (s)r_1{r}_2-\log Z(s)),$$ (81)

where Z(s) is a normalization constant,

$$Z(s)={\displaystyle \sum _{r_1,r_2=0,1}\exp \left({\displaystyle \sum _{i=1}^2{\theta }_i(s)r_i+\Theta (s)r_1{r}_2}\right)}.$$ (82)

The probability distribution for T observations of neural responses r can be written as

$$\begin{array}{l}p\left({\overline{r}}_1,{\overline{r}}_2,{\overline{R}}_{12}\right)=W\left({\overline{r}}_1,{\overline{r}}_2,{\overline{R}}_{12}\right)\exp \left(T({\theta }_1{\overline{r}}_1+{\theta }_2{\overline{r}}_2+\Theta {\overline{R}}_{12}-\log Z)\right),\\ =\exp \left(T\left({\theta }_1{\overline{r}}_1+{\theta }_2{\overline{r}}_2+\Theta {\overline{R}}_{12}+\frac{\log W\left({\overline{r}}_1,{\overline{r}}_2,{\overline{R}}_{12}\right)}{T}-\log Z\right)\right),\end{array}$$ (83)

where ${\overline{r}}_2,{\overline{r}}_1,$ and ${\overline{R}}_{12}$ are sufficient statistics,

$${\overline{r}}_1=\frac{1}{T}{\displaystyle \sum _{t=1}^T{r}_1^{(t)}},$$ (84)

$${\overline{r}}_2=\frac{1}{T}{\displaystyle \sum _{t=1}^T{r}_2^{(t)}},$$ (85)

$${\overline{R}}_{12}=\frac{1}{T}{\displaystyle \sum _{t=1}^T{r}_1^{(t)}{r}_2^{(t)}},$$ (86)

and $W({\overline{r}}_1,{\overline{r}}_2,{\overline{R}}_{12})$ is the number of configurations of (r⁽¹⁾, r⁽²⁾,…,r^(T)) where the sufficient statistics take the specific values ${\overline{r}}_1,\text{\hspace{0.17em}\hspace{0.05em}}{\overline{r}}_2,\text{\hspace{0.17em}\hspace{0.05em}and\hspace{0.17em}\hspace{0.05em}}{\overline{R}}_{12}.$ $W({\overline{r}}_1,{\overline{r}}_2,{\overline{R}}_{12})$ can be expressed as

$$\begin{array}{l}W\left({\overline{r}}_1,{\overline{r}}_2,{\overline{R}}_{12}\right)\\ =\frac{T!}{\left(T{\overline{R}}_{12}\right)!\left(T\left({\overline{r}}_1-{\overline{R}}_{12}\right)\right)!\left(T\left({\overline{r}}_2-{\overline{R}}_{12}\right)\right)!\left(T\left(1-{\overline{r}}_1-{\overline{r}}_2+{\overline{R}}_{12}\right)\right)!}.\end{array}$$ (87)

In the limit of T → ∞, by using Stirling's formula, $W({\overline{r}}_1,{\overline{r}}_2,{\overline{R}}_{12})$ can be approximated as

$$\begin{array}{c}\log W=-T({\overline{R}}_{12}\log {\overline{R}}_{12}+\left({\overline{r}}_1-{\overline{R}}_{12}\right)\log \left({\overline{r}}_1-{\overline{R}}_{12}\right)\\ +\left({\overline{r}}_2-{\overline{R}}_{12}\right)\log \left({\overline{r}}_2-{\overline{R}}_{12}\right))\\ +\left(1-{\overline{r}}_1-{\overline{r}}_2+{\overline{R}}_{12}\right)\log \left(1-{\overline{r}}_1-{\overline{r}}_2+{\overline{R}}_{12}\right)).\end{array}$$ (88)

Inference 1 We first compute the Fisher information in the Inference 1 case. The Fisher information matrix with respect to the natural parameters is given by

$$-\frac{{\partial }^2\log p({\overline{r}}_1,{\overline{r}}_2,{\overline{R}}_{12})}{\partial {\theta }_i\partial {\theta }_j}=T\left(\langle r_i{r}_j\rangle -\langle r_i\rangle \langle r_j\rangle \right),$$ (89)

$$-\frac{{\partial }^2\log p\left({\overline{r}}_1,{\overline{r}}_2,{\overline{R}}_{12}\right)}{\partial {\Theta }^2}=T\left(\langle r_1{r}_2\rangle -{\langle r_1{r}_2\rangle }^2\right),$$ (90)

$$-\frac{{\partial }^2\log p\left({\overline{r}}_1,{\overline{r}}_2,{\overline{R}}_{12}\right)}{\partial {\theta }_i\partial \Theta }=T\left(\langle r_1{r}_2\rangle -\langle r_i\rangle \langle r_1{r}_2\rangle \right),$$ (91)

where 〈x〉 = E_p[x] = ∑_rxp(r₁, r₂). By using the Fisher information matrix with respect to the natural parameters, we can obtain the Fisher information with respect to stimulus s from Eq. 25:

$$\begin{array}{l}\frac{I_{F1}\left(s;{\overline{r}}_1,{\overline{r}}_2,{\overline{R}}_{12}\right)}{T}\\ ={\displaystyle \sum _{i=1}^2\langle r_i\rangle \left(1-\langle r_i\rangle \right){\left(\frac{\partial {\theta }_i}{\partial s}\right)}^2+\langle r_1{r}_2\rangle \left(1-\langle r_1{r}_2\rangle \right){\left(\frac{\partial \Theta }{\partial s}\right)}^2}\\ +2\left(\langle r_1{r}_2\rangle -\langle r_1\rangle \langle r_2\rangle \right)\frac{\partial {\theta }_1}{\partial s}\frac{\partial {\theta }_2}{\partial s}+2{\displaystyle \sum _{i=1}^2\langle r_1{r}_2\rangle \left(1-\langle r_i\rangle \right)\frac{\partial {\theta }_i}{\partial s}\frac{\partial \Theta }{\partial s}.}\end{array}$$ (92)

Inference 2 We next compute the Fisher information when the data of neural correlation ${\overline{R}}_{12}$ are lost. When T is finite, it is difficult to marginalize the probability distribution $p({\overline{r}}_1,{\overline{r}}_2,{\overline{R}}_{12})$ over ${\overline{R}}_{12}$ because there are many possible ${\overline{R}}_{12}$ when specific values of ${\overline{r}}_1$ and ${\overline{r}}_2$ are given. However, in the T → ∞ case, we only need to consider the most probable ${\overline{R}}_{12}$ when ${\overline{r}}_1$ and ${\overline{r}}_2$ are given. By differentiating the argument of the exponential function in Eq. 83 with respect to ${\overline{R}}_{12},$ we can obtain the equation for the most probable ${\overline{R}}_{12}$:

$$\Theta -\log \frac{{\overline{R}}_{12}\left(1-{\overline{r}}_1-{\overline{r}}_2+{\overline{R}}_{12}\right)}{\left({\overline{r}}_1-{\overline{R}}_{12}\right)\left({\overline{r}}_2-{\overline{R}}_{12}\right)}=0.$$ (93)

We denote the solution of this equation by ${\overline{R}}_{12}^{\ast }$ Note that ${\overline{R}}_{12}^{\ast }$ depends on θ but does not depend on θ₁ and θ₂. The marginalized probability distribution is written as

$$\begin{array}{l}p\left({\overline{r}}_1,{\overline{r}}_2\right)\\ =\exp \left(T\left({\theta }_1{\overline{r}}_1+{\theta }_2{\overline{r}}_2+\Theta {\overline{R}}_{12}^{\ast }+\frac{\log W\left({\overline{r}}_1,{\overline{r}}_2,{\overline{R}}_{12}^{*}\right)}{T}-\log Z\right)\right).\end{array}$$ (94)

By using Eq. 94, we can compute the Fisher information when ${\overline{R}}_{12}$ is lost as

$$\begin{array}{l}\frac{I_{F2}\left(s;{\overline{r}}_1,{\overline{r}}_2\right)}{T}=\frac{I_{F1}\left(s;{\overline{r}}_1,{\overline{r}}_2,{\overline{R}}_{12}\right)}{T}\\ -E_p\left({\overline{r}}_1,{\overline{r}}_2\right)\left[\frac{{\partial }^2}{\partial {\Theta }^2}\left(\Theta {\overline{R}}_{12}^{\ast }+\frac{1}{T}\log W\left({\overline{r}}_1,{\overline{r}}_2,{\overline{R}}_{12}^{\ast }\right)\right){\left(\frac{\partial \Theta }{\partial s}\right)}^2,\right]\end{array}$$ (95)

where we used the fact that ${\overline{R}}_{12}^{\ast }$ only depends on θ. The information loss is given by

$$\begin{array}{c}\frac{\Delta I_{F2}(s)}{T}=E_{p({\overline{r}}_1,{\overline{r}}_2)}\left[\frac{{\partial }^2}{\partial {\Theta }^2}\left(\Theta {\overline{R}}_{12}^{\ast }+\frac{1}{T}\log W\left({\overline{r}}_1,{\overline{r}}_2,{\overline{R}}_{12}^{\ast }\right)\right)\right]d{\left(\frac{\partial \Theta }{\partial s}\right)}^2,\\ =(\frac{1}{\langle r_1,r_2\rangle }+\frac{1}{\langle r_1\rangle -\langle r_1{r}_2\rangle }+\frac{1}{\langle r_2\rangle -\langle r_1{r}_2\rangle }\\ +\frac{1}{1-\langle r_1\rangle -\langle r_2\rangle +\langle r_1{r}_2\rangle }{\left(\frac{\partial \Theta }{\partial s}\right)}^2).\end{array}$$ (96)

This information loss only depends on ∂θ/∂s. Thus, when θ is independent of s, there is no information loss even if data ${\overline{R}}_{12}$ are lost.

Inference 3 We next compute the Fisher information when neural correlation is ignored in decoding. We consider the mismatched decoding model q(r₁, r₂) that is the product of the marginal probability distributions of the actual distribution p(r₁, r₂):

$$\begin{array}{l}q(r_1,r_2;s)={\displaystyle \prod _{i=1}^{2}p\left(r_i\right)},\\ ={\displaystyle \prod _{i=1}^2\exp \left({\theta }_i^D(s)r_i-\log Z_i^D(s)\right)},\end{array}$$ (97)

where p(r_i) is the marginalized distribution over r_j, i.e., $p(r_i)={\sum }_{r_j}p(r_i,r_j),$ and the normalization constant $Z_i^D(s)$ is given by

$$Z_i^D(s)=1+\exp \left({\theta }_i^D(s)\right).$$ (98)

From the definition of q(r₁, r₂), the averaged values of r₁ and r₂ over the mismatched model q(r₁, r₂) are equal to those over the actual model p(r₁, r₂). Thus, the following relationship holds between ${\theta }_i^D(s)$ and the natural parameters in the actual probability distribution p(r₁, r₂):

$$\begin{array}{l}\langle r_i\rangle =\frac{1}{Z(s)}{\displaystyle \sum _{r_1,r_2}{r}_i\exp \left({\theta }_1{r}_1+{\theta }_2{r}_2+\Theta r_1{r}_2\right)},\\ =\frac{\exp \left({\theta }_i^D(s)\right)}{1+\exp \left({\theta }_i^D(s)\right)}.(\text{for }i=1,2)\end{array}$$ (99)

The maximum likelihood estimation based on this mismatched decoding model q(r₁, r₂) is shown to be consistent as follows:

$$E_p\left[\frac{\partial \log q(r_1,r_2)}{\partial s}\right]=E_p\left[{\displaystyle \sum _{i=1}^2\frac{\partial {\theta }_i^D(s)}{\partial s}}\left(r_i-\frac{\exp \left({\theta }_i^D(s)\right)}{1+\exp \left({\theta }_i^D(s)\right)}\right)\right],$$ (100)

$$=0.$$ (101)

When the estimation by a mismatched decoding model is consistent, the Fisher information obtained by the mismatched decoding model can be computed by Eq. 36. The Fisher information is given by

$$\frac{I_{F3}(s)}{T}=\frac{{\left({\displaystyle \sum _{i=1}^2{\left(\frac{\partial {\theta }_i^D}{\partial s}\right)}^2\langle r_i\rangle \left(1-\langle r_i\rangle \right)}\right)}^2}{{\displaystyle \sum _{i=1}^2{\left(\frac{\partial {\theta }_i^D}{\partial s}\right)}^2\langle r_i\rangle \left(1-\langle r_i\rangle \right)+2\frac{\partial {\theta }_1^D}{\partial s}\frac{\partial {\theta }_2^D}{\partial s}\left(\langle r_1{r}_2\rangle -\langle r_1\rangle \langle r_2\rangle \right)}},$$ (102)

where

$$\begin{array}{l}\frac{\partial {\theta }_i^D}{\partial s}=\frac{\partial {\theta }_i}{\partial s}\langle r_i\rangle \left(1-\langle r_i\rangle \right)+\frac{\partial {\theta }_j}{\partial s}\left(\langle r_1{r}_2\rangle -\langle r_1\rangle \langle r_2\rangle \right)\\ +\frac{\partial \Theta }{\partial s}\langle r_1{r}_2\rangle \left(1-\langle r_i\rangle \right).(\text{for }i,j=1,2)\end{array}$$ (103)

As discussed in the previous section, I_F3(s) is always smaller than I_F2(s). Although the information loss associated with the loss of data $\overline{R}$ only depends on ∂Θ/∂s, the information loss associated with ignoring correlation in decoding depends not only on ∂Θ/∂s but also on ∂θ₁/∂s and ∂θ₂/∂s. Thus, when neural correlation is ignored in decoding, the information is lost even if Θ is independent of s.

As a special case, if θ₁ = θ₂, which means 〈r₁〉 = 〈r₂〉 = 〈r〉, the information loss only depends on ∂Θ/∂s and is given by

$$\frac{\Delta I_{F3}(s)}{T}=\frac{\langle r_1{r}_2\rangle \left(\langle r\rangle -\langle r_1{r}_2\rangle \right)\left(1-2\langle r\rangle +\langle r_1{r}_2\rangle \right)}{\langle r\rangle -2{\langle r\rangle }^2+\langle r_1{r}_2\rangle }{\left(\frac{\partial \Theta }{\partial s}\right)}^2.$$ (104)

In this case, ΔI_F3(s) is equal to ΔI_F2(s) (Eq. 96). This is because if θ₁ = θ₂ = θ, only two parameters, namely θ and Θ, are in the statistical model. This is the same situation as in the one-dimensional Gaussian case illustrated in Figure 6, where ΔI_F3(s) is also equal to ΔI_F2(s).

Homogeneous n neuron model

We next consider the case with a large number of neurons. In this case, the Fisher information cannot be analytically computed in general. To restrict ourselves to dealing with an analytically tractable model, we here only deal with a probabilistic model of a homogeneous neural population. In this model, θi(s) = θ(s) for any i and θij(s) = Θ(s) for any pair of i and j, i.e.,

$$p(r;s)=\exp \left(\theta (s){\displaystyle \sum _{i=1}^N{r}_i+\frac{\Theta (s)}{N}}{\displaystyle \sum _{i<j}{r}_i{r}_j-\log Z(s)}\right),$$ (105)

where the normalization constant Z(s) is given by

$$Z(s)=\text{Tr}\exp \left(\theta (s){\displaystyle \sum _{i=1}^N{r}_i}+\frac{\Theta (s)}{N}{\displaystyle \sum _{i<j}{r}_i{r}_j}\right),$$ (106)

where Tr stands for the sum over all possible combinations of the neuron state variables (r₁, r₂,…,r_N). The probability distribution of T observations is given by

$$p\left(\overline{r},\overline{R}\right)=\exp \left(T\left(\theta (s){\displaystyle \sum _{i=1}^N{\overline{r}}_i}+\frac{\Theta (s)}{N}{\displaystyle \sum _{i<j}{\overline{R}}_{ij}}-\log Z(s)\right)\right),$$ (107)

where $\overline{r}=\left\{{\overline{r}}_i\right\}$ and $\overline{R}=\left\{{\overline{R}}_{ij}\right\}$ are sufficient statistics,

$${\overline{r}}_i=\frac{1}{T}{\displaystyle \sum _{t=1}^T{r}_i(t)},$$ (108)

$${\overline{R}}_{ij}=\frac{1}{T}{\displaystyle \sum _{t=1}^T{r}_i(t)r_j(t)}.$$ (109)

Inference 1 We first compute the Fisher information when data are complete and the decoding is optimal. As Eq. 23 shows, the Fisher information can be computed if we evaluate log Z. For analytical tractability, we consider the limit of N→∞. In this case, Z can be calculated as

$$\begin{array}{l}Z=\text{Trexp}\left(\theta \{{\displaystyle \sum _i{r}_i}+\frac{\Theta }{N}{\displaystyle \sum _{i<j}}{r}_i{r}_j\right),\\ =\text{Trexp}\left(\left(\theta -\frac{\Theta }{2N}\right){\displaystyle \sum _i{r}_i}+\frac{\Theta }{2N}{\left({\displaystyle \sum _i}{r}_i\right)}^2\right),\\ ={\displaystyle {\int }_{-\infty }^{\infty }dm}\text{\hspace{0.17em}exp}\left(N\left(\frac{\Theta }{2}{m}^2+\theta m\right)+\text{log}W(m)\right),\end{array}$$ (110)

where m is

$$m=\frac{1}{N}{\displaystyle \sum _{i=1}^N}{r}_i$$ (111)

and W(m) is the number of states where m takes a certain value, which is given by

$$W(m)=\frac{N!}{(Nm)!(N(1-m))!}.$$ (112)

In the limit of N→∞, by using Stirling's formula, W(m) can be approximated as

$$\log W(m)=-N(m\log m+(1-m)\log (1-m)).$$ (113)

We denote the argument of the exponential function in Eq. 110 by F, where

$$F=N\left(\frac{\Theta }{2}{m}^2+\theta m-m\log m-(1-m)\log (1-m)\right).$$ (114)

In the limit of N→∞, the integral in Eq. 110 can be approximated as

$${\displaystyle {\int }_{-\infty }^{\infty }dm\exp }(F)=\exp (F^{\ast }),$$ (115)

where F* is the maximum of the function F. From ∂F/∂m = 0, the value of m that maximizes the function F is the solution of the self-consistent equation

$$m=\frac{\exp (\theta +\Theta m)}{1+\exp (\theta +\Theta m)}.$$ (116)

We set the solution of Eq. 116 to m*(θ, θ). The Fisher information matrix with respect to the natural parameters θ and θ is given by

$$-\frac{{\partial }^2{F}^{\ast }}{\partial {\theta }^2}=N\frac{m^{\ast }(1-m^{\ast })}{1-\Theta m^{\ast }(1-m^{\ast })},$$ (117)

$$-\frac{{\partial }^2{F}^{\ast }}{\partial \theta \partial \Theta }=N\frac{m^{\ast 2}(1-m^{\ast })}{1-\Theta m^{\ast }(1-m^{\ast })},$$ (118)

$$-\frac{{\partial }^2{F}^{\ast }}{\partial {\Theta }^2}=N\frac{m^{\ast 3}(1-m^{\ast })}{1-\Theta m^{\ast }(1-m^{\ast })}.$$ (119)

By using the Fisher information matrix, we obtain the Fisher information with respect to stimulus s from Eq. 25:

$$\begin{array}{l}\frac{I_{F1}(s)}{N}=\frac{m^{\ast }(1-m^{\ast })}{1-\Theta m^{\ast }(1-m^{\ast })}{\left(\frac{\partial \theta }{\partial s}\right)}^2+2\frac{m^{\ast 2}(1-m^{\ast })}{1-\Theta m^{\ast }(1-m^{\ast })}\frac{\partial \theta }{\partial s}\frac{\partial \Theta }{\partial s}\\ +\frac{m^{\ast 3}(1-m^{\ast })}{1-\Theta m^{\ast }(1-m^{\ast })}{\left(\frac{\partial \Theta }{\partial s}\right)}^2.\end{array}$$ (120)

Inferences 2 and 3 We next consider the situation where the data of correlation, $\overline{\text{R}}$ are lost and a mismatched decoding model that ignores neural correlation q(r; s) is used. We first consider the inference with a mismatched decoding model (Inference 3). This model q(r; s) is defined from the actual probability distribution p(r; s) as follows:

$$q(r;s)={\displaystyle \prod _{i=1}^{N}p(r_i),}$$

$$={\displaystyle \prod _{i=1}^N\exp ({\theta }_D(s)r_i-\log Z_D(s)),}$$ (121)

where the normalization constant Z_D is given by

$$Z_D(s)=1+\exp ({\theta }_D(s)).$$ (122)

The relationship between m* in the previous section and θD is

$$m^{\ast }=\frac{\exp ({\theta }_D)}{1+\exp ({\theta }_D)}.$$ (123)

By comparing Eq. 123 with Eq. 116, we obtain

$${\theta }_D=\theta +\Theta m^{\ast }.$$ (124)

Similar to in the previous section, the maximum likelihood estimation with the independent decoding model q(r; s) can be shown to be consistent as follows:

$$\begin{array}{c}{E}_p[{\partial }_s{l}_q(r;s)]=E_p\left[\left({\displaystyle \sum _{i=1}^N}{r}_i\right)\frac{\partial {\theta }_D}{\partial s}-N\frac{\exp ({\theta }_D)}{1+\exp ({\theta }_D)}\frac{\partial {\theta }_D}{\partial s}\right],\\ =0.\end{array}$$ (125)

By using Eq. 36, we can compute the Fisher information obtained by the independent decoding model as

$$\begin{array}{c}\frac{I_{F3}(s)}{N}=\frac{m^{\ast }(1-m^{\ast })}{1-\Theta m^{\ast }(1-m^{\ast })}{\left(\frac{\partial \theta }{\partial s}\right)}^2+2\frac{m^{\ast 2}(1-m^{\ast })}{1-\Theta m^{\ast }(1-m^{\ast })}\frac{\partial \theta }{\partial s}\frac{\partial \Theta }{\partial s}\\ +\frac{m^{\ast 3}(1-m^{\ast })}{1-\Theta m^{\ast }(1-m^{\ast })}{\left(\frac{\partial \Theta }{\partial s}\right)}^2.\end{array}$$ (126)

Since I_F1(s) = I_F3(s) (see Eqs. 120 and 126), there is no information loss when the independent model is used for decoding in a homogeneous neural population (Wu et al., 2001). As for the Gaussian model, the Fisher information when the data of neural correlation $\overline{R}$ are lost, I_F2, is larger than I_F3 and is smaller than I_F1. Since I_F1(s) = I_F3(s) in this case, I_F2(s) is also equal to I_F1(s). Thus,

$$I_{F1}(s)=I_{F2}(s)=I_{F3}(s).$$ (127)

In summary, no information is lost in a homogeneous neural population even when the data of neural correlation, $\overline{R}$ are lost or when the mismatched model that ignores neural correlation is used for decoding.

Discussion

In this work, we introduced a novel framework for investigating information processing in the brain where we studied information loss caused by two situations: imperfect observations and mismatched decoding. By evaluating the information loss caused by non-simultaneous observations of neural responses, we can quantify the importance of correlated activity. This can also be quantified by similarly evaluating the information loss caused by mismatched decoding that ignores neural correlation. We discussed these two types of loss by giving the information geometric interpretations of inferences with partially observed data and those with a mismatched decoding model and elucidated their relationship. We showed that the information loss associated with ignoring correlation in decoding is always larger than that caused by non-simultaneous observations of neural responses. This is because the inference based on an independent decoding model with complete data is equivalent to the inference based on an independent decoding model with “partial” data where the data of neural correlation are lost, which is naturally worse than the inference based on a correct decoding model with the partial data. This also can be intuitively understood by considering that decoding without the data of correlation considers all possible models of neural correlations, including the correct one, whereas decoding with a mismatched model locks it within the wrong domain.

Taking account of the relationship between the two inference methods, we give a simple guide on how to use the two different measures for quantifying the importance of correlation. To address the importance of coincidence detection by downstream neurons without making any specific assumption about the decoding process in higher-order areas in the brain, we should evaluate the information loss caused by non-simultaneous observations. The information loss quantified in this way can be used as the lower bound on the information conveyed by correlated activity. In contrast, the information loss quantified by using the independent decoding model can be used as the upper bound on the information conveyed by correlated activity because neural correlation is ignored not only in the observations but also in the decoding in this quantification. In summary, we consider that both measures should be computed when quantifying the importance of correlations and should be used as the lower bound and upper bound for the importance of correlations.

We considered the case that the stimulus is represented as a continuous variable. In this case, the Fisher information is a suitable measure for quantifying the maximal amount of information that can be extracted from neural responses. When considering a set of discrete stimuli, which is mostly the case for neurophysiological experiments, we should use the mutual information instead of the Fisher information. Similar to the Fisher information, the mutual information obtained by the inference with partially observed data can be computed by marginalizing the probability distribution of neural responses over the sufficient statistics that correspond to the lost data. In addition, the mutual information obtained by the mismatched decoding model can be computed using information derived by Merhav et al. (Merhav et al., 1994; Latham and Nirenberg, 2005; Amari and Nakahara, 2006; Oizumi et al., 2010). Thus, when we use the mutual information, we can also quantify the importance of correlation with the concepts of missing data and mismatched decoding. We analyzed the binary probabilistic model with a homogeneous neural population and found that neural correlation does not carry any information in this particular case. It is important to study how much information correlated activity could carry in an inhomogeneous neural population such as the hypercolumn model in V1. Another interesting future direction would be to evaluate the information conveyed by higher-order correlation (Amari et al., 2003); we considered only pair-wise correlation in this paper. These issues remain as future work.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Appendix

Calculation of fisher information

In this appendix, we provide derivations for Eqs. 61–64, 70–73, and 89–91. First, we derive a useful relationship which holds for the exponential family. From the definition of the exponential family (Eq. 2), we obtain

$$\frac{\partial \log p(\mathbf{\text{r}};s)}{\partial {\theta }_I}=R_I(r)-\frac{{\partial }^2\psi (s)}{\partial {\theta }_I}.$$ (128)

Taking the average of this equation, we find

$$E_p\left[R_I(\mathbf{\text{r}})\right]=\frac{\partial \psi (s)}{\partial {\theta }_I}.$$ (129)

By using this relationship, we can derive Eqs. 61–63 as follows,

$$\begin{array}{c}-\frac{{\partial }^2\log p(\overline{\mathbf{\text{r}}},\overline{\mathbf{\text{R}}})}{\partial {\theta }_i\partial {\theta }_j}=T\frac{{\partial }^2\psi (s)}{\partial {\theta }_i\partial {\theta }_j},\\ =T\frac{\partial f_i}{\partial {\theta }_j},\\ =T{C}_{ij},\end{array}$$ (130)

$$\begin{array}{l}-\frac{{\partial }^2\log p(\overline{\mathbf{\text{r}}},\overline{\mathbf{\text{R}}})}{\partial {\theta }_i\partial {\Theta }_{jk}}-T\frac{\partial (f_i{f}_k+C_{jk})}{\partial {\theta }_i},\\ =T\left(\frac{\partial f_i}{\partial {\theta }_i}{f}_k+\frac{\partial f_k}{\partial {\theta }_i}{f}_j\right),\\ =T(C_{ji}{f}_k+C_{ki}{f}_j),\end{array}$$ (131)

and

$$\begin{array}{c}-\frac{{\partial }^2\log p(\overline{\mathbf{\text{r}}},\overline{\mathbf{\text{R}}})}{\partial {\Theta }_{ij}\partial {\Theta }_{kl}}=T\frac{\partial (f_i{f}_j+C_{ij})}{\partial {\Theta }_{kl}},\\ =\left(\frac{\partial f_i}{\partial {\Theta }_{kl}}{f}_j+\frac{\partial f_j}{\partial {\Theta }_{kl}}{f}_i+\frac{\partial C_{ij}}{\partial {\Theta }_{kl}}\right),\\ =T(2C_{ik}{f}_l{f}_j+2C_{jk}{f}_l{f}_i+2C_{ik}{C}_{lj}).\end{array}$$ (132)

where we used Eqs. 58 and 59. We compute the Fisher information with respect to stimulus s by the following equation,

$$\begin{array}{l}{I}_F(s)=-{\displaystyle \sum _{i,j}\frac{{\partial }^2\log p(\overline{\mathbf{\text{r}}},\overline{\mathbf{\text{R}}})}{\partial {\theta }_i\partial {\partial }_j}}\frac{\partial {\theta }_i(s)}{\partial s}\frac{\partial {\theta }_j(s)}{\partial s}\\ -2{\displaystyle \sum _{i,j,k}\frac{{\partial }^2\log p(\overline{\mathbf{\text{r}}},\overline{\mathbf{\text{R}}})}{\partial {\theta }_i\partial {\Theta }_{jk}}}\frac{\partial {\theta }_i(s)}{\partial s}\frac{\partial {\Theta }_{jk}(s)}{\partial s}\\ -{\displaystyle \sum _{i,j,k,l}\frac{{\partial }^2\log p(\overline{\mathbf{\text{r}}},\overline{\mathbf{\text{R}}})}{\partial {\Theta }_{ij}\partial {\Theta }_{kl}}}\frac{\partial {\Theta }_{ij}(s)}{\partial s}\frac{\partial {\Theta }_{kl}(s)}{\partial s}.\end{array}$$ (133)

By using Eqs. 130–132, we have

$$\begin{array}{l}-{\displaystyle \sum _{i,j}\frac{{\partial }^2\log p(\overline{\mathbf{\text{r}}},\overline{\mathbf{\text{R}}})}{\partial {\theta }_i\partial {\theta }_j}\frac{\partial {\theta }_i(s)}{\partial s}\frac{\partial {\theta }_j(s)}{\partial s}}\\ =T{\displaystyle \sum _{i,j,k,l}{C}_{ij}(C_{ik}^{-1}{f}_k^{\prime }+{C^{\prime }}_{ik}^{-1}fk)(C_{jl}^{-1}{f^{\prime }}_l+{C^{\prime }}_{jl}^{-1}{f}_l),}\\ =T({{\mathbf{\text{f}}}^{\prime }}^T{\mathbf{\text{C}}}^{-1}{\mathbf{\text{f}}}^{\prime }+2{\mathbf{\text{f}}}^T{{\mathbf{\text{C}}}^{\prime }}^{-1}\mathbf{\text{C}}{\mathbf{\text{C}}}^{-1}{\mathbf{\text{f}}}^{\prime }+{\mathbf{\text{f}}}^T{{\mathbf{\text{C}}}^{\prime }}^{-1}\mathbf{\text{C}}{{\mathbf{\text{C}}}^{\prime }}^{-1}\mathbf{\text{f}}),\end{array}$$ (134)

$$\begin{array}{l}-2{\displaystyle \sum _{i,j,k}\frac{{\partial }^2\log p(\overline{\mathbf{\text{r}}},\overline{\mathbf{\text{R}}})}{\partial {\theta }_i\partial {\Theta }_{jk}}\frac{\partial {\theta }_i(s)}{\partial s}\frac{\partial {\Theta }_{jk}(s)}{\partial s}}\\ =2T{\displaystyle \sum _{i,j,k,l}\left(C_{ji}{f}_k+C_{ki}{f}_j)(C_{il}^{-1}{f^{\prime }}_l+{C^{\prime }}_{il}^{-1}{f}_l\right)\left(-\frac{1}{2}{C^{\prime }}_{jk}^{-1}\right)},\\ =T(2{\mathbf{\text{f}}}^T{{\mathbf{\text{C}}}^{\prime }}^{-1}\mathbf{\text{C}}{\mathbf{\text{C}}}^{-1}{\mathbf{\text{f}}}^{\prime }+2{\mathbf{\text{f}}}^T{C^{\prime }}^{-1}\mathbf{\text{C}}{{\mathbf{\text{C}}}^{\prime }}^{-1}\mathbf{\text{f}}),\end{array}$$ (135)

$$\begin{array}{l}-{\displaystyle \sum _{i,j,k,l}\frac{{\partial }^2\log p(\overline{\mathbf{\text{r}}},\overline{\mathbf{\text{R}}})}{\partial {\Theta }_{ij}\partial {\Theta }_{kl}}\frac{\partial {\Theta }_{ij}(s)}{\partial s}\frac{\partial {\Theta }_{kl}(s)}{\partial s}}\\ -T{\displaystyle \sum _{i,j,k,l}(2C_{ik}{f}_l{f}_j+2C_{jk}{f}_l{f}_i+2C_{ik}{C}_{lj})(-\frac{1}{2}{C^{\prime }}_{ij}^{-1})(-\frac{1}{2}{C^{\prime }}_{kl}^{-1})},\\ =T\left(\frac{1}{2}\text{Tr}\left[{\mathbf{\text{C}}}^{\prime }{\mathbf{\text{C}}}^{-1}{\mathbf{\text{C}}}^{\prime }{\mathbf{\text{C}}}^{-1}\right]+{\mathbf{\text{f}}}^T{{\mathbf{\text{C}}}^{\prime }}^{-1}\mathbf{\text{C}}{{\mathbf{\text{C}}}^{\prime }}^{-1}\mathbf{\text{f}}\right).\end{array}$$ (136)

Taken together, we can obtain Eq. 64. By similar computations, we can also obtain Eqs. 70–73.

Eqs. 88–91 can be derived as follows,

$$\begin{array}{c}-\frac{{\partial }^2\log p({\overline{r}}_1,{\overline{r}}_2,{\overline{R}}_{12})}{\partial {\theta }_i\partial {\theta }_j}\\ =T\frac{\partial }{\partial {\theta }_j}\left(\frac{1}{Z}{\displaystyle \sum _{r_1,r_2=0,1}}{r}_i\exp \left({\displaystyle \sum _{i=1}^2{\theta }_i(s)r_i+\Theta (s)r_1{r}_2}\right)\right)\\ =T\left(\langle r_i{r}_j\rangle -\langle r_i\rangle \langle r_j\rangle \right).\end{array}$$ (137)

$$\begin{array}{c}-\frac{{\partial }^2\log p({\overline{r}}_1,{\overline{r}}_2,{\overline{R}}_{12})}{\partial {\Theta }^2}\\ =T\frac{\partial }{\partial \Theta }\left(\frac{1}{Z}{\displaystyle \sum _{r_1,r_2=0,1}}{r}_1{r}_2\exp \left({\displaystyle \sum _{i=1}^2}{\theta }_i(s)r_i+\Theta (s)r_1{r}_2\right)\right)\\ =T\left(\langle r_1{r}_2\rangle -{\langle r_1{r}_2\rangle }^2\right),\end{array}$$ (138)

$$\begin{array}{l}-\frac{{\partial }^2\log p({\overline{r}}_1,{\overline{r}}_2,{\overline{R}}_{12})}{\partial {\theta }_i\partial \Theta }\\ =T\frac{\partial }{\partial {\theta }_i}\left(\frac{1}{Z}{\displaystyle \sum _{r_1,r_2=0,1}}{r}_1{r}_2\exp \left({\displaystyle \sum _{i=1}^2{\theta }_i(s)r_i+\Theta (s)r_1{r}_2}\right)\right)\\ =T\left(\langle r_1{r}_2\rangle -\langle r_i\rangle \langle r_1{r}_2\rangle \right).\end{array}$$ (139)