Error-Based Analysis of Optimal Tuning Functions Explains Phenomena Observed in Sensory Neurons

Abstract

Biological systems display impressive capabilities in effectively responding to environmental signals in real time. There is increasing evidence that organisms may indeed be employing near optimal Bayesian calculations in their decision-making. An intriguing question relates to the properties of optimal encoding methods, namely determining the properties of neural populations in sensory layers that optimize performance, subject to physiological constraints. Within an ecological theory of neural encoding/decoding, we show that optimal Bayesian performance requires neural adaptation which reflects environmental changes. Specifically, we predict that neuronal tuning functions possess an optimal width, which increases with prior uncertainty and environmental noise, and decreases with the decoding time window. Furthermore, even for static stimuli, we demonstrate that dynamic sensory tuning functions, acting at relatively short time scales, lead to improved performance. Interestingly, the narrowing of tuning functions as a function of time was recently observed in several biological systems. Such results set the stage for a functional theory which may explain the high reliability of sensory systems, and the utility of neuronal adaptation occurring at multiple time scales.

Introduction

Behaving organisms often exhibit optimal, or nearly optimal performance (e.g., Jacobs, 1999; Ernst and Banks, 2002; Jacobs et al., 2009), despite physiological constraints and noise which is inherent to both the environment and the neural activity. Experimental work performed over the past few years has suggested that one means by which such effective behavior is achieved is through adaptation to the environment occurring across multiple time scales (e.g., milliseconds – Wang et al., 2005; seconds – Dean et al., 2008; minutes – Pettet and Gilbert, 1992). A major challenge pertaining to such results relates to the construction of a coherent theoretical framework within which the efficacy of such adaptation processes can be motivated and assessed. We believe that a carefully articulated reverse engineering perspective (see Marom et al., 2009), based on the utilization of well founded engineering principles, can yield significant insight about these phenomena. More specifically, such approaches can explain how the environmental state may be effectively estimated from the neural spike trains and from prior knowledge through a process of neural decoding. Within this context, many psychophysical and neurophysiological results can be explained by Bayesian models, suggesting that organisms may indeed be employing Bayesian calculations in their decision-making (Knill and Richards, 1996; Knill and Pouget, 2004; Rao, 2004; Ma et al., 2006). Optimal real-time decoding methods have been proposed over the past decade for static (Zemel et al., 1998; Deneve et al., 2001; Pouget et al., 2002, 2003; Averbeck et al., 2006; Ma et al., 2006; Beck et al., 2008) and dynamic (Twum-Danso and Brockett, 2001; Eden et al., 2004; Beck and Pouget, 2007; Huys et al., 2007; Pitkow et al., 2007; Deneve, 2008; Bobrowski et al., 2009) environments. While much of this work has been performed in the context of sensory systems, it has also been widely used in the context of higher brain regions (e.g., hippocampus – Barbieri et al., 2004; Eden et al., 2004), and may therefore be relevant across multiple physiological levels.

Given the well developed theory of Bayesian decoding, an intriguing follow-up question relates to the properties of optimal encoding methods, namely determining the properties of a neural population in the sensory layer that optimizes performance, subject to physiological constraints. Sensory neurons are often characterized by their tuning functions (sometimes referred to as “tuning curves”– e.g., Korte and Rauschecker, 1993; Anderson et al., 2000; Brenner et al., 2000; Dragoi et al., 2000; Harper and McAlpine, 2004), which quantify the relationship between an external stimulus and the evoked activities of each neuron, typically measured by the probability, or frequency, of emitting spikes. From an ecological point of view it is speculated that optimal tuning functions are not universal, but rather adapt themselves to the specific context and to the statistical nature of the environment. The neurophysiological literature offers much experimental evidence for neurons in many areas, which respond to changes in the statistical attributes of stimuli by modulating their response properties (e.g., Pettet and Gilbert, 1992; Brenner et al., 2000; Dragoi et al., 2000; Dean et al., 2005; Hosoya et al., 2005).

Theoretical studies of optimal tuning functions hinge on the notion of optimality, which requires the definition of an appropriate cost function. Arguably, a natural cost function is some measure of distance between the true and the estimated environmental state. In this context a reasonable choice is the mean squared error (MSE), or the average Euclidean distance between the two signals. While other distance measures can be envisaged, and should indeed be studied in appropriate contexts, there are many advantages to using this specific measure, not least of which is its relative analytic tractability. The optimality criterion then becomes minimal MSE (MMSE), where it is assumed that the spike trains generated by the sensory cells are later processed by an optimal decoder. Unfortunately, even in this relatively simple setting, analytical examination of the dependence of the MMSE on the tuning functions is infeasible in the general case, and many researchers draw conclusions about optimality of tuning functions from lower bounds on the MSE, especially those which make use of Fisher information (Seung and Sompolinsky, 1993; Pouget et al., 1999; Zhang and Sejnowski, 1999; Eurich et al., 2000; Harper and McAlpine, 2004; Johnson and Ray, 2004; Seriès et al., 2004; Lánsky and Greenwood, 2005; Brown and Bäcker, 2006; Toyoizumi et al., 2006). Unfortunately, in many cases reaching conclusions based on, often loose, bounds can lead to very misleading results, which stand in distinct opposition to predictions arising from analyzing the MMSE itself (see Discussion). Along different lines, other researchers consider information theoretic quantities (Brunel and Nadal, 1998; Panzeri et al., 1999; McDonnell and Stocks, 2008; Geisler et al., 2009; Nikitin et al., 2009), attempting to determine conditions under which the maximal amount of information is conveyed, subject to various constraints. However, a direct relation (as opposed to bounds) between such quantities and the MSE has only been established in very specific situations (Duncan, 1970; Guo et al., 2005) and does not hold in general. As far as we aware, Brunel and Nadal (1998) were the first to obtain an optimal width in the MMSE sense, for triangular tuning functions in a special case where the entire population emits a single spike. Interestingly, they showed that the error based result differs from the one based on mutual information, although a qualitative similarity was noted. Other notable exceptions are the work of Bethge et al. (2002, 2003), who employed Monte Carlo simulations in order to assess the MMSE directly in specific scenarios, and the work of Bobrowski et al. (2009), who computed the MMSE analytically to find optimal tuning functions in a concrete setting. We note that in the context of two-dimensional rate-based coding, MMSE-optimal receptive fields were investigated by Liu et al. (2009), who focused on the relationship between ganglion cell mosaics and receptive field shapes irregularities.

In this paper we directly address the issue of MMSE-optimal tuning functions, using well-justified approximations which lead to explicit analytic results. We examine various scenarios, including some that were not previously treated, and make novel predictions that can be tested experimentally. We begin in Section “Fisher-Optimal Width” by analyzing optimality in terms of Fisher information and demonstrate why drawing qualitative and quantitative conclusions based on bounds can be misleading. In fact, bound-based predictions can sometimes be diametrically opposed to the predictions based on the true error. This notion is of great importance due to the very prevalent use of approximations and bounds on the true error. We then move in Section “MMSE-Based Optimal Width” to discuss the implications of directly minimizing the MMSE, focusing on the effects of noise and multimodality. The advantages of dynamic real-time modification of tuning function properties are analyzed in Section “Dynamic Optimal Width”, and concrete experimental predictions are made. In fact, some of these predictions have already been observed in existing experimental data. Specifically, we predict that neuronal tuning functions possess an optimal width, which increases with prior uncertainty and environmental noise, and decreases with the decoding time window. The results of the paper are summarized and discussed in Section “Discussion”, and the mathematical details appear in Section “Materials and Methods”.

Materials and Methods

We investigate the problem of neural encoding and decoding in a static environment. More formally, consider an environment described by a random variable X, taking values in some space χ, and characterized by a probability density function p(·) (in order to simplify notation, we refrain from indexing probability distributions with the corresponding random variable, e.g., p_X(·), as the random variable will be clear from the context). In general, X may represent a stochastic dynamic process (e.g., Bobrowski et al., 2009) but we limit ourselves in this study to the static case. In typical cases X may be a random vector, e.g., the spatial location of an object, the frequency and intensity of a sound and so on. More generally, in the case of neurons in associative cortical regions, X can represent more abstract quantities. Suppose that X is sensed by a population of M sensory neurons which emit spike trains corresponding to the counting processes ${\mathbf{\text{N}}}_t=\{N_t^1,\dots ,N_t^M\},$ where $N_t^m$ represents the number of spikes emitted by cell m up to time t. Denote by λ_m(·) the tuning function of the m-th sensory cell. We further assume that, given the input, these spike trains are conditionally independent Poisson processes, namely

$$N_t^m|X\sim \text{Pois}\left({\lambda }_m\left(X\right)t\right),(m=1,\dots ,M),$$

implying that $P(N_t^m=n|X)=e^{-{\lambda }_m(X)t}{({\lambda }_m(X)t)}^n/n!$ (in a more general case of dynamic tuning functions {λ_m(t, ·)}, with which we deal later, the parameter of the Poisson distribution is ${\int }_0^t{\text{λ}}_m(s,X)ds$ – see Bobrowski et al., 2009). Following this encoding process, the goal of an optimal decoder is to find the best reconstruction of the input X, based on observations of the M spike trains N_t. In this paper we focus on the converse problem, namely the selection of tuning functions for encoding, that facilitate optimal decoding of X.

We formulate our results within a Bayesian setting, where it is assumed that the input signal X is drawn from some prior probability density function (pdf) p(·), and the task is to estimate the true state X, based on the prior distribution and the spike trains N_t observed up to time t. For any estimator $\widehat{X}$ we consider the mean squared error $\text{MSE}(\widehat{X})=\mathbb{E}{[(X-\widehat{X})]}^2,$, where the expectation is taken over both the observations N_t and the state X. It is well known that ${\widehat{X}}^{\text{opt}},$, the estimator minimizing the MSE, is given by the conditional mean 𝔼[X|N_t] (e.g., Van Trees, 1968), which depends on the parameters defining the tuning functions {λ_m(x)}.

As a specific example, assuming that the tuning functions are Gaussians with centers c₁, …, c_M and widths α₁, …, α_M, the estimator $\widehat{X}$ and the corresponding MMSE, depend on θ = (c₁, …, c_M, α₁, …, α_M). The optimal values of the parameters are then given by a further minimization process,

$${\mathbf{\theta }}^{\text{opt}}=\underset{\theta }{\arg \min }\{\text{MMSE(}\theta \text{)}\}.$$

In other words, θ^opt represents the tuning function parameters which lead to minimal reconstruction error. In this paper we focus on fixed centers of the tuning functions, and study the optimal widths of the sensory tuning functions that minimize MMSE(θ).

Optimal decoding of neural spike trains

As stated before, the optimal estimator minimizing the MSE is given by ${\widehat{X}}^{\text{opt}}\left({\mathbf{\text{N}}}_t\right)=\mathbb{E}[X|{\mathbf{\text{N}}}_t].$. This estimator can be directly computed using the posterior distribution, obtained from Bayes’ theorem

$$p(x|{\mathbf{\text{N}}}_t)=\frac{p(x)P({\mathbf{\text{N}}}_t|x)}{P({\mathbf{\text{N}}}_t)}=Kp(x)e^{-{\displaystyle \sum _{m=1}^M{\lambda }_m(x)t}}{\displaystyle \prod _{m=1}^M{({\lambda }_m(x)t)}^{N_t^m}},$$ (1)

where K is a normalization constant. We comment that here, and in the sequel, we use the symbol K to denote a generic normalization constant. Since such constants will play no role in the analysis, we do not bother to distinguish between them. For analytical tractability we restrict ourselves to the family of Gaussian prior distributions $X\sim p(\cdot )=\mathcal{N}({\mu }_x,{\sigma }_x^2),$ and consider Gaussian tuning functions

$${\lambda }_m(x)={\lambda }_{\max }{e}^{-\frac{{(x-c_m)}^2}{2{\alpha }_m^2}},(m=1,\cdots ,M)$$ (2)

which in many cases set a fair approximation to biological tuning functions (e.g., Anderson et al., 2000; Pouget et al., 2000). From Eq. 1 we have

$$p\text{(x|}{\mathbf{\text{N}}}_t\text{)=}K{e}^{-\frac{{(x-{\mu }_x)}^2}{2{\sigma }_x^2}}{e}^{-{\displaystyle \sum _{m=1}^M{\lambda }_m(x)t}}{e}^{-{\displaystyle \sum _{m=1}^M\frac{{(x-c_m)}^2}{2{\alpha }_m^2}}{N}_t^m}.$$ (3)

In the multi-dimensional case X ∼ 𝓝(μ_x, Σ_x) and the Gaussian tuning functions are of the form

$${\lambda }_m(x)={\lambda }_{\max }{e}^{-\frac{1}{2}{(\text{x}-{\text{c}}_m)}^{\text{T}}{A}_m^{-1}({\text{x-c}}_m)},(m=1,\dots ,M).$$ (4)

Bayesian cramér–rao bound

Since the MMSE is often difficult to compute, an alternative approach which has been widely used in the neural computation literature is based on minimizing a more tractable lower bound on the MSE, related to Fisher information of the population of sensory neurons. The Fisher information is defined by

$$\mathcal{J}(X)=\mathbb{E}\left[{\left(\frac{\partial }{\partial X}\text{ln}P({\mathbf{\text{N}}}_t|X)\right)}^2\right].$$ (5)

If X is deterministic, as in the classic (non-Bayesian) case, and $B(X)\triangleq \mathbb{E}[(\widehat{X}-X)]$ is the estimation bias, then the error variance of any non-Bayesian estimator $\widehat{X}({\mathbf{\text{N}}}_t)$ is lower bounded by ${(1+\frac{d}{dX}B(X))}^2{\mathcal{J}}^{-1}(X),$ which is the Cramér–Rao bound (Van Trees, 1968). The extension to the Bayesian setting, where X is a random variable with probability density function p(·), often referred to as the Bayesian Cramér–Rao bound (BCRB), states that

$$\mathbb{E}\left[{\left(X-\widehat{X}\right)}^2\right]\ge \frac{1}{\mathbb{E}[\mathcal{J}(X)]+ℐ(p(x))},$$ (6)

where $ℐ(p(x))=\mathbb{E}[{(\frac{d}{dX}\text{ln}p(X))}^2]$ and the bounded quantity is the MSE of any estimator (section 1.1.2 in Van Trees and Bell, 2007). Note that the expectation in Eq. 6 is taken with respect to both the observations N_t and the state X, in contrast to the non-Bayesian case where the unknown state X is deterministic. Note that in the asymptotic limit, t → ∞ the prior term, ℐ(p(x)), can be neglected with respect to the expected Fisher information, so that the asymptotic BCRB is 1/𝔼[𝒥(X)].

The second term in the denominator of Eq. 6 is independent of the tuning functions, and in the case of a univariate Gaussian prior is given by

$$ℐ(p(x))=\mathbb{E}\left[\frac{{\left(x-{\mu }_x\right)}^2}{{\sigma }_x^4}\right]=\frac{1}{{\sigma }_x^2}.$$

In our setting, the expected value of the population's Fisher information is given by

$$\mathbb{E}\left[\mathcal{J}(X)\right]={\lambda }_{\max }t{\displaystyle \sum _{m=1}^M\frac{e^{-\frac{{\left(c_m-{\mu }_x\right)}^2}{2\left({\alpha }_m^2+{\sigma }_x^2\right)}}}{{\alpha }_m{\left({\alpha }_m^2+{\sigma }_x^2\right)}^{5/2}}\left[{\sigma }_x^2\left({\sigma }_x^2+{\alpha }_m^2\right)+{\alpha }_m^2{\left(c_m-{\mu }_x\right)}^2\right]}$$ (7)

(see Section “Expectation of Population Fisher Information” in Appendix).

In the multi-dimensional case the right hand side of Eq. 6 is replaced by the inverse of the matrix J = 𝔼[𝒥(X)] + ℐ(p(x)), where

$$\begin{array}{c}\mathcal{J}(\mathbf{\text{X}})=\mathbb{E}\left[(\nabla \text{\hspace{0.17em}ln\hspace{0.17em}}P(N_t|\mathbf{\text{X}})){(\nabla \text{\hspace{0.17em}ln\hspace{0.17em}}P(N_t|\mathbf{\text{X}}))}^T\right],\\ ℐ(p(\mathbf{\text{x}}))=\mathbb{E}\left[(\nabla \text{\hspace{0.17em}ln\hspace{0.17em}}P(\mathbf{\text{X}})){(\nabla \text{\hspace{0.17em}ln\hspace{0.17em}}P(\mathbf{\text{X}}))}^T\right],\end{array}$$

and the left hand side is replaced by the error correlation matrix $\mathbf{\text{R\hspace{0.17em}}}=\mathbb{E}\left[\right(\mathbf{\text{X}}-\widehat{\mathbf{\text{X}}}){(\mathbf{\text{X}}-\widehat{\mathbf{\text{X}}})}^T\text{].}$ The interpretation of the resulting inequality is twofold. First, the matrix R − J⁻¹ is non-negative definite and second,

$$\mathbb{E}\left[{\left({\mathbf{\text{X}}}_k-{\widehat{\mathbf{\text{X}}}}_k\right)}^2\right]\ge {({\mathbf{J}}^{-1})}_{kk},(k=1,\cdots ,d),$$ (8)

where d is the dimension of X (Van Trees, 1968).

It is straightforward to show that in this case

$$\mathcal{I}(p(x))=\mathbb{E}\left[{\sum }_x^{-1}(\mathbf{X}-{\mu }_x){(\mathbf{X}-{\mu }_x)}^T{\sum }_x^{-1}\right]={\sum }_x^{-1}.$$

The expression for 𝔼[𝒥(X)] is similar to Eq. 7 and is derived in Section “Expectation of Population Fisher Information” in Appendix, yielding

$$\begin{array}{c}\mathbb{E}\left[\mathcal{J}(\mathbf{X})\right]={\lambda }_{\max }t{\displaystyle \sum _{m=1}^M{e}^{-\frac{1}{2}\xi m}}\frac{\sqrt{\left|A_m\right|A_m^{-1}}}{\left|A_m+{\sum }_x\right|}{\left(A_m+{\sum }_x\right)}^{-2}\\ \times \left[{\sum }_x(A_m+{\sum }_x)+A_m({\text{c}}_m-{\mu }_x){({\text{c}}_m-{\mu }_x)}^T\right],\end{array}$$ (9)

where ξ_m = (c_m − µ_x)^T(A_m + Σ_x)⁻¹(c_m − m_x).

Analytical derivation of the MMSE

We now proceed to establish closed form expressions for the MMSE. In order to maintain analytical tractability we analyze the simple case of equally spaced tuning functions (c_m+1 − c_m ≡ Δc) with uniform width (α_m ≡ α). When the width is of the order of Δc or higher, and M is large enough with respect to σ_x, the mean firing rate of the entire population is practically uniform for any “reasonable” X. As shown in Section “Uniform Mean Firing Rate of Population” in Appendix, the sum Σ_m λ_m(x) is well approximated in this case by $\lambda (\alpha )=\sqrt{2\pi }(\alpha /△c){\lambda }_{\max }.$ Under these conditions the posterior (Eq. 3) takes the form

$$p\left(x|{\mathbf{N}}_t\right)=K\exp \left\{-\frac{1}{2}\left(\frac{{\left(x-{\mu }_x\right)}^2}{{\sigma }_x^2}+{\displaystyle \sum _{m=1}^M\frac{{\left(x-c_m\right)}^2}{{\alpha }^2}}{N}_t^m\right)\right\},$$ (10)

implying that [X|N_t] ∼𝒩$\left(\widehat{\mu },{\widehat{\sigma }}^2\right),$ where

$${\widehat{\sigma }}^2={\left(\frac{1}{{\sigma }_x^2}+\frac{{\sum }_m{N}_t^m}{{\alpha }^2}\right)}^{-1};\widehat{\mu }={\widehat{\sigma }}^2\left(\frac{{\mu }_x}{{\sigma }_x^2}+\frac{{\sum }_m{c}_m{N}_t^m}{{\alpha }^2}\right).$$

Recalling that the spike trains are independent Poisson processes, namely ${\Sigma }_m{N}_t^m|X~\text{Pois(}{\Sigma }_m{\lambda }_m\text{(}X\text{)}t\text{),}$ and in light of the above approximation, $Y\triangleq {\Sigma }_m{N}_t^m$ is independent of X and Y ∼ Pois(αt_eff), where $t_{\text{eff}}\triangleq \frac{\sqrt{2\pi }}{\Delta c}{\lambda }_{\max }t.$ Since ${\widehat{X}}^{\text{opt}}({\mathbf{N}}_t)=\mathbb{E}[X|{\mathbf{N}}_t]\equiv \widehat{\mu },$ the MMSE is given by

$$\text{MMSE}=\mathbb{E}\left[{\left(X-{\widehat{X}}^{\text{opt}}\right)}^2\right]=\mathbb{E}\left[\mathbb{E}\left[{(X-\widehat{\mu })}^2|{\mathbf{N}}_t\right]\right]\equiv \mathbb{E}[{\widehat{\sigma }}^2],$$

and thus we get an explicit expression for the normalized MMSE,

$$\frac{\text{MMSE}}{{\sigma }_x^{\text{2}}}=\mathbb{E}\left[\frac{1}{1+Y{\sigma }_x^{\text{2}}/{\alpha }^2}\right]={\displaystyle \sum _{k=0}^{\infty }\frac{1}{1+k{\sigma }_x^{\text{2}}/{\alpha }^2}}\frac{{(\alpha t_{\text{eff}})}^k{e}^{-\alpha t_{\text{eff}}}}{k!}$$ (11)

In the multivariate case we assume that the centers of the tuning functions form a dense multi-dimensional lattice with equal spacing Δc along all axes, and Eq. 10 becomes

$$\begin{array}{l}p(\mathbf{\text{x|}}{\mathbf{N}}_t)=K\exp \{-\frac{1}{2}({(\mathbf{\text{x}}-{\mu }_{\mathbf{\text{x}}})}^T{\sum }_{\mathbf{\text{x}}}^{-1}(\mathbf{\text{x}}-{\mu }_{\mathbf{\text{x}}})\\ +{\displaystyle \sum _{m=1}^M{N}_t^m{(\mathbf{\text{x}}-{\mathbf{\text{c}}}_m)}^T{A}^{-1}(\mathbf{\text{x}}-{\mathbf{\text{c}}}_m)})\},\end{array}$$ (12)

namely [X|N_t] ∼ 𝒩$(\widehat{\mu },\widehat{\Sigma }),$ where

$$\widehat{\Sigma }={\left({\sum }_x^{-1}+A^{-1}{\displaystyle {\sum }_m{N}_t^m}\right)}^{-1};\widehat{\mu }=\widehat{\Sigma }\left({\sum }_x^{-1}{\mu }_x+A^{-1}{\displaystyle {\sum }_m{\mathbf{\text{c}}}_m{N}_t^m}\right).$$ (13)

In this case $Y\triangleq {\Sigma }_m{N}_t^m~\text{Pois(}\sqrt{\left|A\right|}{t}_{\text{eff}}^d\text{),}{t}_{\text{eff}}^d\triangleq {(\sqrt{2\pi }/\Delta c)}^d{\lambda }_{\max }t$(see Section “Uniform Mean Firing Rate of Population” in Appendix), and

$$\begin{array}{c}\text{MMSE}=\mathbb{E}\left[{\Vert \mathbf{\text{X}}-{\widehat{\mathbf{\text{X}}}}^{opt}\Vert }^2\right]=\mathbb{E}\left[\mathbb{E}\left[{(\mathbf{\text{X}}-\widehat{\mu })}^T(\mathbf{\text{X}}-\widehat{\mu })|{\mathbf{N}}_t\right]\right]\\ =\mathbb{E}\left[\mathbb{E}\left[{\displaystyle \sum _{k=1}^d{({\mathbf{\text{X}}}_k-{\widehat{\mu }}_k)}^2}|N_t\right]\right]={\displaystyle \sum _{k=1}^d\mathbb{E}\left[{\widehat{\sum }}_{kk}\right]}.\end{array}$$ (14)

Incorporation of noise

Suppose that due to environmental noise the value of X is not directly available to the sensory system, which must respond to a noisy version of the environmental state –$\tilde{X}.$ For simplicity we assume additive Gaussian noise:$\tilde{X}=X+W,W~\mathcal{N}(0,{\sigma }_w^2).$ For a dense layer of sensory neurons the joint posterior distribution of the environmental state and noise is

$$\begin{array}{c}p(x,w|{\mathbf{N}}_t)=\frac{p(x,w)P\left({\mathbf{N}}_t|x,w\right)}{P\left({\mathbf{N}}_t\right)}=\frac{p(x)p(w)P\left({\mathbf{N}}_t|x+w\right)}{P\left({\mathbf{N}}_t\right)}\\ =K{e}^{-\frac{{\left(x-{\mu }_x\right)}^2}{2{\sigma }_x^2}}{e}^{-\frac{w^2}{2{\sigma }_{\text{w}}^2}}{e}^{-{\displaystyle \sum _{m=1}^M\frac{{\left(x+w-c_m\right)}^2}{2{\alpha }^2}{N}_t^m}}\\ =K{e}^{-\frac{1}{2{\widehat{\sigma }}_w^2}{\left(w-{\widehat{\sigma }}_w^2{\displaystyle \sum _{m=1}^M\frac{\left(c_m-x\right)}{{\alpha }^2}}{N}_t^m\right)}^2}\\ \times e^{-\frac{1}{2}\left\{\frac{{\left(x-{\mu }_x\right)}^2}{{\sigma }_x^2}+{\displaystyle \sum _{m=1}^M\frac{{\left(x-c_m\right)}^2}{{\alpha }^2}{N}_t^m-{\widehat{\sigma }}_w^2{\left({\displaystyle \sum _{m=1}^M\frac{\left(c_m-x\right)}{{\alpha }^2}}{N}_t^m\right)}^2}\right\}},\end{array}$$

which is Gaussian with respect to W, where ${\widehat{\sigma }}_w^2={({\sigma }_w^{-2}+{\alpha }^{-2}{\Sigma }_m{N}_t^m)}^{-1}.$ Integration over w gives the marginal distribution

$$p(x|{\mathbf{N}}_t)=K{e}^{-\frac{1}{2}\left\{\frac{{(x-{\mu }_x)}^2}{{\sigma }_x^2}+{\displaystyle \sum _{m=1}^M\frac{{(x-c_m)}^2}{{\alpha }^2}{N}_t^m-{\stackrel{\wedge }{\sigma }}_w^2{\left({\displaystyle \sum _{m=1}^M\frac{(c_m-x)}{{\alpha }^2}{N}_t^m}\right)}^2}\right\}},$$

which is Gaussian with variance

$${\widehat{\sigma }}^2={\left(\frac{1}{{\sigma }_x^2}+\frac{Y}{{\alpha }^2}-\frac{{\widehat{\sigma }}_w^2{Y}^2}{{\alpha }^4}\right)}^{-1}={\left(\frac{1}{{\sigma }_x^2}+\frac{Y}{{\alpha }^2}-\frac{{\sigma }_w^2{Y}^2}{{\alpha }^2({\alpha }^2+{\sigma }_w^{2}Y)}\right)}^{-1}.$$ (15)

where $Y={\Sigma }_m{N}_t^m\sim \text{Pois(}\alpha t_{\text{eff}}\text{)}$ as before. The MMSE is computed directly from the posterior variance:

$$\text{MMSE}=\mathbb{E}[{\widehat{\sigma }}^2]=\mathbb{E}\left[{\left(\frac{1}{{\sigma }_x^2}+\frac{Y}{{\alpha }^2+{\sigma }_w^{2}Y}\right)}^{-1}\right].$$ (16)

Note that when σ_w → ∞ the sensory information becomes irrelevant for estimating X, and indeed in such case $\text{MMSE}\to {\sigma }_x^2.$

Multisensory integration

Consider the case where the environmental state is estimated based on observations from two different sensory modalities (e.g., visual and auditory). The spike trains in each channel are generated in a similar manner to the unimodal case, but are then integrated to yield enhanced neural decoding. All quantities that are related to the first and second modalities will be indexed by v and a, respectively, although these modalities are not necessarily the visual and the auditory.

Each sensory modality detects a different noisy version of the environmental state: ${\tilde{X}}_v=X+W_v$ in the “visual” pathway and ${\tilde{X}}_a=X+W_a$ in the “auditory” pathway. The noise variables are Gaussian with zero-mean and variances ${\sigma }_{w,v}^2$ and ${\sigma }_{w,a}^2$, respectively, and independent of each other as they emerge from different physical processes. The noisy versions of the environmental state are encoded by M_v + M_a sensory neurons with conditional Poisson statistics:

$$\{\begin{array}{cc}{N}_t^{m,v}|{\tilde{X}}_v~\text{Pois}\left({\lambda }_m^v({\tilde{X}}_v)t\right),& (m=1,\dots ,M_v)\\ N_t^{m,a}|{\tilde{X}}_a~\text{Pois}\left({\lambda }_m^a({\tilde{X}}_a)t\right),& (m=1,\dots ,M_a),\end{array}$$

where the tuning functions in each modality are Gaussian with uniform width,

$$\{\begin{array}{cc}{\lambda }_m^v(x)={\lambda }_{\max }^v{e}^{-\frac{{(x-c_m^v)}^2}{2{\alpha }_v^2}},& (m=1,\dots ,M_v)\\ {\lambda }_m^a(x)={\lambda }_{\max }^a{e}^{-\frac{{(x-c_m^a)}^2}{2{\alpha }_a^2}},& (m=1,\dots ,M_a).\end{array}$$

We assume that the tuning functions are dense enough, namely the widths are not significantly smaller than the spacing between the centers, and equally spaced $(c_{m+1}^v-c_m^v\equiv \Delta c_v,c_{m+1}^a-c_m^a\equiv \Delta c_a).$

The full expression for the MMSE in this case is

$$\text{MMSE}=\mathbb{E}\left[{\left(\frac{1}{{\sigma }_x^2}+\frac{Y_v}{{\alpha }_v^2+{\sigma }_{w,v}^2{Y}_v}+\frac{Y_a}{{\alpha }_a^2+{\sigma }_{w,a}^2{Y}_a}\right)}^{-1}\right],$$ (17)

where $Y_v\triangleq {\Sigma }_m{N}_t^{m,v}~\text{Pois(}{\alpha }_v{t}_{\text{eff}}^v\text{)}$ and $Y_a\triangleq {\Sigma }_m{N}_t^{m,a}~\text{Pois(}{\alpha }_a{t}_{\text{eff}}^a\text{)}$(the derivation appears in Section “MMSE for Multisensory Integration” in Appendix).

Results

As argued in Section “Materials and Methods”, obtaining optimal tuning function parameters (or simply optimal width, in the simplified scenario presented in Section “Analytical Derivation of the MMSE”) requires computation of the MMSE for every given set of parameters, which is often not even numerically feasible. In the context of classical estimation, in order to maintain analytical tractability, the Cramér–Rao bound has been traditionally analyzed in place of the MMSE. In this text we employ a theoretical framework of optimal Bayesian estimation, in an attempt to find the optimal tuning functions that facilitate optimal estimation of X from the observations (the neural spike trains). We start by analyzing optimal width as predicted from the BCRB and proceed to scrutinize the properties of optimal width derived directly from the MMSE.

Fisher-optimal width

In a Bayesian context, it is well known (Van Trees, 1968) that for any estimator $\widehat{X},$ the MSE is lower bounded by the Bayesian Cramér–Rao lower bound (BCRB) given in Eq. 6. An interesting question is the following: considering that the MSE of any estimator (and thus the MMSE itself) is lower bounded by the BCRB, do the optimal widths also minimize the BCRB? In other words, can the BCRB be used as a proxy to the MMSE in selecting optimal widths? If this were the case, we could analyze MMSE-optimality by searching for widths that maximize the expected value of the Fisher information defined in Eq. 5. This alternative analysis would be favorable, because in most cases analytical computation of 𝔼[𝒥(X)] is much simpler than that of the MMSE, especially when the conditional likelihood is separable, in which case the population's log likelihood reduces to a sum of individual log likelihood functions. Moreover, it remains analytically tractable under much broader conditions. Unfortunately, as we show below, the answer to the above question is negative. Despite the fact that for any encoding ensemble the BCRB bounds the MMSE from below, and may even approximate it in the asymptotic limit (if, for example, 𝒥(X) is independent of X), the behavior of the two (as a function of tuning function widths) is very different, as we demonstrate below.

In terms of the BCRB the existence of an optimal set of widths critically depends on the dimensionality of the environmental state. We consider first the univariate case, and then discuss the extension to multiple dimensions.

The univariate case

In the scalar case, using the tuning functions in Eq. 2, it follows from Eq. 7 that it is best to employ infinitesimally narrow tuning functions, since 𝔼[𝒥(X)] → ∞ most rapidly when α_m → 0 for all m, although Fisher information is undefined when one of the width parameters is exactly 0. A similar result was suggested previously for non-Bayesian estimation schemes (Seung and Sompolinsky, 1993; Zhang and Sejnowski, 1999; Brown and Bäcker, 2006), but within the framework that was employed there the result was not valid, as we elucidate in Section “Discussion”. Why does Fisher information predict that “narrower is better”? As a tuning function narrows, the probability that the stimulus will appear within its effective receptive field and evoke any activity decreases, but more and more information could be extracted about the stimulus if the cell did emit a spike. Evidently, in the limit α_m → 0 the gain in information dominates the low probability and BCRB → 0; however, this is precisely the regime where the bound vanishes and is therefore trivial. This result is in complete contrast with the MSE, which is minimal for non-zero values of α as we show below. More disturbingly, as argued in Section “The Univariate Case”, the performance is in fact the worst possible when α → 0.

We note that Bethge et al. (2002) were the first to address the shortcomings of lower bounds on the MSE in a Bayesian setting. These authors stressed the fact that performance bounds are usually tight only in the limit of infinitely long time windows, and cannot be expected to reflect the behavior of the bounded quantities for finite decoding times. They observed that, for any X, the minimal conditional MSE asymptotically equals 1/𝒥(X), and by taking expectations on both sides they obtained 𝔼[1/𝒥(X)] as the asymptotic value of the MMSE, which they assessed using Monte Carlo simulations.

The multivariate case

In this case, using the d-dimensional Gaussian tuning functions (Eq. 4) characterized by the matrices (A_m), the problem becomes more complex as is clear from Eq. 9, because the tuning functions may have different widths along different axes or even non-diagonal matrices. For simplicity we focus on the case where the matrices A_m are diagonal. By definition, the MSE of any estimator is the trace of its error correlation matrix, and in accordance with Eq. 8, Fisher-optimal tuning functions minimize the trace of J⁻¹, which equals the sum of its eigenvalues. If the widths in all dimensions are vanishingly small (A_m ≡ εI where ε ≪ 1), A_m becomes negligible with respect to Σ_x $\mathbb{E}[\mathcal{J}(X)]\approx {\lambda }_{\max }t{\Sigma }_m{e}^{-\frac{1}{2}{\xi }_m}|{\Sigma }_x{|}^{-1/2}\sqrt{\left|A_m\right|}{A}_m^{-1}.$ and In 3D and in higher dimensions the matrix $\sqrt{\left|A_m\right|}{A}_m^{-1}=\text{diag}({\epsilon }^{d/2-1},\mathrm{\dots },{\epsilon }^{d/2-1})$ is close to the zero matrix, and consequently the eigenvalues of 𝔼[𝒥(X)] are vanishingly small. If all widths are large (A_m ≡ (1/ε)>I where ε ≪ 1), A_m dominates Σ_x and $\mathbb{E}[\mathcal{J}(X)]\approx {\lambda }_{\max }t\Sigma e^{-\frac{1}{2}{\xi }_m}{A}_m^{-2}[{\Sigma }_x+(c_m-{\mu }_x)$ (c_m − μ_x)^T], namely the eigenvalues of 𝔼[𝒥(X)] are still vanishingly small. This means that when the widths are too small or too large, the BCRB dictates poor estimation performance in high dimensional settings (specifically, the information gain no longer dominates the low probability in the limit of vanishing widths). Therefore, there exists an optimal set of finite positive widths that minimize the BCRB. As an exception, in 2D the eigenvalues of $\sqrt{\left|A_m\right|}{A}_m^{-1}$ are finite when the widths approach 0, and 𝔼[𝒥(X)] cannot be said to have infinitesimally small eigenvalues. In fact, numerical calculations for radial prior distribution reveal that in 2D “narrower is still better”, as in the univariate case.

An interesting observation concerning the BCRB based optimal widths is that they cannot depend on the available decoding time t, since 𝔼[𝒥(X)] is simply proportional to t. As we will show, an important consequence of the present work is to show that the optimal tuning function widths, based on minimizing the MMSE, depend explicitly on time. In this sense, choosing optimal tuning functions based on the BCRB leads to a qualitatively incorrect prediction. We return to this issue in Section “Discussion”, where further difficulties arising from the utilization of lower bounds are presented.

MMSE-based optimal width

Having discussed predictions about optimal widths based on BCRB minimization, we wish to test their reliability by finding optimal widths through direct MMSE minimization. As explained in Section “Materials and Methods”, in order to facilitate an analytic derivation of the MMSE, we examine here the case of dense equally spaced tuning functions with uniform widths. We consider first the univariate case and then proceed to the multivariate setting. As far as we are aware, Bethge et al. (2002) were the first to observe, using Monte Carlo simulations, that the MSE-optimal width of tuning functions may depend on decoding time.

The univariate case

Based on Eq. 11, we plot the normalized MMSE as a function of the width (Figure 1A) for different combinations of effective decoding time and prior standard deviation. The effective time, t_eff, defined in Section “Analytical Derivation of the MMSE”, is proportional to the time over which spikes accumulate. The existence of an optimal width, which is not only positive but also varies with σ_x and t_eff, is clearly exhibited. A similar result was first demonstrated by Bobrowski et al. (2009), but the dependence of optimal width on the two other parameters was not examined. Note that the derivation in Eq. 11 is not reliable in the vicinity of the y-axis, because the approximation of uniform population firing rate (see Analytical Derivation of the MMSE) is not valid when α → 0. Nevertheless, it can be proved that when α → 0, P(N_t = 0) → 1 and as a consequence ${\widehat{X}}^{\text{opt}}({\text{N}}_t)\to {\mu }_x$ and indeed $\text{MMSE/}{\sigma }_x^{\text{2}}\to 1.$

Figure 1.

(A) The normalized MMSE of a dense population of Gaussian tuning functions against tuning function width, for different values of effective decoding time (shortest – uppermost triplet, longest – lowermost triplet). In each triplet three values of prior standard deviation were examined –σ_x = 1 (solid), σ_x = 2 (dashed), and σ_x = 3 (dotted), where the optimal width is indicated by squares, circles and diamonds, respectively. (B) The normalized MMSE for different values of effective decoding time (σ_x = 1), obtained theoretically (dashed lines, optimal width indicated by circles) and in simulation (solid lines, optimal width indicated by squares). Simulation parameters: N = 251, X ∈ [−4,4], $P(\text{x})\propto \mathcal{N}(0,{\sigma }_x^2),$ M = 250, Δc = 0.034, λ_max = 50.

The existence of an optimal width is very intuitive, considering the tradeoff between multiplicity of spikes and informativeness of observations. When tuning functions are too wide, multiple spikes will be generated that loosely depend on the stimulus, and thus the observations carry little information. When tuning functions are too narrow, an observed spike will be informative about the stimulus but the generation of a spike is very unlikely. Thus, an intermediate optimal width emerges. To verify that the results were not affected by the approximation in Eq. 10, we compare in Figure 1B the theoretical MMSE with the MMSE obtained in simulations, which make no assumptions about uniform population firing rate. As can be seen in the figure, the two functions coincide, and in particular, the optimal widths are nearly identical. Recall that at each moment it is assumed that α is not significantly smaller than Δc (this is required for the uniform population firing rate approximation). For the values obtained in Figure 1A this implies Δc smaller than about 0.5. Obviously, the validity of Eq. 10 would break down for sufficiently long decoding time windows. For instance, when t_eff > 9(1.715Δc^-1(i.e., when α^opt < 0.583Δc) the relative ripple of the population firing rate rises above 0.5% (see Section “Uniform Mean Firing Rate of Population” in Appendix). Nevertheless, as can be observed in Figure 1B, the theoretical approximation for the MMSE in Eq. 11 holds even for vanishing width, despite the non-uniformity in population firing rate (this appears more clearly in Figure 8).

We now turn to analyze the effect of the two parameters, σ_x and t_eff, in Eq. 11. The effective time is a measure for the number of spikes generated by the neural population, since it is proportional to λ_max(where λ_max is the maximal mean firing rate of a single neuron) and also to 1/Δc (which is proportional to the population size when the population is required to “cover” a certain fraction of the stimulus space). Longer effective times increase the likelihood of spikes under all circumstances, and therefore the drawback of narrow tuning functions is somewhat mitigated, leading to a reduction in optimal width (as illustrated in Figure 2A).

Figure 2.

Optimal tuning functions width as a function of (A) effective decoding time, and (B) prior standard deviation. All curves are well-fitted by the same function: $\widehat{\alpha }(t_{\text{eff}},{\sigma }_x)={(\frac{1}{9}{t}_{\text{eff}}+{\sigma }_x^{-1})}^{-1}.$

The prior standard deviation reflects the initial uncertainty in the environment: the larger it is – the less is known a priori about the identity of the stimulus. Confidence about stimulus identity (associated with small MMSE) may stem from either a deterministic environment or from numerous observations for which the likelihood is very sharp. When the environment is less certain, many more observations are needed in order to obtain a narrow posterior distribution leading to smaller MMSE, and thus the optimal width increases (as illustrated in Figure 2B). Consider the condition discussed above for the uniform rate approximation to hold with high accuracy, namely $t_{\text{eff}}>9(1.715\Delta c^{-1}-{\sigma }_x^{-1})$. Assuming, for example, that σ_x ≥ Δc, implies that t_eff ≤ 6/Δc suffices to guarantee the above condition. Under these conditions we obtain a simple empirical expression relating the optimal width to the effective time and prior uncertainty,

$${\alpha }^{\text{opt}}\approx \widehat{\alpha }(t_{\text{eff}},{\sigma }_x)={\left(\frac{t_{\text{eff}}}{\text{9}}+\frac{1}{{\sigma }_x}\right)}^{-1}.$$ (18)

Figure 2 demonstrates the quality of this relation.

Finally, an interesting limit to consider is that of short decoding times. In this case the tuning functions are broad and the uniform sum rate approximation is expected to hold accurately. More precisely, when t → 0 the Poisson random variable $Y={\Sigma }_m{N}_t^m$ converges in probability to a Bernoulli random variable with “success” probability αt_eff, in which case the expression for the normalized MMSE (Eq. 11) greatly simplifies,

$$\frac{\text{MMSE}}{{\sigma }_x^2}=\frac{1-\alpha t_{\text{eff}}}{1}+\frac{\alpha t_{\text{eff}}}{\text{1+\hspace{0.05em}}{\sigma }_x^{\text{2}}/{\alpha }^2}=\frac{{\alpha }^2+(1-\alpha t_{\text{eff}}){\sigma }_x^2}{{\alpha }^2+{\sigma }_x^2}.$$

Differentiating with respect to α and setting the result to 0 yields

$${\left[t_{\text{eff}}{\sigma }_x^2\left({\alpha }^2-{\sigma }_x^2\right)\right]}_{\alpha ={\alpha }^{\text{opt}}}=0\Rightarrow {\alpha }^{\text{opt}}={\sigma }_x.$$

Note that by taking the appropriate limit in the empirical formula (Eq. 18) we see that lim_t→0α^opt = σ_x, consistently with the above exact derivation.

Interestingly, when the width equals the prior standard deviation, the normalized MMSE can be calculated exactly for any effective time,

$$\begin{array}{c}\frac{\text{MMSE}}{{\sigma }_x^2}={\displaystyle \sum _{k=0}^{\infty }\frac{1}{1+k}\frac{{({\sigma }_x{t}_{\text{eff}})}^k{e}^{-{\sigma }_x{t}_{\text{eff}}}}{k!}}=\frac{e^{-{\sigma }_x{t}_{\text{eff}}}}{{\sigma }_x{t}_{\text{eff}}}{\displaystyle \sum _{k=0}^{\infty }\frac{{({\sigma }_x{t}_{\text{eff}})}^{k+1}}{\text{(}k+\text{1)!}}}\\ =\frac{e^{-{\sigma }_x{t}_{\text{eff}}}}{{\sigma }_x{t}_{\text{eff}}}\left(e^{{\sigma }_x{t}_{\text{eff}}}-1\right)=\frac{1-e^{-{\sigma }_x{t}_{\text{eff}}}}{{\sigma }_x{t}_{\text{eff}}},\end{array}$$

which for very short time windows converges to the optimal normalized MMSE.

The multi-dimensional case

We consider multivariate normal distributions for both the prior and the tuning functions. We start by considering a radial prior distribution and radial tuning functions, namely Gaussians with covariance matrices ${\Sigma }_x={\sigma }_x^2\mathbf{\text{I}}$ and A_m = α²I, respectively, where I is the d × d identity matrix. In this simple scenario the posterior covariance matrix in Eq. 13 becomes

$$\widehat{\Sigma }={\left(\frac{1}{{\sigma }_x^2}+\frac{Y}{{\alpha }^2}\right)}^{-1}I,$$

where $Y={\Sigma }_m{N}_t^m\sim \text{Pois}({\alpha }^d{t}_{\text{eff}}^d)=\text{Pois(}\alpha t_{\text{eff}}{(\sqrt{2\pi }\alpha \text{/}\Delta c)}^{d-1}\text{)}.$ From Eq. 14 we see that

$$\frac{\text{MMSE}}{{\sigma }_x^{\text{2}}}=\frac{{\sum }_{k=1}^d\mathbb{E}\left[{\widehat{\Sigma }}_{kk}\right]}{{\sigma }_x^2}=d\mathbb{E}\left[\frac{1}{1+Y{\sigma }_x^2/{\alpha }^2}\right].$$

As long as $\sqrt{2\pi }{\alpha }^{\text{opt}}/\Delta c>1$ the parameter of the Poisson random variable Y increases with the dimensionality d, in a manner that is equivalent to longer effective time (albeit width-dependent). Consequently, recalling (Eq. 18), the optimal width decreases with the dimensionality of the stimulus. Numerical calculations show that indeed this is the case in 2D and 3D for effective times which are not extremely long. This is in contrast with the predictions made by Zhang and Sejnowski (1999), where it was speculated that performance is indifferent to the width of the tuning functions in 2D, and improves with infinitely increasing width in higher dimensions.

When the prior covariance matrix and tuning functions shape matrix A are diagonal with, possibly different, elements ${\left\{{\sigma }_{x,k}^2\right\}}_{k=1}^d$ and ${\left\{{\alpha }_k^2\right\}}_{k=1}^d,$ respectively, the MMSE is given by

$$\text{MMSE}={\displaystyle \sum _{k=1}^d\mathbb{E}}\left[\frac{1}{{\sigma }_{x,k}^{-2}+{\alpha }_k^{-2}Y}\right],$$

where $Y={\Sigma }_m{N}_t^m~\text{Pois(}{t}_{\text{eff}}^d{\prod }_{k=1}^d{\alpha }_k\text{).}$ Since the random variable Y appears in every term in the sum, combined with different prior variances, the optimal width vector will adjust itself to the vector of prior variances, namely the optimal width will be largest (smallest) in the dimension where the prior variance is largest (smallest).

Biological implications

An important feature of the theoretical results is the ability to predict the qualitative behavior that biological tuning functions should adopt, if they are to perform optimally. From Eq. 18 we see that if the environmental uncertainty (expressed by σ_x) decreases, the tuning functions width is expected to reduce accordingly (this holds in 2D as well). This prediction provides a possible theoretical explanation for some results obtained in psychophysical experiments (Yeshurun and Carrasco, 1999). In these experiments human subjects were tested on spatial resolution tasks where targets appeared at random locations on a computer screen. In each trial the subjects were instructed to fixate on the screen center and the target was presented for a brief moment, with or without a preceding brief spatial cue marking the location of the target but being neutral with respect to the spatial resolution task. The authors found that the existence of the preceding cue improved both reaction times and success rates, concluding that spatial resolution was enhanced following the cue by reducing the size of neuronal receptive fields. Following their interpretation, we argue that the spatial cue reduces the uncertainty about the stimulus by bounding the region where the stimulus is likely to appear (i.e., σ_x is reduced). In light of our results, an optimal sensory system would then respond to the decrease in prior standard deviation by narrowing the tuning functions toward stimulus onset.

Our prediction may also relate to the neurophysiological experiments of Martinez-Trujillo and Treue (2004), who recorded the activity of direction-sensitive MT neurons in awake macaque monkeys in response to random dot patterns. These patterns appeared in two distinct regions of the visual field, one inside the neuron's receptive field and another on the opposite hemifield, but both were governed by the same general movement direction. The direction-selective tuning function was measured when the animal's attention was directed to a fixation point between these regions (“neutral”) and when its attention was directed to the other region with the same movement direction (“same”). In the “same” condition, neuronal response was enhanced near the tuning function center and suppressed away from the center with respect to the “neutral” condition, in a manner that is equivalent to a reduction in tuning function width (albeit followed by a modulation of maximal firing rate, which is fixed in our model). A possible interpretation is that attending the random dot pattern outside the receptive field and observing its movement direction strengthens the belief that the movement direction inside the receptive field is the same, thereby narrowing the prior distribution of directions and thus requiring narrow tuning functions.

The effect of environmental noise

In Figure 2B we saw that the optimal width increases with σ_x, the prior environmental uncertainty. Since external noise adds further uncertainty to the environment, we expect the optimal tuning functions to broaden with increasing noise levels, but the exact effect of noise is difficult to predict because even in the case of additive Gaussian noise with standard deviation σ_w the expression for the MMSE (Eq. 16) becomes slightly more complex. The optimal width in this case is plotted against effective time (Figure 3A) and against prior standard deviation (Figure 3B) for different values of noise levels. All curves are very well-fitted empirically by the function

Figure 3.

Optimal tuning functions width in the presence of noise, for different values of parameters: (A) σ_x = 1 (solid), σ_x = 2 (dashed), and σ_x = 3 (dotted); (B) t_eff = 0. 1 (solid), t_eff = 1 (dashed), and t_eff = 10 (dotted).

$${\widehat{\alpha }}^{\text{opt}}\left(t_{\text{eff}},{\sigma }_x,{\sigma }_w\right)={\left(\frac{t_{\text{eff}}}{\text{9}}+\frac{1}{\sqrt{{\sigma }_x^2+{\sigma }_w^2}}\right)}^{-1},$$

where the average squared error of each fit is less than 1.3 × 10⁻³. This result implies that the effect of noise boils down to increasing the prior uncertainty. Since the noise is additive, and is independent of the stimulus, the term $\sqrt{{\sigma }_x^2+{\sigma }_w^2}$ is precisely the standard deviation of the noisy stimulus meaning that the optimal width for estimating X is the same as for estimating $\tilde{X}$ This might not seem trivial by looking at Eq. 16, where the two standard deviations are not interchangeable, but is nonetheless very intuitive: seeing that all spikes depend only on $\tilde{X}$ it is the only quantity that can be estimated directly from the spike trains, whereas X is then estimated solely from the estimator of $\tilde{X}$ Therefore, optimal estimation of X necessitates first estimating $\tilde{X}$ optimally.

An indication for the symmetric role played by the prior and the noise, can be seen in Figure 3A, where the dashed red curve (σ_x = 2, σ_w = 1) merges with the solid green curve (σ_x = 1, σ_w = 2). Note also that the greatest effect of noise is observed when the prior standard deviation is small, whereas for large Σ_x the relative contribution of noise becomes more and more negligible (Figure 3B).

The effect of multimodality

Integration of sensory information from different modalities (e.g., visual, auditory, somatosensory) has the advantage of increasing estimation reliability and accuracy since each channel provides an independent sampling of the environment (we focus here on the simple case where tuning functions are bell-shaped in all modalities). In the absence of noise, this improvement is merely reflected by an increased number of observations, but the main advantage manifests itself in the noisy case, where multimodal integration has the potential of noise reduction since the noise variables in the two modalities are independent. Considering two sensory modalities, denoted by v and a, and indexing the respective parameters of each modality by v and a, we provide a closed form expression for the MMSE in Eq. 17. When integrating the observations, the spike trains in each modality are weighted according to their reliability, reflected in the predictability of the stimulus based on the estimated input (related to σ_w,v, σ_w,a) and in the discriminability of the tuning functions (related to α_v, α_a). For instance, when “visual” noise has infinite variance or when “visual” tuning functions are flat, the spike trains in the “visual” pathway do not bear any information about the stimulus and are thus ignored by the optimal decoder. Indeed, when substituting σ_w,v = ∞ or α_v = ∞ in Eq. 17, it reduces to Eq. 16 with α_a, σ_w,a in place of α, σ_w. Note that in the absence of noise the MMSE reduces to $\mathbb{E}[{({\sigma }_x^{-2}+{\alpha }^{-2}(Y_v+Y_a))}^{-1}],$ which is identical to the unimodal expression for the MMSE with twice the expected number of observations (Y_v + Y_a ∼ Pois(α2t_eff)).

In a bimodal setting, the optimal width in each modality adapts itself to the standard deviation of the noise associated with that modality. To see that, we fix σ_w,v = 0.5, and plot the optimal widths against the ratio of noise standard deviations σ_w,a/σ_w,v (Figure 4A). When the “visual” channel is noisier (σ_w,a/σ_w,v < 1), the optimal “auditory” tuning functions are narrower than their counterparts, and when the “auditory” channel is noisier the situation is reversed. Interestingly, even when the “visual” noise variance is fixed, the optimal width in the “visual” modality is slightly affected by the “auditory” noise variance.

Figure 4.

(A) Optimal tuning function widths in different sensory modalities as a function of the ratio between the noise standard deviations of the underlying physical phenomena (t_eff = 1, σ_x = 1, and σ_w,v = 0. 5). (B) Optimal tuning function widths in symmetrical bimodal and in unimodal settings (t_eff = 1 and σ_x = 1). The numbers in parentheses denote sizes of populations.

In the symmetric case we can easily compare the unimodal and bimodal settings. Assuming that the number of tuning functions in each modality is the same, integrating two modalities doubles the expected number of observations, naturally resulting in a smaller optimal width (Figure 4B). When the overall number of tuning functions is maintained constant, splitting them into two populations of equal size in each modality is preferred in terms of MMSE (results not shown), because the bimodal setting also has the advantage of partial noise-cancellation. But what can be said about the optimizing width in each setting? In Figure 4B we observe that 2M optimal tuning functions in a single modality are narrower than M + M optimal tuning functions for two different modalities. What is more intriguing is the fact that single-modality optimal tuning functions are still narrower than their double-modality counterparts even when they are outnumbered, as long as the noise standard deviation is sufficiently large.

Biological implications

The latter prediction may be related to experimental results dealing with multisensory compensation. Neuronal tuning functions are observed not only in the sensory areas but also in associative areas in the cortex, where cortical maps are adaptable and not strictly defined for a single modality. Korte and Rauschecker (1993) found that tuning functions in the auditory cortex of cats with early blindness are narrower than those in normal cats. We hypothesize that this phenomenon may subserve more accurate decoding of neural spike trains. Although optimal auditory tuning functions are wider in the absence of visual information (Figure 4B), our results predict that if the blindness is compensated by an increase in the number of auditory neurons (for instance, if some visual neurons are rewired to respond to auditory stimuli), the auditory tuning functions will in fact be narrower than in the case of bimodal sensory integration. Indeed, neurobiological experiments verify that the specificity of cortical neurons can be modified under conditions of sensory deprivation: auditory neurons of deaf subjects react to visual stimuli (Finney et al., 2001) and visual neurons of blind subjects react to sounds (Rauschecker and Harris, 1983; Weeks et al., 2000). We predict that even when the compensation is partial, namely when only a portion of the visual population transforms to sound-sensitive neurons, the optimal tuning functions in a noisy environment are still narrower than the “original” tuning functions in two functioning modalities (Figure 4B).

Dynamic optimal width

Two important features of the decoding process pertain to the time available for decoding and to the prior information at hand. In principle, we may expect that different attributes are required of the optimal tuning functions under different conditions. We have seen in Figure 2 (see also Eq. 1) that the optimal width increases with the initial uncertainty and decreases with effective time. In this case the width is set in advance and the quantity being optimized is the decoding performance at the end of the time window. We conclude that tuning functions should be narrower when more knowledge about the stimulus is expected to be available at time t. However, in realistic situations the decoding time may depend on external circumstances which may not be known in advance. It is more natural to assume that since observations accumulate over time, the tuning functions should adapt in real time based on past information, in such a way that performance is optimal at any time instance, as opposed to some pre-specified time. To address this possibility, we now allow the width to be a function of time and seek an optimal width function. Moreover, it seems more functionally reasonable to set the MMSE process as an optimality criterion rather than the MMSE at an arbitrary time point. As a consequence, the optimal width function may depend on the random accumulation of observations, namely it becomes an optimal width process.

We begin by analyzing the simple case of a piecewise-constant process of the form

α(t:iΔt ≤ t < (i + 1)Δt) = α_i, (i = 0, 1, …)

and search for optimal (possibly random) variables ${\left\{{\alpha }_i\right\}}_{i=1}^{\infty }.$ At each moment, we assume that α(t) is large enough with respect to Δc so that the mean firing rate of the population, Σ_m λ_m(x,t), is independent of the stimulus. Therefore, the posterior distribution is now

$$p(x|{\mathbf{\text{N}}}_t)=K\exp \left\{-\frac{1}{2}\left(\frac{{(x-{\mu }_x)}^2}{{\sigma }_x^2}+{\displaystyle \sum _{m=1}^M{\displaystyle \sum _{i(m)=1}^{N_t^m}\frac{{(x-c_m)}^2}{{[\alpha (t_{i(m)})]}^2}}}\right)\right\},$$

where t_i(m) is the time of the i(m)-th spike generated by the m-th sensory cell. Summing over all spike-times in each Δt-long interval, it is simple to show that at the end of the L-th interval the posterior density function is Gaussian with variance

$${\widehat{\sigma }}^2={\left(\frac{1}{{\sigma }_x^2}+{\displaystyle \sum _{i=0}^{L-1}\frac{Y_i}{{\alpha }_i^2}}\right)}^{-1},$$

where $Y_i={\Sigma }_m(N_{(i+1)\Delta t}^m-N_{i\Delta t}^m)~\text{Pois(}{\alpha }_i\Delta t_{\text{eff}}\text{)}$ for all i and $\Delta t_{\text{eff}}=$ $(\sqrt{2\pi }/\Delta c){\lambda }_{\max }\Delta t.$

The value of ${\alpha }_{\text{0}}^{\text{opt}}$ is obtained by minimizing $\mathbb{E}[{({\sigma }_x^{-2}+{\alpha }^{-2}{Y}_0)}^{-1}].$ When “choosing” the optimal width for the second interval, the population can rely on the spikes observed so far to modify its width so as to minimize $\mathbb{E}[{({\sigma }_1^{-2}+{\alpha }^{-2}{Y}_1)}^{-1}],$ where ${\sigma }_1^{-2}={\sigma }_x^{-2}+{\alpha }_0^{-2}{Y}_0.$ By recursion, at time t = iΔt the optimal width parameter α_i is determined by taking into account all spikes in the interval [0,(i − 1)Δt] and minimizing $\mathbb{E}[{({\sigma }_i^{-2}+{\alpha }^{-2}{Y}_i)}^{-1}],$ where

$${\sigma }_i^2={\left(\frac{1}{{\sigma }_x^2}+{\displaystyle \sum _{\mathcal{J}=0}^{i-1}\frac{Y_{\mathcal{J}}}{{\alpha }_{\mathcal{J}}^2}}\right)}^{-1}$$

reflects the effective uncertainty at time iΔt after integrating prior knowledge and information from observations. We see that the optimal width process is monotonically non-increasing (since the sequence $\left\{{\sigma }_i^2\right\}$ is non-increasing), and the rate of reduction in tuning functions width is affected by the rate of spike arrivals. Examples for both slow and fast spiking processes can be seen in Figure 5A, where the average optimal process (averaged over 1000 trials) is plotted as well. In general, an optimal width process is unlikely to drastically deviate from the average due to an internal “control” mechanism: if few spikes were obtained then the width is still relatively large, increasing the chances of future spikes, whereas multiplicity of spikes results in small width, limiting the probability of observing more spikes. In the limit Δt → 0, the optimal width process starts exactly at σ_x and jumps to a value $\sqrt{2}$ -times smaller than its previous value at the occurrence of each spike (see Section “Dynamics of the Optimal Width Process” in Appendix for proof).

Figure 5.

(A) Sample realizations of optimal width processes for fast (bottom) and slow (top) spiking processes. The average over 1000 trials is plotted in the middle (σ_x = 1 and Δt_eff = 0.1). (B) Average optimal width processes for different values of prior standard deviation, all coinciding with the function $\widehat{\alpha }(t,{\sigma }_x)={(\frac{1}{3}{t}_{\text{eff}}(t)+{\sigma }_x^{-1})}^{-1}.$

When we examine the average optimal width process for different values of prior standard deviation (Figure 5B), we see that prior to the encoding onset it is always best to keep the tuning functions practically as wide as the probability density function of the stimulus (i.e., at t = 0, before any spikes are obtained, set α^opt (t = 0) = σ_x for all realizations). The average rate of dynamic narrowing is then related to the initial uncertainty, where the fastest narrowing occurs in the most uncertain environment. Note that under the conditions stated in Section “Analytical Derivation of the MMSE”, the mean population activity Σ_mλ_m(x) is proportional to the width parameter α. Thus, Figure 5B can be interpreted as reflecting a dynamic decrease in population activity as a function of time. Furthermore, there is an excellent empirical fit between the average optimal process to the simple function

$$\widehat{\alpha }(t;{\sigma }_x)={\left(\frac{t_{\text{eff}}(t)}{3}+\frac{1}{{\sigma }_x}\right)}^{-1}.$$

Implementation of an optimal width process seems biologically implausible, due to its discontinuity and its perfect synchronization with the spikes. It is more likely for biological tuning functions to adopt the average optimal width process strategy, namely to automatically commence a dynamic narrowing at the beginning of an encoding period, independently of the generated spikes. But is the average optimal width process still more efficient than constant width? To address this question, we simulated the dynamics of four populations: two with constant widths tuned for optimal performance at the middle and at the end of the decoding time window, respectively, one with a piecewise-optimal width process and another one with a width function corresponding to the average optimal width process. As expected, each of the constant-width populations outperforms the other at the time for which it is tuned (Figure 6). The dynamic-width populations significantly outperform them both for short times, where for later times the differences become negligible. Therefore, implementation of dynamic width offers a clear advantage over operation with constant width, especially since there is seldom a reference time of interest to which the tuning functions could be tuned in advance. Interestingly, the predetermined width function is as good as the optimal width process for short times and even outperforms it for longer times, meaning that the performance is not being compromised when a predetermined width function is employed in place of the optimal width process.

Figure 6.

The MMSE associated with four different populations, each one tuned to a different optimality criterion. The MMSE was assessed using Monte Carlo methods. Inset: Enlargement for late times.

Biological Implications

Our prediction that tuning functions should dynamically narrow during the course of encoding a stimulus, coincides with some reported experimental phenomena. Wang et al. (2005) recorded the activity of auditory cortical neurons in awake marmoset monkeys in response to a sound stimulus, which is the preferred stimulus of only one of the neurons. When comparing the activity profiles of the two neurons, they found that both fire intensively immediately following stimulus presentation, but only the neuron which is tuned to the stimulus maintained high activity throughout stimulus duration. Such behavior is equivalent to (and can stem from) dynamic narrowing of receptive fields. Widespread onset responses (caused by initially large receptive fields) subserve fast detection of the occurrence of sounds, whereas sustained firing in specific neurons facilitates extraction of the exact features of the sound. Dynamic narrowing of receptive fields following stimulus application was also observed in area 17 of anesthetized cats by Wörgötter et al. (1998). In the context of one-dimensional tuning functions, an illustrative example for dynamic narrowing was observed by Gutfreund et al. (2002) in the external nucleus of the inferior colliculus in anesthetized barn owls. The authors recorded from a population of neurons tuned to a short range of inter-aural time differences (ITD) in response to stimuli with different ITD values. While initial responses were intensive for all stimuli, intensive ongoing responses were restricted to stimuli within the ITD range preferred by the population, implying a dynamic reduction in tuning function width. A similar phenomenon was also obtained for single neurons. Benucci et al. (2009) quantified the instantaneous tuning functions of direction-sensitive V1 neurons in anesthetized cats and observed a dynamic narrowing following stimulus application, accompanied by a later recovery of the tuning function width, after the stimulus was changed or removed.

Discussion

In this paper we have studied optimal encoding of environmental signals by a population of spiking neurons, characterized by tuning functions which quantify the firing probability as a function of the input. Within the framework of optimal Bayesian decoding, based on the mean square reconstruction error criterion, we investigated the properties of optimal encoding. Based on the well known inequality, bounding the MMSE by the Bayesian Cramér–Rao lower bound (Eq. 6), we tested the hypothesis that the tuning function width minimizing the MMSE can be recovered by minimizing the lower bound. This hidden assumption was implied in previous studies dealing with classical, non-Bayesian, estimators based on neural decoding (Seung and Sompolinsky, 1993; Zhang and Sejnowski, 1999; Brown and Backer, 2006). Unfortunately, as argued in Section “Results”, the predictions were often incompatible with the assumptions required in the derivation of the bound, and thus could not be utilized to compare bound-based predictions with results based on direct minimization of the MMSE. It should be noted that in the non-Bayesian setting an optimal estimator does not necessarily exist, because the performance of each estimator depends on the stimulus, which is an unknown parameter. For instance, $\widehat{X}({\mathbf{\text{N}}}_t)\equiv a$ is the optimal estimator if and only if X = a. The above-mentioned studies overcame this problem by assuming that the tuning functions are dense enough so that the population's Fisher information can be approximated by an integral and thus becomes independent of the stimulus. They all concluded that in one dimension estimation performance improves with narrower tuning curves. However, the underlying assumption is valid only when the width is large enough with respect to Δc, and obviously breaks down as α → 0. Therefore, values in the proximity of α = 0 are not part of the solution space, and lim_α→0𝒥(X) cannot be estimated using the suggested approximation. Moreover, realistic environments are dynamic and it seems more reasonable to model their features as random variables rather than unknown parameters. This means that optimal tuning curves may be more fruitfully examined within a Bayesian framework. Indeed, neurobiological evidence indicates that the nervous system can adapt to changes in the statistical nature of the environment and its random features at multiple time scales (e.g., Pettet and Gilbert, 1992; Brenner et al., 2000; Dragoi et al., 2000; Dean et al., 2005; Hosoya et al., 2005).

Starting from the Bayesian Cramér–Rao lower bound (Eq. 6) we have studied predictions about tuning function properties based on this bound which, under appropriate conditions, is asymptotically tight. As we demonstrated, this bound-based approach has little value in predicting the true optimal tuning functions for finite decoding time. Even though the performance inequalities hold in all scenarios, optimizing the bound does not guarantee the same behavior for the bounded quantity. Moreover, an important, and often overlooked observation is the following. Performance bounds might be too model-dependent, and when the model is misspecified (as is often the case) they lose their operative meaning. For instance, when the model is incorrect, the error does not converge asymptotically to the BCRB, even though the estimator itself might converge to the true value of the state (White, 1982). Thus, given that models are often inaccurate, the danger of using bounds is apparent even in the asymptotic limit.

By deriving analytical expressions for the minimal attainable MSE under various scenarios we have obtained optimal widths directly by minimizing the MMSE, without relying on bounds. Our analysis uncovers the dependence of optimal width on decoding time window, prior standard deviation and environmental noise. The relation between optimal width and prior standard deviation may very well explain physiological responses induced by attention, as noted in Section “MMSE-Based Optimal Width”, where further experimental evidence supporting the predictions of this paper was discussed. In particular, the work of Korte and Rauschecker (1993) related to tuning functions in the auditory cortex of cats with early blindness was discussed, and specific predictions were made. We also showed that when the constant-width restriction is removed, tuning functions should narrow during the course of encoding a stimulus, a phenomenon which has already been demonstrated for A1 neurons in marmoset monkeys (Wang et al., 2005) and in the external nucleus of the inferior colliculus in barn owls (e.g., Gutfreund et al., 2002).

In summary, in this paper we examined optimal neuronal tuning functions within a Bayesian setting based on the MMSE criterion. Careful calculations were followed by theoretical results, based on a natural criterion of optimality which is commonly employed, but which has seldom been analyzed in the context of neural encoding. Our analysis yielded novel predictions about the context-dependence of optimal widths, stating that optimal tuning curves should adapt to the statistical properties of the environment – in accordance with ecological theories of sensory processing. Interestingly, the results predict at least two time scales of change. For example, when the statistical properties of the environment change (e.g., a change in the noise level or prior distribution) the optimal encoding should adapt on the environmental time scale in such a way that the tuning function widths increases with noise level. However, even in the context of a fixed stimulus, we predict that tuning functions should change on the fast time scale of stimulus presentation. Interestingly, recent results, summarized by Gollisch and Meister (2010), consider contrast adaptation in the retina taking place on a fast and slow time scale (see also Baccus and Meister, 2002). The fast time scale, related to “contrast gain control” occurs on time scales of tens of millisecond, while the slow time-scale, lasting many seconds, has been referred to as “contrast adaptation”. While our results do not directly relate to contrast adaptation, the prediction that optimal adaptation takes place on at least two, widely separated, time scales is promising. Moreover, the circuit mechanisms proposed in (Gollisch and Meister, 2010) may well subserve some of the computations related to tuning function narrowing. These results, and the above-mentioned experimentally observed phenomena, suggest that biological sensory systems indeed adapt to changes in environmental statistics on multiple time scales. The theory provided in this paper offers a clear functional explanation of these phenomena, based on the notion of optimal dynamic encoding.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Appendix

Expectation of population fisher information

In this section we calculate the expected value of the Fisher information for a population of sensory neurons following Poisson spiking statistics, having Gaussian tuning functions of the form

$${\lambda }_m(x)={\lambda }_{\max }{e}^{-\frac{{(x-c_m)}^2}{2{\alpha }_m^2}},(m=1,\dots ,M)$$

and encoding a stimulus with Gaussian prior distribution $(X~p(\cdot )=\mathcal{N}({\mu }_x,{\sigma }_x^2)).$ We start by calculating the population's local Fisher information, taking into account the conditional independence of the Poisson processes:

$$\begin{array}{c}\mathcal{J}(X)=\mathbb{E}\left[{\left(\frac{\partial }{\partial X}\text{ln}\left({\displaystyle \prod _{m=1}^{M}P(N_t^m|X)}\right)\right)}^2\right]=\mathbb{E}\left[{\left({\displaystyle \sum _{m=1}^M\frac{\partial }{\partial X}\text{ln}P\left(N_t^m|X\right)}\right)}^2\right]\\ =\mathbb{E}\left[{\left({\displaystyle \sum _{m=1}^M\left(\frac{N_t^m}{{\lambda }_m(X)}-t\right)\frac{\partial {\lambda }_m(X)}{\partial X}}\right)}^2\right]\\ =\mathbb{E}\left[{\left({\displaystyle \sum _{m=1}^M\left(N_t^m-{\lambda }_m\left(X\right)t\right)\left(\frac{c_m-X}{{\alpha }_m^2}\right)}\right)}^2\right]\\ ={\displaystyle \sum _{m=1}^M{\displaystyle \sum _{n=1}^M\mathbb{E}\left[\left(N_t^m-{\lambda }_m\left(X\right)t\right)\left(N_t^n-{\lambda }_n\left(X\right)t\right)|X\right]}}\\ \times \left(\frac{c_m-X}{{\alpha }_m^2}\right)\left(\frac{c_n-X}{{\alpha }_n^2}\right).\end{array}$$

Due to the conditional independence the expectation is separable for every m ≠ n, and equals 0 because $\mathbb{E}[N_t^m|X]={\lambda }_m(X)t.$ Thus,

$$\begin{array}{c}\mathcal{J}\left(X\right)={\displaystyle \sum _{m=1}^M\stackrel{\mathrm{var}\left(N_t^m|X\right)={\lambda }_m(X)t}{\overbrace{\mathbb{E}\left[{\left(N_t^m-{\lambda }_m\left(X\right)t\right)}^2|X\right]}}}\frac{{\left(X-c_m\right)}^2}{{\alpha }_m^4}\\ ={\lambda }_{\max }t{\displaystyle \sum _{m=1}^M\frac{{\left(X-c_m\right)}^2}{{\alpha }_m^4}}{e}^{-\frac{{\left(x-c_m\right)}^2}{2{\alpha }_m^2}}.\end{array}$$

Taking the expectation over the prior p(x) yields:

$$\begin{array}{c}\mathbb{E}\left[\mathcal{J}\left(X\right)\right]={\displaystyle \underset{-\infty }{\overset{\infty }{\int }}{\lambda }_{\max }t}{\displaystyle \sum _{m=1}^M\left(\frac{{\left(x-c_m\right)}^2}{{\alpha }_m^4}{e}^{-\frac{{\left(x-c_m\right)}^2}{2{\alpha }_m^2}}\right)\frac{1}{\sqrt{2\pi {\sigma }_x^2}}{e}^{-\frac{{\left(x-{\mu }_x\right)}^2}{2{\sigma }_x^2}}dx}\\ =\frac{{\lambda }_{\max }t}{{\sigma }_x}{\displaystyle \sum _{m=1}^M\left(\frac{{\sigma }_m}{{\alpha }_m^4}{e}^{-\frac{{\left(c_m-{\mu }_x\right)}^2}{2\left({\alpha }_m^2+{\sigma }_x^2\right)}}{\displaystyle \underset{-\infty }{\overset{\infty }{\int }}{\left(x-c_m\right)}^2}\frac{e^{-\frac{{\left(x-{\mu }_m\right)}^2}{2{\sigma }_m^2}}}{\sqrt{2\pi {\sigma }_m^2}}dx\right)},\end{array}$$

where

$${\mu }_m=\frac{{\alpha }_m^2{\mu }_x+{\sigma }_x^2{c}_m}{{\alpha }_m^2+{\sigma }_x^2};{\sigma }_m^2=\frac{{\alpha }_m^2{\sigma }_x^2}{{\alpha }_m^2+{\sigma }_x^2}.$$

By defining $Y_m~\mathcal{N}\left({\mu }_m,{\sigma }_m^2\right),\left(m=1,2,\dots ,M\right),$ the integral becomes $\mathbb{E}\left[Y_m^2-2c_m{Y}_m+c_m^2\right]={\sigma }_m^2+{\mu }_m^2-2c_m{\mu }_m+c_m^2,$ and by substituting ${\mu }_m,{\sigma }_m^2$ we get

$$\mathbb{E}\left[\mathcal{J}\left(X\right)\right]={\displaystyle \sum _{m=1}^M\frac{{\lambda }_{\max }t\cdot e^{-\frac{{\left(c_m-{\mu }_x\right)}^2}{2\left({\alpha }_m^2+{\sigma }_x^2\right)}}}{{\alpha }_m{\left({\alpha }_m^2+{\sigma }_x^2\right)}^{\raisebox{1ex}{$5$}\!\left/ \!\raisebox{-1ex}{$2$}\right.}}}\left[{\sigma }_x^2\left({\sigma }_x^2+{\alpha }_m^2\right)+{\alpha }_m^2{\left(c_m-{\mu }_x\right)}^2\right].$$

In the multi-dimensional case X ∼ 𝒩(μ_x, Σ_x), the tuning functions are of the form

$${\lambda }_m\left(x\right)={\lambda }_{\max }{e}^{-\frac{1}{2}{\left(x-c_m\right)}^T{A}_m^{-1}\left(\text{x}-{\text{c}}_m\right)},\left(m=1,\dots ,M\right),$$

and the population's Fisher information is given by

$$\begin{array}{c}\mathcal{J}\left(\mathbf{\text{X}}\right)=\mathbb{E}[\left({\displaystyle \sum _{m=1}^M\left(N_t^m-{\lambda }_m\left(\mathbf{\text{X}}\right)t\right)A_m^{-1}\left(\mathbf{\text{X}}-{\mathbf{\text{c}}}_m\right)}\right)\\ {\left({\displaystyle \sum _{n=1}^M\left(N_t^n-{\lambda }_n\left(\mathbf{\text{X}}\right)t\right)A_n^{-1}\left(\mathbf{\text{X}}-{\mathbf{\text{c}}}_n\right)}\right)}^T]\\ ={\displaystyle \sum _{m=1}^M{\displaystyle \sum _{n=1}^M\mathbb{E}\left[\left(N_t^m-{\lambda }_m\left(\mathbf{\text{X}}\right)t\right)\left(N_t^n-{\lambda }_n\left(\mathbf{\text{X}}\right)t\right)|\mathbf{\text{X}}\right]}}{A}_m^{-1}\\ \left(\mathbf{\text{X}}-{\mathbf{\text{c}}}_m\right){\left(\mathbf{\text{X}}-{\mathbf{\text{c}}}_n\right)}^T{A}_n^{-1}\\ ={\lambda }_{\max }t{\displaystyle \sum _{m=1}^M{A}_m^{-1}}\left\{\left(\mathbf{\text{X}}-{\mathbf{\text{c}}}_m\right){\left(\mathbf{\text{X}}-{\mathbf{\text{c}}}_m\right)}^T{e}^{-\frac{1}{2}{\left(\mathbf{\text{X}}-{\mathbf{\text{c}}}_m\right)}^T{A}_m^{-1}\left(\mathbf{\text{X}}-{\mathbf{\text{c}}}_m\right)}\right\}{A}_m^{-1}.\end{array}$$

By virtue of the linearity, we calculate the expected value of 𝒥(X) with respect to X by taking expectation over the expression in curly brackets:

$$\begin{array}{c}\mathbb{E}\{\cdot \}={\displaystyle \int \left(\mathbf{\text{X}}-{\text{c}}_m\right){\left(\mathbf{\text{X}}-{\text{c}}_m\right)}^T{e}^{-\frac{1}{2}{\left(\mathbf{\text{X}}-{\text{c}}_m\right)}^T{A}_m^{-1}\left(\mathbf{\text{X}}-{\text{c}}_m\right)}}\frac{e^{-\frac{1}{2}{\left(\mathbf{\text{X}}-{\mu }_{\mathbf{\text{X}}}\right)}^T{\sum }_{\mathbf{\text{X}}}^{-1}\left(\mathbf{\text{X}}-{\mu }_{\mathbf{\text{X}}}\right)}}{{\left(2\pi \right)}^{D/2}\sqrt{\left|{\sum }_{\mathbf{\text{X}}}\right|}}dX\\ =e^{-\frac{1}{2}{\xi }_m}\frac{\sqrt{\left|{\sum }_m\right|}}{\sqrt{\left|{\sum }_x\right|}}{\displaystyle \int \left(\mathbf{\text{X}}-{\text{c}}_m\right){\left(\mathbf{\text{X}}-{\text{c}}_m\right)}^T}\frac{e^{-\frac{1}{2}{\left(\mathbf{\text{X}}-{\mu }_m\right)}^T{\sum }_m^{-1}\left(\mathbf{\text{X}}-{\mu }_m\right)}}{{\left(2\pi \right)}^{D/2}\sqrt{\left|{\sum }_m\right|}}d\mathbf{\text{X}}\\ =e^{-\frac{1}{2}{\xi }_m}\frac{\sqrt{\left|{\sum }_m\right|}}{\sqrt{\left|{\sum }_{\mathbf{\text{X}}}\right|}}\left({\sum }_m+{\mu }_m{\mu }_m^T-2{\text{c}}_m{\mu }_m^T+{\text{c}}_m{\text{c}}_m^T\right),\end{array}$$

where

$${\sum }_m={\left(A_m^{-1}+{\sum }_x^{-1}\right)}^{-1};{\mu }_m={\sum }_m\left(A_m^{-1}{\text{c}}_m+{\sum }_x^{-1}{\mu }_x\right);$$

$${\xi }_m={\text{c}}_m^T{A}_m^{-1}{\text{c}}_m+{\mu }_x^T{\sum }_x^{-1}{\mu }_x-{\mu }_m^T{\sum }_m^{-1}{\mu }_m.$$

Using the identity (A⁻¹ + B⁻¹)⁻¹ = A(A + B)⁻¹ B (Searle, 1982) and the symmetry of all matrices it is a simple mathematical exercise to show that

$$\begin{array}{l}\mathbb{E}\left[\mathcal{J}\left(\mathbf{\text{X}}\right)\right]={\lambda }_{\max }t{\displaystyle \sum _{m=1}^M{e}^{-\frac{1}{2}{\xi }_m}}\frac{\sqrt{\left|A_m\right|}{A}_m^{-1}}{\sqrt{\left|A_m+{\sum }_x\right|}}{\left(A_m+{\sum }_x\right)}^{-2}\\ \left[{\sum }_x\left(A_m+{\sum }_x\right)+A_m\left({\text{c}}_m-{\mu }_x\right){\left({\text{c}}_m-{\mu }_x\right)}^T\right]\end{array}$$

and ξ_m = (c_m − μ_x)^T(A_m + Σ_x)⁻¹(c_m − μ_x).

Uniform mean firing rate of population

In this section we approximate the mean firing rate of the entire population of sensory cells for the case of equally spaced tuning functions with uniform width. First, consider the case M = ∞, for which the population's mean firing rate is given by

$$\lambda \left(x,\alpha \right)={\lambda }_{\max }{\displaystyle \sum _{m=-\infty }^{\infty }{e}^{-\frac{1}{2}{\left(\frac{x-m\Delta \text{c}}{\alpha }\right)}^2}}.$$

The function λ(x,α) is continuous with respect to x, and therefore when the spacing goes to 0, Δc·λ(x,α) converges to the Riemann integral

$$\underset{\Delta c\to 0}{\lim }\Delta c\cdot \lambda \left(x,\alpha \right)={\lambda }_{\max }{\displaystyle \underset{-\infty }{\overset{\infty }{\int }}{e}^{-\frac{1}{2}{\left(\frac{x-c}{\alpha }\right)}^2}}dc={\lambda }_{\max }\sqrt{2\pi }\alpha$$

which is independent of x. This means that for any ε > 0 there exists δ > 0 such that if Δc/α < δ then the relative ripple (digression from uniformity) in the total firing rate

$$\frac{\max \lambda \left(x,\alpha \right)-\min \lambda \left(x,\alpha \right)}{\min \lambda \left(x,\alpha \right)}$$

is smaller than ε.

Now consider the case of finite M,

$$\lambda \left(x,\alpha ;M\right)={\lambda }_{\max }{\displaystyle \sum _{m=\lceil \frac{M-1}{\text{2}}\rceil }^{\lceil \frac{M-1}{2}\rceil }{e}^{-\frac{1}{2}\left(\frac{x-m\Delta c}{\alpha }\right)}}.$$

Since the limit at M → ∞ exists, for every x and ε > 0 there exists M_ε(x) > 0 such that for all M > M_ε(x) the difference between the infinite sum to the finite sum is less than ε, and therefore λ(x, α; M) can be well approximated by

$${\displaystyle \sum _{m=-\infty }^{\infty }{\lambda }_m\left(x\right)\approx \frac{{\lambda }_{\max }}{\Delta c}}{\displaystyle \underset{-\infty }{\overset{\infty }{\int }}{e}^{-\frac{1}{2}{\left(\frac{x-c}{\alpha }\right)}^2}}dc=\frac{{\lambda }_{\max }\sqrt{2\pi }\alpha }{\Delta c}\triangleq \lambda (\alpha ).$$

To quantify the desired ratio between the width and the spacing we plot the relative ripple (Figure 7A). For α > 0.583Δc the relative ripple falls down below 0.5%, rendering the approximation λ(x, α;M) ≅ λ(α) quite accurate. Note that the approximation fails near the most extreme tuning functions, but for a sufficiently large population this would not have any effect on the results, because the probability of X falling into these regions would be vanishingly small for any rapidly decaying prior.

Figure 7.

The relative ripple in the mean firing rate of a population of neurons with Gaussian tuning functions: (A) in single dimension, (B) in two dimensions.

The proof for the multi-dimensional case is similar, where the population's mean firing rate is approximated by

$${\displaystyle \sum _{m=-\infty }^{\infty }{\lambda }_m\left(x\right)}\approx \frac{{\lambda }_{\max }}{{\left(\Delta c\right)}^d}{\displaystyle \underset{-\infty }{\overset{\infty }{\int }}{e}^{-\frac{1}{2}{\left(\text{x}-\text{c}\right)}^T{A}^{-1}\left(\text{x}-\text{c}\right)}d\text{c}={\lambda }_{\max }}{\left(\frac{\sqrt{2\pi }}{\Delta c}\right)}^d\sqrt{\left|A\right|}\triangleq \lambda \left(A\right).$$

The conditions which make the approximation valid vary with the dimensionality of the problem – as the dimension increases the width-spacing ratio must be larger in order to maintain the same approximation accuracy. For instance, in two dimensions where A = αI (Figure 7B), the relative ripple falls down below 0.5% only for α > 1.446Δc.

As mentioned in Section “The Univariate Case”, even when the approximation in Eq. 10 fails (i.e., when the mean population firing rate is highly non-uniform) the relationship in Eq. 11 still holds, namely the MMSE can be accurately estimated using the derivation for uniform mean population firing rate. While the normalized MMSE can be shown analytically to converge to 1 in the limit α → 0 (see The Univariate Case), Figure 8 verifies that the approximation for the normalized MMSE practically coincides with the simulated normalized MMSE for finite values of α that do not satisfy α ≫ c.

Figure 8.

Normalized MMSE of a sparse population of Gaussian tuning functions against tuning function width, for different values of effective decoding time, obtained theoretically (dashed lines) and in simulation (solid lines). Simulation parameters: N = 251, X ∈ [−4,4], P(x)∝N(0,1), M = 250, Δc = 0.034, λ_max = 50.

MMSE for multisensory integration

In this section we derive the MMSE for the case of decoding bimodal spike trains. Similarly to the unimodal noisy case, it is straightforward to show that the joint posterior distribution of the environmental state and noise variables is given by

$$p\left(x,w_v,w_a|{\mathbf{\text{N}}}_t^v,{\mathbf{\text{N}}}_t^a\right)=K{e}^{-\frac{{(x-{\mu }_x)}^2}{2{\sigma }_x^2}}{e}^{-\frac{w_v^2}{2{\sigma }_{w,v}^2}}{e}^{-\frac{w_a^2}{2{\sigma }_{w,a}^2}}{e}^{-{\displaystyle \sum _{m=1}^{Mv}\frac{(x+w_v-c_m^v)}{2{\alpha }_v^2}{N}_t^{m,v}}}\times e^{-{\displaystyle \sum _{m=1}^{Ma}\frac{{(x+w_a-c_m^a)}^2}{2{\alpha }_a^2}}{N}_t^{m,a}}$$

$$=K{e}^{-\frac{{(x-{\mu }_x)}^2}{2{\sigma }_x^2}-\frac{w_v^2}{2{\sigma }_{w,v}^2}-{\displaystyle \sum _{m=1}^{Mv}\frac{{(x+w_v-c_m^v)}^2}{2{\alpha }_v^2}{N}_t^{m,v}}}\times e^{-\frac{1}{2}\left(\frac{w_a^2}{{\sigma }_{w,a}^2}+{\displaystyle \sum _{m=1}^{M_a}\frac{(x+w_a-c_m^a)}{{\alpha }_a^2}{N}_t^{m,a}}\right)},$$

which is Gaussian with respect to W_a, seeing that the argument of the rightmost exponent is

$$\begin{array}{l}-\frac{1}{2{\widehat{\sigma }}_{w,a}^2}{\left(w_a-{\widehat{\sigma }}_{w,a}^2{\displaystyle \sum _{m=1}^{M_a}\frac{c_m^a-x}{{\alpha }_a^2}{N}_t^{m,a}}\right)}^2\\ -\frac{1}{2}\left({\displaystyle \sum _{m=1}^{M_a}\frac{\left(c_m^a-x\right)}{{\alpha }_a^2}{N}_t^{m,a}}-{\widehat{\sigma }}_{w,a}^2{\left({\displaystyle \sum _{m=1}^{M_a}\frac{c_m^a-x}{{\alpha }_a^2}{N}_t^{m,a}}\right)}^2\right),\end{array}$$

where ${\widehat{\sigma }}_{w,a}^2={\left({\sigma }_{w,a}^{-2}+{\alpha }_a^{-2}{\sum }_m{N}_t^{m,a}\right)}^{-1}.$ Integration over w_a gives the joint posterior distribution of the state and the “visual” noise:

$$\begin{array}{c}p\left(x,w_v|{\mathbf{\text{N}}}_t^v,{\mathbf{\text{N}}}_t^a\right)=K{e}^{-\frac{{\left(x-{\mu }_x\right)}^2}{2{\sigma }_x^2}-\frac{w_v^2}{2{\sigma }_{w,v}^2}-{\displaystyle \sum _{m=1}^{M_v}\frac{{\left(x+wv-c_m^v\right)}^2}{2{\alpha }_v^2}}{N}_t^{m,v}}\\ \times e^{-\frac{1}{2}\left({\displaystyle \sum _{m=1}^{M_a}\frac{{\left(c_m^a-x\right)}^2}{{\alpha }_a^2}}{N}_t^{m,a}-{\stackrel{\wedge }{\sigma }}_{w,a}^2{\left({\displaystyle \sum _{m=1}^{M_a}\frac{\left(c_m^a-x\right)}{{\alpha }_a^2}{N}_t^{m,a}}\right)}^2\right)}\end{array}$$

$$=K{e}^{-\frac{{(x-{\mu }_x)}^2}{2{\sigma }_x^2}-{\displaystyle \sum _{m=1}^{M_a}\frac{(c_m^a-x)}{2{\alpha }_a^2}{N}_t^{m,a}+{\hat{\sigma }}_{w,a}^2{\left({\displaystyle \sum _{m=1}^{M_a}\frac{c_m^a-x}{2{\alpha }_a^2}{N}_t^{m,a}}\right)}^2}}\times e^{-\frac{1}{2}\left(\frac{w_v^2}{{\sigma }_{w,v}^2}+{\displaystyle \sum _{m=1}^{M_v}\frac{{\left(x+w_v-c_m^v\right)}^2}{{\alpha }_v^2}{N}_t^{m,v}}\right)},$$

which is Gaussian with respect to W_v, seeing that the argument of the rightmost exponent is

$$\begin{array}{l}-\frac{1}{2{\widehat{\sigma }}_{w,v}^2}{\left(w_v-{\widehat{\sigma }}_{w,v}^2{\displaystyle \sum _{m=1}^{M_v}\frac{c_m^v-x}{{\alpha }_v^2}{N}_t^{m,v}}\right)}^2\\ -\frac{1}{2}\left({\displaystyle \sum _{m=1}^{M_v}\frac{{\left(c_m^v-x\right)}^2}{{\alpha }_v^2}}{N}_t^{m,v}-{\widehat{\sigma }}_{w,v}^2{\left({\displaystyle \sum _{m=1}^{M_v}\frac{c_m^v-x}{{\alpha }_v^2}{N}_t^{m,v}}\right)}^2\right),\end{array}$$

where ${\widehat{\sigma }}_{w,v}^2={\left({\sigma }_{w,v}^{-2}+{\alpha }_v^{-2}{\sum }_m{N}_t^{m,v}\right)}^{-1}.$ Integration over w_v gives the marginal distribution of the state:

$$\begin{array}{l}p\left(x|{\mathbf{\text{N}}}_t^v,{\mathbf{\text{N}}}_t^a\right)=\\ K{e}^{-\frac{{\left(x-{\mu }_x\right)}^2}{2{\sigma }_x^2}-{\displaystyle \sum _{m=1}^{M_a}\frac{{\left(c_m^a-x\right)}^2}{2{\alpha }_a^2}}{N}_t^{m,a}+{\stackrel{\wedge }{\sigma }}_{w,a}^2{\left({\displaystyle \sum _{m=1}^{M_a}\frac{c_m^a-x}{2{\alpha }_a^2}{N}_t^{m,a}}\right)}^2-{\displaystyle \sum _{m=1}^{M_v}\frac{{\left(c_m^v-x\right)}^2}{2{\alpha }_v^2}{N}_t^{m,v}+{\stackrel{\wedge }{\sigma }}_{w,v}^2{\left({\displaystyle \sum _{m=1}^{M_v}\frac{c_m^v-x}{2{\alpha }_v^2}{N}_t^{m,v}}\right)}^2}},\end{array}$$

which can be shown to be Gaussian with variance

$${\widehat{\sigma }}^2={\left(\frac{1}{{\sigma }_x^2}+\frac{{\sum }_m{N}_t^{m,v}}{{\alpha }_v^2}+\frac{{\sum }_m{N}_t^{m,a}}{{\alpha }_a^2}-\frac{{\widehat{\sigma }}_{w,v}^2{\left({\sum }_m{N}_t^{m,v}\right)}^2}{{\alpha }_v^4}-\frac{{\widehat{\sigma }}_{w,a}^2{\left({\sum }_m{N}_t^{m,a}\right)}^2}{{\alpha }_a^4}\right)}^{-1}.$$

By defining $Y_v\triangleq {\sum }_m{N}_t^{m,v}~\text{Pois}\left({\alpha }_v{t}_{\text{eff}}^v\right)$ and $Y_a\triangleq {\sum }_m{N}_t^{m,a}~\text{Pois}\left({\alpha }_a{t}_{\text{eff}}^a\right),$ we get a relatively simple expression for the MMSE:

$$\begin{array}{c}\text{MMSE}=\mathbb{E}\left[{\left(\frac{1}{{\sigma }_x^2}+\frac{Y_v}{{\alpha }_v^2}+\frac{Y_a}{{\alpha }_a^2}-\frac{{\sigma }_{w,v}^2{Y}_v^2}{{\alpha }_v^2\left({\alpha }_v^2+{\sigma }_{w,v}^2{Y}_v\right)}-\frac{{\sigma }_{w,a}^2{Y}_a^2}{{\alpha }_a^2\left({\alpha }_a^2+{\sigma }_{w,a}^2{Y}_a\right)}\right)}^{-1}\right]\\ =\mathbb{E}\left[{\left(\frac{1}{{\sigma }_x^2}+\frac{Y_v}{{\alpha }_v^2+{\sigma }_{w,v}^2{Y}_v}+\frac{Y_a}{{\alpha }_a^2+{\sigma }_{w,a}^2{Y}_a}\right)}^{-1}\right].\end{array}$$

Dynamics of the optimal width process

In this section we prove that for the optimal piecewise-constant width process defined in Section “Dynamic Optimal Width”,

$$\underset{\Delta t\to 0}{\lim }{\alpha }_0={\sigma }_x$$

and

$$\underset{\Delta t\to 0}{\lim }\frac{{\alpha }_i}{{\alpha }_{i+1}}=\sqrt{2},(i=0,1,\mathrm{\dots },)$$

where α_i is the value of the width process after obtaining a total of i spikes.

For short intervals, the Poisson random variable $Y_{\Delta t}={\sum }_m{N}_{\Delta t}^m$ converges in probability to a Bernoulli random variable with success probability α₀Δt_eff. The MMSE at t = Δt then takes the form

$$\begin{array}{c}\text{MMSE}={\displaystyle \sum _{k=0}^1\frac{1}{{\sigma }_x^{-2}+k{\alpha }_0^{-2}}P\left(Y_{\Delta t}=k\right)=\frac{1-{\alpha }_0\Delta t_{\text{eff}}}{{\sigma }_x^{-2}}}+\frac{{\alpha }_0\Delta t_{\text{eff}}}{{\sigma }_x^{-2}+{\alpha }_0^{-2}}\\ =\frac{{\sigma }_x^{-2}+{\alpha }_0^{-2}-{\alpha }_0^{-1}\Delta t_{\text{eff}}}{{\sigma }_x^{-2}+{\alpha }_0^{-2}}=1-\Delta t_{\text{eff}}\frac{{\alpha }_0^{-1}}{{\alpha }_0^{-2}+{\sigma }_x^{-2}}.\end{array}$$

The MMSE is minimized when ${\alpha }_0^{-1}/\left({\alpha }_0^{-2}+{\sigma }_x^{-2}\right)$ is maximized, and thus ${\alpha }_0^{\text{opt}}={\sigma }_x$. Given the spikes history at t = Δt, if Y_Δt = 0 then the MMSE expression for the next interval remains unchanged, whereas if Y_Δt = 1 then the MMSE at the end of the next interval is given by

$$\text{MMSE}=\mathbb{E}\left[\frac{\text{1}}{{\sigma }_x^{-2}+{\left({\alpha }_{\text{0}}^{\text{opt}}\right)}^{-2}+\left(Y_{2\Delta t}-1\right){\alpha }_1^{-2}}\right]=\mathbb{E}\left[\frac{\text{1}}{{\sigma }_1^{-2}+\left(Y_{2\Delta t}-1\right){\alpha }_1^{-2}}\right],$$

where ${\sigma }_1={\sigma }_x/\sqrt{2}$ is the effective uncertainty after observing a single spike, and Y_2Δt − 1 is a Bernoulli random variable with success probability α₁Δt_eff. Applying the same analysis as before, ${\sigma }_1^{\text{opt}}={\sigma }_1={\alpha }_0^{\text{opt}}/\sqrt{2}\text{.}$ By recursion, one can see that as long as k spikes were observed the optimal width for the next interval remains ${\alpha }_k^{\text{opt}},$ and if another spike is observed the optimal width for the next interval becomes ${\alpha }_{k+1}^{\text{opt}}={\alpha }_k^{\text{opt}}/\sqrt{2}.$