Optimization and variability can coexist

Abstract

Many biological systems perform close to their physical limits, but promoting this optimality to a general principle seems to require implausibly fine tuning of parameters. Using examples from a wide range of systems, we show that this intuition is wrong. Near an optimum, functional performance depends on parameters in a “sloppy” way, with some combinations of parameters being only weakly constrained. Absent any other constraints, this predicts that we should observe widely varying parameters, and we make this precise: the entropy in parameter space can be extensive even if performance on average is very close to optimal. This removes a major objection to optimization as a general principle, and rationalizes the observed variability.

Humans can count single photons on a dark night [1], insects see the world with a resolution close to the limits set by diffraction through the lenses of the compound eye [2, 3], and bacteria navigate chemical gradients with a precision limited by random arrival of molecules at their surface receptors [4]. In these and other examples, living systems approach fundamental physical limits to their performance, and in this sense are close to optimal. In many different contexts it has been suggested that this sort of optimization should be elevated to a principle from which the behavior and underlying mechanisms of these complex systems can be derived [5, 6]. In contrast, others have argued that evolution cannot find optimal solutions, and emphasize examples in which the assignment of complex structures and mechanisms to functional behavior is misleading [7]. One important observation is that the parameters of biological systems, from the copy numbers of proteins in single cells to the strengths of synapses between neurons, can be highly variable [8–10], and this seems to be direct evidence against the idea that these parameters have been tuned to optimal values.

Intuitions about the ability of evolution to find optima and about the significance of parameter variations both depend on hypotheses about the landscape for optimization. If optima are small, sharply defined regions in a rugged terrain [11, 12], then plausible dynamics for the exploration of high dimensional parameter spaces are unlikely to find the optimum; similarly, even small variations in parameters would drive the system far from optimality. We will see that this picture of the landscape is wrong in many cases. Instead, the dependence of functional performance on the underlying parameters is very gentle or “soft,” so that at least some combinations of parameters can vary substantially with very little effect. As a result, many of the arguments against optimization lose their force. In particular, in the absence of any other constraints, a theory of systems that are very close to optimal predicts that we should observe widely varying parameters. We try to make this precise, exploring several different biological systems as well as biologically inspired model neural networks.

Our analysis is inspired in part by the idea of sloppy models [13]. When we fit multi–parameter models to the observed behavior of complex systems, we often find that some combinations of parameters are determined precisely, and others are not [14, 15]. Sethna and colleagues have argued that this is generic [16]; that the smoothness of models implies that the sensitivity to parameters is distributed more or less uniformly on a logarithmic scale, so that the fit is orders of magnitude more sensitive to some combinations of parameters than to others [17]; and that this behavior provides a path to model simplification [18].

To address the issue of optimization in living systems, we consider not the quality of a model’s fit to data, but the functional performance of the organism at some task essential for its survival. The parameters are not something we choose, but rather the physical properties of the organism’s internal mechanisms—concentrations of molecules, binding constants, positions of cells, strengths of synapses—that are adjusted through adaptation, learning, and evolution. Sloppiness or the existence of soft modes then means that organisms can be very close to optimal performance while its physical properties can vary along many dimensions.

It is tempting to think that soft modes appear just because there are many parameters feeding in to a single functional output, so that there is a kind of redundancy. While this plays a role, we will see that soft modes appear in addition to any true redundancies. We also will see that these modes emerge in part from the structure of the problem that needs to be solved by the biological system rather than being generic features of the underlying mechanisms.

In analyzing complex models, sloppiness suggests a path to simplification. Similarly, it might seem that mechanisms with parameters that have little effect on functional performance are somehow inefficient, but this notion of parameter efficiency then is something we should have added to the performance measure from the start. We also will see that soft modes in parameter space generally are not aligned with the microscopic components; for example, sloppiness of a genetic network crucial for fly development does not mean that we can knock out genes and maintain the same level of functional performance, even after readjustment of all other parameters [19]. These and other considerations mean that soft modes or sloppiness have consequences for the organism, not just for our models.

Evolution, learning, and adaptation allow organisms to explore their parameter spaces. A natural hypothesis is that this exploration ranges as widely as possible while maintaining, on average, some high level of performance. Our results on the spectrum of soft modes allows us to turn this hypothesis into quantitative predictions about observable variations in parameters, especially in the limit that parameter spaces are very high–dimensional. Quantitatively, we argue that the entropy of the allowed variations in parameter space can be extensive even if performance is on average arbitrarily close to optimal. Thus variability is not a retreat from optimization to a weaker notion of being “good enough;” rather it is a feature of these systems that optimization and variability coexist.

FORMULATING THE PROBLEM

In different biological systems, the relevant measure of functional performance is different. If we can think about the organism as a whole in the context of evolution, then by definition the relevant measure is fitness. In many cases we want to think more explicitly about particular systems within the organism to which we can ascribe less abstract goals. In a neural circuit that generates a rhythm what might be important is the frequency of the rhythm; in a developing embryo what might matter is the ability of cells to adopt fates that are appropriate to their positions; for an array of receptor cells the measure of performance might be the amount of sensory information that these cells can capture from the outside world. In the same spirit, the underlying parameters of different systems are different. The dynamics of a neural circuit are determined by the number of each different kind of ion channel in each cell and by the strengths of the synapses between cells; the ability of a genetic network to read out positional information in the embryo is determined in part by the thresholds at which different genes are activated or repressed; the information gathered by receptor arrays depends on the positions of the cells in the sensory epithelium. We want to address these different problems in a common language.

We will write the measure of functional performance as $ℱ$, where larger $ℱ$ means better performance. We write the parameters of the underlying mechanism in a vector $\mathit{\theta }\equiv \left\{{\theta }_1,{\theta }_2,\cdots ,{\theta }_K\right\}$, where the number of parameters $K$ may be quite large. There is some function $ℱ\left(\theta \right)$ that determines how performance depends on parameters, and there is some parameter setting $\mathit{\theta }={\mathit{\theta }}^{\mathrm{*}}$ that maximizes performance. In the neighborhood of the optimum we can write

$$ℱ(\mathit{\theta })={ℱ}_{\mathrm{m}\mathrm{a}\mathrm{x}}-\frac{1}{2}\sum _{\mathrm{i},\mathrm{j}=1}^K{H}_{\mathrm{i}\mathrm{j}}({\theta }_{\mathrm{i}}-{\theta }_{\mathrm{i}}^{\mathrm{*}})({\theta }_{\mathrm{j}}-{\theta }_{\mathrm{j}}^{\mathrm{*}})+\cdots ,$$ (1)

where the Hessian matrix

$$H_{\mathrm{i}\mathrm{j}}\equiv -{\left.\frac{{\partial }^2ℱ\left(\mathit{\theta }\right)}{\partial {\theta }_{\mathrm{i}}\partial {\theta }_{\mathrm{j}}}\right|}_{\mathit{\theta }={\mathit{\theta }}^{\mathrm{*}}}.$$ (2)

The eigenvectors of the Hessian define combinations of parameters that have independent effects on the performance, and the corresponding eigenvalues ${\mathrm{\lambda }}_{\mu }$ measure the strength of these effects. If we think of the (negative) performance as an energy function then the eigenvalues measure the stiffness of springs that hold the parameters to their optimal setting; small eigenvalues correspond to soft springs. Our goal is to understand the spectrum of spring constants that arise in different systems, using the examples summarized in Table I.

TABLE I:

Summary of the systems used to explore how optimizing a performance function, which we can think of as a surrogate for fitness, impacts variation in system parameters. Each of the five examples show variation near their optima, but in different ways across a wide range of problems and solutions.

system	performance $ℱ$	parameters $\mathit{\theta }$
photoreceptor array	information about visual input	receptor positions
transcription factors in fly embryo	positional information	readout thresholds
small neural circuit (STG)	match to desired period or duty cycle	channel protein copy numbers
recurrent neural network	match to desired filter	synaptic strengths
deep artificial network	image classification performance	synaptic strengths

SAMPLING THE VISUAL WORLD

The lenses of the insect compound eye have a nearly crystalline arrangement, as seen in Fig 1A, and this structure is repeated through several layers of visual processing circuitry [20]. It is an old idea that this structure optimizes the image forming or information gathering capacity of the eye [2, 3]. More generally, we expect that regular sampling will maximize the capture of information, which suggests that the retinae of humans and other vertebrates—with rather random packing of photoreceptor cells, as in Fig 1B [21–24] —must be far from optimal. Perhaps surprisingly, this is not true.

FIG. 1:

Sampling the visual world. (a) A segment of the fly’s eye showing the regular lattice of lenses, which is echoed by a regular lattice of receptors and processing circuitry [25]. (b) Photoreceptor array in a human retina, with cells colored by their spectral sensitivity; note the more random arrangement [21]. (c) Array of $N=1141$ receptors on a regular lattice. (d) The same number of receptors scattered at random. (e) Functional performance—here, information per cell—from Eqs (5, 6), as a function of the signal–to–noise ratio for individual cells in the regular (blue) or random (red) arrangements. The range of SNR is chosen to match estimates for human cones in moderate daylight [26]. Inset shows the spectrum of eigenvalues of the Hessian of the functional performance, from Eq (2).

In this problem, we will take the functional measure of performance to be the information that the array of $N$ receptor responses convey about the visual input, normalized by the number of cells, so that $ℱ=I/N$. The parameters are the positions of the cells on the retina, $\mathit{\theta }=\{{\overrightarrow{x}}_1,{\overrightarrow{x}}_2,\cdots ,{\overrightarrow{x}}_N\}$, so the dimensionality of the parameter space $K=2N$. To calculate $ℱ(\theta )$ we need to describe the statistics of the visual input and the nature of cells’ responses. As explained in Box 1, we assume that receptors respond linearly to the light intensity as seen through an optical blurring function, and that input images are described by the power spectrum of spatial fluctuations in contrast measured in natural scenes [26].

Box 1: Information in a receptor array.

The retina samples the image intensity or contrast $\varphi (\overrightarrow{x})$, which varies across position $\overrightarrow{x}$, with $N$ receptors centered at locations ${\overrightarrow{x}}_{\mathrm{n}},\mathrm{n}=1,2,\cdots ,N$. This sampling involves local spatial averaging, at a minimum from the optics of the eye’s lens(es), and the receptor response $R_{\mathrm{n}}$ will be noisy, so we can write

$$R_{\mathrm{n}}=\int d^{2}xf(\overrightarrow{x}-{\overrightarrow{x}}_{\mathrm{n}})\varphi (\overrightarrow{x})+{\xi }_{\mathrm{n}}.$$ (3)

For simplicity we will assume that both signal and noise are Gaussian, with the noise independent in each receptor, $⟨{\xi }_{\mathrm{n}}{\xi }_{\mathrm{m}}⟩={\delta }_{\mathrm{n}\mathrm{m}}{\sigma }^2$, and the image statistics described by their power spectrum $S(\overrightarrow{k})$. Measurements on natural scenes show that the spectrum is scale invariant [26], with

$$S(\overrightarrow{k})=\frac{A}{|\overrightarrow{k}{|}^{2-\eta }},$$ (4)

and $\eta ~0.2$. We treat the spatial averaging as Gaussian, $f(\overrightarrow{x})=\left(1/2\pi x_0^2\right)\mathrm{e}\mathrm{x}\mathrm{p}\left[-|\overrightarrow{x}{|}^2/2x_0^2\right]$, and assume that the width of this blur is matched to the overall density of receptors $\rho =1/x_0^2$. In this setting we find that the mutual information between the set of responses $\left\{R_{\mathrm{n}}\right\}$ and the image $\varphi (\overrightarrow{x})$ is

$$I=\frac{1}{2}{\log }_2\mathrm{d}\mathrm{e}\mathrm{t}(𝟙+\hat{C}),$$ (5)

and the $N\times N$ matrix $\hat{C}$ has elements

$$C_{\mathrm{n}\mathrm{m}}=\frac{A}{x_0^{\eta }{\sigma }^2}{\int }_0^{\mathrm{\infty }}\frac{dz}{2\pi }\frac{e^{-z^2}}{z^{1-\eta }}{J}_0\left(z{r}_{\mathrm{n}\mathrm{m}}/x_0\right),$$ (6)

where $J_0$ is a Bessel function and $r_{\mathrm{n}\mathrm{m}}=\left|{\overrightarrow{x}}_{\mathrm{n}}-{\overrightarrow{x}}_{\mathrm{m}}\right|$. We can choose $x_0$ as the unit of distance, so the only free parameter is a dimensionless signal–to–noise ratio $SNR=C_{\mathrm{n}\mathrm{n}}$. A natural performance measure is the information per receptor cell, $ℱ=I/N$.

In Figure 1 we see an array of $N=1141$ receptors arranged in a regular lattice (Fig 1C) or completely at random (Fig 1D) across the same area. The resulting information per cell is shown as a function of SNR in Fig 1E. The lattice solution outperforms all random solutions; however, it does so only narrowly across all signal–to–noise ratios. This implies that the space of near-optimal retinal arrays is enormous and is on a scale similar to the entire volume of the configuration space.

Figure 1E provides a global view of the space available for parameter variation with rather small changes in the functional performance of the system. We can give a local analysis, computing the Hessian as in Eq (2), with the spectrum of eigenvalues shown in the inset to Fig 1E, around the optimal crystalline arrangement. The eigenvectors of the Hessian are waves of receptor displacements across the array, as expected from translation invariance with corrections for the boundaries. The eigenvalues scale roughly as $\mathrm{\lambda }~1/{\ell }^2$ for long wavelengths $\ell$, giving a finite density of soft modes, close to $\mathrm{\lambda }=0$, for parameter variation. This means that performance is only a weakly varying function of the parameters, consistent with the result in Fig 1E that even random parameter choices are not so far from optimal.

These results allow us to understand how the human retina can provide near optimal gathering of visual information while being disordered, allowing different instantiations of disorder in different individuals. In this precise sense, optimality and variability can coexist. In the compound eye, the size of the lens is the distance between receptors; formally this would be described in Eq (3) by having a different blurring function $f(\overrightarrow{x})$ for each cell, with a width related to the distances to neighboring cells. But changing the size of the lens has a large effect on the degree of blur at each point in the retina [2, 3], so that disorder in the receptor lattice would drive each lens away from its optimal size. Because of these local constraints on the optics, optimality and variability cannot coexist in compound eyes the way they do in vertebrate eyes, and correspondingly we see near crystalline structures.

INFORMATION AND TRANSCRIPTIONAL REGULATION

Levels of gene expression provide information about variables relevant in the life of the organism. In a developing embryo, for example, it is an old idea that concentrations of crucial “morphogen” molecules carry information about position, driving cell fate decisions that are appropriate to their locations [27], and this can be made precise [28–30]. In the fruit fly all of the relevant molecules have been identified [31, 32], and the concentrations of just four “gap gene” products are sufficient to determine position with an accuracy of ~ 1%, comparable to the precision of downstream events [28, 33].

There is a classical view of gap gene expression as occurring in domains that are essentially on or off [34]. If we interpret the expression levels in this way, as binary variables, then much of the information about position is lost, which means that cells must “measure” intermediate levels [28]. But any realistic mechanism for cells to respond to gap gene expression will be noisy. This points to an optimization problem: Given a limited capacity to measure the concentrations of the relevant molecules, how can cells use this capacity to best capture information about position [35, 36]?

We can think of a limited capacity as being equivalent to discretization, mapping continuous gene expression values $\mathit{g}$ into a small set of alternatives $Z=1,2,···,‖Z‖$. The functional performance measure is the information that this discrete variable provides about a cell’s position $x$ in the embryo, $ℱ=I(Z;x)$, and the parameters $\mathit{\theta }$ are the locations of the thresholds or boundaries that define these discrete alternatives. Importantly, if we have enough samples from the joint distribution of $\mathit{g}$ and $x$ then we can compute $ℱ(\mathit{\theta })$ directly from data. In Figure 2 we do this for the genes hunchback and Krüppel in the early fruit fly embryo; for this pair of genes we know that this simple discretization is a very good approximation to a more general problem of preserving positional information through limited capacity measurements of the gene expression levels (Box 2).

FIG. 2:

Optimizing the response to morphogens. (a) (left) Expression levels of Hb (red) and Kr (blue) as a function of position along the anterior–posterior axis of the embryo [33]. Solid lines are means over many embryos, and shaded regions show the standard deviation at each position. Expression of each gene is normalized so that $0\le ⟨g⟩\le 1$, and position $x$ is measured in units of the embryo length $L$. (right) An image of three “pair–rule” genes [37], whose expression depends on enhancers reading information from all gap genes and hence on the (hypothetical) compressed variable $Z$. (b) Discretizing combinations of expression levels. This is the optimal solution for $\left|\right|Z\left|\right|=10$, with regions bounded by solid lines. Grey levels illustrate the probability density across all embryos and all positions. (c) Hessian matrices (insets), from Eq (2), and their eigenvalues for $\left|\right|Z\left|\right|=3$ and $\left|\right|Z\left|\right|=10$. The peak is normalized to 1.

Box 2: Compression of expression levels.

Cells in the fly embryo respond to the proteins encoded by hunchback and Krüppel as they bind to enhancer elements along the genome, and these in turn interact with promoters to influence the expression of other genes [38]. We can think of this abstractly as a mapping from concentrations $g_{\mathrm{H}\mathrm{b}},g_{\mathrm{K}\mathrm{r}}$ into some internal variable $Z$, which might represent the occupancy of enhancer binding sites or the resulting activity of the promoter [35, 39]. Noise and limited dynamic range in the cellular response imply that this variable captures only limited information $I\left(Z;g_{\mathrm{H}\mathrm{b}},g_{\mathrm{K}\mathrm{r}}\right)$. For each value of this information there is a maximum information that can be represented about the position $x$ of the cell, $I(Z;x)$, and this is determined by the structure of expression profiles and their variability. Formally this problem can be phrased as

$$\underset{P\left(Z\mid g_{\mathrm{H}\mathrm{b}},g_{\mathrm{K}\mathrm{r}}\right)}{\mathrm{m}\mathrm{a}\mathrm{x}}\left[I(Z;x)-\mu I\left(Z;g_{\mathrm{H}\mathrm{b}},g_{\mathrm{K}\mathrm{r}}\right)\right],$$ (7)

where $\mu$ is a Lagrange multiplier and in general the mapping $\left(g_{\mathrm{H}\mathrm{b}},g_{\mathrm{K}\mathrm{r}}\right)\to Z$ can be probabilistic; this is an instance of the information bottleneck problem [40]. We have solved this optimization problem [35], and a very good approximation is to divide the space of expression levels into Vornoi polygons, deterministically (Fig 2B). The Vornoi construction is parameterized by the positions of the central points

$$\mathit{\theta }=\left\{{\mathit{g}}_1,{\mathit{g}}_2,\cdots ,{\mathit{g}}_N\right\};$$ (8)

each ${\mathit{g}}_n$ is a vector with two components, one for hunchback and one for Krüppel, so that $\mathit{\theta }$ has $||Z||=2N$ dimensions. Capturing more information defines a more reliable and complex body plan [6], so we take $ℱ(\theta )=I(Z;x)$ as a measure of functional performance.

Figure 2A shows the spatial profiles of hunchback and Krüppel in a small time window 40−45min into the fourteenth cycle of nuclear division in the embryo. Figure 2B shows the optimal discretization into $\left|\right|Z\left|\right|=10$ regions. Figure 2C shows the eigenvalues of the Hessian in the neighborhood of this optimum, both for $\left|\right|Z\left|\right|=3$ and 10. We see that these eigenvalues are distributed over a decade or more. We emphasize that this emerges directly from the expression level data—mean spatial profiles and the (co)variances of fluctuations around these means—and thus represents a feature of the problem faced by the organism, not a model.

ELECTRICAL DYNAMICS OF SINGLE NEURONS

The electrical dynamics of neurons are determined by the molecular dynamics of ion channels in the cell membrane [41, 42]. A typical neuron expresses several kinds of channels, chosen from roughly one hundred possibilities encoded in the genome. Channels interact with one another through the voltage across the membrane, and these are the protein networks for which we have the most precise mathematical descriptions. The models of these networks predict that stabilizing even the qualitative behavior of neurons requires feedback from electrical activity to channel copy number [43], and these feedback mechanisms now are well established. Here we consider a small network of neurons that generate a rhythm in the crab stomatogastric ganglion, as schematized in Figs 3A and B [9]; these coordinated oscillations create crucial feeding patterns in the crab. A description of the channel dynamics in the simplest three cell network requires twenty parameters $\mathit{\theta }$ that define the number of each type of channel, and we measure functional performance $ℱ$ by the closeness of the period and duty cycle to target values (Box 3).

FIG. 3:

Ion channel copy numbers in a small neural circuit. (a) Schematic of the core rhythm generating network in the crab stomatogastric ganglion [9], showing the anterior burster and pyloric dilator neurons (AB/PD), the lateral pyloric neuron (LP) and the pyloric neurons (PY), as well as their synaptic connections. (b) Voltage vs time for the three neurons, from the model described in Box 3. (c) Eigenvalues $\mathrm{\lambda }$ of the Hessian in Eq (2), where the functional performance is measured by the fractional closeness of the rhythm period (black) or duty cycle of the individual neurons (blue, green, yellow) and the parameters are the copy numbers of the different channel types in each cell.

Box 3: Ion channel dynamics.

Electrical dynamics in single neurons are described by generalized Hodgkin–Huxley equations [44]:

$$C_n\frac{d{V}_n}{dt}=-g_n^{\text{leak}}{V}_n-\sum _i{g}_{ni}^{\mathrm{m}\mathrm{a}\mathrm{x}}{m}_{ni}^{{\alpha }_i}{h}_{ni}^{{\beta }_i}\left(V_n-{\bar{V}}_{ni}\right)+I_n^{\text{syn}},$$ (9)

$$\frac{d{m}_{ni}}{dt}=-\frac{1}{{\tau }_i^m\left(V_n\right)}\left[m_{ni}-{\bar{m}}_i\left(V_n\right)\right],$$ (10)

$$\frac{d{h}_{ni}}{dt}=-\frac{1}{{\tau }_i^h\left(V_n\right)}\left[h_{ni}-{\bar{h}}_i\left(V_n\right)\right].$$ (11)

The variable $V_n$ is the voltage across the membrane of cell $n$ and $C_n$ is its capacitance. There are many types of ion channel proteins that can insert into the cell membranes, and these are labeled by $i;m_{ni}$ and $h_{ni}$ are the “activation” and “inactivation” variables, respectively, for these channels, reflecting the states of the proteins. Given the ionic selectivity of each channel type and the chemical potential differences of the ions across each cell membrane, current through channels of type $i$ reverses at a voltage ${\bar{V}}_{ni}$ in cell $n$. If these channels are fully open the maximal conductance is $g_{ni}^{\mathrm{m}\mathrm{a}\mathrm{x}}$, proportional to the number of copies of that protein in the membrane. When the voltage across the membrane is $V$, the equilibrium values of the activation and inactivation variables are ${\bar{m}}_i(V)$ and ${\bar{h}}_i(V)$, respectively, and relaxation toward these equilibrium values occurs with time constants ${\tau }_i^m(V)$ and ${\tau }_i^h(V)$; these voltage dependencies are well characterized for each channel type [41]. Each cell has a “leak” conductance $g_n^{\text{leak}}$ that pulls the voltage back toward zero. Current is injected into each cell by synapses from other neurons,

$$I_n^{\mathrm{s}\mathrm{y}\mathrm{n}}=\sum _k{g}_{nk}^s{s}_k\left(V_n-E_{nk}^s\right)$$ (12)

$$\frac{d{s}_k}{dt}=-\frac{1}{{\tau }_k^s\left(V_k\right)}\left[s_k-{\bar{s}}_k\left(V_k\right)\right].$$ (13)

The input current to the postsynaptic neuron $n$ depends on the momentary activation of the synapse from cell $k,s_k$, the maximal synapse strength $g_{nk}^s$, and the difference between the membrane potential of the postsynaptic neuron and the reversal potential of the synapse, $E_{nk}$. The activation of the synapse relaxes on a timescale ${\tau }_k^s\left(V_k\right)$ toward a steady state ${\bar{s}}_k\left(V_k\right)$. The functions ${\bar{s}}_k\left(V_k\right)$ and ${\tau }_k^s\left(V_k\right)$ are modeled as in Ref [45], with ${\bar{s}}_k\left(V_k\right)$ taking the form of a sigmoid In small circuits as in Fig 3 we find solutions to Eqs (9–13) that are periodic, so that $V_n(t)=V_n(t+T)$. One measure of fitness is the (fractional) approach of this period to some desired value, $ℱ={ℱ}_{\text{max}}-{\left(T-T_{\mathrm{*}}\right)}^2/T_{\mathrm{*}}^2$. During the period $T$ each neuron spends part of the cycle in firing state and part of the cycle not firing. We can define a duty cycle $D$ for each neuron by first inferring the timing $t_i$ of each of the firing events within a single burst of firing, and then finding the length $\tau =t_f-t_1$ of the firing burst. Finally, we can define the duty cycle as $D=\tau /T$. Then, we define a fitness as the fractional approach of this duty cycle to a desired value for each cell $n,{ℱ}_n={ℱ}_{\text{max}}-{\left(D_n-D_{n,\mathrm{*}}\right)}^2/D_{n,\mathrm{*}}^2$. We take the parameters to be the channel protein copy numbers or maximal conductances, $\mathit{\theta }=\left\{g_{ni}^{\mathrm{m}\mathrm{a}\mathrm{x}}\right\}$. Hessians were estimated through numerical simulation using the Xolotl package [46], using the “example pyloric network” to define $D_{n,\mathrm{*}}$ and $T_{\mathrm{*}}$.

Figure 3C shows the eigenvalues of the Hessian, from Eq (2). Whether we focus on the period of the oscillation or the duty cycles of individual neurons, the results are strikingly similar, and show that eigenvalues are distributed almost uniformly on a logarithmic scale. The dynamic range is ~ 10¹⁰, which means that some combinations of parameters can vary by one hundred thousand times as much as others while having the same effect on the performance. Said another way, to hold all parameters fixed would require that variations of ~ 10⁻¹⁰ are significant.

A RECURRENT NETWORK

Going beyond single neurons to explore the dynamics of networks we can see more examples of these ideas. We consider first a network with recurrent internal connections and a linear readout (Fig 4A). Nonlinear versions of these networks have been widely discussed as generators of complex outputs $z(t)$, as in motor control [47–49]. A much simpler case is where the dynamics are linear and there is a single input $u(t)$ (Box 4), in which case the output is a linearly filtered version of this signal,

$$z\left(t\right)=\int d{t}^{\prime }f\left(t-t^{\prime }\right)u\left(t^{\prime }\right).$$ (14)

FIG. 4:

A linear recurrent network of $N=100$ neurons. (a) Schematic network architecture (left). A network with internal connections $J_{\mathrm{i}\mathrm{j}}$, is driven by a signal $\mathit{u}(\mathit{t})$ via an input weight vector $\mathbf{m}$, and the output $\mathit{z}(\mathit{t})$ is a weighted sum of network activity via the readout weights $\mathbf{n}$. Visualization of the network filter (right), from Eq (14). (b) Hessian eigenvalues $\mathrm{\lambda }$ of the cost function, Eq (16), for a random filter $f_{\mathrm{*}}(\tau )$ when perturbing the readout weights $n_i$; different values of $\mathrm{g}$ in colors. (c) As in (B) when perturbing the recurrent connectivity weights $J_{\mathrm{i}\mathrm{j}}$.

Box 4: A linear network.

We consider a linear network of $N$ neurons in which the activity of each neuron is described by a variable $x_{\mathrm{i}}$. This activity relaxes to its resting state on time scale ${\tau }_0$, and $J_{\mathrm{i}\mathrm{j}}$ measures the strength of the connection from neuron $\mathrm{j}$ to neuron $\mathrm{i}$. There is a single input $u(t)$ that drives each neuron $\mathrm{i}$ with a fixed weight $m_{\mathrm{i}}$, so the full dynamics are

$${\tau }_0\frac{d{x}_{\mathrm{i}}}{dt}=-x_{\mathrm{i}}+\sum _{\mathrm{j}=1}^N{J}_{\mathrm{i}\mathrm{j}}{x}_{\mathrm{j}}+m_{\mathrm{i}}u\left(t\right).$$ (15)

The output of the network is some linear combination of activities, $z(t)=\sum _{\mathrm{i}}{n}_{\mathrm{i}}{x}_{\mathrm{i}}(t)$. Linearity of the dynamics implies that $z(t)$ is a linearly filtered version of $u(t)$, as in Eq (14). We measure functional performance as the similarity of this filter to some target,

$$ℱ(\overrightarrow{\mathit{\theta }})={ℱ}_{\mathrm{m}\mathrm{a}\mathrm{x}}-\int d\tau {\left|f\left(\tau \right)-f_{\mathrm{*}}\left(\tau \right)\right|}^2.$$ (16)

We can make a generic choice of the target by drawing each synaptic strength $J_{\mathrm{i}\mathrm{j}}$ independently from a Gaussian distribution with zero mean and $⟨{\left(J_{\mathrm{i}\mathrm{j}}\right)}^2⟩=g^2/N$, and similarly picking weights $\left\{m_{\mathrm{i}};n_{\mathrm{i}}\right\}$ from zero mean unit variance Gaussians. In the large $N$ limit, the eigenvalues of $J_{\mathrm{i}\mathrm{j}}$ are uniformly distributed on a disc centered at the origin of the complex plane, with radius $g$. Through Equation (15) this means that when $g\to 1$ the network dynamics have increasingly longer timescales, and when $g>1$ the dynamics destabilize.

We can take as a measure of functional behavior $ℱ$ the closeness of this filter to some desired target $f_{\mathrm{*}}(\tau )$. The parameters are the strengths of internal synapses and the readout weight, $\mathit{\theta }=\left\{J_{\mathrm{i}\mathrm{j}};n_{\mathrm{i}}\right\}$.

We can make a generic choice of the target filter $f_{\mathrm{*}}(\tau )$ by setting all the parameters to random values. Note that as we increase the variance of the synaptic strengths ($g/N$ from Box 4), the network has access to a wider range of time scales and these will be reflected in $f_{\mathrm{*}}(\tau )$. We can gain some insight by first varying only the $N$ readout weights $\left\{n_{\mathrm{i}}\right\}$. We find the eigenvalues of the Hessian matrix shown in Fig 4B, illustrating once again the tendency for these values to be equally spaced on a logarithmic axis. At larger $g$, and hence a wider range of time scales, we find a larger number of eigenvalues that are significantly nonzero, but at all values of $g$ there is a range of more than ten orders of magnitude. Indeed, as $g\to 1$ the spectrum becomes almost perfectly uniform across this range.

When we change the $N^2$ synaptic strengths we find the eigenvalues of the Hessian shown in Fig 4C. To understand this result we can think not about the individual elements of the synaptic matrix $J_{\mathrm{i}\mathrm{j}}$ but about the eigenvalues and eigenvectors of this matrix. If perturbations to the network parameters change only the eigenvectors, then we can compensate by adjusting the readout weights. But this means that there are $N(N-1)$ parameters of the synaptic matrix that are effectively redundant, so we see only $N$ significantly nonzero eigenvalues. Beyond true redundancy or over–parameterization, even the nonzero eigenvalues are widely distributed (Fig 4C). These eigenvalues span roughly five orders of magnitude, and except for a few they are distributed uniformly on a logarithmic axis, as in our other examples.

A DEEP NETWORK

Modern artificial neural networks began as schematic models for networks of real neurons. While not faithful to the details of the brain’s dynamics, they capture important functional behaviors that are at the heart of the current revolution in artificial intelligence. Although it often is emphasized these networks have very large numbers of parameters, networks of real neurons have even more. Here we use deep networks [50] as a controlled test case to further explore the interplay of optimization and variability.

Networks that are trained on classification tasks are models for a conditional distribution $Q_{\mathit{\theta }}(C\mid \mathbf{x})$, where $C$ is the class, $\mathbf{x}$ is the $D$–dimensional input such as an image, and $\mathit{\theta }$ are the (many) parameters. The inputs themselves are drawn from a distribution $P(\mathit{x})$. The performance of the model can be measured by the average (log) probability that the model can explain the data we have seen,

$$ℱ\left(\mathit{\theta }\right)=+\frac{1}{N}\sum _{\mathrm{i}=1}^N\log Q_{\mathit{\theta }}\left(C_{\mathrm{i}}\mid {\mathbf{x}}_{\mathrm{i}}\right),$$ (17)

where $C_{\mathrm{i}}$ is the class to which the input ${\mathbf{x}}_{\mathrm{i}}$ belongs. As usual in neural networks the parameters $\mathit{\theta }$ are the connection weights and biases for activation of individual units. Note that our distinction between functional performance and model fit becomes blurred in this example.

Because the network implements a probabilistic model, and the model is large enough to achieve perfect performance, the Hessian matrix of $ℱ$ is equal to the Fisher information matrix [51] at the optimum, but the Fisher matrix is well defined and positive definite even away from the optimum. This allows us to ask, for example, how the soft modes emerge over the course of learning.

In most modern applications of neural networks, and perhaps in the brain itself, there are more parameters than examples from which to learn, and so we expect that some combinations of parameters are completely unconstrained, generating true zero eigenvalues in the Hessian matrix. But there are hints in the literature that beyond these genuinely undetermined parameters there is a near continuum soft modes [52]. To explore this further we consider ResNet20, a twenty layer network [53, 54], that is trained to 100% accuracy—and hence the absolute maximum $ℱ$—on the CIFAR–10 task, in which 32 × 32 color images must be assigned to $\left|C\right|=10$ classes [55]. A schematic of the network is shown in Fig 5A, and Fig 5B shows that learning indeed reaches the optimum, with perfect classification of the training data. This network has $K~\mathrm{270,000}$ parameters, and the Hessian matrix is $K\times K$. As explained in Box 5, we have developed finite rank approximations to the Fisher information matrix that involve a direct average over samples.

FIG. 5:

Eigenvalue spectrum of the Fisher information matrix for a network that learns the CIFAR–10 task. (a) Schematic of the network. (b) Performance of the network as function of time (“epoch”) during training. Training reaches a true optimum—zero loss—both for the natural task and for an artificial task in which the labels on images are randomized. (c) Eigenvalues of the Fisher information matrix $F$ estimated from $\tilde{F}$ as in Eq (24). These estimates are based on $n=1500$ samples, probing the spectrum at different stages in the learning process. The distribution of eigenvalues converges as we increase $n$, especially in the tail of small eigenvalues.

Box 5: The Hessian in large networks.

As before we are interested in the Hessian

$$H_{\mu \nu }\equiv -{\left.\frac{{\partial }^2ℱ\left(\mathit{\theta }\right)}{\partial {\theta }_{\mu }\partial {\theta }_{\nu }}\right|}_{\mathit{\theta }={\mathit{\theta }}^{\mathrm{*}}}.$$ (18)

At large $N$ we can replace the average over samples with an average over the true distribution $P(C|\mathit{x})$. But in modern deep networks, the very large number of parameters makes it possible to achieve perfect classification on the set of training examples, so that $P(C\mid \mathbf{x})=Q_{{\theta }^{\mathrm{*}}}(C\mid \mathbf{x})$. When this is true, the Hessian at the optimum is the Fisher information matrix,

$$H_{\mu \nu }=F_{\mu \nu }({\mathit{\theta }}^{\mathrm{*}})$$ (19)

$$F_{\mu \nu }(\theta )=⟨\frac{\partial \mathrm{l}\mathrm{o}\mathrm{g}{Q}_{\mathit{\theta }}(C\mid \mathbf{x})}{\partial {\theta }_{\mu }}\frac{\partial \mathrm{l}\mathrm{o}\mathrm{g}{Q}_{\mathit{\theta }}(C\mid \mathbf{x})}{\partial {\theta }_{\nu }}⟩$$ (20)

$$⟨\dots ⟩=\int d^{D}xP\left(\mathbf{x}\right)\sum _C{Q}_{\mathit{\theta }}\left(C∣\mathbf{x}\right)\left(\cdots \right).$$ (21)

We note that the Fisher information matrix is a property of the model and not of the training data, except insofar as the training drives us to a particular set of parameters ${\mathit{\theta }}^{\mathrm{*}}$. Because $F_{\mu \nu }$ is an expectation value we can approximate it by drawing $n$ examples out of the true distribution (not necessarily examples used for training). Concretely we can construct

$$F_{\mu \nu }^{(n)}=\frac{1}{n}\sum _{\mathrm{i}=1}^n\sum _C{A}_{\mu ,(\mathrm{i}C)}{A}_{\nu ,(\mathrm{i}C)}$$ (22)

$$A_{\mu ,(\mathrm{i}C)}=\frac{\partial \mathrm{l}\mathrm{o}\mathrm{g}{Q}_{\mathit{\theta }}\left(C\mid {\mathbf{x}}_{\mathrm{i}}\right)}{\partial {\theta }_{\mu }}$$ (23)

This is a rank $n|C|$ approximation to the true matrix $F_{\mu \nu }$ and the eigenvalues of $F_{\mu \nu }^{(n)}$ are the same as the $n$ nonzero eigenvalues of

$${\tilde{F}}_{(\mathrm{i}C),\left(\mathrm{j}{C}^{\prime }\right)}^{(n)}=\frac{1}{n}{[{\hat{A}}^T\hat{A}]}_{\mathrm{i}\mathrm{j}}=\frac{1}{n}\sum _{\mu =1}^K{A}_{\mu ,(\mathrm{i}C)}{A}_{\mu ,\left(\mathrm{j}{C}^{\prime }\right)}.$$ (24)

Handling the $n|C|\times n|C|$ matrix $\tilde{F}$ is a manageable task for reasonably large $n$, so long as $n|C|\ll K$. The eigenvalues of $F^{(n)}$ approach those of $F$ as $n$ increases, and we can find these eigenvalues by diagonalizing ${\tilde{F}}^{(n)}$. If the eigenvalue spectrum of ${\tilde{F}}^{(n)}$ stabilizes as $n$ becomes large, then we are finding a good approximation to the spectrum of $F$ itself.

We see in Figure 5C that the Hessian has just a handful of large eigenvalues; as noted previously the number of these “stiff” directions is equal to the number of classes $\left|C\right|$ [52, 56]. As learning progresses, the density of moderate eigenvalues is depleted and added to the tail at small eigenvalues, which becomes nearly scale invariant $\rho (\mathrm{\lambda })~1/\mathrm{\lambda }$ so that the spectrum is barely integrable; there is no obvious limit to how soft these modes can be. Importantly, the spectrum we see converges rapidly as we increase the rank of our approximation.

The classical examples of sloppy models involve eigenvalues that are distributed almost uniformly on a logarithmic scale, which is equivalent to the scale invariant distribution $\rho (\mathrm{\lambda })~1/\mathrm{\lambda }$ that we see here. It is striking that we see this not only across many decades in $\mathrm{\lambda }$, but across many thousands of dimensions. We emphasize that this uniform distribution across scales is not a property of the generic deep network, but emerges during training, as we can see by comparing different colors in Fig 5C. More subtly, if we change the task by randomly permuting the labels we can still reach optimal performance (Fig 5B), but in Fig 5C we see that the density of moderate and large eigenvalues now grows over the course of training at the expense of the very smallest eigenvalues.

SYNTHESIS

Across all of the systems highlighted here, we observe a large number of soft modes in parameter space. This describes quantitatively the flexibility of parameter choices that allow equivalent and very nearly optimal performance. As parameter spaces expand, the scaling of the eigenvalues that we observe admits yet more soft modes. Our framework allows us to see the generality of these relationships across vastly different biological substrates and cost functions.

In thinking about the possibility of parameter variation it is useful to imagine a population of cells, networks, or organisms that choose their underlying parameters $\mathit{\theta }$ at random from a distribution $P(\mathit{\theta })$. This population will exhibit an average performance or fitness $\overline{ℱ}$,

$$\overline{ℱ}=\int d^K\theta P\left(\mathit{\theta }\right)ℱ\left(\mathit{\theta }\right).$$ (25)

Environmental and evolutionary pressures will push for this average to reach some characteristic level. Unless there is some mechanistic or historical constraint beyond the fitness itself, we expect that the parameters will be as variable as possible while obeying the mean value in Eq (25).

The only consistent way to make “as variable as possible” mathematically precise is to ask for the distribution $P(\mathit{\theta })$ that has the maximum entropy consistent with the constraint on $\overline{ℱ}$ [57, 58]. This maximum entropy distribution has the form

$$P\left(\mathit{\theta }\right)=\frac{1}{Z\left(T\right)}\exp \left[+\frac{1}{T}ℱ\left(\mathit{\theta }\right)\right],$$ (26)

where $T$ is a parameter that must be set so that Eq (25) is satisfied and $Z(T)$ enforces the normalization of the distribution. We recognize this as the Boltzmann distribution with the performance measure playing the role of the (negative) energy and $T$ the temperature.

A seemingly different view of the problem is that there is some added cost in controlling the parameters, and this competes with the fitness advantage that can be gained by tuning towards an optimum. If the cost is proportional to the information, in bits, that is used to specify the parameters, then achieving a certain mean fitness at minimal cost means that the distribution of parameters will again be drawn from Eq (26). In this picture the effective temperature $T$ measures the relative cost of the information needed for control vs the gain in fitness.

If the evolutionary pressure is strong enough to keep the mean performance close to its maximum, then we can use the expansion in Eq (1), and in agreement with the equipartition theorem, we find

$$\overline{ℱ}={ℱ}_{\mathrm{m}\mathrm{a}\mathrm{x}}-\frac{K}{2}T,$$ (27)

independent of the details of the eigenvalue spectrum. We can explicitly insert our knowledge of the eigenvalue spectrum into the expression for the entropy of the distribution,

$$S=-\int d^K\theta P(\mathit{\theta })\mathrm{l}\mathrm{n}P(\mathit{\theta })=\frac{1}{2}\sum _{\mu =1}^K\mathrm{l}\mathrm{n}\left(2\pi eT/{\lambda }_{\mu }\right).$$ (28)

If we have a sloppy spectrum with

$${\mathrm{\lambda }}_{\mu }={\mathrm{\lambda }}_{\mathrm{m}\mathrm{a}\mathrm{x}}{e}^{-\alpha \mu }.$$ (29)

then

$$S=\frac{K}{2}\mathrm{l}\mathrm{n}\left(2\pi eT/{\lambda }_{\mathrm{m}\mathrm{a}\mathrm{x}}\right)+\frac{\alpha }{4}K(K+1).$$ (30)

In this way, we can relate the entropy to the temperature and dimensionality, $K$, of the system. Solving for $T$ and inserting the expression into Eq (27), we can also probe the scaling of the cost function with dimensionality. Putting things together we have

$$\mathrm{\Delta }ℱ\equiv {\overline{ℱ}}_{\mathrm{m}\mathrm{a}\mathrm{x}}-\overline{ℱ}=\frac{K{\mathrm{\lambda }}_{\mathrm{m}\mathrm{a}\mathrm{x}}}{4\pi e}\exp \left[\frac{2S}{K}-\frac{\alpha }{2}\left(K+1\right)\right].$$ (31)

This implies that if the the dimensionality of the parameter space is large ($K\to \infty$) we can get arbitrarily close to optimal performance, on average, even though the variability corresponds to a finite entropy per parameter.

It is useful to think about the case where parameter spaces are so large that the discrete Hessian eigenvalues are drawn from a nearly continuous distribution with normalized density $\rho (\mathrm{\lambda })$, as in Fig 5. Then the analog of Eq (31) is

$$\mathrm{\Delta }ℱ=\frac{K{\mathrm{\lambda }}_{\mathrm{m}\mathrm{a}\mathrm{x}}}{4\pi e}\mathrm{e}\mathrm{x}\mathrm{p}\left[\frac{2S}{K}+{\int }_{{\mathrm{\lambda }}_{\mathrm{m}\mathrm{i}\mathrm{n}}}^{{\mathrm{\lambda }}_{\mathrm{m}\mathrm{a}\mathrm{x}}}d\lambda \rho (\lambda )\mathrm{l}\mathrm{n}\left(\mathrm{\lambda }/{\mathrm{\lambda }}_{\mathrm{m}\mathrm{a}\mathrm{x}}\right)\right].$$ (32)

The condition to have finite entropy per parameter at vanishing fitness cost is that the dynamic range of $\mathrm{l}\mathrm{n}\mathrm{\lambda }$ grow with the dimensionality $K$ of the parameter space. This growth is linear if the sloppy spectrum in Eq (29) is exact, but this can happen in a wider range of systems.

Similar considerations appear in many examples beyond those discussed here. In some of these problems the parameters are discrete, as for the mapping of amino acid sequences into protein structure and function, but the principles are the same. The fact that proteins form families is a hint that the mapping from sequence to structure is many–to–one [59–61], and this observation is at the root of a large literature that builds statistical physics models for the distribution of sequences consistent with a given structure [62–66]. Within these models one can estimate the entropy of sequence variations at fixed structure, analogous to the entropy at fixed mean performance discussed here; again this entropy appears to be extensive in the number of amino acids [67]. This is not a generic feature of proteins, and it has been suggested that evolution selects for structures that are the stable folded states of many sequences, in effect maximizing the entropy in parameter space [68, 69]. Direct experimental explorations of sequence space also suggest that the landscape may be more easily navigable than previously expected [70, 71].

On a higher level of organization, many cognitive tasks can be thought of as sequential decision making, leading to a natural discrete parameterization of the space of strategies. There is a regime in which tasks are complex enough to be interesting but small enough that this space can be explored exhaustively. For foraging tasks, this enumeration reveals a broad swath of strategies that differ significantly in form but have negligible loss in performance, very much analogous to the results above [72].

CONCLUSION

Contrary to a widely held intuition, near–optimal performance of biological systems does not require fine tuning of parameters. We have seen this in examples across scales, from the control of gene expression to the dynamics of large neural networks. This is not a generic feature of high–dimensional, “over–parameterized” systems, but rather emerges from an interplay between the underlying mechanistic architecture and the structure of the problem that the system is selected to solve. Nor is the tolerance for parameter variation a retreat from optimality to “good enough” solutions; rather it is a feature of the landscape for optimization itself. Optimization principles for the function of living systems thus predict that there should be substantial variation in the underlying parameters. We should not be surprised by optimality, but rather only by those rare cases where microscopic parameters are controlled beyond what seems necessary for function.