Balanced excitation and inhibition are required for high-capacity, noise-robust neuronal selectivity

Significance

Neurons and networks in the cerebral cortex must operate reliably despite multiple sources of noise. Using a mathematical analysis and model simulations, we show that noise robustness requires synaptic connections to be in a balanced regime in which excitation and inhibition are strong and largely cancel each other. Our theory predicts an optimal ratio for the number of excitatory and inhibitory synapses that depends on the statistics of afferent activity and is consistent with data. This distinct form of excitation–inhibition balance is essential for robust neuronal selectivity and crucial for stability in associative memory networks, and it emerges automatically from learning in the presence of noise.

Abstract

Neurons and networks in the cerebral cortex must operate reliably despite multiple sources of noise. To evaluate the impact of both input and output noise, we determine the robustness of single-neuron stimulus selective responses, as well as the robustness of attractor states of networks of neurons performing memory tasks. We find that robustness to output noise requires synaptic connections to be in a balanced regime in which excitation and inhibition are strong and largely cancel each other. We evaluate the conditions required for this regime to exist and determine the properties of networks operating within it. A plausible synaptic plasticity rule for learning that balances weight configurations is presented. Our theory predicts an optimal ratio of the number of excitatory and inhibitory synapses for maximizing the encoding capacity of balanced networks for given statistics of afferent activations. Previous work has shown that balanced networks amplify spatiotemporal variability and account for observed asynchronous irregular states. Here we present a distinct type of balanced network that amplifies small changes in the impinging signals and emerges automatically from learning to perform neuronal and network functions robustly.

The response properties of neurons in many brain areas including cerebral cortex are shaped by the balance between coactivated inhibitory and excitatory synaptic inputs (1–5) (for a review see ref. 6). Excitation–inhibition balance may have different forms in different brain areas or species and its emergence likely arises from multiple mechanisms. Theoretical work has shown that, when externally driven, circuits of recurrently connected excitatory and inhibitory neurons with strong synapses settle rapidly into a state in which population activity levels ensure a balance of excitatory and inhibitory currents (7, 8). Experimental evidence in some systems indicates that synaptic plasticity plays a role in maintaining this balance (9–12). Here we address the question of what computational benefits are conferred by the excitation–inhibition balance properties of balanced and unbalanced neuronal circuits. Although it has been shown that networks in the balanced states have advantages in generating a fast and linear response to changing stimuli (7, 8, 13, 14), the advantages and disadvantages of excitation–inhibition balance for general information processing have not been elucidated [except in special architectures (15–17)]. Here we compare the computational properties of neurons operating with and without excitation–inhibition balance and present a constructive computational reason for strong, balanced excitation and inhibition: It is needed for neurons to generate selective responses that are robust to output noise, and it is crucial for the stability of memory states in associative memory networks. The distinct balanced networks we present here naturally and automatically emerge from synaptic learning that endows neurons and networks with robust functionality.

We begin our analysis by considering a single neuron receiving input from a large number of afferents. We characterize its basic task as discriminating patterns of input activation to which it should respond by firing action potentials from other patterns which should leave it quiescent. Neurons implement this form of response selectivity by applying a threshold to the sum of inputs from their presynaptic afferents. The simplest (parsimonious) model that captures these basic elements is the binary model neuron (18, 19), which has been studied extensively (20–23) and used to model a variety of neuronal circuits (24–28). Our work is based on including and analyzing the implications of four fundamental neuronal features not previously considered together: (i) nonnegative input, corresponding to the fact that neuronal activity is characterized by firing rates; (ii) a membrane potential threshold for neuronal firing above the resting potential (and hence a silent resting state); (iii) sign-constrained and bounded synaptic weights, meaning that individual synapses are either excitatory or inhibitory and the total synaptic strength is limited; and (iv) two sources of noise, input and output noise, representing fluctuations arising from variable stimuli and inputs and from processes within the neuron. As will be shown, these features imply that, when the number of input afferents is large, synaptic input must be strong and balanced if the neuron’s response selectivity is to be robust. We extend our analysis to recurrently connected networks storing long-term memory and find that similar balanced synaptic patterns are required for the stability of the memory states against noise. In addition, maximizing the performance of neurons and networks in the balanced state yields a prediction for the optimal ratio of excitatory to inhibitory inputs in cortical circuits.

Results

Our model neuron is a binary unit that is either active or quiescent, depending on whether its membrane potential is above or below a firing threshold. The potential, labeled $V_{\mathrm{PSP}}$, is a weighted sum of inputs $x_i$, $i=\mathrm{\hspace{0.17em}1},2,\mathrm{\dots },N$, that represent afferent firing rates and are thus nonnegative,

$$V_{\mathrm{PSP}}(\text{𝒙},\text{𝒘})=V_{\mathrm{rest}}+\sum _{i=1}^N{w}_i{x}_i,$$ [1]

where $V_{\mathrm{rest}}$ is the resting potential of the neuron and x and w are $N$-component vectors with elements $x_i$ and $w_i$, respectively. The weight $w_i$ represents the synaptic efficacy of the $i$th input. If $V_{\mathrm{PSP}}\ge V_{\mathrm{th}}$ the neuron is in an active state; otherwise, it is in a quiescent state. To implement the segregation of excitatory and inhibitory inputs, each weight is constrained so that $w_i\ge \mathrm{\hspace{0.17em}0}$ if input $i$ is excitatory and $w_i\le \mathrm{\hspace{0.17em}0}$ if input $i$ is inhibitory.

To function properly in a circuit, a neuron must respond selectively to an appropriate set of inputs. To characterize selectivity, we define a set of $P$ exemplar input vectors ${\text{𝒙}}^{\mu }$, with $\mu =\mathrm{1,2},\mathrm{\dots },P$, and randomly assign them to two classes, denoted as “plus” and “minus.” The neuron must respond to inputs belonging to the plus class by firing (active state) and to the minus class by remaining quiescent. This means that the neuron is acting as a perceptron (18–22, 25, 27, 29). We assume the $P$ input activations, ${\text{𝒙}}^{\mu }$, are drawn identically and independently from a distribution with nonnegative means, $\overline{\text{𝒙}}$, and covariance matrix, $C$ (when $N$ is large, higher moments of the distribution of x have negligible effect). For simplicity we assume that the stimulus average activities are the same for all input neurons within a population, so that ${\overline{x}}_i={\overline{x}}_{\mathrm{exc}\left(\mathrm{inh}\right)}\ge \mathrm{\hspace{0.17em}0}$, and that $C$ is diagonal with equal variances within a population, ${\sigma }_i^2={\sigma }_{\mathrm{exc}\left(\mathrm{inh}\right)}^2$. Note that synaptic weights are in units of membrane potential over input activity levels (firing rates) and hence will be measured in units of $\left(V_{\mathrm{th}}-V_{\mathrm{rest}}\right)/{\sigma }_{\mathrm{exc}}$.

We call weight vectors that correctly categorize the $P$ exemplar input patterns, ${\text{𝒙}}^{\mu }$ for $\mu =\mathrm{\hspace{0.17em}1},2,\mathrm{\dots },P$, solutions of the categorization task presented to the neuron. Before describing in detail the properties of the solutions, we outline a broad distinction between two types of possible solutions. One type is characterized by weak synapses, i.e., individual synaptic weights that are inversely proportional to the total number of synaptic inputs, $w_i\sim \mathrm{\hspace{0.17em}1}/N$ [note that weights weaker than $\mathcal{O}\left(1/N\right)$ will not enable the neuron to cross the threshold]. For this solution type, the total excitatory and inhibitory parts of the membrane potential are of the same order as the neuron’s threshold. An alternative scenario is a solution in which individual synaptic weights are relatively strong, $w_i\sim \mathrm{\hspace{0.17em}1}/\sqrt{N}$. In this case, both the total excitatory and inhibitory parts of the potential are, individually, much greater than the threshold, but they make approximately equal contributions, so that excitation and inhibition tend to cancel, and the mean $V_{\mathrm{PSP}}$ is close to threshold. We call the first type of solution unbalanced and the second type balanced. Importantly, since both balanced and unbalanced solutions solve the categorization task with the same value of $V_{\mathrm{th}}$, the two solution types are not related to each other by a global scaling of the weights but represent different patterns of $\left\{w_i\right\}$. Note that the norm of the weight vector, $\left|\text{𝒘}\right|=\sqrt{{\sum }_{i=1}^N{w}_i^2}$, serves to distinguish the two types of solutions. This norm is of order of $1/\sqrt{N}$ for unbalanced solutions and of order $1$ in the balanced case. Weights with norms stronger than $\mathcal{O}\left(1\right)$ lead to membrane potential values that are much larger in magnitude than the neuron’s threshold. For biological neurons postsynaptic potentials of such magnitude can result in very high, unreasonable firing rates (although see ref. 30). We therefore impose an upper bound of the weight norm $\left|\text{𝒘}\right|\le \mathrm{\Gamma }$, where $\mathrm{\Gamma }$ is of order 1. We now argue that the differences between unbalanced and balanced solutions have important consequences for the way the system copes with noise.

As mentioned above, neurons in the central nervous system are subject to multiple sources of noise, and their performance must be robust to its effects. We distinguish two biologically relevant types of noise: input noise resulting from the fluctuations of the stimuli and sensory processes that generate the stimulus-related input x and output noise arising from afferents unrelated to a particular task or from biophysical processes internal to the neuron, including fluctuations in the effective threshold due to spiking history and adaptation (31–33) (for theoretical modeling see ref. 34). Both sources of noise result in trial-by-trial fluctuations of the membrane potential $V_{\mathrm{PSP}}$ and, for a robust solution, the probability of changing the state of the output neuron relative to the noise-free condition must be low. The two sources of noise differ in their dependence on the magnitude of the synaptic weights. Because input noise is filtered through the same set of synaptic weights as the signal, its effect on the membrane potential is sensitive to the magnitude of those weights. Specifically, if the trial-to-trial variability of each input $x_i^{\mu }$ is characterized by SD ${\sigma }_{\mathrm{in}}$, the fluctuations it generates in the membrane potential have SD $|\text{𝒘}|{\sigma }_{\mathrm{in}}$ (Fig. 1, Top Left and Top Right). On the other hand, the effect of output noise is independent of the synaptic weights w. Output noise characterized by SD ${\sigma }_{\mathrm{out}}$ induces membrane potential fluctuations with the same SD ${\sigma }_{\mathrm{out}}$ for both types of solutions (Fig. 1, Bottom Left and Bottom Right).

Fig. 1.

Only balanced solutions can be robust to both input and output noise. Each panel depicts membrane potentials resulting from different input patterns in a classification task. Weights are unbalanced [$\left|\text{𝒘}\right|=\mathcal{O}(1/\sqrt{N})$, Top Left and Bottom Left] or balanced [$\left|\text{𝒘}\right|=\mathcal{O}\left(1\right)$, Top Right and Bottom Right]. The neuron is in an active state only if the membrane potential is greater than the threshold $V_{\mathrm{th}}$. The input pattern class (plus or minus) is specified by the squares underneath the horizontal axis. Each input pattern determines a membrane potential (mean, horizontal bars) that fluctuates from one presentation to another due to input noise (Top Left and Top Right) and output noise (Bottom Left and Bottom Right). Vertical bars depict the magnitude of the noise in each case. The variability of the mean $V_{\mathrm{PSP}}$ across input patterns (which is the signal differentiating input pattern classes) is proportional to $\left|\text{𝒘}\right|$. As a result, the mean $V_{\mathrm{PSP}}$s for unbalanced solutions (Top Left and Bottom Left) cluster close to the threshold [difference from threshold $\mathcal{O}(1/\sqrt{N})$]. For balanced solutions (Top Right and Bottom Right), the mean $V_{\mathrm{PSP}}$s have a larger spread [potential difference $\mathcal{O}\left(1\right)$]. Input noise (fluctuations of $x_i$, Top Left and Top Right) produces membrane potential fluctuations with SD that is proportional to $|\text{𝒘}|$, which is of $\mathcal{O}(1/\sqrt{N})$ for unbalanced solutions (Top Left) and of $\mathcal{O}\left(1\right)$ for balanced solutions (Top Right). Output noise (Bottom Left and Bottom Right) produces membrane potential fluctuations that are independent of $\left|\text{𝒘}\right|$, so it is of the same magnitude for both solution types. Thus, while both balanced and unbalanced solutions can be robust to input noise, only balanced solutions can also be robust to substantial output noise.

We can now appreciate the basis for the difference in the noise robustness of the two types of solutions. For unbalanced solutions, the difference between the potential induced by typical plus and minus noise-free inputs (the signal) is of the order of $\left|\text{𝒘}\right|=\mathcal{O}(1/\sqrt{N})$ (Fig. 1, Top Left and Bottom Left). Although the fluctuations induced by input noise are of this same order (Fig. 1, Top Left), output noise yields fluctuations in the membrane potential of order $1,$ which is much larger than the magnitude of the weak signal (Fig. 1, Bottom Left). In contrast, for balanced solutions, the signal differentiating plus and minus patterns is of order $\left|\text{𝒘}\right|=\mathcal{O}\left(1\right)$, which is the same order as the fluctuations induced by both types of noise (Fig. 1, Top Right and Bottom Right). Thus, we are led to the important observation that the balanced solution provides the only hope for producing selectivity that is robust against both types of noise. However, there is no guarantee that robust, balanced solutions exist or that they can be found and maintained in a manner that can be implemented by a biological system. Key questions, therefore, are, Under what conditions does a balanced solution to the selectivity task exist? And what are, in detail, its robustness properties? Below, we derive conditions for the existence of a balanced solution, analyze its properties, and study the implications for single-neuron and network computation. We show that, subject to a small reduction of the total information stored in the network, robust and balanced solutions exist and can emerge naturally when learning occurs in the presence of output noise.

Balanced and Unbalanced Solutions.

We begin by presenting the results of an analytic approach (20–22) for determining existence conditions and analyzing properties of weights that generate a specified selectivity, independent of the particular method or learning algorithm used to find the weights (SI Replica Theory for Sign- and Norm-Constrained Perceptron). We validate the theoretical results by using numerical methods that can determine the existence of such weights and find them if they exist (SI Materials and Methods).

When the number of patterns $P$ is too large, solutions may not exist. The maximal value of $P$ that permits solutions is proportional to the number of synapses, $N$, so a useful measure is the ratio $\alpha =P/N$, which we call the load. The capacity, denoted as ${\alpha }_{\mathrm{c}}$, is the maximal load that permits solutions to the task. The capacity depends on the relative number of plus and minus input patterns. For simplicity we assume throughout that the two classes are equal in size (but see SI Capacity for Noneven Split of Plus and Minus Patterns). A classic result for the perceptron with weights that are not sign constrained is that the capacity is ${\alpha }_{\mathrm{c}}=\mathrm{\hspace{0.17em}2}$ (20, 35, 36). For the “constrained perceptron” considered here, we find that ${\alpha }_{\mathrm{c}}$ depends also on the fraction of excitatory afferents, denoted by $f_{\mathrm{exc}}$. This fraction is an important architectural feature of neuronal circuits and varies in different brain systems. For $f_{\mathrm{exc}}=\mathrm{\hspace{0.17em}0}$, namely a purely inhibitory circuit, the capacity vanishes, because when all of the input to the neuron is inhibitory, $V_{\mathrm{PSP}}$ cannot reach threshold and the neuron is quiescent for all stimuli. When the circuit includes excitatory synapses, the task can be solved by appropriate shaping of the strength of the excitatory and inhibitory synapses, and this ability increases the larger the fraction of excitatory synapses is. Therefore, for $f_{\mathrm{exc}}>\mathrm{\hspace{0.17em}0}$, ${\alpha }_{\mathrm{c}}$ increases with $f_{\mathrm{exc}}$ up to a maximum of ${\alpha }_{\mathrm{c}}=\mathrm{\hspace{0.17em}1}$ (half the capacity of an unconstrained perceptron) for fractions equal to or greater than a critical fraction $f_{\mathrm{exc}}=f_{\mathrm{exc}}^{\star }$. This dependence can be summarized by the capacity curve ${\alpha }_{\mathrm{c}}\left(f_{\mathrm{exc}}\right)$ (Fig. 2A, solid line) bounding the range of loads which admit solutions for the different excitatory/inhibitory ratios.

Fig. 2.

Balanced and unbalanced solutions. (A) Perceptron solutions as a function of load and fraction of excitatory weights. Above the capacity line [${\alpha }_{\mathrm{c}}\left(f_{\mathrm{exc}}\right)$, solid line] no solution exists. Balanced solutions exist only below the balanced capacity line [${\alpha }_{\mathrm{b}}\left(f_{\mathrm{exc}}\right)$, dashed shaded line]. Between the balanced capacity and maximum capacity lines, only unbalanced solutions exist (U). On the other hand, below the balanced capacity line, unbalanced solutions coexist with balanced ones (B+U). (B) The norm of the synaptic weight vector of typical solutions as a function the load [in units of $\left(V_{\mathrm{th}}-V_{\mathrm{rest}}\right)/{\sigma }_{\mathrm{exc}}$]. Below ${\alpha }_{\mathrm{b}}$ the norm is clipped at its upper bound $\mathrm{\Gamma }$ (in this case $\mathrm{\Gamma }=\mathrm{\hspace{0.17em}1}$). Above ${\alpha }_{\mathrm{b}}$ the norm collapses and is of order $1/\sqrt{N}$ (shown here for $N=\mathrm{\hspace{0.17em}3}\text{,}000$). (C) The input imbalance index (IB, Eq. 3) of typical solutions as a function of the load. Note the sharp onset of imbalance above ${\alpha }_{\mathrm{b}}$. In B and C $f_{\mathrm{exc}}=\mathrm{\hspace{0.17em}0.8}$, yielding ${\alpha }_{\mathrm{c}}=\mathrm{\hspace{0.17em}1}$. See SI Materials and Methods for other parameters used. For simulation results see Fig. S1.

Interestingly, $f_{\mathrm{exc}}^{\star }$ depends on the statistics of the inputs (SI Replica Theory for Sign- and Norm-Constrained Perceptron). We denote the coefficient of variation (CV) of the excitatory and inhibitory input activities by ${\mathrm{CV}}_{\mathrm{exc}}={\sigma }_{\mathrm{exc}}/{\overline{x}}_{\mathrm{exc}}$ and ${\mathrm{CV}}_{\mathrm{inh}}={\sigma }_{\mathrm{inh}}/{\overline{x}}_{\mathrm{inh}}$, respectively. These measure the degree of stimulus tuning of the two afferent populations. In terms of these quantities, the critical excitatory fraction is

$$f_{\mathrm{exc}}^{\star }=\frac{{\mathrm{CV}}_{\mathrm{exc}}}{{\mathrm{CV}}_{\mathrm{exc}}+{\mathrm{CV}}_{\mathrm{inh}}}.$$ [2]

In other words, the critical ratio between the number of excitatory and inhibitory afferents [$f_{\mathrm{exc}}^{\star }$/(1 − $f_{\mathrm{exc}}^{\star }$)] equals the ratio of their degree of tuning. To understand the origin of this result, we note that to maximize the encoding capacity, the relative strength of the weights should be inversely proportional to the SD of their afferents, ${\overline{w}}_{\mathrm{exc}\left(\mathrm{inh}\right)}\propto \mathrm{\hspace{0.17em}1}/{\sigma }_{\mathrm{exc}\left(\mathrm{inh}\right)}$, implying that the mean total synaptic inputs are proportional to $f_{\mathrm{exc}}{\overline{w}}_{\mathrm{exc}}{\overline{x}}_{\mathrm{exc}}+f_{\mathrm{inh}}{\overline{w}}_{\mathrm{inh}}{\overline{x}}_{\mathrm{inh}}=f_{\mathrm{exc}}/{\mathrm{CV}}_{\mathrm{exc}}-f_{\mathrm{inh}}/{\mathrm{CV}}_{\mathrm{inh}}$, where $f_{\mathrm{inh}}=\mathrm{\hspace{0.17em}1}-f_{\mathrm{exc}}$. For excitatory fraction $f_{\mathrm{exc}}>f_{\mathrm{exc}}^{\star }$ this mean total synaptic inputs are positive, allowing the voltage to reach the threshold and the neuron to implement the required selectivity task with optimally scaled weights. Thus, the capacity of the neuron is unaffected by changes in $f_{\mathrm{exc}}$ in the range $f_{\mathrm{exc}}^{\star }\le f_{\mathrm{exc}}\le \mathrm{\hspace{0.17em}1}$. For excitatory fraction $f_{\mathrm{exc}}<f_{\mathrm{exc}}^{\star }$ the neuron cannot remain responsive (reach threshold) with optimally scaled weights, and thus the capacity is reduced.

In cortical circuits, inhibitory neurons tend to fire at higher firing rates and are thought to be more broadly tuned than excitatory neurons (4, 37, 38), implying $f_{\mathrm{exc}}^{\star }>\mathrm{\hspace{0.17em}0.5}$ (SI Effects of E and I Input Statistics). This is consistent with the abundance of excitatory synapses in cortex. However, input statistics that make $f_{\mathrm{exc}}^{\star }<\mathrm{\hspace{0.17em}0.5}$ do not change the qualitative behavior we discuss (SI Effects of E and I Input Statistics and Fig. S2A).

Fig. S2.

Effects of input statistics. (A) Solution type vs. $f_{\mathrm{exc}}$ and $\alpha$ (as a fraction of ${\alpha }_{\mathrm{c}}^{\mathrm{Unconst}}$) (Capacity for Noneven Split of Plus and Minus Patterns) for different values of $\varphi ={\mathrm{CV}}_{\mathrm{exc}}/{\mathrm{CV}}_{\mathrm{inh}}$. From Left to Right $\varphi =\mathrm{\hspace{0.17em}1}/\sqrt{2}\text{,}\mathrm{\hspace{0.25em}1}\text{,}\sqrt{2}$. Lines are as in Fig. 2A. (B) Type of maximal ${\kappa }_{\mathrm{in}}$ solutions vs. $\varphi$ and ${\kappa }_{\mathrm{in}}^{\max }$ for different values of $\lambda ={\sigma }_{\mathrm{inh}}/{\sigma }_{\mathrm{exc}}$. For a wide range of $\varphi$ and $\lambda$ these solutions are unbalanced for all values of ${\kappa }_{\mathrm{in}}^{\max }$. Here $f_{\mathrm{exc}}=0.8$ and $\lambda =\mathrm{\hspace{0.17em}1}/2\text{,}\mathrm{\hspace{0.25em}1}\text{,}\mathrm{\hspace{0.25em}2}$ from Left to Right. (C) Fraction of silent weights for maximal ${\kappa }_{\mathrm{out}}$ solutions vs. the load for different values of $\lambda$. Fraction of E silent weights is shown in blue and fraction of I silent weights is depicted in red. Here $f_{\mathrm{exc}}=0.8\text{,}\varphi =\sqrt{2}$, $p_{\mathrm{out}}=0.1$, and $\lambda =\mathrm{\hspace{0.17em}1}/2\text{,}\mathrm{\hspace{0.25em}1}\text{,}\mathrm{\hspace{0.25em}2}$ from Left to Right. Notably, for unbalanced, maximal ${\kappa }_{\mathrm{in}}$ solutions the fraction of silent weights is constant and equals 0.5 for both E and I inputs (Saddle-point equations for the maximal $\kappa$in solution and Distribution of synaptic weights).

For load levels below the capacity, many synaptic weight vectors solve the selectivity task and we now describe the properties of the different solutions. In particular, we investigate the parameter regimes where balanced or unbalanced solutions exist. We find that unbalanced solutions with weight vector norms of order $1/\sqrt{N}$ exist for all load values below ${\alpha }_{\mathrm{c}}$. As for the balanced solutions with weight vector norms of order $1$, they exist below a critical value ${\alpha }_{\mathrm{b}}$ which may be smaller than ${\alpha }_{\mathrm{c}}$. Specifically, for $f_{\mathrm{exc}}\le f_{\mathrm{exc}}^{\star }$ balanced solutions exist for all load values below capacity; i.e., ${\alpha }_{\mathrm{b}}={\alpha }_{\mathrm{c}}$. For $f_{\mathrm{exc}}>f_{\mathrm{exc}}^{\star }$, ${\alpha }_{\mathrm{b}}$ is smaller than ${\alpha }_{\mathrm{c}}$ and decreases with $f_{\mathrm{exc}}$ until it vanishes at $f_{\mathrm{exc}}=\mathrm{\hspace{0.17em}1}$ (Fig. 2A, dashed shaded line). The absence of balanced solutions for $f_{\mathrm{exc}}=\mathrm{\hspace{0.17em}1}$ is clear, as there is no inhibition to balance the excitatory inputs. Furthermore, the synaptic excitatory weights must be weak (scaling as $1/N$) to ensure that $V_{\mathrm{PSP}}$ remains close to threshold (slightly above it for plus patterns and slightly below it for minus ones). For $1\ge f_{\mathrm{exc}}>f_{\mathrm{exc}}^{\star }$ the predominance of excitatory afferents precludes a balanced solution if the load is high; i.e., ${\alpha }_{\mathrm{b}}\le \alpha \le {\alpha }_{\mathrm{c}}$. As argued above and shown below, balanced solutions are more robust than unbalanced solutions. Hence, we can identify $f_{\mathrm{exc}}^{\star }$ as the optimal fraction of excitatory input, because it is the fraction of excitatory afferents for which the capacity of balanced solutions is maximal.

For loads below ${\alpha }_{\mathrm{b}}$ both balanced and unbalanced solutions exist, raising the question, What would be the character of a weight vector that is sampled randomly from the space of all possible solutions? Our theory predicts that whenever the balanced solutions exist, the vast majority of the solutions are balanced and furthermore have a weight vector norm that is saturated at the upper bound $\mathrm{\Gamma }$. This is a consequence of the geometry of high-dimensional spaces in which volumes are dominated by the volume elements with the largest radii (see SI Replica Theory for Sign- and Norm-Constrained Perceptron for details). Thus, for $f_{\mathrm{exc}}>f_{\mathrm{exc}}^{\star }$, the typical solution undergoes a transition from balanced to unbalanced weights as $\alpha$ crosses the balanced capacity line ${\alpha }_{\mathrm{b}}\left(f_{\mathrm{exc}}\right)$. At this point the norm of the solution collapses from $\mathrm{\Gamma }$ to $\left|\text{𝒘}\right|\sim \mathrm{\hspace{0.17em}1}/\sqrt{N}$ (Fig. 2B).

As explained above, for balanced solutions we expect to find a near cancellation of the total excitatory (E) and inhibitory (I) inputs. Our theory confirms this expectation. To measure the degree of E-I cancellation for any solution, we introduce the imbalance index,

$$\text{IB}=\frac{{\sum }_i{w}_i{\overline{x}}_i}{{\sum }_{i\in \mathrm{exc}}{w}_i{\overline{x}}_i-{\sum }_{i\in \mathrm{inh}}{w}_i{\overline{x}}_i},$$ [3]

where the overbar symbol denotes an average over all of the input patterns ($\mu$) and, as mentioned above, E weights are nonnegative ($w_i\ge \mathrm{\hspace{0.17em}0}$) and I weights are nonpositive ($w_i\le \mathrm{\hspace{0.17em}0}$). Whereas for the unbalanced solution the IB is of order $1$, for the balanced solution it is small, of order $1/\sqrt{N}$. Thus, the typical solution below ${\alpha }_{\mathrm{b}}$ has zero imbalance (to leading order in $N)$, but the imbalance increases sharply as $\alpha$ increases beyond ${\alpha }_{\mathrm{b}}$ (Fig. 2C).

Noise Robustness of Balanced and Unbalanced Solutions.

To characterize the effect of noise on the different solutions, we introduce two measures, input robustness ${\kappa }_{\mathrm{in}}$ and output robustness ${\kappa }_{\mathrm{out}}$, which characterize the robustness of the noise-free solutions to the addition of two types of noise. To ensure robustness to output noise, the noise-free membrane potential that is the closest to the threshold must be sufficiently far from it. Thus, we define

$${\kappa }_{\mathrm{out}}=\underset{\mu }{\min }\left|\sum _{i=1}^N{w}_i{x}_i^{\mu }-1\right|,$$ [4]

where the minimum is taken over all of the input patterns in the task and the threshold is 1 [because we measure the weights in units of $\left(V_{\mathrm{th}}-V_{\mathrm{rest}}\right)/{\sigma }_{\mathrm{exc}}$]. The second measure, which characterizes robustness to input noise, must take into account the fact that the fluctuations in the membrane potential induced by this form of noise scale with the size of the synaptic weights. Hence, ${\kappa }_{\mathrm{in}}={\kappa }_{\mathrm{out}}/\left|\text{𝒘}\right|$ [${\kappa }_{\mathrm{in}}$ corresponds to the notion of margin in machine learning (39)]. Efficient algorithms for finding the solution with a maximum possible value of ${\kappa }_{\mathrm{in}}$ have been studied extensively (39, 40). We have developed an efficient algorithm for finding solutions with maximal ${\kappa }_{\mathrm{out}}$ (SI Materials and Methods).

We now ask, What are the possible values of the input and output robustness of unbalanced and balanced solutions? Our theory predicts that the majority of both balanced and unbalanced solutions have vanishingly small values of ${\kappa }_{\mathrm{in}}$ and ${\kappa }_{\mathrm{out}}$ and are thus very sensitive to noise. However, for a given load (below capacity) robust solutions do exist, with a spectrum of robustness values up to maximal values, ${\kappa }_{\mathrm{in}}^{\max }>\mathrm{\hspace{0.17em}0}$ and ${\kappa }_{\mathrm{out}}^{\max }>\mathrm{\hspace{0.17em}0}$. Since the magnitude of w scales both signal and noise in the inputs, ${\kappa }_{\mathrm{in}}^{\max }$ is not sensitive to $\left|\text{𝒘}\right|$ and hence is of $\mathcal{O}\left(1\right)$ for both unbalanced and balanced solutions. On the other hand, ${\kappa }_{\mathrm{out}}^{\max }={\kappa }_{\mathrm{in}}^{\max }\left|\text{𝒘}\right|$ is proportional to $\left|\text{𝒘}\right|$. Thus, we expect ${\kappa }_{\mathrm{out}}^{\max }$ to be of $\mathcal{O}(1)$ when balanced solutions exist and of $\mathcal{O}(1/\sqrt{N})$ when only unbalanced solutions exist. In addition, we expect that increasing the load will reduce the value of ${\kappa }_{\mathrm{in}}^{\max }$ and ${\kappa }_{\mathrm{out}}^{\max }$ as the number of constraints that need to be satisfied by the synaptic weights increases.

In Fig. 3 we present the values of ${\kappa }_{\mathrm{in}}^{\max }$ and ${\kappa }_{\mathrm{out}}^{\max }$ vs. the load. As expected, we find that the values of both ${\kappa }_{\mathrm{in}}^{\max }$ and ${\kappa }_{\mathrm{out}}^{\max }$ reach zero as the load approaches the capacity, ${\alpha }_{\mathrm{c}}$ (and diverges, as $N\to \mathrm{\infty }$, for vanishingly small loads). However, ${\kappa }_{\mathrm{out}}^{\max }$ is only substantial (of order 1) and proportional to $\mathrm{\Gamma }$ below ${\alpha }_{\mathrm{b}}$ where balanced solutions exist (Fig. 3 A and B). In contrast, ${\kappa }_{\mathrm{in}}^{\max }$ remains of order 1 up to the full capacity, ${\alpha }_{\mathrm{c}}$ (Fig. 3C). What are the properties of “optimal” solutions that achieve the maximal robustness to either input or output noise? We find that the solutions that achieve the maximal output robustness, ${\kappa }_{\mathrm{out}}^{\max }$, are balanced for all $\alpha \le {\alpha }_{\mathrm{b}}$ and their norm saturates the upper bound, $\mathrm{\Gamma }$ (Fig. S3B). Interestingly, for a wide range of input parameters (SI Replica Theory for Sign- and Norm-Constrained Perceptron, Effects if E and I Input Statistics, and Fig. S2B), solutions that achieve the maximal input robustness, ${\kappa }_{\mathrm{in}}^{\max }$, are unbalanced solutions (Fig. S3C). Nevertheless, we find that below the critical balance load, ${\alpha }_{\mathrm{b}}$, the ${\kappa }_{\mathrm{in}}$ values of the balanced maximal ${\kappa }_{\mathrm{out}}$ solutions are of the same order as, and indeed close to, ${\kappa }_{\mathrm{in}}^{\max }$ (Fig. 3C, dashed shaded line). In fact, the balanced solution with maximal ${\kappa }_{\mathrm{out}}$ also poses the maximal value of ${\kappa }_{\mathrm{in}}$ that is possible for balanced solutions.

Fig. 3.

Maximal values of input and output robustness. (A) Maximal value of ${\kappa }_{\mathrm{out}}$ vs. load [in units of $\mathrm{\Gamma }{\sigma }_{\mathrm{exc}}/\left(V_{\mathrm{th}}-V_{\mathrm{rest}}\right)$]. No solutions exist above the maximal ${\kappa }_{\mathrm{out}}$ line (${\kappa }_{\mathrm{out}}^{\max }$, solid line). Below ${\kappa }_{\mathrm{out}}^{\max }$, for output robustness that is of order 1, only balanced solutions exist. (B) Maximal value of ${\kappa }_{\mathrm{out}}$ for loads between ${\alpha }_{\mathrm{b}}$ and ${\alpha }_{\mathrm{c}}$ (in units of ${\sigma }_{\mathrm{exc}}/{\overline{x}}_{\mathrm{exc}}$). In this range only unbalanced solutions exist and the maximal ${\kappa }_{\mathrm{out}}$ values (solid line) scale as $1/\sqrt{N}$. (C) Maximal value of ${\kappa }_{\mathrm{in}}$ vs. load (in units of ${\sigma }_{\mathrm{exc}}$). No solutions exist above the maximal ${\kappa }_{\mathrm{in}}$ line (${\kappa }_{\mathrm{in}}^{\max }$, solid line). For the parameters used, solutions that achieve ${\kappa }_{\mathrm{in}}^{\max }$ are unbalanced. The maximal value of ${\kappa }_{\mathrm{in}}$ for balanced solutions (dashed shaded line) is not far from the ${\kappa }_{\mathrm{in}}^{\max }$ and is attained by solutions that maximize ${\kappa }_{\mathrm{out}}$ for $\alpha <{\alpha }_{\mathrm{b}}$. In A–C, theory and numerical results are depicted in solid or shaded lines and shaded circles, respectively. Error bars depict SE of the mean. See SI Materials and Methods for parameters used. For further simulation results see Fig. S3.

Fig. S3.

Properties of maximal output and input robustness solutions. (A) Input robustness, ${\kappa }_{\mathrm{in}}$, vs. the load for the maximal ${\kappa }_{\mathrm{in}}$ solution (red) and the maximal ${\kappa }_{\mathrm{out}}$ solution (blue). (B) Norm of synaptic weight vector vs. the load for the maximal ${\kappa }_{\mathrm{out}}$ solution. In the balanced regime ($\alpha <{\alpha }_{\mathrm{b}}$) the norm saturates its upper bound $\mathrm{\Gamma }=1$. Since the norm is constant, maximizing ${\kappa }_{\mathrm{out}}$ in the balanced regime is equivalent to maximizing ${\kappa }_{\mathrm{in}}$ under the constraint $\left|\text{𝒘}\right|=\mathrm{\Gamma }$. (C) Rescaled norm of synaptic weight vector ($\sqrt{N}\left|\text{𝒘}\right|$) vs. the load for the maximal ${\kappa }_{\mathrm{in}}$ solution. To demonstrate the $1/\sqrt{N}$ scaling of the weight vector norm, colors depict results for $N=750$ (gray), $N=1\text{,}500$ (green), and $N=3\text{,}000$ (red). In A–C lines depict theoretical prediction. $f_{\mathrm{exc}}=0.8$, $p_{\mathrm{out}}=0.1$, $\varphi ={\mathrm{CV}}_{\mathrm{exc}}/{\mathrm{CV}}_{\mathrm{inh}}=\sqrt{2},$ and $\lambda ={\sigma }_{\mathrm{inh}}/{\sigma }_{\mathrm{exc}}=2$; results are averaged over 100 samples.

We conclude that solutions that are robust to both input and output noise exist for loads less than ${\alpha }_{\mathrm{b}}$ which for $f_{\mathrm{exc}}>f_{\mathrm{exc}}^{\star }$ is smaller than ${\alpha }_{\mathrm{c}}$. However, as long as $f_{\mathrm{exc}}$ is close to $f_{\mathrm{exc}}^{\star }$, the reduction in capacity from ${\alpha }_{\mathrm{c}}$ to ${\alpha }_{\mathrm{b}}$ imposed by the requirement of robustness is small.

Balanced and Unbalanced Solutions for Spiking Neurons.

Neurons typically receive their input and communicate their output through action potentials. Thus, a fundamental question is, How will the introduction of spike-based input and spiking output affect our results? Here we show that the main properties of balanced and unbalanced synaptic efficacies, as discussed above, remain when the inputs are spike trains and the model neuron implements spiking and membrane potential reset mechanisms.

We consider a leaky integrate-and-fire (LIF) neuron that is required to perform the same binary classification task we considered using the perceptron. Each input is characterized by a vector of firing rates, ${\text{𝒙}}^{\mu }$. Each afferent generates a Poisson spike train over an interval from time $t=\mathrm{\hspace{0.17em}0}$ to $t=T$, with mean rate $r_i\propto x_i^{\mu }$. The LIF neuron integrates these input spikes (SI Materials and Methods) and emits an output spike whenever its membrane potential crosses a firing threshold. After each output spike, the membrane potential is reset to the resting potential, and the integration of inputs continues. We define the output state of the LIF neuron, using the total number of output spikes $n_{\mathrm{spikes}}$: The neuron is quiescent if $n_{\mathrm{spikes}}\le n_{\mathrm{thr}}$ and active if $n_{\mathrm{spikes}}>n_{\mathrm{thr}}$, where $n_{\mathrm{thr}}$ is chosen to maximize classification performance. We do not discuss the properties of learning in LIF neurons (41–45), but instead test the properties of the solutions (weights) obtained from the perceptron model when they are used for the LIF neuron. In particular, we compare the performance of the balanced, maximal ${\kappa }_{\mathrm{out}}$ solution and the unbalanced, maximal ${\kappa }_{\mathrm{in}}$ solution. When the synaptic weights of the LIF neuron are set according to the two perceptron solutions, the mean output of the LIF neuron correctly classifies the input patterns (according to the desired classification; Fig. S4). Consistent with the results for the perceptron, we find that with no output noise the performance of both solutions is good, even in the presence of the substantial input noise caused by Poisson fluctuations in the number of input spikes and their timings (Fig. 4 A–C). When the output noise magnitude is increased (SI Materials and Methods), however, the performance of the unbalanced maximal ${\kappa }_{\mathrm{in}}$ solution quickly deteriorates, whereas the performance of the balanced maximal ${\kappa }_{\mathrm{out}}$ solution remains largely unaffected (Fig. 4 D–F). Thus, the spiking model recapitulates the general results found for the perceptron.

Fig. S4.

Neuronal selectivity for a spiking neuron. Both panels depict the histograms of the mean output spike count for patterns belonging to the plus (blue) and minus (red) classes of an LIF neuron with balanced weights maximizing ${\kappa }_{\mathrm{out}}$ (Left) and unbalanced weights maximizing ${\kappa }_{\mathrm{in}}$ (Right). Here the magnitude of the output noise is zero. In both cases the mean output spike count can be used to correctly classify the patterns. For parameters used see Fig. 3.

Fig. 4.

Selectivity in a spiking model. A and B (D and E) depict the output of an LIF neuron with no (high) output noise for the balanced maximal ${\kappa }_{\mathrm{out}}$ solution (A and D) and the unbalanced maximal ${\kappa }_{\mathrm{in}}$ solution (B and E). C and F depict the receiver operating characteristic (ROC) curves for the two solutions under the no output noise (C) and high output noise (F) conditions obtained as the decision threshold ($n_{\mathrm{thr}}$) is modified from 0 to $\mathrm{\infty }$. Consistent with the results of the perceptron, the performances of the two solutions with no output noise are very similar with a slight advantage for the maximal ${\kappa }_{\mathrm{in}}$ solution. With higher levels of output noise, the performance of the unbalanced maximal ${\kappa }_{\mathrm{in}}$ solution quickly deteriorates, whereas the performance of the balanced maximal ${\kappa }_{\mathrm{out}}$ solution is only slightly affected. $\left|\text{𝒘}\right|$ of the balanced solution was chosen to equalize the mean output spike count across all patterns in both solutions (mean $n_{\mathrm{spike}}\sim \mathrm{\hspace{0.17em}4}$). See SI Materials and Methods for parameters used.

Balanced and Unbalanced Synaptic Weights in Associative Memory Networks.

Thus far, we have considered the selectivity of a single neuron, but our results also have important implications for recurrently connected neuronal networks, in particular recurrent networks implementing associative memory functions. Models of associative memory in which stable fixed points of the network dynamics represent memories, and memory retrieval corresponds to the dynamic transformation of an initial state to one of the memory-representing fixed points, have been a major focus of memory research for many years (24, 27, 28, 46–48). For the network to function as an associative memory, memory states must have large basins of attraction so that the network can perform pattern completion, recalling a memory from an initial state that is similar but not identical to it. In addition, memory retrieval must be robust to output noise. As we will show, the variables ${\kappa }_{\mathrm{in}}$ and ${\kappa }_{\mathrm{out}}$ for the synaptic weights projecting onto individual neurons in the network are closely related to the sizes of the basins of attraction of the memories and the robustness to output noise, respectively.

We consider a network that consists of $N{f}_{\mathrm{exc}}$ E and $N\left(1-f_{\mathrm{exc}}\right)$ I, recurrently connected binary neurons. The network operates in discrete time steps and at each step the state of one randomly chosen neuron, $i$, is updated according to

$$s_i\left(t+1\right)=\mathrm{\Theta }\left[\sum _{j\ne i}{J}_{ij}{s}_j\left(t\right)+{\eta }_{\mathrm{out}}\left(t\right)-1\right].$$ [5]

Here $\mathrm{\Theta }(x)=\mathrm{\hspace{0.17em}1}$ for $x\ge \mathrm{\hspace{0.17em}0}$ and 0 otherwise, $J_{ij}$ is the weight of the synapse from neuron $j$ to neuron $i$, and ${\eta }_{\mathrm{out}}(t)$, the output noise, is a Gaussian random variable with SD ${\sigma }_{\mathrm{out}}$. $P$ randomly chosen binary activity patterns $\left\{{𝐬}^{\mu }\right\},\mu =\mathrm{\hspace{0.17em}1},2,\mathrm{\dots },P$ (where each $s_i^{\mu }=\left\{\mathrm{0,1}\right\}$) representing the stored memories are encoded in the recurrent synaptic matrix $J$. This is achieved by treating each neuron, say $i,$ as a perceptron with a weight vector ${\text{𝒘}}^i={\{J_{ij}\}}_{j\ne i}$ that maps its inputs $\{s_j^{\mu }$} from all other neurons to its desired output $s_i^{\mu }$ for each memory state (Fig. 5 A and B and SI Materials and Methods). This creates an attractor network in which the memory states are fixed points of the dynamics in the noise-free condition (${\sigma }_{\mathrm{out}}=\mathrm{\hspace{0.17em}0}$) (20).

Fig. 5.

Recurrent associative memory network constructed using single-neuron feedforward learning. (A) A fully connected recurrent network of E and I neurons in a particular memory state. Active (quiescent) neurons are shown in black (white). E and I synaptic connections ($J_{ij}$) are shown in yellow and blue, respectively (not all connections are depicted). Lines symbolize axons, and synapses are shown as small circles. (B) To find an appropriate $J_{ij}$, the postsynaptic weights of each neuron are set using the memory-state activities of the other neurons as input and its own memory state as the desired output. In this example, neuron 4 will implement its desire memory state through modification of the weights $J_{4j}$ for $j=1\text{,}2\text{,}3\text{,}5\text{,}6\text{,}7$. C and E show the fraction of erroneous (different from a given memory pattern) neurons in the network as a function of time. (C) Network dynamics with ${\sigma }_{\mathrm{out}}=\mathrm{\hspace{0.17em}0}$. An initial state of the network can either converge to the memory state (blue) or diverge to other network states (red). (D) Probability of converging to a memory state vs. initial pattern distortion (SI Materials and Methods) for a network with unbalanced maximal ${\kappa }_{\mathrm{in}}$ weights (green), a network with balanced maximal ${\kappa }_{\mathrm{out}}$ weights (black), and a network with balanced maximal ${\kappa }_{\mathrm{out}}$ weights with unlearned inhibition (gray, main text). (E) Network dynamics with ${\sigma }_{\mathrm{out}}>\mathrm{\hspace{0.17em}0}$. The network is initialized at the memory state. The dynamics can be stable (blue; the network remains close to the memory state), or unstable (red; the network diverges to another state). (F) Probability of stable dynamics for at least $500N$ time steps for networks initialized at the memory state in the presence of output noise vs. ${\sigma }_{\mathrm{out}}$. Colors are the same as in D. (G) Maximal output noise magnitude vs. load for networks with balanced synaptic weights matrix maximizing ${\kappa }_{\mathrm{out}}$. Similar to ${\kappa }_{\mathrm{out}}$, the maximal output noise magnitude is of order 1 only below ${\alpha }_{\mathrm{b}}$. Above it, even though solutions exist they are extremely sensitive to output noise. Results are shown for $f_{\mathrm{exc}}=\mathrm{\hspace{0.17em}0.8}$ (green) and $f_{\mathrm{exc}}=\mathrm{\hspace{0.17em}0.9}$ (magenta). See SI Materials and Methods for parameters used.

We do not attempt to perform a complete analysis of the effects of input and output noise in recurrent networks, a difficult challenge. Instead, we link observations from our single-neuron analysis to key features of a recurrent network performing a memory function. The capacity of such a memory network is defined as the maximal load for which the memory patterns can be fixed points of the noise-free dynamics, stable against single-neuron perturbations. This condition is met as long as the single-neuron synaptic weights possess substantial ${\kappa }_{\mathrm{in}}$ (i.e., ${\kappa }_{\mathrm{in}}\sim \mathcal{O}\left(1\right)$) for all neurons. Thus, the single-neuron capacities will determine the overall network capacity. As we showed before, the capacity of a single-neuron perceptron depends on the statistics of its desired output (which in our case is the sparsity of activity across memory states). Since this statistic may be different in E and I populations, the single-neuron capacity of the two populations may vary, and hence the global capacity of the recurrent network is the minimum of the single-neuron capacities of the two neuron types. As long as $P$ is smaller than this critical capacity, a recurrent weight matrix exists for which all $P$ memory states are stable fixed points of the noiseless dynamics. However, such solutions are not unique, and the choice of a particular matrix can endow the network with different robustness properties. As stated above, to properly function as an associative memory the fixed points must have large basins of attraction. Corruption of the initial state away from the parent memory pattern introduces variability into the inputs of each neuron for subsequent dynamic iterations and hence is equivalent to injecting input noise in the single-neuron feedforward case. The network propagates this initial input noise in a nontrivial way; however, its magnitude always remains proportional to the magnitude of the norm of the neurons’ synaptic weights. We therefore expect that a large basin of attraction is achieved when the matrix $J$ yields a large input noise robustness for each neuron in the (noise-free) fixed points (49, 50). When output noise is introduced to the network dynamics (${\sigma }_{\mathrm{out}}>\mathrm{\hspace{0.17em}0}$), the network may propagate it as input noise to other neurons in subsequent time steps. However, initially its magnitude is proportional only to ${\sigma }_{\mathrm{out}}$ and is unaffected by the scale of the synaptic weights. Thus, we expect that the requirement that the memory states and retrieval will be robust against output noise is satisfied when $J$ yields a large output noise robustness for each neuron in the (noise-free) fixed points. We therefore consider two types of recurrent connections: one in which each row of $J$ is a weight vector that maximizes ${\kappa }_{\mathrm{in}}$ and hence, in the chosen parameter regime, is necessarily unbalanced and a second one in which the rows of the connection matrix correspond to balanced solutions that maximize ${\kappa }_{\mathrm{out}}$.

We estimate the basins of attraction of the memory patterns numerically by initializing the network in states that are corrupted versions of the memory states (SI Materials and Methods) and observing whether the network, with ${\sigma }_{\mathrm{out}}=0$, converges to the parent memory state (Fig. 5C, blue) or diverges away from it (Fig. 5C, red). We define the size of the basin of attraction as the maximum distortion in the initial state that ensures convergence to the parent memory with high probability.

Comparing the basins of attraction of the two types of networks, we find that the mean basin of attraction of the unbalanced network is moderately larger than that of the balanced one (Fig. 5D), consistent with the slightly lower value of ${\kappa }_{\mathrm{in}}$ in the balanced case (Fig. 5D). On the other hand, the behavior of the two networks is strikingly different in the presence of output noise. To illustrate this, we start each network at a memory state and determine whether it is stable (remains in the vicinity of this state for an extended period), despite the noise in the dynamics (Fig. 5E). We estimate the output noise tolerance of the network by measuring the maximal value of ${\sigma }_{\mathrm{out}}$ for which the memory states are stable (Fig. 5F). We find that memory states in the balanced solution with maximal ${\kappa }_{\mathrm{out}}$ are stable for noise levels that (for the network sizes used in the simulation) are an order of magnitude larger than for the unbalanced network with maximal ${\kappa }_{\mathrm{in}}$ (Fig. 5F).

Finally, we ask how the noise robustness of the memory states in the balanced network depends on the number of memories. As shown in Fig. 5F, for a fixed level of load below capacity, memory patterns are stable ($P_{\mathrm{stable}}>\mathrm{\hspace{0.17em}0.5}$) as long as levels of noise remain below a threshold value, which we denote as ${\sigma }_{\mathrm{out}}^{\max }\left(\alpha \right)$. When ${\sigma }_{\mathrm{out}}$ increases beyond ${\sigma }_{\mathrm{out}}^{\max }\left(\alpha \right)$, stability of the memory states rapidly deteriorates. The critical noise function ${\sigma }_{\mathrm{out}}^{\max }\left(\alpha \right)$ decreases smoothly from a large value at small $\alpha$ to zero at a level of load, ${\alpha }_{\mathrm{b}}$. This load coincides with the maximal load for which both E and I neurons have balanced solutions (Fig. 5G). For loads ${\alpha }_{\mathrm{b}}<\alpha <{\alpha }_{\mathrm{c}}$, all solutions are unbalanced, and hence the magnitude of the stochastic dynamical component can be at most of order $1/\sqrt{N}$.

The Role of Inhibition in Associative Memory Networks.

In our associative memory network model, we assumed that both E and I neurons code desired memory states and that all network connections are modified by learning. Most previous models of associative memory that separate excitation and inhibition assume that memory patterns are restricted to the E population, whereas inhibition provides stabilizing inputs (14, 48, 51–54). To address the emergence of balanced solution in scenarios where the I neurons do not represent long-term memories, we studied an architecture where I to E, I to I, and E to I connections are random sparse matrices with large amplitudes, resulting in I activity patterns driven by the E memory states. In such conditions, the I subnetwork exhibits irregular asynchronous activity with an overall mean activity that is proportional to the mean activity of the driving E population (7, 55, 56). Although the mean I feedback provided to the E neurons can balance the mean excitation, the variability in this feedback injects substantial noise onto the E neurons, which degrades system performance (SI Recurrent Networks with Nonlearned Inhibition). This variability stems from the differences in I activity patterns generated by the different E memory states (albeit with the same mean). Additional noise is caused by the temporal irregular activity of the chaotic I dynamics. Next we ask whether the system’s performance can be improved through plasticity in the I to E connections for which some experimental evidence exists (23, 57–60). Indeed, we find an appropriate plasticity rule for this pathway (SI Recurrent Networks with Nonlearned Inhibition) that suppresses the spatiotemporal fluctuations in the I feedback, yielding a balanced state that behaves similarly to the fully learned networks described above (Fig. 5 D and F, gray lines). Interestingly, in this case the basins of attraction of the balanced network are comparable to or even larger than the basins of the unbalanced fully learned network (compare gray to green curves in Fig. 5D). Despite the fact that no explicit memory patterns are assigned to the the I populations, the I activity plays a computational role that goes beyond providing global I feedback; when the weights of the I to E connections are shuffled, the network’s performance significantly degrades (Fig. S5).

Fig. S5.

Effect of shuffling learned I weights in recurrent networks with nonlearned I activity. Shaded line depicts the performance of the network with random E to I and I to I connections and learned E to E and I to E connections (Recurrent Networks with Nonlearned Inhibition) (same as gray line in Fig. 5D). Solid line depicts the performance of the same network with the I weights of each E neuron randomly shuffled. Thus, the distribution of I synaptic weights for each E neuron is identical in both cases. This result shows that the learned I weights are important for network performance and stability.

Learning Robust Solutions.

Thus far, we have presented analytical and numerical investigations of solutions that support selectivity or associative memory and provide substantial robustness to noise. However, we did not address the way in which these robust solutions could be learned by a biological system. In fact, as stated above, the majority of solutions for these tasks have vanishingly small output and input robustness and the above maximum robustness solutions are found numerically by special learning algorithms. Therefore, an important question is whether noise robust weights can emerge naturally from synaptic learning rules that are appropriate for neuronal circuits.

The actual algorithms used for learning in the neural circuits are generally unknown, especially within a supervised learning scenario. Experiments suggest that learning rules may depend on brain area and both pre- and postsynaptic neuron types (for example, refs. 57–59, 61; for reviews see refs. 60, 62–64). From a theoretical perspective, the properties of the solutions found through learning, and in particular their noise robustness, depend on both the type and parameters of the algorithm and the properties of the space of possible solutions. However, our theory suggests that a general, simple way to ensure that learning arrives at a robust solution is to introduce noise during learning. Indeed, this is a common practice in machine learning for increasing generalization abilities [a specific form of data augmentation (65, 66)]. The rationale is that learning algorithms that achieve low error in the presence of noise necessarily lead to solutions that are robust against noise levels at least as large as those present during learning. In the case we are considering, learning in the presence of substantial input noise should lead to solutions that have substantial ${\kappa }_{\mathrm{in}}$ and introducing output noise during learning should lead to solutions with substantial ${\kappa }_{\mathrm{out}}$. We note that ${\kappa }_{\mathrm{in}}$ may be large even if ${\kappa }_{\mathrm{out}}$ remains small (for example, in unbalanced solutions with maximal ${\kappa }_{\mathrm{in}}$) but not vice versa [because ${\kappa }_{\mathrm{out}}$ of order 1 implies $\left|\text{𝒘}\right|$ (and as a result ${\kappa }_{\mathrm{in}}$) of order 1 as well]. Therefore, learning in the presence of significant output noise should lead to solutions that are robust to both input and output noise, whereas learning in the presence of input noise alone may lead to unbalanced solutions that are sensitive to output noise, depending on the details of the learning algorithm. We therefore predict that performing successful learning in the presence of output noise is a sufficient condition for the emergence of excitation–inhibition balance.

To demonstrate that robust balanced solutions emerge in the presence of output noise, we consider a variant of the perceptron learning algorithm (18) in which we have forced the sign constraints on the weights (29) and, in addition, added a weight decay term implementing a soft constraint on the magnitude of the weights (SI Materials and Methods). This supervised learning rule possesses several important properties that are required for biological plausibility: It is online, and weights are modified incrementally after each pattern presentation; it is history independent so that each weight update depends only on the current pattern and error signal; and finally, it is simple and local, and weight updates are a function of the error signal and quantities that are available locally at the synapse (presynaptic activity and synaptic efficacy). When this learning rule is applied to train a selectivity task in the presence of substantial output noise, the resulting solution has a balanced weight vector with substantial ${\kappa }_{\mathrm{out}}$ and ${\kappa }_{\mathrm{in}}$ (Fig. 6, shaded lines). In contrast, if learning occurs with weak output noise, the algorithm’s tendency to reduce the magnitude of the weights causes the resulting solution to be unbalanced with small ${\kappa }_{\mathrm{out}}$, while its ${\kappa }_{\mathrm{in}}$ may be large if substantial input noise is present during learning (Fig. 6, solid lines). When this learning rule is applied in the load regime where only unbalanced solutions exist (${\alpha }_{\mathrm{b}}<\alpha <{\alpha }_{\mathrm{c}}$), learning fails to achieve reasonable performance when applied in the presence of large output noise. When noise is scaled down to the value allowed by ${\kappa }_{\mathrm{out}}^{\max }\propto \mathrm{\hspace{0.17em}1}/\sqrt{N}$, learning yields unbalanced solutions with robustness values of the order of the maximum allowed in this region (Fig. S6).

Fig. 6.

Emergence of E-I balance from learning in the presence of output noise. All panels show the outcome of perceptron learning for a noisy neuron (SI Materials and Methods) under low (${\sigma }_{\mathrm{out}}=\mathrm{\hspace{0.17em}0.01}$, solid lines) and high (${\sigma }_{\mathrm{out}}=\mathrm{\hspace{0.17em}0.1}$, shaded lines) output noise conditions. Except for ${\sigma }_{\mathrm{out}}$, all model and learning parameters are identical for the two conditions (including ${\sigma }_{\mathrm{in}}=\mathrm{\hspace{0.17em}0.1}$). (A) Mean training error vs. learning cycle. On each cycle, all of the input patterns to be learned are presented once. The error decays and plateaus at its minimal value under both low and high output noise conditions. (B) Mean IB (Eq. 3) vs. learning cycle. IB remains of order 1 under low output noise conditions and drops close to zero under high output noise conditions. (C) Mean input robustness (${\kappa }_{\mathrm{in}}$) vs. learning cycle. Input robustness is high under both output noise conditions. (D) Mean output robustness (${\kappa }_{\mathrm{out}})$ vs. learning cycle. Output robustness is substantial only under the high output noise learning condition. These results demonstrate that robust balanced solutions naturally emerge under learning in the presence of high output noise. See SI Materials and Methods for other parameters used.

Fig. S6.

Perceptron learning with input and output noise for ${\alpha }_{\mathrm{b}}<\alpha <{\alpha }_{\mathrm{c}}$. A–D depict the outcome of simple perceptron learning for a noisy neuron (Materials and Methods) under low output noise conditions (${\sigma }_{\mathrm{out}}=0.01/\sqrt{N}$, solid lines) and high output noise conditions (${\sigma }_{\mathrm{out}}=0.01$, shaded lines). Except for ${\sigma }_{\mathrm{out}}$ all model and learning parameters are identical for the two conditions (including ${\sigma }_{\mathrm{in}}=0.01$). (A) Mean training error vs. learning cycle. At each cycle all of the learned input patterns are presented once. (B) Mean imbalance index vs. learning cycle. IB remains of order 1 under low output noise conditions and drops to lower values under high output noise conditions. (C) Mean input robustness (${\kappa }_{\mathrm{in}}$) vs. learning cycle. (D) Mean rescaled output robustness ($\sqrt{N}{\kappa }_{\mathrm{out}}$) vs. learning cycle. The error decays and plateaus at its minimal value under both low and high output noise conditions; however, for high output noise the error remains substantial. Both output and input robustness are negative under the high output noise conditions. (The learning does not find a weights vector that performs the classification of the noise-free patterns correctly.) Input and output robustness are positive when the output noise scales at most as $1/\sqrt{N}$. Random patterns are binary pattern $x_i^{\mu }\in \left\{0\text{,}\mathrm{\hspace{0.17em}1}\right\}$ with equal probabilities and an even split of plus and minus patterns. $N=3\text{,}000$, $P=2\text{,}400$. Learning algorithm parameters: $\epsilon ={\mathrm{\hspace{0.17em}10}}^{-8}$, $\rho =0.02/N$, ${\sigma }_{\mathrm{in}}=0.01$. Results are averaged over 50 samples.

SI Materials and Methods

Finding Perceptron Solutions.

There are a number of numerical methods for choosing a weight vector w that generates a specified selectivity (25, 27, 29, 39). For numerical simulations we developed algorithms that find the maximal ${\kappa }_{\mathrm{out}}$ and maximal ${\kappa }_{\mathrm{in}}$ solutions that obey the imposed biological constraints. These solutions can be found directly by solving conic programming optimization problems for which efficient algorithms exist and are widely available (73). For details see Finding Maximal $\kappa$in and $\kappa$out Solutions.

Random Patterns in Numerical Estimation of ${𝜿}_{\text{out}}^{\text{max}}$ and ${𝜿}_{\text{in}}^{\text{max}}$ Solutions.

In numerical experiments for Figs. 3 and 4 and Figs. S1 and S3, E inputs for the random patterns were drawn i.i.d. from an exponential distribution with unity mean and SD. I inputs were drawn from a Gamma distribution with shape parameter $k$ and scale parameter $\theta$ [the probability density function of the Gamma distribution is given by $P\left(x\right)=\frac{1}{\mathrm{\Gamma }\left(k\right)\theta }{\left(\frac{x}{\theta }\right)}^{k-1}{e}^{-\frac{x}{\theta }}$, where $\mathrm{\Gamma }\left(k\right)$ is the Gamma function].

Fig. S1.

Numerical measurement of capacity and balanced capacity. (A) Capacity of sign-constrained weights perceptron, ${\alpha }_{\mathrm{c}}$, vs. the fraction of excitatory inputs, $f_{\mathrm{exc}}$, as a fraction of the capacity of an unconstrained perceptron (Capacity for Noneven Split of Plus and Minus Patterns). Theory is depicted in black. Simulation results are shown in blue for $p_{\mathrm{out}}=0.5$ and red for $p_{\mathrm{out}}=0.1$. To measure ${\alpha }_{\mathrm{c}}$ we measure the probability of the existence of a solution as a function of $\alpha$. We estimate ${\alpha }_{\mathrm{c}}$ by the load at which this probability is 1/2. (B) Capacity of balanced solutions, ${\alpha }_{\mathrm{b}}$, as a fraction of ${\alpha }_{\mathrm{c}}$ vs. $f_{\mathrm{exc}}>f_{\mathrm{exc}}^{\star }$. Since ${\kappa }_{\mathrm{out}}^{\max }$ solutions are balanced whenever balanced solutions exist, to measure ${\alpha }_{\mathrm{b}}$ we measure the probability of finding a balanced ${\kappa }_{\mathrm{out}}^{\max }$ solution, i.e., a solution that saturates the upper bound on $\left|\text{𝒘}\right|$. We estimate ${\alpha }_{\mathrm{b}}$ by the load at which this probability is 1/2. In both A and B, $N=3\text{,}000$, ${\mathrm{CV}}_{\mathrm{exc}}/{\mathrm{CV}}_{\mathrm{inh}}=\sqrt{2}$.

Dynamics of LIF Neuron.

Input spike trains.

For each input pattern ${\text{𝒙}}^{\mu }$ input spike trains of input afferent $i=1,2,\mathrm{\dots },N$ were drawn randomly from a Poisson process with rate $r_i=A{x}_i^{\mu },$ for duration $T$.

Synaptic input.

Given the set of input spike trains $\left\{t_i\right\},$ $i=\mathrm{1,2},\mathrm{\dots },N$ the contribution of synaptic input to the membrane potential is given by $V_{\mathrm{syn}}\left(t\right)={\sum }_i{w}_i{\sum }_{t_i}K\left(t-t_i\right)$, where $w_i$ is the synaptic efficacy of the synapse from the $i$th input afferent and $K\left(t\right)$ is a postsynaptic potential kernel. $K\left(t\right)=0$ for $t<0$ and is given by $K\left(t\right)=V_0\left(e^{-\frac{t}{{\tau }_{\mathrm{m}}}}-e^{-\frac{t}{{\tau }_{\mathrm{s}}}}\right)$ for $t>0$, where ${\tau }_{\mathrm{m}}$ and ${\tau }_{\mathrm{s}}$ are the membrane and synaptic time constants, respectively, and $V_0$ is such that the maximal value of $K\left(t\right)$ is one.

Output noise.

Output noise was added to the neuron’s membrane potential as random synaptic input $V_{\mathrm{o}.\mathrm{n}.}\left(t\right)={\sum }_{j=1}^{N_{\mathrm{noise}}}{g}_{j}K\left(t-t_j\right)$, where $g_j$ was randomly drawn from a zero mean Gaussian distribution with SD ${\sigma }_{\mathrm{n}}$ and $t_j\in (0,T)$ was randomly drawn from a uniform distribution.

Voltage reset.

After each threshold crossing the membrane potential was reset to its resting potential. Given the set of output spike times $\left\{t_{\mathrm{spike}}\right\}$, the total contribution of voltage reset to the membrane potential can be written as $V_{\mathrm{reset}}\left(t\right)=-\left(V_{\mathrm{th}}-V_{\mathrm{rest}}\right){\sum }_{t_{\mathrm{spike}}}R\left(t-t_{\mathrm{spike}}\right)$, where $V_{\mathrm{rest}}$ and $V_{\mathrm{th}}$ are the neuron’s resting and threshold potentials, respectively, and $R\left(t\right)$ implements the postspike voltage reset. $R\left(t\right)=0$ for $t<0$ and is given by $R\left(t\right)=e^{-\frac{t}{{\tau }_{\mathrm{m}}}}$ for $t\ge 0$. This form ensures the voltage is reset to the resting potential immediately after an output spike.

Membrane potential.

Finally, the neuron’s membrane potential is given by $V\left(t\right)=V_{\mathrm{rest}}+V_{\mathrm{syn}}\left(t\right)+V_{\mathrm{o}.\mathrm{n}.}\left(t\right)+V_{\mathrm{reset}}\left(t\right)$, where $V_{\mathrm{reset}}$ is computed given $V_{\mathrm{syn}}$ and $V_{\mathrm{o}.\mathrm{n}.}$.

Simulations of Recurrent Networks.

Memory states.

Networks were trained to implement a set of $P$ memory states, specified by $x_i^{\mu }\in \left\{\mathrm{0,1}\right\},$ $i=1,2,\mathrm{\dots },N$, $\mu =1,2,\mathrm{\dots },P$, as stable fixed points of the noise-free dynamics. Memory states were randomly chosen i.i.d. from binary distributions with parameter $p_{\mathrm{exc}/\mathrm{inh}}$ according to the type of the $i$th input afferent; i.e., $\mathrm{Pr}(x_i^{\mu }=1)=p_{\mathrm{exc}/\mathrm{inh}}$ and $\mathrm{Pr}(x_i^{\mu }=0)=1-p_{\mathrm{exc}/\mathrm{inh}}$.

Initial pattern distortion.

To start the network close to a memory state ${\text{𝒙}}^{\mu }$, the initial state of the network, $s_i(t=0)$ for $i=1,2,\mathrm{\dots },N$, was randomly chosen according to $\mathrm{Pr}(s_i=1)=(1-\delta )\mathrm{\Theta }(2x_i^{\mu }-1)+\delta \frac{p_{\mathrm{exc}/\mathrm{inh}}}{1-p_{\mathrm{exc}/\mathrm{inh}}}\mathrm{\Theta }(-2x_i^{\mu }+1)$, where $\delta$ is the initial pattern distortion level (Fig. 5B) and $\mathrm{\Theta }\left(x\right)=1$ for $x\ge 0$ and 0 otherwise. This procedure ensures that the mean activity levels of E and I neurons in the initial state are the same as their mean activity levels in the memory state (74).

Perceptron Learning Algorithm.

The perceptron learning algorithm (Fig. 6 and Fig. S6) learns to classify a set of $P$ labeled patterns. At learning time-step $t$ one pattern ${\text{𝒙}}_t$ with desired output $y_t=\pm 1$ is presented to the neuron. The output of the perceptron $s_t$ is given by $s_t=\mathrm{sign}({\text{𝒘}}_t^{\mathrm{T}}{\text{𝒙}}_t+{\eta }_t-1),$ where ${\eta }_t$ is a Gaussian random variable with zero mean and variance ${\left|{\text{𝒘}}_t\right|}^2{\sigma }_{\mathrm{in}}^2+{\sigma }_{\mathrm{out}}^2$. The error signal is defined as $e_t=y_t\mathrm{\Theta }\left(-s_t{y}_t\right)$, where $\mathrm{\Theta }\left(x\right)=1$ for $x>0$ and zero otherwise. After each pattern presentation all synapses are updated. The synaptic weights of E inputs are updated according to $w_{i,t+1}={\left[\left(1-\epsilon \right)w_{i,t}+\rho e_t{x}_{i,t}\right]}_{+}$ and weights of I inputs are updated according to $w_{i,t+1}={\left[\left(1-\epsilon \right)w_{i,t}+\rho e_t{x}_{i,t}\right]}_{-}$, where ${\left[x\right]}_{\pm }=x\mathrm{\Theta }\left(\pm x\right)$, $\epsilon$ is a weight decay constant, and $\rho$ is a constant learning rate. At each learning cycle ($P$ learning time steps) all patterns are presented sequentially in a random order (randomized at each learning cycle).

Figure Parameters.

Fig. 2.

In all panels ${\sigma }_{\mathrm{inh}}/{\sigma }_{\mathrm{exc}}=2$ and ${\mathrm{CV}}_{\mathrm{exc}}/{\mathrm{CV}}_{\mathrm{inh}}=\sqrt{2}$. ${\sigma }_{\mathrm{inh}}/{\sigma }_{\mathrm{exc}}=2,{\mathrm{CV}}_{\mathrm{exc}}/{\mathrm{CV}}_{\mathrm{inh}}=\sqrt{2}$ with an even split between responsive/unresponsive labels. In B and C $f_{\mathrm{exc}}=0.8$.

Fig. 3.

In all panels $N=3,000$, $k=2$, and $\theta =\sqrt{2}$ (Random Patterns in Numerical Estimation of ${\kappa }_{\mathrm{out}}^{\max }$ and ${\kappa }_{\mathrm{in}}^{\max }$ Solutions), leading to ${\sigma }_{\mathrm{inh}}/{\sigma }_{\mathrm{exc}}=\mathrm{\hspace{0.17em}2}{\mathrm{CV}}_{\mathrm{exc}}/{\mathrm{CV}}_{\mathrm{inh}}=\sqrt{2}$, $f_{\mathrm{exc}}=0.8$ with an even split between responsive/unresponsive labels. Numerical results are averaged over 100 samples.

Fig. 4.

In all panels $N=1,000$, $P=1,000$, fraction of plus patterns $p_{\mathrm{out}}=0.1,$ $f_{\mathrm{exc}}=0.8$, $V_{\mathrm{rest}}=0$, $V_{\mathrm{thr}}=1$, ${\tau }_{\mathrm{m}}=\mathrm{\hspace{0.17em}30}\mathrm{ms},$ ${\tau }_{\mathrm{s}}=\mathrm{\hspace{0.17em}10}\mathrm{ms}$, $T=\mathrm{\hspace{0.17em}200}\mathrm{ms},$ $A=\mathrm{\hspace{0.17em}30}\mathrm{Hz}$ (Dynamics of LIF Neuron). Random patterns were drawn as described in Random Patterns in Numerical Estimation of ${\kappa }_{\mathrm{out}}^{\max }$ and ${\kappa }_{\mathrm{in}}^{\max }$ Solutions with $k=2$ and $\theta =\sqrt{2}$. Maximal ${\kappa }_{\mathrm{out}}$ solutions were found with $\mathrm{\Gamma }=1.5$ in units of $\left(V_{\mathrm{th}}-V_{\mathrm{rest}}\right)/{\sigma }_{\mathrm{exc}}$. No output noise was added in A–C. In D and E output noise was added with $N_{\mathrm{noise}}=30,000$ and ${\sigma }_{\mathrm{n}}=2/\sqrt{N_{\mathrm{noise}}}$ (Dynamics of LIF Neuron).

Fig. 5.

In C–F $N=2,000$, $P=1,000$, $f_{\mathrm{exc}}=0.8$, $p_{\mathrm{exc}}=0.1$, $p_{\mathrm{inh}}=0.2$, $\mathrm{\Gamma }=10\sqrt{p_{\mathrm{exc}}\left(1-p_{\mathrm{exc}}\right)}$ in units of $\left(V_{\mathrm{th}}-V_{\mathrm{rest}}\right)/{\sigma }_{\mathrm{exc}}$. In D and F results are averaged over 10 networks and 10 patterns from each network. See Recurrent Networks with Nonlearned Inhibition for parameters of I connectivity of the nonlearned inhibition networks (gray lines). In G maximal output noise magnitude is defined as the value of ${\sigma }_{\mathrm{out}}$ for which the stable pattern probability is 1/2. To minimize finite-size effects in simulations we used $N=3,000$, $f_{\mathrm{exc}}=0.8,$ $p_{\mathrm{exc}}=0.5$, $p_{\mathrm{inh}}=0.5$, $\mathrm{\Gamma }=10\sqrt{p_{\mathrm{exc}}\left(1-p_{\mathrm{exc}}\right)}$ in units of $\left(V_{\mathrm{th}}-V_{\mathrm{rest}}\right)/{\sigma }_{\mathrm{exc}}$. Stable pattern probability for each load and noise level was estimated by averaging over five networks and 20 patterns from each network.

Fig. 6.

Random patterns are binary pattern $x_i^{\mu }\in \left\{\mathrm{0,1}\right\}$ with equal probabilities and an even split of plus and minus patterns. $N=3,000$, $P=900$. Learning algorithm parameters are $\epsilon =\mathrm{\hspace{0.17em}5}\cdot {\mathrm{\hspace{0.17em}10}}^{-7}$, $\rho =0.1/N$, ${\sigma }_{\mathrm{in}}=0.1$ (Perceptron Learning Algorithm). Results are averaged over 50 samples.

Finding Maximal ${𝜿}_{\text{in}}$ and Maximal ${𝜿}_{\text{out}}$ Solutions.

Here we describe how finding the maximal ${\kappa }_{\mathrm{in}}$ and maximal ${\kappa }_{\mathrm{out}}$ solutions can be expressed as convex conic optimization problems. This allows us to efficiently validate the theoretical results. As noted in the main text, maximizing ${\kappa }_{\mathrm{in}}$ is equivalent to maximizing the margin of the solution’s weight vector as is done by support vector machines (39). However, to our knowledge, the application of conic optimization tools for maximizing ${\kappa }_{\mathrm{out}}$ is a unique contribution of our work.

Solution weight vectors, w, with input robustness ${\kappa }_{\mathrm{in}}$ or output robustness ${\kappa }_{\mathrm{out}}$, satisfy the inequalities

$$\forall \mu y^{\mu }\left({\text{𝒘}}^T{\text{𝒙}}^{\mu }-V_{\mathrm{th}}\right)\ge D,$$ [S1]

where $D_{\mathrm{in}}=\left|\text{𝒘}\right|{\kappa }_{\mathrm{in}}$ and $D_{\mathrm{out}}={\kappa }_{\mathrm{out}}$ (here we assume without loss of generality that $V_{\mathrm{rest}}=0$).

For each solution w we define effective weights, u, and effective threshold $b$ [the so-called canonical weights and threshold (39)] given by

$$u_i=\mathrm{\Lambda }{w}_i$$ [S2]

$$b=\mathrm{\Lambda }{V}_{\mathrm{th}},$$ [S3]

where $\mathrm{\Lambda }>0$ is chosen such that $\mathrm{\Lambda }D=1$ (for either $D_{\mathrm{in}}$ or $D_{\mathrm{out}}$).

Together with the sign and norm constraints on the weights, u and $b$ must satisfy the linear constraints

$$\forall \mu y^{\mu }\left({\text{𝒖}}^T{\text{𝒙}}^{\mu }-b\right)\ge 1\forall i{s}_i{u}_i\ge 0b\ge 0,$$ [S4]

where $s_i=1$ if $w_i$ is excitatory and $s_i=-1$ if $w_i$ is inhibitory, and the quadratic constraint

$${\left|\text{𝒖}\right|}^2\le b^2{\mathrm{\Gamma }}^2/V_{\mathrm{th}}^2,$$ [S5]

which enforces the constraint $\left|\text{𝒘}\right|\le \mathrm{\Gamma }$.

For the effective weights and threshold, ${\kappa }_{\mathrm{in}}$ is given by ${\kappa }_{\mathrm{in}}=\frac{1}{\left|\text{𝒖}\right|}$ and ${\kappa }_{\mathrm{out}}$ is given by ${\kappa }_{\mathrm{out}}=\frac{V_{\mathrm{th}}}{b}$. Thus, maximizing ${\kappa }_{\mathrm{in}}$ is equivalent to minimizing $\left|\text{𝒖}\right|$ and maximizing ${\kappa }_{\mathrm{out}}$ is equivalent to minimizing $b$. We therefore define a minimization cost function $E(\text{𝒖},b)$ that is given by

$$E_{\text{in}}(\text{𝒖},b)=\frac{1}{2}{\text{𝒖}}^T\text{𝒖},$$ [S6]

for the ${\kappa }_{\mathrm{in}}^{\max }$ solution, and

$$E_{\text{out}}(\text{𝒖},b)=b,$$ [S7]

for the ${\kappa }_{\mathrm{out}}^{\max }$ solution.

To find the maximal ${\kappa }_{\mathrm{in}}$ or maximal ${\kappa }_{\mathrm{out}}$ solution we solve the conic program

$$\underset{u,b,\tau }{\min }E(\text{𝒖},b)+\beta \tau$$ [S8]

in the limit of $\beta \to \mathrm{\infty }$, subject to

$$\forall \mu y^{\mu }\left({\text{𝒖}}^T{\text{𝒙}}^{\mu }-b\right)\ge 1-\tau \forall i{s}_i{u}_i\ge 0b\ge 0\tau \ge 0b^2{\mathrm{\Gamma }}^2/V_{\mathrm{th}}^2\ge {\left|\text{𝒖}\right|}^2.$$ [S9]

$\tau$ is a global regularization variable that ensures the existence of a solution to the optimization problem (Eqs. S8 and S9) even when the linear constraints S4 are not realizable. In practice it is sufficient to set $\beta$ to be a large constant (we set $\beta ={\mathrm{\hspace{0.17em}10}}^5$). If the optimal value of $\tau$ is zero, the solution corresponds to the optimal perceptron solution for the classification task. If the optimal value of $\tau$ is greater then zero, it indicates that the labeled patterns are not linearly separable and that there is no zero error solution to the classification task. Given that a solution with $\tau =0$ is found, the optimal weights are given by $\text{𝒘}=V_{\mathrm{th}}\text{𝒖}/b$.

SI Capacity for Noneven Split of Plus and Minus Patterns

The capacity of a perceptron with no sign constraints on synaptic weights for classification of random patterns is a function of the fraction of plus patterns in the desired classification, $p_{\mathrm{out}}$ (20–22), and is given by

$${\alpha }_{\mathrm{c}}^{\mathrm{Unconst}.}={\left[p_{\mathrm{out}}{\int }_{-\mathrm{\infty }}^{\mathrm{\Delta }}Dt{\left(t-\mathrm{\Delta }\right)}^2+\left(1-p_{\mathrm{out}}\right){\int }_{\mathrm{\Delta }}^{\mathrm{\infty }}Dt{\left(t-\mathrm{\Delta }\right)}^2\right]}^{-1},$$

where $Dt$ is the Gaussian integration measure, $Dt=\frac{e^{-\frac{t^2}{2}}}{\sqrt{2\pi }}\mathrm{d}t$, and the order parameter $\mathrm{\Delta }$ is given by the solution to the equation

$$0=p_{\mathrm{out}}{\int }_{-\mathrm{\infty }}^{\mathrm{\Delta }}Dt\left(t-\mathrm{\Delta }\right)+\left(1-p_{\mathrm{out}}\right){\int }_{\mathrm{\Delta }}^{\mathrm{\infty }}Dt\left(t-\mathrm{\Delta }\right).$$

Fig. S1A depicts the theoretical and measured ${\alpha }_{\mathrm{c}}$ of our “constrained” perceptron as a fraction of the corresponding unconstrained capacity vs. $f_{\mathrm{exc}}$ for two values of $p_{\mathrm{out}}$. Fig. S1B depicts theoretical and measured ${\alpha }_{\mathrm{b}}$ as a fraction of ${\alpha }_{\mathrm{c}}$ for two values of $p_{\mathrm{out}}$.

SI Effects of E and I Input Statistics

Our results depend, of course, on parameters, but in a fairly reduced way. In particular, the properties we discuss depend on the ratio of the inputs standard deviations, $\lambda ={\sigma }_{\mathrm{inh}}/{\sigma }_{\mathrm{exc}}$, and the ratio of their coefficients of variation, $\varphi ={\mathrm{CV}}_{\mathrm{exc}}/{\mathrm{CV}}_{\mathrm{inh}}$ (Repica Theory for Sign- and Norm-Constrained Perceptron). As discussed in the main text $\varphi$ determines the value of the optimal fraction of excitatory synapses, $f_{\mathrm{exc}}^{\star }$, which can be written as $f_{\mathrm{exc}}^{\star }=\varphi /\left(1+\varphi \right)$ (Eq. 2 in the main text). Thus, the shape of the phase diagram changes with $\varphi$ (Fig. S2A). The parameter $\lambda$ has more subtle effects. We note here the main effect $\lambda$ has on the maximal ${\kappa }_{\mathrm{in}}$ and maximal ${\kappa }_{\mathrm{out}}$ solutions.

Balanced and Unbalanced Maximal ${𝜿}_{\text{in}}$ Solutions.

The maximal ${\kappa }_{\mathrm{in}}$ solutions can be either balanced or unbalanced, depending on $f_{\mathrm{exc}},\varphi ,\lambda$ and the value of ${\kappa }_{\mathrm{in}}^{\max }$ (example in Fig. S2B). Importantly, for a wide range of reasonable parameters [for example, $\varphi \le f_{\mathrm{exc}}/\left(1-f_{\mathrm{exc}}\right)$ and $\lambda \ge 1$] the ${\kappa }_{\mathrm{in}}^{\max }$ solution is unbalanced for all values of ${\kappa }_{\mathrm{in}}^{\max }$.

Fraction of “Silent” Weights in Maximal ${𝜿}_{\text{in}}$ and Maximal ${𝜿}_{\text{out}}$ Solutions.

As noted in previous studies (25, 27), a prominent feature of “critical” solutions with sign-constraint weights, such as the maximal ${\kappa }_{\mathrm{in}}$ and maximal ${\kappa }_{\mathrm{out}}$ solutions, is that a finite fraction of the synapses are silent; i.e., $w_i=0$. Our theory allows us to derive the full distribution of synaptic efficacies (Distribution of synaptic weights) and calculate the fraction of silent weights for each solution. For the maximal ${\kappa }_{\mathrm{out}}$ solutions in the unbalanced regime (${\alpha }_{\mathrm{b}}<\alpha <{\alpha }_{\mathrm{c}}$), the fraction of E (I) silent weights is always larger (smaller) then 1/2 (Fig. S2C). However, in the balanced regime ($\alpha <{\alpha }_{\mathrm{b}}$) the qualitative behavior depends on $\lambda$ (Fig. S2C). Interestingly, for unbalanced maximal ${\kappa }_{\mathrm{in}}$ solutions the fraction of silent weights is constant and equals 1/2 for both E and I inputs (Saddle-point equations for the maximal $\kappa$in solution and Distribution of synaptic weights).

Tuning Properties of Cortical Neurons Suggest That in Cortex $f_{\text{exc}}^{\mathbf{\star }}\mathbf{>}0.5$.

In cortical circuits, I neurons tend to fire with higher firing rates and are thought to be more broadly tuned than E neurons, implying, under reasonable assumptions, that both $\lambda$ and $\varphi$ are greater than 1, leading to $f_{\mathrm{exc}}^{\star }>0.5$.

To see this, we consider input neurons with Gaussian tuning curves to some external stimulus variable $\phi \in \left[\mathrm{0,1}\right]$; i.e., the mean response, $x_i$, of neuron $i$ to stimulus $\phi$ is given by

$$x_i=A_i\exp \left[-{\left(\phi -{\phi }_i^{\mathrm{pref}}\right)}^2/\left(2{\delta }_i^2\right)\right],$$ [S10]

where $A_i$, ${\phi }_i^{\mathrm{pref}}$, and ${\delta }_i$ characterize the response properties of the neuron. Assuming that $\phi$ is distributed uniformly and, for simplicity, that ${\delta }_i\ll 1$, the mean and variance of the neurons’ responses are given by

$${\overline{x}}_i=\sqrt{2\pi }{A}_i{\delta }_i,$$ [S11]

and

$${\sigma }_i^2\simeq A_i^2{\delta }_i\sqrt{\pi },$$ [S12]

where we neglect terms of order ${\delta }_i^2$. We now assume that $A_i=A_{\mathrm{exc}}$ and ${\delta }_i={\delta }_{\mathrm{exc}}$ if neuron $i$ is E and that $A_i=A_{\mathrm{inh}}$ and ${\delta }_i={\delta }_{\mathrm{inh}}$ if neuron $i$ is I. Further, we assume that I neurons respond with a higher firing rate ($A_{\mathrm{inh}}>A_{\mathrm{exc}}$) and are more broadly tuned (${\delta }_{\mathrm{inh}}>{\delta }_{\mathrm{exc}})$. In this case we have

$$\lambda =\frac{A_{\mathrm{inh}}\sqrt{{\delta }_{\mathrm{inh}}}}{A_{\mathrm{exc}}\sqrt{{\delta }_{\mathrm{exc}}}}>1,$$ [S13]

and

$$\varphi =\sqrt{\frac{{\delta }_{\mathrm{inh}}}{{\delta }_{\mathrm{exc}}}}>1.$$ [S14]

SI Recurrent Networks with Nonlearned Inhibition

In our basic model for an associative memory network we assume that the activity of both E and I neurons is specified in the desired memory states and that all network connections are learned. Both of these assumptions can be modified, creating new scenarios with different computational properties.

First, we assume that memory state is specified only by the activity of E neurons and that the memory is recalled when the activity of E neurons matches the memory state regardless of the activity of I neurons. The problem of learning in such a network is computationally hard since the learning needs to optimize the activity of the I neurons, using the full connectivity matrix. In our work we do not address this scenario. Instead we forgo the assumption that E and I connections onto I neurons are learned and replace them with randomly chosen connections, i.e., assume that E to I and I to I connections are not learned and random.

Choosing Random Synapses for I Neurons.

In this scenario the activity of I neurons is determined by the network dynamics. We consider random I to I and E to I weights with means $J_{\mathrm{II}}$ and $J_{\mathrm{IE}}$ and standard deviations ${\sigma }_{J_{\mathrm{II}}}$ and ${\sigma }_{J_{\mathrm{IE}}}$. We examine the distribution of I neurons’ membrane potential, given that the activity of E neurons is held at a memory state in which $p_{\mathrm{out}}^{\mathrm{exc}}N$ neurons are active. When $N$ is large, this distribution is Gaussian and we assume correlations are weak. Thus, the mean activity in the network is the probability that the membrane potential is above threshold and is given by the equation

$$m_{\mathrm{I}}=H\left(\frac{V_{\mathrm{th}}-⟨V⟩}{\sqrt{{\sigma }^2\left(V\right)}}\right),$$ [S15]

where $H\left(x\right)={\int }_x^{\mathrm{\infty }}\frac{e^{-\frac{y^2}{2}}}{\sqrt{2\pi }}\mathrm{d}y$, and $⟨V⟩$ and ${\sigma }^2\left(V\right)$ are the mean and variance of the membrane potential of I neurons, respectively.

On the other hand, given the mean activity, $m_{\mathrm{I}}$, the mean and variance of the membrane potentials are given by

$$⟨V⟩=N\left(p_{\mathrm{out}}^{\mathrm{exc}}{g}_{\mathrm{exc}}{J}_{\mathrm{IE}}-m_{\mathrm{I}}\left(1-g_{\mathrm{exc}}\right)J_{\mathrm{II}}\right)$$ [S16]

$${\sigma }^2\left(V\right)=N\left(\left[{\sigma }_{J_{\mathrm{IE}}}^2+\left(1-p_{\mathrm{out}}^{\mathrm{exc}}\right)J_{\mathrm{IE}}^2\right]p_{\mathrm{out}}^{\mathrm{exc}}{g}_{\mathrm{exc}}+\left[{\sigma }_{J_{\mathrm{II}}}^2+\left(1-m_{\mathrm{I}}\right)J_{\mathrm{II}}^2\right]m_{\mathrm{I}}\left(1-g_{\mathrm{exc}}\right)\right).$$ [S17]

Together, Eqs. S15–S17 define the relations between $m_{\mathrm{I}}$, $J_{\mathrm{II}}$, $J_{\mathrm{IE}}$, ${\sigma }_{J_{\mathrm{IE}}}$, and ${\sigma }_{J_{\mathrm{II}}}$.

In our simulations we set $m_{\mathrm{I}}$ and the mean and variance of the I to I connections and choose the mean and variance of the E to I connections according to the solution of Eq. S15 [when $N$ is large, $J_{\mathrm{IE}}$ is given by $N{J}_{\mathrm{IE}}\simeq \left(V_{\mathrm{th}}+m_{\mathrm{I}}N\left(1-g_{\mathrm{exc}}\right)J_{\mathrm{II}}\right)/\left(g_{\mathrm{exc}}{p}_{\mathrm{out}}^{\mathrm{exc}}\right)$]. In particular, we choose an I network with binary weights in which each I neuron projects to another I neuron with probability $p_{\mathrm{II}}$ with synaptic efficacy $j_{\mathrm{II}}=1/\left(\sqrt{N}{p}_{\mathrm{II}}\right)$. Each E neuron projects to an I neuron with probability $p_{\mathrm{IE}}$ with synaptic efficacy $j_{\mathrm{IE}}$ that ensures that the mean I activity level at the memory states is $m_{\mathrm{I}}$.

In this parameter regime, the I subnetwork exhibits asynchronous activity, with mean activity $m_{\mathrm{I}}$, at the E memory states. However, different memory states lead to different asynchronous states.

Training Set Definition.

E neurons need to learn to remain stationary at the desired memory states, given the network activity at this state. However, since the activity of the I subnetwork is not stationary at the desired memory states, the training set for learning is not well defined.

To properly define the training set, we sample $n_{\mathrm{sample}}$ instances of the generated I activity for each memory state when the activity of the E neuron is clipped to this memory state. Sampling was performed by running the I network dynamics and recording the state of the I neurons after $T=100N$ time steps. We then use the sampled activity patterns together with the E memory states as an extended training set (with $P{n}_{\mathrm{sample}}$ patterns) for the E neurons.

Learned Network Stability.

The nonfixed point dynamics of the I subnetwork imply that the convergence of the learning on the training set does not entail that the memory states themselves are dynamically stable, in contrast to our prior model in which I neurons learn their synaptic weights. Therefore, after training we measure the probabilities that patterns are stable. This is done by the following procedure: First, we run the network dynamics (with ${\sigma }_{\mathrm{out}}=0$) when the E neurons’ activity is clipped to the memory state, for $T_{\mathrm{init}}=50N$ time steps. We then release the E neurons to evolve according to the natural network dynamics and observe whether their activity remains in the vicinity of the memory state for $T=500N$ time steps. In a similar way we test the basins of attraction, starting the E network from a distorted version of the memory state instead of the memory state itself.

Learning Only E to E Connections.

First, we consider the case in which I to E connections are random: Each I neuron projects to an E neuron with probability $p_{\mathrm{EI}}$ with synaptic efficacy $j_{\mathrm{EI}}=1/\left(\sqrt{N}{p}_{\mathrm{EI}}\right)$. We then try to find an appropriate E to E connection, using the learning scheme described above. We find that the pattern to pattern fluctuations in the I feedback due to the variance of the I to E connections and the variance in the I network neurons’ activation are substantial and of the same order of the signal differentiating the memory states. In fact, in this scenario the parameters we consider ($N=2,000$, $P=1,000,$ $g_{\mathrm{exc}}=0.8$, $p_{\mathrm{out}}^{\mathrm{exc}}=0.15,$ $p_{\mathrm{II}}=1/2,$ $p_{IE}=1/2,$ $m_{\mathrm{I}}=0.4$, $p_{\mathrm{EI}}=1/2$, $n_{\mathrm{sample}}=40$) are above the system’s memory capacity and we are unable to find appropriate E weights which implement the desired memory states for the training set. We conclude that this form of balancing I feedback is too restrictive due to the heterogeneity of I to E connections and variability of I neurons’ activity.

Learning Both E to E and I to E Connections.

In this scenario we find the maximal ${\kappa }_{\mathrm{out}}$ solution for the extended training set described above. For the parameters used ($N=2,000$, $P=1,000,$ $g_{\mathrm{exc}}=0.8$, $p_{\mathrm{out}}^{\mathrm{exc}}=0.15,$ $p_{\mathrm{II}}=1/2,$ $p_{\mathrm{IE}}=1/2,$ $m_{\mathrm{I}}=0.4,$ $n_{\mathrm{sample}}=40$) we are able to find solutions that implement all of the desired memory states for the extended training set. In addition, we find that the E memory states are dynamically stable with very high probability (we did not observe any unstable pattern). For numerical results see Fig. 5 and Fig S5.

SI Replica Theory for Sign- and Norm-Constrained Perceptron

We use the replica method (75) to calculate the system’s typical properties. For the perceptron architecture the replica symmetric solution has been shown to be stable and exact (20–22).

Given a set of $P$ patterns, ${\text{𝒙}}^{\mu },$ and desired labels $y^{\mu }=\pm 1$ for $\mu =1,2,\mathrm{\dots },P$, the Gardner volume is given by

$$V_{\mathrm{G}}=\int \mathcal{D}\left(\text{𝒘}\right)\prod _{\mu =1}^P\mathrm{\Theta }\left[y^{\mu }\left(w^{\mathrm{T}}{\text{𝒙}}^{\mu }-V_{\mathrm{th}}\right)-K\right],$$ [S18]

where $\mathrm{\Theta }\left[x\right]$ is the Heaviside step function and $\mathcal{D}\left(\text{𝒘}\right)$ is an integration domain obeying the sign and norm constraint $\left|\text{𝒘}\right|\le \mathrm{\Gamma }$.

We assume input pattern and labels are drawn independently from distributions with nonnegative means ${\overline{x}}_{\mathrm{exc}\left(\mathrm{inh}\right)}$ and SD ${\sigma }_{\mathrm{exc}\left(\mathrm{inh}\right)}$. Labels are independently drawn from a binary distribution with $\mathrm{Pr}(y^{\mu }=1)=p_{\mathrm{out}}$ and $\mathrm{Pr}(y^{\mu }=-1)=1-p_{\mathrm{out}}$.

We handle both input and output robustness criteria by using different $K$ for each case,

$$K_{\text{in}}=\left|\text{𝒘}\right|{\sigma }_{\mathrm{exc}}{\kappa }_{\mathrm{in}}{K}_{\text{out}}=V_{\mathrm{th}}{\kappa }_{\mathrm{out}},$$ [S19]

where here, ${\kappa }_{\mathrm{in}}$ and ${\kappa }_{\mathrm{out}}$ are dimensionless numbers representing the input robustness in units of ${\sigma }_{\mathrm{exc}}$ and the output robustness in units of $V_{\mathrm{th}}$, respectively.

Further, we define the parameters

$$\lambda =\frac{{\sigma }_{\mathrm{inh}}}{{\sigma }_{\mathrm{exc}}},\eta =\frac{{\overline{x}}_{\mathrm{inh}}}{{\overline{x}}_{\mathrm{exc}}},$$ [S20]

and

$$\varphi =\frac{{\mathrm{CV}}_{\mathrm{exc}}}{{\mathrm{CV}}_{\mathrm{inh}}}.$$ [S21]

The Order Parameters.

We calculate the mean logarithm of the Gardner volume ${⟨{⟨\ln V_{\mathrm{G}}⟩}_x⟩}_y$ averaged over the E and I input distributions and the desired label distribution. The result of the calculation is expressing ${⟨{⟨\ln V_{\mathrm{G}}⟩}_x⟩}_y$ as a stationary-phase integral over a free energy that is a function of several order parameters. The value of the order parameters is determined by the saddle-point equations of the free energy.

In our model the saddle-point equations are a system of six equations for the six order parameters: $q,Q,\theta ,\mathrm{\Delta },B$, and $C$.

The order parameters $q,Q,\theta ,\mathrm{and}\mathrm{\Delta }$ have a straightforward physical interpretation.

The parameter $q$ is the mean typical correlation coefficient between the $V_{\mathrm{PSP}}$s elicited by two different solutions to the same classification task: Given two typical solution weight vectors ${\text{𝒘}}^{\alpha }$ and ${\text{𝒘}}^{\beta }$, $q$ is given by

$$q=\frac{{\sum }_{i=1}^N{\lambda }_i^2{w}_i^{\alpha }{w}_i^{\beta }}{\sqrt{\left({\sum }_{i=1}^N{\lambda }_i^2{w}_i^{\alpha 2}\right)\left({\sum }_{i=1}^N{\lambda }_i^2{w}_i^{\beta 2}\right)}},$$ [S22]

where ${\lambda }_i=1$ if $w_i$ is E and ${\lambda }_i=\lambda$ if $w_i$ is I.

Given a typical solution w, the physical interpretation of $Q$ and $\theta$ is given by

$$Q=\frac{{\sum }_{i=1}^N{\lambda }_i^2{w}_i^2}{{\sum }_{i=1}^N{w}_i^2}$$ [S23]

$$\theta =\frac{V_{\mathrm{th}}}{{\sigma }_{\mathrm{exc}}}{\left(\sum _{i=1}^N{\lambda }_i^2{w}_i^2\right)}^{-\frac{1}{2}}.$$ [S24]

The norm constraint on the weights is satisfied as long as

$$\theta \ge \frac{V_{\mathrm{th}}}{{\sigma }_{\mathrm{exc}}\mathrm{\Gamma }\sqrt{Q}}.$$ [S25]

Thus, there are two types of solutions: one in which the value of $\theta$ is determined by the saddle-point equation (unbalanced solutions) and the other in which $\theta$ is clipped to its lower-bound value (balanced solutions). Note that $q$ and $Q$ remain of order 1 for any scaling of $\left|\text{𝒘}\right|$ while $\theta$ scales as $\sqrt{N}$ when $\left|\text{𝒘}\right|$ is of order $1/\sqrt{N}$ and is of order $1$ when $\left|\text{𝒘}\right|$ is of order 1.

The physical interpretation of $\mathrm{\Delta }$ can be expressed through the relation

$$\mathrm{\Delta }=\theta \left(1-\frac{{\overline{x}}_{\mathrm{exc}}{\sum }_{i=1}^N{\eta }_i{w}_i}{V_{\mathrm{th}}}\right),$$ [S26]

where ${\eta }_i=1$ if $w_i$ is E and ${\eta }_i=\eta$ if $w_i$ is I.

Summary of Main Results.

Before describing the full saddle-point (SP) equations and their various solutions in detail, we provide a brief general summary of the results that will hopefully provide some flavor of the derivations for the interested reader.

Since $\theta$ is bounded from below by $V_{\mathrm{th}}/\left({\sigma }_{\mathrm{exc}}\mathrm{\Gamma }\sqrt{Q}\right)$, we have two sets of SP equations which we term the balanced and the unbalanced sets. In both sets, given the free energy $\mathcal{F}(Q,q,\mathrm{\Delta },\theta ,B,C)$, five of the SP equations are given by

$$\frac{\partial \mathcal{F}}{\partial Q}=\frac{\partial \mathcal{F}}{\partial q}=\frac{\partial \mathcal{F}}{\partial \mathrm{\Delta }}=\frac{\partial \mathcal{F}}{\partial B}=\frac{\partial \mathcal{F}}{\partial C}=0.$$ [S27]

The sixth equation is

$$\frac{\partial \mathcal{F}}{\partial \theta }=0$$ [S28]

in the unbalanced set and is

$$\theta =\frac{V_{\mathrm{th}}}{{\sigma }_{\mathrm{exc}}\mathrm{\Gamma }\sqrt{Q}}$$ [S29]

in the balanced set. Importantly, we find that Eq. S28 has solutions only when $\theta \sim \sqrt{N}$, which implies $\left|\text{𝒘}\right|\sim 1/\sqrt{N}$, and Eq. S29 implies $\left|\text{𝒘}\right|=\mathrm{\Gamma }\sim \mathcal{O}\left(1\right)$, justifying the naming of the two sets. The solutions to the two sets of SP equations define the range of possible values of $\alpha$, ${\kappa }_{\mathrm{in}}$, and ${\kappa }_{\mathrm{out}}$ that permits the existence of solution weight vectors. There are a number of interesting cases that we analyze below.

We first consider the solutions of the SP equations for zero ${\kappa }_{\mathrm{out}}$ and ${\kappa }_{\mathrm{in}}$. In this case the SP describes the typical solutions that dominate the Gardner volume. Since the $N$-dim. volume of balanced solutions with $\left|\text{𝒘}\right|\sim 1$ is exponentially larger than the volume of unbalanced solutions with $\left|\text{𝒘}\right|\sim 1/\sqrt{N}$, we expect that balanced solutions will dominate the Gardner volume whenever they exist. Indeed, solving the two sets of SP equations we find that solutions to the balanced set exist only for $\alpha <{\alpha }_{\mathrm{b}}$ while solutions for the unbalanced set exist only for ${\alpha }_{\mathrm{b}}<\alpha <{\alpha }_{\mathrm{c}}$.

Next, we examine the values of ${\kappa }_{\mathrm{out}}$ that permit solutions to the balanced and unbalanced sets of SP equations. Importantly, we show that the unbalanced set can be solved only for ${\kappa }_{\mathrm{out}}\propto 1/\sqrt{N}$. Thus, unbalanced solutions cannot have ${\kappa }_{\mathrm{out}}$ of $\mathcal{O}\left(1\right)$ or conversely all solutions with ${\kappa }_{\mathrm{out}}$ of $\mathcal{O}\left(1\right)$ are balanced.

Of particular interest are the so-called critical solutions for which $q\to 1$. At this limit the typical correlation coefficient between the $V_{\mathrm{PSP}}$s elicited by two different solutions to the same classification task approaches unity, which implies that only one solution exists and the Gardner volume shrinks to zero. Thus, for a given ${\kappa }_{\mathrm{in}}$ or ${\kappa }_{\mathrm{out}}$, the value of $\alpha$ for which $q\to 1$ is the maximal load for which solutions exist. In this case, the SP describes the properties of the maximal ${\kappa }_{\mathrm{out}}$ or ${\kappa }_{\mathrm{in}}$ solutions.

The structure of the equations in this limit is relatively simple. First, the order parameter $\mathrm{\Delta }$ is given by the solutions to

$$0=p_{\mathrm{out}}{\int }_{-\mathrm{\infty }}^{\mathrm{\Delta }+\stackrel{\sim }{K}}Dt\left(t-\mathrm{\Delta }-\stackrel{\sim }{K}\right)+\left(1-p_{\mathrm{out}}\right){\int }_{\mathrm{\Delta }-\stackrel{\sim }{K}}^{\mathrm{\infty }}Dt\left(t-\mathrm{\Delta }+\stackrel{\sim }{K}\right)$$ [S30]

with the robustness parameter $\stackrel{\sim }{K}$ being ${\stackrel{\sim }{K}}_{\text{in}}={\kappa }_{\mathrm{in}}/\sqrt{Q}$ or ${\stackrel{\sim }{K}}_{\text{out}}=\theta {\kappa }_{\mathrm{out}}$ and the integration measure, $Dt$, is given by $Dt=\frac{e^{-\frac{t^2}{2}}}{\sqrt{2\pi }}\mathrm{d}t$. Second, we find a simple relation between critical loads of the constrained perceptron considered here and the critical loads of the classic unconstrained perceptron, ${\alpha }^{\mathrm{Unconst}.}$:

$$\alpha =2C{\alpha }^{\mathrm{Unconst}.}.$$ [S31]

${\alpha }^{\mathrm{Unconst}.}$ is given by

$${\alpha }^{\mathrm{Unconst}.}={\left[p_{\mathrm{out}}{\int }_{-\mathrm{\infty }}^{\mathrm{\Delta }+\stackrel{\sim }{K}}Dt{\left(t-\mathrm{\Delta }-\stackrel{\sim }{K}\right)}^2+\left(1-p_{\mathrm{out}}\right){\int }_{\mathrm{\Delta }-\stackrel{\sim }{K}}^{\mathrm{\infty }}Dt{\left(t-\mathrm{\Delta }+\stackrel{\sim }{K}\right)}^2\right]}^{-1},$$ [S32]

which is indeed the critical load of an unconstrained perceptron with a given margin $\stackrel{\sim }{K}$ (21). Finding the critical load is then reduced to solving for the order parameter $C$. For each value of ${\kappa }_{\mathrm{in}}>0$ or ${\kappa }_{\mathrm{out}}>0$ only one set of SP equations can be solved, determining whether the maximal ${\kappa }_{\mathrm{in}}$ or ${\kappa }_{\mathrm{out}}$ solutions are balanced or unbalanced. By examining the range of solutions for each set we can find the value of ${\kappa }_{\mathrm{in}}^{\max }$ and ${\kappa }_{\mathrm{out}}^{\max }$ for any $\alpha$ and determine that (i) the maximal ${\kappa }_{\mathrm{out}}$ solution is balanced for $\alpha \le {\alpha }_{\mathrm{b}}$ and unbalanced for ${\alpha }_{\mathrm{b}}<\alpha <{\alpha }_{\mathrm{c}}$ and (ii) for a wide range of parameters the maximal ${\kappa }_{\mathrm{in}}$ is unbalanced for all $\alpha <{\alpha }_{\mathrm{c}}$. In addition, we find that for ${\alpha }_{\mathrm{b}}<\alpha <{\alpha }_{\mathrm{c}}$, ${\kappa }_{\mathrm{out}}^{\max }$ is given by

$${\kappa }_{\mathrm{out}}^{\max }=\frac{{\sigma }_{\mathrm{exc}}}{{\overline{x}}_{\mathrm{exc}}\sqrt{N}}{\kappa }_0,$$ [S33]

where ${\kappa }_0$ is finite and larger than zero when $\alpha$ approaches ${\alpha }_{\mathrm{b}}$ from above and ${\kappa }_0$ approaches zero when $\alpha$ approaches ${\alpha }_{\mathrm{c}}$. The above result implies that output robustness can be increased when the tuning of the input is increased. As we discuss in the main text in the context of neuronal selectivity in purely E circuits, sparse input activity is one way to increase the input tuning. If we consider sparse binary inputs with mean activity level $s\ll 1$, the output robustness will be given by ${\kappa }_0\sqrt{\left(1-s\right)/sN}\simeq {\kappa }_0/\sqrt{sN}$.

Finally, we consider the solutions of the SP equations in the critical limit ($q\to 1$) for ${\kappa }_{\mathrm{in}}={\kappa }_{\mathrm{out}}=0$. In this limit the SP describes the capacity and balanced capacity. We note that for ${\kappa }_{\mathrm{out}}={\kappa }_{\mathrm{in}}=0$, $\mathrm{\Delta }$ (and as a result ${\alpha }_{\mathrm{c}}^{\mathrm{Unconst}.}$) is independent of all of the other order parameters, simplifying the equations. In this case, we have only two coupled SP equations (for the order parameters $B$, $C$, and $\theta$), given by

$${\left(1-\frac{{\sigma }_{\mathrm{exc}}B\theta }{{\overline{x}}_{\mathrm{exc}}\sqrt{2CN}}\right)}^{2}C=f_{\mathrm{exc}}{\gamma }_{+}\left(B\right)+\left(1-f_{\mathrm{exc}}\right){\gamma }_{-}\left(B\varphi \right)$$ [S34]

$$\frac{{\sigma }_{\mathrm{exc}}\theta }{{\overline{x}}_{\mathrm{exc}}\sqrt{N}}=-\frac{f_{\mathrm{exc}}{\gamma }_{+}^{\prime }\left(B\right)+\left(1-f_{\mathrm{exc}}\right)\varphi {\gamma }_{-}^{\prime }\left(B\varphi \right)}{\sqrt{2C}},$$ [S35]

where we defined the functions

$${\gamma }_{\pm }\left(x\right)=\frac{x^2+1}{2}-\frac{1}{2}\int Du{\left(x+u\right)}^2\mathrm{\Theta }\left[\pm \left(x+u\right)\right]$$ [S36]

$${\gamma }_{\pm }^{\prime }\left(x\right)=x-\int Du\left(x+u\right)\mathrm{\Theta }\left[\pm \left(x+u\right)\right].$$ [S37]

For the balanced SP equations we have $\theta =V_{\mathrm{th}}/\left({\sigma }_{\mathrm{exc}}\mathrm{\Gamma }\sqrt{Q}\right)$ and for the unbalanced SP equations, Eq. S28 reduces to $B=0$. Finally, $\alpha$ is given by Eq. S31.

For the unbalanced set ($B=0$) we have ${\gamma }_{\pm }\left(0\right)=\frac{1}{2}-\frac{1}{2}{\int }_0^{\mathrm{\infty }}Du{\left(u\right)}^2=\frac{1}{4}$ and ${\gamma }_{\pm }^{\prime }\left(0\right)=\mp {\int }_0^{\mathrm{\infty }}Du\left(u\right)=\mp \frac{1}{\sqrt{\pi }}$. We immediately get $C=\frac{1}{4}$ and

$$\theta =\sqrt{\frac{2N}{\pi }}\left[f_{\mathrm{exc}}/{\mathrm{CV}}_{\mathrm{exc}}-\left(1-{\mathrm{f}}_{\mathrm{exc}}\right)/{\mathrm{CV}}_{\mathrm{inh}}\right].$$ [S38]

This solution suggests that at capacity the solutions are unbalanced $\left(\theta \sim \sqrt{N}\Rightarrow \left|\text{𝒘}\right|\sim 1/\sqrt{N}\right)$ and that capacity as a function of $f_{\mathrm{exc}}$ is constant with

$${\alpha }_{\mathrm{c}}=\frac{1}{2}{\alpha }_{\mathrm{c}}^{\mathrm{Unconst}.}.$$ [S39]

However, this solution is valid only as long as $\theta >V_{\mathrm{th}}/\left({\sigma }_{\mathrm{exc}}\mathrm{\Gamma }\sqrt{Q}\right)$, which is true only as long as

$$\left[f_{\mathrm{exc}}/{\mathrm{CV}}_{\mathrm{exc}}-\left(1-f_{\mathrm{exc}}\right)/{\mathrm{CV}}_{\mathrm{inh}}\right]>0,$$ [S40]

which implies

$$f_{\mathrm{exc}}>f_{\mathrm{exc}}^{\star }=\frac{{\mathrm{CV}}_{\mathrm{exc}}}{{\mathrm{CV}}_{\mathrm{exc}}+{\mathrm{CV}}_{\mathrm{inh}}}.$$ [S41]

For the solution for the balanced set $\left(\theta =V_{\mathrm{th}}/\left({\sigma }_{\mathrm{exc}}\mathrm{\Gamma }\sqrt{Q}\right)\right)$ terms with $\theta /\sqrt{N}$ can be neglected and we have the equation

$$0=f_{\mathrm{exc}}{\gamma }_{+}^{\prime }\left(B\right)+\left(1-f_{\mathrm{exc}}\right)\varphi {\gamma }_{-}^{\prime }\left(B\varphi \right)$$ [S42]

for the order parameter $B$. $C$ and $Q$ are given by

$$C=f_{\mathrm{exc}}{\gamma }_{+}\left(B\right)+\left(1-f_{\mathrm{exc}}\right){\gamma }_{-}\left(B\varphi \right)$$ [S43]

$$Q=\frac{f_{\mathrm{exc}}{\gamma }_{+}\left(B\right)+\left(1-f_{\mathrm{exc}}\right){\gamma }_{-}\left(B\varphi \right)}{f_{\mathrm{exc}}{\gamma }_{+}\left(B\right)+\left(\frac{1-f_{\mathrm{exc}}}{{\lambda }^2}\right){\gamma }_{-}\left(B\varphi \right)}.$$ [S44]

This solution gives us the balanced capacity line

$${\alpha }_{\mathrm{b}}\left(f_{\mathrm{exc}}\right)=2\left[f_{\mathrm{exc}}{\gamma }_{+}\left(B\right)+\left(1-f_{\mathrm{exc}}\right){\gamma }_{-}\left(B\varphi \right)\right]{{\alpha }^{\mathrm{Uncont}.}}_{\left(p_{\mathrm{out}}\right)},$$ [S45]

where $B$ is given by the solution of Eq. S42 and ${{\alpha }^{\mathrm{Uncont}.}}_{\left(p_{\mathrm{out}}\right)}$ is given by Eqs. S32 and S30 with $\stackrel{\sim }{K}=0$.

Detailed Solutions of the SP Equations.

Below we provide the SP equations and their solutions under various conditions. We also provide the derived form of the distributions of synaptic weights for critical solutions.

The general SP equations.

We define the following:

$${\mathcal{F}}_h=p_{\mathrm{out}}\int Dt\ln H\left[-X_{+}\left(t\right)\right]+\left(1-p_{\mathrm{out}}\right)\int Dt\ln H\left[X_{-}\left(t\right)\right]$$ [S46]

$$Dt=\frac{e^{-\frac{t^2}{2}}}{\sqrt{2\pi }}\mathrm{d}t$$ [S47]

$$H\left(x\right)={\int }_x^{\mathrm{\infty }}Dt$$ [S48]

$$X_{\pm }\left(t\right)=\frac{\sqrt{q}t-\mathrm{\Delta }\mp \stackrel{\sim }{K}}{\sqrt{1-q}}$$ [S49]

$${\stackrel{\sim }{K}}_{\text{in}}={\kappa }_{\mathrm{in}}/\sqrt{Q}$$ [S50]

$${\stackrel{\sim }{K}}_{\text{out}}=\theta {\kappa }_{\mathrm{out}}$$ [S51]

$${\varphi }_{+}=1,{\varphi }_{-}=\varphi$$ [S52]

$${\lambda }_{+}=1,{\lambda }_{-}=\lambda$$ [S53]

$$f_{+}=f_{\mathrm{exc}},f_{-}=1-f_{\mathrm{exc}}$$ [S54]

$$\alpha =\frac{P}{N}$$ [S55]

$$\stackrel{\sim }{\theta }=\frac{{\sigma }_{\mathrm{exc}}\left(\theta -\mathrm{\Delta }\right)}{{\overline{x}}_{\mathrm{exc}}\sqrt{N}}$$ [S56]

$$Z_{\pm }=2C{\left[2C-\sqrt{2C}B\stackrel{\sim }{\theta }+\left(1-q\right)\times \left(1-2\alpha Q\frac{\partial {\mathcal{F}}_h}{\partial Q}\left(1-\frac{Q}{{\lambda }_{\pm }^2}\right)\right)\right]}^{-1}$$ [S57]

$${\mathrm{\Phi }}_{\pm }(x,z,q)=\frac{z\left(x^2+1\right)}{2}+\frac{1-q}{2}\times \left[1+\int Du{J}_1\left(\pm \frac{\sqrt{z}\left(x+u\right)}{\sqrt{\left(1-q\right)}}\right)\right]$$ [S58]

$${\mathrm{\Phi }}_{\pm }^{\prime }(x,z,q)=zx\pm \sqrt{z\left(1-q\right)}\int Du{J}_2\left(\pm \frac{\sqrt{z}\left(x+u\right)}{\sqrt{\left(1-q\right)}}\right)$$ [S59]

$$J_1\left(x\right)=\frac{H\prime \left(x\right)}{H\left(x\right)}x$$ [S60]

$$J_2\left(x\right)=\frac{H\prime \left(x\right)}{H\left(x\right)}.$$ [S61]

The SP equations are given by

$$C/Q=\sum _{\pm }\frac{f_{\pm }}{{\lambda }_{\pm }^2}{Z}_{\pm }{\mathrm{\Phi }}_{\pm }(B\varphi ,Z_{\pm },q)$$ [S62]

$$C=\sum _{\pm }{f}_{\pm }{Z}_{\pm }{\mathrm{\Phi }}_{\pm }(B{\varphi }_{\pm },Z_{\pm },q)$$ [S63]

$$\sqrt{2C}\stackrel{\sim }{\theta }=-\sum _{\pm }{f}_{\pm }{\varphi }_{\pm }{\mathrm{\Phi }}_{\pm }^{\prime }(B{\varphi }_{\pm },Z_{\pm },q)$$ [S64]

$$\alpha =-\frac{C}{{\left(1-q\right)}^2}{\left(\frac{\partial {\mathcal{F}}_h}{\partial q}\right)}^{-1}$$ [S65]

$$\frac{{\sigma }_{\mathrm{exc}}B}{{\overline{x}}_{\mathrm{exc}}\sqrt{2CN}}=-\frac{1}{2\left(1-q\right)}\frac{\partial {\mathcal{F}}_h}{\partial \mathrm{\Delta }}{\left(\frac{\partial {\mathcal{F}}_h}{\partial q}\right)}^{-1}$$ [S66]

$$\frac{{\sigma }_{\mathrm{exc}}B}{{\overline{x}}_{\mathrm{exc}}\sqrt{2CN}}=\frac{1-q}{2\theta C}+\frac{1}{2\left(1-q\right)}\frac{\partial {\mathcal{F}}_h}{\partial \theta }{\left(\frac{\partial {\mathcal{F}}_h}{\partial q}\right)}^{-1}\text{𝑶𝑹}\theta =\frac{V_{\mathrm{th}}}{{\sigma }_{\mathrm{exc}}\mathrm{\Gamma }\sqrt{Q}}.$$ [S67]

It is important to note the relation between $\theta$ and $\stackrel{\sim }{\theta }$. $\stackrel{\sim }{\theta }$ is of the order of $\theta /\sqrt{N}$ ($\mathrm{\Delta }$ remains of order 1 under all conditions). Thus, for unbalanced solutions $\theta \sim \sqrt{N}$ and $\stackrel{\sim }{\theta }$ is of $\mathcal{O}\left(1\right)$ while for balanced solutions $\theta$ is of $\mathcal{O}\left(1\right)$ and $\stackrel{\sim }{\theta }$ is of $\mathcal{O}\left(1/\sqrt{N}\right)$ and can be neglected.

SP equations for typical solutions.

For typical solutions we solve the SP equations for ${\kappa }_{\mathrm{in}}=0$ or ${\kappa }_{\mathrm{out}}=0$, leading to $K=0$. In this case we have $\frac{\partial {\mathcal{F}}_h}{\partial \theta }=\frac{\partial {\mathcal{F}}_h}{\partial Q}=0$ and thus

$$Z_{\pm }=2C{\left[2C-\sqrt{2C}B\stackrel{\sim }{\theta }+\left(1-q\right)\right]}^{-1}$$ [S68]

and the SP equation for $\theta$ is

$$\frac{{\sigma }_{\mathrm{exc}}\sqrt{2C}B\theta }{{\overline{x}}_{\mathrm{exc}}\sqrt{N}}=1-q\text{𝑶𝑹}\theta =\frac{V_{\mathrm{th}}}{{\sigma }_{\mathrm{exc}}\mathrm{\Gamma }\sqrt{Q}}.$$ [S69]

We now can solve the SP equations for the unbalanced case with

$$Z_{\pm }=1\mathrm{and}\theta >\frac{V_{\mathrm{th}}}{{\sigma }_{\mathrm{exc}}\mathrm{\Gamma }\sqrt{Q}},\stackrel{\sim }{\theta }>0$$ [S70]

and for the balanced case with

$$Z_{\pm }=2C{\left[2C+\left(1-q\right)\right]}^{-1}\mathrm{and}\theta =\frac{V_{\mathrm{th}}}{{\sigma }_{\mathrm{exc}}\mathrm{\Gamma }\sqrt{Q}},\stackrel{\sim }{\theta }=0.$$ [S71]

We find that for $\alpha <{\alpha }_{\mathrm{b}}$ a solution exists only for equations of the balanced case while for $\alpha >{\alpha }_{\mathrm{b}}$ a solution exists for the equations for the unbalanced case. Thus, typical solutions are balanced below ${\alpha }_{\mathrm{b}}$ and unbalanced above it. The norm of the weight and the IB depicted in Fig. 2 B and C are given by

$$\left|\text{𝒘}\right|=\frac{1}{\sqrt{Q}\theta }$$ [S72]

$$\mathrm{IB}=\frac{{\sum }_{\pm }{f}_{\pm }{\varphi }_{\pm }{\mathrm{\Phi }}_{\pm }^{\prime }(B{\varphi }_{\pm },Z,q)}{{\sum }_{\pm }\pm f_{\pm }{\varphi }_{\pm }{\mathrm{\Phi }}_{\pm }^{\prime }(B{\varphi }_{\pm },Z,q)}.$$ [S73]

Solutions with significant ${𝜿}_{\mathrm{𝐨𝐮𝐭}}$ are balanced.

In this section we show that all unbalanced solutions have output robustness of order $1/\sqrt{N}$ and, equivalently, solutions with ${\kappa }_{\mathrm{out}}$ of order 1 are balanced.

Theorem.

All unbalanced solutions have output robustness of the order of $1/\sqrt{N}.$

Proof.

In the case of output robustness we have $\stackrel{\sim }{K}={\stackrel{\sim }{K}}_{\mathrm{out}}=\theta {\kappa }_{\mathrm{out}}$ and thus $\frac{\partial {\mathcal{F}}_h}{\partial Q}=0$. We are looking for unbalanced solutions $\left(\stackrel{\sim }{\theta }>0,\theta \sim \mathcal{O}(\sqrt{N})\right)$ so we have the equations

$$\frac{{\sigma }_{\mathrm{exc}}B}{{\overline{x}}_{\mathrm{exc}}\sqrt{2CN}}=-\frac{1}{2\left(1-q\right)}\frac{\partial {\mathcal{F}}_h}{\partial \mathrm{\Delta }}{\left(\frac{\partial {\mathcal{F}}_h}{\partial q}\right)}^{-1}$$ [S74]

$$\frac{{\sigma }_{\mathrm{exc}}B}{{\overline{x}}_{\mathrm{exc}}\sqrt{2CN}}=\frac{1-q}{2\theta C}+\frac{1}{2\left(1-q\right)}\frac{\partial {\mathcal{F}}_h}{\partial \theta }{\left(\frac{\partial {\mathcal{F}}_h}{\partial q}\right)}^{-1}.$$ [S75]

Both equations must be satisfied and therefore we have (using Eqs. S65, S74, and S75)

$$0=\frac{\partial {\mathcal{F}}_h}{\partial \mathrm{\Delta }}+\frac{\partial {\mathcal{F}}_h}{\partial \theta }-\frac{1}{\alpha \theta }.$$ [S76]

Performing the derivatives, we get

$$0=\left({\kappa }_{\mathrm{out}}+1\right)p_{\mathrm{out}}\int Dt{J}_2\left(-X_{+}\right)+\left({\kappa }_{\mathrm{out}}-1\right)\left(1-p_{\mathrm{out}}\right)\int Dt{J}_2\left(X_{-}\right)-\frac{\sqrt{1-q}}{\alpha \theta }.$$ [S77]

Now, we use Eq. S74, leading to

$$p_{\mathrm{out}}\int Dt{J}_2\left(-X_{+}\right)=\left(1-p_{\mathrm{out}}\right)\int Dt{J}_2\left(X_{-}\right)-M/\sqrt{N},$$ [S78]

where we defined $M$ as

$$M=\frac{\sqrt{2}B{\sigma }_{\mathrm{exc}}\sqrt{1-q}}{\sqrt{C}{\overline{x}}_{\mathrm{exc}}}\frac{\partial {\mathcal{F}}_h}{\partial q},$$ [S79]

which remains of $\mathcal{O}\left(1\right)$. Thus, we are left with

$$0=2{\kappa }_{\mathrm{out}}\left(1-p_{\mathrm{out}}\right)\int Dt{J}_2\left(X_{-}\right)-\left({\kappa }_{\mathrm{out}}+1\right)\frac{M}{\sqrt{N}}-\frac{\sqrt{1-q}}{\alpha \theta }.$$ [S80]

Note that $J_2\left(x\right)<0$ and the first term is negative (nonzero). The other two terms scale as $1/\sqrt{N}$ and therefore the equation can be satisfied only if $\kappa =\frac{{\kappa }_0}{\sqrt{N}}$.

SP equations for critical solutions.

To find the capacity, balanced capacity, and solutions with maximal output and input robustness we consider the limit $q\to 1$.

We define

$$G_Q=\underset{q\to 1}{lim}\frac{Q}{\left(1-q\right)}{\left(\frac{\partial {\mathcal{F}}_h}{\partial q}\right)}^{-1}\frac{\partial {\mathcal{F}}_h}{\partial Q},$$ [S81]

and thus in this limit $Z_{\pm }$ is given by

$$Z_{\pm }={\left[1-\frac{B\stackrel{\sim }{\theta }}{\sqrt{2C}}+\left(1-\frac{Q}{{\lambda }_{\pm }^2}\right)G_Q\right]}^{-1}.$$ [S82]

In addition,

$$\underset{q\to 1}{lim}{\mathrm{\Phi }}_{\pm }(x,z,q)=z{\gamma }_{\pm }\left(x\right)$$ [S83]

$${\gamma }_{\pm }\left(x\right)=\frac{x^2+1}{2}-\frac{1}{2}\int Du{\left(x+u\right)}^2\mathrm{\Theta }\left[\pm \left(x+u\right)\right]$$ [S84]

$$\underset{q\to 1}{lim}{\mathrm{\Phi }}_{\pm }^{\prime }(x,z,q)=z{\gamma }_{\pm }^{\prime }\left(x\right)$$ [S85]

$${\gamma }_{\pm }^{\prime }\left(x\right)=x-\int Du\left(x+u\right)\mathrm{\Theta }\left[\pm \left(x+u\right)\right],$$ [S86]

and, in the $q\to 1$ limit we have

$$\left(1-q\right)\frac{\partial {\mathcal{F}}_h}{\partial \mathrm{\Delta }}=M(\mathrm{\Delta },\stackrel{\sim }{K})$$ [S87]

$$M(\mathrm{\Delta },\stackrel{\sim }{K})=p_{\mathrm{out}}{\int }_{-\mathrm{\infty }}^{\mathrm{\Delta }+\stackrel{\sim }{K}}Dt\left(t-\mathrm{\Delta }-\stackrel{\sim }{K}\right)+\left(1-p_{\mathrm{out}}\right){\int }_{\mathrm{\Delta }-\stackrel{\sim }{K}}^{\mathrm{\infty }}Dt\left(t-\mathrm{\Delta }+\stackrel{\sim }{K}\right)$$ [S88]

$${\left(1-q\right)}^2\frac{\partial {\mathcal{F}}_h}{\partial q}=-\frac{1}{2}{\alpha }^{\mathrm{Unconst}.}(\mathrm{\Delta },\stackrel{\sim }{K})$$ [S89]

$${\alpha }^{\mathrm{Unconst}.}(\mathrm{\Delta },\stackrel{\sim }{K})={\left[p_{\mathrm{out}}{\int }_{-\mathrm{\infty }}^{\mathrm{\Delta }+\stackrel{\sim }{K}}Dt{\left(t-\mathrm{\Delta }-\stackrel{\sim }{K}\right)}^2+\left(1-p_{\mathrm{out}}\right){\int }_{\mathrm{\Delta }-\stackrel{\sim }{K}}^{\mathrm{\infty }}Dt{\left(t-\mathrm{\Delta }+\stackrel{\sim }{K}\right)}^2\right]}^{-1}.$$ [S90]

We now write the final form of the SP equations for critical solutions:

$$C=\sum _{\pm }{f}_{\pm }{Z}_{\pm }^2{\gamma }_{\pm }\left(B{\varphi }_{\pm }\right)$$ [S91]

$$Q=\frac{{\sum }_{\pm }{f}_{\pm }{Z}_{\pm }^2{\gamma }_{\pm }\left(B{\varphi }_{\pm }\right)}{{\sum }_{\pm }\frac{f_{\pm }}{{\lambda }_{\pm }^2}{Z}_{\pm }^2{\gamma }_{\pm }\left(B{\varphi }_{\pm }\right)}$$ [S92]

$$\stackrel{\sim }{\theta }=-\frac{{\sum }_{\pm }{f}_{\pm }{\varphi }_{\pm }{Z}_{\pm }{\gamma }_{\pm }^{\prime }\left(B{\varphi }_{\pm }\right)}{\sqrt{2{\sum }_{\pm }{f}_{\pm }{Z}_{\pm }^2{\gamma }_{\pm }\left(B{\varphi }_{\pm }\right)}}$$ [S93]

$$0=M(\mathrm{\Delta },\stackrel{\sim }{K})$$ [S94]

$$\frac{B}{\sqrt{2CN}}=-\frac{{\overline{x}}_{\mathrm{ex}}}{{\sigma }_{\mathrm{ex}}}2C{\alpha }^{\mathrm{Unconst}.}(\mathrm{\Delta },\stackrel{\sim }{K})\underset{q\to 1}{lim}\left(1-q\right)\frac{\partial {\mathcal{F}}_h}{\partial \theta }\text{𝑶𝑹}\theta =\frac{V_{\mathrm{th}}}{{\sigma }_{\mathrm{exc}}\mathrm{\Gamma }\sqrt{Q}}.$$ [S95]

Finally $\alpha$ is given by

$$\alpha =2C{\alpha }^{\mathrm{Unconst}.}(\mathrm{\Delta },\stackrel{\sim }{K}).$$ [S96]

Capacity and balanced capacity.

The capacity is given for $\delta =0$ and $\kappa =0$. In this case both $\frac{\partial {\mathcal{F}}_h}{\partial \theta }$ and $\frac{\partial {\mathcal{F}}_h}{\partial Q}$ are zero and Eq. S95 has two possible solutions:

$$B=0,\theta >\frac{V_{\mathrm{th}}}{{\sigma }_{\mathrm{exc}}\mathrm{\Gamma }\sqrt{Q}},$$ [S97]

for unbalanced solutions or

$$\theta =\frac{V_{\mathrm{th}}}{{\sigma }_{\mathrm{exc}}\mathrm{\Gamma }\sqrt{Q}},\stackrel{\sim }{\theta }=0,$$ [S98]

for balanced solutions. In both cases we have $Z_{\pm }=1$.

Unbalanced solution.

The SP equations become

$$Q=\frac{1}{{\sum }_{\pm }\frac{1}{{\lambda }_{\pm }^2}{f}_{\pm }}$$ [S99]

$$\stackrel{\sim }{\theta }=\frac{1}{\sqrt{\pi }}\sum _{\pm }\pm f_{\pm }{\varphi }_{\pm }$$ [S100]

$$C=\frac{1}{4}$$ [S101]

and the capacity is given by

$${\alpha }_{\mathrm{c}}=\frac{1}{2}{\alpha }^{\mathrm{Unconst}.}(\mathrm{\Delta },0),$$ [S102]

where $\mathrm{\Delta }$ is given by $M(\mathrm{\Delta },0)=0$.

This solution is valid only when $\theta$ is larger than its $\mathcal{O}\left(1\right)$ lower bound, which is guarantied in the large $N$ limit as long as $\stackrel{\sim }{\theta }>0$. Using Eq. S100, this entails that

$$f_{\mathrm{exc}}>f_{\mathrm{exc}}^{\star }$$ [S103]

with

$$f_{\mathrm{exc}}^{\star }=\frac{\varphi }{1+\varphi }$$ [S104]

or conversely $\varphi <\frac{f_{\mathrm{exc}}}{1-f_{\mathrm{exc}}}.$

Balanced solution.

In this solution we have $\theta =\frac{V_{\mathrm{th}}}{{\sigma }_{\mathrm{exc}}\mathrm{\Gamma }\sqrt{Q}},\stackrel{\sim }{\theta }=0$.

$B$ is given by the solution to

$$\sum _{\pm }{f}_{\pm }{\varphi }_{\pm }{\gamma }_{\pm }^{\prime }\left(B{\varphi }_{\pm }\right)=0$$ [S105]

and we have

$$Q=\frac{{\sum }_{\pm }{f}_{\pm }{\gamma }_{\pm }\left(B{\varphi }_{\pm }\right)}{{\sum }_{\pm }\frac{1}{{\lambda }_{\pm }^2}{f}_{\pm }{\gamma }_{\pm }\left(B{\varphi }_{\pm }\right)}$$ [S106]

$$C=\sum _{\pm }{f}_{\pm }{\gamma }_{\pm }\left(B{\varphi }_{\pm }\right)$$ [S107]

and

$${\alpha }_{\mathrm{b}}=2C{\alpha }^{\mathrm{Unconst}.}(\mathrm{\Delta },0),$$ [S108]

where $\mathrm{\Delta }$ is given by $M(\mathrm{\Delta },0)=0$.

This gives the balanced capacity line. For $f_{\mathrm{exc}}<f_{\mathrm{exc}}^{\star }$ this is the capacity line as well. Thus, for $f_{\mathrm{exc}}<f_{\mathrm{exc}}^{\star }$, at capacity the solution is balanced.

Coexistence of balanced and unbalanced solutions below the balanced capacity line.

To show that unbalanced solutions coexist with balanced solutions for any $\alpha <{\alpha }_{\mathrm{b}}$, we calculate the capacity of unbalanced solutions with a given norm. This can be done by solving Eqs. S91–S94 while imposing the condition $\left|\text{𝒘}\right|=\frac{V_{\mathrm{thr}}W}{\sqrt{N}{\sigma }_{\mathrm{exc}}}$ through the SP equation of $\theta$:

$$\theta =\frac{\sqrt{N}}{W\sqrt{Q}}.$$ [S109]

We therefore have

$$\stackrel{\sim }{\theta }=\frac{{\sigma }_{\mathrm{exc}}\left(\theta -\mathrm{\Delta }\right)}{{\overline{x}}_{\mathrm{exc}}\sqrt{N}}\simeq \frac{{\sigma }_{\mathrm{exc}}}{{\overline{x}}_{\mathrm{exc}}W\sqrt{Q}}=\frac{1}{\stackrel{\sim }{W}\sqrt{Q}}.$$ [S110]

We are interested in the capacity and therefore we take $K=0$. As a result we have

$$Z_{\pm }={\left[1-\frac{B}{\sqrt{2CQ{\stackrel{\sim }{W}}^2}}\right]}^{-1}$$ [S111]

and the SP equations become

$$\frac{1}{\stackrel{\sim }{W}}=-\frac{{\sum }_{\pm }{f}_{\pm }{\varphi }_{\pm }{\gamma }_{\pm }^{\prime }\left(B{\varphi }_{\pm }\right)}{\sqrt{2{\sum }_{\pm }\frac{f_{\pm }}{{\lambda }_{\pm }^2}{\gamma }_{\pm }\left(B{\varphi }_{\pm }\right)}}$$ [S112]

$$C={\left[1-\frac{B}{\stackrel{\sim }{W}\sqrt{2CQ}}\right]}^{-2}\sum _{\pm }{f}_{\pm }{\gamma }_{\pm }\left(B{\varphi }_{\pm }\right)$$ [S113]

$$0=M(\mathrm{\Delta },0),$$ [S114]

where $Q$ is given by

$$Q=\frac{{\sum }_{\pm }{f}_{\pm }{\gamma }_{\pm }\left(B{\varphi }_{\pm }\right)}{{\sum }_{\pm }\frac{f_{\pm }}{{\lambda }_{\pm }^2}{\gamma }_{\pm }\left(B{\varphi }_{\pm }\right)}.$$ [S115]

Given the value of $B$ and $Q$ the equation for $C$ can be solved for $\sqrt{C}$ and we get

$$\sqrt{C}=\frac{B}{\sqrt{2Q}\stackrel{\sim }{W}}+\sqrt{\sum _{\pm }{f}_{\pm }{\gamma }_{\pm }\left(B{\varphi }_{\pm }\right)-{\left(\frac{B}{\sqrt{2Q}\stackrel{\sim }{W}}\right)}^2}.$$ [S116]

This solution is valid as long as $\sqrt{C}\ge 0$. The conditional capacity ${\alpha }_{\mathrm{c}}\left(\stackrel{\sim }{W}\right)$ is then given by

$${\alpha }_{\mathrm{c}}\left(\stackrel{\sim }{W}\right)=2C{\alpha }^{\mathrm{Unconst}.}(\mathrm{\Delta },0).$$ [S117]

It is easy to see that for $\stackrel{\sim }{W}\to \mathrm{\infty }$, the SP equations converge to the equations of the balanced capacity and thus ${\alpha }_{\mathrm{c}}\left(\stackrel{\sim }{W}\right)$ approaches ${\alpha }_{\mathrm{b}}$. In addition, we find that for $f_{\mathrm{exc}}<f_{\mathrm{exc}}^{\star }$, ${\alpha }_{\mathrm{c}}\left(\stackrel{\sim }{W}\right)$ is a monotonically increasing function of $\stackrel{\sim }{W}$. Another way to interpret this result is to “invert the function” and ask, What is the minimal value of $\stackrel{\sim }{W}$ that permits solutions given $\alpha$? Our result implies that strictly below ${\alpha }_{\mathrm{b}}$ the minimal value of $\stackrel{\sim }{W}$ that permits solutions is of $\mathcal{O}\left(1\right)$ [i.e., $\left|\text{𝒘}\right|$ of $\mathcal{O}(1/\sqrt{N})$] and unbalanced solutions exist. The minimal $\stackrel{\sim }{W}$ diverges as $\alpha$ approaches ${\alpha }_{\mathrm{b}}$ and hence the solution at ${\alpha }_{\mathrm{b}}$ is balanced [$\left|\text{𝒘}\right|$ of $\mathcal{O}\left(1\right)$].

SP equations for the maximal ${𝜿}_{\mathrm{𝐢𝐧}}$ solution.

In this case we have $\stackrel{\sim }{K}={\stackrel{\sim }{K}}_{\text{in}}=\frac{\delta }{\sqrt{Q}}$ and therefore $\frac{\partial {\mathcal{F}}_h}{\partial \theta }=0$. For unbalanced solutions we have

$$B=0,\theta >\frac{V_{\mathrm{th}}}{{\sigma }_{\mathrm{exc}}\mathrm{\Gamma }\sqrt{Q}},\stackrel{\sim }{\theta }>0$$ [S118]

and for balanced solutions we have

$$\theta =\frac{V_{\mathrm{th}}}{{\sigma }_{\mathrm{exc}}\mathrm{\Gamma }\sqrt{Q}},\stackrel{\sim }{\theta }=0.$$ [S119]

In both cases $Z_{\pm }$ is given by

$$Z_{\pm }={\left[1+\left(1-\frac{Q}{{\lambda }_{\pm }^2}\right)G_Q\right]}^{-1}$$ [S120]

with

$$G_Q={\kappa }_{\mathrm{in}}/\sqrt{Q}{\alpha }^{\mathrm{Unconst}.}(\mathrm{\Delta },{\kappa }_{\mathrm{in}}/\sqrt{Q})\times \left[p_{\mathrm{out}}{\int }_{-\mathrm{\infty }}^{\mathrm{\Delta }+\frac{{\kappa }_{\mathrm{in}}}{\sqrt{Q}}}Dt\left(t-\mathrm{\Delta }-\frac{{\kappa }_{\mathrm{in}}}{\sqrt{Q}}\right)-\left(1-p_{\mathrm{out}}\right){\int }_{\mathrm{\Delta }-\frac{{\kappa }_{\mathrm{in}}}{\sqrt{Q}}}^{\mathrm{\infty }}Dt\left(t-\mathrm{\Delta }+\frac{{\kappa }_{\mathrm{in}}}{\sqrt{Q}}\right)\right].$$ [S121]

Unbalanced solution.

In this case we have equations for $\mathrm{\Delta }$ and $Q$:

$$M(\mathrm{\Delta },{\kappa }_{\mathrm{in}}/\sqrt{Q})=0$$ [S122]

$$Q=\frac{{\sum }_{\pm }{f}_{\pm }{Z}_{\pm }^2}{{\sum }_{\pm }\frac{1}{{\lambda }_{\pm }^2}{f}_{\pm }{Z}_{\pm }^2}.$$ [S123]

We then have

$$\stackrel{\sim }{\theta }=\frac{\frac{1}{\sqrt{2\pi }}{\sum }_{\pm }\pm f_{\pm }{\varphi }_{\pm }{Z}_{\pm }}{\sqrt{\frac{1}{2}{\sum }_{\pm }{f}_{\pm }{Z}_{\pm }^2}}$$ [S124]

$$C=\frac{1}{4}\sum _{\pm }{f}_{\pm }{Z}_{\pm }^2$$ [S125]

and

$$\alpha =2C{\alpha }^{\mathrm{Unconst}.}(\mathrm{\Delta },{\kappa }_{\mathrm{in}}/\sqrt{Q}).$$ [S126]

Balanced solution.

In this case we have equations for $\mathrm{\Delta }$, $B$, and $Q$:

$$M(\mathrm{\Delta },{\kappa }_{\mathrm{in}}/\sqrt{Q})=0$$ [S127]

$$Q=\frac{{\sum }_{\pm }{f}_{\pm }{\gamma }_{\pm }\left(B{\varphi }_{\pm }\right)Z_{\pm }^2}{{\sum }_{\pm }\frac{1}{{\lambda }_{\pm }^2}{f}_{\pm }{\gamma }_{\pm }\left(B{\varphi }_{\pm }\right)Z_{\pm }^2}$$ [S128]

$$0=\sum _{\pm }{f}_{\pm }{\varphi }_{\pm }{\gamma }_{\pm }^{\prime }\left(B{\varphi }_{\pm }\right)Z_{\pm }.$$ [S129]

The equations for $G_Q$ (Eq. S121) and $\alpha$ (Eq. S126) remain the same; however, $C$ is given by

$$C=\sum _{\pm }{f}_{\pm }{\gamma }_{\pm }\left(B{\varphi }_{\pm }\right)Z_{\pm }^2.$$ [S130]

Transition between balanced and unbalanced solutions.

Transition points between balanced and unbalanced solutions depend on the value of $\varphi ,$ $\lambda$, and $f_{\mathrm{exc}}$. Transition points are points in which both $B=0$ and $\stackrel{\sim }{\theta }=0$. Thus, we have

$${\varphi }^{\star }=\frac{f_{\mathrm{exc}}{Z}_{+}}{\left(1-f_{\mathrm{exc}}\right)Z_{-}},$$ [S131]

where $Q$ and $\mathrm{\Delta }$ are given by Eqs. S123 and S122. Thus, ${\varphi }^{\star }$ is a function of ${\kappa }_{\mathrm{in}}$ and $\lambda$. Solutions are balanced for $\varphi >{\varphi }^{\star }$ and unbalanced for $\varphi <{\varphi }^{\star }$ (Fig. S2B).

SP equations for the maximal ${𝜿}_{\mathrm{𝐨𝐮𝐭}}$ solution.

Unbalanced solution.

This solution is valid for $\alpha >{\alpha }_{\mathrm{b}}$, $f_{\mathrm{exc}}>f_{\mathrm{exc}}^{\star }$. We look for a solution with $\stackrel{\sim }{\theta }>0$ and thus $\theta$ must scale as $\sqrt{N}$.

In this case $\stackrel{\sim }{K}=\theta \kappa$ and so $\frac{\partial {\mathcal{F}}_h}{\partial Q}=0$ and $Z_{\pm }={\left[1-\frac{B\stackrel{\sim }{\theta }}{\sqrt{2C}}\right]}^{-1}$.We then have

$$Q=\frac{{\sum }_{\pm }{f}_{\pm }{\gamma }_{\pm }\left(B{\varphi }_{\pm }\right)}{{\sum }_{\pm }\frac{1}{{\lambda }_{\pm }^2}{f}_{\pm }{\gamma }_{\pm }\left(B{\varphi }_{\pm }\right)}$$ [S132]

and we are left with equations to solve for $\stackrel{\sim }{\theta },$ $B$, and $\mathrm{\Delta }$:

$$\stackrel{\sim }{\theta }=-\frac{{\sum }_{\pm }{f}_{\pm }{\varphi }_{\pm }{\gamma }_{\pm }^{\prime }\left(B{\varphi }_{\pm }\right)}{\sqrt{2{\sum }_{\pm }{f}_{\pm }{\gamma }_{\pm }\left(B{\varphi }_{\pm }\right)}}$$ [S133]

$$M(\mathrm{\Delta },\stackrel{\sim }{K})=0$$ [S134]

and

$$\frac{B}{\sqrt{2CN}}=-\frac{{\overline{x}}_{\mathrm{ex}}}{{\sigma }_{\mathrm{ex}}}\kappa {\alpha }^{\mathrm{Unconst}.}(\mathrm{\Delta },\stackrel{\sim }{K})\left[f{\int }_{-\mathrm{\infty }}^{\mathrm{\Delta }+\stackrel{\sim }{K}}Dt\left(t-\mathrm{\Delta }-\stackrel{\sim }{K}\right)-\left(1-f\right){\int }_{\mathrm{\Delta }-\stackrel{\sim }{K}}^{\mathrm{\infty }}Dt\left(t-\mathrm{\Delta }+\stackrel{\sim }{K}\right)\right].$$ [S135]

There is a solution only if

$$\kappa =\frac{{\sigma }_{\mathrm{exc}}}{{\overline{x}}_{\mathrm{exc}}\sqrt{N}}{\kappa }_0,\theta \simeq \sqrt{N}\frac{{\overline{x}}_{\mathrm{ex}}}{{\sigma }_{\mathrm{ex}}}\stackrel{\sim }{\theta }$$ [S136]

so we have $\stackrel{\sim }{K}=\stackrel{\sim }{\theta }{\kappa }_0$ and

$$\frac{B}{\sqrt{2C}}=-{\kappa }_0{\alpha }^{\mathrm{Unconst}.}(\mathrm{\Delta },\stackrel{\sim }{\theta }{\kappa }_0)\times \left[f{\int }_{-\mathrm{\infty }}^{\mathrm{\Delta }+\stackrel{\sim }{\theta }{\kappa }_0}Dt\left(t-\mathrm{\Delta }-\stackrel{\sim }{\theta }{\kappa }_0\right)-\left(1-f\right){\int }_{\mathrm{\Delta }-\stackrel{\sim }{\theta }{\kappa }_0}^{\mathrm{\infty }}Dt\left(t-\mathrm{\Delta }+\stackrel{\sim }{\theta }{\kappa }_0\right)\right].$$ [S137]

Finally, we have $C=Z^2{\sum }_{\pm }{f}_{\pm }{\gamma }_{\pm }\left(B{\varphi }_{\pm }\right),$ from which we can isolate $C$ to have

$$C=\left[\sum _{\pm }{f}_{\pm }{\gamma }_{\pm }\left(B{\varphi }_{\pm }\right)\right]{\left[1-\frac{B{\sum }_{\pm }{f}_{\pm }{\varphi }_{\pm }{\gamma }_{\pm }^{\prime }\left(B{\varphi }_{\pm }\right)}{2{\sum }_{\pm }{f}_{\pm }{\gamma }_{\pm }\left(B{\varphi }_{\pm }\right)}\right]}^2.$$ [S138]

$\alpha$ is given as before, $\alpha =2C{\alpha }^{\mathrm{Unconst}.}(\mathrm{\Delta },\stackrel{\sim }{\theta }{\kappa }_0)$.

The equations given in this section are equivalent to the ones derived in Eq. S27.

Balanced solution.

We look for balanced solutions with $\theta =\frac{V_{\mathrm{th}}}{{\sigma }_{\mathrm{exc}}\mathrm{\Gamma }\sqrt{Q}},\stackrel{\sim }{\theta }=0$. The SP equations in this case are given by the same equations as the balanced solution described in SP equations for the maximal $\kappa$in solution with ${\kappa }_{\mathrm{out}}{V}_{\mathrm{th}}/{\sigma }_{\mathrm{exc}}\mathrm{\Gamma }$ replacing ${\kappa }_{\mathrm{in}}$.

Distribution of Synaptic Weights.

We derive the mean distribution of synaptic weights for critical solutions ($q\to 1$)

$$P_{\pm }\left(w\right)=H\left(\mp B{\varphi }_{\pm }\right)\delta \left(w\right)+\sqrt{\frac{{\theta }^{2}N{\lambda }_{\pm }^2}{2\pi {\sigma }_{w\pm }^2}}\exp \times \left[-\frac{{\left(\theta \sqrt{N}{\lambda }_{\pm }w+B{\varphi }_{\pm }{\sigma }_{w\pm }\right)}^2}{2{\sigma }_{w\pm }^2}\right],$$ [S139]

with ${\sigma }_{w\pm }=\frac{Z_{\pm }}{\sqrt{2C}}$, where $P_{+}$ and $P_{-}$ denote the probability densities for E and I synaptic weights, respectively, $\delta \left(x\right)$ is the Dirac delta function, and weights are given in units of $V_{\mathrm{th}}/{\sigma }_{\mathrm{exc}}$. The fraction of silent synapses is given by $H\left(-B\right)$ for E synapses and by $H\left(B\varphi \right)$ for I synapses.

Discussion

The results we have presented come from imposing a set of fundamental biological constraints: fixed-sign synaptic weights, nonnegative afferent activities, a positive firing threshold (relative to the resting potential), and both input and output forms of noise. Amit et al. (23) studied the maximal margin solution for the sign-constrained perceptron and showed that it has half the capacity of the unconstrained perceptron. However, this previous work considered afferent activities that were centered around zero and a neuron with zero firing threshold, features that preclude the presence of the behavior exhibited by the more biologically constrained model studied here. Chapeton et al. (27) studied perceptron learning with sign-constrained weights and a preassigned level of robustness, but considered only solutions in the unbalanced regime which, as we have shown, are extremely sensitive to output noise.

Learning in neural circuits involves a trade-off between exhausting the system’s capacity for implementing complex input–output functions on the one hand and ensuring good generalization properties on the other. A well-known approach in machine learning has been to search for solutions that fit the training examples while maximizing the distance of samples from the decision surface, a strategy known as maximizing the margin (21, 23, 39). The margin being maximized in this case corresponds, in our framework, to ${\kappa }_{\mathrm{in}}$. Work in computational neuroscience has implicitly optimized a robustness parameter equivalent to our ${\kappa }_{\mathrm{out}}$ (25, 27). To our knowledge, the two approaches have not been distinguished before or shown to result in solutions with dramatically different noise sensitivities. In particular, over a wide parameter range, we have shown that maximizing ${\kappa }_{\mathrm{out}}$ leads to a balanced solution with minimal sensitivity to output noise and robustness to input noise that is almost as good as that of the maximal margin solution, with only a modest trade-off in capacity. On the other hand, maximizing the margin (${\kappa }_{\mathrm{in}}$) often leads to unbalanced solutions with extreme sensitivity to output noise.

The perceptron has long been considered a model of cerebellar learning and computation (67, 68). More recently, Brunel et al. (25) investigated the capacity and robustness of a perceptron model of a cerebellar Purkinje cell, taking all weights to be E. In view of the analysis presented here, balanced solutions are not possible in this case ($f_{\mathrm{exc}}=\mathrm{\hspace{0.17em}1}$), and solutions that maximize either input-noise or output-noise robustness both have ${\kappa }_{\mathrm{out}}\propto \mathrm{\hspace{0.17em}1}/\sqrt{N}$. These two types of solutions differ in their weight distributions, with experimentally testable consequences for the predicted circuit structure [SI ${\kappa }_{\mathrm{out}}^{\max }$ $\mathrm{and}$ ${\kappa }_{\mathrm{in}}^{\max }$ Solutions in Purely E Networks and Fig. S2C; Brunel et al. (25) considered only solutions that maximize ${\kappa }_{\mathrm{out}}$]. Output robustness of the unbalanced solutions can be increased by making the input activity patterns sparse. Denoting by $s$ the mean fraction of active neurons in the input, maximum output robustness scales as ${\kappa }_{\mathrm{out}}\sim \mathrm{\hspace{0.17em}1}/\sqrt{Ns}$ (Fig. 3B and SI Replica Theory for Sign- and Norm-Constrained Perceptron). Thus, the high sparsity in input activation (granule cell activity) of the cerebellum relative to the modest sparsity in the neocortex is consistent with the former being dominated by E modifiable synapses.

Interestingly, our results suggest an optimal ratio of E to I synapses. Capacity in the balanced regime is optimal when $f_{\mathrm{exc}}=f_{\mathrm{exc}}^{\star }$, with $f_{\mathrm{exc}}^{\star }$ determined by the CVs (with respect to stimulus) of the E and I inputs (Eq. 2). Thus, optimality predicts a simple relation between the fraction of E and I inputs and their degree of tuning. Estimating the CVs from existing data is difficult, but it would be interesting to check whether input statistics and connectivity ratios in different brain areas are consistent with this prediction. The commonly observed value in cortex, $f_{\mathrm{exc}}\simeq \mathrm{\hspace{0.17em}0.8}$, would be optimal for input statistics with ${\mathrm{CV}}_{\mathrm{exc}}/{\mathrm{CV}}_{\mathrm{inh}}\simeq \mathrm{\hspace{0.17em}4}$. In general, we expect that ${\mathrm{CV}}_{\mathrm{exc}}/{\mathrm{CV}}_{\mathrm{inh}}>\mathrm{\hspace{0.17em}1}$, which implies that $f_{\mathrm{exc}}^{\star }>\mathrm{\hspace{0.17em}1}/2$.

For most of our work, we assumed that I neurons learn to represent specific sensory and long-term memory information, the same as the E ones, and that all synaptic pathways are learned using similar learning rules. While plasticity in both E and I pathways has been observed (57–59, 61, 63, 64, 69), accumulating experimental evidence indicates a high degree of cell-type and synaptic-type specificity of the plasticity rules. In addition, synaptic plasticity is under tight control of neuromodulatory systems. At present, it is unclear how to interpret our learning rules in terms of concrete experimentally observed synaptic plasticity. Other functional models of neural learning assume learning only within the E population with inhibition acting as a global stabilizing force. In the case of sensory processing, our approach is consistent with the observation of a similar stimulus tuning of E and I postsynaptic currents in many cortical sensory areas. The role of I neurons in memory representations is less known (but see ref. 70). Importantly, we have shown that our main results are valid also in the case in which I neurons do not explicitly participate in the coding of the memories. Interestingly, our work suggests that even if I neurons are only passive observers during learning processes, learning of I synapses onto E cells can amplify the memory stability of the system against fluctuations in the I feedback. Given the diversity of I cell types it is likely that in the real circuits inhibition plays multiple roles, including both conveying information and providing stability.

Several previous models of associative memory have incorporated biological constraints on the sign of the synapses, Dale’s law, assuming variants of Hebbian plasticity in the E to E synapses (14, 48, 51–54). The capacity of the these Hebbian models is relatively poor, and their basins of attraction are small, except at extremely sparse activity levels. In contrast, our model applies a more powerful learning rule that, while keeping the sign constraints on the synapses, exhibits significantly superior performance: with high capacity even for moderate sparsity levels, large basins of attraction, and high robustness to output noise.

From a dynamical systems perspective, the associative memory networks we construct exhibit unusual properties. In most associative memory network models large basins of attractions endow the memory state with robustness against stochasticity in the dynamics (i.e., output noise). Here, we found that, for the same set of fixed-point memories, the synaptic weights with the largest possible basins (the unbalanced solutions with maximal ${\kappa }_{\mathrm{in}}$) are very sensitive to even mild levels of stochasticity, whereas the balanced synaptic weights with somewhat reduced basins have substantially increased output noise robustness.

At the network level, as at the single-neuron level, imposing basic features of neural circuitry—positive inputs, bounded synapses of fixed sign, a positive firing threshold, and sources of noise—forces neural circuits into the balanced regime. A recent class of models showing computational benefits of balanced inputs uses extremely strong synapses, which are outside the range we have discussed (16). These models are stabilized by instantaneous transmission of signals between neurons which are not required in the range of synaptic strength we consider.

Previous models of balanced networks have highlighted the ability of networks with strong E and I recurrent synapses to settle into a state in which the total input is dynamically balanced without special tuning of the synaptic strengths. Such a state is characterized by a high degree of intrinsically generated spatiotemporal variability (7). Mean population activities respond fast and in a linear fashion to external inputs. Typically, these networks lack the population-level nonlinearity required to generate multiple attractors. In contrast, we have explored the capacity of the balanced network to support multiple stable fixed points by tuning the synaptic strengths through appropriate learning. We note that fully understanding and characterizing the dynamic properties of these networks and their relation to previously studied models remains an important challenge. Despite the dynamic and functional differences in the two classes of networks, the balancing of excitation and inhibition plays a similar role in both. In the first scenario, synaptic balance amplifies small changes in the spatial or temporal properties of the external drive. Similarly, in the present scenario, balanced synaptic architecture leads to enhanced robustness by amplifying the small variations in the synaptic inputs induced by changes in the stimulus or memory identity. It would be very interesting to combine fast dynamics with robust associative memory capabilities.

In conclusion, we have uncovered a fundamental principle of neuronal learning under basic biological constraints. Our work reveals that excitation–inhibition balance may have a critical computational role in producing robust neuronal functionality that is insensitive to output noise. We showed that this balance is important at the single-neuron level for both spiking and nonspiking neurons and at the level of recurrently connected neural networks. Further, the theory suggests that excitation–inhibition balance may be a collective, self-maintaining, emergent phenomenon of synaptic plasticity. Any successful neuronal learning process in the presence of substantial output noise will lead to strong balanced synaptic efficacies with noise robustness features. The fundamental nature of this result suggests that it should apply across a variety of neuronal circuits that learn in the presence of noise.

Materials and Methods

Detailed methods and simulation parameters are given in SI Materials and Methods.

Software.

To acknowledge their contribution to scientific work we cite the open source projects that directly and most crucially contributed to the current work: The Python stack of scientific computing [CPython, Numpy, Scipy, Matplotlib (71), Jupyter/Ipython (72), and others], CVXOPT (73) (convex conic optimization), and IPyparallel (parallelization).

Code Availability.

Python code for simulations and numerical solution of saddle-point equations is available upon request.

SI ${𝜿}_{\text{out}}^{\text{max}}$ and ${𝜿}_{\text{in}}^{\text{max}}$ Solutions in Purely E Networks

In purely E networks ($f_{\mathrm{exc}}=1$) all solutions are unbalanced and output robustness can be achieved by sparse input (25) or tonic inhibition (28). However, the distinction between output robustness and input robustness still applies and, surprisingly, maximizing either ${\kappa }_{\mathrm{in}}$ or ${\kappa }_{\mathrm{out}}$ leads to two different solutions with qualitatively different properties.

In particular, as noted in refs. 25 and 27, the fraction of silent weights of the ${\kappa }_{\mathrm{out}}^{\max }$ solutions increases as the load decreases. Thus, if the network implements the maximal ${\kappa }_{\mathrm{out}}$ solution, network connectivity, as measured in a pairwise stimulation experiment, is expected to be sparse. However, for the maximal ${\kappa }_{\mathrm{in}}$ solution the fraction of silent weights is constant and remains $1/2$ for all values of the load. Thus, measured network connectivity is expected to be higher.

Establishing correspondence between theory and experiment in this case is confounded by the difficulty to experimentally distinguish between silent synapses and completely absent synapses that were never available as inputs for the postsynaptic neuron during learning.