Geometric framework to predict structure from function in neural networks

Abstract

Neural computation in biological and artificial networks relies on the nonlinear summation of many inputs. The structural connectivity matrix of synaptic weights between neurons is a critical determinant of overall network function, but quantitative links between neural network structure and function are complex and subtle. For example, many networks can give rise to similar functional responses, and the same network can function differently depending on context. Whether certain patterns of synaptic connectivity are required to generate specific network-level computations is largely unknown. Here we introduce a geometric framework for identifying synaptic connections required by steady-state responses in recurrent networks of threshold-linear neurons. Assuming that the number of specified response patterns does not exceed the number of input synapses, we analytically calculate the solution space of all feedforward and recurrent connectivity matrices that can generate the specified responses from the network inputs. A generalization accounting for noise further reveals that the solution space geometry can undergo topological transitions as the allowed error increases, which could provide insight into both neuroscience and machine learning. We ultimately use this geometric characterization to derive certainty conditions guaranteeing a nonzero synapse between neurons. Our theoretical framework could thus be applied to neural activity data to make rigorous anatomical predictions that follow generally from the model architecture.

I. INTRODUCTION

Structure-function relationships are fundamental to biology [1–3]. In neural networks, the structure of synaptic connectivity critically shapes the functional responses of neurons [4,5], and large-scale techniques for measuring neural network structure and function provide exciting opportunities for examining this link quantitatively [6–15]. The ellipsoid body in the central complex of Drosophila is a beautiful example where modeling showed how the structural pattern of excitatory and inhibitory connections enables a persistent representation of heading direction [16–19]. Lucid structure-function links have also been found in several other neural networks [20–23]. However, it is generally hard to predict either neural network structure or function from the other [5,24]. For example, functionally inferred connectivity can capture neuronal response correlations without matching structural connectivity [25–28], and network simulations with structural constraints do not automatically reproduce function [29–31]. Two broad modeling difficulties hinder the establishment of robust structure-function links. First, models with too much detail are difficult to adequately constrain and analyze. Second, models with too little detail may poorly match biological mechanisms, the model mismatch problem. Here we propose a rigorous theoretical framework that attempts to balance these competing factors to predict components of network structure required for function.

Neural network function probably does not depend on the exact strength of every synapse. Indeed, multiple network connectivity structures can generate the same functional responses [32,33], as illustrated by structural variability across individual animals [24,34] and artificial neural networks [29,35–37]. Such redundancy may be a general feature of emergent phenomena in physics, biology, and neuroscience [38–40]. Nevertheless, some important details may be consistent despite this variability, and here we find well-constrained structure-function links by characterizing all connectivity structures that are consistent with the desired functional responses [24]. We also account for ambiguities caused by measurement noise. Our goal is not to find degenerate networks that perform equivalently in all possible scenarios. We instead seek a framework that finds connectivity required for specific functional responses, independently of whatever else the network might do.

The model mismatch problem has at least two facets. First, neurons and synapses are incredibly complex [41–44], but which complexities are needed to elucidate specific structure-function relationships is unclear [5,45,46]. This issue is very hard to address in full generality, and here we seek a theoretical framework that makes clear experimental predictions that can adjudicate candidate models empirically. In particular, we predict neural network structure only when it occurs in all networks generating the functional responses. This high bar precludes the analysis of biophysically-detailed network models, which require numerical exploration of the connectivity space that is typically incomplete [24,32,47–49]. We instead focus on recurrent firing rate networks of threshold-linear neurons, which are growing in popularity because they strike an appealing balance between biological realism, computational power, and mathematical tractability [12,16,18,20,22,23,29,30,37,50–55].

The second facet of the model mismatch problem is hidden variables, such as missing neurons, neuromodulator levels, and physiological states [5,56–58]. Here we take inspiration from whole-brain imaging in small organisms [15], such as Caenorhabditis elegans [9], larval zebrafish [8,12,57], and larval Drosophila [11], and assume access to all relevant neurons. Our model neglects neuromodulators and other state variables, which would be interesting to consider in the future. Furthermore, many experiments indirectly assess neuronal spiking activity, such as by calcium florescence [58–61] or hemodynamic responses [25,62–64]. We restrict our analysis to steady-state responses to mitigate mismatch between fast firing rate changes and these inherently slow measurement techniques.

Our analysis begins with an analytical characterization of synaptic weight matrices that realize specified steady-state responses as fixed points of neural network dynamics [Figs. 1(a) and 1(b)]. A key insight is that asymmetrically constrained dimensions appear as a consequence of the threshold nonlinearity. Synaptic weight components in these semiconstrained dimensions are completely uncertain in one half of the dimension but well-constrained in the other. We then compute error surfaces by finding weight matrices with fixed points near the desired ones. This error landscape has a continuum of local and global minima, and constant-error surfaces exhibit topological transitions that add semiconstrained dimensions as the error increases. This may help explain the importance of weight initialization in machine learning, as poorly initialized models can get stuck in semiconstrained dimensions that abruptly vanish at nonzero error. By studying the geometric structure of the neural network ensemble that can approximate the functional responses, we derive analytical formulas that pinpoint a subset of connections, which we term certain synapses, that must exist for the model to work [Fig. 1(c)]. These analytical results are especially useful for studying high-dimensional synaptic weight spaces that are otherwise intractable. Since the presence of a synapse is readily measurable, our theory generates accessible experimental predictions [Fig. 1(c)]. Tests of these predictions assess the utility of the modeling framework itself, as the predictions hold across model parameters. Their successes and failures can thus move us forward toward identifying the mechanistic principles governing how neural networks implement brain computations.

FIG. 1.

Cartoon of theoretical framework. (a) We first specify some steady-state responses of a recurrent threshold-linear neural network receiving feedforward input. (b) We then find all synaptic weight matrices that have fixed points at the specified responses. Red (blue) matrix elements are positive (negative) synaptic weights. (c) When a weight is consistently positive (or consistently negative) across all possibilities, then the model needs a nonzero synaptic connection to generate the responses. We therefore make the experimental prediction that this synapse must exist. We also predict whether the synapse is excitatory or inhibitory.

The rest of the paper begins in Sec. II with a toy problem that concretely demonstrates the approach illustrated in Fig. 1 and relates the geometry of the solution space (all synaptic weight matrices that realize a given set of response patterns) to the concept of a certain synapse. In Sec. III, we explain how the solution space for a limited number of response patterns can be calculated for an arbitrarily large threshold-linear recurrent neural network. Section IV is devoted to three simple toy problems that provide additional insights into how the geometry of the solution space can help us to identify certain synapses. This is followed by Sec. V, where we explain and numerically test the precise algebraic relation that must be satisfied for a synapse to be certain when the response patterns are orthonormal. Section VI generalizes our analyses to include noise, including numerical tests via simulation. Finally, Sec. VII concludes the paper by summarizing our main results and discussing important future directions.

II. AN ILLUSTRATIVE TOY PROBLEM

To gain intuition on how robust structure-function links can be established, including the effects of nonlinearity, we begin by analyzing the structural implications of functional responses in a very simple threshold-linear feedforward network [Fig. 2(a)]. We assume that two input neurons, x₁ and x₂, provide signals to a single driven neuron, y, via synaptic weights, w₁ and w₂. The weights are unknown, and we constrain their possible values using two neuronal response patterns, labeled μ = + and μ = −. We suppose that steady-state activities of the input neurons and driven neuron are nonlinearly related according to

$$y=\Phi \left(w_1{x}_1+w_2{x}_2\right),$$ (1)

where x₁, x₂, and y denote firing rates of the corresponding neurons, and

$$\Phi (s)=\text{max}(0,s)$$ (2)

is the threshold-linear transfer function. The driven neuron responds (y = 1) when x₁ = x₂ = 1 in the μ = + pattern. In contrast, the driven neuron does not respond (y = 0) when x₁ = −x₂ = 1 in the μ = − pattern. If the transfer function were linear, then it is easy to see that there is a unique set of weights, $w_1=w_2=\frac{1}{2}$, that produces these driven neuron responses, the brown dot in Fig. 2(b).

FIG. 2.

An illustrative two-dimensional problem. (a) Cartoon depicting two stimulus response patterns in a simple feedforward network with two input neurons and one driven neuron. (b) Since the driven neuron in panel (a) responds in one condition but not the other, we have one constrained dimension (magenta axis) and one semiconstrained dimension (green axis). The yellow ray depicts the space of weights, (w₁, w₂), that generate the stimulus transformation. The weight vector $\left(\frac{1}{2},\frac{1}{2}\right)$ (brown dot) would uniquely generate the neural responses in a linear network. We assume that the magnitude of the weight vector is bounded by W, such that all candidate weight vectors lie within a circle of that radius. A nonzero synapse x₂ → y exists in all solutions, but the x₁ → y synapse can be zero because the yellow ray intersects the w₁ = 0 axis.

How does the nonlinearity change the solution space of weights that reproduce the driven neuron responses? To answer this question, we define two linear combinations of weights,

$${\eta }_{\pm }=w_1\pm w_2,$$ (3)

which correspond to the driven neuron’s input drive in patterns μ = ±. Equation (1) now yields rather simple algebraic constraints for the two patterns:

$$y_{+}=1=\Phi \left({\eta }_{+}\right)\Rightarrow {\eta }_{+}=1,$$ (4)

$$y_{-}=0=\Phi \left({\eta }_{-}\right)\Rightarrow {\eta }_{-}⩽0.$$ (5)

Note that η₋ would have had to be zero if Φ were linear, but because the threshold-linear transfer function turns everything negative into a null response, η₋ can now also be any negative number. However, sufficiently negative values of η₋ correspond to implausibly large weight vectors, and hence we focus on solutions with norm bounded above by some value,W. The nonlinearity thus turns the unique linear solution [brown dot in Fig. 2(b)] into a continuum of solutions [yellow line segment in Fig. 2(b)]. This continuum lies along what we will refer to as a semiconstrained dimension. Indeed, this will turn out to be a generic feature of threshold-linear neural networks: every time there is a null response, a semiconstrained dimension emerges in the solution space.¹

Although we found infinitely many weight vectors that solve the problem, all solutions to the problem have a synaptic connection x₂ → y, and this connection is always excitatory [Fig. 2(b)]. Positive, negative, or zero connection weights are all possible for x₁ → y. However, this reveals why the value of the synaptic weight bound, W, has important implications for the solution space. For example, all solutions in Fig. 2(b) with $|\overrightarrow{w}|<1$ have w₁ > 0, whereas larger magnitude weight vectors have w₁ ≤ 0. Therefore, one would be certain that an excitatory x₁ → y synapse exists if the weight bound were biologically known to be less than W_cr = 1. We refer to this weight bound as W-critical. Looser weight bounds raise the possibility that the synapse is absent or inhibitory. Note that too tight weight bounds, here less than $W_{\text{min}}=1/\sqrt{2}$, can exclude all solutions.

The example of Fig. 2 concretely illustrates the general procedure diagramed in Fig. 1. First, we specified a network architecture and steady-state response patterns [Figs. 1(a) and 2(a)]. Second, we found all synaptic weight vectors that can implement the nonlinear transformation [Figs. 1(b) and 2(b)]. Finally, we determined whether individual synaptic weights varied in sign across the solution space [Figs. 1(c) and 2(b)]. Section III will generalize the first two parts of this procedure to characterize the solution space of any threshold-linear recurrent neural network, assuming that the number of response patterns is at most the dimensionality of the weight vectors. Sections IV and V will then generalize the final part of this procedure to pinpoint synaptic connections that are critical for generating any specified set of orthonormal responses.

III. SOLUTION SPACE GEOMETRY

A. Neural network structure and dynamics

Consider a neural network of ℐ input neurons that send signals to a recurrently connected population of 𝒟 driven neurons [Fig. 3(a)]. We compactly represent the network connectivity with a matrix of synaptic weights, w_im, where i = 1, …, 𝒟 indexes the driven neurons, and m = 1, …, 𝒟 + ℐ indexes presynaptic neurons from both the driven and input populations. We suppose that activity in the population of driven neurons dynamically evolves according to

$${\tau }_i\frac{d{y}_i}{dt}=-y_i+\Phi \left(\sum _{m=1}^{𝒟}{w}_{im}{y}_m+\sum _{m=𝒟+1}^{𝒟+ℐ}{w}_{im}{x}_{m-𝒟}\right),$$ (6)

where y_i is the firing rate of the ith driven neuron, x_m is the firing rate of the mth input neuron, and τ_i is the time constant that determines how long the ith driven neuron integrates its presynaptic signals. It is possible that prior biological knowledge dictates that certain synapses appearing in Eq. (6) are absent. For notational convenience, in this paper we will assume that the number of synapses onto each driven neuron remains the same,² and we will denote this number of the incoming synapses as 𝒩. Note that 𝒩 = ℐ + 𝒟 for a general recurrent network, 𝒩 = ℐ + 𝒟 − 1 for recurrent networks without self-synapses, and 𝒩 = ℐ for feedforward networks. We suppose that the network functionally maps input patterns, x_μm, to steady-state driven signals, y_μi ⩾ 0, where μ = 1, …, 𝒫 labels the patterns [Fig. 3(b)]. We assume throughout that 𝒫 ⩽ 𝒩, as the number of known response patterns is typically small, and the number of possible synaptic inputs is large. Experimentally, different response patterns often correspond to different stimulus conditions, so we will often refer to μ as a stimulus index and x_μm → y_μi as a stimulus transformation.

FIG. 3.

Finding network structure that implements functional responses. (a) Cartoon depicting a recurrent network of driven neurons (blue) receiving feedforward input from a population of input neurons (orange). (b) The μth pattern of input neuron activity (x_μm) appears at t = 0 and drives the recurrent neurons to approach the steady-state response pattern (y_μi) via feedforward and recurrent network connectivity (w_im). (c) (Left) We focus on one driven neuron at a time, referred to henceforth as the target neuron, to determine its possible incoming synaptic weights, w_m. (Right) These weights must reproduce the target neuron’s 𝒫 steady-state responses from the steady-state activity patterns of all 𝒩 presynaptic neurons. (d) The yellow planes depict the subspace of incoming weights that can exactly reproduce all nonzero responses of the target neuron, and the subregion shaded dark yellow indicates weights that also reproduce the target neuron’s zero responses. The top graph depicts the weight space parametrized by physically meaningful w coordinates, but the solution space is more simply parametrized by abstract η coordinates (bottom). The η coordinates depend on the specified stimulus transformation (x_μm → y_μi), and η_c, η_s, and η_u are coordinates in 𝒞-dimensional constrained, 𝒮-dimensional semiconstrained, and 𝒰-dimensional unconstrained subspaces, respectively.

B. Decomposing a recurrent network into 𝒟 feedforward networks

Our goal is to find features of the synaptic weight matrix that are required for the stimulus transformation discussed above. For notational simplicity, let us consider the case where we potentially have all-to-all connectivity, so that 𝒩 = 𝒟 + ℐ, but we will later explain how our arguments generalize. Since all time-derivatives are zero at steady-state, the response properties provide 𝒟 × 𝒫 nonlinear equations for 𝒟 × 𝒩 unknown parameters³:

$$y_{\mu i}=\Phi \left(\sum _{m=1}^{𝒟}{w}_{im}{y}_{\mu m}+\sum _{m=𝒟+1}^{𝒟+ℐ}{w}_{im}{x}_{\mu ,m-𝒟}\right).$$ (7)

Inspection of the above equation, however, reveals that each neuron’s steady-state activity depends only on a single row of the connectivity matrix [Fig. 3(c)]; the responses of the ith driven neuron, {y_μi, μ = 1, …, P}, are only affected by its incoming synaptic weights, {w_im, m = 1, …, 𝒩}. Thus, the above equations separate into 𝒟 independent sets of equations, one for each driven neuron. In other words, we now have to solve 𝒟 feedforward problems, each of which will characterize the incoming synaptic weights of a particular driven neuron, which we term the target neuron. Note that since a generic target neuron receives signals from both the input and the driven populations, the activities of both input and driven neurons serve to produce the presynaptic input patterns that drive the responses of the target neuron in the reduced feedforward problem.

C. Solution space for feedforward networks

We have just seen how we can solve the problem of finding synaptic weights consistent with steady-state responses of a recurrent population of neurons, provided we know how to solve the equivalent problem for feedforward networks. Accordingly, we will now focus on a feedforward network, where a single target neuron, y, receives inputs from 𝒩 neurons {x_m; m = 1, …, 𝒩}, to find the ensemble of synaptic weights that reproduce this target neuron’s observed responses. The constraint equations are

$$y_{\mu }=\Phi \left(\sum _{m=1}^{𝒩}{x}_{\mu m}{w}_m\right),$$ (8)

where y_μ now stands for the activity of the target neuron driven by the μth input pattern, and $\overrightarrow{w}$ is the 𝒩-vector of synaptic weights onto the target neuron. Assuming that the 𝒫 × 𝒩 matrix x is rank P, we let the 𝒩 × 𝒩 matrix X be rank 𝒩 with X_μm = x_μm for μ = 1, …, 𝒫. This implies that the last 𝒩 − 𝒫 rows of X span the null space of x, and X defines a basis transformation on the weight space,

$${\eta }_{\mu }=\sum _{m=1}^{𝒩}{X}_{\mu m}{w}_m\iff w_m=\sum _{\mu =1}^{𝒩}{X}_{m\mu }^{-1}{\eta }_{\mu }.$$ (9)

The 𝒩 linearly independent columns of X⁻¹ define the basis vectors corresponding to the η coordinates,

$$X^{-1}=\left(\begin{array}{ccccc}{\overrightarrow{\epsilon }}_1& \cdots & {\overrightarrow{\epsilon }}_{\mu }& \cdots & {\overrightarrow{\epsilon }}_{𝒩}\end{array}\right).$$ (10)

In other words,

$${\overrightarrow{\epsilon }}_{\mu }=\sum _{m=1}^{𝒩}{\widehat{e}}_m{X}_{m\mu }^{-1},$$ (11)

where $\left\{{\widehat{e}}_m\right\}$ is the physical orthonormal basis whose coordinates, {w_m}, correspond to the material substrates of network connectivity. These basis vectors can be obtained from $\left\{{\overrightarrow{\epsilon }}_{\mu }\right\}$ by an inverse basis transformation:

$${\widehat{e}}_m=\sum _{\mu =1}^{𝒩}{\overrightarrow{\epsilon }}_{\mu }{X}_{\mu m}.$$ (12)

We can thus write any vector of incoming weights as

$$\overrightarrow{w}=\sum _{m=1}^{𝒩}{w}_m{\widehat{e}}_m=\sum _{\mu =1}^{𝒩}{\eta }_{\mu }{\overrightarrow{\epsilon }}_{\mu }.$$ (13)

In terms of η coordinates, the nonlinear constraint equations take a rather simple form:

$$y_{\mu }=\Phi \left({\eta }_{\mu }\right)\hspace{1em}\text{for}\mu =1,\cdots ,𝒫.$$ (14)

Accordingly, η coordinates succinctly parametrize the solution space of all weight matrices that support the specified fixed points [Fig. 3(d)]. Each η dimension can be neatly categorized into one of three types. First, for each stimulus condition μ where y_μ > 0, we must have η_μ > 0. This in turn implies that Φ(η_μ) = η_μ = y_μ. Because the coordinate η_μ must adopt a specific value to generate the transformation, we say that μ defines a constrained dimension. We denote the number of constrained dimensions as 𝒞 ⩽ 𝒫. Second, note that the threshold in the transfer function implies that Φ(a) = 0 for all a ⩽ 0. Therefore, for any stimulus condition such that y_μ = 0, we have a solution whenever η_μ 0. Because positive values of η_μ are excluded but all negative values are equally consistent with the transformation, we say that μ defines a semiconstrained dimension. We denote the number of semiconstrained dimensions as 𝒮 = 𝒫 – 𝒞. Finally, we have no constraint equations for η_μ if μ = 𝒫 + 1,···, 𝒩. Because all positive or negative values of η_μ are equally consistent with the stimulus transformation, we say that μ defines an unconstrained dimension. We denote the number of unconstrained dimensions as 𝒰 = 𝒩 − 𝒫. Altogether, the stimulus transformation is consistent with every incoming weight vector that satisfies

$$\begin{array}{cc}\hfill {\eta }_{\mu }=y_{\mu }\hfill & \text{if}{y}_{\mu }>0,\mu ⩽𝒫\\ -\infty <{\eta }_{\mu }⩽0& \hfill \text{if}{y}_{\mu }=0,\mu ⩽𝒫\hfill \\ -\infty <{\eta }_{\mu }<\infty & \text{if}\mu >𝒫.\hfill \end{array}$$ (15)

Note that one can enumerate the solutions in the physically meaningful w coordinates by simply applying the inverse basis transformation in Eq. (9) to any solution found in η coordinates.

Going forward, it will be convenient to extend the 𝒫-dimensional vector of target neuron activity to an 𝒩-dimensional vector whose components along the unconstrained dimensions are equal to zero, because this will allow us to compactly write equations in terms of dot products between the activity vector and vectors in the 𝒩-dimensional weight space. Rather than introducing a new notation for this extended 𝒩-dimensional vector, we simply write $\overrightarrow{y}$ with y_μ = 0 for μ = 𝒫 + 1, …, 𝒩. It is critical to remember that this is merely a notational convenience, and the solution space distinguishes between semiconstrained dimensions and unconstrained dimensions according to Eq. (15). In particular, y_μ = 0 is a constraint equation for semiconstrained dimensions, but y_μ = 0 is a notational convenience for unconstrained dimensions.

D. Back to the recurrent network

To understand how the solution space geometry of the feedforward network can be translated back to the recurrent network, it is useful to group together the steady-state activities of all input and driven neurons that are presynaptic to the ith driven neuron as a 𝒫 × 𝒩 input pattern matrix, z⁽ⁱ⁾.⁴ The entries of the matrix, $z_{\mu m}^{(i)}$, correspond to the responses of the mth presynaptic neuron to the μth stimulus. At this point it is easy to see that when biological constraints dictate that some of the synapses are absent, then one should just exclude those presynaptic neurons when constructing z⁽ⁱ⁾, such that the m index excludes those presynaptic neurons. Similarly, by a suitable reordering, which will depend on the driven neuron, we can always ensure that m = 1, …, 𝒩 runs only over the neurons that are presynaptic to the given driven neuron.

Once the input patterns feeding into the ith neuron are known, we can follow the steps outlined in the previous subsection to define the 𝒩 × 𝒩 full rank extension of z⁽ⁱ⁾, Z⁽ⁱ⁾, and the η⁽ⁱ⁾ coordinates via

$${\eta }_{\mu }^{(i)}=\sum _{m=1}^{𝒩}{Z}_{\mu m}^{(i)}{w}_{im}.$$ (16)

The nature of the ${\eta }_{\mu }^{(i)}$ coordinates, that is whether they are constrained, semiconstrained, or unconstrained, is determined by how the ith neuron responded to the stimulus conditions, as in Eq. (15). Repeating this process for all driven neurons provides a geometric characterization of the entire recurrent network solution space, which involves all elements of the synaptic weight matrix, w_im.

An important special case is all-to-all network connectivity. In this case, the Z⁽ⁱ⁾ matrices are the same for all driven neurons, and therefore the directions corresponding to the η coordinates are also preserved.⁵ In particular, the orientation of the unconstrained subspace with respect to the physical basis does not change from one driven neuron to another. However, how a given driven neuron responds to a particular stimulus determines whether the corresponding η direction is going to be constrained or semiconstrained for the feedforward network associated with that driven neuron.

IV. CERTAIN SYNAPSES IN ILLUSTRATIVE 3D EXAMPLES

Although we have found infinitely many weight matrices that produce a given stimulus transformation, it is nevertheless possible that the solutions imply firm anatomical constraints (e.g., Sec. II). In this paper we focus on finding synapses that must be nonzero in order for the response patterns to be fixed points of the neural network dynamics. We refer to such synapses as certain, because the synapse must exist in the model, and its sign is identifiable from the response patterns. It is clear from the geometry of the solution space that the relative orientations between the η coordinates and the physical w coordinates are significant determinants of synapse certainty. To build quantitative intuition for how the solution space geometry precisely determines synapse certainty, we begin by first analyzing a few illustrative toy problems. In the next section we will describe the more general treatment of high-dimensional networks. Importantly, we select and parametrize each toy problem to introduce concepts and notations that will reappear in the general solution.

More specifically, we first consider three feedforward examples with 𝒩 = 3 [Fig. 4(a)]. The first two examples have 𝒫 = 3, and the third has 𝒫 = 2. In the first example, we will assume that the driven neuron does not respond to the first two stimulus patterns, but responds positively to the third pattern. So we have two semiconstrained and one constrained dimension,

$$\begin{array}{cccc}{\eta }_1⩽0,& {\eta }_2⩽0,& \text{and}& {\eta }_3=y_3>0.\end{array}$$ (17)

In contrast, in the second example we will have two constrained and one semiconstrained dimension,

$$\begin{array}{cccc}{\eta }_1=y_1>0,& {\eta }_2=y_2>0,& \text{and}& {\eta }_3⩽0.\end{array}$$ (18)

The final example will feature one unconstrained, one semiconstrained, and one constrained dimension,

$$\begin{array}{cccc}{\eta }_1⩽0,& {\eta }_2=y_2>0,& \text{and}& -\infty <{\eta }_3<\infty .\end{array}$$ (19)

For technical simplicity we will consider orthonormal input patterns, X⁻¹ = X^T , which implies that

$$\sum _{m=1}^{𝒩}{X}_{\mu m}{X}_{vm}={\delta }_{\mu v}={\overrightarrow{\epsilon }}_{\mu }\cdot {\overrightarrow{\epsilon }}_v,$$ (20)

where δ_μν is the Kronecker δ function, which equals 1 if μ = ν and 0 if μ ≠ ν, so ${\widehat{\epsilon }}_{\mu }={\overrightarrow{\epsilon }}_{\mu }$. This trivially implies that the η coordinates are related to the synaptic coordinates via a rotation, so the spherical biological bound on the physical coordinates transforms to an identical spherical bound on the η coordinates:

$$\sum _{\mu =1}^{𝒩=3}{\eta }_{\mu }^2=\sum _{m=1}^{𝒩=3}{w}_m^2⩽W^2.$$ (21)

FIG. 4.

Geometric quantities determining whether neurons must be synaptically connected in several three-dimensional toy problems. (a) Cartoon depicting the 𝒩 = 3 feedforward network corresponding to the toy problems. (b), (c) Geometrically determining whether a synapse is nonzero when the target neuron responds to one input pattern but does not to two other patterns. A synapse can only vanish if the w₁ = 0 plane (orange circle) intersects the solution space (dark yellow wedge) within the weight bounds (bounding sphere). For example, this intersection occurs in panel (b), so the synapse is not required for the responses. For every synapse one can associate a direction in synaptic weight space (orange arrow) that is normal to the planes with constant synaptic weight. This synapse vector can be decomposed into its projections into the semiconstrained subspace (green arrow, $\overrightarrow{s}$) and along the constrained dimension (pink arrow, $\overrightarrow{c}$). In this example, whether the synapse is certain is determined by the size of the bounding synapse space, W [see panel (b)], the angle θ between the synapse direction (orange arrow) and the closest axis of the constrained dimension $\left(-{\overrightarrow{\epsilon }}_3\right)$ [see panel (b)], and the angle γ between $\overrightarrow{s}$ and its closest vector in the solution space $\left({\overrightarrow{s}}_{\ast }\right)$ [see panel (c)]. In panel (c), d_s depicts the perpendicular distance from the origin of the yellow semiconstrained plane in panel (b) to its intersection line with the w₁ = 0 orange plane. If this distance is sufficiently large, then the orange line will not intersect the target neuron responds to two input patterns but not the third pattern. In panel (d), the orange w₁ = 0 plane intersects the solution space (deep solution space within the yellow plane’s circular bound of radius $\tilde{W}$. (d), (e) Geometrically determining whether a synapse is nonzero when the yellow line) within the bounding sphere, so the synapse is not certain. In this example, the factors that determine synapse certainty are W [see panel (d)], the angle θ that the synapse vector (orange arrow) makes with its projection along the constrained subspace (pink arrow) [see panel (d)], and the angle α between the target response vector (brown arrow) and the pink arrow [see panel (e)]. The angle β does not ultimately matter, but it is included in the diagrams to aid the derivation. Here d_s is the distance from the brown dot to the point of intersection between the yellow line and the orange plane. Again this point will lie outside the bounding sphere if d_s is large enough, and this signals a certain synapse. (f) Geometrically determining whether a synapse is nonzero when the target neuron responds to one input pattern but does not to a second pattern. In the figure shown, the w₁ = 0 orange plane intersects the solution space (deep yellow semicircle) within the bounding sphere, so the synapse is not certain. In this example, apart from W, what determines synapse certainty are the angles θ and φ, which encode how the synapse vector (orange arrow) can be decomposed into its projections along the constrained direction (pink arrow), semiconstrained direction (green arrow) and unconstrained direction (purple arrow).

A. Problem 1

Let us first focus on the example with two semiconstrained and one constrained dimension, whose solution space is depicted in deep yellow in Fig. 4(b). Suppose we are interested in assessing whether the w₁ synapse is certain. Since the w₁ = 0 plane divides the weight space into the positive and the negative halves, the synapse will be certain if this plane does not intersect with the solution space, which clearly depends on the orientation of the plane relative to the various η directions [Fig. 4(b)]. It is thus useful to consider how the w₁ = 0 plane’s unit normal vector pointing toward positive weights, $\widehat{e}\equiv {\widehat{e}}_1$, is oriented relative to the η directions. For ease of graphical illustration, here we assume the specific orientation diagramed in Figs. 4(b) and 4(c). Using Eq. (12) and the orthogonality of X, we can parametrize $\widehat{e}$ as

$$\widehat{e}=\sum _{\mu =1}^{𝒩=3}{X}_{\mu 1}{\widehat{\epsilon }}_{\mu }=\text{cos}\theta \widehat{c}+\text{sin}\theta \widehat{s},$$ (22)

where

$$\widehat{c}=-{\widehat{\epsilon }}_3,\hspace{1em}\text{and}\hspace{1em}\widehat{s}=-\text{cos}\gamma {\widehat{\epsilon }}_1+\text{sin}\gamma {\widehat{\epsilon }}_2$$ (23)

[Figs. 4(b) and 4(c)]. Geometrically, $\widehat{c}$ and $\widehat{s}$ are unit vectors along the projections of $\widehat{e}$ onto the constrained and semiconstrained subspaces [Fig. 4(b)]. Thus, cos θ ⩾ 0 and sin θ ⩾ 0, making θ an acute angle. In this example, γ is also an acute angle, as depicted in Fig. 4(c).

Note that all solutions lie within the two-dimensional semiconstrained subspace having η₃ = y₃. The w₁ = 0 plane intersects this semiconstrained subspace as a line [Figs. 4(b) and 4(c)], and its equation in η coordinates is

$$w_1=\widehat{e}\cdot \overrightarrow{w}=\text{sin}\theta \left(-\text{cos}\gamma {\eta }_1+\text{sin}\gamma {\eta }_2\right)-\text{cos}\theta y_3=0.$$ (24)

From the geometry of the problem [Fig. 4(c)], it is clear that if the perpendicular distance, d_s, from the origin to this line is large enough, then it will not intersect the all-negative quadrant of the semiconstrained subspace within the weight bound. According to simple trigonometry, this occurs when

$$d_s>\tilde{W}\text{cos}\gamma =\sqrt{W^2-y_3^2}\text{cos}\gamma ,$$ (25)

where $\tilde{W}=\sqrt{W^2-y_3^2}$ is the radius of the semiconstrained subspace containing the solutions. The perpendicular distance can be identified from Eq. (24) as

$$d_s=y_3\text{cot}\theta .$$ (26)

Substituting this expression for d_s into Eq. (25), one finds through simple algebra that the w₁ = 0 hyperplane does not intersect the solution space, and hence the synapse is certain, if the response magnitude exceeds a critical value,

$$y_3>y_{\text{cr}}=W\sqrt{\frac{{\text{sin}}^2\theta {\text{\hspace{0.17em}cos}}^2\gamma }{{\text{cos}}^2\theta +{\text{sin}}^2\theta {\text{\hspace{0.17em}cos}}^2\gamma }},$$ (27)

which we generally refer to as y-critical.

Notice that if θ increases in Fig. 4(b), then the orange line in Fig. 4(c) comes closer to the origin, making it intersect with the solution space for more γ angles. Therefore, the synapse is more difficult to identify, and indeed Eq. (27) shows that y_cr increases. However, if γ increases, then the orange line in Fig. 4(c) rotates away from the solution space, making the synapse easier to identify with small d_s. Accordingly, y_cr decreases.

It will turn out that the concept of y-critical is general, and y_cr can always be expressed in terms of projections of ê along several specific directions. In this example, if we define e_s* and e_y to be projections of $\widehat{e}$ along, ${\widehat{s}}_{\ast }=-\overrightarrow{\epsilon }$ and $\widehat{y}={\overrightarrow{\epsilon }}_3$ respectively, then it is easy to check that one can re-express y_cr as

$$y_{\text{cr}}=W\sqrt{\frac{e_{s\ast }^2}{e_y^2+e_{s\ast }^2}}.$$ (28)

We will later discover that these projections are closely related to correlations between pre-synaptic and postsynaptic neuronal activity patterns. Thus, the expressions in Eq. (28) will provide a deeper understanding of the determinants of synapse certainty.

B. Problem 2

Having identified two key angles, θ and γ, that play a role in synapse certainty, let us look at the example of two constrained and one semiconstrained dimensions to uncover other important geometric quantities. In this case, the solution space is a ray defined by η₁ = y₁, η₂ = y₂, and −∞ < η₃ 0, and the magnitude of η₃ is at most

$$\tilde{W}=\sqrt{W^2-y_1^2-y_2^2}$$ (29)

for solutions within the weight bound [Fig. 4(d)]. Figure 4(d) shows a geometry where the w₁ = 0 plane intersects the solution space at the point

$${\overrightarrow{w}}_{\text{int}}=y_1{\overrightarrow{\epsilon }}_1+y_2{\overrightarrow{\epsilon }}_2+{\eta }_3{\overrightarrow{\epsilon }}_3.$$ (30)

Now we must have

$$\widehat{e}\cdot {\overrightarrow{w}}_{\text{int}}=0,$$ (31)

as the intersection point lies on the w₁ = 0 plane by definition, where we have defined $\widehat{e}\equiv {\widehat{e}}_1$ as in the previous toy problem. The projection directions of $\widehat{e}$ onto the constrained and semiconstrained subspaces are given by

$$\widehat{c}=\text{cos}\beta {\widehat{\epsilon }}_1+\text{sin}\beta {\widehat{\epsilon }}_2,\hspace{1em}\text{and}\hspace{1em}\widehat{s}={\widehat{\epsilon }}_3,$$ (32)

[Fig. 4(d)]. Then combining Eqs. (22) and (32), we can find an equation to determine η₃ at the intersection point

$$\widehat{e}\cdot {\overrightarrow{w}}_{\text{int}}=\text{cos}\theta \left(y_1\text{cos}\beta +y_2\text{sin}\beta \right)+{\eta }_3\text{sin}\theta =0.$$ (33)

We next introduce α to represent the angle between $\widehat{c}$ and $\overrightarrow{y}$ [Fig. 4(e)], such that

$$y_1=y\text{cos}(\beta -\alpha )\hspace{1em}\text{and}\hspace{1em}{y}_2=y\text{\hspace{0.17em}sin}(\beta -\alpha ),$$ (34)

where $y=\left|\overrightarrow{y}\right|$. The first two terms in Eq. (33) can then be trigonometrically combined with a difference of angles identity to arrive at

$$y\text{cos}\theta \text{cos}\alpha +{\eta }_3\text{sin}\theta =0\Rightarrow {\eta }_3=-y\text{cot}\theta \text{cos}\alpha .$$ (35)

To be able to identify the sign of w₁, this intersection point must lie beyond the weight bounds of the solution line segment, so ${\eta }_3<-\tilde{W}$. After some straightforward algebra we obtain the certainty condition as

$$y>y_{\text{cr}}\equiv W\sqrt{\frac{{\text{sin}}^2\theta }{{\text{cos}}^2\theta {\text{cos}}^2\alpha +{\text{sin}}^2\theta }}.$$ (36)

From the geometry of the problem in Figs. 4(d) and 4(e), one sees that as θ or α increases, the point where the orange hyperplane intersects the yellow line is closer to the origin. Indeed, y_cr increases, making it more difficult to identify the synapse sign. Again, one can re-express y_cr as Eq. (28) in terms of projections, with the role of ${\widehat{s}}_{\ast }$ being played by ${\overrightarrow{\epsilon }}_3$.

C. Problem 3

Through the two above examples we found three angles, θ,α, and γ, that determine how large the response of the driven neuron has to be in order for a given synapse to be certain. However in both examples the number of patterns were equal to the number of synapses, 𝒫 = 𝒩. When 𝒫 < 𝒩, we have unconstrained dimensions, and the projection of the $\widehat{e}\equiv {\widehat{e}}_1$ vector into the unconstrained subspace will also matter, because it relates to how much we do not know about the response properties of the driven neuron.

Here we consider a 𝒩 = 3 example with one constrained, one semiconstrained, and one unconstrained dimension rection as a linear combination of its projections along the [Fig. 4(f)]. In this case, we can express the $\widehat{e}$ synaptic diconstrained, semiconstrained and unconstrained dimension as

$$\widehat{e}=\sum _{\mu =1}^{𝒩}{X}_{\mu 1}{\overrightarrow{\epsilon }}_{\mu }=\text{cos}\theta \widehat{c}+\text{sin}\theta \text{cos}\varphi \widehat{s}+\text{sin}\theta \text{sin}\varphi \widehat{u},$$ (37)

where we can always choose the directions of the unit vectors to make θ and φ acute angles. For the example shown in Fig. 4(f), this is achieved by choosing

$$\widehat{c}={\overrightarrow{\epsilon }}_2,\widehat{s}=-{\overrightarrow{\epsilon }}_1,\hspace{1em}\text{and}\hspace{1em}\widehat{u}={\overrightarrow{\epsilon }}_3.$$ (38)

Obtaining the certainty condition again involves ascertaining whether the w₁ = 0 hyperplane intersects the deep yellow solution space [Fig. 4(f)]. In the example of Fig. 4(f), one can see that increasing the driven neuron response moves the yellow plane up, and there will come a critical point when the orange w₁ = 0 plane just touches the solution space at the corner (η₁ = 0, η₂ = y_cr, η₃). Thus,

$${\overrightarrow{w}}_{\text{int}}=y_{\text{cr}}{\overrightarrow{\epsilon }}_2+{\eta }_3{\overrightarrow{\epsilon }}_3.$$ (39)

Since this corner point has a negative η₃ component and lies on the bounding sphere, we must also have

$${\eta }_3=-\sqrt{W^2-y_{\text{cr}}^2},$$ (40)

[Fig. 4(f)]. Substituting ${\overrightarrow{w}}_{\text{int}}$ in the w₁ = 0 plane equation,

$$\widehat{e}\cdot {\overrightarrow{w}}_{\text{int}}=-\text{sin}\theta \text{sin}\varphi \sqrt{W^2-y_{\text{cr}}^2}+\text{cos}\theta y_{\text{cr}}=0,$$ (41)

we can then determine y_cr through simple algebra as

$$y_{\text{cr}}=W\sqrt{\frac{{\text{sin}}^2\theta {\text{sin}}^2\varphi }{{\text{cos}}^2\theta +{\text{sin}}^2\theta {\text{sin}}^2\varphi }}.$$ (42)

The final result now depends on the two acute orientation angles, θ and φ. By inspection of Fig. 4(f) or Eq. (42), it is clear that y_cr increases if either θ or φ increases toward π/2. One therefore needs a larger response (y₃) to make the synapse certain. We can again express y_cr in terms of projections

$$y_{\text{cr}}=W\sqrt{\frac{e_u^2}{e_y^2+e_u^2}},$$ (43)

where e_u is the projection of $\widehat{e}$ along $\widehat{u}$, and e_s* does not appear because the intersection occurred at the origin of the semiconstrained subspace.

V. CERTAIN SYNAPSES, THE GENERAL TREATMENT

A. High-dimensional feedforward networks

We have seen in the previous section how geometric considerations can identify synapses that must be present to generate observed response patterns in small networks. One can similarly ask when a synapse is required in high-dimensional networks [Fig. 5(a)].

FIG. 5.

Identifying certain synapses in high-dimensional networks. (a) Cartoon depicting the high-dimensional feedforward network under consideration. (b) Geometrically determining whether a synapse is nonzero throughout a high-dimensional solution space. A synapse can only vanish if the w_m = 0 hyperplane (orange circle) intersects the solution space (dark yellow wedge) within the weight bounds (bounding sphere). In the example shown, this intersection does not occur, so the synapse must be present. For orthonormal neural responses, only a few parameters determine whether this intersection occurs (Appendix A). First, the magnitude of the weight bound, W, controls the extent of the solution space. Second, there are three projections of the synapse direction (orange arrow) whose lengths are important determinants of the certainty condition: e_y, the length of projection along the target response vector (pink arrow); e_s*, the length of projection along the closest boundary vector in the semiconstrained solution subspace [green arrow, see also ${\overrightarrow{s}}_{\ast }$ in Fig. 4(c)]; and e_u, the length of projection into the unconstrained subspace (purple arrow). Note that the shown example would have had an intersection if the solution space (dark yellow wedge) were moved down (along $\widehat{c}$) to lie below the hyperplane (orange circle). The solution space’s height is proportional to the magnitude of the postsynaptic responses, y. Thus, the solution space does not intersect the hyperplane only if y exceeds a critical value, y_cr. (c) Plots of the certainty condition, Eq. (57), for W = 1. The red, blue, and purple curves plot y_cr as a function of r_y = e_y/e_p for e_p = 0.3, 0.6, and 0.9, respectively. Different purple shades correspond to different values of $r_{s\ast }=e_{s\ast }/\sqrt{e_p^2-e_y^2}$. As this ratio increases, nonlinear effects increase y_cr and make the sign harder to determine. The red and blue curves are for the maximally nonlinear case when $r_{s\ast }=1\Rightarrow e_{s\ast }=\sqrt{e_p^2-e_y^2}$. The dashed black curves represent y_cr in a linear model, which cannot exceed the nonlinear y_cr.

Although the rigorous derivation is intricate, this certainty condition is remarkably simple for orthonormal X (Appendix A). Quantitatively, orthonormal X imply that only a few parameters matter for the certainty condition, each illustrated in the previous section and abstractly summarized in Fig. 5(b).

For any given synapse, its physical basis vector, ${\widehat{e}}_m$, can always be written as a sum of components in the constrained, semiconstrained, and unconstrained subspaces,

$${\widehat{e}}_m=\sum _{\mu =1}^{𝒩}{X}_{\mu m}{\overrightarrow{\epsilon }}_{\mu }={\overrightarrow{c}}_m+{\overrightarrow{s}}_m+{\overrightarrow{u}}_m,$$ (44)

where ${\overrightarrow{c}}_m$, ${\overrightarrow{s}}_m$ and ${\overrightarrow{u}}_m$ denote the partial sums over μ in the constrained, semiconstrained, and unconstrained subspaces, respectively. Note that $\left\{{\overrightarrow{\epsilon }}_{\mu }\right\}$ are orthogonal unit vectors if and only if X is an orthogonal matrix. In this case, the decomposition of ${\widehat{e}}_m$ is a sum of three orthogonal vectors that can be parameterized by two angles,

$${\widehat{e}}_m=\text{cos}\theta {\widehat{c}}_m+\text{sin}\theta \text{cos}\varphi {\widehat{s}}_m+\text{sin}\theta \text{sin}\varphi {\widehat{u}}_m,$$ (45)

where ${\widehat{c}}_m$, ${\widehat{s}}_m$, and ${\widehat{u}}_m$ are unit vectors in the constrained, semiconstrained, and unconstrained subspaces, and (θ, ϕ) are spherical coordinates⁶ specifying the orientation of ${\widehat{e}}_m$ with respect to these subspaces [e.g., Fig. 4(f)]. In particular,

$$\begin{gathered} \text{cos}\theta =\sqrt{\sum _{\left\{\mu \mid y_{\mu }>0\right\}}^{𝒫}{X}_{\mu m}^2,} \\ \text{sin}\theta \text{cos}\varphi =\sqrt{\sum _{\left\{\mu \mid y_{\mu }=0\right\}}^{𝒫}{X}_{\mu m}^2,} \\ \text{sin}\theta \text{sin}\varphi =\sqrt{\sum _{\mu =𝒫+1}^{𝒩}{X}_{\mu m}^2}. \end{gathered}$$ (46)

As we have seen in the toy examples, these two orientation angles heavily influence whether the synapse is certain.

Additionally, because the solution space’s height along $\widehat{c_m}$ [e.g., Fig. 4(b)] is controlled = by the angle between $\widehat{c_m}$ and $\widehat{y}$, the equation for the w_m = 0 hyperplane that divides the positive and negative synaptic regions in the solution space depends on

$$\overrightarrow{y}\cdot {\widehat{c}}_m=y\text{cos}\alpha ,$$ (47)

where y is the length of $\overrightarrow{y}$ and α is the angle between $\overrightarrow{y}$ and $\widehat{c_m}$ [Fig. 4(e)]. Finally, there is another critical angle, which we call γ, that encodes how ${\widehat{s}}_m$ is oriented with respect to the solution space in the semiconstrained subspace. Using a more convenient direction, ${\widehat{s}}_m^{\prime }\equiv -\text{Sgn}(\text{cos}\alpha ){\widehat{s}}_m$, which is either along or opposite to the ${\widehat{s}}_m$ direction, we define γ to be the minimal angle between ${\widehat{s}}_m^{\prime }$ and the solution space [e.g., Fig. 4(b)]. It is generally given by

$$\text{cos}\gamma =\sqrt{\sum _{\left\{\mu \mid {\widehat{s}}_{\mu }^{\prime }<0\right\}}{\widehat{s}}_{\mu }^{\prime 2}}$$ (48)

(Appendix A), where ${\widehat{s}}_{\mu }^{\prime }$ is the μth component of ${\widehat{s}}_m^{\prime }$, and we have suppressed m to avoid cluttered notation. Although this definition and equation for γ may initially appear opaque, we soon clarify its meaning in terms of interpretable projections of the synapse vector.

Putting all the pieces together, we find that the mth synapse must be present, and its sign is unambiguous, if and only if y exceeds the critical value

$$y_{\text{cr}}=W\sqrt{\frac{{\text{cos}}^2\gamma {\text{sin}}^2\theta {\text{cos}}^2\varphi +{\text{sin}}^2\theta {\text{sin}}^2\varphi }{{\text{cos}}^2\alpha {\text{cos}}^2\theta +{\text{cos}}^2\gamma {\text{sin}}^2\theta {\text{cos}}^2\varphi +{\text{sin}}^2\theta {\text{sin}}^2\varphi }}$$ (49)

(Appendix A). Intuitively, W bounds the magnitude of weight vectors, and largeW increase y_cr by admitting more solutions. Note that a synapse is certain, for a given y, when the weight bound is less than a critical value,

$$W_{\text{cr}}=y\sqrt{\frac{{\text{cos}}^2\alpha {\text{cos}}^2\theta +{\text{cos}}^2\gamma {\text{sin}}^2\theta {\text{cos}}^2\varphi +{\text{sin}}^2\theta {\text{sin}}^2\varphi }{{\text{cos}}^2\gamma {\text{sin}}^2\theta {\text{cos}}^2\varphi +{\text{sin}}^2\theta {\text{sin}}^2\varphi }}.$$ (50)

Finally, we note that we must have W ⩾ y for any solutions to exist. One can straightforwardly obtain the special cases Eqs. (27), (36), and (42), by substituting α = φ = 0, γ = φ = 0, and α = cosγ = 0 in the general expression given by Eq. (49).

The geometric description of Eq. (49) can be written more intuitively as

$$y_{\text{cr}}=W\sqrt{\frac{e_{s\ast }^2+e_u^2}{e_y^2+e_{s\ast }^2+e_u^2}}=W\sqrt{\frac{1}{1+e_y^2/\left(e_{s\ast }^2+e_u^2\right)}}$$ (51)

(Appendix A), where $\widehat{s_{\ast }}$ is the unit vector in the solution space that is most aligned with ${\widehat{s}}_m^{\prime }$ [e.g., Fig. 4(b)], and e_y, e_s*, and e_u are the projections of $\widehat{e_m}$ onto $\widehat{y}=\overrightarrow{y}/y$, $\widehat{s_{\ast }}$, and ${\widehat{u}}_m$ [Fig. 5(b)]. Indeed, Eqs. (28) and (43) canbe readily recognized as special cases of the above general expression.

Each of these projections is interpretable in light of the fact that x_μm represents the activity level of the mth presynaptic neuron in the μth response pattern. Most simply,

$$e_y={\widehat{e}}_m\cdot \widehat{y}=\frac{\sum _{\mu =1}^{𝒫}{y}_{\mu }{x}_{\mu m}}{\sqrt{\sum _{\nu =1}^{𝒫}{y}_v^2}}$$ (52)

is a normalized correlation of the pre- and postsynaptic activity (note that ${\sum }_{\rho =1}^{𝒩}{X}_{\rho m}^2=1$ As expected, synapse certainty is aided by large magnitudes of e_y. Moreover, the sign of a certain synapse is the sign of this correlation, or equivalently the sign of e_y. Synapse sign identifiability is hindered by large values of

$$e_u={\widehat{e}}_m\cdot {\widehat{u}}_m=\sqrt{1-\sum _{\mu =1}^{𝒫}{x}_{\mu m}^2},$$ (53)

which effectively measures the weakness of the presynaptic neuron’s activity, as it is the amount of presynaptic drive for which we do not have any information on the target neuron’s response. The more subtle quantity is

$$\begin{gathered} e_{s\ast }={\widehat{e}}_m\cdot {\widehat{s}}_{\ast }=-\text{Sgn}(\text{cos}\alpha )\sqrt{\sum _{\left\{\mu \mid s_{\mu }^{\prime }<0\right\}}^{𝒫}{x}_{\mu m}^2} \\ \Rightarrow e_{s\ast }^2=\sum _{\left\{\mu \mid s_{\mu }^{\prime }<0\right\}}^{𝒫}{x}_{\mu m}^2. \end{gathered}$$ (54)

The condition that $s_{\mu }^{\prime }<0$ selects for patterns where the sign of the presynaptic activity is Sgn(cosα) = Sgn(e_y), but the postsynaptic neuron does not respond. In other words, presynaptic activity should have promoted a response in the target neuron according to the observed activity correlation. That it does not generates uncertainty in the sign of the synapse. See Appendix A for a heuristic derivation of y_cr based on this argument.

We can gain more useful intuition by interpreting our result in relation to what we would obtain in a linear neural network. In the linear problem, there are only constrained and unconstrained dimensions; every dimension that was semiconstrained in the nonlinear problem becomes constrained, with all solutions having η_μ = y_μ for μ = 1, …, 𝒫. This implies that

$$y_{\text{cr},\text{lin}}=W\sqrt{\frac{e_u^2}{e_y^2+e_u^2}}.$$ (55)

Returning to the nonlinear problem, recall that the certainty condition finds the largest y for which the solution space and w_m = 0 hyperplane intersect within the weight bound, and this intersection is simply a point when y = y_cr. Importantly, each semiconstrained dimension can either behave like a linear constrained dimension with η_μ = y_μ = 0 at this intersection point (toy problems 1 and 3), or like an unconstrained dimension with η_μ < 0 at the intersection point (toy problems 1 and 2).⁷ The first case occurs when ${\widehat{e}}_m\cdot {\overrightarrow{\epsilon }}_{\mu }$ and $e_y={\widehat{e}}_m\cdot \widehat{y}$ have opposite signs and $s_{\mu }^{\prime }>0$; the second case occurs when they have the same sign and $s_{\mu }^{\prime }<0$. This means that one could compute the nonlinear theory’s y-critical from y_cr,lin by appending the second class of semiconstrained dimensions onto the unconstrained dimensions. Mathematically, this corresponds to the replacement

$$e_u^2\to e_u^2+e_{s\ast }^2=\sum _{\mu =𝒫+1}^{𝒩}{x}_{\mu m}^2+\sum _{\left\{\mu \mid s_{\mu }^{\prime }<0\right\}}^{𝒫}{x}_{\mu m}^2,$$ (56)

which indeed transforms Eq. (55) to Eq. (51). The role of $e_{s\ast }^2$ is to quantify the uncertainty introduced by the subset of semiconstrained dimensions that do not behave as constrained at the intersection point.

Since the parameters e_y,e_s*, and e_u cannot be set independently, it is convenient to reparameterize Eq. (51) as

$$y_{\text{cr}}=W\sqrt{\frac{1}{1+r_y^2{e}_p^2/\left\{1-e_p^2\left[1-r_{s\ast }^2\left(1-r_y^2\right)\right]\right\}}},$$ (57)

where $e_p^2=1-e_u^2,r_y^2=e_y^2/e_p^2,r_{s\ast }^2=e_{s\ast }^2/\left(e_p^2-e_y^2\right)$, and all three composite parameters can be independently set between 0 and 1. Conceptually, r_y and r_s* merely normalize e_y and e_s* by their maximal values, and e_p is the projection of ${\widehat{e}}_m$ into the activity-constrained subspace spanned by both constrained and semiconstrained dimensions. One could also interpret r_s* as quantifying the effect of threshold nonlinearity. For instance, r_s* = 0 describes the case where all semiconstrained dimensions are effectively constrained, but r_s* increases as some of the semiconstrained dimensions start to behave like unconstrained dimensions. As expected, y_cr is a decreasing function of $r_y^2$ and $e_p^2$ and an increasing function of $r_{s\ast }^2$ [Fig. 5(c)].

B. Regarding nonorthogonal input patterns

While a complete treatment of the certainty condition for generally correlated input patterns is beyond the scope of this paper, we could find a conservative bound for y-critical that may be useful when patterns are close to being orthogonal. The details of the derivation are discussed in the final subsection of Appendix A.

The major challenge caused by nonorthogonal patterns is that the spherical weight space becomes elliptical in terms of the η coordinates. Thus, the main idea behind the bound is that one can always find the sphere that just encompasses this ellipse. We can then use our formalism to obtain a conservative y-critical, such that if the norm of $\overrightarrow{y}$ is larger than this value then all solutions within the encompassing sphere have a consistent sign for the synapse under consideration. An interesting insight that emerges from our analysis is that the relative orientations between

$${\overrightarrow{n}}_m\equiv \sum _{\mu =1}^{𝒩}{X}_{m\mu }^{-1}{\overrightarrow{\epsilon }}_{\mu }$$ (58)

and the various important η directions play the role of θ, φ, α and γ (Appendix A). Note that ${\overrightarrow{n}}_m={\widehat{e}}_m$ when X is an orthogonal matrix. We anticipate that $\overrightarrow{n}$ will also be an important player in a more comprehensive treatment of nonorthogonal patterns.

C. Application to recurrent networks

As we explained in Sec. III, to find the ensemble of all incoming weight vectors onto the ith driven neuron, one can use the results obtained for the feedforward network and just substitute X with the Z⁽ⁱ⁾ matrix. Consequently, identifying certain synapses onto the ith neuron can follow the route outlined for the feedforward scenario as long as Z⁽ⁱ⁾ is orthogonal. So for example, if we want to ascertain whether any incoming synapse to the ith neuron is certain, we have to replace $x_{\mu m}\to z_{\mu m}^{(i)}$ and y_μ → y_μi in Eqs. (51)–(54) to compute y_cr.

D. Numerical illustration of the certainty condition

To illustrate and test the theory numerically, we first considered a small neural network of three input neurons and three driven neurons [Fig. 6(a)]. This small number of synapses meant that we could comprehensively scan the entire spherical weight space without relying on a numerical algorithm to find solutions.⁸ This is important because numerical techniques, such as gradient descent learning, potentially find a biased set of solutions that incompletely test the theory. We supposed that each driven neuron has three inputs, and we constrained weights with two orthonormal stimulus responses. We set W = 1 for all simulations and numerically screened weights randomly. See Appendix F for complete simulation details.

FIG. 6.

Testing the certainty condition with exhaustive low-dimensional simulations. (a) A simple recurrent network with three input neurons and three driven neurons (Appendix E). (b) We plot the theoretically derived y_cr for feedforward synapses to y₁ as we vary θ (green curves) or φ (magenta curves), keeping the other angle fixed at 45°. The lighter shades correspond to cosγ = 1 ⇒ r_s* = 1. The darker shades correspond to cosγ = 0 ⇒ r_s* = 0, where the predictions from the nonlinear network match those of a linear network. The dots represent y_cr estimated through simulations, and they agree well with the theory. (c) (Bottom) Bar graph of the fraction of solutions with positive (red) and negative (blue) self-couplings (y₃ → y₃) as a function of θ. (Top) As predicted, all solutions have positive w_y3,y3 when y − y_cr > 0.

The first driven neuron in Fig. 6(a), y₁, receives only feedforward drive, and we suppose that it responds to one stimulus condition with response y (μ = 2), but it does not respond to the other (μ = 1). Its synapses thus have one constrained, one semiconstrained, and one unconstrained dimension, and all of the terms in Eq. (49) contribute to y-critical. We could thus use y₁ to verify Eq. (49). Moreover, this scenario includes the illustrative example of Fig. 4(f) as a special case, so we could also use y₁ to verify Eq. (42).

To these ends, we decided to focus on a two-parameter family of input patterns,

$$x=\left(\begin{array}{ccc}-\text{sin}\psi \text{cos}\chi & \text{cos}\psi \text{cos}\chi & \text{sin}\chi \\ \text{cos}\psi & \text{sin}\psi & 0\end{array}\right),$$ (59)

where rows correspond to different input patterns and columns correspond to different input neurons, as usual, and we extend x to the full-rank orthogonal matrix

$$X=\left(\begin{array}{ccc}-\text{sin}\psi \text{cos}\chi & \text{cos}\psi \text{cos}\chi & \text{sin}\chi \\ \text{cos}\psi & \text{sin}\psi & 0\\ \text{sin}\psi \text{sin}\chi & -\text{cos}\psi \text{sin}\chi & \text{cos}\chi \end{array}\right).$$ (60)

By Eq. (44), the physical basis vector corresponding to the synapse from the first input neuron is thus

$${\widehat{e}}_1=\text{cos}\psi {\overrightarrow{\epsilon }}_2+\text{sin}\psi \text{cos}\chi \left(-{\overrightarrow{\epsilon }}_1\right)+\text{sin}\psi \text{sin}\chi {\overrightarrow{\epsilon }}_3,$$ (61)

and it has the same general form as Eqs. (37) and (45), where ${\overrightarrow{\epsilon }}_3$ plays the role of $\widehat{u}$. If ψ and χ are both acute, then one can identify them with θ and φ in Fig. 4(f), and the roles of $\widehat{c}$ and $\widehat{s}$ are played by ${\overrightarrow{\epsilon }}_2$ and $-{\overrightarrow{\epsilon }}_1$, respectively. In this case α = 0, cosγ = 0, and the theoretical dependencies of y_cr on θ and φ are given by Eq. (42). Figure 6(b) illustrates these dependencies as the purple and dark green curves. If ψ is acute, but χ is obtuse, then according to our conventions, θ = ψ, φ = π – χ, $\widehat{c}={\overrightarrow{\epsilon }}_2$, and $\widehat{s}={\overrightarrow{\epsilon }}_1$. Now α = 0 and cosγ = 1, and our general formula, Eq. (49), implies

$$y_{\text{cr}}=W\text{sin}\theta =W\text{sin}\psi .$$ (62)

These dependencies are plotted as the pink and the light green curves in Fig. 6(b). We do not plot cases where ψ is obtuse, because obtuse and acute ψ result in equivalent y_cr formulas. Whether ψ is acute or obtuse nevertheless matters because it determines the sign of the w₁ synapse when it is certain.

The black dots in Fig. 6(b) show the largest response magnitude, y, for which we numerically found solutions with both positive and negative w₁ (see Appendix F for numerical methods), thereby providing a numerical estimate of y_cr. The theoretical curves and numerical points precisely aligned in all cases. The differences between the light and dark theoretical curves illustrates the effect of nonlinearity. When χ is obtuse, the semiconstrained dimension effectively behaves as unconstrained, and the mixing angle between the semiconstrained and unconstrained dimension is irrelevant to y-critical. When χ is acute, the semiconstrained dimension effectively behaves as constrained, as if its coordinate were set to zero. Moreover, these results confirmed that stronger responses were needed to make synapses fixed sign when the synaptic direction was less aligned with the constrained dimension [Fig. 6(b), purple and pink]. Furthermore, smaller y-critical values occurred when the synaptic direction anti-aligned with the semiconstrained dimension [Fig. 6(b), purple versus pink, dark green versus light green].

We next wanted to check the validity of our results for the recurrently connected neurons in Fig. 6(a). We therefore needed to tailor the steady-state activity levels of the recurrent network to result in orthogonal presynaptic input patterns for each driven neuron. In mathematical terms, Z⁽ⁱ⁾ must be an orthogonal matrix for i = 1, 2, 3. We achieved this by considering a two-parameter family of driven neuronal responses in which the activity patterns of y₂ and y₃ were matched to those of x₁ and x₃, respectively. This construction means that all three driven neurons receive the same input patterns. To ensure positivity of driven neuronal responses, we set χ as an acute angle and ψ as the negative of an acute angle.

Although y₂ has both feedforward and recurrent inputs, we can analyze its connectivity in exactly the same way as y₁. Recurrence only complicates the analysis for neurons that synapse onto themselves, like y₃, since changing the output activity also changes the input drive. So $\left|{\overrightarrow{y}}_3\right|$ and y_cr are not independent. Here we focused on the certainty condition for the self-synapse, w_y3,y3, for which y_cr = cos χ, and $\left|{\overrightarrow{y}}_3\right|=\text{sin}\chi$. Therefore, the synapse should be certain if 45° < χ ⩽ 90°. Since θ = π/2 − χ according to our conventions,⁹ this is equivalent to 0⩽ θ < 45° [Fig. 6(c), top]. Our numerical results precisely recapitulated these theoretical expectations [Fig. 6(c), bottom], as the self-connection was consistently positive across all simulations whenever this condition on θ was met. See Appendix E for certainty condition analyses for other synapses onto y₃ and Appendix F for complete simulation details.

VI. ACCOUNTING FOR NOISE

A. Finding the solution space in the presence of noise

So far we have only considered exact solutions to the fixed point equations. However, it is also important to determine weights that lead to fixed points near the specified ones. For example, biological variability and measurement noise generally make it infeasible to specify exact biological responses. Furthermore, numerical optimization typically produces model networks that only approximate the specified computation. We therefore define the ℰ-error surface as those weights that generate fixed points a distance ℰ from the specified ones,

$${𝒱}_{ℰ}=\left\{w\mid \sum _{\mu =1}^{𝒫}\sum _{i=1}^{𝒟}{\left(y_{\mu i}-{\tilde{y}}_{\mu i}(w)\right)}^2={ℰ}^2\right\},$$ (63)

where y_μi is the specified activity of the ith driven neuron in in the μth fixed point, and ${\tilde{y}}_{\mu i}$ is the corresponding activity level in the fixed point approached by the model network when it is initialized as y_i(t = 0) = y_μi. If the network dynamics do not approach a fixed point, perhaps oscillating or diverging instead [54], we say ℰ = ∞.

Each ℰ-error surface can be found exactly for feedforward networks. For illustrative purposes, let us first consider the 𝒟 = 1 feedforward scenario in which the driven neuron is active in every response pattern. This means that y_μ > 0 for all μ = 1, …, 𝒫, and we can reorder the μ indices to sort the driven neuron responses in ascending order, 0 < y₁ < y₂ < ⋯ < y_𝒫. Here we assumed that no two response levels are exactly equal, as is typical of noisy responses. Since all responses are positive, the zero-error solution space has no semiconstrained dimensions, and the only freedom for choosing w is in the 𝒰 = 𝒩 − 𝒫 unconstrained dimensions. Therefore, the zero-error surface of exact solutions, 𝒱₀, is a 𝒰-dimensional linear subspace, and 𝒱₀ is a point in the 𝒫-dimensional activity-constrained subspace.

How does this geometry change as we allow error? For 0 < ℰ < y₁, we must have ${\tilde{y}}_{\mu }>0$ for all μ. Therefore, the nonlinearity is irrelevant, and ℰ-error surfaces are spherical in the activity-constrained η coordinates [Eq. (63), Fig. 7(a(i))]. However, once ℰ = y₁ it becomes possible that ${\tilde{y}}_1=0$, and suddenly a semi-infinite line of solutions appears with η₁ ⩽ 0. As ℰ further increases, this line dilates to a high-dimensional cylinder [Fig. 7(a(ii))]. A similar transition happens at ℰ = y₂, whereafter two cylinders cap the sphere [Fig. 7(a(iii))]. Things get more interesting as ℰ increases further because two transitions are possible. A third cylinder appears at ℰ′ = y₃. However, at ${ℰ}^{″}=\sqrt{{\left(y_1\right)}^2+{\left(y_2\right)}^2}$ it is possible for both ${\tilde{y}}_1$ and ${\tilde{y}}_2$ to be zero, and the two cylindrical axes merge into a semi-infinite hyperplane defined by η₁ ⩽ 0, η₂ ⩽ 0. Thus, when ℰ″ < ℰ′ the error surface grows to attach a third cylinder [Fig. 7(a(iv))], and when ℰ″ < ℰ′ the two cylindrical surfaces merge to also include planar surfaces in between [Fig. 7(av)]. These topological transitions continue by adding new cylinders and merging existing ones, and the sequence is easily calculable from {y_μ}. Note that we use the terminology “topological transition” to emphasize that the structure of the error surface changes discontinuously at these values of error. The geometric transitions we observe here also relate to topological changes in a formal mathematical sense. For instance, while there are no incontractible circles in Fig. 7(a(ii)), one develops as we transition to Fig. 7(a(iii)).

FIG. 7.

The solution space geometry changes as the allowed error increases. (a) Error surface contours in a three-dimensional subspace corresponding to η₁, η₂, and η₃. Several topological transitions occur as the error increases. (i) We consider the case where all responses are positive, so the contours are spherical for small errors, just like in a linear neural network. (ii), (iii) Two cylindrical dimensions sequentially open up when the error is large enough for some η coordinates to become negative. (iv), (v) After that, either a third cylindrical dimension can open up, or the two cylindrical axes can join to form a plane. Which transition occurs at lower error depends on the pattern of neural responses. (b) (Left) We illustrate a case where there is a unique exact solution to the problem (brown dot). Allowing error but neglecting topological transitions would expand the solution space to an ellipse (here, brown circle), but the signs of w₁ and w₂ remains positive. Including topological transitions in the error surface can cap the ellipse with a cylinder (full yellow solution space). Now we can say with certainty that the sign of w₂ is positive, but negative values of w₁ become possible. (Right) Graphical conventions are the same. However, in this case all solutions inside the cylinder have w₂ > 0. Therefore, the topological transition breaks a near symmetry between positive and negative weights.

In general, y_μ may also be zero or negative in the presence of noise. Whenever y_μ = 0, the μth response pattern generates a semiconstrained dimension in 𝒱₀. However, if some response levels are negative, then there are no exact solutions at all. However, it becomes possible to find solutions when $ℰ=\sqrt{\sum _{\left\{\mu \mid y_{\mu }<0\right\}}{y}_{\mu }^2},$, and each response pattern associated with a negative y_μ acts as a semiconstrained dimension in 𝒱_ℇ. As illustrated above, more semiconstrained dimensions open up as more error is allowed in each of these cases.

This geometry only approximates ℰ-error surfaces for recurrent networks (Appendix C). For instance, displacing y_μi from its specified value changes the input pattern that define the ${\overrightarrow{\epsilon }}_{\mu }$ directions for downstream driven neurons, but this effect is neglected here. We will nevertheless find that this feedforward approximation to ℰ-error surfaces is practically useful for predicting synaptic connectivity in recurrent networks as well.

B. Predicting connectivity in the presence of noise

The threshold nonlinearity and error-induced topological transitions can have a major impact on synapse certainty [Fig. 7(b)]. For example, one might model a neuronal dataset with a linear neural network and find that models with acceptably low error consistently have positive signs for some synapses. However, if measured neural activity was sometimes comparable to the noise level, then semiconstrained dimensions could open up that suddenly make some of these synapse signs ambiguous [Fig. 7(b), left]. Although semiconstrained dimensions can never make an ambiguous synapse fully unambiguous, semiconstrained dimensions can heavily affect the distribution of synapse signs across the model ensemble by providing a large number of solutions that have consistent anatomical features [Fig. 7(b), right].

We therefore generalized the certainty condition to include the effects of error, including topological transitions in the error surface (Appendix C). As before, finding the certainty condition amounts to determining when the w_m = 0 hyperplane intersects the solution space within the weight bound, but to account for noise of magnitude ε, we must now check whether an intersection occurs with any ℰ-error surface with ℰ ⩽ ε. No intersections will occur if and only if every nonnegative $\overrightarrow{\tilde{y}}$ within ε of the provided 𝒫-vector of noisy target neuron activity [Fig. 3(c)] satisfies its zero-error certainty condition, and each $\overrightarrow{\tilde{y}}$ is a possible denoised version of it [Eq. (63)]. We thus define y-critical in the presence of noise as the maximal y_cr [Eq. (51)] among this set of $\overrightarrow{\tilde{y}}$.

Although we lack an exact expression for y-critical in the presence of noise, we derived several useful bounds and approximations (Appendix C). We usually focus on a theoretical upper bound for y-critical, y_cr,max. Note that this upper bound suffices for making rigorous predictions for certain synapses, because y > y_cr,max ⇒ y > y-critical. In the absence of topological transitions, this formula is

$$y_{\text{cr},\text{max}}=W[\sqrt{\left(\frac{e_{s\ast }^2+e_u^2}{e_y^2+e_{s\ast }^2+e_u^2}\right)+\frac{{\epsilon }^2}{W^2}\left(1+\frac{e_y^2\left(e_p^2-e_y^2\right)}{{\left(e_y^2+e_{s\ast }^2+e_u^2\right)}^2}\right)}+\frac{\epsilon }{W}\sqrt{1+\frac{e_y^2\left(e_p^2-e_y^2\right)}{{\left(e_y^2+e_{s\ast }^2+e_u^2\right)}^2}]}.$$ (64)

We also computed a lower bound, y_cr,min, to assess the tightness of the upper bound. This bound is

$$y_{\text{cr},\text{min}}=W\left[\sqrt{\frac{e_{s\ast }^2+e_u^2}{e_y^2+e_{s\ast }^2+e_u^2}}+\frac{\epsilon }{W}\right]$$ (65)

without topological transitions. Both bounds increase with error and should be considered to be bounded above byW. As expected, both expressions reduce to Eq. (51) as ε/W → 0. We also note that the two bounds coincide, to leading order in ε/W, if e_y ≪ max(e_s*, e_u) and e_p/max(e_s*, e_u) = 𝒪(1), and we argue in Appendix B that this is typical when the network size is large.

The effect of topological transitions is that y_cr,max and y_cr,min become the maximums of several terms, each corresponding to a way that constrained dimensions could behave as semiconstrained within the error bound (Appendix C). We compute each term from generalizations of Eqs. (64) and (65) that account for the amount of error needed to open up semiconstrained dimensions.

C. Testing the theory with simulations

To examine our theory’s validity, we assessed its predictions with numerical simulations of feedforward and recurrent networks [Fig. 8(a)]. Each assessment used gradient descent learning to find neural networks whose late time activity approximated some specified orthogonal configuration of input neuron activity and driven neuron activity (Appendix F). We then used our analytically derived certainty condition with noise to identify a subset of synapses that were predicted to not vary in sign across the model ensemble (W = 1), and we checked these predictions using the numerical ensemble. We similarly checked predictions from simpler certainty conditions that ignored the nonlinearity or neglected topological transitions in the error surface (Appendix C). Note that we expected gradient descent learning to often fail at finding good solutions in high dimensions, as our theory predicts that each semiconstrained dimension induces local minima in the error surface [Fig. 7(a)]. Since we did not want the theory to bias our numerical verification of it, we focused our simulations on small to moderately sized networks, where we could reasonably sample the initial weight distribution randomly. Future work will consider more realistic neural network applications.

FIG. 8.

The theory accounting for error explains numerical ensembles of feedforward and recurrent networks. (a) Cartoon of a recurrent neural network. We disallow recurrent connectivity of neurons onto themselves throughout this figure. 𝒟 = 1 corresponds to the feedforward case, and W = 1 for all panels. (b) Comparison of numerical and theoretical y-critical values for 102 random configurations of input-output activity (Appendix F). We considered a feedforward network with ℐ = 6, 𝒫 = 5, 𝒞 = 2. For each configuration and postsynaptic activity level y, we used gradient descent learning to numerically find many solutions to the problem with ℇ ≈ 0.1. The black dots correspond to the maximal value of y in our simulations that resulted in an inconsistent sign for the synaptic weight under consideration. The continuous curves show theoretical values for y-critical that upper bound the true y-critical (y_cr,max, black), that neglect topological transitions in the error surface (yellow), or that neglect the threshold nonlinearity (cyan). Only the black curve successfully upper bounded the numerical points. Configurations were sorted by the y_cr,max value predicted by the black curve. (c) The number of certain synapses increased with the total number of synapses in feedforward networks. Purple and brown correspond to 𝒩 = 2𝒫 = 4𝒞 and 𝒩 = 𝒫 = 4𝒞, respectively. The solid lines plot the predicted number of certain synapses. The circles represent the number of correctly predicted synapse signs in the simulations. The dashed brown and purple lines are best-fit linear curves with slopes 0.16(±0.01) and 0.07(±0.01) at 95% confidence level, significantly less than the zero error theoretical estimates of 0.28 and 0.18 (Appendix B). (d), (e) Testing the theory in a recurrent neural network with 𝒩 = 10, ℐ = 7, 𝒟 = 4, 𝒫 = 8, and 𝒞 = 3. Each dot shows a model found with gradient descent learning. (d) x and y axes show two η coordinates predicted to be constrained and semiconstrained, respectively, and the color axis shows the model’s root-mean-square error over neurons, $ℰ/\sqrt{𝒟}$. Although our theory for error surfaces is approximate for recurrent networks, the solution space was well explained by the constrained and semiconstrained dimensions. Note that the numerical solutions tend to have constrained coordinates smaller than the theoretical value (vertical line) because the learning procedure is initialized with small weights and stops at nonzero error. (e) The x axis shows the model’s error, and the y axis shows the number of synapse signs correctly predicted by the nonlinear theory (yellow dots or red crosses) or linear theory (cyan dots or blue crosses). Dots denote models for which every model prediction was accurate, and crosses denote models for which some predictions failed.

We first considered feedforward network architectures, for which our analytical treatment of noise is exact. To illustrate how nonlinearity and noise affect synapse certainty, we calculated the magnitude of postsynaptic activity needed to make a particular synapse sign certain [Fig. 8(b)]. We specifically considered 102 random input-output configurations of a small feedforward network with 6 input neurons (𝒫 = 5, 𝒞 = 2), which were tailored to have orthonormal input patterns and generate one topological error surface transition at small errors. In particular, we generated random orthogonal matrices by exponentiating random antisymmetric matrices, we set one element of $\widehat{y}$ to a small random value to encourage the topological transition, and we ensured that the other nonzero random element of $\widehat{y}$ was large enough to preclude additional transitions (Appendices C and F). For each input-output configuration, we then systematically varied the magnitude of driven neuron activity, y, finding 10⁵ synaptic weight matrices with moderate error, ℰ² ≈ ε², for each magnitude y. Since randomly screening a six-dimensional synaptic weight space is not numerically efficient, we applied gradient descent learning. Nevertheless, the small network size meant that we could comprehensively sample the solution space and numerically probe the distinct predictions made by each bound or approximation used to estimate y-critical.

As expected, the maximum value of y that produced numerical solutions with mixed synapse signs [Fig. 8(b), black dots] was always below the theoretical upper bound for y-critical [Fig. 8(b), black line]. In contrast, mixed-sign numerical ensembles were often found above theoretical y-critical values that neglected topological transitions in the error surface [Fig. 8(b), yellow line] or that neglected the nonlinearity entirely [Fig. 8(b), cyan line]. This means that these simplified calculations for estimating y-critical make erroneous predictions, because the synapse sign is supposed to be exclusively positive or negative whenever y exceeds y-critical, by definition. Therefore, we were able to accurately assess synapse certainty, and this generally required us to include both the nonlinearity and noise-induced topological transitions in the error surface.

We next asked how often we could identify certain synapses in larger networks. For this purpose, we generated 25 random input-output configurations in the feedforward setting (Appendix F), again with orthonormal input patterns, but this time we increased the number of input neurons from 4 to 100 across the configurations [Fig. 8(c)]. As we increased the size of the network, we kept 𝒞/𝒩 fixed at 0.25 and 𝒫/𝒩 fixed at 1 [Fig. 8(c), brown] or 0.5 [Fig. 8(c), purple]. These scaling relationships put our simulations in the setting of high-dimensional statistics [65], where both the number of parameters and the number of constraints increase with the size of the network. In this high-dimensional regime, a simple heuristic argument suggests that the number of zero-error certain synapses should scale linearly with the number of synapses (Appendix B), because y_cr and the typical magnitude of y scale equivalently with 𝒩. Here we tested this prediction by setting $\widehat{y}$ randomly, setting y = 1 − ln 2/-𝒞 to approximate the median norm of vectors in the unit 𝒞-ball (Appendix B), and numerically finding a small error solutionC for each configuration (ℰ²/𝒫 ≈ 10⁻⁶).

As expected, we empirically found that the number of certain synapses predicted by the theory [Fig. 8(c), solid lines] scaled with the network size linearly [Fig. 8(c), dashed lines]. The jaggedness of the solid curves reflect the fact that each point is specific to the random input-output configuration constructed for that value of 𝒩. The purple curve corresponds to the case when 𝒩 = 2𝒫 = 4𝒞 and the brown curve when 𝒩 = 𝒫 = 4𝒞. Furthermore, for every certain synapse predicted, we verified that its predicted sign was realized in the numerical solution we found [Fig. 8(c), circles]. These results suggest that the theory will predict many synapses to be certain in realistically large neural systems.

Finally, we empirically tested our theory for a recurrent network [Figs. 8(d) and 8(e)], where our treatment of noise is only approximate. For this purpose, we considered networks without the self-coupling terms, 𝒩 = ℐ + 𝒟 − 1. We constructed a single random configuration with nonnegative driven neuron responses and orthogonal presynaptic patterns for one of the driven neurons¹⁰ (Appendix F). This driven neuron could thus serve as the target neuron for our analyses. Note that it is sometimes possible to orthogonalize the input patterns for more than one driven neuron, but this is irrelevant to our analysis and is not pursued here. We then used gradient descent learning to find around 4500 networks that approximated the desired fixed points with variable accuracy. For technical simplicity, we first found connectivity matrices using a proxy cost function that treated the network as if it were feedforward. We then simulated the neural network dynamics with these weights and correctly evaluated the model’s error as prescribed by Eq. (63).

This network ensemble revealed that constrained and semiconstrained dimensions accurately explained the structure of the solution space for recurrent networks with nonzero error. Figure 8(d) shows the projection of the corresponding solution space along two η directions, one predicted to be constrained by the feedforward theory and the other predicted to be semiconstrained. As predicted, the extension of the solution space along the negative semiconstrained direction was clearly discernible. However, recurrence implies that the exact solution space is not perfectly cylindrical around the semiconstrained axes (Appendix C), because the driven neuron inputs to the target neuron can themselves vary due to noise. Here this effect was empirically insignificant, and the geometric structure of the solution space conformed rather well to our feedforward prediction. One might have expected the error [color in Fig. 8(d)] to increase monotonically as one moves away from the center of semiconstrained cylinder, but this expectation is incorrect for two reasons. First, we are visualizing the error surface as a projection along two dimensions, yet variations in other η coordinates add variation to the error.¹¹ Second, we are visualizing the solution space for one target neuron, but other driven neurons in the recurrent network contribute to the summed error represented by the color.

Moreover, the theory correctly predicted how the number of certain synapses would decrease as a function of ε [Fig. 8(e)], and we never found a numerical violation of the theoretical certainty condition that included nonlinearity and noise. In Fig. 8(e), the yellow circles represent the number of certain synapses that were predicted by the theory and verified to have synapse signs that agreed with the theoretical prediction. Here accurate predictions did not require us to account for topological error surface transitions. In contrast, although our simulations usually agreed with the predictions of the linear theory [Fig. 8(e), cyan circles], they could also disagree. In Fig. 8(e), the blue crosses indicate configurations where the linear theory incorrectly predicted some synapse signs. The absence of red crosses reiterates the consistency of predictions coming from the nonlinear treatment.

VII. DISCUSSION

In summary, we enumerated all threshold-linear recurrent neural networks that generate specified sets of fixed points, under the assumption that the number of candidate synapses onto a neuron is at least the specified number of fixed points. We found that the geometry of the solution space was elegantly simple, and we described a coordinate transformation that permits easy classification of weight-space dimensions into constrained, semiconstrained, and unconstrained varieties. This geometric approach also generalized to approximate error-surfaces of model parameters that imprecisely generate the fixed points. We used this geometric description of the error surface to analyze structure-function links in neural networks. In particular, we found that it is often possible to identify synapses that must be present for the network to perform its task, and we verified the theory with simulations of feedforward and recurrent neural networks.

Rectified-linear units are also popular in state of the art machine learning models [29,66–68], so the fundamental insights we provide into the effects of neuronal thresholds on neural network error landscapes may have practical significance. For example, machine learning often works by taking a model that initially has high error and gradually improving it by modifying its parameters in the gradient direction [69]. However, error surfaces with high error can have semiconstrained dimensions that abruptly vanish at lower errors (Fig. 7). Local parameter changes typically cannot move the model through these topological transitions, because models that wander deeply into semiconstrained dimensions are far from where they must be to move down the error surface. The model has continua of local and global minima, and the network needs to be initialized correctly to reach its lowest possible errors. This could provide insight into deep learning theories that view its success as a consequence of weight subspaces that happen to be initialized well [70,71].

The geometric simplicity of the zero-error solution space provides several insights into neural network computation. Every time a neuron has a vanishing response, half of a dimension remains part of the solution space, which the network could explore to perform other tasks. In other words, by replacing an equality constraint with an inequality constraint, simple thresholding nonlinearities effectively increase the computational capacity of the network [72,73]. The flexibility afforded by vanishing neuronal responses thereby provides an intuitive way to understand the impressive computational power of sparse neural representations [50,74–76]. Furthermore, the brain could potentially use this flexibility to set some synaptic strengths to zero, thereby improving wiring efficiency. This would link sparse connectivity to sparse response patterns, both of which are observed ubiquitously in neural systems.

Our theory could be extended in several important ways. First, we only derived the certainty condition to identify critical synapses from orthonormal sets of fixed points. Although our orthogonal analysis also provides a conservative bound for a general set of fixed points (Appendix A), a more precise analysis will be needed to pinpoint synapses in realistic biological settings where stimulus-induced activity patterns may be strongly correlated. Since our error surface description made no orthonormality assumptions, this analysis will only require more complicated geometrical calculations to discern whether the synapse sign is consistent across the space of low-error models. Furthermore, we could use the error surfaces to identify multisynapse anatomical motifs that are required for function, or to estimate the fraction of models in which an uncertain synapse is excitatory versus inhibitory. It would also be interesting to relax the assumption that the number of fixed points is small. This would allow us to consider scenarios where the fixed points can only be generated nonlinearly. We could also consider cases where no exact solution exists at all. Here we assumed that we knew the activity level of every neuron in the circuit. This is not always the case, and it will be important to determine how unobserved neurons alter the error landscape for synaptic weights connecting the observed neurons. The error landscape geometry will also be affected by recurrent network effects that we ignored here (Appendix C). It will be interesting to see whether the geometric toolbox of theoretical physics can provide insights into the nontrivial effects of unobserved neurons and recurrent network dynamics. Finally, we note that it will sometimes be important to analyze networks with alternate nonlinear transfer functions. Our analyses already apply exactly to recurrent networks with arbitrary threshold-monotonic nonlinear transfer functions (Appendix D). Moreover, our analyses can approximate any nonlinearity by treating its departures from threshold-linearity as noise (Appendix D). An extension to capped rectified linear units [67], which saturate above a second threshold, would also be straightforward. In particular, semiconstrained dimensions would emerge from any condition where the target neuron is inactive or saturated.

Our primary motivation for undertaking this study was to find rigorous theoretical methods for predicting neural circuit structure from its functional responses. This identification can be used to corroborate or broaden circuit models that posit specific connectivity patterns, such as center-surround excitation-inhibition in ring attractors [16–18] or contralateral relay neuron connectivity in zebrafish binocular vision [12,77]. More generally, if an experimental test violates the certainty conditions we derived using our ensemble modeling approach, it will suggest that some aspect of model mismatch is important. We could then move on to the development of qualitatively improved models that might modify neuronal nonlinearities, relax weight bounds, incorporate subcellular processes or neuromodulation, or hypothesize hidden cell populations. However, we hope that our focus on predictions that follow with certainty from simple network assumptions will enable predictions that are relatively insensitive to minor mismatches between our abstract model and the real biological brain. More nuanced predictions may require more nuanced models.

An important parameter of the theory is the weight bound. In particular, W bounds the magnitude of synaptic weight vectors in biological networks, and our certainty condition declares a synapse to be necessary when the ratio y/W exceeds a critical value. It is not a priori clear how to set this scale parameter without additional biological data. Nevertheless, one could use the neuronal activity data to compute each synapse’s W-critical value, below which the certainty condition is satisfied, and rank-order the synapses according to decreasing W-critical values. Until we know the value of W, we do not know where to draw the line between certain synapses and uncertain synapses. However, our theory predicts that all of the certain synapses will be at the top of the list, which specifies a sequence of experimentally testable predictions and may already provide biological insights into the important synaptic connections. Testing these predictions can help constrain the theory’s biological bound parameter.

Our theory describes function at the level of neural representations. This description is useful because many systems neuroscience experiments measure representations directly, and it is important to build mechanistic models that explain these data in terms of neural network interactions [12,15,18,77]. However, it would also be interesting to link structure to function at the higher levels of behavior and cognition. This is a significantly different problem because multiple representations can support the same high-level functions, and both neural network structure and representation can change over time [78–85]. Consequently, experimental tests of our current framework must measure network structure and representation on timescales shorter than the network’s representational dynamics, and certain synapses may be most biologically meaningful in innate circuits with limited plasticity. Extensions to our framework may also be useful for relating structural and representational dynamics in circuits for learning [86].

An exciting prospect is to explore how our ensemble modeling framework can be combined with other theoretical principles and biological constraints to obtain more refined structure-function links. For instance, we could refine our ensemble by restricting to stable fixed points. Alternatively, once the sign of a given synapse is identified, Dale’s principle might allow us to fix the signs of all other synapses from this neuron [87]. This would restrict the solution space and could make other synapses certain. Utilizing limited connectomic data to impose similar restrictions might also be a fruitful way to benefit from large-scale anatomical efforts [7,10,13,14]. Finally, rather than restricting the magnitude of the incoming synaptic weight vector, we could consider alternate biologically relevant constraints, such as limiting the number of synapses, minimizing the total wiring length, or positing that the network operates at capacity [88,89]. These changes would modify the certainty conditions in our framework, as well as our experimental predictions. We could therefore assess candidate optimization principles and biological priors experimentally. While the base framework developed here was designed to identify crucial network connections required for function, we hope that our approach will eventually allow us to assess theoretical principles that determine how neural network structure follows from function.

APPENDIX A: A CERTAINTY CONDITION TO PINPOINT SYNAPSES REQUIRED FOR SPECIFIED RESPONSE PATTERNS

1. Preliminaries

For completeness, we begin by briefly reviewing a few central concepts from the main manuscript.

a. From recurrent to feedforward networks

Let us consider a neural network of ℐ input neurons that send signals to an interconnected population of 𝒟 driven neurons governed by dynamical equations (6), as described in the main manuscript. At steady-state, since all time-derivatives are zero, Eq. (6) yields

$$\begin{array}{l}{y}_{\mu i}=\Phi \left(\sum _{m=1}^{𝒟}{w}_{im}{y}_{\mu m}+\sum _{m=𝒟+1}^{𝒟+ℐ}{w}_{im}{x}_{\mu ,m-𝒟}\right)\\ =\Phi \left(\sum _{m=1}^{𝒩}{w}_{im}{z}_{\mu m}\right),\end{array}$$ (A1)

where, as prescribed in the main manuscript, y_μi and x_μm denote steady-state activity levels of the driven and input neurons to the μth stimulus, which we have combined into z_μm, and 𝒩 is the number of incoming synapses onto each of the driven neurons. Equation (A1) provides 𝒟 × 𝒫 nonlinear equations for 𝒟 × 𝒩 unknown parameters. However, we immediately notice that the steady-state activity of neuron i depends only on the ith row of the connectivity matrix, so these equations separate into 𝒟 independent sets of 𝒫 equations with 𝒩 unknowns, the weights onto a given driven neuron. In other words, the recurrent network involving 𝒟 driven and ℐ input neurons decomposes into 𝒟 feedforward networks with 𝒩 = 𝒟 + ℐ feedforward inputs. The steady-state equations for these feedforward networks are given by

$$y_{\mu }=\Phi \left(\sum _{m=1}^{𝒩}{z}_{\mu m}{w}_m\right),$$ (A2)

where we have now suppressed the i index in y_μi and in w_im. For this feedforward network we will refer the ith neuron as the target neuron, and it is as if that all the neurons (driven and input) are providing feedforward inputs to it. As long as we only consider exact solutions to the fixed point equations, the problem of identifying synaptic connectivity in a recurrent network reduces to solving the problem for feedforward networks. Thus, in the rest of this Appendix we will focus on identifying w_m’s satisfying Eq. (A2).

Note that the main text used the notation $z_{\mu m}^{(i)}$ to emphasize that the set of presynaptic neurons may depend on the target neuron, but we simply write z_μm throughout the Appendices with the understanding that the formalism applies to a specified target neuron whose index is suppressed. Furthermore, for conceptual simplicity the main text first stated many results in a feedforward setting with a single driven neuron, but the Appendices immediately treat the general case where presynaptic partners may come from either the input or driven populations of neurons.

b. A convenient set of variables

In all our discussions in this section the input neuronal response matrix, z_μm, will be assumed to be fixed. Note that z_μm connects synaptic weight vectors to the target response vector and can be used to define 𝒫 weight combinations, the η coordinates. Each η coordinate controls the target response to a single stimulus condition:

$$y_{\mu }=\Phi \left({\eta }_{\mu }\right),\hspace{1em}\text{where}{\eta }_{\mu }\equiv \sum _{m=1}^{𝒩}{z}_{\mu m}{w}_m.$$ (A3)

It is rather convenient to extend this set of 𝒫 η coordinates to a basis set of 𝒩 η coordinates, such that all synaptic weights can be uniquely expressed as a linear combination of these η coordinates, and vice versa. To see how this can be done, we will henceforth make the simplifying assumption that the 𝒫 × 𝒩 matrix has the maximal rank, 𝒫, although we anticipate that much of our framework, results, and insights will apply more generally. If z has maximal rank, then its kernel will be an (𝒩 − 𝒫)-dimensional linear subspace spanned by (𝒩 − 𝒫) orthogonal basis vectors, denoted by ${\overrightarrow{\epsilon }}_{\mu }$ for μ = 𝒫 + 1 …𝒩. We can now extend z to an 𝒩 × 𝒩 matrix, Z, as follows:

$$\begin{gathered} Z_{\mu m}=z_{\mu m}\hspace{1em}\text{for}\hspace{1em}\mu =1\dots 𝒫,\text{ and }\forall m, \\ {Z}_{\mu m}={\epsilon }_{\mu m}\hspace{1em}\text{for }\mu =𝒫+1\dots 𝒩,\text{ and }\forall m, \end{gathered}$$ (A4)

where ε_μm is the mth component of the null vector ${\overrightarrow{\epsilon }}_{\mu }$. With this construction, it is easy to see that the new η coordinates,

$${\eta }_{\mu }=\sum _{m=1}^{𝒩}{Z}_{\mu m}{w}_m,\hspace{1em}\text{for }\mu =𝒫+1,\dots ,𝒩,$$ (A5)

remain completely unconstrained by the specified response patterns, as these linear combinations do not contribute to any of the target responses. In contrast, the original η coordinates,

$${\eta }_{\mu }=\sum _{m=1}^{𝒩}{Z}_{\mu m}{w}_m=\sum _{m=1}^{𝒩}{z}_{\mu m}{w}_m,\hspace{1em}\text{for}\mu =1,\dots ,𝒫,$$ (A6)

are all constrained by the data:

$${\eta }_{\mu }\{\begin{array}{ll}=y_{\mu }\hfill & \text{for}\mu =1,\dots ,𝒞,\text{the constrained dimensions},\hfill \\ ⩽0\hfill & \text{for}\mu =𝒞+1,\dots ,𝒫\text{, the semiconstrained dimensions},\hfill \end{array}$$ (A7)

where for notational simplicity we have ordered the response patterns such that y_μ = 0 only for μ = 1, …, 𝒞. Also, we extend the y_μ’s to an 𝒩-dimensional vector, $\overrightarrow{y}$, by assigning y_μ = 0 for μ = 𝒫 + 1 …𝒩.

The extended response matrix Z defines a basis transdirections formation connecting physical synaptic directions, $\widehat{e_m}$, with directions

$${\overrightarrow{\epsilon }}_{\mu }\equiv \sum _{m=1}^{𝒩}{\widehat{e}}_m{Z}_{m\mu }^{-1},$$ (A8)

along which the η coordinates change. These $\overrightarrow{\epsilon }$ vectors clearly differentiate directions in the weight space that are activity-constrained by neuronal responses (μ = 1, …, 𝒫) from those that are not (μ = 𝒫 + 1, …, 𝒩). We can express any weight vector in either the $\left\{{\widehat{e}}_m\right\}$ basis or the $\left\{{\overrightarrow{\epsilon }}_{\mu }\right\}$ basis:

$$\overrightarrow{w}=\sum _{m=1}^{𝒩}{w}_m{\widehat{e}}_m=\sum _{\mu =1}^{𝒩}{\eta }_{\mu }{\overrightarrow{\epsilon }}_{\mu },$$

where

$$\begin{array}{l}{w}_m=\sum _{\mu =1}^{𝒩}{Z}_{m\mu }^{-1}{\eta }_{\mu },\hspace{1em}{\eta }_{\mu }=\sum _{m=1}^{𝒩}{Z}_{\mu m}{w}_m,\\ {\overrightarrow{\epsilon }}_{\mu }\equiv \sum _{m=1}^{𝒩}{\widehat{e}}_m{Z}_{m\mu }^{-1},\hspace{1em}{\widehat{e}}_m=\sum _{\mu =1}^{𝒩}{\overrightarrow{\epsilon }}_{\mu }{Z}_{\mu m}.\end{array}$$ (A9)

For later convenience we also define the number of semiconstrained and unconstrained dimensions as, 𝒮 = 𝒫 − 𝒞 and 𝒰 = 𝒩 − 𝒫, respectively.

2. Derivation of the certainty condition for orthogonal input patterns

Our goal here is to use the solution space (i.e., ensemble of weights that are precisely able to recover the specified target responses) to derive a condition for when we can be certain that a given synapse must be nonzero. For technical simplicity, we will specialize to the case when all the response patterns are orthonormal, i.e.,

$$\sum _{m=1}^{𝒩}{z}_{\mu m}{z}_{vm}={\delta }_{\mu \nu }\iff z{z}^T=I,$$ (A10)

where I is the identity matrix. Then we can always choose the extended Z matrix to be an 𝒩 × 𝒩 orthogonal matrix, such that Z⁻¹ = Z^T and the ${\overrightarrow{\epsilon }}_{\mu }$ vectors now form an orthonormal basis. Motivated by biological constraints, we will impose a bound on the magnitude of the synaptic weight vector. For orthonormal response patterns, this translates into a spherical bound on η coordinates as well [see Fig. 5(b)]:

$$|\overrightarrow{w}{|}^2=\sum _{m=1}^{𝒩}{w}_m^2=\sum _{\mu =1}^{𝒩}{\eta }_{\mu }^2⩽W^2.$$ (A11)

We refer to this 𝒩-dimensional ball, in which all admissable synaptic weights reside, as the weight space.

a. A heuristic argument for y-critical

Before diving into the rigorous and technical derivation, in this subsection we first try to intuitively understand how the certainty condition (51) can arise. For this purpose, let us start with a linear theory with no unconstrained dimension, so 𝒮 = 𝒰 = 0. In this case, there is a unique set of weights that can precisely reproduce the observed responses:

$$w_m=\sum _{\mu =1}^{𝒩}{Z}_{m\mu }^{-1}{y}_{\mu }=\sum _{\mu =1}^{𝒩}{Z}_{\mu m}{y}_{\mu }.$$ (A12)

Since Z_μm = z_μm represents the responses of the mth presynaptic neuron, the solution for the mth synaptic weight (A12) is simply the correlation between the pre and post synaptic activity. In a linear theory, the sign of the synapse is thus dictated by the sign of the correlation between the pre and post synaptic neuron.

Let us now allow a single (𝒩th) unconstrained direction. One can think of this situation as if we do not have the information on how the target neuron would respond to the unconstrained stimulus pattern. If we knew that this response was say, y_u, then we would have been able to determine the sign of w_m:

$$\text{Sgn}\left(w_m\right)=\text{Sgn}\left(\sum _{\mu =1}^{𝒩-1}{Z}_{\mu m}{y}_{\mu }+Z_{𝒩m}{y}_u\right).$$ (A13)

However, since we do not know what the last term is, if it can cancel the first term for some allowed value of y_u then the overall sign becomes ambiguous. Conversely, w_m becomes certain if

$$\left|\sum _{\mu =1}^{𝒩-1}{Z}_{\mu m}{y}_{\mu }\right|>\left|Z_{𝒩m}{y}_u\right|\forall y_u.$$ (A14)

Now, it is easy to recognize that the first term is just $\widehat{e}\cdot \overrightarrow{y}=y{e}_y$, where we have suppressed the m index on $\widehat{e_m}$ here to reduce notational clutter and will continue to doso while referring to the synapse direction whose we are considering.¹² $e_y=\widehat{e}\cdot \widehat{y}$ refers to the projection of $\widehat{e}$ along $\widehat{y}$. Also, note that in this simple case with one unconstrained direction, the projection of $\widehat{e}$ along the unconstrained subspace is just given by $e_u=\widehat{e}\cdot {\overrightarrow{\epsilon }}_{𝒩}=Z_{𝒩m}$. Further, since Z is orthogonal, to have any solution at all

$$y^2+y_u^2⩽W^2$$ (A15)

Substituting the maximum |y_u| from Eq. (A15) into Eq. (A14), after some algebra we get the condition for sign certainty as

$$y>y_{\text{cr}}=W\sqrt{\frac{e_u^2}{e_u^2+e_y^2}}.$$ (A16)

The same argument applies if the 𝒩th direction is semiconstrained instead of unconstrained, with one notable difference. If the 𝒩th pattern was semiconstrained that means y 𝒩 = 0, and the nonlinear thresholding is masking how the target neuron would have responded in a linear model.¹³ However, the ambiguity in sign can only arise if the second term has a sign opposite to the first term, or a sign opposite to Sgn(e_y). Moreover, for the thresholding to act, the target response for the semiconstrained pattern must be negative in the linear theory, so Z_𝒩m has to have the same sign as e_y to generate the ambiguity. And, if it is indeed so, then we obtain a certainty condition that is identical to Eq. (A16) except that e_u → e_s, the projection of $\widehat{e}$ along the semiconstrained direction:

$$y>y_{\text{cr}}=W\sqrt{\frac{e_s^2}{e_s^2+e_y^2}}.$$ (A17)

If Z_𝒩m and e_y have opposite signs, then the synapse always has the same sign throughout the solution space.

While this derivation of y_cr is heuristic and only deals with a single semiconstrained or unconstrained dimension, it provides intuition for the general result (51). Essentially, whether the sign of a given synapse is constant across the solution space depends on two competing quantities: the correlation between the pre- and postsynaptic responses; and the strength of the postsynaptic drive for patterns where the target response is either unknown or masked by the thresholding nonlinearity.

b. Hyperplane dividing excitatory and inhibitory synaptic regions

Having gained some intuition about the certainty condition, let us now proceed to a rigorous derivation of the result. Since the constrained coordinates are fixed for the weight vectors that belong to the solution space [the deep yellow wedge in Fig. 5(b)], we must have

$$y^2\equiv \sum _{\mu =1}^{𝒞}{y}_{\mu }^2=\sum _{\mu =1}^{𝒞}{\eta }_{\mu }^2,$$ (A18)

so that the solution space resides within an (𝒰 + 𝒮)-dimensional ball with radius

$$\overline{W}\equiv \sqrt{W^2-y^2},$$ (A19)

as depicted in Fig. 5(b), by the yellow region. We refer to this semiconstrained plus unconstrained subspace as the flexible subspace.

Now, the synaptic direction of interest, $\widehat{e}$, can be decomposed into its projections along constrained, semiconstrained and unconstrained subspaces. For notational simplicity, let us denote $e_{\mu }\equiv \left(\widehat{e}\cdot {\overrightarrow{\epsilon }}_{\mu }\right)=Z_{\mu m}$ as the component of $\widehat{e}$ along ${\overrightarrow{\epsilon }}_{\mu }$. Note that the second equality follows from the orthogonality assumption and Eq. (A9). In general, in this manuscript we will subscripts on e to denote projections of $\widehat{e}$ along different directions or subspaces. We can now write¹⁴

$$\widehat{e}=\sum _{\mu =1}^{𝒩}{e}_{\mu }{\overrightarrow{\epsilon }}_{\mu }=\sum _{\mu =1}^{𝒞}{e}_{\mu }{\overrightarrow{\epsilon }}_{\mu }+\sum _{\mu =𝒞+1}^{𝒫}{e}_{\mu }{\overrightarrow{\epsilon }}_{\mu }+\sum _{\mu =𝒫+1}^{𝒩}{e}_{\mu }{\overrightarrow{\epsilon }}_{\mu }=\text{cos}\theta \widehat{c}+\text{sin}\theta \text{cos}\varphi \widehat{s}+\text{sin}\theta \text{sin}\varphi \widehat{u},$$ (A20)

where

$$\widehat{c}\equiv \frac{{\sum }_{\mu =1}^{𝒞}{e}_{\mu }{\overrightarrow{\epsilon }}_{\mu }}{\sqrt{{\sum }_{\mu =1}^{𝒞}{e}_{\mu }^2}},\widehat{s}\equiv \frac{{\sum }_{\mu =𝒞+1}^{𝒫}{e}_{\mu }{\overrightarrow{\epsilon }}_{\mu }}{\sqrt{{\sum }_{\mu =𝒞+1}^{𝒫}{e}_{\mu }^2}},\widehat{u}\equiv \frac{{\sum }_{\mu =𝒫+1}^{𝒩}{e}_{\mu }{\overrightarrow{\epsilon }}_{\mu }}{\sqrt{{\sum }_{\mu =𝒫+1}^{𝒩}{e}_{\mu }^2}},$$ (A21)

are unit vectors that lie within the constrained, semiconstrained and unconstrained subspaces, and

$$\begin{array}{l}{e}_c\equiv \widehat{c}\cdot \widehat{e}=\text{cos}\theta =\sqrt{\sum _{\mu =1}^{𝒞}{e}_{\mu }^2}⩾0,\hspace{1em}{e}_s\equiv \widehat{s}\cdot \widehat{e}=\text{sin}\theta \text{cos}\varphi =\sqrt{\sum _{\mu =𝒞+1}^{𝒫}{e}_{\mu }^2}⩾0,\\ e_u\equiv \widehat{u}\cdot \widehat{e}=\text{sin}\theta \text{sin}\varphi =\sqrt{\sum _{\mu =𝒫+1}^{𝒩}{e}_{\mu }^2}⩾0\end{array}$$ (A22)

are the projections of $\widehat{e}$ along these directions. One could think of θ, φ as representing a spherical coordinate system where the role of x, y, and z axesare played by $\widehat{s}$, $\widehat{u}$, and $\widehat{c}$ respectively, and our definitions (A20)–(A22) imply the convention, 0 ⩽ {θ, φ} < π/2. For later convenience, let us also introduce the projection of $\widehat{e}$ onto the activity-constrained subspace

$$e_p\equiv \widehat{p}\cdot \widehat{e}=\sqrt{{\text{cos}}^2\theta +{\text{sin}}^2\theta {\text{cos}}^2\varphi }=\sqrt{\sum _{\mu =1}^{𝒫}{e}_{\mu }^2}⩾0,\hspace{1em}\text{where }\widehat{p}\equiv \frac{{\sum }_{\mu =1}^{𝒫}{e}_{\mu }{\overrightarrow{\epsilon }}_{\mu }}{\sqrt{{\sum }_{\mu =1}^{𝒫}{e}_{\mu }^2}}.$$ (A23)

We would also like to emphasize that we can compute θ, φ just from the knowledge of the neuronal responses, z_μm = e_μ, which is particularly useful for numerical calculations:

$$\theta \equiv {\text{cos}}^{-1}\left(\sqrt{\sum _{\mu =1}^{𝒞}{z}_{\mu m}^2}\right),\hspace{1em}\text{and}\hspace{1em}\varphi \equiv {\text{cos}}^{-1}\left(\sqrt{\sum _{\mu =𝒞+1}^{𝒫}{z}_{\mu m}^2}/\sqrt{1-\sum _{\mu =1}^{𝒞}{z}_{\mu m}^2}\right).$$ (A24)

Now, any weight vector in the solution space can be written as

$$\overrightarrow{w}=\overrightarrow{y}+{\overrightarrow{w}}_s+{\overrightarrow{w}}_u,$$ (A25)

where ${\overrightarrow{w}}_s$ and ${\overrightarrow{w}}_u$ are the projections of $\overrightarrow{w}$ onto the semiconstrained and unconstrained subspaces, and the constrained part of $\overrightarrow{w}$ is fixed at $\overrightarrow{y}$. Using Eqs. (A20) and (A25), one then finds that the w_m = 0 hyperplane dividing the excitatory and inhibitory regions in the flexible subspace satisfies the equation

$$\begin{array}{l}w=\overrightarrow{w}\cdot \widehat{e}\\ =y\text{cos}\theta \text{cos}\alpha +\text{sin}\theta \text{cos}\varphi \widehat{s}\cdot {\overrightarrow{w}}_s+\text{sin}\theta \text{sin}\varphi \widehat{u}\cdot {\overrightarrow{w}}_u=0,\end{array}$$ (A26)

where we have now also suppressed the index m in w_m. Also, we have defined α ∈ [0, π] to be the angle between $\overrightarrow{y}$ and $\widehat{c}$. We now notice that the origin of the flexible subspace, ${\overrightarrow{w}}_s={\overrightarrow{w}}_u=0$, is in the solution space and the sign of w for this solution point is given by

$$\text{Sgn}(w)=\text{Sgn}(\text{cos}\alpha )=\text{Sgn}\left(e_y\right)=\text{Sgn}\left(\sum _{\mu =1}^{𝒞}{z}_{\mu m}{y}_{\mu }\right).$$ (A27)

In other words, if the sign of the synapse is certain, this certain sign must be Sgn(cos α), which corresponds to the sign of the correlation between the target neuron and the presynaptic neuron. Intuitively, positive correlations point to an excitatory connection, and negative correlations point to an inhibitory connection.

c. Special case without unconstrained dimensions

To derive the certainty condition, let us start by looking at the case when 𝒫 = 𝒩, so that there are no unconstrained directions, or equivalently, ϕ = 0. In this case, the solution space is just the all-negative orthant in the 𝒮-dimensional semiconstrained hypersphere (Fig. 9), and the equation for the w = 0 hyperplane can be written as

$$\text{sin}\theta \left({\widehat{s}}^{\prime }\cdot {\overrightarrow{w}}_s\right)=y\text{cos}\theta \left|\text{cos}\alpha \right|,$$ (A28)

where the right-hand side (RHS) is positive, and we have introduced

$${\widehat{s}}^{\prime }\equiv -\text{Sgn}(\text{cos}\alpha )\widehat{s},$$ (A29)

which flips the direction of $\widehat{s}$ if cos α > 0, or equivalently, if e_y > 0. Now, if the w = 0 hyperplane (orange lines in Fig. 9) is far enough along ${\widehat{s}}^{\prime }$ from the origin that it does not intersect with the all-negative orthant within the weight bounds, then we can be certain that w is nonzero and always has a consistent sign. To check this, we need to compare the cone angle that the orange hyperplane subtends at the center, φ, with the minimum angle, γ, that the ${\widehat{s}}^{\prime }$ vector makes with the all-negative orthant.

FIG. 9.

Cartoons depicting the orientation of the semiconstrained projection of a given synaptic weight direction $\left({\widehat{s}}^{\prime }\right)$ within the semiconstrained subspace and its impact on determining the sign of the given weight. In these plots, the yellow wedges represent the solution space, η₁, η₂ ⩽ 0. d_s is the distance of the w_m = 0 orange line (hyperplane in higher dimension) from the origin. If d_s is small, as in the left plot (a), then the projection angle γ is smaller than φ, half of the angle subtended by the orange line to the origin, and therefore the orange line and the yellow cone intersect. This means that solutions with both positive and negative w’s are present. In the right plot (b), d_s is sufficiently large such that γ > φ and consequently, all the solutions must have consistent sign.

First, φ can easily be inferred from trigonometry:

$$\text{cos}\phi =\frac{d_s}{\overline{W}}=\frac{y\text{cot}\theta \left|\text{cos}\alpha \right|}{\sqrt{W^2-y^2}},$$ (A30)

where d_s = y cot θ|cos α| represents the distance from the center of the semiconstrained sphere to the hyperplane. The expression for d_s follows from the general mathematical result that if

$$\overrightarrow{\beta }\cdot \overrightarrow{x}+{\beta }_0=0$$ (A31)

is an equation for a hyperplane, where $\overrightarrow{x}$ denotes the coordinate vector and $\overrightarrow{\beta }=\text{sin}\theta {\widehat{s}}^{\prime }\Rightarrow |\overrightarrow{\beta }|=\text{sin}\theta$, β₀ are constants, then the perpendicular distance, d_⊥, to it from a point $\overrightarrow{x}=\overrightarrow{\zeta }$ is given by

$$d_{\perp }=\frac{\left|\overrightarrow{\beta }\cdot \overrightarrow{\zeta }+{\beta }_0\right|}{\left|\overrightarrow{\beta }\right|}$$ (A32)

Note, we are interested in the distance from the origin, $\overrightarrow{\zeta }=0$, to the hyperplane satisfying Eq. (A28), so $\overrightarrow{\beta }=\text{sin}\theta {\widehat{s}}^{\prime }\Rightarrow |\overrightarrow{\beta }|=\text{sin}\theta$ and ${\beta }_0=-y\text{cos}\theta |\text{cos}\alpha |$.

To provide a geometric intuition for γ, let us first assume that ${\widehat{s}}^{\prime }$ does not point into the all-negative orthant. If we can find the projection of ${\widehat{s}}^{\prime }$ on the correct boundary of the solution space, then γ will be given by the angle between ${\widehat{s}}^{\prime }$ and the appropriate semiconstrained boundary vector, ${\overrightarrow{s}}_{*}$ (Fig. 9). Since all the components in the solution space (all-negative orthant) have to be negative or zero, to find the appropriate projection vector of ${\widehat{s}}^{\prime }$ onto the boundary of solution space, we essentially have toset all the positive components to zero:

$${\overrightarrow{s}}_{\ast }=\sum _{\mu =𝒞+1}^{𝒫}{s}_{\mu }^{\prime }\Theta \left(-s_{\mu }^{\prime }\right){\overrightarrow{\epsilon }}_{\mu }=\mp \sum _{\mu =𝒞+1}^{𝒫}{s}_{\mu }\Theta \left(\pm s_{\mu }\right){\overrightarrow{\epsilon }}_{\mu },$$ (A33)

depending upon whether Sgn(e_y) = ±. Here $s_{\mu }^{\prime }$, s_μ are just the μth components of ${\widehat{s}}^{\prime }$ and $\widehat{s}$ vectors, and ϴ(x) is the Heaviside step function, whichis one if x is positive and zero otherwise. Then, γ is given by

$$\begin{gathered} \text{cos}\gamma ={\widehat{s}}^{\prime }\cdot {\widehat{s}}_{\ast }=\frac{{\widehat{s}}^{\prime }\cdot {\overrightarrow{s}}_{\ast }}{\left|{\overrightarrow{s}}_{\ast }\right|}=\sqrt{\sum _{\mu =𝒞+1}^{𝒫}{s}_{\mu }^{\prime 2}\Theta \left(-s_{\mu }^{\prime }\right)} \\ =\sqrt{\sum _{\mu =𝒞+1}^{𝒫}{s}_{\mu }^2\Theta \left(\pm s_{\mu }\right)}=\left|{\overrightarrow{s}}_{\ast }\right|, \end{gathered}$$ (A34)

where again the sign in Θ is determined by the sign of e_y.

A formal way to see that γ is indeed given by Eq. (A34) is to start with any unit vector, ${\overrightarrow{w}}_s$, lying in the solution space. Then, the angle, γ, between ${\widehat{s}}^{\prime }$ and ${\overrightarrow{w}}_s$ is given by

$$\text{cos}\gamma =\sum _{\mu =𝒞+1}^{𝒫}{s}_{\mu }^{\prime }{w}_{s\mu }=\sum _{\mu \in A_{+}}{s}_{\mu }^{\prime }{w}_{s\mu }+\sum _{\mu \in A_{-}}{s}_{\mu }^{\prime }{w}_{s\mu },$$ (A35)

where we have defined A_± as the set of all μ indices for which $s_{\mu }^{\prime }$ is positive/negative, respectively. Since ${\overrightarrow{w}}_s$ is in the solution space, w_sμ ⩽ 0, and therefore the second term sums positive quantities while the first term subtracts. Thus,

$$\text{cos}\gamma ⩽\sum _{\mu \in A_{-}}{s}_{\mu }^{\prime }{w}_{s\mu }={\overrightarrow{s}}_{\ast }\cdot {\overrightarrow{w}}_s⩽\left|{\overrightarrow{s}}_{\ast }\right|$$ (A36)

where both the equalities are achieved when ${\overrightarrow{w}}_s$ is aligned with the boundary semiconstrained vector, ${\overrightarrow{s}}_{\ast }$, or ${\overrightarrow{w}}_s={\widehat{s}}_{\ast }$, as argued previously. Note also, that this formal proof did not assume any restrictions on $\widehat{s^{\prime }}$ direction and thus Eq. (A34) turns out to be a general result that also holds if $\widehat{s^{\prime }}$ points into the all-negative orthant.

Combining Eqs. (A34) and (A30), the certainty condition now reads

$$\begin{gathered} \phi \langle \gamma \Rightarrow \frac{y^2{\text{cot}}^2\theta {\text{cos}}^2\alpha }{W^2-y^2}\rangle {\text{cos}}^2\gamma \\ \Rightarrow y>y_{\text{cr}}\equiv W\sqrt{\frac{{\text{cos}}^2\gamma {\text{sin}}^2\theta }{{\text{cos}}^2\alpha {\text{cos}}^2\theta +{\text{cos}}^2\gamma {\text{sin}}^2\theta }}. \end{gathered}$$ (A37)

d. General case with unconstrained dimensions

We can extend the above analysis to the case when we have unconstrained dimensions by noting that, for a given set of unconstrained coordinates, the solution space is again the all-negative orthant in a semiconstrained hypersphere. Isometry along unconstrained dimensions ensures that it is always possible to make one of the null directions, lets say ${\overrightarrow{\epsilon }}_{𝒩}$, align with $\widehat{u}$. Then, the w = 0 hyperplane Eq. (A26) reads

$$w=\text{sin}\theta \text{cos}\varphi \widehat{s}\cdot {\overrightarrow{w}}_s+y\text{cos}\theta \text{cos}\alpha +{\eta }_u\text{sin}\theta \text{sin}\varphi =0,$$ (A38)

which can be rewritten as

$$\text{sin}\theta \text{cos}\varphi {\widehat{s}}^{\prime }\cdot {\overrightarrow{w}}_s=y\text{cos}\theta \left|\text{cos}\alpha \right|-{\eta }_u^{\prime }\text{sin}\theta \text{sin}\varphi ,$$ (A39)

where we have introduced ${\eta }_u^{\prime }=-\text{Sgn}(\text{cos}\alpha ){\eta }_u$. To have a certain synapse, the w = 0 hyperplane cannot intersect the solution space for any allowed value of ${\eta }_u^{\prime }$.

The direction of ${\widehat{s}}^{\prime }$ is independent of the unconstrained coordinates and hence the value of γ remains unchanged. However, the cone-angle, φ, does depend on the unconstrained coordinates in two ways. First, the radius, $\tilde{W}$, of the 𝒮-dimensional spherical subspace containing admissible solutions is now

$$\tilde{W}=\sqrt{W^2-y^2-{\eta }_{\perp }^2-{\eta }_u^{\prime 2}},$$ (A40)

where η_⊥ is the magnitude of the weight-vector in the (𝒰 − 1)-dimensional subspace that is perpendicular to ${\overrightarrow{\epsilon }}_{𝒩}=\widehat{u}$. We note in passing that Eq. (A40) implies ${\eta }_{\perp },{\eta }_u^{\prime }⩽\sqrt{W^2-y^2}=\overline{W}$. Second, the distance of the hyperplane from origin that follows from Eq. (A39) is now a function of ${\eta }_u^{\prime }$:

$$d_s=\frac{y\left|\text{cos}\theta \text{cos}\alpha \right|-{\eta }_u^{\prime }\text{sin}\theta \text{sin}\varphi }{\text{sin}\theta \text{cos}\varphi }.$$ (A41)

Strictly speaking, this expression for the distance is only valid as long as the numerator in the d_s expression stays positive. However, if there exists an allowed ${\eta }_u^{\prime }⩽\overline{W}$ (let us call it η_u0) for which the numerator can vanish, then the synapse cannot have a certain sign, because at that point d_s = 0, the hyperplane intersects the origin, and the weight can vanish even for a linear theory. In fact, the d_s = 0 condition provides us with the y-critical value below which the synapse sign becomes uncertain in a linear theory:

$${\eta }_{u0}=\frac{y\text{cos}\theta \left|\text{cos}\alpha \right|}{\text{sin}\theta \text{sin}\varphi }⩽\overline{W}\Rightarrow y_{\text{cr},\text{lin}}=W\sqrt{\frac{{\text{sin}}^2\theta {\text{sin}}^2\varphi }{{\text{cos}}^2\theta {\text{cos}}^2\alpha +{\text{sin}}^2\theta {\text{sin}}^2\varphi }}.$$ (A42)

So we will now look into cases when y ⩾ y_cr,lin which also means that Eq. (A41) will remain valid.

Combining Eqs. (A40) and (A41) we get

$$\text{cos}\phi =\frac{d_s}{\tilde{W}}=\frac{y\left|\text{cos}\theta \text{cos}\alpha \right|-{\eta }_u^{\prime }\text{sin}\theta \text{sin}\varphi }{\text{sin}\theta \text{cos}\varphi \sqrt{W^2-y^2-{\eta }_{\perp }^2-{\eta }_u^{\prime 2}}}.$$ (A43)

For us to be certain that w is nonzero, we have to make sure that even the largest φ that one can obtain by varying η_⊥ and ${\eta }_u^{\prime }$ is still smaller than γ. Clearly, to make φ large it is best to make η_⊥ = 0. Also, it is clear from inspection that cos φ starts to initially decrease as ${\eta }_u^{\prime }$ increases from zero, being dominated by the linear term. However, as the quadratic term in ${\eta }_u^{\prime }$ in the denominator becomes more and more important, cos φ reaches a minimum and starts to increase. Imposing $d\text{cos}\phi /d{\eta }_u^{\prime }=0$, we can find that this minimum is reached at

$${\eta }_u^{\prime }=\frac{\text{sin}\theta \text{sin}\varphi \left(W^2-y^2\right)}{y\text{cos}\theta \left|\text{cos}\alpha \right|}=\overline{W}\sqrt{\frac{{(W/y)}^2-1}{{\left(W/y_{\text{cr},\text{lin}}\right)}^2-1}}⩽\overline{W},$$ (A44)

where we substituted y_cr,lin from Eq. (A42) and used the fact that W ⩾ y ⩾ y_cr,lin to obtain the inequality. This proves that the minimum cos φ indeed occurs at an allowed positive value of ${\eta }_u^{\prime }⩽\overline{W}$. Substituting the above ${\eta }_u^{\prime }$ in Eq. (A43), after some algebra we find that this minimum value of cos φ, or equivalently the maximum φ, is given by

$$\text{cos}\phi =\sqrt{\frac{y^2{\text{cos}}^2\theta {\text{cos}}^2\alpha -\left(W^2-y^2\right){\text{sin}}^2\theta {\text{sin}}^2\varphi }{\left(W^2-y^2\right){\text{cos}}^2\varphi {\text{sin}}^2\theta }}.$$ (A45)

The certainty condition then requires

$${\text{cos}}^2\phi =\frac{y^2{\text{cos}}^2\theta {\text{cos}}^2\alpha -\left(W^2-y^2\right){\text{sin}}^2\theta {\text{sin}}^2\varphi }{\left(W^2-y^2\right){\text{cos}}^2\varphi {\text{sin}}^2\theta }>{\text{cos}}^2\gamma ,$$ (A46)

which can be recast as

$$y>y_{\text{cr}}\equiv W\sqrt{\frac{{\text{cos}}^2\gamma {\text{sin}}^2\theta {\text{cos}}^2\varphi +{\text{sin}}^2\theta {\text{sin}}^2\varphi }{{\text{cos}}^2\alpha {\text{cos}}^2\theta +{\text{cos}}^2\gamma {\text{sin}}^2\theta {\text{cos}}^2\varphi +{\text{sin}}^2\theta {\text{sin}}^2\varphi }},$$ (A47)

It is illuminating to express y-critical in terms of the projections, e_y, e_u, e_s∗, of the synaptic direction, $\widehat{e}$, respectively along the data vector, $\widehat{y}$, the unconstrained unit vector, $\widehat{u}$, and the semiconstrained boundary vector, ${\overrightarrow{s}}_{\ast }$:

$$y_{\text{cr}}=W\sqrt{\frac{e_{s*}^2+e_u^2}{e_y^2+e_{s*}^2+e_u^2}},$$ (A48)

where

$$e_y\equiv \widehat{e}\cdot \widehat{y}=\frac{{\sum }_{\mu =1}^{𝒞}{y}_{\mu }{e}_{\mu }}{\sqrt{{\sum }_{\mu =1}^{𝒞}{y}_{\mu }^2}}=\text{cos}\theta \text{cos}\alpha ,\hspace{1em}{e}_{s*}\equiv \widehat{e}\cdot {\widehat{s}}_{*}=\sqrt{\sum _{\mu \in A_{-}}{e}_{\mu }^2}=-\text{Sgn}(\text{cos}\alpha )\text{sin}\theta \text{cos}\varphi \text{cos}\gamma ,$$ (A49)

and e_u is given by Eq. (A22). We note that setting ϕ = 0 precisely reproduces the correct limit with no unconstrained directions (A37).

3. Regarding orthogonal input patterns in recurrent networks

While our analysis of the solution space and the certainty condition (A48) translate directly to recurrent networks, the requirement of orthogonality for the derivation of our certainty condition imposes certain technical restrictions on its scope when it comes to recurrent neural networks.

The certainty condition we derived for feedforward networks can be applied to two different recurrent neural network set ups. First, let us consider networks where neurons have self-couplings. A consequence of having orthogonal response patterns in this case is that the certainty condition can only be satisfied for self-couplings w_ii, as long as W ⩾ 1. This is because the imposition of orthogonality in response patterns also restricts the correlation between the target neuron and the other neurons:

$$\sum _{m=1}^{𝒩}{Z}_{\mu m}{Z}_{vm}={\delta }_{\mu \nu }\Rightarrow \sum _{\mu =1}^{𝒩}{Z}_{\mu m}{Z}_{\mu n}={\delta }_{mn}.$$ (A50)

However, for the synapse-sign to be certain, the responses of pre and postsynaptic neurons need to be correlated. To see the problem more quantitatively, suppose we are interested in constraining the synapse from the mth neuron onto the ith neuron, as before. Now, the first P elements of the unit vectors, ${\widehat{e}}_i$ and ${\widehat{e}}_m$ contain the responses of the ith and the mth neuron in the 𝒫 patterns. We have already derived a decomposition of e_m in terms of its projections onto the constrained, semiconstrained and unconstrained subspaces (A20). Similarly, ${\widehat{e}}_i$ can be decomposed as

$${\widehat{e}}_i=y\widehat{y}+\sqrt{1-y^2}{\widehat{y}}_{\perp },$$ (A51)

where $\widehat{y}$ lies entirely along the constrained directions, and ${\widehat{y}}_{\perp }$ is orthogonal to it and only has components along unconstrained directions. Then, orthogonality implies

$$\begin{gathered} {\widehat{e}}_i\cdot {\widehat{e}}_m=y\text{cos}\theta \text{cos}\alpha +\sqrt{1-y^2}\text{sin}\theta \text{sin}\varphi \left({\widehat{y}}_{\perp }\cdot \widehat{u}\right) \\ =0\Rightarrow \text{sin}\theta \text{sin}\varphi =-\frac{y\text{cos}\theta \text{cos}\alpha }{\left({\widehat{y}}_{\perp }\cdot \widehat{u}\right)\sqrt{1-y^2}}. \end{gathered}$$ (A52)

Starting from the certainty condition (A48), we can now go through a sequence of (in)equalities:

$$\begin{gathered} y^2>W^2\frac{{\text{sin}}^2\theta {\text{sin}}^2\varphi +{\text{sin}}^2\theta {\text{cos}}^2\varphi {\text{cos}}^2\gamma }{{\text{sin}}^2\theta {\text{sin}}^2\varphi +{\text{sin}}^2\theta {\text{cos}}^2\varphi {\text{cos}}^2\gamma +{\text{cos}}^2\theta {\text{cos}}^2\alpha } \\ ⩾\frac{W^2{\text{sin}}^2\theta {\text{sin}}^2\varphi }{{\text{sin}}^2\theta {\text{sin}}^2\varphi +{\text{cos}}^2\theta {\text{cos}}^2\alpha } \\ =\frac{W^2{y}^2{\text{cos}}^2\theta {\text{cos}}^2\alpha }{y^2{\text{cos}}^2\theta {\text{cos}}^2\alpha +\left(1-y^2\right){\left({\widehat{y}}_{\perp }\cdot \widehat{u}\right)}^2{\text{cos}}^2\theta {\text{cos}}^2\alpha } \\ =\frac{W^2{y}^2}{y^2+\left(1-y^2\right){\left({\widehat{y}}_{\perp }\cdot \widehat{u}\right)}^2}⩾W^2{y}^2, \end{gathered}$$ (A53)

where we substituted sin θ sin ϕ from Eq. (A52). Note that the RHS is minimized when $\widehat{u}$ and ${\widehat{y}}_{\perp }$ are either aligned or antialigned. Even in this case, RHS = W²y², and thus the certainty condition cannot be satisfied if W ⩾ 1. One can check that when i = m, because the RHS in the first equation of (A52) is one and not zero, no similar constraints appear. Indeed, the certainty condition may be satisfied depending upon the specific response patterns.

As a second possibility, suppose that no self-couplings are present. Then to be able to apply our framework and determine the couplings w_im for a given i, we only need the truncated row vectors of z whose ith column entry is absent, to be orthonormal. Therefore, the response of the ith driven neuron, which consists of the entries of the ith column, can now be chosen independently from the responses of its input neurons. In other words, ${\widehat{e}}_i$ and ${\widehat{e}}_m$, m ≠ i, no longer need to satisfy orthogonality constraint of Eq. (A52). Consequently, the w_im weights can indeed satisfy the certainty condition, just as in the feedforward case.

4. Implied conservative bound on the certainty condition for nonorthogonal input patterns

A complete treatment of the certainty conditions for nonorthogonal fixed point patterns is beyond the scope of this work. However, here we provide some preliminary results and insights by explaining how our formalism for analyzing orthogonal fixed-point patterns can be simply adapted to derive an exact, but conservative, upper bound for y-critical that applies to general sets of patterns.

Conceptually speaking, deriving the certainty condition amounts to determining when the w_m = 0 hyperplane intersects the solution space within the sphere of weight vectors with norm at most W. Because the solution space is exceedingly simple in η coordinates, our orthogonal analysis used η coordinates to conveniently recast the equations for the bounding sphere and w_m = 0 hyperplane. To adapt this analysis to the nonorthogonal case, it is important to account for three important changes to the geometry of the problem. Most fundamentally, the ${\overrightarrow{\epsilon }}_{\mu }$ directions corresponding to η coordinates are no longer orthonormal. However, the mathematical notions of orthogonality and normality are implicitly defined with respect to the inner-product structure imposed on the vector space, and our geometrical calculations from the orthogonal case easily carry over to the nonorthogonal case if we redefine the inner product structure of the weight space to give an orthonormal coordinate system with respect to the η coordinates rather than the physical coordinates.¹⁵ In practice, all that this will entail is interpreting the η coordinates as if they define coordinates along orthogonal axes, and we will never need to explicitly write down the associated inner product. Second, the equation for the weight bound in the orthogonal system of η coordinates is elliptical around the origin, rather than spherical. However, for any ellipse one can find a sphere that just encompasses it. If we can find the radius of this bounding sphere, then one can look for an intersection anywhere within this sphere and our geometrical approach for deriving Eq. (A48) will carry over and provide a conservative bound for y-critical. This bound will poorly approximate the true y-critical when some axes of the ellipse are much longer than others. Third, the normal vector to the w_m = 0 hyperplane is no longer ${\widehat{e}}_m$ in the orthogonal system of η coordinates. Therefore, the projections of ${\widehat{e}}_m$ in Eq. (A48) must be generalized to become projections of the hyperplane’s normal vector.

To obtain the radius of the bounding sphere, consider the SVD decomposition of the data-matrix:

$$z=L\Lambda R^T,$$ (A54)

where L and R are 𝒫 × 𝒫 and 𝒩 × 𝒩 orthogonal rotation matrices, and Λ is a 𝒫 × 𝒩 rectangular diagonal matrix whose only nonzero entries are given by

$${\Lambda }_{aa}={\lambda }_a⩾0,\hspace{1em}a=1\dots 𝒫.$$ (A55)

Note that λ₁,·⋯ ,λ_𝒫 are called the singular values of z. We can now define rotated coordinates:

$${\eta }^{\prime }=L^T\eta ,\hspace{1em}\text{and}{w}^{\prime }=R^{T}w,$$ (A56)

so that

$${\eta }^{\prime }=L^{T}L\Lambda R^{T}w=\Lambda w^{\prime }\Rightarrow {\eta }_a^{\prime }={\lambda }_a{w}_a^{\prime }\forall a=1\dots 𝒫.$$ (A57)

Note that η and η′ are 𝒫 vectors in the current notation. We also note that since w′ is just a rotation of the original synaptic coordinates, the biological bound does not change as we go from w to w′ coordinates:

$$\sum _a{w}_a^{\prime 2}=\sum _m{w}_m^2⩽W^2.$$ (A58)

This makes it possible to find an inequality in terms of the η′ coordinates:

$$\begin{gathered} W^2⩾\sum _{a=1}^{𝒩}{w}_a^{\prime 2}=\sum _{a=1}^{𝒫}\frac{{\eta }_a^{\prime 2}}{{\lambda }_a^2}+\sum _{a=𝒫+1}^{𝒩}{w}_a^{\prime 2} \\ ⩾\frac{1}{{\lambda }_{\text{max}}^2}\sum _{a=1}^{𝒫}{\eta }_a^{\prime 2}+\sum _{a=𝒫+1}^{𝒩}{w}_a^{\prime 2} \\ =\frac{1}{{\lambda }_{\text{max}}^2}\sum _{\mu =1}^{𝒫}{\eta }_{\mu }^2+\sum _{a=𝒫+1}^{𝒩}{w}_a^{\prime 2}, \end{gathered}$$ (A59)

where in the last step we have used the fact that the orthogonal matrix L does not change the L2-norm as one goes from η′ to η coordinates, and λ_max is defined as the maximal singular value. To obtain spherical symmetry, we thus define unconstrained coordinates via

$${\eta }_{\mu }={\lambda }_{\text{max}}{w}_{\mu }^{\prime },\hspace{1em}\forall \mu >𝒫,$$ (A60)

such that

$$W^{\prime 2}\equiv {\lambda }_{\text{max}}^2{W}^2⩾\sum _{\mu =1}^{𝒩}{\eta }_{\mu }^2.$$ (A61)

We now realize that the problem of finding y-critical using this conservative bound can be recast into the problem of the orthogonal case: As we just described, the η coordinates satisfy the conservative spherical bound. Our goal can then be to find the minimum value of y for which the hyperplane satisfying, w_m = 0, does not intersect the solution space. Now, if we define Z to be, as in the orthogonal case, a full rank extension¹⁶ of z:

$$Z_{\mu m}=\{\begin{array}{ll}{z}_{\mu m},\hfill & \forall \mu ⩽𝒫,\hfill \\ {\lambda }_{\text{max}}{R}_{\mu m}^T,\hfill & \forall \mu >𝒫,\hfill \end{array}$$ (A63)

so that for all value of μ,

$${\eta }_{\mu }=\sum _{m=1}^{𝒩}{Z}_{\mu m}{w}_m,$$ (A64)

then the hyperplane equation can be rewritten as

$$w_m=\sum _{\mu =1}^{𝒩}{Z}_{m\mu }^{-1}{\eta }_{\mu }=0.$$ (A65)

From Eq. (A65) it is clear that

$${\overrightarrow{n}}_m=\sum _{\mu =1}^{𝒩}{Z}_{m\mu }^{-1}{\overrightarrow{\epsilon }}_{\mu }$$ (A66)

is perpendicular to the w_m = 0 hyperplane, where we remind the readers that ${\overrightarrow{\epsilon }}_{\mu }$’s were defined by Eq. (A9). ${\widehat{n}}_m$ thus plays the role of ${\widehat{e}}_m$ whose orientation with respect to the constrained, semiconstrained and unconstrained dimensions determines the certainty condition. Specifically,

$$y_{\text{cr}}=W^{\prime }\sqrt{\frac{n_{s*}^2+n_u^2}{n_y^2+n_{s*}^2+n_u^2}}=W{\lambda }_{\text{max}}\sqrt{\frac{n_{s*}^2+n_u^2}{n_y^2+n_{s*}^2+n_u^2}},$$ (A67)

where the various projections of ${\widehat{n}}_m$ are given by

$$\begin{gathered} n_y\equiv \frac{{\sum }_{\mu =1}^{𝒞}{y}_{\mu }{Z}_{m\mu }^{-1}}{\sqrt{{\sum }_{\mu =1}^{𝒞}{y}_{\mu }^2}\sqrt{{\sum }_{\mu =1}^{𝒩}{\left(Z_{m\mu }^{-1}\right)}^2}}, \\ {n}_{s*}\equiv \frac{\sqrt{{\sum }_{\mu \in A_{-}}{\left(Z_{m\mu }^{-1}\right)}^2}}{\sqrt{{\sum }_{\mu =1}^{𝒩}{\left(Z_{m\mu }^{-1}\right)}^2}}, \\ \text{and}\hspace{1em}{n}_u\equiv \frac{\sqrt{{\sum }_{\mu =𝒫+1}^{𝒩}{\left(Z_{m\mu }^{-1}\right)}^2}}{\sqrt{{\sum }_{\mu =1}^{𝒩}{\left(Z_{m\mu }^{-1}\right)}^2}}. \end{gathered}$$ (A68)

We remind the readers that in the orthogonal case the set A₋ contained each semiconstrained μ index along which the component of ${\widehat{e}}_m$ had the same sign as e_y. Similarly, here A₋, contains those semiconstrained μ indices along which the component of ${\overrightarrow{n}}_m$ have the same sign as n_y. To summarize, the above analysis suggests that both Z and Z⁻¹ will play an important role in generalizing the certainty condition to nonorthogonal patterns, and especially the relative orientation of ${\widehat{n}}_m$ (defined by Z⁻¹) with respect to the η directions.

Appendix B: ESTIMATING THE PROBABILITY THAT A SYNAPSE IS CERTAIN IN LARGE FEEDFORWARD NETWORKS

For given values of (= ℐ), 𝒞, 𝒫 in a feedforward setting we will here try to assess how likely it is that noiseless orthonormal neuronal responses require a given synapse to be nonzero. As we have seen in Eq. (49), whether a synapse is certain to exist depends on six parameters, θ, φ, γ, α, W, and y. The first four quantities depend on how $\widehat{e}$ is oriented with respect to various directions in the weight space. Since $\widehat{e}$ is a unit vector, typically we expect its component along any given direction to be $~𝒪(1/\sqrt{𝒩})$. Thus, we typically expect

$$\begin{gathered} e_y^2={\text{cos}}^2\theta {\text{cos}}^2\alpha ~\frac{1}{𝒩};\hspace{1em}{e}_s^2={\text{sin}}^2\theta {\text{cos}}^2\varphi ~\frac{𝒮}{𝒩}; \\ {e}_u^2={\text{sin}}^2\theta {\text{sin}}^2\varphi ~\frac{𝒰}{𝒩}\hspace{1em}\text{and}{e}_{s*}^2={\text{cos}}^2\gamma e_s^2~\frac{𝒮}{2𝒩}. \end{gathered}$$ (B1)

Hence, we approximate the typical y_cr as

$$y_{\text{cr}}=W\sqrt{\frac{𝒮/(2𝒩)+𝒰/𝒩}{1/𝒩+𝒮/(2𝒩)+𝒰/𝒩}}=W\sqrt{\frac{𝒮+2𝒰}{2+𝒮+2𝒰}}.$$ (B2)

Let us now suppose that all the dimensions scale with the network size, such that

$$𝒮=\sigma 𝒩\hspace{1em}\text{and}𝒰=v𝒩.$$ (B3)

Then, we find that as the network size increases y_cr behaves as

$$\frac{y_{\text{cr}}}{W}\approx 1-\left(\frac{1}{\sigma +2v}\right)\frac{1}{𝒩},$$ (B4)

and y_cr is essentially pushed up toward W.

However, the typical scale of y behaves similarly as the dimensions increase. To see this concretely, let us define ${\overrightarrow{y}}_{\text{cons}}\equiv \left\{y_{\mu }\mid \mu =1\dots 𝒞\right\}$ as a 𝒞-dimensional vector, and assume that every possible ${\overrightarrow{y}}_{\text{cons}}$ is equally likely within a sphere of radius W (larger activity levels of the target neuron admit no solutions). Then the average and median values of $y=\left|{\overrightarrow{y}}_{\text{cons}}\right|$ are given by

$$\begin{gathered} \langle y\rangle =\frac{{\int }_0^{W}dy{y}^{𝒞}}{{\int }_0^{W}dy{y}^{𝒞-1}}=W\left(\frac{𝒞}{𝒞+1}\right) \\ \Rightarrow \frac{\langle y\rangle }{W}=\frac{1}{1+1/(\eta 𝒩)}\approx 1-\frac{1}{\eta 𝒩},\hspace{1em}\text{where}\eta \equiv \frac{𝒞}{𝒩}, \\ \text{and}\hspace{1em}\frac{{\int }_0^{y_M}dy{y}^{𝒞-1}}{{\int }_0^{W}dy{y}^{𝒞-1}}=\frac{1}{2}\Rightarrow \frac{y_M}{W}={\left(\frac{1}{2}\right)}^{\frac{1}{𝒞}} \\ \Rightarrow \frac{y_M}{W}\approx 1-\frac{\text{ln}2}{\eta 𝒩}, \end{gathered}$$ (B5)

respectively. Since y and y_cr scale similarity as one increases the network size, the probability of a synapse being certain should not change as the network size increases. In Fig. 8(c), we show that if we choose, y = 1 − ln 2/𝒞, as the approximate median value in simulations with random input-output configurations (see Appendix F for details), then the number of certain synapses does indeed increase linearly with 𝒩.

To quantitatively estimate the probability of finding a certain synapse, we can compute the fraction of volume of ${\overrightarrow{y}}_{\text{cons}}$’s for which the synapse is certain for the typical projections (B2), as compared to the volume of ${\overrightarrow{y}}_{\text{cons}}$’s for which solutions to the steady-state equations exist. We know that ${\overrightarrow{y}}_{\text{cons}}$ has to lie within a 𝒞-dimensional sphere of radius W in order for there to be any solutions to the problem.¹⁷ However, for the synapse sign to be certain, we need $W⩾\left|{\overrightarrow{y}}_{\text{cons}}\right|=y>y_{\text{cr}}$, where y_cr is given by Eq. (B2). So, we need to compare the spherical shell volume, $V_{W⩾y>y_{\text{cr}}}$, with the volume of the 𝒞-dimensional sphere, V_y⩽W. To find $V_{W⩾y>y_{\text{cr}}}$, we have to subtract the 𝒞-dimensional spherical volume with radius y_cr from the spherical volume with radius W. Since n-dimensional spherical volumes scale as the nth power of the radius, the probability, P, of ascertaining the sign of the synapse is approximately given by

$$P\approx \frac{V_{W⩾y>y_{\text{cr}}}}{V_{y⩽W}}=\frac{W^{𝒞}-y_{\text{cr}}^{𝒞}}{W^{𝒞}}=1-{\left(\frac{y_{\text{cr}}}{W}\right)}^{𝒞}.$$ (B6)

Now, when 𝒮, 𝒰 ≫ 1 we can approximately evaluate the RHS as follows:

$$\begin{gathered} {\left(\frac{y_{\text{cr}}}{W}\right)}^{𝒞}={\left(\frac{𝒮+2𝒰}{2+𝒮+2𝒰}\right)}^{𝒞/2}={\left(1+\frac{2}{𝒮+2𝒰}\right)}^{-𝒞/2} \\ \Rightarrow \text{ln}\left[{\left(\frac{y_{\text{cr}}}{W}\right)}^{𝒞}\right]=-\frac{𝒞}{2}\text{ln}\left(1+\frac{2}{𝒮+2𝒰}\right) \\ =\frac{-𝒞}{𝒮+2𝒰}\left[1+𝒪\left(\frac{1}{𝒮+2𝒰}\right)\right]. \end{gathered}$$ (B7)

Thus, we get

$$P\approx 1-e^{-\frac{𝒞}{𝒮+2𝒰}}.$$ (B8)

The most prominent feature of Eq. (B8) is that the probability only depends on the ratios of the various dimensions. Hence, it does not change as we increase the size of the network as long as the ratios are kept constant.

For the purpose of illustration and numerically testing this feature we assessed how certainty predictions changed when the network size is increased while holding the ratios between 𝒞, 𝒮, and 𝒰 fixed. In Fig. 8(c) we have plotted the number of certain synapses in simulations generated from random data as we scale up 𝒩 maintaining the ratios between 𝒞, 𝒮, and 𝒰 (see Appendix F for more details). We illustrate two cases. In the first example, no unconstrained directions were present, and 𝒮 = 3𝒞. Then P = 1 − e^−1/3 ≈ 0.28, so one has a 28% chance of being able to determine the sign of the connections. This answer incidentally is the same as an example with 𝒞 = 𝒮 = 𝒰. As another example, Fig. 8(c) considered the case when 𝒰 = 2𝒞 = 2𝒮. According to Eq. (B8), then P = 1 − e^−1/5 ≈ 0.18, so the chance of determining the sign drops to about 18%. We only expect these numbers to be approximate. For example, our arguments relied on the assumption that all target responses admitting solutions are equally likely, an assumption that definitely needs to be revisited for realistic networks. However, the scaling behavior should hold for other probabilistic distributions as long as the scale of ${\overrightarrow{y}}_{\text{cons}}$ behaves similar to Eq. (B5) with increasing 𝒩.

APPENDIX C: NONZERO-ERROR CERTAINTY CONDITIONS

There are various reasons why we may want to not only consider weights that exactly reproduce the specified neuronal responses, but also weights that do so approximately. For instance, we are always limited by the accuracy of the measurement apparatus. More importantly, there are various sources of biological noise that typically lead to uncertainties in observed values of neuronal responses. For the purpose of this paper we will consider any set of weights to be part of the ε-error solution space if it is able to reproduce the specified neuronal responses with an error ⩽ ε [see Eq. (63) for definition of E]. We will neglect uncertainties in the input responses to the target neuron, but we will comment on their possible effects toward the end of this Appendix.

1. Errors in feedforward networks

Let us first focus on feedforward networks. Allowing for error increases the value of y_cr by expanding the solution space. One way to think about this is to realize that we have to now make sure that Eq. (51) is satisfied for any nonnegative $\overrightarrow{\tilde{y}}=\overrightarrow{y}+\overrightarrow{\delta }$, where $\overrightarrow{y}$ and $\overrightarrow{\delta }$ are vectors in the 𝒫-dimensional activity-constrained subspace, the former representing the observed responses, and the latter coming from noise. We will initially assume that all the observed responses are nonnegative, so a zero-error solution is possible and the noise is bounded by $|\overrightarrow{\delta }|⩽\epsilon$. Our strategy will be to first seek the minimum y needed to have a certain synapse for a given $\overrightarrow{\delta }$. We then find the maximum among these y-critical values as we let $\overrightarrow{\delta }$ vary within the ε-ball. Since this procedure will guarantee that the w = 0 hyperplane does not intersect the entire solution space with ℇ ⩽ ε, this means that the synapse must exist for the network to generate the specified responses patterns. The synapse’s sign will match the zero-error analysis. We will first estimate y-critical when the error is small enough to not induce topological transitions in the error surface. In the subsequent sections, we will include the effects of topological transitions, as well as explain how to deal with situations where some of the observed responses are negative, which is possible due to noise.

a. When all observed responses are nonnegative and no topological transitions occur

To understand how errors affect the certainty conditions, let us consider the case where the observed responses are nonnegative and the allowed error satisfies 0 < ε < min{y_μ}_{μ=1, …, 𝒞}, so that no topological transitions can occur. If some responses that were zero in $\overrightarrow{y}$ are now nonzero in $\overrightarrow{\tilde{y}}$, then both e_y and e_s∗ can change due to the noise. Withoutloss of generality, let us assume that $\overrightarrow{\delta }$ only has nonzero components along μ = 𝒞 + 1, …, 𝒬 semiconstrained dimensions,¹⁸ as well as along some (or all) of the constrained dimensions. Then e_y changes to

$${\tilde{e}}_y=\frac{\widehat{e}\cdot (\overrightarrow{y}+\overrightarrow{\delta })}{|\overrightarrow{y}+\overrightarrow{\delta }|}.$$ (C1)

Furthermore, if some previously semiconstrained components that contributed to $\widehat{s_{*}}$ have now become constrained,¹⁹ then $\widehat{s_{*}}$ no longer has thosecomponents. This means that we haveto subtract these components from e_s∗:

$${\widehat{\tilde{s}}}_{\ast }=\widehat{s_{\ast }}-\sum _{\mu =𝒞+1}^{𝒬}{A}_{\mu }{e}_{\mu }{\overrightarrow{\epsilon }}_{\mu }\Rightarrow {\tilde{e}}_{s\ast }^2=e_{s\ast }^2-\sum _{\mu =𝒞+1}^{𝒬}{A}_{\mu }{e}_{\mu }^2,$$ (C2)

where A_μ is 1 if $(\widehat{y}\cdot \widehat{c})$ and e_μ have the same sign and 0 otherwise. This follows from the definition of the boundary projection vector (A33) and e_s∗ (A49). Thus, for a given $\overrightarrow{\delta }$ the certainty condition (A48) yields

$$\begin{gathered} |\overrightarrow{\tilde{y}}{|}^2={\left|\overrightarrow{y}+\overrightarrow{\delta }\right|}^2>W^2\left(\frac{{\tilde{e}}_{s\ast }^2+e_u^2}{{\tilde{e}}_y^2+{\tilde{e}}_{s\ast }^2+e_u^2}\right) \\ \Rightarrow |\overrightarrow{y}+\overrightarrow{\delta }{|}^2\left(\frac{{\tilde{e}}_y^2}{{\tilde{e}}_{s\ast }^2+e_u^2}+1\right)>W^2 \\ \Rightarrow \frac{|\widehat{e}\cdot (\overrightarrow{y}+\overrightarrow{\delta }){|}^2}{{\tilde{e}}_{s\ast }^2+e_u^2}+|\overrightarrow{y}+\overrightarrow{\delta }{|}^2>W^2. \end{gathered}$$ (C3)

As before, one can interpret the above inequality as equivalently specifying either y-critical or W-critical. For a fixed $\overrightarrow{y}$, one can obtain a minimum value of the left-hand side (LHS) of the latter inequality by varying $\overrightarrow{\delta }$ within the ε ball. The square root of this is W-critical. Then as long as W is less than W-critical, we will have a certain synapse. Inverting the relation, one finds y-critical as the minimum y needed to make the synapse sign certain for all $\overrightarrow{\delta }$ and given $\widehat{y}$ and W. More explicitly, equating the two sides of the inequality forany given $\widehat{y}$, $\overrightarrow{\delta }$, and W, we get a minimal y that depends on $\overrightarrow{\delta }$. To find y-critical, we have to take the maximum of the minimal y as we vary over all possible $\overrightarrow{\delta }$ in the ε-ball.

Let us first obtain a lower bound on y-critical. By inspection of the LHS of the above inequality, it is clear that the more the $\overrightarrow{\delta }$-dependent terms can cancel the $\overrightarrow{y}$-dependent terms, the harder it is to satisfy the certainty condition. We observe that in Eq. (C3), the second term is minimized when $\overrightarrow{\delta }=-\epsilon \widehat{y}$.²⁰ Accordingly, one can obtain a lower bound on y-critical by substituting $\delta =-\epsilon \widehat{y}$ in Eq. (C3):

$$\begin{gathered} {\left(\widehat{e}\cdot \widehat{y}\right)}^2{\left(y-\epsilon \right)}^2+\left(e_{s\ast }^2+e_u^2\right){\left(y-\epsilon \right)}^2>W^2\left(e_{s\ast }^2+e_u^2\right),\text{or}, \\ y>y_{\text{cr},\text{min},0}\equiv W\sqrt{\frac{e_{s\ast }^2+e_u^2}{e_y^2+e_{s\ast }^2+e_u^2}}+\epsilon , \end{gathered}$$ (C4)

where we have used $e_y=\widehat{e}\cdot \widehat{y}$ and ${\tilde{e}}_{s*}^2=e_{s*}^2$, since $\overrightarrow{\delta }$ has no components along the semiconstrained directions. We will see later that this simple lower bound can approximate the actual y-critical very well in many situations. Notice that we used a subscript “0” to denote this lower bound. This is because, as we will soon see, when noise allows for topological transitions, one may be able to obtain stricter lower bounds by allowing some constrained dimensions to behave as semiconstrained. This “0” emphasized that no constrained indices behave as semiconstrained. Next, we can find an upper bound for y-critical by noting

$$\begin{gathered} \frac{|\widehat{e}\cdot (\overrightarrow{y}+\overrightarrow{\delta }){|}^2}{{\tilde{e}}_{s\ast }^2+e_u^2}+|\overrightarrow{y}+\overrightarrow{\delta }{|}^2 \\ ⩾\frac{|\widehat{e}\cdot (\overrightarrow{y}+\overrightarrow{\delta }){|}^2}{e_{s\ast }^2+e_u^2}+|\overrightarrow{y}+\overrightarrow{\delta }{|}^2 \\ ⩾\frac{{(\widehat{e}\cdot \overrightarrow{y})}^2+2(\widehat{e}\cdot \overrightarrow{y})(\widehat{e}\cdot \overrightarrow{\delta })}{e_{s\ast }^2+e_u^2}+y_0^2+2\overrightarrow{y}\cdot \overrightarrow{\delta }, \end{gathered}$$ (C5)

where the first inequality is true because ${\tilde{e}}_{s*}^2⩽e_{s*}^2$, and the second inequality because we have dropped positive (δ²) terms. Then we can obtain an upper bound on y-critical by finding a y such that even the last expression on the RHS is greater than W². Specifically,

$${\left(\widehat{e}\cdot \overrightarrow{y}\right)}^2+2\left(\widehat{e}\cdot \overrightarrow{y}\right)\left(\widehat{e}\cdot \overrightarrow{\delta }\right)+\left(e_{s\ast }^2+e_u^2\right)\left(y_0^2+2\overrightarrow{y}\cdot \overrightarrow{\delta }\right)>W^2\left(e_{s\ast }^2+e_u^2\right).$$ (C6)

So, let us try to find the $\overrightarrow{\delta }$ that minimizes the LHS:

$$\begin{gathered} \text{LHS}=y^2{e}_y^2+2y{e}_y(\widehat{e}\cdot \overrightarrow{\delta })+\left(e_{s\ast }^2+e_u^2\right)\left(y^2+2y\widehat{y}\cdot \overrightarrow{\delta }\right) \\ =y^2\left(e_y^2+e_{s\ast }^2+e_u^2\right)+2y\left(e_y\widehat{e}+\left(e_{s\ast }^2+e_u^2\right)\widehat{y}\right)\cdot \overrightarrow{\delta } \\ =y^2\left(e_y^2+e_{s\ast }^2+e_u^2\right)+2y\overrightarrow{\xi }\cdot \overrightarrow{\delta }, \end{gathered}$$ (C7)

where $\overrightarrow{\xi }\equiv e_y{\sum }_{\mu =1}^{𝒫}{e}_{\mu }{\overrightarrow{\epsilon }}_{\mu }+\left(e_{s\ast }^2+e_u^2\right)\widehat{y}$, and we have noted that $\widehat{u}\cdot \overrightarrow{\delta }=0$ because $\overrightarrow{\delta }$ must be in the activity-constrained subspace. It is now clear that LHS is minimized if $\overrightarrow{\delta }$ anti-aligns with $\overrightarrow{\xi }$. Then Eq. (C6) yields

$$y^2\left(e_y^2+e_{s\ast }^2+e_u^2\right)-2y\left|\overrightarrow{\xi }\right|\epsilon >W^2\left(e_{s\ast }^2+e_u^2\right).$$ (C8)

Equating the two sides of Eq. (C8) and solving for y,²¹ we now get an upper bound for y-critical:

$$y_{\text{cr},\text{max},0}\equiv \sqrt{W^2\left(\frac{e_{s\ast }^2+e_u^2}{e_y^2+e_{s\ast }^2+e_u^2}\right)+\frac{{\epsilon }^2{\xi }^2}{{\left(e_y^2+e_{s\ast }^2+e_u^2\right)}^2}}+\frac{\epsilon \xi }{e_{s\ast }^2+e_u^2+e_y^2},$$ (C9)

where ξ is the norm of $\overrightarrow{\xi }$ and can be simplified as

$$\begin{gathered} {\xi }^2=\sum _{\mu =1}^{𝒫}{\left[e_y{e}_{\mu }+\left(e_{s\ast }^2+e_u^2\right){\widehat{y}}_{\mu }\right]}^2 \\ =e_y^2\sum _{\mu =1}^{𝒫}{e}_{\mu }^2+{\left(e_{s\ast }^2+e_u^2\right)}^2\sum _{\mu =1}^{𝒞}{\widehat{y}}_{\mu }^2+2\left(e_{s\ast }^2+e_u^2\right)e_y\sum _{\mu =1}^{𝒞}{\widehat{y}}_{\mu }{e}_{\mu } \\ =e_y^2{e}_p^2+{\left(e_{s\ast }^2+e_u^2\right)}^2+2\left(e_{s\ast }^2+e_u^2\right)e_y^2 \\ ={\left(e_y^2+e_{s\ast }^2+e_u^2\right)}^2+e_y^2\left(e_p^2-e_y^2\right). \end{gathered}$$ (C10)

Thus, we have

$$y_{\text{cr},\text{max},0}\equiv \sqrt{W^2\left(\frac{e_{s\ast }^2+e_u^2}{e_y^2+e_{s\ast }^2+e_u^2}\right)+{\epsilon }^2\left(1+\frac{e_y^2\left(e_p^2-e_y^2\right)}{{\left(e_y^2+e_{s\ast }^2+e_u^2\right)}^2}\right)}+\epsilon \sqrt{1+\frac{e_y^2\left(e_p^2-e_y^2\right)}{{\left(e_y^2+e_{s\ast }^2+e_u^2\right)}^2}}.$$ (C11)

As with lower bound, we will see that to obtain the correct upper bound in presence of topological transitions, one has to maximize over several upper bounds. Hence, we refer the above upper bound that does not include any effects from topological transitions with an index “0.”

Finally, we would like to point out that for small errors one can also obtain an approximate correction to y-critical that lies in between y_cr,min,0 and y_cr,max,0. To obtain this estimate, let us first write down the bound on y that one would obtain from Eq. (C3) as δ → 0:

$$y>W\sqrt{\frac{{\tilde{e}}_{s\ast }^2+e_u^2}{{\tilde{e}}_y^2+{\tilde{e}}_{s\ast }^2+e_u^2}}.$$ (C12)

If $\overrightarrow{\delta }$ has components along any semiconstrained direction that contributes toward the original ${\widehat{e}}_{s\ast }$ vector, then ${\tilde{e}}_{s\ast }^2<e_{s\ast }^2$, and comparing Eqs. (A48) and (C12) we see that, as δ → 0, the bound on y will be less than the zero-error y_cr. In other words, for sufficiently small errors, if $\overrightarrow{\delta }$ explores directions that contribute to e_s∗, then the corresponding bound on y is going to be smaller than even the zero-error y_cr. Thus, for these small errors the leading order corrections to Eq. (A48) is obtained only if δ do not have any components along these semiconstrained directions. This means ${\tilde{e}}_{s\ast }^2=e_{s\ast }^2$, and we can reorder the indices such that the semiconstrained directions along which excursions of $\overrightarrow{\delta }$ will be considered range from 𝒞 + 1, …, 𝒬; i.e., $\text{Sgn}\left({\widehat{s}}_{\mu }^{\prime }\right)$ is negative for these, and only these, semiconstrained indices. To obtain the certainty condition, one can then follow steps (C5)²² through (C9) except that $\overrightarrow{\delta }$ is restricted to only have nonzero components along constrained directions those semiconstrained directions that do not contribute to ${\widehat{e}}_{s\ast }$, i.e., for μ = 1 … 𝒬. In other words, it can at the most anti-align with a truncated $\overrightarrow{\xi }$,

$$\begin{gathered} {\overrightarrow{\xi }}_{\text{trunc}}\equiv \sum _{\mu =1}^{𝒬}{\xi }_{\mu }{\overrightarrow{\epsilon }}_{\mu }=\sum _{\mu =1}^{𝒬}\left[e_y{e}_{\mu }+\left(e_{s\ast }^2+e_u^2\right){\widehat{y}}_{\mu }\right]{\overrightarrow{\epsilon }}_{\mu },\hspace{1em}\text{and} \\ {\left(\overrightarrow{\xi }\cdot \overrightarrow{\delta }\right)}_{\text{min}}=-\epsilon {\xi }_{\text{trunc}}, \end{gathered}$$ (C13)

where ξ_trunc is the norm of ${\overrightarrow{\xi }}_{\text{trunc}}$ and can be simplified as

$$\begin{gathered} {\xi }_{\text{trunc}}^2=\sum _{\mu =1}^{𝒬}{\left[e_y{e}_{\mu }+\left(e_{s\ast }^2+e_u^2\right){\widehat{y}}_{\mu }\right]}^2 \\ =e_y^2\sum _{\mu =1}^{𝒬}{e}_{\mu }^2+{\left(e_{s\ast }^2+e_u^2\right)}^2\sum _{\mu =1}^{𝒞}{\widehat{y}}_{\mu }^2 \\ +2\left(e_{s\ast }^2+e_u^2\right)e_y\sum _{\mu =1}^{𝒞}{\widehat{y}}_{\mu }{e}_{\mu } \\ =e_y^2\left(e_p^2-e_{s\ast }^2\right)+{\left(e_{s\ast }^2+e_u^2\right)}^2+2\left(e_{s\ast }^2+e_u^2\right)e_y^2 \\ ={\left(e_y^2+e_{s\ast }^2+e_u^2\right)}^2+e_y^2\left(e_p^2-e_{s\ast }^2-e_y^2\right). \end{gathered}$$ (C14)

Substituting ξ = ξ_trunc into the counterpart of Eq. (C9), and keeping only the linear terms in ε, we thus get the leading order correction to Eq. (A48):

$$\begin{gathered} y_{\text{cr},\text{appr},0}\approx W\sqrt{\frac{e_{s\ast }^2+e_u^2}{e_y^2+e_{s\ast }^2+e_u^2}}+\frac{\epsilon {\xi }_{\text{trunc}}}{e_y^2+e_{s\ast }^2+e_u^2} \\ =W\sqrt{\frac{e_{s\ast }^2+e_u^2}{e_y^2+e_{s\ast }^2+e_u^2}}+\epsilon \sqrt{1+\frac{e_y^2\left(e_p^2-e_{s\ast }^2-e_y^2\right)}{{\left(e_y^2+e_{s\ast }^2+e_u^2\right)}^2}}. \end{gathered}$$ (C15)

We will see later how y_cr,appr,0 can be generalized to provide an approximation, y_cr,appr, to y-critical that accounts for topological transitions.

Reassuringly, we see that at ε = 0, y_cr,appr,0, y_cr,max,0, and y_cr,min,0, all reduce to the zero-error y_cr Eq. (A48). Also it is obvious that the coefficient of ε in y_cr,appr,0 is greater than that of y_cr,min,0 but less than that of y_cr,max,0. Finally, note that y_cr,appr,0 coincides with y_cr,min in the maximally nonlinear case where $e_{s\ast }^2=e_p^2-e_y^2$. In Fig. 10(a), we have plotted how these different quantities depend on e_p, e_y and e_s∗. In particular we note that as the network size increases, these curves typically come closer together [Fig. 10(b)], so that they provide a good approximation for y-critical. Finally, for future reference we point out that for a given set of input patterns, z_μm, the various y-criticals that we have computed above depend on the orientation of the target response vector, or $\widehat{y}$, and the total noise budget, ε. In other words, $y_{\text{cr},\text{min},0}=y_{\text{cr},\text{min},0}(\widehat{y},\epsilon )$, $y_{\text{cr},\text{max},0}=y_{\text{cr},\text{max},0}(\widehat{y},\epsilon )$, and $y_{\text{cr},\text{appr},0}=y_{\text{cr},\text{appr}.0}(\widehat{y},\epsilon )$.

FIG. 10.

Dependence of y-critical on various parameters for nonzero errors. (a) The red, blue, and purple curves track y-critical as a function of e_y/e_p for e_p = 0.5, 0.7, and 0.9, respectively. The dotted, dashed and bold curves represent the lower bound, leading-order and upper bound y-critical curves for a fixed error, ε = 0.1W. The darker shade correspond to the most nonlinear case when $e_{s\ast }/\sqrt{e_p^2-e_y^2}=1$, while the lighter shade correspond to e_s∗ = 0. These latter curves are also the ones that one obtains in a linear theory. Clearly, the difference between the linear and nonlinear theory increases as e_p increases. In all these cases y-critical decreases with increase of e_p, and for a given e_p, as e_y/e_p increases. Also, as e_s∗ increases and the semiconstrained dimensions become more important, it becomes harder to constrain the synapse sign, and therefore y-critical increases. (b) The green, brown, and orange curves again track y-critical, but this time as a function of ε, for networks with 𝒩 = 3,9, and 27 input neurons, respectively. The dotted, dashed and bold curves plot the lower bound, leading-order and upper bound on y-critical for typical values of e_p, e_y, and e_s∗ that one expects in these networks (B1). We see that these curves come closer together as the network size increases. The dot-dashed curves correspond to the linear theory (e_s∗ = 0), which remains clearly separated from the nonlinear curves. In each of these networks, 𝒫/𝒩 = 2/3 and 𝒞/𝒫 = 1/2.

b. Comparing predictions from linear and nonlinear models

To assess the effects of nonlinearity it is useful to compare the predictions for certain-synapses between the linear and nonlinear theory. In a linear theory, there are no semiconstrained directions, and therefore, a lower bound, leading order and upper bound on y-critical can be obtained from Eqs. (C4), (C15), and (C11), respectively, by setting e_s∗ = 0:

$$y_{\text{cr},\text{min},\text{lin}}=W\sqrt{\frac{e_u^2}{e_y^2+e_u^2}}+\epsilon ,$$ (C16)

$$y_{\text{cr},\text{appr ,lin}}=W\sqrt{\frac{e_u^2}{e_y^2+e_u^2}}+\epsilon \sqrt{1+\frac{e_y^2\left(e_p^2-e_y^2\right)}{{\left(e_y^2+e_u^2\right)}^2}},$$ (C17)

$$y_{\text{cr},\text{max},\text{lin}}=\sqrt{W^2\left(\frac{e_u^2}{e_y^2+e_u^2}\right)+{\epsilon }^2\left(1+\frac{e_y^2\left(e_p^2-e_y^2\right)}{{\left(e_y^2+e_u^2\right)}^2}\right)}+\epsilon \sqrt{1+\frac{e_y^2\left(e_p^2-e_y^2\right)}{{\left(e_y^2+e_u^2\right)}^2}}.$$ (C18)

We note that since all these quantities are increasing function of e_s∗, the linear values are always less than or equal to the nonlinear counterparts. Since no topological transitions are possible in a linear theory, these expressions do not need a qualifying “0” index. In Fig. 10, we show a comparison between the upper bound on y-critical obtained in the linear and the nonlinear theories.

c. y_cr,max, when one and only one nonzero response is smaller than noise

In the previous section we have considered responses which are either zero, or positive and greater than the noise bound, ε. In this section, we consider a situation where one and only one of the observed responses is smaller than the noise, |y₁| < ε.

Note that once we admit noise, it is possible for the small observed response to be negative. In this case, there is no zero-error solution as Φ(η₁) cannot be negative, and therefore we need a minimum noise, and incur a minimum error:

$${\delta }_1=-y_1\Rightarrow {ℰ}_{\text{min}}=y_1^2.$$ (C19)

In fact, since the noise for this observation has to be positive, we must have

$$\Phi \left({\eta }_1\right)=y_1+{\delta }_1\equiv {\delta }_1^{\prime }⩾0.$$ (C20)

Then using

$${\delta }_1^2={\delta }_1^{\prime 2}+y_1^2-2y_1{\delta }_1^{\prime },$$ (C21)

we obtain a modified bound on the noise:

$$\begin{gathered} \sum _{\mu =1}^{𝒫}{\delta }_{\mu }^2=\sum _{\mu =2}^{𝒫}{\delta }_{\mu }^2+{\delta }_1^{\prime 2}+y_1^2-2y_1{\delta }_1^{\prime }⩽{\epsilon }^2 \\ \Rightarrow {\delta }_1^{\prime 2}+\sum _{\mu =2}^{𝒫}{\delta }_{\mu }^2⩽{\epsilon }^2-y_1^2+2y_1{\delta }_1^{\prime }⩽{\epsilon }^2-y_1^2, \end{gathered}$$ (C22)

since $y_1{\delta }_1^{\prime }<0$. Or,

$${\delta }_1^{\prime 2}+\sum _{\mu =1}^{𝒫}{\delta }_{\mu }^2⩽{\epsilon }^{\prime 2}\equiv {\epsilon }^2-y_1^2.$$ (C23)

Let us now introduce a new reduced response vector whose response to the first pattern is set to zero:

$${\overrightarrow{y}}^{\prime }\equiv \sum _{\mu =2}^{𝒞}{y}_{\mu }{\overrightarrow{\epsilon }}_{\mu }\text{.}$$ (C24)

We can then identify ${\delta }_1^{\prime }$ to be the noise associated with the μ = 1 response in this new feedforward problem, while the other δ_μ’s can continue to represent the noise associated with all the other responses. Thus, a sufficient condition for a given synapse to be certain is

$$\left|{\overrightarrow{y}}^{\prime }\right|>y_{\text{cr},\text{max},0}\left({\widehat{y}}^{\prime },{\epsilon }^{\prime }\right).$$ (C25)

A very similar condition arises if y₁ is positive but small enough to admit a topological transition. To see how, let us first remember that in order for a synapse to be certain, the solution space should not intersect with the w = 0 hyperplane. Now, let us look at the solution space coming from denoised $\overrightarrow{\tilde{y}}$’s that have ${\tilde{y}}_1>0$. Since, the solution space corresponding to these $\overrightarrow{\tilde{y}}$’s do not have any additional semiconstrained dimension as compared to the observed response, $\overrightarrow{y}$, the condition for no intersection with this part of the solution space is simply given by

$$\left|\overrightarrow{y}\right|>y_{\text{cr},\text{max},0}(\widehat{y},\epsilon ),$$ (C26)

a condition that guarantees a certain synapse when no topological transitions are considered. Next consider the solution space for denoised $\overrightarrow{\tilde{y}}$’s with ${\tilde{y}}_1=0$. The solution space for these $\overrightarrow{\tilde{y}}$’s have an additional semiconstrained dimension corresponding to the first pattern. We can therefore use the reduced response vector, ${\overrightarrow{y}}^{\prime }$ (C24), so that the solution space corresponding this new response vector with the error bound, ε′ (C23), along with the solution space with ${\tilde{y}}_1>0$ accounts for the full solution space of $\overrightarrow{y}$ with error ε. Note, that the noise budget is again reduced according to Eq. (C23) since we are committed to making at least an error of y₁ to convert the first response to a semiconstrained dimension. To ensure that there is no intersection of the w = 0 hyperplane with the ${\overrightarrow{y}}^{\prime }$ solution space we must therefore also satisfy Eq. (C25). We note that to calculate the right-hand side using Eq. (C11), the various projections have to be recalculated according to

$$\begin{gathered} e_{y^{\prime }}=\frac{{\sum }_{\mu =2}^{𝒞}{y}_{\mu }{e}_{\mu }}{\sqrt{{\sum }_{\mu =2}^{𝒞}{y}_{\mu }^2}}\text{and} \\ {e}_{s^{\prime }\ast }=\sqrt{\sum _{\mu \in A_{-}}{e}_{\mu }^2+\Theta \left(\text{Sgn}\left(e_{y^{\prime }}\right)e_1\right)e_1^2}. \end{gathered}$$ (C27)

The condition (C25) on $\left|{\overrightarrow{y}}^{\prime }\right|$ translates to a condition on $|\overrightarrow{y}|$:

$$|\overrightarrow{y}|>\sqrt{y_{\text{cr},\text{max},0}^2\left(\widehat{y},{\epsilon }^{\prime }\right)+y_1^2}\Rightarrow |\overrightarrow{y}|>\frac{y_{\text{cr},\text{max},0}\left({\widehat{y}}^{\prime },{\epsilon }^{\prime }\right)}{\sqrt{1-{\widehat{y}}_1^2}},$$ (C28)

where we have defined ${\widehat{y}}_{\mu }$’s to be the μth component of $\widehat{y}$.

We note that this is a nonlinear inequality as the right-hand side depends on $|\overrightarrow{y}|$ through its implicit dependence on ε′. When we have a negative y₁, only Eq. (C28) needs to be satisfied to guarantee a certain synapse, but if y₁ is positive, both Eqs. (C26) and (C28) have to be satisfied. It is not hard to see how this process should be continued if one has more than one topological transition within the allowed error. Since we know the precise sequence of topological transitions, all the sequential certainty-conditions can in principle be obtained. A synapse is certain if all of its certainty-conditions are satisfied.

So far, we have obtained a way to check whether a synapse is certain given the response data, $\overrightarrow{y}$. We also have an upper bound of y-critical, y_cr,max,0, ignoring effects from topological transitions when all the observed responses are nonnegative. We will now investigate how topological transitions can change this upper bound. We will start by quantifying effects from a single topological transition by finding potentially a new upper bound for y-critical, y_cr,max, such that we can say that if $|\overrightarrow{y}|>y_{\text{cr},\text{max}}$, then the synapse is certain. Suppose we start out with a data vector whose norm is so large that there are no topological transitions. Then as we decrease the strained norm, but keep its orientation, $\widehat{y}$, fixed, eventually a semiconstrained dimension will open up in the solution space, in our example, the first direction. If we keep decreasing further, then at some point another response dimension will become semiconstrained due to the presence of noise. Let us however consider the situation where y_cr,max (that is yet to be computed) is going to turn out to be larger than the norm when the second transition occurs. In this case, we do not have to consider this possibility (and any other transitions) because then if $|\overrightarrow{y}|>y_{\text{cr},\text{max}}$ the second transition cannot occur. We will later find a condition that guarantees this. Since we are trying to find the smallest value of y_cr,max that we can find, what all this means is that at ${\overrightarrow{y}}_{\text{cr},\text{max}}=y_{\text{cr},\text{max}}\widehat{y}$, one of the two inequalities, (C26) or (C25), becomes an equality. While the first equality is trivial to solve as the right-hand side does not depend on y_cr,max, the second equation is highly nonlinear²³:

$$y_{\text{cr},\text{max},1}=\frac{y_{\text{cr},\text{max},0}\left({\widehat{y}}^{\prime },{\epsilon }^{\prime }\right)}{\sqrt{1-{\widehat{y}}_1^2}},$$ (C29)

where

$${\epsilon }^{\prime }=\sqrt{{\epsilon }^2-y_{\text{cr},\text{max},1}^2{\widehat{y}}_1^2}.$$ (C30)

In particular, we notice that there are two competing effects that ultimately determine y_cr,max,1. The numerator depends on ε′, which decreases as y_cr,max,1 increases and therefore has an overall effect of decreasing y_cr,max,1. However, the presence of $\widehat{y_1}$ in the denominator within the square root tends to increase y_cr,max,1. To determine the correct upper bound for y-critical one has to compare the $y_{\text{cr},\text{max},1}\left({\widehat{y}}^{\prime },{\widehat{y}}_1,{\epsilon }^{\prime }\right)$ obtained from Eq. (C29) with $y_{\text{cr},\text{max},0}(\widehat{y},\epsilon )$, and then choose the maximum because then both the inequalities (C26) and (C25) will be satisfied. For the negative response case, we simply need to solve Eq. (C29) to obtain y_cr,max,1.

Now, determining y_cr,max,1 from Eq. (C28) involves solving a quartic equation leading to expressions that are not particularly insightful. However, we can obtain a relatively simple conservative estimate bypassing the nonlinearity if we have a lower bound on y-critical, y_cr,min because we can use this bound to overestimate ε′:

$$\begin{gathered} y_{\text{cr},\text{max}}\equiv \text{max}\left\{\frac{y_{\text{cr},\text{max},0}\left({\widehat{y}}^{\prime },{\epsilon }^{″}\right)}{\sqrt{1-{\widehat{y}}_1^2}},y_{\text{cr},\text{max},0}(\widehat{y},\epsilon )\right\}\text{where} \\ {\epsilon }^{″}\equiv \sqrt{{\epsilon }^2-y_{\text{cr},\text{min}}^2{\widehat{y}}_1^2}⩾{\epsilon }^{\prime }. \end{gathered}$$ (C31)

Before we describe how we can obtain y_cr,min, let us note that if

$$y_{\text{cr},\text{min}}{\widehat{y}}_{\mu }>\epsilon ,\hspace{1em}\forall \mu >1,$$ (C32)

then the second transition occurs at a magnitude that is lower than y_cr,max,1, and therefore does not need to be incorporated in the y_cr,max,1 calculation. Indeed, Eq. (C32) is a sufficient condition but not a necessary one.

d. y_cr,min, when one and only one nonzero response is smaller than noise

When we have a small response, |y₁| < ε, we have seen that we have to consider solution space around a reduced response vector, ${\overrightarrow{y}}^{\prime }$ (C24), with a smaller error budget, ε′ (C23). Accordingly, we can obtain an equation for a lower bound on y-critical using Eq. (C4),²⁴

$$y_{\text{cr},\text{min},1}=y_{\text{cr}}\left({\widehat{y}}^{\prime }\right)+{\epsilon }^{\prime },$$ (C33)

where μ = 1 along with μ = 𝒞 + 1 … 𝒫 are all treated as semiconstrained. As before, since the norms along $\overrightarrow{y}$ and ${\overrightarrow{y}}^{\prime }$ are related via

$$\left|\overrightarrow{y}\right|\sqrt{1-{\widehat{y}}_1}=\left|{\overrightarrow{y}}^{\prime }\right|,$$ (C34)

we get an equation for y_cr,min,1 very similar to Eq. (C29) for y_cr,max,1:

$$y_{\text{cr},\text{min},1}=\frac{y_{\text{cr}}\left({\widehat{y}}^{\prime }\right)+{\epsilon }^{″}}{\sqrt{1-{\widehat{y}}_1^2}},\hspace{1em}\text{where}{\epsilon }^{″}=\sqrt{{\epsilon }^2-y_{\text{cr},\text{min},1}^2{\widehat{y}}_1^2}.$$ (C35)

Although nonlinear, the above equation reduces to a quadratic equation for y_cr,min,1,

$$y_{\text{cr},\text{min},1}^2-2y_{\text{cr},\text{min},1}{y}_{\text{cr}}\sqrt{1-{\widehat{y}}_1^2}+y_{\text{cr}}^2-{\epsilon }^2=0,$$ (C36)

solving which we get²⁵

$$y_{\text{cr},\text{min},1}=y_{\text{cr}}\left({\widehat{y}}^{\prime }\right)\sqrt{1-{\widehat{y}}_1^2}+\sqrt{{\epsilon }^2-y_{\text{cr}}^2\left({\widehat{y}}^{\prime }\right){\widehat{y}}_1^2}.$$ (C37)

For positive y₁ the above expression provides another lower bound along with the one obtained without the transition (C4). To ensure we have the tightest possible lower bound we thus maximize:

$$y_{\text{cr},\text{min}}\equiv \text{max}\left\{y_{\text{cr},\text{min},0},y_{\text{cr},\text{min},1}\right\}.$$ (C38)

e. When more than one nonzero responses are smaller than allowed error

It is not difficult to see how the arguments above generalize if we have more than one small (< ε) observed response. We have to consider cases where all the negative observed responses, and different possible combinations of the positive responses, are set to zero. Let us denote T to be one such possible set of μ indices. As before, we define a reduced response vector, which is now indexed by T:

$${\overrightarrow{y}}_T\equiv \sum _{\mu \notin T}{y}_{\mu }{\overrightarrow{\epsilon }}_{\mu },$$ (C39)

so y_T,μ = 0 for all μ ∈ T. Then, essentially following the same algebraic manipulations as above we obtain a lower bound according to

$$y_{\text{cr},\text{min},T}=y_{\text{cr}}\left({\widehat{y}}_T\right)\sqrt{1-\sum _{\mu \in T}{\widehat{y}}_{\mu }^2}+\sqrt{{\epsilon }^2-y_{\text{cr}}^2\left({\widehat{y}}_T\right)\sum _{\mu \in T}{\widehat{y}}_{\mu }^2}.$$ (C40)

To reiterate, whenever an observed response is negative, which is inconsistent with a threshold linear transfer function, this means that some of the noise budget has to be used up to bring this response up to zero, and the same noise reduction occurs if one wants to consider topological transitions. Each y_cr,min,T evaluated this way provides us with a lower bound, and hence we have to take a maximum over all these to find the tightest lower bound, y_cr,min. To make things explicit, let us also enumerate the new expressions for the various projections of $\widehat{e}$ that one needs to calculate $y_{\text{cr}}\left({\widehat{y}}_T\right)$:

$$\begin{gathered} e_{y,T}=\left(\sum _{\mu \notin T}{y}_{\mu }{e}_{\mu }\right)/\sqrt{\sum _{\mu \notin T}{y}_{\mu }^2}\hspace{1em}\text{and} \\ {e}_{s\ast ,T}=\sqrt{\sum _{\mu \in T}\Theta \left(e_{\mu }{e}_{y,T}\right)e_{\mu }^2}. \end{gathered}$$ (C41)

Once we have a lower bound, we can obtain conservative upper bounds analogous to Eq. (C31) for each T:

$$\begin{gathered} y_{\text{cr},\text{max},T}\equiv \frac{y_{\text{cr},\text{max},0}\left({\widehat{y}}_T,{\epsilon }_T\right)}{\sqrt{1-{\sum }_{\mu \in T}{\widehat{y}}_{\mu }^2}},\hspace{1em}\text{where} \\ {\epsilon }_T\equiv \sqrt{{\epsilon }^2-y_{\text{cr},\text{min}}^2\sum _{\mu \in T}{\widehat{y}}_{\mu }^2}. \end{gathered}$$ (C42)

As before, for a consistent upper bound for y-critical, we need to take the maximum over all y_cr,max,T ’s.

Finally, we note that we do not need to consider all possible transitions. While going through the sequence of transitions, as soon as we find a T such that $y_{\text{cr},\text{min},T}{\widehat{y}}_{\mu }>\epsilon$ for all μ ∉ T we can stop as this means that by the time $|\overrightarrow{y}|$ is small enough that any additional y_μ’s can be set to zero, the synapse is already uncertain.

f. Numerical simulation

To illustrate the behavior of the various y-critical functions and check their utility, in Fig. 11 we have plotted y_cr,min (light gray curve) and y_cr,max (black curve) for the same 102 configurations as the ones depicted in Fig. 8(b) involving a feedforward simulation with 𝒩 = 6, 𝒫 = 5, 𝒞 = 2, and ℇ < ε = 0.1. We also defined, y_cr,appr, as a maximum over different approximations, y_cr,appr,T’s, that incorporate topological transitions and are defined as natural generalizations of Eq. (C15):

$$y_{\text{cr},\text{appr},\text{T}}=W\sqrt{\frac{e_{s\ast ,T}^2+e_u^2}{e_{y,T}^2+e_{s\ast ,T}^2+e_u^2}}+{\epsilon }_T\sqrt{1+\frac{e_{y,T}^2\left(e_p^2-e_{s\ast ,T}^2-e_{y,T}^2\right)}{{\left(e_{y,T}^2+e_{s\ast ,T}^2+e_u^2\right)}^2}}.$$ (C43)

We have plotted y_cr,appr, the approximation of y-critical, in green in Fig. 11. As in Fig. 8(b), the black dots here denote the maximum value of y in our simulations that still admitted mixed signs for the synapse under consideration, for details on the simulations, please see Appendix F. As one can see, most of the black dots seem to closely track the y_cr,min curve, but some of the dots lie between the y_cr,appr and y_cr,min curves.

FIG. 11.

Testing bounds on y-critical for solutions with error. We show the same 102 random configurations of input-output activity as Fig. 8(b). The bold black, green, and gray curves represent the upper bound y_cr,max, approximate y_cr,appr, and lower bound y-critical y_cr,min, values, respectively. The black dots correspond to the maximum value of y in our simulations that resulted in mixed signs for the synaptic weights under consideration.

2. New sources of corrections in recurrent neural networks

It is clear that recurrent neural networks inherit error corrections to y-critical that were already present in the feedforward case. There are two additional sources of error that one could consider as one moves from feedforward to recurrent networks. However, our numerical simulations of recurrent networks suggest that these are sometimes small effects, and we leave their systematic study for the future.

First, we could account for the fact that the ${\overrightarrow{\epsilon }}_{\mu }$ directions themselves can change. This is because the inputs driving any given driven neuron can no longer be assumed to be fixed at z_μm if the other driven neurons suffer from noise. However, these activity patterns define the ${\overrightarrow{\epsilon }}_{\mu }$ directions and η_μ coordinates. Allowing noise in input neurons would lead to similar corrections.

Second, the total error in Eq. (63) may be unevenly distributed across the driven neurons. If the total squared error summed over all responses and neurons is ${\epsilon }_{\text{tot}}^2$, then on average, the root-mean-square error associated with each driven neuron is ${\epsilon }_{\text{tot}}/\sqrt{𝒟}$. We can thus hope that a substitution of $\epsilon ={\epsilon }_{\text{tot}}/\sqrt{𝒟}$ in the various y-critical formulas will provide a good approximation. However, it is also possible that a few neurons will incur most of the error (up to ε_tot), potentially leading to violation of the certainty conditions computed from the root-mean-square error over neurons.

APPENDIX D: BEYOND THRESHOLD-LINEAR TRANSFER FUNCTIONS

So far, we have always modeled the firing rate as a threshold-linear function applied to the input drive. Here we will explain how our analyses of y-critical with noise also provide a formalism to analyze a much more general class of nonlinear transfer functions.

1. Bounded deviations from the threshold-linear function

Let us start by considering transfer functions with bounded differences from the threshold-linear function:

$$\Psi (x)=\Phi (x)+\Delta (x),\text{with}|\Delta (x)|<{\Delta }_0\forall x\text{.}$$ (D1)

In this case, the fixed-point equations become

$$\begin{gathered} y_{\mu }=\Psi \left(\sum _{m=1}^{𝒩}{z}_{\mu m}{w}_m\right)=\Phi \left(\sum _{m=1}^{𝒩}{z}_{\mu m}{w}_m\right)+{\delta }_{\mu },\hspace{1em}\text{where} \\ {\delta }_{\mu }\equiv \Delta \left(\sum _{m=1}^{𝒩}{z}_{\mu m}{w}_m\right). \end{gathered}$$ (D2)

Since |Δ(x)| is bounded by Δ₀, we have a bound on the squared norm of $\overrightarrow{\delta }$:

$$|\overrightarrow{\delta }{|}^2<𝒫{\Delta }_0^2.$$ (D3)

It is therefore clear that we can estimate y-critical for the Ψ nonlinearity with exactly the same formalism that we used to estimate y-critical for the threshold nonlinearity in the presence of noise. In particular, all the y-critical estimates [(64), (65), (C15)] are valid with the substitution, $\epsilon =\sqrt{𝒫}{\Delta }_0$. Moreover, one can account for other sources of noise (bounded by ε₀) by instead substituting

$$\epsilon =\sqrt{{\epsilon }_0^2+𝒫{\Delta }_0^2}$$ (D4)

to obtain estimates and bounds on y-critical.

2. Bounded departures from any threshold-monotonic nonlinearity

Let us now consider transfer functions, Ψ(x), that are close to a function, Ξ(x), that monotonically increases above a threshold, x_T :

$$\begin{gathered} \Xi (x)=0,\text{if}x⩽x_T, \\ \Xi (x)>\Xi (y)>0,\text{if}x>y>x_T,\text{and} \\ \Psi (x)=\Xi (x)+\Delta (x),\text{with}|\Delta (x)|<{\Delta }_0\forall x. \end{gathered}$$ (D5)

Accordingly, we find

$$\begin{gathered} y_{\mu }=\Psi \left(\sum _{m=1}^{𝒩}{z}_{\mu m}{w}_m\right)=\Xi \left({\eta }_{\mu }\right)+\Delta \left({\eta }_{\mu }\right)\Rightarrow \Xi \left({\eta }_{\mu }\right) \\ =y_{\mu }-\Delta \left({\eta }_{\mu }\right)\text{.} \end{gathered}$$ (D6)

Since the monotonicity condition ensures that Ξ⁻¹ is well defined above threshold, and, we then have the upper bound,

$$0⩽\Xi \left({\eta }_{\mu }\right)<y_{\mu }+{\Delta }_0\Rightarrow {\eta }_{\mu }<{\Xi }^{-1}\left(y_{\mu }+{\Delta }_0\right).$$ (D7)

Additionally, if y_μ > Δ₀, then we also have a lower bound:

$$0<y_{\mu }-{\Delta }_0<\Xi \left({\eta }_{\mu }\right)\Rightarrow {\eta }_{\mu }{>\Xi }^{-1}\left(y_{\mu }-{\Delta }_0\right).$$ (D8)

Thus, combining the upper and lower bounds, we find

$${\Xi }^{-1}\left(y_{\mu }+{\Delta }_0\right)>{\eta }_{\mu }>{\Xi }^{-1}\left(y_{\mu }-{\Delta }_0\right)>0$$ (D9)

However, if y_μ ⩽ Δ₀, then there is no lower bound, and any η_μ satisfying the upper bound (D7) is allowed. Now, we can introduce effective responses, representing the midpoint of possible superthreshold input drives,

$${\overline{y}}_{\mu }\equiv \{\begin{array}{ll}\frac{1}{2}\left[{\Xi }^{-1}\left(y_{\mu }+{\Delta }_0\right)+{\Xi }^{-1}\left(y_{\mu }-{\Delta }_0\right)\right],\hfill & \text{if}{y}_{\mu }>{\Delta }_0,\hfill \\ \frac{1}{2}{\Xi }^{-1}\left(y_{\mu }+{\Delta }_0\right),\hfill & \text{if}{y}_{\mu }⩽{\Delta }_0,\hfill \end{array}$$ (D10)

and effective noise limits,

$${\overline{\epsilon }}_{\mu }\equiv \{\begin{array}{ll}\frac{1}{2}\left[{\Xi }^{-1}\left(y_{\mu }+{\Delta }_0\right)-{\Xi }^{-1}\left(y_{\mu }-{\Delta }_0\right)\right],\hfill & \text{if}{y}_{\mu }>{\Delta }_0,\hfill \\ \frac{1}{2}{\Xi }^{-1}\left(y_{\mu }+{\Delta }_0\right),\hfill & \text{if}{y}_{\mu }⩽{\Delta }_0,\hfill \end{array}$$ (D11)

which allow η_μ to span the full allowed range. By inspection, we now see that the solution space is equivalent to the solution space of a threshold-linear problem:

$$\Phi \left({\eta }_{\mu }\right)={\overline{y}}_{\mu }+{\overline{\delta }}_{\mu },\hspace{1em}\text{with}\left|{\overline{\delta }}_{\mu }\right|⩽{\overline{\epsilon }}_{\mu }.$$

Thus, again all the y-critical estimates [(64), (65), (C15)] will be valid with the substitution $y_{\mu }\to {\overline{y}}_{\mu }$ and a conservative error bound

$${\epsilon }^2=\sum _{\mu =1}^{𝒫}{\overline{\epsilon }}_{\mu }^2.$$ (D13)

APPENDIX E: CERTAIN SYNAPSES IN LOW-DIMENSIONAL RECURRENT NETWORKS WITH SELF-CONNECTIONS

As discussed in Appendix A, when one moves from feedforward to recurrent neural networks with self-synapses, the input patterns can no longer be considered independent from the target neuron responses. How then does one assess synapse certainty for driven neurons with self-synapses, such as y₃ in Fig. 6(a), y in Fig. 12(a), y₁ in Fig. 12(c), and y₂ in Fig. 12(c)?

FIG. 12.

Comparing simulation and theoretical results in 𝒩 = 3 recurrent network. (a) A simple 𝒩 = 3 recurrent neural network with one driven and two input neurons. Note that the y₁ neuron shown here maps onto the y₃ neuron in Fig. 6(a) by interpreting the x₁ and x₂ neurons shown here as the x₃ and y₂ neurons shown in Fig. 6(a). (b) Bar graphs depicting the fraction of positive (red) and negative (blue) weights from the network depicted in panel (a). (c) Another 𝒩 = 3 recurrent neural network, this time with two driven and one input neuron. (d) Bar graphs depicting the fraction of positive (red) and negative (blue) weights from the network depicted in panel (c). (e) The black bars depict y − y_cr for the corresponding synapses.

We begin by concretely analyzing neuron y in Fig. 12(a), because this is the conceptually simplest example, and fundamentally the mathematical analyses are the same for the other examples. In particular, to test our formalism and analytical results using this neuron, we performed low-dimensional simulations where the 𝒩 × 𝒩 extended input pattern matrix was

$$Z=\left(\begin{array}{rrr}\hfill x_1& \hfill x_2& \hfill y\\ \hfill -\text{sin}\psi \text{cos}\chi & \hfill \text{cos}\psi \text{cos}\chi & \hfill \text{sin}\chi \\ \hfill \text{cos}\psi & \hfill \text{sin}\psi & \hfill 0\\ \hfill \text{sin}\psi \text{sin}\chi & \hfill -\text{cos}\psi \text{sin}\chi & \hfill \text{cos}\chi \end{array}\right)$$ (E1)

which is the same as X in Eq. (60), except that the role of x₃ is now played by y itself.²⁶ The third column of Eq. (E1) corresponds to the responses of the driven neuron, but it also provides a self-input. The two input neuron responses are given by the first two columns. Equation (E1) is meant to correspond to the case where 𝒫 = 2 and 𝒞 = 1, such that μ = 1, 2, 3 correspond to the constrained, semiconstrained, and unconstrained response patterns, respectively. For the purpose of numerical testing, we assumed that χ ∈ (0°, 90°) and ψ ∈ (−90°, 0°), as this range of angles ensures that driven neuron responses were nonnegative.²⁷ We set W = 1. See Appendix F for numerical simulation details.

This example problem has one self-coupling, w, and two feedforward couplings, u₁ and u₂. For this response structure, one can use Eq. (51) to calculate y_cr for each of these three couplings. We find

$$\begin{gathered} y_{\text{cr},w}=W\text{cos}\chi , \\ {y}_{\text{cr},u_1}=W\sqrt{{\text{cos}}^2\psi +{\text{cos}}^2\chi {\text{sin}}^2\psi }, \\ {y}_{\text{cr},u_2}=W\text{sin}\chi . \end{gathered}$$ (E2)

We assess synapse certainty by checking whether these formulas for y_cr are smaller than the magnitude of $\overrightarrow{y}$,

$$y=\text{sin}\chi .$$ (E3)

Since W = 1, we see that the self-coupling becomes certain only if

$$\text{sin}\chi >\text{cos}\chi ,\hspace{1em}\text{or},0°<\theta \equiv \pi /2-\chi <45°$$ (E4)

[Fig. 12(b), left], where θ has been defined to be the angle between ${\widehat{e}}_{yy}$ and $\widehat{c}=\widehat{y}$²⁸. consistent with the conventions of Eq. (37). Next, basic trigonometric manipulations tell us that the certainty condition can never be satisfied for u₁ [Fig. 12(b), middle]. Finally, we see that the condition for certainty is always just not satisfied for u₂. Here this implies that all solutions have u₂ ⩾ 0 [Fig. 12(b), right]. This is because it is a very special case where cos γ = e_s∗ = 0 and ${\widehat{y}}_{\perp }$ in Eq. (A51) is aligned with $\widehat{u}$, so that both the inequalities in Eq. (A53) turn into equalities. In each panel of Fig. 12(b), we tracked the fraction of positive and negative synapse signs across the simulations, as we varied θ. In particular, we see that w had a unique sign as long as θ < 45°, u₁ always had mixed signs, and u₂ was nonnegative.²⁹

The same exact response matrix (E1) can also be used to consider neurons y₃ in Fig. 6(a) and y₂ in Fig. 12(c). The only difference is that the three columns respectively encode the responses of the second driven neuron, second input neuron, and third driven neuron in Fig. 6(a) and the responses of the single input neuron and two driven neurons, y₁ and y₂, in Fig. 12(c). This correspondence can be seen by comparing the numerical results in Figs. 12(b) and 12(d). We use this correspondence to avoid having to simulate the full network in Fig. 6(a), and the numerical results in Fig. 6(c) are the same as those in Fig. 12(b).

APPENDIX F: NUMERICAL METHODS

1. Low-dimensional numerical methods

Here we detail the numerical methods relevant for Figs. 6 and 12.

a. Feedforward analysis

To test the analytic dependence in Fig. 6(b), we wanted to simulate solutions without biasing ourselves by the particular search algorithm used to find solutions. Accordingly, to find solutions to the fixed point equations (A2) with very small error (ℇ < ε = 0.01) we performed a random screen where each weight was chosen randomly from a uniform distribution between −1 and +1. For feedforward circuits, given the synaptic weights, one can obtain the fixed point responses of the target neuron by direct substitution of the known input responses in Eq. (A2) and then comparing these simulated target responses with the known target responses. We varied ψ, χ in the response data (E1) systematically in steps of 6°.³⁰ For the light and dark green curves, ψ was fixed at 45°, and χ was varied between (0°, 90°) and (90°, 180°), respectively, while for the pink and purple curves χ was fixed at 45°, 135°, respectively, and ψ was varied between (0°, 90°). Finally, for a given choice of ψ, χ, we systematically varied y between 0 and 1, in intervals of Δy = 0.01. For each value of ψ, χ, and y, we obtained ~(10²–10⁴) solutions³¹ satisfying the error and the biological bound (A11) from five to ten million different trial weight vectors. We then identified the maximal value of y for which the solutions had both positive and negative w₁’s. This simulation point should lie beneath the theoretical y_cr if no error is allowed. However, since the error is small but nonzero, occasionally the y-criticals determined from simulations did slightly exceed the theoretical value. Also, since we vary y by small amounts y = 0.01, we expected the simulated y-criticals to be discrete but close to the theoretical predictions, which is exactly what we found in Fig. 6(b).

b. Recurrent analysis

Because the recurrent network solution space separates into several feedforward solution spaces at zero error, we numerically treated the driven neurons one at a time. To find solutions for the recurrent neurons in Figs. 6(a), 12(a), and 12(c), we fixed χ, ψ and then performed screens with random weights, selected in the same manner as the feedforward simulations discussed earlier. For each set of weights, and for each μ = 1, 2, we obtained the late time values of y by solving the time evolution equation [Eq. (6) with τ_i = 20 ms] using Euler’s method starting with initial conditions y_i(0) = y_μi, for μ = 1, 2. We used a time step of Δt = 0.2 ms. The ${\tilde{y}}_{\mu }$’s obtained from the simulation at late times, t ~ 600 ms, were then compared with y_μ to obtain ℇ. If the weights satisfied, $ℰ<0.05\sqrt{𝒟}$ and the biological bound (A11), then we considered the weights as solutions and checked the sign of the synaptic weights. For every value of ψ, χ, we found at least 50 solutions³² to test the certainty predictions.

2. High-dimensional numerical methods

Here we detail the numerical methods relevant for Figs. 8 and 11.

a. Generating random orthogonal matrices

In several simulations we had to generate orthogonal response matrices. This meant that we had to obtain 𝒫 orthonormal 𝒩-dimensional vectors. This was done by first generating an 𝒩 × 𝒩 matrix, G, where each of its entries was randomly selected from a uniform distribution between −1 and +1. We then antisymmetrized the matrix, G → (G − G^T)/2, and a random orthogonal response matrix was then obtained via matrix exponentiation, Z = e^G, where the matrix exponential is defined by substituting the matrix G into the power series expansion of the exponential function and is distinct from the simple exponentiation of individual matrix elements. The first 𝒫 rows of Z could then correspond to the 𝒫 orthogonal patterns, and Z can be interpreted as the orthogonal extension of z as discussed before.

b. Generating random orthogonal matrices with nonnegativity constraints

In recurrent networks, all driven neurons must have nonnegative responses for all patterns. Accordingly, when the input response pattern includes responses of driven neurons, we follow a different procedure for generating the response matrix, which works as long as ℐ ⩾ 𝒫 – 1. We started by choosing a (𝒫 × 𝒩)-dimensional matrix, z, containing the responses of 𝒟 driven neurons and ℐ ⩾ 𝒫 – 1 input neurons. The first columns of z corresponded to driven neuron responses, and the last columns to input neuron responses, such that z = (y x). We made sure that the responses of the driven neurons were all nonnegative, as the threshold nonlinearity dictates, by choosing them to lie randomly between 0 and 1. To mimic a sparse response pattern, we set driven responses to 0 with 50% probability. The feedforward inputs, however, were randomly selected between −1 and 1. We then orthogonalized the input responses to the target neuron as follows. We start by normalizing the ν = 1 pattern:

$$z_{1m}\to \frac{z_{1m}}{\sqrt{{\sum }_{n=1}^{𝒩}{z}_{1n}^2}}.$$ (F1)

Then, for each row, ν = 2 … 𝒫, in a sequential order we performed the following operations:

1. We started by defining a (ν − 1)-dimensional square matrix, x′:

   $$x_{\mu m}^{\prime }\equiv x_{\mu m}\hspace{1em}\text{for}\mu ,m=1\dots (v-1).$$ (F2)
2. We next changed the first m = 1 … ν − 1 elements of the νth row of x

   $$x_{vm}\equiv -\sum _{\mu =1}^{v-1}{x}_{m\mu }^{\prime -1}\left(\sum _{i=1}^{𝒟}{y}_{\mu i}{y}_{vi}+\sum _{n=v}^{ℐ}{x}_{\mu n}{x}_{vn}\right),$$ (F3)

   and thus z. The other elements of the νth row of z were left unchanged. In particular, none of the driven neuron responses changed during this step.
3. Finally, we rescaled all the elements of the νth row of z for normalization:

   $$z_{vm}\to \frac{z_{vm}}{\sqrt{{\sum }_{n=1}^{𝒩}{z}_{vn}^2}}.$$ (F4)

   This algorithm essentially uses the responses of the input neurons to the νth stimulus to ensure that the full νth response pattern involving both the driven and input neurons is orthogonal to all μ ⩽ ν − 1 patterns.

c. Generating target responses and response directions

To generate 𝒫 target responses with 𝒮 null responses, we simply randomly selected numbers between 0 and 1 for the 𝒞 = 𝒫 − 𝒮 nonzero responses.

In some simulations, we wanted to consider situations where one has to account for a single topological transition to compute y-critical. Accordingly, we tailored the responses as follows. First, we set one nonzero response of the target neuron to a small value, 0.1ε. $\widehat{y}$ was then obtained by dividing the response vector by its norm. We then only considered those $\widehat{y}$’s whose other entries were large enough to prevent additional topological transitions from affecting y-critical. This was done by: evaluating y_cr,min, the theoretical lower bound for y-critical that includes the first topological transition (Appendix C); constructing $\overrightarrow{y}=y_{\text{cr},\text{min}}\widehat{y}$, which approximates the activity vector right below y-critical; and ensuring that all the other entries of $\overrightarrow{y}$ were greater than the allowed error, which guarantees that no other constrained dimensions can become semiconstrained in between y_cr,min and the true y-critical. This way, typically one and only one constrained direction became semiconstrained when we allowed solutions with errors ≲ε.

d. Finding solutions using gradient descent learning in feedforward networks

In all the high-dimensional simulations, we had to find solutions to the fixed point equations (A2). Since scanning a high-dimensional synaptic weight space randomly is not numerically efficient, we applied gradient descent learning³³ to obtain solutions. For feedforward networks, this meant using the loss function

$${ℰ}^2=\sum _{\mu =1}^{𝒫}{\left({\tilde{y}}_{\mu }-y_{\mu }\right)}^2,\hspace{1em}\text{where}{\tilde{y}}_{\mu }=\Phi \left(\sum _{m=1}^{𝒩}{z}_{\mu m}{w}_m\right).$$ (F5)

We performed gradient descent optimization until we reached the desired error bound, ℇ < ε. The initial weights were first chosen randomly from a uniform distribution between −1 and 1. The initial weight vector was then rescaled to have a norm between 0 and W = 1, chosen uniformly.

e. Finding solutions using gradient descent learning in recurrent networks

To find solutions for recurrent neural networks, we used the modified loss function,

$$\begin{gathered} {\overline{ℰ}}^2\equiv \sum _{i=1}^{𝒟}\sum _{\mu =1}^{𝒫}{\left({\tilde{y}}_{\mu i}-y_{\mu i}\right)}^2\equiv \sum _{i=1}^{𝒟}{\overline{ℰ}}_i^2,\hspace{1em}\text{where} \\ {\tilde{y}}_{\mu i}=\Phi \left(\sum _{m=1}^{𝒩}{z}_{\mu m}{w}_{im}\right), \end{gathered}$$ (F6)

to perform gradient descent, instead of Eq. (63). Since the responses of the driven neurons can vary for nonzero errors, the two loss functions, ℇ and $\overline{ℰ}$, differ. However, it is numerically a lot quicker to obtain solutions via gradient descent with $\overline{ℰ}$ as compared to using back-propagation through time to consider the entire time evolution of the network. Thus, the strategy we adopted to find solutions with ℇ ≲ ε was to first find weights satisfying $\overline{ℰ}\lesssim \overline{\epsilon }=\epsilon /10$. Also, the gradient descent was done in two stages. In the first stage we minimized the error associated with each individual driven neuron, ${\overline{ℰ}}_i$ treating it as a feedforward problem. Once each of these errors were less than $\overline{\epsilon }$, we performed a second stage of gradient descent to minimize $\overline{ℰ}$ down to $\overline{\epsilon }$. Next, we obtained the late time values of y_i’s by solving the time evolution equations (6) with τ_i = 20 ms using Euler’s method with step time Δt = 0.2 ms for the weights obtained via gradient descent and starting with initial conditions y_i(0) = y_μi, ∀μ,i. The ${\tilde{y}}_{\mu i}$’s obtained at late times, t ~ 600 ms, this way were compared with y_μi to obtain ℇ. Finally, we checked that the weights satisfied the biological bound³⁴ (A11).

f. Other minor simulation details

In Fig. 8(b), we show results from a simulation with 𝒩 = 6, 𝒫 = 5, 𝒞 = 2, and ℇ < ε = 0.1. The solutions were obtained using a gradient descent learning rate of 0.02. We varied the norm of the target response vector, $\overrightarrow{y}=y\widehat{y}$, systematically by Δy = 0.01 in a manner similar to the low-dimensional simulations.

In Fig. 8(c), we considered a single input-output configuration for a given value of 𝒩 and 𝒫, and we found a single solution with $ℰ<0.001\sqrt{𝒫}$ using a gradient descent learning rate of 0.005.

In Figs. 8(d)–8(e), we show results of a simulation for a 𝒩 = 10, 𝒟 = 4, ℐ = 7, 𝒫 = 8, and 𝒞 = 3 network where the norm of $\overrightarrow{y}$ was fixed to 0.79, which approximates the median value of y for the given values of 𝒩, 𝒫, and 𝒞 [Eq. (B5)]. Our solutions were obtained using a gradient descent learning rate of 0.004 and the overall error satisfied $0.017\sqrt{𝒟}\lesssim ℰ\lesssim 0.25\sqrt{𝒟}$.