Theory of Gating in Recurrent Neural Networks

Abstract

Recurrent neural networks (RNNs) are powerful dynamical models, widely used in machine learning (ML) and neuroscience. Prior theoretical work has focused on RNNs with additive interactions. However gating i.e., multiplicative interactions are ubiquitous in real neurons and also the central feature of the best-performing RNNs in ML. Here, we show that gating offers flexible control of two salient features of the collective dynamics: (i) timescales and (ii) dimensionality. The gate controlling timescales leads to a novel marginally stable state, where the network functions as a flexible integrator. Unlike previous approaches, gating permits this important function without parameter fine-tuning or special symmetries. Gates also provide a flexible, context-dependent mechanism to reset the memory trace, thus complementing the memory function. The gate modulating the dimensionality can induce a novel, discontinuous chaotic transition, where inputs push a stable system to strong chaotic activity, in contrast to the typically stabilizing effect of inputs. At this transition, unlike additive RNNs, the proliferation of critical points (topological complexity) is decoupled from the appearance of chaotic dynamics (dynamical complexity). The rich dynamics are summarized in phase diagrams, thus providing a map for principled parameter initialization choices to ML practitioners.

I. INTRODUCTION

Recurrent neural networks (RNNs) are powerful dynamical systems that can represent a rich repertoire of trajectories and are popular models in neuroscience and machine learning. In modern machine learning, RNNs are used to learn complex dynamics from data with rich sequential or temporal structure such as speech [1,2], turbulent flows [3–5], or text sequences [6]. RNNs are also influential in neuroscience as models to study the collective behavior of a large network of neurons [7] (and references therein). For instance, they have been used to explain the dynamics and temporally irregular fluctuations observed in cortical networks [8,9] and how the motor-cortex network generates movement sequences [10,11].

Classical RNN models typically involve units that interact with each other in an additive fashion—i.e., each unit integrates a weighted sum of the output of the rest of the network. However, researchers in machine learning have empirically found that RNNs with gating—a form of multiplicative interaction—can be trained to perform significantly more complex tasks than classical RNNs [6,12]. Gating interactions are also ubiquitous in real neurons due to mechanisms such as shunting inhibition [13]. Moreover, when single-neuron models are endowed with more realistic conductance dynamics, the effective interactions at the network level have gating effects, which confer robustness to time-warped inputs [14]. Thus, RNNs with gating interactions not only have superior information processing capabilities, but they also embody a prominent feature found in real neurons.

Prior theoretical work on understanding the dynamics and functional capabilities of RNNs has mostly focused on RNNs with additive interactions. The original work by Sompolinsky, Crisanti, and Sommers [15] identifies a phase transition in the autonomous dynamics of randomly connected RNNs from stability to chaos. Subsequent work extends this analysis to cases where the random connectivity additionally has correlations [16], a low-rank structured component [17,18], strong self-interaction [19], and heterogeneous variance across blocks [20]. The role of sparse connectivity and the single-neuron nonlinearity is studied in Ref. [9]. The effect of a Gaussian noise input is analyzed in Ref. [21].

In this work, we study the consequences of gating interactions on the dynamics of RNNs. We introduce a gated RNN model that naturally extends a classical RNN by augmenting it with two kinds of gating interactions: (i) an update gate that acts like an adaptive time constant and (ii) an output gate which modulates the output of a neuron. The choice of these forms for gates are motivated by biophysical considerations (e.g., Refs. [14,22]) and retain the most functionally important aspects of the gated RNNs in machine learning. Our gated RNN reduces to the classical RNN [15,23] when the gates are open and is closely related to the state-of-the-art gated RNNs in machine learning when the dynamics are discretized [24]. We further elaborate on this connection in Sec. VIII.

We develop a theory for the gated RNN based on non-Hermitian random matrix techniques [25,26] and the Martin–Siggia–Rose–De Dominicis-Janssen (MSRDJ) formalism [21,27–32] and use the theory to map out, in a phase diagram, the rich, functionally significant dynamical phenomena produced by gating.

We show that the update gate produces slow modes and a marginally stable critical state. Marginally stable systems are of special interest in the context of biological information processing (e.g., Ref. [33]). Moreover, the network in this marginally stable state can function as a robust integrator—a function that is critical for memory capabilities in biological systems [34–37] but has been hard to achieve without parameter fine-tuning and handcrafted symmetries [38]. Gating permits the network to serve this function without any symmetries or fine-tuning. For a detailed discussion of these issues, we refer the reader to Ref. [39] (pp. 329–350) and Refs. [38,40]. Integratorlike dynamics are also empirically observed in gated machine learning (ML) RNNs successfully trained on complex sequential tasks [41]; our theory shows how gates allow for this robustly.

The output gate allows fine control over the dimensionality of the network activity; control of the dimensionality can be useful during learning tasks [42]. In certain regimes, this gate can mediate an input-driven chaotic transition, where static inputs can push a stable system abruptly to a chaotic state. This behavior with gating is in stark contrast to the typically stabilizing effect of inputs in high-dimensional systems [21,43,44]. The output gate also leads to a novel, discontinuous chaotic transition, where the proliferation of critical points (a static property) is decoupled from the appearance of chaotic transients (a dynamical property); this is in contrast to the tight link between the two properties in additive RNNs as shown by Wainrib and Touboul [45]. This transition is also characterized by a nontrivial state where a stable fixed point coexists with long chaotic transients. Gates also provide a flexible, context-dependent way to reset the state, thus providing a way to selectively erase the memory trace of past inputs.

We summarize these functionally significant phenomena in phase diagrams, which are also practically useful for ML practitioners—indeed, the choice of parameter initialization is known to be one of the most important factors deciding the success of training [46], with best outcomes occurring near critical lines [10,47–49]. Phase diagrams, thus, allow a principled and exhaustive exploration of dynamically distinct initializations.

II. A RECURRENT NEURAL NETWORK MODEL TO STUDY GATING

We study an extension of a classical RNN [15,23] by augmenting it with multiplicative gating interactions. Specifically, we consider two gates: (i) an update (or z) gate which controls the rate of integration and (ii) an output (or r) gate which modulates the strength of the output. The equations describing the gated RNN are given by

$${\dot{h}}_i(t)={\sigma }_z\left(z_i\right)\left[-h_i(t)+R_i(t)\right]+I_i^h(t),$$ (1)

where h_i represents the internal state of the ith unit and σ_(·)(x) = [1 + exp(−α_(·)x + β_(·))]⁻¹ are sigmoidal gating functions. The recurrent input to a neuron is $R_i(t)={\sum }_{j=1}^N{J}_{ij}^h\varphi \left[h_j(t)\right]\cdot {\sigma }_r\left[r_j(t)\right]$, where $J_{ij}^h$ are the coupling strengths between the units and ϕ(x) = tanh(g_hx + β_h) is the activation function. ϕ and σ_z,r are parametrized by gain parameters (g_h, α_z,r) and biases (β_h,z,r), which constitute the parameters of the gated RNN. Finally, I^h represents external input to the network. The gating variables z_i(t) and r_i(t) evolve according to dynamics driven by the output ϕ[h(t)] of the network:

$${\tau }_x{\dot{x}}_i(t)=-x_i(t)+\sum _{j=1}^N{J}_{ij}^x\varphi \left[h_j(t)\right]+I_i^x,$$ (2)

where x ∈ {z, r}. Note that the coupling matrices J^z,r for z, r are distinct from J^h. We also allow for different inputs I^r and I^z being fed to the gates. For instance, they might be zero, or they might be equal up to a scaling factor to I^h.

The value of σ_z(z_i) can be viewed as a dynamical time constant for the ith unit, while the output gate σ_r(r_i) modulates the output strength of unit i. In the presence of external input, the r gate can control the relative strengths of the internal (recurrent) activity and the external input I^h. In the limit σ_z, σ_r → 1, we recover the dynamics of the classical RNN.

We choose the coupling weights from a Gaussian distribution with variance scaled such that the input to each unit remains O(1). Specifically, $J_{ij}^{h,z,r}~𝒩\left(0,N^{-1}\right)$. This choice of couplings is a popular initialization scheme for RNNs in machine learning [6,46] and also in models of cortical neural circuits [15,20]. If the gating variables are purely internal, then (J^z,r) is diagonal; however, we do not consider this case below. In the rest of the paper, we analyze the various dynamical regimes the gated RNN exhibits and their functional significance.

III. HOW THE GATES SHAPE THE LINEARIZED DYNAMICS

We first study the linearized dynamics of the gated RNN through the lens of the instantaneous Jacobian and describe how these dynamics are shaped by the gates. The instantaneous Jacobian describes the linearized dynamics about an operating point, and the eigenvalues of the Jacobian inform us about the timescales of growth and decay of perturbations and the local stability of the dynamics. As we show below, the spectral density of the Jacobian depends on equal-time correlation functions, which are the order parameters in the mean-field picture of the dynamics, developed in the Appendix C. We study how the gates shape the support and the density of Jacobian eigenvalues in the steady state, through their influence on the correlation functions.

The linearized dynamics in the tangent space at an operating point x = (h, z, r) is given by

$$\delta \dot{x}=𝒟(t)\delta x,$$ (3)

where 𝒟 is the 3N × 3N-dimensional instantaneous Jacobian of the full network equations. Linearization of Eqs. (1) and (2) yields

$$𝒟=\left(\begin{array}{ccc}\left[{\sigma }_z\right]\left(-\mathbb{1}+J^h\left[{\varphi }^{\prime }{\sigma }_r\right]\right)& \mathbb{D}& \left[{\sigma }_z\right]J^h\left[\varphi {\sigma }_r^{\prime }\right]\\ {\tau }_z^{-1}{J}^z\left[{\varphi }^{\prime }\right]& -{\tau }_z^{-1}\mathbb{1}& 0\\ {\tau }_r^{-1}{J}^r\left[{\varphi }^{\prime }\right]& 0& -{\tau }_r^{-1}\mathbb{1}\end{array}\right),$$ (4)

where [x] denotes a diagonal matrix with the diagonal entries given by the vector x. The term ${\mathbb{D}}_{ij}={\delta }_{ij}{\sigma }_z^{\prime }\left(z_i\right)\left(-h_i+{\sum }_j{J}_{ij}^h\varphi \left(h_j\right){\sigma }_r\left(r_j\right)\right)=\left[-{\sigma }_z^{\prime }(z)h\right]+\left[{\sigma }_z^{\prime }\odot J^h\left(\varphi \odot {\sigma }_r\right)\right]$ arises when we linearize about a time-varying state and is zero for fixed points. We introduce the additional shorthand ϕ′(t) = ϕ′(h(t)) and ${\sigma }_{r/z}^{\prime }={\sigma }_{r/z}^{\prime }(r/z(t))$.

The Jacobian is a block-structured matrix involving random elements (J^z,h,r) and functions of various state variables. We need additional tools from non-Hermitian random matrix theory (RMT) [26] and dynamical mean-field theory (DMFT) [15] to analyze the spectrum of the Jacobian 𝒟. We provide a detailed, self-contained derivation of the calculations in Appendix C (DMFT) and Appendix A (RMT). Here, we state only the main results derived from these formalisms.

One of the main results is an analytical expression for the spectral curve, which describes the boundary of the Jacobian spectrum, in terms of the moments of the state variables. The most general expression for the spectral curve [Appendix A, Eq. (A34)] involves empirical averages over the 3N-dimensional state variables. However, for large N, we can appeal to a concentration of measure argument to replace these discrete sums with averages over the steady-state distribution from the DMFT (cf. Appendix C)—i.e., we can replace empirical averages of any function of the state variables (1/N) Σ_i F(h_i, z_i, r_i) with 〈F[h(t), z(t), r(t)]〉, where the brackets indicate average over the steady-state distribution. The DMFT + RMT prediction for the spectral curve for a generic steady-state point is given in Appendix A, Eq. (A35). Strictly speaking, the analysis of the DMFT around a generic time-dependent steady state is complicated by the fact that the distribution for h is not Gaussian (while r and z are Gaussian). For fixed points, however, the distributions of h, z, and r are all Gaussian, and the expression for the spectral curve reduces simplifies. It is given by the set of $\lambda \in ℂ$ which satisfy

$$\langle {\varphi }^{\prime 2}\rangle \left(\langle {\sigma }_r^2\rangle +\frac{\langle {\varphi }^2\rangle \langle {\sigma }_r^{\prime 2}\rangle }{{\left|1+{\tau }_r\lambda \right|}^2}\right){\langle \frac{{\sigma }_z^2}{{\left|\lambda +{\sigma }_z\right|}^2}\rangle }_z=1.$$ (5)

Here, the averages are taken over the Gaussian fixed-point distributions (h, z, r) ~ 𝒩(0, Δ_h,z,r) which follow from the MFT [Eq. (C26)]. For example, $\langle {\varphi }^{\prime 2}\rangle ={\mathbb{E}}_{h~𝒩\left(0,{\Delta }_h\right)}\left[{\varphi }^{\prime }{(h)}^2\right]$.

We make two comments on the Jacobian of a time-varying state: (i) One might wonder if any useful information can be gleaned by studying the Jacobian at a time-varying state where the Hartman-Grobman theorem is not valid. Indeed, as we see below, the limiting form of the Jacobian in steady state crucially informs us about the suppression of unstable directions and the emergence of slow dynamics due to pinching and marginal stability in certain parameter regimes (also see Ref. [50]). In other words, the instantaneous Jacobian charts the approach to marginal stability and provides a quantitative justification for the approximate integrator functionality exhibited in Sec. IV B. (ii) Interestingly, the spectral curve calculated using the MFT [Eq. (5)] for a time-varying steady state not deep in the chaotic regime is a very good approximation for the true spectral curve (see Fig. 8 in Appendix A).

Figures 1(a)–1(d) show that the RMT prediction of the spectral support (dark outline) agrees well with the numerically calculated spectrum (red dots) in different dynamical regimes. As a consequence of Eq. (5), we get a condition for the stability of the zero fixed point. The leading edge of the spectral curve for the zero fixed point (FP) crosses the origin when $g_h<1+e^{-{\beta }_r}$. So, in the absence of biases, g_h > 2 makes the zero FP unstable. More generally, the leading edge of the spectrum crossing the origin gives us the condition for the FP to become unstable:

$$\langle {\varphi }^{\prime 2}\rangle \left(\langle {\varphi }^2\rangle \langle {\sigma }_r^{\prime 2}\rangle +\langle {\sigma }_r^2\rangle \right)>1\Rightarrow \text{ unstable FP}.$$ (6)

FIG. 1.

How gates shape the Jacobian spectrum. (a)–(d) Jacobian eigenvalues (red dots) of the gated RNN in (time-varying) steady state. The dark outline is the spectral support curve predicted by Eq. (5). The bottom row corresponds to larger α_z, and the right column corresponds to large α_r. (e) Cumulative distribution function of Jacobian eigenvalues in a disk of radius r = 0.05 centered at the origin plotted against α_z. Circles are numerical density calculated from the true network Jacobian (averaged over ten instances), and the dashed line is a fit from Eq. (7). (f) Intercept of the spectral curve on the imaginary axis, plotted against α_z for three different values of g_h (α_r = 0). For network simulations, N = 2000, g_h = 3, and τ_r = τ_z = 1 unless otherwise stated, and all biases are zero.

We see later on that the time-varying state corresponding to this regime is chaotic. We now proceed to analyze how the two gates shape the Jacobian spectrum via the equation for the spectral curve.

A. Update gate facilitates slow modes and output gate causes instability

To understand how each gate shapes the local dynamics, we study their effect on the density of Jacobian eigenvalues and the shape of the spectral support curve—the eigenvalues tell us about the rate of growth or decay of small perturbations and, thus, timescales in the local dynamics, and the spectral curve informs us about stability. For ease of exposition, we consider the case without biases in the main text (β_r,z,h = 0); we discuss the role of biases in Appendix H.

Figure 1 shows how the gain parameters of the update and output gates—α_z and α_r, respectively—shape the Jacobian spectrum. In Figs. 1(a)–1(d), we see that α_z has two salient effects on the spectrum: Increasing α_z leads to (i) an accumulation of eigenvalues near zero and (ii) a pinching of the spectral curve for certain values of g_h wherein the intercept on the imaginary axis gets smaller [Fig. 1(f); also see Sec. IVA]. In Figs. 1(a)–1(d), we also see that increasing the value of α_r leads to an increase in the spectral radius, thus pushing the leading edge (max Reλ_i) to the right and thereby increasing the local dimensionality of the unstable manifold. This behavior of the linearized dynamics is also reflected in the nonlinear dynamics, where, as we show in Sec. V, α_r has the effect of controlling the dimensionality of full phase-space dynamics.

The accumulation of eigenvalues near zero with increasing α_z suggests the emergence of a wide spectrum of timescales in the local dynamics. To understand this accumulation quantitatively, it is helpful to consider the scenario where α_z is large and we replace the tanh activation functions with a piecewise linear approximation. In this limit, the density of eigenvalues within a radius δ of the origin is well approximated by the following functional form (details in Appendix B):

$$P\left[\left|\lambda \left({𝒟}_x\right)\right|<\delta \right]~c_0\text{erf}\left(\frac{c_1}{{\alpha }_z}\right),$$ (7)

where c₀ and c₁ are constants that, in general, depend on a_r, δ, and g_h. Figure 1(e) shows this scaling for a specific value of δ: The dashed line shows the predicted curve, and the circles indicate the actual eigenvalue density calculated using the full Jacobian. In the limit of α_z → ∞, we get an extensive number of eigenvalues at zero, and the eigenvalue density converges to (see Appendix B)

$$\mu (\lambda )=\left(1-f_z\right)\delta (\lambda )+f_z\left(1-f_h\right)\delta (\lambda +1)+\frac{4}{\pi g_h^2}{\mathbb{I}}_{\left\{|\lambda |\le g_h^2/4\right\}},$$

where f_z = 〈σ_z(z)〉 is the fraction of update gates which are nonzero and f_h is the fraction of unsaturated activation functions ϕ(h). For other choices of saturating nonlinearities, the extensive number of eigenvalues at zero remains; however, the expressions are more complicated. Analogous phenomena are observed for discrete-time gated RNNs in Ref. [51], using a similar combination of analytical and numerical techniques [52].

In Sec. VA, we show that the slow modes, as seen from linearization, persist asymptotically (i.e., in the nonlinear regime). This can be seen from the Lyapunov spectrum in Fig. 3(a), which for large α_z exhibits an analogous accumulation of Lyapunov exponents near zero.

FIG. 3.

Lyapunov spectra and dimensionality of the gated RNN. (a),(b) The first 50 ordered Lyapunov exponents for a gated RNN (N = 2000) as a function of varying (a) α_z and (b) α_r. The Lyapunov spectrum is calculated as described in Appendix D. (c) The Kaplan-Yorke dimensionality of the dynamics as a function of α_r. (d) The maximal Lyapunov exponent λ_max predicted by the DMFT [solving Eqs. (10) and (11); solid line] and obtained numerically using the QR method (circles; N = 2000 and α_z = 0). Note that the transition for α_r = 20 is sharp; also cf. Fig. 5(c). τ_z = τ_t = 2.0 here.

In the next section, we study the profound functional consequences of the combination of spectral pinching and accumulation of eigenvalues near zero.

IV. MARGINAL STABILITY AND ITS CONSEQUENCES

As the update gate becomes more switchlike (higher α_z), we see an accumulation of slow modes and a pinching of the spectral curve which drastically suppresses the unstable directions. In the limit α_z → ∞, this can make previously unstable points marginally stable by pinning the leading edge of the spectral curve exactly at zero. Marginally stable systems are of significant interest because of the potential benefits in information processing—for instance, they can generate long timescales in their collective modes [33,39]. Moreover, achieving marginal stability often requires fine-tuning parameters close to a bifurcation point. As we see, gating allows us to achieve a marginally stable critical state over a wide range of parameters; this has been typically highly nontrivial to achieve (e.g., Ref. [39], pp. 329–350, and Ref. [33]). We first investigate the conditions under which marginal stability arises, and then we touch on one of its important functional consequences: the appearance of “line attractors” which allow the system to be used as a robust integrator.

A. Condition for marginal stability

Marginal stability is a consequence of pinching of the spectral curve with increasing α_z, wherein the (positive) leading edge of the spectrum and the intercept of the spectral curve on the imaginary axis both shrink with α_z [e.g., Fig. 1(f) and compare Figs. 1(a) and 1(c)]. However, we see in Fig. 1(f) (via the intercept) that pinching does not happen if g_h is sufficiently large (even as α_z → ∞). Here, we provide the conditions when pinching can occur and, thus, marginal stability can emerge. For simplicity, let us consider the case where τ_r = 1 and there are no biases.

Marginal stability strictly exists only for α_z = ∞. We first examine the conditions under which the system can become marginally stable in this limit, and then we explain the route to marginal stability for large but finite α_z, i.e., how a time-varying state ends up as a marginally stable fixed point. For α_z = ∞, the spectral density has an extensive number N[1 − 〈σ_z(z)〉] of zero eigenvalues, and the remaining eigenvalues are distributed in a disk centered at λ = −1 with radius ρ. If ρ < 1, then the spectral density has two topologically disconnected configurations (the disk and the zero modes) and the system is marginally stable. If ρ > 1, the zero modes get absorbed in the interior of the disk and the system is unstable with fast, chaotic dynamics. The radius ρ is given by ${\rho }^2=\frac{1}{2}a+\frac{1}{2}\sqrt{4b+a^2}<1$, where $a=\langle {{\varphi }^{\prime }}^2\rangle \langle {\sigma }_z\rangle \langle {\sigma }_r^2\rangle$ and $b=\langle {\varphi }^{\prime 2}\rangle \langle {\sigma }_z\rangle \langle {\varphi }^2\rangle \langle {\sigma }_r^{\prime 2}\rangle$. This follows from Eq. (5) by evaluating the z-expectation value assuming σ_z is a binary variable. Thus, the system is marginally stable in the limit α_z = ∞ as long as

$$\langle {\varphi }^{\prime 2}\rangle \left(\langle {\varphi }^2\rangle \langle {\sigma }_r^{\prime 2}\rangle +\langle {\sigma }_r^2\rangle \right)<{\langle {\sigma }_z\rangle }^{-1}.$$ (8)

The crucial difference between this expression and Eq. (6) is that the rhs now has a factor of 〈σ_z〉⁻¹ which can be greater than unity, thus pushing the transition to chaos further out along the g_h and α_r directions, as depicted in the phase diagram (Fig. 7). For concreteness, we report here how the transition changes at α_r = 0. In this setting, the transition to chaos moves from g_h = 2 to g_h ⪅ 6.2, and the system is marginally stable for 2 < g_h ⪅ 6.2.

FIG. 7.

Phase diagram for the gated RNN. (a) (no biases) In regions 1 and 2, the zero FP is the global attractor of dynamics; however, in region 2, there is a proliferation of unstable FPs without any asymptotic dynamical signatures. In region 3, the (stable) zero FP coexists with chaotic dynamics. Note that the plotted curve separating regions 2 and 3 is computed for α_z = 0 and remains valid for sufficiently small values of α_z. In region 4, the zero FP is unstable, and dynamics are chaotic. For all parameter values in region 5, a previously unstable or chaotic state can be made marginally stable when α_z = ∞. For any given parameter values in region 5, there are infinitely many marginally stable points in the phase space to which the dynamics converge. The red dashed line indicates the critical transition between a stable fixed point (below the line) and chaos (above the line) in the presence of static random input (to the h variable) with standard deviation σ_h = 0.5. Note that, while chaos is suppressed for small α_r along the g_h axis, for larger α_r there are regions of stable FPs that become chaotic with finite input. This leads to the phenomenon of input-induced chaos.

Having identified the region in the phase diagram that can be made marginally stable for α_z = ∞, we can now discuss the route to marginal stability for large but finite α_z. In other words, how does an unstable chaotic state become marginally stable with increasing α_z? Since the marginally stable region is characterized by a disconnected spectral density, evidently increasing α_z must lead to singular behavior in the spectral curve. This takes the form of a pinching at the origin. We show that, for values of g_h supporting marginal stability, the leading edge λ_e of the spectrum for the time-varying state gets pinched exponentially fast with α_z as ${\lambda }_e~e^{-c{\alpha }_z\sqrt{{\Delta }_h}}$ (see Appendix B). This accounts for the fact that, already for α_z = 15, we observe the pinching in Fig. 1(c). In contrast, the parameters in Fig. 1(d) lie outside the marginally stable region, and, thus, there is no pinching, since the zero modes are asymptotically (in α_z) buried in the bulk of the spectrum.

In summary, as α_z → ∞ the Jacobian spectrum undergoes a topological transition from a single simply connected domain to two domains, both containing an extensive number of eigenvalues. A finite fraction of eigenvalues end up sitting exactly at zero, while the rest occupy a finite circular region. If the leading edge of the circular region crosses zero in this limit, then the state remains unstable; otherwise, the state becomes marginally stable. The latter case is achieved through a gradual pinching of the spectrum near zero; there is no pinching in the former case.

We emphasize that marginal stability requires more than just an accumulation of eigenvalues near zero. Indeed, this happens even when g_h is outside the range supporting marginal stability as α_z → ∞, but there is no pinching and the system remains unstable [e.g., see Fig. 1(d)]. We return to this when we describe the phase diagram for the gated RNN (Sec. VII). There, we see that the marginally stable region occupies a macroscopic volume in the parameter space adjoining the critical lines on one side.

B. Functional consequences of marginal stability

The marginally stable critical state produced by gating can subserve the function of a robust integrator. This integratorlike function is crucial for a variety of computational functions such as motor control [34–36], decision making [37], and auditory processing [53]. However, achieving this function has typically required fine-tuning or special handcrafted architectures [38], but gating permits the integrator function over a range of parameters and without any specific symmetries in J^h,z,r. Specifically, for large α_z, any perturbation in the span of the eigenvectors corresponding to the eigenvalues with a magnitude close to zero is integrated by the network, and, once the input perturbation ceases, the memory trace of the input is retained for a duration much longer than the intrinsic time constant of the neurons; perturbations along other directions, however, relax with a spectrum of timescales dictated by the inverse of (the real part of) their eigenvalues. Thus, the manifold of slow directions forms an approximate continuous attractor on which input can effortlessly move the state vector around. These approximate continuous attractor dynamics are illustrated in Fig. 2. At time t = 0, an input I^h (with I^r = I^z = 0) is applied till t = 10 (between dashed vertical lines) along an eigenvector of the Jacobian with an eigenvalue close to zero. Inputs along this slow manifold with varying strengths (different shades of red) are integrated by the network as evidenced by the excess projection of the network activity on the left eigenvector u_λ corresponding to the slow mode; on the other hand, inputs not aligned with the slow modes decay away quickly (dashed black line). Recall that the intrinsic time constant of the neurons here is set to one unit. The exponentially fast (in α_z) pinching of the spectral curve (discussed above in Sec. IVA) suggests this slow-manifold behavior should also hold for moderately large α_z (as in Fig. 2). Therefore, even though the state is technically unstable, the local structure of the Jacobian is responsible for giving rise to extremely long timescales and allows the network to operate as an approximate integrator within relatively long windows of time, as demonstrated in Fig. 2.

FIG. 2.

Network in the marginally stable state functions as an integrator. (a) Sample traces from a network with switchlike update gates (α_z = 30, g_h = 3) show slow evolution (time on x axis is relative to τ_h). (b) An input is applied to the same network in (a) from t = 0 till t = 10, either aligned with a slow eigenvector u_λ (red traces) or unaligned with slow modes (black dashed trace). The plot shows the excess projection of the network state on the left eigenvector u_λ. Different shades of red correspond to different input strengths. If the input is along the slow manifold, the trace of the input is retained for a long time after the cessation of input. [The traces in (a) are for the network with an input along the manifold.].

Of course, after sufficiently long times, the instability causes the state to evolve and the memory is lost. Exactly how long the memory lasts depends on the asymptotic stability of the network, which is revealed by the Lyapunov spectrum, discussed below in Sec. VA.

V. OUTPUT GATE CONTROLS DIMENSIONALITY AND LEADS TO A NOVEL CHAOTIC TRANSITION

We thus far use insights from local dynamics to study the functional consequences of the gates. To study the salient features of the output gate, it is useful to analyze the effect of inputs and the long-time behavior of the network through the lens of Lyapunov spectra. We see that the output gate controls the dimensionality of the dynamics in the phase space; dimensionality is a salient aspect of the dynamics for task function [42]. The output gate also gives rise to a novel discontinuous chaotic transition, near which inputs (even static ones) can abruptly push a stable system into strongly chaotic behavior—contrary to the typically stabilizing effect of inputs. Below, we begin with the Lyapunov analyses of the dynamics and then proceed to study the chaotic transition.

A. Long-time behavior of the network

We study the asymptotic behavior of the network and the nature of the time-varying state through the lens of its Lyapunov spectra. In this section, where we study the effects of α_z, our results are numerical except in cases where α_z = 0 [e.g., in Fig. 3(d)]. Lyapunov exponents specify how infinitesimal perturbations δx(t) grow or shrink along the trajectories of the dynamics—in particular, if the growth or decay is exponentially fast, then the rate is dictated by the maximal Lyapunov exponent defined as [54] λ_max ≔ lim_T→∞ T⁻¹ lim_{‖δx(0)‖→0} ln[‖δx(T)‖/‖δx(0)‖]. More generally, the set of all Lyapunov exponents—the Lyapunov spectrum—yields the rates at which perturbations along different directions shrink or diverge and, thus, provide a fuller characterization of asymptotic behavior. We first numerically study how the gates shape the full Lyapunov spectrum (details in Appendix D) and derive an analytical prediction for the maximum Lyapunov exponent using the DMFT (Sec. VA1) [55].

Figures 3(a) and 3(b) show how the update (z) and output (r) gates shape the Lyapunov spectrum. We see that, as the update gets more sensitive (larger α_z), the Lyapunov spectrum flattens, pushing more exponents closer to zero, generating long timescales. As the output gate becomes more sensitive (larger α_r), all Lyapunov exponents increase, thus increasing the rate of growth in unstable directions.

We can estimate the dimensionality of the activity in the chaotic state by calculating an upper bound D_A on the dimension according to a conjecture by Kaplan and Yorke [54]. The Kaplan-Yorke upper bound for the attractor dimension D_A is given by

$$D_A=M+\frac{{\sum }_{i=1}^M{\lambda }_i}{\left|{\lambda }_{M+1}\right|}\text{, where }M=\underset{j}{\text{max}}\left\{\sum _{i=1}^j{\lambda }_i\ge 0\right\},$$ (9)

where λ_i are the rank-ordered Lyapunov exponents. We see in Fig. 3(c) that the sensitivity of the output gate (α_r) can shape the dimensionality of the dynamics—a more sensitive output gate leads to higher dimensionality. As we see below, this effect of the output gate is different from how the gain g_h shapes dimensionality and can lead to a novel chaotic transition. Even more directly, if the r gate for neurons i₁…i_K is set to zero, then the activity is constrained to evolve in an N − K-dimensional subspace; however, the r gate allows the possibility—i.e., the “inductive bias”—of doing this dynamically.

1. DMFT prediction for λ_max

We would also like to study the chaotic nature of the time-varying phase by means of the maximal Lyapunov exponent and characterize when the transition to chaos occurs. We extend the DMFT for the gated RNN to calculate the maximum Lyapunov exponent, and, to do this, we make use of a technique suggested by Refs. [56,57] and clearly elucidated in Ref. [21]. The details are provided in Appendix E, and the end result of the calculation is the DMFT prediction for λ_max as the solution to a generalized eigenvalue problem for κ involving the correlation functions of the state variables:

$$\left[{\left(\langle {\sigma }_z\rangle +\kappa \right)}^2-{\partial }_{\tau }^2+C_{{\sigma }_z}(\tau )-{\langle {\sigma }_z\rangle }^2\right]{\chi }_h(\tau )=C_{{\sigma }_z^{\prime }}(\tau )\left[C_{\varphi \cdot {\sigma }_r}(\tau )-C_h(\tau )\right]{\chi }_z(\tau )+C_{{\sigma }_z}(\tau )\frac{\partial C_{\varphi \cdot {\sigma }_r}(\tau )}{\partial C_h}{\chi }_h(\tau ),$$ (10)

$$\left[{\left(1+{\tau }_{z/r}\kappa \right)}^2-{\tau }_{z/r}^2{\partial }_{\tau }^2\right]{\chi }_{z/r}(\tau )=\frac{\partial C_{\varphi }(\tau )}{\partial C_h}{\chi }_h(\tau ),$$ (11)

where we denote the two-time correlation function C_x(t, t′) ≡ 〈x(t)x(t′)〉 for different (functions of) state variables x(t) [see Eq. (C25) for more context]. The largest eigenvalue solution to this problem is the required maximal Lyapunov exponent [58]. Note that this is the analog of the Schrodinger equation for the maximal Lyapunov exponent in the vanilla RNN. When α_z = 0 (or small), the h field is Gaussian, and we can use Price’s theorem for Gaussian integrals to replace the variational derivatives on the rhs of Eqs. (10) and (11) by simple correlation functions, for instance, ∂C_ϕ(τ)/∂C_h(τ) = C_ϕ′(τ). In this limit, we see good agreement between the numerically calculated maximal Lyapunov exponent [Fig. 3(c), dots] compared to the DMFT prediction [Fig. 3(c), solid line] obtained by solving the eigenvalue problem [Eqs. (10) and (11)]. For large values of α_z, we see quantitative deviations between the DMFT prediction and the true λ_max. Indeed, for large α_z, the distribution of h is strongly non-Gaussian, and there is no reason to expect that variational formulas given by Price’s theorem are even approximately correct. For more on this point, see the discussion toward the end of Appendix C.

2. Condition for continuous transition to chaos

The value of α_z affects the precise value of the maximal Lyapunov exponent λ_max; however, numerics suggest that, across a continuous transition to chaos, the point at which λ_max becomes positive is not dependent on α_z (data not shown). We can see this more clearly by calculating the transition to chaos when the leading edge of the spectral curve (for a FP) crosses zero. This condition is given by Eq. (6), and we see that it has no dependence on α_z or the update gate. We stress that this condition [Eq. (6)] for the transition to chaos—when the stable fixed point becomes unstable—is valid when the chaotic attractor emerges continuously from the fixed point [Fig. 3(c), α_r = 0, 2]. However, in the gated RNN, there is another discontinuous transition to chaos [Fig. 3(c), α_r = 20]: For sufficiently large α_r, the transition to chaos is discontinuous and occurs at a value of g_h where the zero FP is still stable (g_h < 2 with no biases). To our knowledge, this is a novel type of transition which is not present in the vanilla RNN and not visible from an analysis that considers only the stability of fixed points. We characterize this phenomenon in detail below.

B. Output gate induces a novel chaotic transition

Here, we describe a novel phase, characterized by a proliferation of unstable fixed points and the coexistence of a stable fixed point with chaotic dynamics. It is the appearance of this state that gives rise to the discontinuous transition observed in Fig. 3(c). The appearance of this state is mediated by the output gate becoming more switchlike (i.e., increasing α_r) in the quiescent region for g_h. To our knowledge, no such comparable phenomenon exists in RNNs with additive interactions. The full details of the calculations for this transition are provided in Appendix G. Here, we simply state and describe the salient features. For ease of presentation, the rest of the section assumes that all biases are zero. The results in this section are strictly valid only for α_z = 0. In Appendix G3, we argue that they should also hold for moderate α_z.

This discontinuous transition is characterized by a few noteworthy features.

1. Spontaneous emergence of fixed points

When g_h < 2.0, the zero fixed point is stable. Moreover, for $\sqrt{2}<g_h<2$, when α_r crosses a threshold value ${\alpha }_{r,\text{FP}}^{*}\left(g_h\right)$, unstable fixed points spontaneously appear in the phase space. The only dynamical signature of these unstable FPs are short-lived transients which do not scale with system size (see Fig. 11). Thus, we have a condition for fixed-point transition:

$$\sqrt{2}<g_h\le 2\text{\hspace{1em} and \hspace{1em}}{\alpha }_r>{\alpha }_{r,\text{FP}}^{*}\left(g_h\right).$$ (12)

These unstable fixed points correspond to the emergence of nontrivial solutions to the time-independent MFT. Figure 4(a) shows the appearance of fixed-point MFT solutions for a fixed g_h, and Fig. 4(b) shows the critical ${\alpha }_{r,\text{FP}}^{*}\left(g_h\right)$ as a function of g_h. As g_h → 2⁻, we see that ${\alpha }_{r,\text{FP}}^{*}\to \sqrt{8}$.

FIG. 4.

The discontinuous dynamical transition. (a) Spontaneous appearance of nonzero solutions (dashed and solid red lines) to the FP equations once α_r crosses a critical value ${\alpha }_{r,\text{FP}}^{*}\left(g_h\right)$ at fixed g_h. (b) The critical ${\alpha }_{r,\text{FP}}^{*}\left(g_h\right)$ as a function of g_h. The vertical dashed line represents left critical value $g_c=\sqrt{2}$, below which a bifurcation is not possible. (c) The critical DMFT transition curve ${\alpha }_{r,\text{DMFT}}^{*}\left(g_h\right)$ (red curve) calculated using Eqs. (G8) and (G9). The FP transition curve from (b) is shown in black. The green dashed line corresponds to $g_c=\sqrt{8/3}$, below which the dynamical transition is not possible. (d) Numerically calculated maximum Lyapunov exponent λ_max as a function of α_r for two different values of g_h. The dashed lines correspond to the DMFT prediction for the discontinuous transition from (c). (e) Schematic of the bifurcation transition: For g_h < 2 and ${\alpha }_r<{\alpha }_{r,\text{FP}}^{*}$, the zero FP is the only (stable) solution (bottom left box); for $\sqrt{2}<g_h<2$ and ${\alpha }_{r,\text{FP}}^{*}<{\alpha }_r<{\alpha }_{r,\text{DMFT}}^{*}$, the zero FP is still stable, but there is a proliferation of unstable FPs without any obvious dynamical signature (top left); for $\sqrt{8/3}<g_h<2$ and ${\alpha }_r>{\alpha }_{r,\text{DMFT}}^{*}$, chaotic dynamics coexist with the stable FP and this transition is discontinuous (top right); finally, for g_h > 2.0, the stable FP becomes unstable, and only the chaotic attractor remains; this transition is continuous (bottom right).

These spontaneous MFT fixed-point solutions are unstable according to the criterion Eq. (6) derived from RMT. Moreover, in Appendix J, using a Kac-Rice analysis, we show that in this region the full 3N-dimensional system does indeed have a number of unstable fixed points that grows exponentially fast with N. Thus, this transition line ${\alpha }_{r,\text{FP}}^{*}$ represents a topological trivialization transition as conceived by, e.g., Refs. [59,60]. Our analysis shows that instability is intimately connected to the proliferation of fixed points. Remarkably, however, a time-dependent solution to the DMFT does not emerge across this transition, and the microscopic dynamics are insensitive to the transition in topological complexity, bringing us to the next point.

2. A delayed dynamical transition that shows a decoupling between topological and dynamical complexity

On increasing α_r beyond ${\alpha }_{r,\text{FP}}^{*}$, there is a second transition when α_r crosses a critical value ${\alpha }_{r,\text{DMFT}}^{*}$. This happens when we satisfy the condition for dynamical transition:

$$\sqrt{\frac{8}{3}}<g_h\le 2\text{\hspace{1em} and \hspace{1em}}{\alpha }_r>{\alpha }_{r,\text{DMFT}}^{*}\left(g_h\right),$$ (13)

derived in Appendix G2. Figure 4(c) shows how ${\alpha }_{r,\text{DMFT}}^{*}\left(g_h\right)$ varies with g_h. As g_h → 2⁻, we see that ${\alpha }_{r,\text{DMFT}}^{*}\to \sqrt{12}$. Across this transition, a dynamical state spontaneously emerges, and the maximum Lyapunov exponent jumps from a negative value to a positive value [Fig. 4(d)]. This state exhibits chaotic dynamics that coexist with the stable zero fixed point. The presence of the stable FP means that the dynamical state is not strictly a chaotic attractor but rather a spontaneously appearing “chaotic set.” On increasing g_h beyond 2.0, for large but fixed α_r, the stable fixed point disappears, and the state smoothly transitions into a full chaotic attractor that is characterized above. This picture is summarized in the schematic in Fig. 4(e). This gap between the proliferation of unstable fixed points and the appearance of the chaotic dynamics differs from the result of Wainrib and Touboul [45] for purely additive RNNs, where the proliferation (topological complexity) is tightly linked to the chaotic dynamics (dynamical complexity). Thus, for gated RNNs, there appears to be another distinct mechanism for the transition to chaos, and the accompanying transition is a discontinuous one.

3. Long chaotic transients

For finite systems, across the transition the dynamics eventually flow into the zero FP after chaotic transients. Moreover, we expect this transient time to scale with the system size, and, in the infinite system size limit, the transient time should diverge in spite of the fact that the stable fixed point still exists. This is because the relative volume of the basin of attraction of the fixed point vanishes as N → ∞. In Appendix G [Figs. 11(c) and 11(d)], we do indeed see that the transient time for a fixed g_h scales with system size [Fig. 11(c)] once α_r is above the second transition (dashed line) and not otherwise [see Figs. 11(a) and 11(e), dashed lines].

4. An input-induced chaotic transition

The discontinuous chaotic transition has a functional consequence: Near the transition, static inputs can push a stable system to strong chaotic activity. This is in contrast to the typically stabilizing effects of inputs on the activity of random additive RNNs [21,43,44]. In Figs. 5(a) and 5(b), we see that, when static input with variance ${\sigma }_{{\beta }_h}$ is applied to a stable system (a) near the discontinuous chaotic transition (in region 2 in Fig. 7), it induces chaotic activity (b); however, for the same input when applied to the system in the chaotic state [Fig. 5(c)], the dynamics are stabilized (d) as reported before.

FIG. 5.

Input-driven chaos. (a),(b) Near the discontinuous chaotic transition (in region 2 in Fig. 7), static input I^h (with I^r = I^z = 0) can push a stable system (a) to chaotic activity (b). (c),(d) In the purely chaotic state [(c), g_h = 3.0], input has the familiar effect of stabilizing the dynamics (d). The elements of the input vector I^h are random Gaussian variables with zero mean and variance ${\sigma }_{{\beta }_h}^2$.

This phenomenon for static inputs can be understood using the phase diagram with nonzero biases, discussed in Sec. VII. There, we see how the transition curves move when a random bias β_h is included. Near the classic chaotic transition (α_r = 0), the bias moves the curve toward larger g_h, thus suppressing chaos. Near the discontinuous chaotic transition ${\alpha }_{r,\text{DMFT}}^{*}$, the bias pulls the curve toward smaller values of α_r, thus promoting chaos. Thus, inputs can have opposite effects of inducing or stabilizing chaos within the same model in different parameter regimes. This phenomenon could, in principle, be leveraged for shaping the interaction between inputs and internal dynamics.

VI. GATES PROVIDE A FLEXIBLE RESET MECHANISM

Here, we discuss how the gates provide another critical function—a mechanism to flexibly reset the memory trace depending on external input or the internal state. This function complements the memory function; a memory that cannot be erased when needed is not very useful. To build intuition, let us consider a linear network ḣ = −h + Jh, where the matrix $-\mathbb{1}+J$ has a few eigenvalues that are zero, while the rest have a negative real part. The slow modes are good for memory function; however, that fact also makes it hard to forget memory traces along the slow modes. This trade-off is pointed out in Ref. [61]. To be functionally useful, it is critical that the memory trace can be erased flexibly in a context-dependent manner. The r gate allows this function naturally. Consider the same net, but now augmented with an r gate: ḣ = −h + Jh ⊙ σ_r. If the gate is turned off (σ_r = 0) for a short duration, the state h is reset to zero. One can actually be more specific: We may choose a $J^r=-\mathbb{1}{u}^T$ with σ_r = σ[J^r(ϕh)], such that the r gate turns off whenever ϕ(h) gets aligned with u, thus providing an internal-context-dependent reset.

Apart from resetting to zero, the z gate also allows the possibility of rapidly scrambling the state to a random value by means of the input-induced chaos. This phenomenon is illustrated in Fig. 6, where the network in the marginally stable state normally functions as a memory (retains traces for long times, as in Fig. 2), but positive inputs I^z (with I^h = I^r = 0) to the z gate above a threshold strength even for a short duration can induce chaos, thereby scrambling the state and erasing the previous memory state (Fig. 6, bottom panel). The mechanism for this scrambling can be understood by appealing to Eq. (8). A finite input I^z with nonzero mean is able to change 〈σ(z)〉 and, thus, push the critical line for marginal stability in one way or the other. For instance, if 〈I^z〉 > 0, 〈σ(z)〉 > 1/2, which (for α_r = 0) moves the transition to marginal stability to a smaller value of g_h. This implies that a marginally stable state can be made chaotic in the presence of I^z with finite mean. This mechanism for input-induced chaos actually appears to be different from that explored in the previous section, which occurs across the discontinuous chaotic transition. We discuss this more in Sec. VII.

FIG. 6.

Gates provide a reset mechanism. Positive static inputs are applied to the z gate when the RNN is in the marginally stable state (g_h = 3.0, α_r = 2.5, and α_z = ∞) for 20 time units at times indicated by dashed lines. The input induces chaos which rapidly scrambles the network state, thus erasing the trace of the previous memory; the bottom panel shows the normalized projection of the state h(t) on the directions h^(1,2,3) aligned with the state in regions 1, 2, and 3.

In summary, gating imbues the RNN with the capacity to flexibly reset memory traces, providing an “inductive bias” for context-dependent reset. The specific method of reset depends on the task or function, and this can be selected, e.g., by gradient-based training. This inductive bias for resetting is found to be critical for performance in ML tasks [62].

VII. PHASE DIAGRAMS FOR THE GATED NETWORK

Here, we summarize the rich dynamical phases of the gated RNN and the critical lines separating them. The key parameters determining the critical lines and the phase diagram are the activation and output-gate gains and the associated biases: (g_h, β_h, α_r, β_r). The update gate does not play a role in determining continuous or critical chaotic transitions. On the other hand, it influences the discontinuous transition to chaos for sufficiently large values of α_z (see Sec. G3 for discussion). Furthermore, the update gate has a strong effect on the dynamical aspects of the states near the critical lines. There are macroscopic regions of the parameter space adjacent to the critical lines where the states can be made marginally stable in the limit of α_z → ∞. The shape of this marginal stability region is influenced by β_z and I^z.

Figure 7(a) shows the dynamical phases for the network with no biases in the (g_h, α_r) plane. When g_h is below 2.0 and ${\alpha }_r<{\alpha }_{r,\text{FP}}^{*}$, the zero fixed point is the only solution (region 1). As discussed in Sec. VB, on crossing the fixed-point bifurcation line [green line, Fig. 7(a)], there is a spontaneous proliferation of unstable fixed points in the phase space (region 2). This can occur only when $g_h>\sqrt{2}$. The proliferation of fixed points is not accompanied by any obvious dynamical signatures. However, if $\sqrt{8/3}<g_h<2$, we can increase α_r further to cross a second discontinuous transition where a dynamical state spontaneously appears featuring the coexistence of chaotic activity and a stable fixed point (region 3). When g_h is increased beyond the critical value of 2.0, the stable zero fixed point becomes unstable for all α_r, and we get a chaotic attractor (region 4). All the critical lines are determined by g_h and α_r, and α_z has no explicit role; however, for large α_z there is a large region of the parameter space on the chaotic side of the chaotic transition that can be made marginally stable [thatched region 5 in Fig. 7(a)].

A. Role of biases and static inputs

Biases have the effect of generating nontrivial fixed points and controlling stability by moving the edge of the spectral curve. Another key feature of biases is the suppression of the discontinuous bifurcation transition observed without biases. This is explained in detail in Appendix H. A particularly illuminating illustration of the effects of a bias can be inferred from the critical line (red dashed) for finite bias shown in Fig. 7. This curve, computed using the FP stability criterion (6) combined with the MFT equations [(C28)–(C30)], represents the transition between stability and chaos for finite bias with zero mean and nonzero variance. Equivalently, we may think of this as the critical line for a network with static input $I_i^h~𝒩\left(0,{\sigma }_h^2\right)$ (with I_r = I_z = 0). Along the g_h axis, we can observe the well-documented phenomena whereby an input suppresses chaos. This corresponds to the region g_h > 2 which lies to the left of the red dashed critical line, which is chaotic in the absence of input and flows to a stable fixed point in the presence of input. However, this behavior is reversed for g_h < 2. Here, we see a significant swath of phase space which is stable in the absence of input but which becomes chaotic when input is present. Thus, the stability-to-chaos phase boundary in the presence of biases (or inputs) reveals that the output (r) gate can facilitate an input-induced transition to chaos.

VIII. DISCUSSION

Gating is a form of multiplicative interaction that is a central feature of the best-performing RNNs in machine learning, and it is also a prominent feature of biological neurons. Prior theoretical work on RNNs has considered only RNNs with additive interactions. Here, we present the first detailed study on the consequences of gating for RNNs and show that gating can produce dramatically richer behavior that have significant functional benefits.

The continuous-time gated RNN (gRNN) we study resembles a popular model used in machine learning applications, the gated recurrent unit (GRU) [see the note below Eq. (C27)]. Previous work [51] looks at the instantaneous Jacobian spectrum for the discrete-time GRU using RMT methods similar to those presented in Appendix A; however, this work does not go beyond time-independent MFT and presents a phase diagram showing only boundaries across which the MFT fixed points become unstable [63]. In the present manuscript, we illuminate the full dynamical phase diagram for our gated RNN, uncovering much richer structure. Both the GRU and our gRNN have a gating function which dynamically scales the time constant, which in both cases leads to a marginally stable phase in the limit of a binary gate. However, the dynamical phase diagram presented in Fig. 7 reveals a novel discontinuous transition to chaos. We conjecture that such a phase transition should also be present in the GRU. Also, Ref. [51] lacks any discussion of the influence of inputs or biases. The present paper includes discussion of the functional significance of the gates in the presence of inputs. We believe these results, combined with the refined dynamical phase diagram, can shed some light on the role of analogous gates in the GRU and other gated ML architectures. We review the significance of the gates in more detail below.

A. Significance of the update gate

The update gate modulates the rate of integration. In single-neuron models, such a modulation is shown to make the neuron’s responses robust to time-warped inputs [14]. Furthermore, normative approaches, requiring time reparametrization invariance in ML RNNs, naturally imply the existence of a mechanism that modulates the integration rate [64]. We show that, for a wide range of parameters, a more sensitive (or switchlike) update gate leads to marginal stability. Marginally stable models of biological function have long been of interest with regard to their benefits for information processing (cf. Ref. [33] and references therein). In the gated RNN, a functional consequence of the marginally stable state is the use of the network as a robust integrator—such integratorlike function is shown to be beneficial for a variety of computational functions such as motor control [34–36], decision making [37], and auditory processing [53]. However, previous models of these integrators often require handcrafted symmetries and fine-tuning [38]. We show that gating allows this function without fine-tuning. Signatures of integratorlike behavior are also empirically observed in successfully trained gated ML RNNs on complex tasks [41]. We provide a theoretical basis for how gating produces these. The update gate facilitates accumulation of slow modes and a pinching of the spectral curve which leads to a suppression of unstable directions and overall slowing of the dynamics over a range of parameters. This is a manifestly self-organized slowing down. Other mechanisms for slowing down dynamics have been proposed where the slow timescales of the network dynamics are inherited from other slow internal processes such as synaptic filtering [65,66]; however, such mechanisms differ from the slowing due to gating; they do not seem to display the pinching and clumping, and they also do not rely on self-organized behavior.

B. Significance of the output gate

The output gate dynamically modulates the outputs of individual neurons. Similar shunting mechanisms are widely observed in real neurons and are crucial for performance in ML tasks [62]. We show that the output gate offers fine control over the dimensionality of the dynamics in phase space, and this ability is implicated in task performance in ML RNNs [42]. This gate also gives rise to a novel discontinuous chaotic transition where inputs can abruptly push stable systems to strongly chaotic activity; this is in contrast to the typically stabilizing role of inputs in additive RNNs. In this transition, there is a decoupling between topological and dynamical complexity. The chaotic state across this transition is also characterized by the coexistence of a stable fixed point with chaotic dynamics—in finite size systems, this manifests as long transients that scale with the system size. We note that there are other systems displaying either a discontinuous chaotic transition or the existence of fixed points overlapping with chaotic (pseudo)attractors [19] or apparent chaotic attractors with finite alignment with particular directions [67]; however, we are not aware of a transition in large RNNs where static inputs can induce strong chaos or the topological and dynamical complexity are decoupled across the transition. In this regard, the chaotic transition mediated by the output gated seems to be fundamentally different. More generally, the output gate is likely to have a significant role in controlling the influence of external inputs on the intrinsic dynamics.

We also show how the gates complement the memory function of the update gate by providing an important, context- and input-dependent reset mechanism. The ability to erase a memory trace flexibly is critical for function [62]. Gates also provide a mechanism to avoid the accuracy-flexibility trade-off noted for purely additive networks—where the stability of a memory comes at the cost of the ease with which it is rewritten [61].

We summarize the rich behavior of the gated RNN via phase diagrams indicating the critical surfaces and regions of marginal stability. From a practical perspective, the phase diagram is useful in light of the observation that it is often easier to train RNNs initialized in the chaotic regime but close to the critical points. This is often referred to as the “edge of chaos” hypothesis [68–70]. Thus, the phase diagrams provide ML practitioners with a map for principled parameter initialization—one of the most critical choices deciding training success.

APPENDIX A: DETAILS OF RANDOM MATRIX THEORY FOR SPECTRUM OF THE JACOBIAN

In this section, we provide details of the calculation of the bounding curve for the Jacobian spectrum for both fixed points and time-varying states. Our approach to the problem utilizes the method of Hermitian reduction [25,26] to deal with non-Hermitian random matrices.The analysis here resembles that in Ref. [51], which also considers Jacobians that are highly structured random matrices arising from discrete-time gated RNNs.

The Jacobian 𝒟 is a block-structured matrix constructed from the random coupling matrices J^h,z,r and diagonal matrices of the state variables. In the limit of large N, we expect the spectrum to be self-averaging—i.e., the distribution of eigenvalues for a random instance of the network approaches the ensemble-averaged distribution. We can, thus, gain insight about typical dynamical behavior by studying the ensemble- (or disorder-) averaged spectrum of the Jacobian. Our starting point is the disorder-averaged spectral density μ(λ) defined as

$$\mu (\lambda )=\frac{1}{3N}\mathbb{E}\left[\sum _{i=1}^{3N}\delta \left(\lambda -{\lambda }_i\right)\right],$$ (A1)

where the λ_i are the eigenvalues of 𝒟 for a given realization of J^h,z,r and the expectation is taken over the distribution of real Ginibre random matrices from which J^h,z,r are drawn. Using an alternate representation for the Dirac delta function in the complex plane $\left[\delta (\lambda )={\pi }^{-1}{\partial }_{\overline{\lambda }}{\lambda }^{-1}\right]$, we can write the average spectral density as

$$\mu (\lambda )=\frac{1}{\pi }\frac{\partial }{\partial \overline{\lambda }}\mathbb{E}\left[\frac{1}{3N}\text{Tr}\left[{\left(\lambda {\mathbb{1}}_{3N}-𝒟\right)}^{-1}\right]\right],$$ (A2)

where ${\mathbb{1}}_{3N}$ is the 3N-dimensional identity matrix. 𝒟 is in general non-Hermitian, so the support of the spectrum is not limited to the real line, and the standard procedure of studying the Green’s function $G(\lambda ,\overline{\lambda })={(3N)}^{-1}\text{Tr}\mathbb{E}\left[{\left(\lambda {\mathbb{1}}_{3N}-𝒟\right)}^{-1}\right]$ by analytic continuation is not applicable, since it is nonholomorphic on the support. Instead, we use the method of Hermitization [25,26] to analyze the resolvent for an expanded 6N × 6N Hermitian matrix H:

$$𝒢(\eta ,\lambda ,\overline{\lambda })=\mathbb{E}\left[{\left(\eta {\mathbb{1}}_{6N}-H\right)}^{-1}\right],$$ (A3)

$$H=\left(\begin{array}{cc}0& \lambda -𝒟\\ \overline{\lambda }-{𝒟}^T& 0\end{array}\right),$$ (A4)

and the Green’s function for the original problem is obtained by considering the lower-left block of 𝒢:

$$G(\lambda ,\overline{\lambda })=\underset{\eta \to i{0}^{+}}{\text{lim}}\frac{1}{3N}\text{Tr}{𝒢}_{21}(\eta ,\lambda ,\overline{\lambda }).$$ (A5)

To make this problem tractable, we invoke an ansatz called the local chaos hypothesis [57,71], which posits that, for large random networks in steady state, the state variables are statistically independent of the random coupling matrices J^z,h,r (also see Ref. [72]). This implies that the Jacobian [Eq. (4)] has an explicit linear dependence only on J^h,z,r, and the state variables are governed by their steady-state distribution from the disorder-averaged DMFT (Appendix C). These assumptions make the random matrix problem tractable, and we can evaluate the Green’s function by using the self-consistent Born approximation, which is exact as N → ∞. We detail this procedure below.

The Jacobian itself can be decomposed into structured (A, L, R) and random parts (𝒥):

$$𝒟=\underset{A}{\underbrace{\left(\begin{array}{ccc}-\left[{\sigma }_z\right]& \mathbb{D}& 0\\ 0& -{\tau }_z^{-1}\mathbb{1}& 0\\ 0& 0& -{\tau }_r^{-1}\mathbb{1}\end{array}\right)}}+\underset{L}{\underbrace{\left(\begin{array}{ccc}\left[{\sigma }_z\right]& 0& 0\\ 0& {\tau }_z^{-1}\mathbb{1}& 0\\ 0& 0& {\tau }_r^{-1}\mathbb{1}\end{array}\right)}}\times \underset{𝒥}{\underbrace{\left(\begin{array}{ccc}{J}^h& 0& 0\\ 0& J^z& 0\\ 0& 0& J^r\end{array}\right)}}\underset{R}{\underbrace{\left(\begin{array}{ccc}\left[{\varphi }^{\prime }{\sigma }_r\right]& 0& \left[\varphi {\sigma }_r^{\prime }\right]\\ \left[{\varphi }^{\prime }\right]& 0& 0\\ \left[{\varphi }^{\prime }\right]& 0& 0\end{array}\right)}}.$$ (A6)

At this point, we must make a crucial assumption: The structured matrices A, L, and R are independent of the random matrices appearing 𝒥. This implies that the dynamics is self-averaging and that the state variables reach a steady-state distribution determined by the DMFT and insensitive to the particular quenched disorder 𝒥. This self-averaging assumption leads to theoretical predictions which are in very good agreement with simulations of large networks, as presented in Fig. 1.

This independence assumption renders 𝒟 a linear function of the random matrix 𝒥, whose entries are Gaussian random variables. The next steps are to develop an asymptotic series in the random components of H, compute the resulting moments, and perform a resummation of the series. This is conveniently accomplished by the self-consistent Born approximation (SCBA). The output of the SCBA is a self-consistently determined self-energy functional Σ[𝒢] which succinctly encapsulates the resummation of moments. With this, the Dyson equation for 𝒢 is given by

$${𝒢}^{-1}={𝒢}_0^{-1}-\Sigma [𝒢],$$ (A7)

where the matrices on the right are defined in terms of 3N × 3N blocks:

$${𝒢}_0^{-1}=\left(\begin{array}{cc}\eta \mathbb{1}& \lambda -A\\ \overline{\lambda }-A^T& \eta \mathbb{1}\end{array}\right),$$ (A8)

$$\Sigma [𝒢]=\left(\begin{array}{cc}LQ\left[R{𝒢}_{22}{R}^T\right]L& 0\\ 0& R^{T}Q\left[L^T{𝒢}_{11}L\right]R\end{array}\right),$$ (A9)

and Q is a superoperator which acts on its argument as follows:

$$Q[M]=\left(\begin{array}{ccc}\frac{1}{N}\text{Tr}{M}_{11}& 0& 0\\ 0& \frac{1}{N}\text{Tr}{M}_{22}& 0\\ 0& 0& \frac{1}{N}\text{Tr}{M}_{33}\end{array}\right).$$ (A10)

Here, we express the self-energy using the 3N × 3N subblocks of the Green’s function 𝒢:

$$𝒢=\left(\begin{array}{ll}{𝒢}_{11}\hfill & {𝒢}_{12}\hfill \\ {𝒢}_{21}\hfill & {𝒢}_{22}\hfill \end{array}\right).$$ (A11)

At this point, we have presented all of the necessary ingredients for computing the Green’s function and, thus, determining the spectral properties of the Jacobian. These are the Dyson equation (A7), along with the free Green’s function (A8) and the self-energy (A9). Most of what is left is complicated linear algebra. However, in the interest of completeness, we proceed to unpack these equations and give a detailed derivation of the main equation of interest, the bounding curve of the spectral density.

To proceed further, it is useful to define the following transformed Green’s functions, which can be written in terms of N × N subblocks:

$${\widetilde{𝒢}}_{11}\equiv L^T{𝒢}_{11}L=\left(\begin{array}{ccc}{\tilde{G}}_{11}& {\tilde{G}}_{12}& {\tilde{G}}_{13}\\ {\tilde{G}}_{21}& {\tilde{G}}_{22}& {\tilde{G}}_{23}\\ {\tilde{G}}_{31}& {\tilde{G}}_{32}& {\tilde{G}}_{33}\end{array}\right),$$ (A12)

$${\widetilde{𝒢}}_{22}\equiv R{𝒢}_{22}{R}^T=\left(\begin{array}{ccc}{\tilde{G}}_{44}& {\tilde{G}}_{45}& {\tilde{G}}_{46}\\ {\tilde{G}}_{54}& {\tilde{G}}_{55}& {\tilde{G}}_{56}\\ {\tilde{G}}_{64}& {\tilde{G}}_{65}& {\tilde{G}}_{66}\end{array}\right).$$ (A13)

Denote also the mean trace of these subblocks as

$${\tilde{g}}_{ij}=\frac{1}{N}\text{Tr}\left[{\tilde{G}}_{ij}\right].$$ (A14)

Then the self-energy matrix in Eq. (A9) is block diagonal, i.e., Σ[𝒢] = bdiag(Σ₁₁, Σ₂₂), with

$${\Sigma }_{11}=\left(\begin{array}{ccc}\left[{\sigma }_z^2\right]{\tilde{g}}_{44}& 0& 0\\ 0& {\tau }_z^{-2}{\tilde{g}}_{55}& 0\\ 0& 0& {\tau }_r^{-2}{\tilde{g}}_{66}\end{array}\right),$$ (A15)

$${\Sigma }_{22}=\left(\begin{array}{ccc}{\left[{\varphi }^{\prime }{\sigma }_r\right]}^2{\tilde{g}}_{11}+{\left[{\varphi }^{\prime }\right]}^2\left({\tilde{g}}_{22}+{\tilde{g}}_{33}\right)& 0& \left[{\varphi }^{\prime }{\sigma }_r\right]\left[\varphi {\sigma }_r^{\prime }\right]{\tilde{g}}_{11}\\ 0& 0& 0\\ \left[{\varphi }^{\prime }{\sigma }_r\right]\left[\varphi {\sigma }_r^{\prime }\right]{\tilde{g}}_{11}& 0& {\left[\varphi {\sigma }_r^{\prime }\right]}^2{\tilde{g}}_{11}\end{array}\right)\text{. }$$ (A16)

With the self-energy in this form, we are able to solve the Dyson equation for the full Green’s function 𝒢 by direct matrix inversion:

$$𝒢={\left(\begin{array}{cc}\eta -{\Sigma }_{11}& \lambda -A\\ \overline{\lambda }-A^T& \eta -{\Sigma }_{22}\end{array}\right)}^{-1},$$ (A17)

which can be carried out easily by symbolic manipulation software. The rhs of Eq. (A17) is a function of ${\tilde{g}}_{ii}$, whereas the lhs is a function of the Green’s function before the transformations (A12) and (A13). Thus, to get a set of equations we can solve, we apply these same transformations to both sides of Eq. (A17) after solving the Dyson equation. The final step is to take the limit η → 0, recovering the problem we originally wished to solve.

The result of these manipulations is a set of six equations for the mean traces of the transformed Green’s function defined in Eq. (A14). In order to write these down, we introduce some additional notation. The self-consistent equations are of the form

$${\tilde{g}}_{ii}=\langle \frac{{\Gamma }_i}{\Gamma }\rangle ,$$ (A18)

where we denote 〈M〉 ≡ N⁻¹TrM for shorthand and i runs from 1 to 6. Denote the state-variable-dependent diagonal matrices as

$$p=\left[{\varphi }^{\prime }\right],q=\left[\varphi {\sigma }_r^{\prime }\right],r=\left[{\varphi }^{\prime }{\sigma }_r\right],$$ (A19)

and, because they appear frequently in the resulting equations, define

$$X={\tilde{g}}_{11}{\left|\lambda {\tau }_r+1\right|}^2{r}^2+\left({\tilde{g}}_{22}+{\tilde{g}}_{33}\right)p^{2}Z,$$ (A20)

$$Y={\mathbb{D}}^2{\tilde{g}}_{55}+{\left|\lambda {\tau }_z+1\right|}^2\left[{\sigma }_z^2\right]{\tilde{g}}_{44},$$ (A21)

$$Z={\left|\lambda {\tau }_r+1\right|}^2-{\tilde{g}}_{11}{\tilde{g}}_{66}{q}^2.$$ (A22)

The denominator in Eq. (A18) is then given by

$$\Gamma ={\left|\lambda {\tau }_z+1\right|}^2{\left|\lambda +{\sigma }_z\right|}^{2}Z-XY,$$ (A23)

and the numerators Γ_i are given by

$${\Gamma }_1={\sigma }_z^2{\left|\lambda {\tau }_z+1\right|}^{2}X,$$ (A24)

$${\Gamma }_2={\mathbb{D}}^{2}X,$$ (A25)

$${\Gamma }_3={\tilde{g}}_{11}{\left|\lambda {\tau }_z+1\right|}^2{\left|\lambda +{\sigma }_z\right|}^2{q}^2-{\tilde{g}}_{11}\left({\tilde{g}}_{22}+{\tilde{g}}_{33}\right)p^2{q}^{2}Y,$$ (A26)

$${\Gamma }_4={\tilde{g}}_{66}{\left|\lambda {\tau }_z+1\right|}^2{\left|\lambda +{\sigma }_z\right|}^2{q}^2+\left[{\left|\lambda {\tau }_r+1\right|}^2{r}^2-{\tilde{g}}_{66}\left({\tilde{g}}_{22}+{\tilde{g}}_{66}\right)p^2{q}^2\right]Y,$$ (A27)

$${\Gamma }_5={\Gamma }_6=p^{2}YZ.$$ (A28)

The numerators and denominator are all diagonal matrices with real entries, which is why we use the simple notation of a ratio when referring to matrix inversion.

Solving these equations gives us the ${\tilde{g}}_{ii}$ as implicit functions of λ. They are, in general, complicated and resist exact solution. However, the situation simplifies considerably when we are looking for the spectral curve. In this case, we are looking for all $\lambda \in ℂ$ that satisfy the self-consistent equations with ${\tilde{g}}_{ii}\to 0$.

We must take this limit carefully, since the ratio of these functions can remain constant. For this reason, it is necessary to define

$$x_2={\tilde{g}}_{22}/{\tilde{g}}_{11},x_3={\tilde{g}}_{33}/{\tilde{g}}_{11}.$$ (A29)

We may do the same for ${\tilde{g}}_{44}$, ${\tilde{g}}_{55}$, and ${\tilde{g}}_{66}$, but it turns out that x₂ and x₃ are sufficient to compute the spectral curve. Next, divide by ${\tilde{g}}_{11}$ and send all ${\tilde{g}}_{ii}\to 0$, keeping the ratios fixed. Applying this to the equation for ${\tilde{g}}_{11}$ results in

$$1=\underset{{\tilde{g}}_{ii}\to 0}{\text{lim}}\frac{1}{{\tilde{g}}_{11}}\langle \frac{{\Gamma }_1}{\Gamma }\rangle ={\gamma }_1+{\gamma }_2\left(x_2+x_3\right).$$ (A30)

Similarly, for ${\tilde{g}}_{22}$ and ${\tilde{g}}_{33}$, we get

$$x_2={\gamma }_3+{\gamma }_4\left(x_2+x_3\right),$$ (A31)

$$x_3={\gamma }_5,$$ (A32)

where the coefficients γ_i, which are functions of λ, are given by

$${\gamma }_1=\langle \frac{{\sigma }_z^2{r}^2}{{\left|\lambda +{\sigma }_z\right|}^2}\rangle ,{\gamma }_2=\langle \frac{p^2{\sigma }_z^2}{{\left|\lambda +{\sigma }_z\right|}^2}\rangle ,{\gamma }_5=\frac{q^2}{{\left|\lambda {\tau }_r+1\right|}^2},$$

$${\gamma }_3=\langle \frac{{\mathbb{D}}^2{r}^2}{{\left|\lambda {\tau }_z+1\right|}^2{\left|\lambda +{\sigma }_z\right|}^2}\rangle ,{\gamma }_4=\langle \frac{{\mathbb{D}}^2{p}^2}{{\left|\lambda {\tau }_z+1\right|}^2{\left|\lambda +{\sigma }_z\right|}^2}\rangle .$$

The linear system of equations (A30)–(A32) is consistent iff

$$\left(1-{\gamma }_1\right)\left(1-{\gamma }_4\right)={\gamma }_2\left({\gamma }_3+{\gamma }_5\right).$$ (A33)

In other words, γ_i must satisfy Eq. (A33) when ${\tilde{g}}_{ii}\to 0$. This expression depends on λ and implicitly defines a curve in $ℂ$, which is the boundary of the support of the spectral density.

Plugging in the explicit expression for γ_i, we get the implicit equation for the spectral curve as all $\lambda \in ℂ$ that satisfy

$$\left\{1-\langle \frac{r^2{\sigma }_z^2}{{\left|\lambda +{\sigma }_z\right|}^2}\rangle \right\}\left\{1-\langle \frac{{\mathbb{D}}^2{p}^2}{{\left|\lambda {\tau }_z+1\right|}^2{\left|\lambda +{\sigma }_z\right|}^2}\rangle \right\}=\langle \frac{{\sigma }_z^2{p}^2}{{\left|\lambda +{\sigma }_z\right|}^2}\rangle \left\{\langle \frac{{\mathbb{D}}^2{r}^2}{{\left|\lambda {\tau }_z+1\right|}^2{\left|\lambda +{\sigma }_z\right|}^2}\rangle +\frac{\langle q^2\rangle }{{\left|\lambda {\tau }_r+1\right|}^2}\right\}.$$ (A34)

For large systems, we can replace the empirical traces of the state variable by their averages given by the DMFT variances. Then, the equation for the curve for a general steady state is given by

$$\left(\langle {\sigma }_r^2\rangle +\frac{\langle {\varphi }^2{\sigma }_r^{\prime 2}\rangle }{{\left|1+{\tau }_r\lambda \right|}^2}\right)\langle \frac{{\varphi }^{\prime 2}{\sigma }_z^2}{{\left|\lambda +{\sigma }_z\right|}^2}\rangle +\frac{1}{{\left|1+{\tau }_z\lambda \right|}^2}\langle \frac{{\mathbb{D}}^2{\varphi }^{\prime 2}}{{\left|\lambda +{\sigma }_z\right|}^2}\rangle =1.$$ (A35)

FIG. 8.

Jacobian spectrum at a time-varying state. Red dots are the Jacobian eigenvalues for the full network in a (time-varying) steady state, and the spectral curve of the Jacobian is calculated using moments from (i) the full state vectors (blue curve) or using the variances from the fixed-point MFT (green). Surprisingly, the agreement is reasonably good. For network simulations, N = 1000, g_h = 2.5, α_r = 1, α_z = 15, and all biases are zero.

For fixed points, we have $\mathbb{D}=0$, which makes γ₃ = γ₄ = 0. The equation for the spectral curve simplifies to that which is quoted in the main text [Eq. (5)]:

$$1=\langle \frac{r^2{\sigma }_z^2}{{\left|\lambda +{\sigma }_z\right|}^2}\rangle +\frac{\langle q^2\rangle }{{\left|\lambda {\tau }_r+1\right|}^2}\langle \frac{{\sigma }_z^2{p}^2}{{\left|\lambda +{\sigma }_z\right|}^2}\rangle .$$ (A36)

1. Jacobian spectrum for the case α_r = 0

In the case when α_r = 0, it is possible to express the Green’s function [Eq. (A5)] in a simpler form. Recall that

$$G(\lambda ,\overline{\lambda })=\underset{\eta \to i{0}^{+}}{\text{lim}}\frac{1}{3N}\text{tr}{𝒢}_{21}(\eta ,\lambda ,\overline{\lambda }).$$ (A37)

Let $\tilde{Y}={\mathbb{D}}^2+{\sigma }_r^2{\sigma }_z^2{\left|\lambda {\tau }_z+1\right|}^2$. Then, the Green’s function is given by

$$G(\lambda ,\overline{\lambda })=\frac{1}{3}\langle \frac{{\left|\lambda {\tau }_z+1\right|}^2\left(\overline{\lambda }+{\sigma }_z\right)}{{\left|\lambda {\tau }_z+1\right|}^2{\left|\lambda +{\sigma }_z\right|}^2-\xi (\lambda ,\overline{\lambda })p^2\tilde{Y}}\rangle$$ (A38)

$$+\frac{1}{3}\langle \frac{\left(\overline{\lambda }+{\tau }_z^{-1}\right)\left({\left|\lambda +{\sigma }_z\right|}^2-\xi (\lambda ,\overline{\lambda })p^2{\sigma }_z^2\right)}{{\left|\lambda +{\tau }_z^{-1}\right|}^2{\left|\lambda +{\sigma }_z\right|}^2-\xi (\lambda ,\overline{\lambda })p^2\tilde{Y}}\rangle$$ (A39)

$$+\frac{1}{3}\frac{1}{\lambda +{\tau }_r^{-1}},$$ (A40)

where $\xi (\lambda ,\overline{\lambda })$ is defined implicitly to satisfy the equation

$$1=\langle \frac{p^2\tilde{Y}}{{\left|\lambda {\tau }_z+1\right|}^2{\left|\lambda +{\sigma }_z\right|}^2-\xi (\lambda ,\overline{\lambda })p^2\tilde{Y}}\rangle .$$ (A41)

The function $\xi (\lambda ,\overline{\lambda })$ acts as a sort of order parameter for the spectral density, indicating the transition on the complex plane between zero and finite density μ. Outside the spectral support, λ ∈ Σ^c, this order parameter vanishes, ξ = 0, and the Green’s function is holomorphic:

$$G(\lambda ,\overline{\lambda })=\frac{1}{3}\left(\langle \frac{1}{\lambda +{\sigma }_z}\rangle +\frac{1}{\lambda +{\tau }_z^{-1}}+\frac{1}{\lambda +{\tau }_r^{-1}}\right),$$ (A42)

which, of course, indicates that the density is zero since $\mu (\lambda )={\partial }_{\overline{\lambda }}G(\lambda ,\overline{\lambda })$. Inside the support λ ∈ Σ, the order parameter ξ ≠ 0, and the Green’s function consequently picks up nonanalytic contributions, proportional to $\overline{\lambda }$. Since the Green’s function is continuous on the complex plane, it must be continuous across the boundary of the spectral support. This must then occur precisely when the holomorphic solution meets the nonanalytic solution, at ξ = 0. This is the condition used to find the boundary curve above.

APPENDIX B: SPECTRAL CLUMPING AND PINCHING IN THE LIMIT α_z → ∞

In this section, we provide details on the accumulation of eigenvalues near zero and the pinching of the leading spectral curve (for certain values of g_h) as the update gate becomes switchlike (α_z → ∞). To focus on the key aspects of these phenomena, we consider the case when the reset gate is off and there are no biases (α_r = 0 and β_r,h,z = 0). Moreover, we consider a piecewise linear approximation—sometimes called “hard” tanh—to the tanh function given by

$${\varphi }_{\text{lin}}(x)=\{\begin{array}{ll}1\hfill & x>1/g_h,\hfill \\ g_{h}x\hfill & |x|\le 1/g_h,\hfill \\ -1\hfill & x<-1/g_h.\hfill \end{array}$$ (B1)

This approximation does not qualitatively change the nature of the clumping.

In the limit α_z → ∞, the update gate σ_z becomes binary with a distribution given by

$$P\left({\sigma }_z=x\right)=f_z\delta (x-1)+\left(1-f_z\right)\delta (x),$$ (B2)

where f_z = 〈σ_z〉 is the fraction of update gates that are open (i.e., equal to one). Using this, along with the assumption that $\mathbb{D}\approx 0$—which is valid in this regime—we can simplify the expression for the Green’s function [Eqs. (A38)–(A42)] to yield

$$G(\lambda ,\overline{\lambda })=\frac{1-f_z}{\lambda }+f_z\left(1-f_h\right)\frac{1}{\lambda +1}+\frac{1}{\lambda +{\tau }_z^{-1}}+\frac{(1+\overline{\lambda })}{g_h^2\sigma {\left({\beta }_r\right)}^2}{\mathbb{I}}_{\left\{|\lambda |<g_h^2\sigma {\left({\beta }_r\right)}^2\right\}},$$ (B3)

where f_h is the fraction of hard tanh activations that are not saturated. In the limit of small τ_z and β_r = 0, we get the expression for the density given in the text:

$$\mu (\lambda )=\left(1-f_z\right)\delta (\lambda )+f_z\left(1-f_h\right)\delta (\lambda +1)+\frac{4}{\pi g_h^2}{\mathbb{I}}_{\left\{|\lambda |\le g_h^2/4\right\}}.$$ (B4)

Thus, we see an extensive number of eigenvalues at zero.

Now, let us study the regime where α_z is large but not infinite. We would like to get the scaling behavior of the leading edge of the spectrum and the density of eigenvalues contained in a radius δ around the origin. We make an ansatz for the spectral edge close to zero $\lambda ~e^{-c{\alpha }_z\sqrt{{\Delta }_h}}$, where c is a positive constant. With this ansatz, the equation for the spectral curve reads

$$\int 𝒟z\frac{{\sigma }_z{\left(\sqrt{{\Delta }_z}\cdot z\right)}^2}{{\left|{\lambda }_0{e}^{-c{\alpha }_z\sqrt{{\Delta }_h}}+{\sigma }_z\left(\sqrt{{\Delta }_z^2}\right)\right|}^2}=\frac{{\sigma }_r{\left({\beta }_r\right)}^{-2}}{\langle {\varphi }^{\prime }{\left(\sqrt{{\Delta }_h}\cdot h\right)}^2\rangle }.$$ (B5)

In the limit of large α_z and β_r = 0, this implies

$$\text{erfc}\left(\frac{c}{\sqrt{2}}\right)\approx \frac{4}{\langle {\varphi }^{\prime }{\left(\sqrt{{\Delta }_h}\cdot h\right)}^2\rangle }.$$ (B6)

If this has a positive solution for c, then the scaling of the spectral edge as $\lambda ~e^{-c{\alpha }_z\sqrt{{\Delta }_h}}$ holds. Moreover, whenever there is a positive solution for c, we also expect pinching of the spectral curve, and in the limit α_z → ∞ we have marginal stability.

Under the same approximation, we can approximate the eigenvalue density in a radius δ around zero as

$$P[|\lambda (𝒟)|<\delta ]=\frac{1}{2\pi i}{\oint }_{𝒞}dzG(z),$$ (B7)

where we choose the contour along $z=e^{-c{\alpha }_z\sqrt{{\Delta }_h}+i\theta }$ for θ ∈ [0, 2π) and $\delta =e^{-c{\alpha }_z\sqrt{{\Delta }_h}}$. In the limit of large α_z (thus, δ ≪ 1), we get the scaling form described in the main text:

$$P[|\lambda (𝒟)|<\delta ]\approx \frac{1}{2}\text{erfc}\left(-\frac{\text{log}(\delta )}{{\alpha }_z\sqrt{2{\Delta }_h}}\right).$$ (B8)

APPENDIX C: DETAILS OF THE DYNAMICAL MEAN-FIELD THEORY

The DMFT is a powerful analytical framework used to study the dynamics of disordered systems, and it traces its origins to the study of dynamical aspects of spin glasses [73,74] and has been later applied to the study of random neural networks [9,15,21,75]. In our case, the DMFT reduces the description of the full 3N-dimensional (deterministic) ordinary differential equations (ODEs) describing (h, z, r) to a set of three coupled stochastic differential equations for scalar variables (h, z, r).

Here, we provide a detailed, self-contained description of the dynamical mean-field theory for the gated RNN using the Martin–Siggia–Rose–De Dominicis–Janssen formalism. The starting point is a generating functional—akin to the generating function of a random variable—which takes an expectation over the paths generated by the dynamics. The generating functional is defined as

$$Z_{𝒥}[\widehat{b},b]=\mathbb{E}\left[\text{exp}\left(i\sum _{j=1}^N\int {\widehat{b}}_j{(t)}^T{x}_j(t)dt\right)\right],$$ (C1)

where x_j(t) ≡ [h_j(t), z_j(t), r_j(t)] is the trajectory and ${\widehat{b}}_j(t)=\left({\widehat{b}}_j^h,{\widehat{b}}_j^z,{\widehat{b}}_j^r\right)$ is the argument of the generating functional. We also include external fields $b_j=\left(b_j^h,b_j^z,b_j^r\right)$, which are used to calculate the response functions. The measure in the expectation is a path integral over the dynamics. The generating functional is then used to calculate correlation and response functions using the appropriate (variational) derivatives. For instance, the two-point function for the h field is given by

$$\langle h_i(t)h_i\left(t^{\prime }\right)\rangle ={\frac{{\delta }^2}{\delta {\widehat{b}}_i^h\left(t^{\prime }\right)\delta {\widehat{b}}_i^h(t)}{Z}_{𝒥}[\widehat{b},b]|}_{b=0}.$$ (C2)

Up until this point, things are quite general and do not rely on the specific form of the dynamics. However, for large random networks, we expect certain quantities such as the population averaged correlation function C_h ≡ N⁻¹ Σ_i〈h_i(t)h_i(t′)〉 to be self-averaging and, thus, not vary much across realizations. Thus, we can study the disorder averaged (over 𝒥), the generating functional $\overline{Z}={\langle Z_{𝒥}\rangle }_{𝒥}$, and approximate $\overline{Z}$ with its value evaluated at the saddle point of the action. This approximation gives us the single-site DMFT picture of dynamics described in Eqs. (C19) and (C20).

To see how this all works, we start with the equations of motion (in vector form)

$${\tau }_z\dot{z}=-z+J^z{\varphi }_z(h),$$ (C3)

$${\tau }_r\dot{r}=-r+J^r{\varphi }_r(h),$$ (C4)

$$\dot{h}={\sigma }_z(z)\odot \left(-h+\left\{J^h\left[{\sigma }_r(r)\odot {\varphi }_h(h)\right]\right\}\right),$$ (C5)

where ⊙ stands for elementwise multiplication.

To write down the MSRDJ generating functional, let us discretize the dynamics (in the Itô convention). Note that in this convention the Jacobian is unity.

$$h_i(t+1)-h_i(t)={\sigma }_{z,i}(t)\{-h_i(t)+\sum _j{J}_{ij}^h{\sigma }_{r,j}(t){\varphi }_j(t)+b_i^h(t)\}\delta t,$$

$${\tau }_z\left[z_i(t+1)-z_i(t)\right]=\left\{-z_i(t)+\sum _i{J}_{ij}^z\varphi (t)+b_i^z(t)\right\}\delta t,$$

$${\tau }_r\left[r_i(t+1)-r_i(t)\right]=\left\{-r_i(t)+\sum _j{J}_{ij}^r\varphi (t)+b_i^r(t)\right\}\delta t,$$

where we introduce external fields in the dynamics $\left\{b_i^h(t)\right\}$, $\left\{b_i^z(t)\right\}$, and $\left\{b_i^r(t)\right\}$. The generating functional is given by

$$Z_{𝒥}[\widehat{b},b]=\mathbb{E}\left[\text{exp}\left(i\sum _{j=1}^N\sum _t{\widehat{b}}_j{(t)}^T{x}_j(t)\delta t\right)\right],$$ (C6)

where $\widehat{b}=\left({\widehat{b}}_j^h,{\widehat{b}}_j^z,{\widehat{b}}_j^r\right)$, $b=\left(b_j^h,b_j^z,b_j^r\right)$, and x_j(t) ≡ [h_j(t), z_j(t), r_j(t)]; also, the expectation is over the dynamics generated by the network. Writing this out explicitly, with δ functions enforcing the dynamics, we get the following integral for the generating functional:

$$Z_{𝒥}[\widehat{b},b]=\int \prod _{i,t}\prod _{k,t^{\prime }}\prod _{m,t^{″}}d{h}_i(t)d{z}_k\left(t^{\prime }\right)d{r}_m\left(t^{″}\right)\cdot \text{exp}\left(i\left\{\sum _{i,t}{\widehat{b}}_i^h(t)h_i(t)+{\widehat{b}}_i^z(t)z_i(t)+{\widehat{b}}_i^r(t)r_i(t)\right\}\delta t\right)\times \delta \left(h_i(t+1)-h_i(t)+\left\{h_i(t){\sigma }_{z,i}(t)-{\sigma }_{z,i}(t)\left[\sum _j{J}_{ij}^h{\sigma }_{r,j}(t){\varphi }_j(t)\right]-b_i^h(t)\right\}\delta t\right)\times \delta \left(z_k\left(t^{\prime }+1\right)-z_k\left(t^{\prime }\right)+\frac{1}{{\tau }_z}\left\{z_k\left(t^{\prime }\right)+\sum _l{J}_{kl}^z{\varphi }_l\left(t^{\prime }\right)+b_k^z\left(t^{\prime }\right)\right\}\delta t\right)\times \delta \left(r_m\left(t^{″}+1\right)-r_m\left(t^{″}\right)+\frac{1}{{\tau }_r}\left\{r_m\left(t^{″}\right)+\sum _n{J}_{mn}^r{\varphi }_n\left(t^{″}\right)+b_m^r\left(t^{\prime }\right)\right\}\delta t\right).$$ (C7)

Now, let us introduce the Fourier representation for the δ function; this introduces an auxiliary field variable, which as we see allows us to calculate the response function in the MSRDJ formalism. The generating functional can then be expressed as

$$Z_{𝒥}[\widehat{b},b]=\int \prod _{i,t}\prod _{k,t^{\prime }}\prod _{m,t^{″}}d{h}_i(t)\frac{d{\widehat{h}}_i(t)}{2\pi }d{z}_k\left(t^{\prime }\right)\frac{d{\widehat{z}}_k\left(t^{\prime }\right)}{2\pi }d{r}_m\left(t^{″}\right)\frac{d{\widehat{r}}_m\left(t^{″}\right)}{2\pi }\times \text{\hspace{0.17em}exp}\left[-i\sum _{i,t}{\widehat{h}}_i(t)\left[h_i(t+1)-h_i(t)-f_h\left(h_i,z_i,r_i\right)\delta t-b_i^h(t)\delta t\right]+i\sum _{i,t}{\widehat{b}}_i^h(t)h_i(t)\delta t\right]\times \text{exp}\left[-i\sum _{k,t^{\prime }}{\widehat{z}}_k(t)\left(z_k\left(t^{\prime }+1\right)-z_k\left(t^{\prime }\right)-f_z\left(h_k,z_k\right)\frac{\delta t}{{\tau }_z}-b_k^z\left(t^{\prime }\right)\frac{\delta t}{{\tau }_z}\right)+i\sum _{k,t^{\prime }}{\widehat{b}}_k^z\left(t^{\prime }\right)z_k\left(t^{\prime }\right)\delta t\right]\times \text{\hspace{0.17em}exp}\left[-i\sum _{m,t^{″}}{\widehat{r}}_m\left(t^{″}\right)\left(r_m\left(t^{″}+1\right)-r_m\left(t^{″}\right)-f_r\left(h_m,r_m\right)\frac{\delta t}{{\tau }_r}-b_m^r\left(t^{″}\right)\frac{\delta t}{{\tau }_r}\right)+i\sum _{m,t^{″}}{\widehat{b}}_m^r\left(t^{″}\right)r_m\left(t^{″}\right)\delta t\right],$$ (C8)

where the functions f_h,z,r summarize the gated RNN dynamics

$$f_h\left(h_i,z_i,r_i\right)={\sigma }_{z,i}(t)\left(-h_i(t)+\sum _j{J}_{ij}^h{\sigma }_{r,j}(t){\varphi }_j(t)\right),$$

$$f_z\left(h_k,z_k\right)=-z_k\left(t^{\prime }\right)+\sum _l{J}_{kl}^z{\varphi }_l\left(t^{\prime }\right),$$

$$f_r\left(h_m,r_m\right)=-r_m\left(t^{″}\right)+\sum J_{mn}^r{\varphi }_n\left(t^{″}\right).$$

Let us now take the continuum limit δt → 0 and formally define the measures 𝒟h_i = lim_δt→0 ∏_t dh_i(t). We can then write the generating functional as a path integral:

$$Z_{𝒥}[\widehat{b},b]=\int \prod _i𝒟h_i𝒟{\widehat{h}}_i𝒟z_i𝒟{\widehat{z}}_i𝒟r_i𝒟{\widehat{r}}_i\text{\hspace{0.17em}exp}\{-S[\widehat{x},x]+i\int dt\left[\widehat{b}{(t)}^{T}x(t)+b{(t)}^T\widehat{x}(t)\right]\},$$ (C9)

where $\widehat{b}=\left({\widehat{b}}_i^h,{\widehat{b}}_i^z,{\widehat{b}}_j^r\right)$, x = (h_i, z_i, r_i), $\widehat{x}=\left({\widehat{h}}_i,{\widehat{z}}_i/{\tau }_z,{\widehat{r}}_i/{\tau }_r\right)$, and the action S which gives weights to the paths is given by

$$S[\widehat{x},x]=i\sum _i\int dt{\widehat{h}}_i(t)\left[{\partial }_t{h}_i(t)-f_h\left(h_i,z_i,r_i\right)\right]+i\sum _k\int dt{\widehat{z}}_k(t)\left[{\partial }_t{z}_k(t)-\frac{f_z\left(h_k,z_k\right)}{{\tau }_z}\right]+i\sum _k\int dt{\widehat{r}}_m(t)\left[{\partial }_t{r}_m(t)-\frac{f_r\left(h_m,r_m\right)}{{\tau }_r}\right].$$ (C10)

The functional is properly normalized, so Z_𝒥[0, b] = l. We can calculate correlation functions and response functions by taking appropriate variational derivatives of the generating functional Z, but first we address the role of the random couplings.

1. Disorder averaging

We are interested in the typical behavior of ensembles of the networks, so we work with the disorder-averaged generating functional $\overline{Z}$; Z_𝒥 is properly normalized, so we are allowed to do this averaging on Z_𝒥. Averaging over $J_{ij}^h$ involves the following integral:

$$\int d{J}_{ij}^h\sqrt{\frac{N}{2\pi }}\text{exp}\left\{-\frac{N{\left(J_{ij}^h\right)}^2}{2}+i\cdot J_{ij}^h\int dt{\widehat{h}}_i(t){\sigma }_{z,i}(t){\varphi }_j(t){\sigma }_{r,j}(t)\right\},$$

which evaluates to

$$\text{exp}\left\{-(1/2N)\cdot {\left[\int dt{\widehat{h}}_i(t){\sigma }_{z,i}(t){\varphi }_j(t){\sigma }_{r,j}(t)\right]}^2\right\},$$

and similarly for J^z and J^r we get terms

$$\text{exp}\left\{-(1/2N)\cdot {\left(\int dt{\widehat{z}}_k(t){\varphi }_l(t)\right)}^2{\tau }_z^{-2}\right\},$$

$$\text{exp}\left\{-(1/2N)\cdot {\left(\int dt{\widehat{r}}_m(t){\varphi }_n(t)\right)}^2{\tau }_r^{-2}\right\}.$$

The disorder-averaged generating functional is then given by

$$\overline{Z}[\widehat{b},b]=\int \prod _i𝒟h_i𝒟{\widehat{h}}_i𝒟z_i𝒟{\widehat{z}}_i𝒟r_i𝒟{\widehat{r}}_i\text{\hspace{0.17em}exp}\left\{-\overline{S}[\widehat{x},x]+i\int dt\left[\widehat{b}{(t)}^{T}x(t)+b{(t)}^T\widehat{x}(t)\right]\right\},$$ (C11)

where the disorder-averaged action $\overline{S}$ is given by

$$\overline{S}[\widehat{x},x]=i\sum _i\int dt{\widehat{h}}_i(t)\left[{\partial }_t{h}_i(t)+h_i(t){\sigma }_{z,i}(t)\right]+\frac{1}{2N}\sum _{i,j}{\left(\int dt{\widehat{h}}_i(t){\sigma }_{z,i}(t){\varphi }_j(t){\sigma }_{r,j}(t)\right)}^2+i\sum _k\int dt{\widehat{z}}_k(t)\left({\partial }_t{z}_k(t)+\frac{z_k(t)}{{\tau }_z}\right)+\frac{1}{2N}\sum _{k,l}{\left(\int dt\frac{{\widehat{z}}_k(t)}{{\tau }_z}\cdot {\varphi }_l(t)\right)}^2+i\sum _m\int dt{\widehat{r}}_m(t)\left({\partial }_t{r}_m(t)+\frac{r_m(t)}{{\tau }_r}\right)+\frac{1}{2N}\sum _{m,n}{\left(\int dt\frac{{\widehat{r}}_m(t)}{{\tau }_r}\cdot {\varphi }_n(t)\right)}^2.$$ (C12)

With some foresight, we see the action is extensive in the system size, and we can try to reduce it to a single-site description. However, the issue now is that we have nonlocal terms (e.g., involving both i and j), and we can introduce the following auxiliary fields to decouple these nonlocal terms:

$$C_{\varphi {\sigma }_r}\left(t,t^{\prime }\right)≔\frac{1}{N}\sum _i{\varphi }_i(t){\varphi }_i\left(t^{\prime }\right){\sigma }_{r,i}(t){\sigma }_{r,i}\left(t^{\prime }\right),$$

$$C_{\varphi }\left(t,t^{\prime }\right)≔\frac{1}{N}\sum _k{\varphi }_k(t){\varphi }_k\left(t^{\prime }\right).$$ (C13)

To make the C’s free fields that we integrate over, we enforce these relations using the Fourier representation of δ functions with additional auxiliary fields:

$$\delta \left(N{C}_{\varphi {\sigma }_r}\left(t,t^{\prime }\right)-\sum _i{\varphi }_i(t){\varphi }_i\left(t^{\prime }\right){\sigma }_{r,i}(t){\sigma }_{r,i}\left(t^{\prime }\right)\right)=\int \frac{N}{\pi }d{\widehat{C}}_{\varphi {\sigma }_r}\left(t,t^{\prime }\right)\exp \left[-\frac{i}{2}{\widehat{C}}_{\varphi {\sigma }_r}\left(t,t^{\prime }\right)\left(N\cdot C_{\varphi {\sigma }_r}\left(t,t^{\prime }\right)-\sum _i{\varphi }_i(t){\varphi }_i\left(t^{\prime }\right){\sigma }_{r,i}(t){\sigma }_{r,i}\left(t^{\prime }\right)\right)\right],$$

$$\delta \left(N{C}_{\varphi }\left(t,t^{\prime }\right)-\sum _k{\varphi }_k(t){\varphi }_k\left(t^{\prime }\right)\right)=\int \frac{N}{\pi }d{\widehat{C}}_{\varphi }\left(t,t^{\prime }\right)\exp \left[-\frac{i}{2}{\widehat{C}}_{\varphi }\left(t,t^{\prime }\right)\left(N\cdot C_{\varphi }\left(t,t^{\prime }\right)-\sum _k{\varphi }_k(t){\varphi }_k\left(t^{\prime }\right)\right)\right].$$

This allows us to make the following transformations to decouple the nonlocal terms in the action $\overline{S}$:

$$\frac{1}{2N}\sum _{i,j}{\left[{\widehat{h}}_i(t){\sigma }_{z,i}(t){\varphi }_j(t){\sigma }_{r,j}(t)\right]}^2\to \frac{1}{2}\sum _i\int dtd{t}^{\prime }{\widehat{h}}_i(t){\sigma }_{z,i}(t)C_{\varphi {\sigma }_r}\left(t,t^{\prime }\right){\widehat{h}}_i\left(t^{\prime }\right){\sigma }_{z,i}\left(t^{\prime }\right),$$

$$\frac{1}{2N}\sum _{k,l}{\left(\int dt\frac{{\widehat{z}}_k(t)}{{\tau }_z}\cdot {\varphi }_l(t)\right)}^2\to \frac{1}{2}\sum _k\int dtd{t}^{\prime }\frac{{\widehat{z}}_k(t)}{{\tau }_z}{C}_{\varphi }\left(t,t^{\prime }\right)\frac{{\widehat{z}}_k\left(t^{\prime }\right)}{{\tau }_z},$$

$$\frac{1}{2N}\sum _{m,n}{\left(\int dt\frac{{\widehat{r}}_m(t)}{{\tau }_r}\cdot {\varphi }_n(t)\right)}^2\to \frac{1}{2}\sum _m\int dtd{t}^{\prime }\frac{{\widehat{r}}_m(t)}{{\tau }_r}{C}_{\varphi }\left(t,t^{\prime }\right)\frac{{\widehat{r}}_m\left(t^{\prime }\right)}{{\tau }_r}.$$

We see clearly that the $C_{\varphi {\sigma }_r}$ and C_ϕ auxiliary fields which represent the (population-averaged) ϕσ_r − ϕσ_r and ϕ − ϕ correlation functions decouple the sites by summarizing all the information present in the rest of the network in terms of two-point functions; two different sites interact only by means of the correlation functions. The disorder-averaged generating functional can now be written as

$$\begin{gathered} \overline{Z}[\widehat{b},b]=\int 𝒟\widehat{C}𝒟C\text{\hspace{0.17em}exp}(-N\cdot ℒ[\widehat{C},C;\widehat{b},b]), \\ ℒ=\frac{i}{2}\int dtd{t}^{\prime }\left[C{\left(t,t^{\prime }\right)}^T\widehat{C}\left(t,t^{\prime }\right)\right]-W[\widehat{C},C;\widehat{b},b], \\ \text{exp}(N\cdot W)=\int \prod _i𝒟h_i𝒟{\widehat{h}}_i𝒟z_i𝒟{\widehat{z}}_i𝒟r_i𝒟{\widehat{r}}_i\times \text{exp}\left\{i\int dt\left[b{(t)}^T\widehat{h}(t)+\widehat{b}{(t)}^{T}h(t)\right]-S_d[\widehat{h},h;\{C,\widehat{C}\}]\right\}, \end{gathered}$$ (C14)

where C = (C_h, C_z, C_r) and Ĉ = (Ĉ_h, Ĉ_z, Ĉ_r). The sitewise decoupled action S_d contains only terms involving a single site (and the C fields). So, for a given value of Ĉ and C, the different sites are decoupled and driven by the sitewise action

$$S_d[\widehat{h},h;\{C,\widehat{C}\}]=i\int dt\left[\widehat{h}{(t)}^T{\partial }_{t}h(t)+{\widehat{h}}_{\tau }{(t)}^{T}h(t)\right]+\frac{1}{2}\int dtd{t}^{\prime }{\widehat{h}}_{\tau }{(t)}^T\mathbb{D}C\left(t,t^{\prime }\right){\widehat{h}}_{\tau }\left(t^{\prime }\right)-\frac{i}{2}\int dtd{t}^{\prime }{S}_x{(t)}^T\mathbb{D}\widehat{C}\left(t,t^{\prime }\right)S_x\left(t^{\prime }\right),$$ (C15)

where

$${\widehat{h}}_{\tau }(t)=\left({\widehat{h}}_i{\sigma }_{z,i},{\widehat{z}}_i/{\tau }_z,{\widehat{r}}_i/{\tau }_r\right),$$

$$\widehat{h}(t)=\left({\widehat{h}}_i,{\widehat{z}}_i,{\widehat{r}}_i\right),$$

$$S_x=\left({\varphi }_i{\sigma }_{r,i},{\varphi }_i,{\varphi }_i\right),$$

$$\mathbb{D}C\left(t,t^{\prime }\right)=\text{Diag}\left[C_{\varphi {\sigma }_r}\left(t,t^{\prime }\right),C_{\varphi }\left(t,t^{\prime }\right),C_{\varphi }\left(t,t^{\prime }\right)\right],$$

$$ℂ\widehat{C}\left(t,t^{\prime }\right)=\text{Diag}\left[{\widehat{C}}_{\varphi {\sigma }_r}\left(t,t^{\prime }\right),{\widehat{C}}_{\varphi }\left(t,t^{\prime }\right),{\widehat{C}}_{\varphi }\left(t,t^{\prime }\right)\right].$$

2. Saddle-point approximation for N → ∞

So far, we do not make any use of the fact that we are considering large networks. However, noting that N appears in the exponent in the expression for the disorder-averaged generating functional, we can approximate it using a saddle-point approximation:

$$\overline{Z}[\widehat{b},b]\simeq e^{N\cdot {ℒ}_0\left[\widehat{b},b;C^0,{\widehat{C}}^0\right]}\int 𝒟\widehat{Q}𝒟Q{e}^{-N\cdot {ℒ}_2[\widehat{Q},Q,\widehat{b},b]}.$$

We approximate the action ℒ in Eq. (C14) by its saddle-point value plus a Hessian term: ℒ ≃ ℒ₀ + ℒ₂ and the Q and Q̂ fields represent Gaussian fluctuations about the saddle-point values C⁰ and Ĉ⁰, respectively. At the saddle-point the action is stationary with respect to variations; thus, the saddle-point values of C fields satisfy

$$\begin{gathered} C_{\varphi {\sigma }_r}^0\left(t,t^{\prime }\right)=\frac{1}{N}\sum _{i=1}^N{\langle {\varphi }_i(t){\sigma }_{r,i}(t){\varphi }_i\left(t^{\prime }\right){\sigma }_{r,i}\left(t^{\prime }\right)\rangle }_0, \\ {\widehat{C}}_{\varphi {\sigma }_r}^0\left(t,t^{\prime }\right)=\frac{1}{N}\sum _{i=1}^N{\langle {\widehat{h}}_i(t){\sigma }_{z,i}(t){\widehat{h}}_i\left(t^{\prime }\right){\sigma }_{z,i}\left(t^{\prime }\right)\rangle }_0 \\ =\frac{{\delta }^2{\langle {\sigma }_{z,i}(t){\sigma }_{z,i}\left(t^{\prime }\right)\rangle }_0}{\delta b_i(t)\delta b_i\left(t^{\prime }\right)}=0, \\ {C}_{\varphi }^0\left(t,t^{\prime }\right)=\frac{1}{N}\sum _{k=1}^N{\langle {\varphi }_k(t){\varphi }_k\left(t^{\prime }\right)\rangle }_0, \\ {\widehat{C}}_{\varphi }^0\left(t,t^{\prime }\right)=0. \end{gathered}$$ (C16)

In evaluating the saddle-point correlation function in the second line, we use the fact that equal-time response functions in the Itô convention are zero [29]. This is perhaps the first significant point of departure from previous studies of disordered neural networks and forces us to confront the multiplicative nature of the z gate. Here, 〈⋯〉₀ denotes averages with respect to paths generated by the saddle-point action; thus, these equations are a self-consistency constraint. With the correlation fields fixed at their saddle-point values, if we neglect the contribution of the fluctuations (i.e., ignore ℒ₂), then the generating functional is given by a product of identical sitewise generating functionals:

$$\overline{Z}[\widehat{b},b]=Z_0{[\widehat{b},b]}^N,$$ (C17)

where the sitewise functionals are given by

$$Z_0[\widehat{b},b]=\int 𝒟h𝒟\widehat{h}𝒟z𝒟\widehat{z}𝒟r𝒟\widehat{r}\times e^{\left(i\int dt\left[b{(t)}^T\widehat{h}(t)+\widehat{b}{(t)}^{T}h(t)\right]-S_d\left[\widehat{h},h;\left\{C^0,0\right\}\right]\right)},$$ (C18)

where $C^0=\left(C_{\varphi \sigma }^0,C_{\varphi }^0\right)$.

The sitewise decoupled action is now quadratic with the correlation functions taking on their saddle-point values. This corresponds to an action for each site containing three scalar variables driven by Gaussian processes. This can be seen explicitly by using a Hubbard-Stratonovich transform which makes the action linear at the cost of introducing three auxiliary Gaussian fields η_h, η_z, and η_r with correlation functions $C_{\varphi {\sigma }_r}^0\left(t,t^{\prime }\right)$, $C_{\varphi }^0\left(t,t^{\prime }\right)$, and $C_{\varphi }^0\left(t,t^{\prime }\right)$, respectively. With this transformation, the action for each site corresponds to stochastic dynamics for three scalar variables given by

$$\dot{h}(t)=-{\sigma }_z(z)\cdot h(t)+{\sigma }_z(z)\cdot {\eta }_h(t),$$ (C19)

$${\tau }_z\dot{z}(t)=-z(t)+{\eta }_z(t),$$ (C20)

$${\tau }_r\dot{r}(t)=-r(t)+{\eta }_r(t),$$ (C21)

where the Gaussian noise processes η_h, η_z, and η_r have correlation functions that must be determined self-consistently:

$$\langle {\eta }_h(t)\cdot {\eta }_h\left(t^{\prime }\right)\rangle =\langle \varphi (t){\sigma }_r(t)\cdot \varphi \left(t^{\prime }\right){\sigma }_r\left(t^{\prime }\right)\rangle ,$$

$$\langle {\eta }_z(t)\cdot {\eta }_z\left(t^{\prime }\right)\rangle =\langle \varphi (t)\cdot \varphi \left(t^{\prime }\right)\rangle ,$$

$$\langle {\eta }_r(t)\cdot {\eta }_r\left(t^{\prime }\right)\rangle =\langle \varphi (t)\cdot \varphi \left(t^{\prime }\right)\rangle .$$

The intuitive picture of the saddle-point approximation is as follows: The sites of the full network become decoupled, and they are each driven by a Gaussian processes whose correlation functions summarize the activity of the rest of the network “felt” by each site. It is possible to argue about the final result heuristically, but one does not have access to the systematic corrections that a field theory formulation affords.

We comment here on the unique difficulty that gating presents to an analysis of the DMFT. While r(t) and z(t) are both described by Gaussian processes in the DMFT, the multiplicative σ_z(z) interaction in Eq. (C19) spoils the Gaussianity of h(t). Note that r(t) is always Gaussian and uncorrelated to h(t). In order to try solving for the correlation functions, we need to make a factorization assumption, justified numerically in Fig. 10. The story simplifies at a fixed point, where h = η^h (since σ_z > 0), and is, thus, Gaussian and independent of r.

In order to solve the DMFT equations, we use a numerical method described in Ref. [76]. Specifically, we generate noise paths η_h,z,r starting with an initial guess for the correlation functions and then iteratively update the correlation functions using the mean-field equations till convergence. The classical method of solving the DMFT by mapping the DMFT equations to a second-order ODE describing the motion of a particle in a potential cannot be used in the presence of multiplicative gates. In Fig. 9, we see that the solution to the mean-field equations agrees well with the true population-averaged correlation function; Fig. 9 also shows the scale of fluctuations around the mean-field solutions (Fig. 9, thin black lines).

The correlation functions in the DMFT picture such as C_h(t, t′) = 〈h(t)h(t′)〉 are the order parameters and correspond to the population-averaged correlation functions in the full network. These turn out to useful in our analysis of the RNN dynamics in some analyses. Qualitative changes in the correlation functions correspond to transitions between dynamical regimes of the RNN.

In general, the non-Gaussian nature of h makes it impossible to get equations governing the correlation functions. However, when α_z is not too large, Eqs. (C19) and (C20) can be extended to get equations of motions for the correlation functions C_h, C_z, and C_r, which proves useful later on. This requires a separation assumption between the h and σ_z correlators, which seems reasonable for moderate α_z (see Fig. 10). “Squaring” Eqs. (C19) and (C20), we get

$$\left[-{\partial }_{\tau }^2+C_{{\sigma }_z}(\tau )\right]C_h(\tau )=C_{{\sigma }_z}(\tau )C_{{\sigma }_r}(\tau )C_{\varphi }(\tau ),$$ (C22)

$$\left[-{\tau }_z^2{\partial }_{\tau }^2+1\right]C_z(\tau )=C_{\varphi }(\tau ),$$ (C23)

$$\left[-{\tau }_r^2{\partial }_{\tau }^2+1\right]C_r(\tau )=C_{\varphi }(\tau ),$$ (C24)

where we use the shorthand σ_z(t) ≡ σ_z[z(t)], ϕ(t) ≡ ϕ[h(t)], and denote the two-time correlation functions as

$$C_x\left(t,t^{\prime }\right)=\langle x(t)x\left(t^{\prime }\right)\rangle ,$$ (C25)

where x ∈ {h, z, r, σ_z, σ_r, ϕ} and the expectation here is over the random Gaussian fields in Eqs. (C19)–(C21). We assume that the network reaches steady state, so that the correlation functions are only a function of the time difference τ = t − t′. The role of the z gate as an adaptive time constant is evident in Eq. (C22).

FIG. 9.

Validating the DMFT. We show the comparison between the population-averaged correlation functions C_ϕ(τ) ≡ 〈ϕ(t)ϕ(t + τ)〉 obtained from the full network simulations of a single instantiation in steady state (purple line) and from solving the DMFT equations (red line) for three distinct parameter values. The lag τ is relative to τ_h (taken to be unity). Thin black lines are the time-averaged correlation functions for individual neurons sampled from the network, to show the scale of fluctuations around the population-averaged correlation functions. N = 5000 for all the panels.

FIG. 10.

The validity of the approximation $C_{h{\sigma }_z}=C_h{C}_{{\sigma }_z}$ for two values of α_z. The correlation functions are calculated numerically in a network with n = 1000, g_h = 3.5, and α_r = 0.

For time-independent solutions, i.e., fixed points, Eqs. (C22)–(C24) simplify to read

$${\Delta }_z\equiv \langle z^2\rangle =\int Dx\varphi {\left(\sqrt{{\Delta }_h}x\right)}^2={\Delta }_r,$$ (C26)

$${\Delta }_h\equiv \langle h^2\rangle =\int \mathit{\text{DxDy}}\varphi {\left(\sqrt{{\Delta }_h}x\right)}^2{\sigma }_r{\left(\sqrt{{\Delta }_r}y\right)}^2,$$ (C27)

where we use Δ instead of C to indicate fixed-point variances and D_x is the standard Gaussian measure. It is interesting to note that these mean-field equations can be mapped to those obtained in Ref. [51] for the discrete-time GRU.

We also make use of the MFT with static random inputs. For completeness, we include the resulting equations here. With $I_i^{h,z,r}~𝒩\left(0,{\sigma }_{h,z,r}^2\right)$, the MFT time-independent solution satisfies

$${\Delta }_z=\int Dx\varphi {\left(\sqrt{{\Delta }_h}x\right)}^2+{\sigma }_z^2,$$ (C28)

$${\Delta }_r=\int Dx\varphi {\left(\sqrt{{\Delta }_h}x\right)}^2+{\sigma }_r^2,$$ (C29)

$${\Delta }_h=\int \mathit{\text{DxDy}}\varphi {\left(\sqrt{{\Delta }_h}x\right)}^2{\sigma }_r{\left(\sqrt{{\Delta }_r}y\right)}^2+{\sigma }_h^2.$$ (C30)

APPENDIX D: DETAILS OF THE NUMERICS FOR THE LYAPUNOV SPECTRUM

The evolution of perturbations δx(t) along a trajectory follow the tangent-space dynamics governed by the Jacobian

$${\partial }_t\delta x(t)=𝒟(t)\delta x(t).$$ (D1)

So, after a time T, the initial perturbation δx(0) is given by

$$\delta x(t)=U(t,0)\delta x(0),U(t,0)=𝒯\left[e^{{\int }_0^{t}ds𝒟(s)}\right],$$ (D2)

where 𝒯[⋯] is the time-ordering operator applied to the contents of the bracket. When the infinitesimal perturbations grow (shrink) exponentially, the rate of this exponential growth (decay) is dictated by the maximal Lyapunov exponent defined as [54]

$${\lambda }_{\text{max}}≔\underset{T\to \infty }{\text{lim}}\frac{1}{T}\underset{\parallel \delta x(0)\parallel \to 0}{\text{lim}}\ln \frac{\parallel \delta x(T)\parallel }{\parallel \delta x(0)\parallel },$$ (D3)

For ergodic systems, this limit is independent of almost all initial conditions, as guaranteed by the Oseledets multiplicative ergodic theorem [54]. Positive values of λ_max imply that the nearby trajectories diverge exponentially fast, and the system is chaotic. More generally, the set of all Lyapunov exponents—the Lyapunov spectrum—yields the rates at which perturbations along different directions shrink or diverge and, thus, provides a fuller characterization of asymptotic behavior. The first k-ordered Lyapunov exponents are given by the growth rates of k linearly independent perturbations. These can be obtained as the logarithms of the eigenvalues of the Oseledets matrix, defined as [54]

$$M(t)=\underset{t\to \infty }{\text{lim}}{\left[U{(t,0)}^{T}U(t,0)\right]}^{1/2t}.$$ (D4)

However, this expression cannot be directly used to calculate the Lyapunov spectra in practice, since M(t) rapidly becomes ill conditioned. We instead employ a method suggested by Ref. [77] (also cf. Ref. [78] for Lyapunov spectra of RNNs). We start with k orthogonal vectors Q⁰ = [q₁, …, q_k] and evolve them using the tangent-space dynamics [Eq. (D1)] for a short time interval t₀. Therefore, the new set of vectors is given by

$$\widehat{Q}=U\left(t_0,0\right)Q^0.$$ (D5)

We now decompose Q̂ = Q¹R¹ using a QR decomposition, into an orthogonal matrix Q¹ and a upper-triangular matrix R¹ with positive diagonal elements, which give the rate of shrinkage or expansion of the volume element along the different directions. We iterate this procedure for a long time, t₀ × N_l, and the first k-ordered Lyapunov exponents are given by

$${\lambda }_i=\underset{N_l\to \infty }{\text{lim}}\frac{1}{N_l{t}_0}\sum _{j=1}^{N_l}\ln R_{ii}^j,i\in \{1,\dots ,k\}.$$ (D6)

APPENDIX E: DETAILS OF THE DMFT PREDICTION FOR λ_max

The starting point of the method to calculate the DMFT prediction for λ_max is two replicas of the system x¹(t) and x²(t) with the same coupling matrices J^h,z,r and the same parameters. If the two systems are started with initial conditions which are close, then the rate of convergence or divergence of the trajectories reveals the maximal Lyapunov exponent. To this end, let us define $d(t,s)≔N^{-1}\sum _i{\left[x_i^1(t)-x_i^2(s)\right]}^2$ and study the growth rate of d(t, t). In the large N limit, we expect population averages like $C^{12}(t,s)≔N^{-1}\sum _i{x}_i^1(t)x_i^2(s)$ to be self-averaging (like in the DMFT for a single system) [79], and, thus, we can write

$$d(t,s)=C^{11}(t,t)+C^{22}(s,s)-C^{12}(t,s)-C^{21}(t,s).$$ (E1)

For trajectories that start nearby, the asymptotic growth rate of d(t) is the maximal Lyapunov exponent. In order to calculate this using the DMFT, we need a way to calculate C¹²—the correlation between replicas—for a typical instantiation of systems in the large N limit. As suggested by Ref. [21], this can be achieved by considering a joint generating functional for the replicated system:

$${\tilde{Z}}_{𝒥}\left[{\widehat{b}}^1,{\widehat{b}}^2,b^1,b^2\right]=\mathbb{E}\left[\text{exp}\left(i\sum _{\mu =1}^2\sum _{j=1}^N\int {\widehat{b}}_j^{\mu }{(t)}^T{x}_j^{\mu }(t)dt\right)\right].$$ (E2)

We then proceed to take the disorder average of this generating functional—in much the same way as a single system—and this introduces correlations between the state vectors of the two replicas. A saddle-point approximation as in the single system case (cf. Appendix C) yields a system of coupled stochastic differential equations (SDEs) (one for each replica), similar to Eq. (C20), but now the noise processes in the two replicas are coupled, so that terms like $\langle {\eta }_h^1(t){\eta }_h^2\left(t^{\prime }\right)\rangle$ need to be considered. As before, the SDEs imply the equations of motion for the correlation functions

$$\left[-{\partial }_{\tau }^2+C_{{\sigma }_z}^{\mu \nu }(\tau )\right]C_h^{\mu \nu }(\tau )=C_{{\sigma }_z}^{\mu \nu }(\tau )C_{\varphi }^{\mu \nu }(\tau )C_{{\sigma }_r}^{\mu \nu }(\tau ),$$ (E3)

$$\left[-{\tau }_z^2{\partial }_{\tau }^2+1\right]C_z^{\mu \nu }(\tau )=C_{\varphi }^{\mu \nu }(\tau ),$$ (E4)

$$\left[-{\tau }_r^2{\partial }_{\tau }^2+1\right]C_r(\tau )=C_{\varphi }^{\mu \nu }(\tau ),$$ (E5)

where μ, v ∈ {1, 2} are the replica indices. Note that the single-replica solution clearly is a solution to this system, reflecting the fact that marginal statistics of each replica is the same as before. When the replicas are started with initial conditions that are ϵ-close, we expect the inter-replica correlation function to diverge from the single-replica steady-state solution, so we expand C¹² to linear order as $C_{h,z,r}^{12}(t,s)\approx C_{h,z,r}(t-s)+ϵ{\tilde{\chi }}_{h,z,r}(t,s)$. From Eq. (E1), we see that $d(t,t)~ϵ\tilde{\chi }(t,t)$, and, thus, the growth rate of $\tilde{\chi }$ yields the required Lyapunov exponent. To this end, we make an ansatz ${\tilde{\chi }}_{h,z,r}=e^{\kappa T}\chi (\tau )$, where 2T = t + s, 2τ = t − s, and κ is the DMFT prediction of the maximum Lyapunov exponent that needs to be solved for. Substituting this back into Eq. (E3), we get a generalized eigenvalue problem for κ as stated in the text [Eqs. (10) and (11)].

APPENDIX F: CALCULATION OF MAXIMAL LYAPUNOV EXPONENT FROM RMT

The DMFT prediction for how gates shape λ_max (via the correlation functions) is somewhat involved; thus, we provide an alternate expression for the maximal Lyapunov exponent λ_max, derived using RMT which relates it to the relaxation time of the dynamics. The starting point to get λ_max is the Oseledets multiplicative ergodic theorem, which guarantees that [80]

$${\lambda }_{\text{max}}=\underset{t\to \infty }{\text{lim}}\frac{1}{2t}\text{log}\frac{\parallel \chi (t){\parallel }^2}{N}$$ (F1)

$$=\underset{t\to \infty }{\text{lim}}\frac{1}{2t}\text{log}\frac{1}{N}\text{Tr}\left[\chi (t)\chi {(t)}^T\right],$$ (F2)

where $\chi (t)=𝒯e^{{\int }_0^{t}d{t}^{\prime }𝒟\left(t^{\prime }\right)}$ and 𝒟 is the Jacobian. For the vanilla RNN, the Jacobian is given by

$$𝒟=-\mathbb{1}+J\left[{\varphi }^{\prime }(t)\right].$$ (F3)

We expect the maximal Lyapunov exponent to be independent of the random network realization and, thus, equal to its value after disorder averaging. Furthermore, to make any progress, we use a short-time approximation for $\chi (t)\approx e^{{\int }_0^{t}d{t}^{\prime }𝒟\left(t^{\prime }\right)}$. Defining the diagonal matrix R(t) = ∫^t [ϕ′(t′)]dt′, these assumptions give

$$\frac{1}{N}\text{Tr}\left[\chi (t)\chi {(t)}^T\right]\approx e^{-2t}\langle \frac{1}{N}\text{Tr}{e}^{JR(t)}{e}^{R(t)J^T}\rangle$$ (F4)

$$=e^{-2t}\sum _{n=0}^{\infty }\frac{1}{{(n!)}^2}{\left(\frac{1}{N}\text{Tr}R{(t)}^2\right)}^n,$$ (F5)

where the second line in Eq. (F5) follows after disorder averaging over J and keeping only terms to leading order in N. Next, we may apply the DMFT to write

$$\frac{1}{N}\text{Tr}R{(t)}^2={\int }^{t}d{t}^{\prime }d{t}^{″}\frac{1}{N}\sum _{i=1}^N{\varphi }_i^{\prime }\left(t^{″}\right){\varphi }_i^{\prime }\left(t^{\prime }\right)$$ (F6)

$$\approx \int d{t}^{\prime }d{t}^{″}{C}_{{\varphi }^{\prime }}\left(t^{\prime },t^{″}\right).$$ (F7)

In steady state, the correlation function depends only on the difference of the two times, and, thus, we can write

$$\int d{t}^{\prime }d{t}^{″}{C}_{{\varphi }^{\prime }}\left(t^{\prime },t^{″}\right)\approx {\int }_0^{2t}\frac{du}{2}{\int }_0^{t}d\tau C_{{\varphi }^{\prime }}(\tau )\equiv t^2{\tau }_R,$$ (F8)

where we define the relaxation time for the C_ϕ′ correlation function

$${\tau }_R\equiv \frac{1}{t}{\int }_0^{t}d\tau C_{{\varphi }^{\prime }}(\tau ).$$ (F9)

Substituting Eq. (F8) in Eq. (F4), we get

$$\frac{1}{N}\text{Tr}\left[\chi (t)\chi {(t)}^T\right]=e^{-2t}{I}_0\left(2t\sqrt{{\tau }_R}\right),$$ (F10)

which for long times behaves like $\text{exp}\left[2\left(\sqrt{{\tau }_R}-1\right)t\right]$. By inserting this into Eq. (F1), we obtain a bound for the maximal Lyapunov exponent for the vanilla RNN:

$${\lambda }_{\text{max}}\ge \sqrt{{\tau }_R}-1,$$ (F11)

$$\text{ where }{\tau }_R\equiv \frac{1}{t}{\int }_0^{t}d\tau C_{{\varphi }^{\prime }}(\tau ).$$ (F12)

This formula relates the asymptotic Lyapunov exponent to relaxation time of a local correlation function in steady state. It is interesting to note that the bound also follows by applying the variational theorem to the potential energy obtained from the Schrodinger equation that arises in computing the Lyapunov exponent using DMFT (e.g., see Refs. [15,32]). Specifically, if one uses the potential obtained in these works V(τ) = 1 − C_ϕ′(τ), and assumes a uniform “ground state wave function,” the variational theorem implies that the true ground state energy E₀ is upper bounded $E_0\le {\lim }_{T\to \infty }(1/T){\int }_{-T/2}^{T/2}V(\tau )d\tau \equiv 1-{\tau }_R$, which consequently implies the bound (F11).

Now we present the derivation for the mean-squared singular value of the susceptibility matrix for the gated RNN with α_z = 0 and β_z = −∞. In this limit, σ_z = 1, and the instantaneous Jacobian becomes the 2N × 2N matrix

$${𝒟}_t=-{\mathbb{1}}_{2N}+\left(\begin{array}{cc}{J}^r& 0\\ 0& J^h\end{array}\right)\left(\begin{array}{cc}0& P_t\\ Q_t& R_t\end{array}\right)\equiv -1_{2N}+\widehat{J}{S}_t,$$ (F13)

$$Q_t=\left[\varphi (h)\odot {\sigma }_r^{\prime }(r)\right],P_t=\left[{\varphi }^{\prime }(h)\right],$$ (F14)

$$R_t=\left[{\varphi }^{\prime }(h)\odot {\sigma }_r(r)\right],$$ (F15)

where h = h(t) and r = r(t) are time dependent.

Let us define the quantity of interest

$${\sigma }_{\chi }^2=\langle \frac{1}{2N}\text{Tr}\left[\chi (t){\chi }^T(t)\right]\rangle$$ (F16)

$$=e^{-2t}\langle \frac{1}{2N}\text{Tr}{e}^{\widehat{J}{\widehat{S}}_t}{e}^{{\widehat{S}}_t^T{\widehat{J}}^T}\rangle ,$$ (F17)

where we additionally define Ŝ_t = ∫^t dt′S_t and the integration is performed elementwise. Expanding the exponentiated matrices and computing moments directly, one finds that the leading order in N moments must have an equal number of Ĵ and Ĵ^T. Thus, we must evaluate

$$c_n=\langle \frac{1}{2N}\text{Tr}\left[{\left(\widehat{J}{\widehat{S}}_t\right)}^n{\left({\widehat{S}}_t^T{\widehat{J}}^T\right)}^n\right]\rangle .$$ (F18)

The ordering of the matrices is important in this expression. Since all of the Ĵ appear to the left of Ĵ^T, the leading-order contributions to the moment come from Wick contractions that are “noncrossing”—in the language of diagrams, the moment is given by a “rainbow” diagram. Consequently, we may evaluate c_n by induction. First, the induction step. Define the expected value of the matrix moment

$${\widehat{c}}_n=\langle {\left(\widehat{J}{\widehat{S}}_t\right)}^n{\left({\widehat{S}}_t^T{\widehat{J}}^T\right)}^n\rangle$$ (F19)

$$=\langle \widehat{J}\left[{\widehat{S}}_t{\left(\widehat{J}{\widehat{S}}_t\right)}^{n-1}{\left({\widehat{S}}_t^T{\widehat{J}}^T\right)}^{n-1}{\widehat{S}}_t^T\right]{\widehat{J}}^T\rangle$$ (F20)

$$=\left(\begin{array}{cc}{a}_n\mathbb{1}& 0\\ 0& b_n\mathbb{1}\end{array}\right)+O\left(N^{-1}\right).$$ (F21)

We wish to determine a_n and b_n. Next, define

$$g_P=\frac{1}{N}\text{Tr}{\int }^{t}d{t}^{\prime }d{t}^{″}{P}_{t^{\prime }}{P}_{t^{″}},$$ (F22)

$$g_Q=\frac{1}{N}\text{Tr}{\int }^{t}d{t}^{\prime }d{t}^{″}{Q}_{t^{\prime }}{Q}_{t^{″}},$$ (F23)

$$g_R=\frac{1}{N}\text{Tr}{\int }^{t}d{t}^{\prime }d{t}^{″}{R}_{t^{\prime }}{R}_{t^{″}}.$$ (F24)

Now we can directly determine the induction step at the level of matrix moments by Wick contraction of the rainbow diagram:

$${\widehat{c}}_n=\langle \widehat{J}{\widehat{S}}_t{\left(\widehat{J}{\widehat{S}}_t\right)}^{n-1}{\left({\widehat{S}}_t^T{\widehat{J}}^T\right)}^{n-1}{\widehat{S}}_t^T{\widehat{J}}^T\rangle$$ (F25)

$$=\langle \widehat{J}{\widehat{S}}_t{\widehat{c}}_{n-1}{\widehat{S}}_t^T{\widehat{J}}^T\rangle +O\left(N^{-1}\right)$$ (F26)

$$=\left(\begin{array}{cc}{b}_{n-1}{g}_P\mathbb{1}& 0\\ 0& \left(a_{n-1}{g}_Q+b_{n-1}{g}_R\right)\mathbb{1}\end{array}\right)+O\left(N^{-1}\right).$$ (F27)

This implies the following recursion for the diagonal elements of ĉ_n:

$$a_n=g_P{b}_{n-1},b_n=g_R{b}_{n-1}+g_Q{a}_{n-1}.$$ (F28)

The initial condition is given by observing that ${\widehat{c}}_0=\mathbb{1}$, which implies a₀ = b₀ = 1. The solution to this recursion relation can be written in terms of a transfer matrix

$$\left(\begin{array}{l}{a}_n\hfill \\ b_n\hfill \end{array}\right)={\left(\begin{array}{ll}0\hfill & g_P\hfill \\ g_O\hfill & g_R\hfill \end{array}\right)}^n\left(\begin{array}{l}1\hfill \\ 1\hfill \end{array}\right),$$ (F29)

which implies the moment $c_n=\frac{1}{2}\left(a_n+b_n\right)$ is given by

$$c_n=\frac{1}{2}\left(\begin{array}{ll}1\hfill & 1\hfill \end{array}\right){\left(\begin{array}{cc}0& g_P\\ g_Q& g_R\end{array}\right)}^n\left(\begin{array}{l}1\hfill \\ 1\hfill \end{array}\right).$$ (F30)

To evaluate this, we use the fact that the eigenvalues of the transfer matrix are

$$v_{\pm }=\frac{1}{2}\left(g_R\pm \sqrt{g_R^2+4g_P{g}_Q}\right),$$ (F31)

which are real valued. The eigenvectors are

$$v_{\pm }=\left(-\frac{v_{\mp }}{g_Q},1\right).$$ (F32)

Then, defining l = (1, 1), the moment can be written

$$c_n=\frac{1}{2}{l}^T\left(v_{+}^n{v}_{+}{v}_{+}^T+v_{-}^n{v}_{-}{v}_{-}^T\right)l$$ (F33)

$$=\frac{1}{2}{\left(1-\frac{v_{-}}{g_Q}\right)}^2{v}_{+}^n+\frac{1}{2}{\left(1-\frac{v_{+}}{g_Q}\right)}^2{v}_{-}^n.$$ (F34)

The final expression for the mean-squared singular value is then

$${\sigma }_{\chi }^2=e^{-2t}\sum _{n=0}^{\infty }\frac{c_n}{{(n!)}^2}.$$ (F35)

After resumming this infinite series, we wind up with an expression in terms of the modified Bessel function:

$${\sigma }_{\chi }^2=\frac{1}{2}{e}^{-2t}\left[{\left(1-\frac{v_{-}}{g_Q}\right)}^2{I}_0\left(2\sqrt{v_{+}}\right)+{\left(1-\frac{v_{+}}{g_Q}\right)}^2{I}_0\left(2\sqrt{v_{-}}\right)\right].$$ (F36)

In the steady state, we approximate these expressions by assuming the correlation functions are time-translation invariant. Thus, we may write, for instance,

$$g_R=\int dtd{t}^{\prime }{R}_t{R}_{t^{\prime }}\approx t^2\frac{1}{t}\int d\tau C_R(\tau )=t^2{\tau }_R,$$ (F37)

and similarly for g_Q and g_P. Then, the eigenvalues of the transfer matrix become

$$v_{\pm }=t^2\frac{1}{2}\left({\tau }_R\pm \sqrt{{\tau }_R^2+4{\tau }_P{\tau }_Q}\right).$$ (F38)

At late times, using the asymptotic behavior of the modified Bessel function, the moment becomes

$${\sigma }_{\chi }^2~\text{exp}\left(-2t+2\sqrt{v_{+}}\right),$$ (F39)

which gives the Lyapunov exponent

$${\lambda }_L\ge {\left(\frac{{\tau }_R+\sqrt{{\tau }_R^2+4{\tau }_P{\tau }_Q}}{2}\right)}^{1/2}-1,$$ (F40)

where the relaxation times τ_A, τ_r, and τ_q are defined as, respectively,

$${\tau }_R=\underset{t\to \infty }{\text{lim}}\frac{1}{t}{\int }_0^{t}d\tau C_{{\varphi }^{\prime }}(\tau )C_{{\sigma }_r}(\tau ),$$ (F41)

$${\tau }_A=\underset{t\to \infty }{\text{lim}}\frac{1}{t}{\int }_0^{t}d\tau C_{{\varphi }^{\prime }}(\tau ),$$ (F42)

$${\tau }_Q=\underset{t\to \infty }{\text{lim}}\frac{1}{t}{\int }_0^{t}d\tau C_{\varphi }(\tau )C_{{\sigma }_r^{\prime }}(\tau ).$$ (F43)

APPENDIX G: DETAILS OF THE DISCONTINUOUS CHAOTIC TRANSITION

In this section, we provide the details for the calculations involved in the discontinuous chaotic transition.

1. Spontaneous emergence of fixed-points

For g_h < 2.0 and small α_r, the zero fixed point is the globally stable state for the dynamics and the only solution to the fixed-point equations [Eq. (C26)] for Δ_h. However, as we increase α_r for a fixed g_h, two additional nonzero solutions to Δ_h spontaneously appear at a critical value ${\alpha }_{\text{FP}}^{*}\left(g_h\right)$ as shown in Fig. 4(a). Numerical solutions to the fixed-point equations reveal the form of the bifurcation curve ${\alpha }_{r,\text{FP}}^{*}\left(g_h\right)$ and the associated value of ${\Delta }_h^{*}\left(g_h\right)$. We see that ${\alpha }_{r,\text{FP}}^{*}\left(g_h\right)$ increases rapidly with decreasing g_h, dividing the parameter space into regions with either one or three solutions for Δ_h. The corresponding ${\Delta }_h^{*}\left(g_h\right)$ vanishes at two boundary values of g_h—one at 2.0 and another, g_c, below 1.5, where ${\alpha }_r^{*}\to \infty$. This naturally leads to the question of whether the fixed-point bifurcation exists for all values of g_h below 2.0.

To answer this, we perturbatively solve the fixed-point equations in two asymptotic regimes: (i) g_h → 2⁻ and (ii) $g_h\to g_c^{+}$. Details of the perturbative treatment are in Appendix I 2. For g_h = 2 − ϵ, we see that the perturbative problem undergoes a bifurcation from one solution (Δ_h = 0) to three when α_r crosses the bifurcation threshold ${\alpha }_r^{*}(2.0)=\sqrt{8}$, and this is the left limit of the bifurcation curve in Fig. 4(b). The larger nonzero solution for the variance at the bifurcation point scales as

$${\Delta }_h^{*}\approx \left({\alpha }_r^2-8\right)\cdot {\xi }_0+{\xi }_1ϵ\text{ for }{\alpha }_r\to {\alpha }_{r,\text{FP}}^{*}(2)=\sqrt{8},$$ (G1)

where ξ₀ and ξ₀ are positive constants (see Appendix I 2).

At the other extreme, to determine the smallest value of g_h for which a bifurcation is possible, we note from Fig. 4(b) that in this limit α_r → ∞, and, thus, we can look for solutions to Δ_h in the limit: Δ_h ≪ 1 and α_r → ∞ and ${\alpha }_r\sqrt{{\Delta }_h}\gg 1$. In this limit, there is a bifurcation in the perturbative solution when $g_h>g_h^{*}=\sqrt{2}$, and, close to the critical point, the fixed-point solution is given by (see Appendix I 2)

$${\Delta }_h^{*}\left({\sqrt{2}}^{+}\right)~\frac{g_h^2-2}{2g_h^4}\text{\hspace{1em} for }{g}_h\to {\sqrt{2}}^{+}.$$ (G2)

Thus, in the region $g_h\in (\sqrt{2},2)$, there exist nonzero solutions to the fixed-point equations once α_r is above a critical value ${\alpha }_r^{*}\left(g_h\right)$. These solutions correspond to unstable fixed points appearing in the phase space.

2. Delayed dynamical transition shows a decoupling between topological and dynamical complexity

The picture from the fixed-point transition above is that. when g_h is in the interval ($\sqrt{2}$, 2), there is a proliferation of unstable fixed points in the phase space provided ${\alpha }_r>{\alpha }_{r,\text{FP}}^{*}\left(g_h\right)$. However, it turns out that the spontaneous appearance of these unstable fixed points is not accompanied by any asymptotic dynamical signatures—as measured by the Lyapunov exponents (see Fig. 4) or by the transient times (see Fig. 11). It is only when α_r is increased further beyond a second critical value ${\alpha }_{r,\text{DMFT}}^{*}\left(g_h\right)$ that we see the appearance of chaotic and long-lived transients. This is significant in regard to a result by Wainrib and Touboul [45], where they show that the transition to chaotic dynamics (dynamical complexity) in random RNNs is tightly linked to the proliferation of critical points (topological complexity), and, in their case, the exponential rate of growth of critical points (a topological property) is the same as the maximal Lyapunov exponent (a dynamical property).

FIG. 11.

Transient times at the bifurcation transition. Transient times (τ_T, relative to τ_h) as a function of g_h, α_r, and system size N. Dashed plot lines correspond to situations where ${\alpha }_r<{\alpha }_{r,\text{DMFT}}^{*}\left(g_h\right)$. Dashed vertical lines are critical values of α_r or g_h. (a) τ_T vs N for g_h = 1.775 and (b) τ_T vs g_h for N = 500; the dashed line indicates g_h such that ${\alpha }_{r,\text{DMFT}}^{*}\left(g_h\right)=40$. (c) τ_T vs α_r for g_h = 1.775; the dashed vertical line is ${\alpha }_{r,\text{DMFT}}^{*}(1.775)$. (d) τ_T vs g_h for α_r = 30; the dashed line is g_h such that ${\alpha }_{r,\text{DMFT}}^{*}\left(g_h\right)=30$. (e) τ_T vs N for α_r = 30; dashed plot lines correspond to situations where $30>{\alpha }_{r,\text{DMFT}}^{\ast }\left(g_h\right)$. (f) τ_T vs α_r for N = 500; the dashed vertical line is ${\alpha }_{r,\text{DMFT}}^{\ast }\left(1.85\right)$. Transient times are averaged over 2000 instances of random networks.

Let us characterize the second dynamical transition curve given by ${\alpha }_{r,\text{DMFT}}^{*}\left(g_h\right)$ [Fig. 4(c), red curve]. For ease of discussion, we turn off the update gate (α_z = 0) and introduce a functional F_ψ for a 2D Gaussian average of a given function ψ(x):

$$F_{\psi }\left[C_h(0),C_h(\tau )\right]=\mathbb{E}\left[\psi \left(z_1\right)\psi \left(z_2\right)\right],$$ (G3)

$$\text{ where }\left(\begin{array}{l}{z}_1\hfill \\ z_2\hfill \end{array}\right)~𝒩\left(0,C_h\right),C_h=\left(\begin{array}{ll}{C}_h(0)\hfill & C_h(\tau )\hfill \\ C_h(\tau )\hfill & C_h(0)\hfill \end{array}\right).$$ (G4)

The DMFT equations for the correlation functions then become

$$\frac{1}{4}{C}_h(\tau )-{\partial }_{\tau }^2{C}_h(\tau )=\frac{1}{4}{F}_{\varphi }\left[C_h(0),C_h(\tau )\right]F_{{\sigma }_r}\left[C_r(0),C_r(\tau )\right],$$

$$C_r(\tau )-{\tau }_r^2{\partial }_{\tau }^2{C}_r(\tau )=F_{\varphi }\left[C_{\varphi }(0),C_{\varphi }(\tau )\right].$$ (G5)

We further make an approximation that τ_r ≪ 1, which, in turn, implies C_r(τ) ≈ C_ϕ(τ). This approximation turns out to hold even for moderately large τ_r. With these approximations, we can integrate the equations for C_h(τ) to arrive at an equation for the variance $C_h^0\equiv C_h(0)$. We do this by multiplying by ∂_τC_h(τ) and integrating from τ to ∞, and we get

$$\frac{1}{2}{\dot{C}}_h{(\tau )}^2=\frac{1}{4}\frac{{\left(C_h\right)}^2}{2}-\frac{1}{4}{\int }_0^{C_h^0}d{C}_h{F}_{\varphi }\left(C_h,C_h^0\right)F_{{\sigma }_r}\left(C_{\varphi },C_{\varphi }^0\right).$$ (G6)

Using the boundary condition that Ċ_h(0) = 0, we get the equation for the variance:

$$\frac{1}{8}{C}_h{(0)}^2-\frac{1}{4}{\int }_0^{C_h^0}d{C}_h{F}_{\varphi }\left(C_h,C_h^0\right)F_{{\sigma }_r}\left(C_{\varphi },C_{\varphi }^0\right)=0.$$ (G7)

Solving this equation gives the DMFT prediction for the variance for any g_h and α_r. Beyond the critical value of α_r, two nonzero solutions for $C_h^0$ spontaneously emerge. In order to use Eq. (G7) to find a prediction for the DMFT bifurcation curve ${\alpha }_{r,\text{DMFT}}^{*}\left(g_h\right)$, we need to use the additional fact that at the bifurcation point the two solutions coincide, and there is only one nonzero solution. To proceed, we can view the lhs of Eq. (G7), as a function of α_r, g_h, and $C_h^0:ℱ\left(g_h,{\alpha }_r,C_h^0\right)$. Then, the equation for the bifurcation curve is obtained by solving the following two equations for $C_h^{0,*}$ and ${\alpha }_r^{*}$:

$$ℱ\left(g_h,{\alpha }_r^{*},C_h^{0,*}\right)=0,$$ (G8)

$${\frac{\partial ℱ\left(g_h,{\alpha }_r,C_h^0\right)}{\partial C_h^0}|}_{{\alpha }_r^{*},C_h^{0,*}}=0.$$ (G9)

To get the condition for the dynamical bifurcation transition, we need to differentiate the lhs of Eq. (G7) $\left[ℱ\left(g_h,{\alpha }_r,C_h^0\right)\right]$ with respect to $C_h^0$ and set it to 0. This involves terms like

$$\frac{\partial F_{\psi }\left(C_h^0,C_h^0\right)}{\partial C_h^0};\frac{\partial F_{\psi }\left(C_h^0,0\right)}{\partial C_h^0}.$$ (G10)

We give a brief outline of calculating the first term. It is easier to work in the Fourier domain:

$$\begin{gathered} F_{\psi }\left(C_h^0,C_h\right) \\ =\mathbb{E}\left[\int \frac{dk}{2\pi }\int \frac{d{k}^{\prime }}{2\pi }\tilde{\psi }(k)e^{-k{z}_1}\tilde{\psi }\left(k^{\prime }\right)e^{-k^{\prime }{z}_2}\right] \\ =\int \frac{dk}{2\pi }\int \frac{d{k}^{\prime }}{2\pi }\tilde{\psi }(k)\tilde{\psi }\left(k^{\prime }\right)\text{exp}\left[-\frac{C_h^0}{2}\left(k^2+k^{\prime 2}\right)-C_h(\tau )k{k}^{\prime }\right]. \end{gathered}$$ (G11)

This immediately gives us

$$\frac{\partial F_{\psi }\left(C_h^0,C_h^0\right)}{\partial C_h^0}=\int 𝒟x\psi \left(\sqrt{c_h^0}x\right){\psi }^{″}\left(\sqrt{c_h^0}x\right)+\int 𝒟x{\psi }^{\prime }{\left(\sqrt{c_h^0}x\right)}^2,$$

$$\frac{\partial F_{\psi }\left(C_h^0,0\right)}{\partial C_h^0}=\int 𝒟x\psi \left(\sqrt{c_h^0}x\right)\int 𝒟x{\psi }^{″}\left(\sqrt{c_h^0}x\right).$$ (G12)

Using this fact, we can calculate the derivative of $ℱ\left(g_h,{\alpha }_r,C_h^0\right)$ as a straightforward (but long) sum of Gaussian integrals. We then numerically solve Eqs. (G8) and (G9) to get the bifurcation curve shown in Fig. 4(c). Figure 4(d) shows the corresponding variance at the bifurcation point $C_h^{0,*}$ (red curves). We note two salient points: (i) The DMFT bifurcation curve is always above the fixed-point bifurcation curve [black, in Fig. 4(a)], and (ii) the lower critical value of g_h which permits a dynamical transition [dashed green curve in Figs. 4(a) and 4(b)] is smaller than the corresponding fixed-point critical value of $\sqrt{2}$.

We now calculate the lower critical value of g_h and provide an analytical description of the asymptotic behavior near the lower and higher critical values of g_h. From the red curve in Fig. 4(c), we know that, as g_h tends toward the lower critical value, ${\alpha }_{r,\text{DMFT}}^{*}\to \infty$ and $C_h^0\to 0$, we can approximate σ_r as a step function in this limit, and $F_{{\sigma }_r}$ is approximated as

$$F_{{\sigma }_r}\left(C_{\varphi }^0,C_{\varphi }\right)\approx \frac{1}{4}+\frac{1}{2\pi }{\text{tan}}^{-1}\left(\frac{x}{\sqrt{1-x^2}}\right),$$ (G13)

$$\text{where }x≔\frac{C_h(\tau )}{C_h(0)}\approx \frac{C_{\varphi }(\tau )}{C_{\varphi }(0)}\text{. }$$ (G14)

The DMFT equation then reads

$$4\ddot{x}=x-g_h^{2}x\left[\frac{1}{4}+\frac{1}{2\pi }{\text{tan}}^{-1}\left(\frac{x}{\sqrt{1-x^2}}\right)\right]+O\left[C_h{(0)}^2\right].$$

Integrating this equation, we get

$$2{\dot{x}}^2=\frac{x^2}{2}\left(1-\frac{g_h^2}{4}\right)+\frac{g_h^2}{8\pi }\left[\left(1-2x^2\right){\text{sin}}^{-1}(x)-x\sqrt{1-x^2}\right],$$

which has O(C_h(0)²) corrections. From the boundary condition Ċ_h(0) = 0, we know that as x → 1 then ẋ → 0. We thus find that these boundary conditions are consistent only to leading order in C_h(0) when g_h is equal to its critical value:

$$g_h^{*}=\sqrt{\frac{8}{3}},$$ (G15)

which indicates that C_h(0) must vanish as $g_h\to {\sqrt{8/3}}^{+}$.

In the other limit when g_h → 2⁻, we see that ${\alpha }_r^{*}$ remains finite and $C_h^{0,*}\to 0$. We assume that, for g_h = 2 − ϵ, $C_h^0$ has a power-series expansion

$$C_h^0=c_0+c_1ϵ+c_2{ϵ}^2+\cdots .$$ (G16)

We also expand F_ϕ and $F_{{\sigma }_r}$ to O[C_h(0)²]:

$$F_{\varphi }\approx g_h^2{C}_h(\tau )-2g_h^4{C}_h^0\cdot C_h(\tau )+5g_h^6{\left(C_h^0\right)}^2\cdot C_h(\tau )$$ (G17)

and look for values of α_r which permit a nonzero value for c₀ in the leading-order solutions to the DMFT. We find that the critical value of α_r from the perturbative solution is given by

$${\alpha }_{r,\text{DMFT}}^{*}(2)=\sqrt{12}.$$ (G18)

The DMFT prediction for the dynamical bifurcation agrees well with the full network simulations. In Fig. 4(e), we see that the maximum Lyapunov exponent experiences a discontinuous transition from a negative value (network activity decays to fixed point) to a positive value (activity is chaotic) at the critical value of α_r predicted by the DMFT (dashed vertical lines).

3. Influence of update gate on the discontinuous transition

Here, we comment briefly on the possible influence of the z gate on the discontinuous dynamical phase transition given by the curve ${\alpha }_{r,\text{DMFT}}^{*}$. Assuming Eq. (C22) is valid (discussed in more detail toward the end of Appendix C), we may rewrite the DMFT equation for the two-point correlation functions as

$$\frac{1}{2}{\dot{C}}_h{(\tau )}^2={\int }_0^{C_h(\tau )}{F}_{{\sigma }_z}\left(C_{\varphi }^0,C_{\varphi }\right)Q\left(C_h,C_h^0\right),$$ (G19)

where

$$Q\left(C_h,C_h^0\right)=C_h-F_{\varphi }\left(C_h,C_h^0\right)F_{{\sigma }_r}\left(C_{\varphi },C_{\varphi }^0\right).$$ (G20)

Noting that a time-dependent solution corresponds to a nonzero solution for C_h(0) and satisfies the boundary condition Ċ_h(0) = 0 then requires

$${ℱ}_{{\alpha }_z}\equiv {\int }_0^{C_h^0}d{C}_h{F}_{{\sigma }_z}\left(C_{\varphi },C_{\varphi }^0\right)Q\left(C_h,C_h^0\right)=0,$$ (G21)

where we define a new “potential” function which is related to that defined above by

$${{ℱ}_{{\alpha }_z}|}_{{\alpha }_z=0}=ℱ=\frac{1}{4}{\int }_0^{C_h^0}d{C}_{h}Q\left(C_h,C_h^0\right).$$ (G22)

We leave the arguments (g_h, α_r, $C_h^0$) implicit, for ease of presentation. We proceed to bound the new potential by establishing bounds on $F_{{\sigma }_z}$. To be explicit, we have

$$F_{{\sigma }_z}=\langle {\sigma }_z(\tau ){\sigma }_z(0)\rangle ={\langle {\sigma }_z(\tau ){\sigma }_z(0)\rangle }_c+{\langle {\sigma }_z\rangle }^2,$$ (G23)

which we express as the sum of a connected component (indicated by a subscript c) and a disconnected component. We can consider two limiting behaviors. When the correlation time tends to zero, the connected component vanishes and (at zero bias βz = 0)

$$F_{{\sigma }_z}\approx {\langle {\sigma }_z\rangle }^2=\frac{1}{4}.$$ (G24)

Increasing the correlation time can serve only to increase the two-point function, since σ ≥ 0. In the extreme limit of very long correlation time, we have that

$$F_{{\sigma }_z}\approx \langle {\sigma }_z^2(0)\rangle \le \frac{1}{2}.$$ (G25)

The inequality is saturated at α_z = ∞, when σ_z becomes a step function of its argument. Therefore, the two-point correlation function of the update gate is bounded above and below:

$$\frac{1}{4}\le F_{{\sigma }_z}\le \frac{1}{2},$$ (G26)

and this bound is uniform in the sense that it holds for all values of the argument $0\le C_h\le C_h^0<\infty$. Consequently, we are able to bound the potential

$$\frac{1}{4}ℱ\le {ℱ}_{{\alpha }_z}\le \frac{1}{2}ℱ.$$ (G27)

It follows immediately that the derivative is similarly bounded. Consequently, the zeros of ${ℱ}_{{\alpha }_z}$ and $\partial {ℱ}_{{\alpha }_z}/\partial C_h^0$ coincide with the zeros of ℱ and $\partial ℱ/\partial C_h^0$, respectively. As a result, the discontinuous transition, determined by Eqs. (G8) and (G9), remain unchanged for values of α_z for which Eq. (C22) is valid. Thus, for moderately large α_z [approximately 10, where Eq. (C22) is valid], the critical line for the discontinuous transition remains unchanged.

APPENDIX H: THE ROLE OF BIASES

We thus far describe the salient dynamical aspects for the gated RNN in the absence of biases. Here, we describe the role of the biases β_h (bias of the activation ϕ) and β_r (bias of the output gate σ_r). We first note that, when β_h = 0, zero is always a fixed point of the dynamics, and the zero fixed point is stable provided

$$-1+{\varphi }^{\prime }(0){\sigma }_r(0)<0,$$ (H1)

where ϕ(x) = tanh(g_hx + β_h). This gives the familiar g_h < 2 condition when β_r = 0 [81]. Thus, in this case, there is an interplay between g_h and β_r in determining the leading edge of the Jacobian around the zero fixed point and, thus, its stability. In the limit β_r → −∞, the leading edge retreats to $-{\tau }_r^{-1}$. When β_h > 0, zero cannot be a fixed point of the dynamics. Therefore, β_h facilitates the appearance of nonzero fixed points, and both β_r and β_h determine the stability of these nonzero fixed points.

FIG. 12.

The role of biases. (a) FP solutions as a function of increasing β_h; different shades of green correspond to different values of β_r. Dashed lines correspond to FP solutions that are unstable (time-varying states). (b) The leading edge of the spectrum corresponding to the FP solutions calculated in (a); the FP solution is unstable when the leading edge is positive. (c) Similar to (a) but for β_r; different shades of blue correspond to different values of β_h. (d) Similar to (b) but for β_r. (e) FP solutions near critical g_c where the zero FP becomes unstable (circles) compared with the perturbative solution predicted by Eq. (H2) (solid lines). (f) FP solution as a function of β_r and β_h. The orange line indicates the stability line—i.e., regions on top of the orange line correspond to unstable or time-varying states.

To gain some insight into the role of β_h in generating fixed points, we treat the mean-field FP equations [Eq. (C26)] perturbatively around the operating point g_c where the zero fixed point becomes unstable [Eq. (H1)]. For small β_h and ϵ = g_h − g_c, we can express the solution Δ_h as a power series in ϵ, and we see that to leading order the fixed-point variance behaves as (details in Appendix I 1)

$${\Delta }_h\approx \{\begin{array}{ll}\frac{{\beta }_h+ϵ}{g_c^2\left(2-g_c^2{a}_1\right)}\hfill & g_c^2{a}_1<2,\hfill \\ \left(g_c^2{a}_1-2\right)f_1+ϵ\cdot f_2\hfill & g_c^2{a}_1>2,\hfill \end{array}$$ (H2)

$$\text{ where }{a}_1=\frac{{\alpha }_r^2}{16}\left[{\varphi }_0^{(1)}{\left({\beta }_r/2\right)}^2+{\varphi }_0\left({\beta }_r/2\right){\varphi }_0^{(2)}\left({\beta }_r/2\right)\right],$$ (H3)

where ϕ₀ ≡ tanh and f₂(α_r, β_r) and f₂(α_r, β_r) are constant functions with respect to ϵ. Therefore, we see that the bias β_h gives rise to nonzero fixed points near the critical point which scale linearly with the bias. In Fig. 12(e), we show this linear scaling of the solution for the case when β_h = ϵ, and we see that the prediction (lines) matches the true solution (circles) over a reasonably wide range.

More generally, away from the critical g_c, an increasing β_h gives rise to fixed-point solutions with increasing variance, and this can arise continuously from zero, or it can arise by stabilizing an unstable, time-varying state depending on the value of β_r. In Fig. 12(a), we see how the Δ_h behaves for increasing β_h for different β_r, and we can see the stabilizing effect of β_h on unstable solutions by looking at its effect on the leading spectral edge [Fig. 12(b)]. In Fig. 12(c), we see that an increasing β_r also gives rise to increasing Δ_h. However, in this case, it has a destabilizing effect by shifting the leading spectral edge to the right. In particular, when β_h = 0, increasing β_r destabilizes the zero fixed point and give rise to a time-varying solution. We note that, when β_h = 0, varying β_r cannot yield stable nonzero FPs. The combined effect of β_h and β_r can been seen in Fig. 12(f), where the nonzero solutions to the left of the orange line indicate unstable (time-varying) solutions. We choose the parameters to illustrate an interesting aspect of the biases: In some cases, increasing β_h can have a nonmonotonic effect on the stability, wherein the solution becomes unstable with increasing β_h and is then eventually stabilized for sufficiently large β_h.

FIG. 13.

How the biases alter the transition between stability and chaos. (a) Critical lines indicating boundaries for stability (solid lines) or marginal stability (dashed lines) for different values of β_h. (b) Similar to (a) but for different values of β_r. (c),(d) How the boundaries of stability (solid lines) or marginal stability (dashed lines) change as we vary α_r (c) or β_r (d).

1. Effect of biases on the phase boundaries

In Figs. 13(a) and 13(b), we look at how the critical line for the chaotic transition, in the α_r − g_h plane, changes as we vary β_h (a) or β_r (b). Positive values of β_r (“open” output gate) tend to make the transition line less dependent on α_r [Fig. 13(b)], and negative values of β_r have a stabilizing effect by requiring larger values of g_h and α_r to transition to chaos. As we see above, higher values of β_h have a stabilizing effect, requiring higher g_h and α_r to make the (nonzero) stable fixed point unstable. In both cases, the critical lines for marginal stability [Figs. 13(a) and 13(b), dashed lines] are also influenced in a similar way. In Figs. 13(c) and 13(d), we see how the stability-to-chaos transition is affected by α_r (c) and β_r (d). Consistent with the discussion above, larger α_r and β_r have a destabilizing effect, requiring a larger β_h to make the system stable.

APPENDIX I: DETAILS OF THE PERTURBATIVE SOLUTIONS TO THE MEAN-FIELD EQUATIONS

1. Perturbative solutions for the fixed-point variance Δ_h with biases

In this section, we derive the perturbative solutions for the fixed-point variance Δ_h with finite biases, near the critical point where the zero fixed point becomes unstable. Recall that fixed-point variances are obtained by solving

$${\Delta }_z\equiv \langle z^2\rangle =\int 𝒟x\varphi {\left(\sqrt{{\Delta }_h}x\right)}^2={\Delta }_r,$$ (I1)

$${\Delta }_h\equiv \langle h^2\rangle =\int 𝒟x𝒟y\varphi {\left(\sqrt{{\Delta }_h}x\right)}^2{\sigma }_r{\left(\sqrt{{\Delta }_r}y\right)}^2.$$ (I2)

The expansion we seek is perturbative in Δ_h. So, expanding the gating and activating functions about their biases under the assumption ${\Delta }_r\approx g_h^2{\Delta }_h$, we have a series expansion to $O\left({\Delta }_h^2\right)$:

$${\langle {\sigma }_r{\left(\sqrt{{\Delta }_r}x\right)}^2\rangle }_x=a_0+a_1{g}_h^2{\Delta }_h+a_2{g}_h^4{\Delta }_h^2,$$

$$a_0=\frac{1}{4}{\left[1+{\varphi }_0\left({\beta }_r/2\right)\right]}^2,$$ (I3)

$$a_1=\frac{{\alpha }_r^2}{16}\left[{\varphi }_0^{(1)}{\left({\beta }_r/2\right)}^2+{\varphi }_0\left({\beta }_r/2\right){\varphi }_0^{(2)}\left({\beta }_r/2\right)+{\varphi }_0^{(2)}\left({\beta }_r/2\right)\right],$$ (I4)

$$a_2=\frac{{\alpha }_r^4}{256}\left[12{\varphi }_0^{(2)}{\left({\beta }_r/2\right)}^2+4{\varphi }_0\left({\beta }_r/2\right){\varphi }_0^{(4)}\left({\beta }_r/2\right)+16{\varphi }_0^{(1)}\left({\beta }_r/2\right){\varphi }_0^{(3)}\left({\beta }_r/2\right)+{\varphi }_0^{(4)}\left({\beta }_r/2\right)\right],$$ (I5)

where we use the following identities involving the derivatives of tanh:

$${\varphi }_0(x)=\text{tanh}(x),$$ (I6)

$${\varphi }_0^{(1)}(x)=1-{\varphi }_0{(x)}^2,$$ (I7)

$${\varphi }_0^{(2)}(x)=-2{\varphi }_0(x)\left[1-{\varphi }_0{(x)}^2\right],$$ (I8)

$${\varphi }_0^{(3)}(x)=2\left[1-{\varphi }_0{(x)}^2\right]\left[3{\varphi }_0{(x)}^2-1\right],$$ (I9)

$${\varphi }_0^{(4)}(x)=-8{\varphi }_0(x)\left[1-{\varphi }_0{(x)}^2\right]\left[3{\varphi }_0{(x)}^2-2\right].$$ (I10)

This gives us to $O\left({\Delta }_h^2\right)$

$${\Delta }_h\approx \left[c_0+c_1{\Delta }_h+c_2{\Delta }_h^2\right]{\langle {\sigma }_r{\left(\sqrt{{\Delta }_r}x\right)}^2\rangle }_x,$$ (I11)

$$c_0={\varphi }_0{\left({\beta }_h\right)}^2,$$ (I12)

$$c_1=g_h^2\left[{\varphi }_0^{(1)}{\left({\beta }_h\right)}^2+{\varphi }_0^{(2)}\left({\beta }_h\right){\varphi }_0\left({\beta }_h\right)\right],$$ (I13)

$$c_2=g_h^4\left[\frac{1}{4}{\varphi }_0\left({\beta }_h\right){\varphi }_0^{(4)}\left({\beta }_h\right)+{\varphi }_0^{(1)}\left({\beta }_h\right){\varphi }_0^{(3)}\left({\beta }_h\right)+\frac{3}{4}{\varphi }_0^{(2)}{\left({\beta }_h\right)}^2\right],$$ (I14)

and, therefore,

$${\Delta }_h\approx \left(c_0+c_1{\Delta }_h+c_2{\Delta }_h^2\right)\left(a_0+a_1{g}_h^2{\Delta }_h+a_2{g}_h^4{\Delta }_h^2\right).$$ (I15)

To proceed further, we study the solutions to this equation for small deviations for a critical value of g_h. Which critical value should we use? Recall that the zero fixed point becomes unstable when

$$-1+{\varphi }^{\prime }(0){\sigma }_r(0)=0.$$ (I16)

Therefore, we expand around this operating point and our small parameter ϵ = g_h − g_c, where g_c = σ_r(0)⁻¹. We make an ansatz that we can express Δ_h as a power series in ϵ:

$${\Delta }_h={ϵ}^{\eta }\left(d_0+d_1ϵ+d_2{ϵ}^2\right),$$ (I17)

where η is the exponent for the prefactor scaling and needs to be determined self-consistently. To get the scaling relations for Δ_h, we need to expand the coefficients in the Taylor series for Δ_h in terms of ϵ. We note that c₀ = tanh(β_h)², and, therefore, these approximations make sense only for small β_h. How small should β_h be relative to ϵ? We make the following ansatz:

$${\beta }_h={\beta }_0{ϵ}^{\delta },$$ (I18)

and, thus, if δ > 1/2, then $c_0~{\beta }_0^2{ϵ}^{2\delta }$ increases slower than ϵ.

We now express the coefficients for small β_h:

$$c_0\approx {\beta }_0^2{ϵ}^{2\delta },$$ (I19)

$$c_1\approx g_h^2\left(1-2{\beta }_h^2\right),$$ (I20)

$$c_2\approx g_h^4\left(-2+17{\beta }_h^2\right).$$ (I21)

After solving Eqs. (I15)–(I19) self-consistently in terms of the expansion parameter ϵ, we get the following perturbative solution for δ ≤ 1:

$${\Delta }_h\approx \{\begin{array}{ll}\frac{2{\beta }_0{ϵ}^{\delta }}{g_c^2\left(2-g_c^2{a}_1\right)}\hfill & g_c^2{a}_1<2,\hfill \\ \left(g_c^2{a}_1-2\right)f_1+ϵ\cdot f_2\hfill & g_c^2{a}_1>2,\hfill \end{array}$$ (I22)

$$\text{where\hspace{0.17em}}{a}_1=\frac{{\alpha }_r^2}{16}\left[{\varphi }_0^{(1)}{\left({\beta }_r/2\right)}^2+{\varphi }_0\left({\beta }_r/2\right){\varphi }_0^{(2)}\left({\beta }_r/2\right)\right]\text{. }$$ (I23)

f₂(α_r, β_r) and f₂(α_r, β_r) are constant functions (with respect to ϵ). Therefore, we see a linear scaling with the bias β_h.

2. Perturbative solutions for the fixed-point variance Δ_h in the bifurcation region with no biases

The perturbative treatment of the fixed-point solutions in this case closely follows that described above. For g_h = 2 − ϵ, we can express Δ_h as a power series in ϵ (Δ_h = c₀ + c₁ϵ + c₂ϵ²) and look for a condition that allows for a nonzero c₀ corresponding to the bifurcation point. Since we expect, Δ_h to be small in this regime, we can expand Δ_r as

$${\Delta }_r\approx g_h^2{\Delta }_h-2g_h^4{\Delta }_h^2+\frac{17}{3}{g}_h^6{\Delta }_h^3+O\left({\Delta }_h^4\right),$$ (I24)

and, similarly, we can also approximate

$${\langle {\sigma }_r{\left(\sqrt{{\Delta }_r}x\right)}^2\rangle }_x\approx \frac{1}{4}\left[1+\frac{{\alpha }_r^2}{4}{\Delta }_r-\frac{{\alpha }_r^4}{8}{\Delta }_r^2\right].$$ (I25)

Now, equating coefficient of powers of ϵ, we get that either c₀ = 0 or

$$c_0=\frac{3\left({\alpha }_r^2-8\right)}{2\left(-136+24{\alpha }_r^2+3{\alpha }_r^4\right)},$$ (I26)

which is a valid solution when ${\alpha }_r\ge \sqrt{8}$. This is the bifurcation curve limit near g_h = 2⁻.

In the other limit, ${\alpha }_r^{*}\to \infty$ and ${\Delta }_h^{*}\to 0$. We can work in the regime where ${\alpha }_r\sqrt{{\Delta }_h}\gg 1$ to see what values of g_h admit a bifurcation in the perturbative solutions. The equation [to $O\left({\Delta }_h^2\right)$] is given by

$${\Delta }_h\approx \frac{1}{2}\left[g_h^2{\Delta }_h-2g_h^4{\Delta }_h^2\right].$$ (I27)

Thus, we get a positive solution for Δ_h, when $g_h>\sqrt{2}$, and, to the leading order, the solution scales as

$${\Delta }_h^{*}\left({\sqrt{2}}^{+}\right)~\frac{g_h^2-2}{2g_h^4}\text{\hspace{1em} for }{g}_h\to {\sqrt{2}}^{+}.$$ (I28)

3. C_h(τ) near critical point

Here, we study the asymptotic behavior of C_h(τ) near the critical point g_h = 2.0 for small α_z. For simplicity, we set the biases to be zero. In this limit, we can assume that C_h(τ) and C_ϕ(τ) are small. Let us begin by approximating $C_{{\sigma }_z}(\tau )$.

We get, up to $O\left(C_z^3\right)$,

$$C_{{\sigma }_z}(\tau )=g_0+g_1{C}_z(\tau )+g_3{C}_z{(\tau )}^3,$$ (I29)

$$\text{ where }{g}_0=\frac{1}{4}\text{, }$$ (I30)

$$g_1=\frac{{\alpha }_z^2}{16}-\frac{{\alpha }_z^4}{32}{C}_z(0)+\frac{5{\alpha }_z^6}{256}{C}_z{(0)}^2,$$ (I31)

$$g_3=\frac{{\alpha }_z^6}{384}-\frac{{\alpha }_z^8}{192}{C}_z(0).$$ (I32)

This can be obtained, for instance, by expanding σ_z[z(t)] and taking the Gaussian averages over the argument z(t) in the steady state. The relation between C_ϕ(τ) and C_z(τ), in general, does not have a simple form; however, when g_h ~ 2, we expect the relaxation time τ_R ≫ 1, and therefore, we can approximate C_z(τ) ≈ C_ϕ(τ). We can then approximate C_ϕ as

$$C_{\varphi }(\tau )=g_0+g_1{C}_h(\tau )+g_3{C}_h{(\tau )}^3,$$ (I33)

$$\text{where }{g}_0=0\left(\text{ for }{\beta }_h=0\right)\text{, }$$ (I34)

$$g_1=g_h^2-2g_h^4{C}_h(0)+5g_h^6{C}_h{(0)}^2,$$ (I35)

$$g_3=\frac{2}{3}{g}_h^6-\frac{16}{3}{g}_h^8{C}_h(0).$$ (I36)

Note that this also gives us an approximation for C_ϕ(0). Putting all this together, the equation governing C_h(τ),

$$\left[-{\partial }_{\tau }^2+C_{{\sigma }_z}(\tau )\right]C_h(\tau )=\frac{1}{4}{C}_{{\sigma }_z}(\tau )C_{\varphi }(\tau ),$$ (I37)

becomes [up to $O\left(C_h^3\right)$]

$${\partial }_{\tau }^2{C}_h(\tau )\simeq a_1{C}_h(\tau )+a_2{C}_h{(\tau )}^2+a_3{C}_h{(\tau )}^3,$$ (I38)

$$\text{ where }{a}_1=\frac{1}{16}(4-\Gamma )\text{, }$$ (I39)

$$a_2=\frac{{\alpha }_z^2}{64}(4-\Gamma )\Gamma ,$$ (I40)

$$a_3=-\frac{g_h^6}{24},$$ (I41)

$$\Gamma =g_h^2-2g_h^4{C}_h(0)+5g_h^6{C}_h{(0)}^2.$$ (I42)

Integrating with respect to τ gives

$${\left[{\partial }_{\tau }{C}_h(\tau )\right]}^2=2\left(\frac{a_1}{2}{C}_h{(\tau )}^2+\frac{a_2}{3}{C}_h{(\tau )}^3+\frac{a_3}{4}{C}_h{(\tau )}^4+\text{ const }\right).$$ (I43)

The boundary conditions are

$${\partial }_{\tau }{C}_h(0)=0,\underset{\tau \to \infty }{\text{lim}}{\partial }_{\tau }{C}_h(\tau )=0.$$ (I44)

The second condition implies the constant is 0. And the first condition implies

$$\frac{a_1}{2}+\frac{a_2}{3}{C}_h(0)+\frac{a_3}{4}{C}_h{(0)}^2=0.$$ (I45)

From this, we can solve for C_h(0) (neglecting terms higher than quadratic) to get a solution that is perturbative in the deviation ϵ from the critical point (g_h = 2 + ϵ). To the leading order, the variance grows as

$$C_h(0)\approx \frac{1}{8}ϵ+O\left({ϵ}^2\right),$$ (I46)

and the α_z enters the timescale-governing term a₁ only at O(ϵ²). At first, it might seem counterintuitive that α_z, which effectively controls the dynamical time constant in the equations of motion, should not influence the relaxation rate to leading order. However, this result is for the dynamical behavior close to the critical point, where the relaxation time is a scaling function of ϵ. Moving away from this critical point, the relaxation time becomes finite, and the z gate, and, thus, α_z, should have a more visible effect.

APPENDIX J: TOPOLOGICAL COMPLEXITY VIA KAC-RICE FORMULA

The arguments here are similar to those presented in Ref. [82], which use a self-averaging assumption to express the topological complexity (defined below) in terms of a spectral integral. Let us begin.

The goal is to estimate the total number of fixed points for a dynamical system ẋ = G(x). The Kac-Rice analysis proceeds by constructing the integral over the state space x whose integrand has delta-functional support only on the fixed points:

$$𝒩=\int dx\mathbb{E}[\delta (G(x))\left|\mathrm{det\hspace{0.17em}}𝒟\right|],$$ (J1)

where 𝒟 = ∂G/∂x is the instantaneous Jacobian. The expectation value here is over the random coupling matrices. The average number of fixed points is related to the so-called topological complexity 𝒞 via the definition

$$𝒩=\text{exp}(N𝒞).$$

We seek a saddle-point approximation of this quantity below.

For the gated RNN, the state space x = (h, z, r), and the fixed points satisfy

$${\sigma }_z\left(z_i\right)\left(-h_i+{\eta }_i^h\right)=0,$$ (J2)

$$-z_i+{\eta }_i^z=0,$$ (J3)

$$-r_i+{\eta }_i^r=0,$$ (J4)

where for notational shorthand we introduce ${\eta }_i^h={\sum }_j{J}_{ij}^h\varphi \left(h_j\right){\sigma }_r\left(r_j\right)$ and ${\eta }_i^{r/z}={\sum }_j{J}_{ij}^{r/z}\varphi \left(h_j\right)$, anticipating the mean-field approximation to come. Notice that only the first equation for h provides a nontrivial constraint. Once h is found, the second and third equations can be used to determine z and r, respectively. Notice, furthermore, that, since σ(z_i) > 0, the solutions h_i to the first equation do not depend on z_i. Indeed, the dependence on σ(z) can be factorized out of the Kac-Rice integral. This requires noting first that, for the fixed point Jacobian, Eq. (A6) implies that the Jacobian can be written (setting τ_r = τ_z = 1 for simplicity)

$$𝒟=A(-1+𝒥R)$$ (J5)

and that the determinant can be factorized:

$$\det \left|𝒟\right|=\det \left|A\right|\times \det \left|-1+𝒥R\right|$$ (J6)

$$=\left(\prod _i\sigma \left(z_i\right)\right)\det \left|-1+𝒥R\right|.$$ (J7)

The product of σ(z_i) produced by the determinant is canceled by the product of delta functions, using the fact that σ(z_i) > 0 and the transformation law

$$\prod _i\delta \left[\sigma \left(z_i\right)\left(-h_i+{\eta }_i^h\right)\right]=\frac{1}{{\prod }_i\sigma \left(z_i\right)}\prod _i\delta \left[-h_i+{\eta }_i^h\right].$$ (J8)

So we see that what evidently matters for the topological complexity is the fixed-point Jacobian:

$${𝒟}^{fp}=-1+𝒥R,$$ (J9)

whose eigenvalues we denote by λ_i for i = 1, …, N and with the spectral density

$$\widehat{\mu }(z)=\frac{1}{N}\sum _i{\delta }^{(2)}\left(z-{\lambda }_i\right).$$ (J10)

The preceding analysis is all basically to show that we could easily have set α_z = 0 and gotten the same answer; i.e., the z gate does not influence the topological properties of the dynamics. For α_z = ∞, the situation changes drastically, and the analysis likely needs to be significantly reworked. Indeed, in this limit, we most likely do not have discrete fixed points anymore, so the very notion of counting fixed points no longer makes sense.

Having introduced the spectral density, we can rewrite the Kac-Rice integral as

$$𝒩=\int \prod _{x\in \{h,r\}}dx\mathbb{E}\left[{\delta }^{(N)}\left(x-{\eta }^x\right)e^{N\int d^{2}z\widehat{\mu }(z)\text{log}\left|z\right|}\right].$$ (J11)

Note that, since the spectral density of 𝒟^fp is independent of z, the integral over z is trivial to perform and leaves only h and r in the integrand.

So far, everything is exact. We begin now to make some approximations. The first crucial approximation is that the spectral density is self-averaging. The RMT analysis in the previous sections shows us furthermore that the spectral density depends only on macroscopic correlation functions of the state variables. Let us denote the spectral integral factor

$$I(x,𝒥)=\exp \left(N\int d^{2}z\widehat{\mu }(z)\text{\hspace{0.17em}log}|z|\right),$$ (J12)

by which we mean that it depends on the particular realization of the random coupling 𝒥 and the state vector x. The self-averaging assumption implies that

$$I(x,𝒥)\approx \mathbb{E}[I(x,𝒥)]\equiv \overline{I}(x);$$ (J13)

i.e., this factor does not depend on the particular realization of ℐ but just on the state vector. Equivalently, we are assuming that the spectral density ${\widehat{\mu }}_{h,r}(z)$ depends only on the configurations h and r and not the particular realization J^h,r. This allows us to pull this factor outside of the expectation value:

$$𝒩\approx \int \prod _{x\in \left\{h,r\right\}}\overline{I}(x)\mathbb{E}\left[{\delta }^{(N)}\left(x-{\mathit{\eta }}^x\right)\right].$$ (J14)

Now we give some nonrigorous arguments for how one might evaluate the remaining expectation value. In order to carry out the average over J^h and J^r, we utilize the Fourier representation of the delta function to write

$$\mathbb{E}\left[\int d\widehat{x}{e}^{i\widehat{x}\left(x-{\mathit{\eta }}^x\right)}\right]$$ (J15)

$$=\int dxd\widehat{x}\mathbb{E}[\text{exp}\sum _{i,j}\{i{\widehat{h}}_i\left[h_i-J_{ij}^h\varphi \left(h_j\right){\sigma }_r\left(r_j\right)\right]$$ (J16)

$$+i{\widehat{r}}_i\left[r_i-J_{ij}^r\varphi \left(h_j\right)\right]\}],$$

which upon disorder averaging yields

$$\int dxd\widehat{x}\mathrm{\hspace{0.17em}exp}\left\{\sum _i\left(i{\widehat{h}}_i{h}_i+i{\widehat{r}}_i{r}_i-\frac{1}{2}{\widehat{h}}_i^2{\widehat{C}}_{\varphi {\sigma }_r}-\frac{1}{2}{\widehat{r}}_i^2{\widehat{C}}_{\varphi }\right)\right\},$$ (J18)

where we define

$${\widehat{C}}_{\varphi {\sigma }_r}=\frac{1}{N}\sum _i\varphi {\left(h_i\right)}^2{\sigma }_r{\left(r_i\right)}^2,{\widehat{C}}_{\varphi }=\frac{1}{N}\sum _i\varphi {\left(h_i\right)}^2.$$ (J19)

This is where we make our second crucial assumption: that the empirical averages appearing in Eq. (J19) converge to their average value

$${\widehat{C}}_{\varphi {\sigma }_r}\to C_{\varphi {\sigma }_r}={\mathbb{E}}_{h,r}\left[\frac{1}{N}\sum _i\varphi {\left(h_i\right)}^2{\sigma }_r{\left(r_i\right)}^2\right],$$ (J20)

$${\widehat{C}}_{\varphi }\to C_{\varphi }\equiv {\mathbb{E}}_h\left[\frac{1}{N}\sum _i\varphi {\left(h_i\right)}^2\right].$$ (J21)

This means we are assuming the strong law of large numbers. With this essential step, the integral in Eq. (J18) evaluates to

$$\frac{1}{\sqrt{2\pi {\Delta }_h}}\frac{1}{\sqrt{2\pi {\Delta }_r}}\text{exp}\left(-\parallel h{\parallel }^2/2{\Delta }_h-\parallel r{\parallel }^2/2{\Delta }_r\right)$$ (J22)

$$=\prod _{i=1}^N{P}_h\left(h_i\right)P_r\left(r_i\right),$$ (J23)

where ${\Delta }_h=C_{\varphi {\sigma }_r}$ and Δ_r = C_ϕ—which are just the time-independent (fixed-point) MFT equations (C26).

Returning to the expression for the complexity, this series of approximations gives us

$$𝒩\approx \int \prod _{i=1}^{N}d{h}_{i}d{r}_i{P}_h\left(h_i\right)P_r\left(r_i\right)\overline{I}(r,h).$$ (J24)

Let us now describe our derivation more intuitively. We start with the formal expression for the Kac-Rice formula, which uses the delta functional integrand to find fixed points and counts them with the weighting factor related to the Jacobian. Our first assumption allows us to simplify the calculation involving the Jacobian, since we argue that this term is self-averaging. The second assumption allows us to deal with the remaining expectation value of the delta functions. The expectation value adds a number of delta functions (however many there may be for that J^h/r) for each configuration of the connectivity. For continuously distributed connectivity, this implies that the expectation value smears out the delta functions and results in a smooth distribution. What should this distribution be? Well, we know from the mean-field analysis that the state vectors are distributed as Gaussians at a fixed point. Furthermore, the mean-field theory becomes exact for large N. Therefore, we should expect that, in this limit, the delta functions are smeared out into the Gaussian distributions determined by the MFT. This is what our derivation shows.

The final step is to recall that the spectral density depends on the state vectors only via empirical averages. For instance, in the absence of an r gate, the spectral density depends on the empirical average Ĉ_ϕ′. Again invoking the strong law of large numbers, we may argue that the self-averaging goes a step further and that

$$𝒩\approx \int \prod _{i}d{P}_h\left(h_i\right)d{P}_r\left(r_i\right)\text{exp}\left(N\int d^{2}z{\widehat{\mu }}_{h,r}(z)\text{\hspace{0.17em}log}\left|z\right|\right)$$ (J25)

$$\approx \text{exp}\left(N\int d^{2}z\overline{\mu }(z)\text{\hspace{0.17em}log}\left|z\right|\right),$$ (J26)

where

$$\overline{\mu }(z)={\mathbb{E}}_{h,r}\left[{\widehat{\mu }}_{h,r}(z)\right]=\int dhdrP(h)P(r){\widehat{\mu }}_{h,r}(z).$$ (J27)

This is precisely the spectral density we study in a preceding Appendix and the one for which we obtain an explicit expression for the spectral curve. These approximations give us the topological complexity

$$𝒞=\int d^{2}z\overline{\mu }(z)\text{\hspace{0.17em}log}\left|z\right|.$$ (J28)

Now we take a closer look at the spectral density. The eigenvalues of 𝒟^fp form a circular droplet of finite radius ρ and centered on −1. Therefore, the eigenvalues have the form λ = −1 + re^iθ, and the spectral density is a function only of r. The value of the radius is found from Eq. (5) by removing the z gate (i.e., setting α_z = 0). After some algebraic steps, we find for the radius

$${\rho }^2=\frac{1}{2}\left(C_1+\sqrt{C_1^2+4C_2}\right),$$ (J29)

$$C_1=C_{{\varphi }^{\prime }}{C}_{{\sigma }_r},C_2=C_{{\varphi }^{\prime }}{C}_{{\sigma }_r^{\prime }}{C}_{\varphi }.$$ (J30)

Using these facts, we can write the topological complexity as

$$𝒞=\int \mathit{\text{rdrd}}\theta \overline{\mu }(r){\mathbb{I}}_{\{r<\rho \}}\text{\hspace{0.17em}log}\left|r{e}^{i\theta }-1\right|$$ (J31)

$$=\{\begin{array}{ll}2\pi {\int }_1^{\rho }rdr\log r\overline{\mu }(r)\ge 0,\hfill & \text{for }\rho >1,\hfill \\ 0\hfill & \text{for }\rho <1,\hfill \end{array}$$ (J32)

where ${\mathbb{I}}_{\{r<\rho \}}$ is the indicator function which is one for r < ρ and vanishes for r > ρ. Thus, we see that the topological complexity is zero for ρ < 1. This is precisely the fixed-point stability condition derived in the main text [Eq. (6)]. Conversely, the topological complexity is nonzero for ρ > 1, which corresponds to unstable fixed points. Thus, we see, under our set of reasonable approximations, unstable MFT fixed points correspond to a finite topological complexity and, consequently, to a number of “microscopic” fixed points that grows exponentially with N.

The final missing ingredient, necessary to show that region 2 in the phase diagram has an exponentially growing number of fixed points, is to show that the MFT fixed points which appear after the bifurcation are indeed unstable. At the moment, we lack any analytical handle on this. However, we confirm numerically that, along the bifurcation curve, the fixed points are unstable and that increasing the variance Δ_h serves only to increase ρ. However, is it possible for the lower branch, on which Δ_h decreases with α_r? Evidently not, since Δ_h scales with α_r in such a way that $C_{{\sigma }_r^{\prime }}$ ends up growing like ${\alpha }_r^2$, thus once again increasing ρ. Therefore, we conclude that the MFT fixed points appearing after the bifurcation are always unstable, with ρ > 1. This concludes our informal proof of the transition in topological complexity between regions 1 and 2 in the phase diagram in Fig. 7.