Properties of networks with partially structured and partially random connectivity

Abstract

Networks studied in many disciplines, including neuroscience and mathematical biology, have connectivity that may be stochastic about some underlying mean connectivity represented by a nonnormal matrix. Furthermore the stochasticity may not be i.i.d. across elements of the connectivity matrix. More generally, the problem of understanding the behavior of stochastic matrices with nontrivial mean structure and correlations arises in many settings. We address this by characterizing large random N × N matrices of the form A = M + LJR, where M, L and R are arbitrary deterministic matrices and J is a random matrix of zero-mean independent and identically distributed elements. M can be nonnormal, and L and R allow correlations that have separable dependence on row and column indices. We first provide a general formula for the eigenvalue density of A. For A nonnormal, the eigenvalues do not suffice to specify the dynamics induced by A, so we also provide general formulae for the transient evolution of the magnitude of activity and frequency power spectrum in an N -dimensional linear dynamical system with a coupling matrix given by A. These quantities can also be thought of as characterizing the stability and the magnitude of the linear response of a nonlinear network to small perturbations about a fixed point. We derive these formulae and work them out analytically for some examples of M, L and R motivated by neurobiological models. We also argue that the persistence as N → ∞ of a finite number of randomly distributed outlying eigenvalues outside the support of the eigenvalue density of A, as previously observed, arises in regions of the complex plane Ω where there are nonzero singular values of L⁻¹(z1 − M)R⁻¹ (for z ∈ Ω) that vanish as N → ∞. When such singular values do not exist and L and R are equal to the identity, there is a correspondence in the normalized Frobenius norm (but not in the operator norm) between the support of the spectrum of A for J of norm σ and the σ-pseudospectrum of M.

I. INTRODUCTION

Knowledge of the statistics of eigenvalues and eigenvectors of random matrices has applications in the modeling of phenomena relevant to a wide range of disciplines [1–3]. In many applications, however, the matrices of interest are not entirely random, but feature substantial deterministic structure. Furthermore this structure, as well as the disorder on top of it, are in general described by nonnormal matrices.

In neuroscience, for example, connections between neurons typically have restricted spatial range and show specificity with respect to neuronal type, location and response properties. Experience-based synaptic plasticity, which underlies learning and memory, naturally gives rise to synaptic connectivity matrices that encode aspects of the statistical structure of the sensory environment, while containing significant randomness partly due to the inherent stochasticity of particular histories of sensory experience. Another simple example of structured neural connectivity is due to what is known as Dale’s principle [4–6]: neurons come in two main types, excitatory and inhibitory. This empirical principle imposes a certain structure on the synaptic connectivity matrix, forcing all elements in each column of the matrix, describing the synaptic projections of a certain neuron, to have the same sign. Particularly when the typical weight magnitude is much larger than typical differences between the magnitudes of excitatory and inhibitory weights, such a matrix can be extremely nonnormal by some measures, much more so than a fully random matrix [7]. Similarly, biological knowledge imparts a great deal of structure to models of biochemical [8–11] or ecological networks [12–16], and matrices characterizing such interactions are typically nonnormal. Yet our knowledge of connectivity or interactions is at best probabilistic. To describe realistic biological behavior, we must generalize from the behavior of a fixed, regular connectivity to the expected behavior of a typical sample from an appropriate connectivity ensemble.

Furthermore, nonnormality can lead to important dynamical properties not seen for normal matrices [17]. In general, networks with a recurrent connectivity pattern described by a nonnormal matrix can be described as having a hidden feedforward connectivity structure between orthogonal activity patterns, each of which can also excite or inhibit itself [7, 18, 19]. In neural networks such hidden feedforward connectivity arises from the natural separation of excitatory and inhibitory neurons, yielding so-called “balanced amplification” of patterns of activity without any dynamical slowing [7]. Underlying this is the phenomenon of “transient amplification”: a small perturbation from a fixed point of a stable system with nonnormal connectivity can lead to a large transient response over finite time [17]. Transient amplification also yields unexpected results in ecological networks [20–22], and has been conjectured to play a key role in many biochemical systems [23]. Networks that yield long hidden feedforward chains can also generate long time scales and provide a substrate for working memory [18, 19]. Systems with nonnormal connectivity can also exhibit pseudo-resonance frequencies in their power-spectrum at which the system responds strongly to external inputs, even though the external frequency is not close to any of the system’s natural frequencies as determined by its eigenvalues [17].

While Hermitian random matrices and fully random non-Hermitian matrices with zero-mean, independent and identically distributed (iid) elements have been widely studied, there is a shortage of results on quantities of interest for nonnormal matrices that fall in between the two extremes of fully random or fully deterministic. A natural departure from a nonnormal deterministic structure, described by a connectivity matrix M, is to additively perturb it with a fully random matrix J with zero-mean, iid elements. In many important examples, however, the strength of disorder (deviations from the mean structure) is not uniform and itself has some structure (e.g. for each connection it can depend on the types of the connected nodes or neurons). Moreover, the deviations of the strength of different connections or interactions from their average need not be independent. Hence it is important to move beyond a simple iid deviation from the mean structure. Here, we study ensembles of large N × N random matrices of the form A = M + LJR where M, L and R are arbitrary (M) or arbitrary invertible (L and R) deterministic matrices that are in general nonnormal, and J is a completely random matrix with zero-mean iid elements of variance 1/N. The matrix M is thus the average of A, and describes average connectivity. Note that when L and R are diagonal, they specify variances that depend separably on the row and column of A; while when they are not diagonal, the elements of A are not statistically independent. As we show in Sec. II C 3, this form arises naturally, for example, in linearizations of dynamical systems involving simple classes of nonlinearities. This type of ensemble is also natural from the random matrix theory viewpoint, as it describes a classical fully random ensemble – an iid random matrix J – modified by the two basic algebraic operations of matrix multiplication and addition.

We study the eigenvalue distribution of such matrices, but also directly study the dynamics of a linear system of differential equations governed by such matrices. Specifically, for matrices of the above type, using the Feynman diagram technique in the large N limit (we follow the particular version of this method developed by Refs. [24, 25]), we have derived a general formula for the density of their eigenvalues in the complex plane, which generalizes the well-known circular law for fully random matrices [26–30]. It also generalizes a result [31] obtained for the case where L and R are scalar multiples of 1, the N -dimensional identity matrix (the same result was obtained in [32] using the methods and language of free probability theory; the eigenvalue density for the case L ∝ R ∝ 1 and a normal M was also calculated in Ref. [24] in the limit N → ∞, and that result was extended to finite N in Ref. [33]). Apart from generalization to arbitrary invertible L and R, we also provide a correct regularizing procedure for finding the support of the eigenvalue density in the limit N → ∞, in certain highly nonnormal cases of M ; the naive interpretation of the formulae fails in these cases, which were not previously discussed. Furthermore, with the aim of studying dynamical signatures of nonnormal connectivity, we focused on the dynamics directly, deriving general formulae for the magnitude of the response of the system to a delta function pulse of input (which provides a measure of the time-course of potential transient amplification), as well as the frequency power spectrum of the system’s response to external time-dependent inputs.

These general results are presented in the next section. There, we also present the explicit results of analytical or numerical calculations based on these general formulae for some specific examples of M, L and R. Sections III and IV contain the detailed derivations of our general formulae for the eigenvalue density and the response magnitude formulae, respectively. Section V contains the detailed analytical calculations of these quantities for the specific examples presented in Sec. II, based on the general formulae. We conclude the paper in Sec. VI.

II. SUMMARY OF RESULTS

We study ensembles of large N × N random matrices of the form

$$A=M+LJR,$$ (2.1)

where M, L and R are arbitrary (M) or arbitrary invertible (L and R) deterministic matrices [59], and J is a random matrix of independent and identically distributed (iid) elements with zero mean and variance 1/N. Since J and therefore LJR have zero mean, M is the ensemble average of A. The random fluctuations of A around its average are given by the matrix LJR, which for general L and/or R has dependent and non-identically distributed elements, due to the possible mixing and non-uniform scaling of the rows (columns) of the iid J by L (R).

We are firstly interested in the statistics of the eigenvalues of Eq. (2.1). While the statistics of the eigenvalues and eigenvectors of A are of interest in their own right, we also directly consider certain properties of the linear dynamical system

$$\frac{d\mathbf{x}\left(t\right)}{dt}=-\gamma \mathbf{x}\left(t\right)+A\mathbf{x}\left(t\right)+\mathbf{I}\left(t\right),$$ (2.2)

for an N -dimensional state vector x(t), when A is a sample of the ensemble Eq. (2.1). Here, γ is a scalar and I(t) is an external, time-dependent input. In studying this system, we generally assume that Eq. (2.2) is asymptotically stable. This means that M, L, R and γ must be chosen such that for any typical realization of J, no eigenvalue of −γ1 + M + LJR has a positive real part; this can normally be achieved, for example, by choosing a large enough γ > 0.

Using the diagrammatic technique in the non-crossing approximation, which is valid for large N, we have derived general formulae for several useful properties of such matrices involving their eigenvalues and eigenvectors (see Sec. III–IV for the details of the derivations and the definition of the non-crossing approximation). We present these results in in this section. In our derivation of these results, we assume the random J belongs to the complex Ginibre ensemble [26], i.e. the distribution of the elements of J is complex Gaussian. However, we emphasize that universality theorems ensure that, for given M, L and R, the obtained result for the eigenvalue density in the limit N → ∞ will not depend on the exact choice of the distribution of the elements of J, beyond its first two moments, and extend to any iid J (including, e.g., J with real binary or log-normal elements) whose elements have the same first two moments, i.e. zero mean and variance 1/N ; the universality of the eigenvalue density for general M, L and R was established in Ref. [34], following earlier work on the universality of the circular law established and successively strengthened in Refs. [26–30]. Furthermore, empirically, from limited simulations, we have thus far found (but have not proved) such universal behavior to also hold for the other quantities we compute here (however, it is possible that universality for these quantities might require the existence of some higher moments beyond the second, as has been found for universality of certain other properties of random matrices; see e.g. Ref. [35]). To demonstrate the universality of our results, we have used non-Gaussian and/or real J ’s in most of the numerical examples below.

Hereinafter, we adopt the following notations. For any matrix B, we denote its operator norm (its maximum singular value) by ∥B∥ and we define its (normalized) Frobenius norm via

$$\parallel B{\parallel }_{\mathrm{F}}^2\equiv \frac{1}{N}\sum _{ij}\mid B_{ij}{\mid }^2=\frac{1}{N}\mathrm{Tr}\left(B{B}^{†}\right)$$ (2.3)

(equivalently, ∥B∥_F is the root mean square of the singular values of B). For general matrices, A and B,

$$\mathrm{tr}\left(A\right)\equiv \frac{1}{N}\mathrm{Tr}\left(A\right),A^{-†}\equiv {\left(A^{†}\right)}^{-1},\frac{1}{A}\equiv A^{-1},\frac{A}{B}\equiv A{B}^{-1},$$

and when adding a scalar to a matrix, it is implied that the scalar is multiplied by the appropriate identity matrix. We denote the identity matrix in any dimension (deduced from the context) by 1. For a complex variable z = x + iy, the Dirac delta function is defined by δ²(z) ≡ δ(x)δ(y), and we define ${\partial }_{\stackrel{‒}{z}}\equiv \partial ∕\partial \stackrel{‒}{z}=(\partial ∕\partial x+i\partial ∕\partial y)∕2$, and ∂_z ≡ ∂/∂z = (∂/∂x − i∂/∂y)/2. For simplicity, we use the notation f (z) (instead of $f(z,\stackrel{‒}{z})$) for general, nonholomorphic functions on the complex plane. We say a quantity is O(f (N)) (resp. Θ(f (N))) when, for large enough N, the absolute value of that quantity is bounded above (resp. above and below) by a fixed positive multiple of |f (N)|. Finally, we say a quantity is o(f (N)) when its ratio to |f (N)| vanishes as N → ∞.

The only conditions we impose on M, L and R are that ∥M∥_F, ∥L∥_F, ∥R∥_F, ∥L⁻¹MR⁻¹_F and ∥(LR)⁻¹∥ are bounded as N → ∞. We use the bound on ∥(LR)⁻¹∥ in Appendices A and B; the Frobenius norm conditions are assumptions in the universality theorem of Ref. [34] which we use as discussed above. Finally, we assume that for all z ∈ ℂ, the distribution of the eigenvalues of $M_z{M}_z^{†}$, where M_z is defined below in Eq. (2.6), tends to a limit distribution as N → ∞. This last condition simply makes precise the requirement that M, L and R are defined consistently as functions of N, such that a limit spectral density for M + LJR is meaningful; in particular, it does not impose any further limits on the growth of the eigenvalues of $M_z{M}_z^{†}$ with N, beyond the various norm bounds imposed above.

A. Spectral density

1. Summary of results

The density of the eigenvalues of M + LJR in the complex plane for a realization of J (also known as the empirical spectral distribution) is defined by

$${\rho }_J\left(z\right)=\frac{1}{N}\sum _{\alpha }{\delta }^2(z-{\lambda }_{\alpha }),$$ (2.4)

where λ_α are the eigenvalues of M + LJR. It is known [34] that ρ_J (z) is asymptotically self-averaging, in the sense that with probability one ρ_J (z) − ρ(z) converges to zero (in the distributional sense) as N → ∞, where ρ(z) ≡ 〈ρ_J (z)〉_J is the ensemble average of ρ_J (z). Thus for large enough N, any typical realization of J yields an eigenvalue density ρ_J (z) that is arbitrarily close to ρ(z).

Our general result is that for large N, with certain cautions and excluding certain special cases as described below (Eqs. (2.19)–(2.20) and preceding discussion), ρ(z) is nonzero in the region of the complex plane satisfying

$$\mathrm{tr}\left[{\left(M_z{M}_z^{†}\right)}^{-1}\right]\ge 1$$ (2.5)

where we defined

$$M_z\equiv L^{-1}(z-M)R^{-1}.$$ (2.6)

Using the definition Eq. (2.3), we can also express Eq. (2.5) as

$$\parallel R\frac{1}{z-M}L{\parallel }_{\mathrm{F}}\ge 1.$$ (2.7)

Inside this region, ρ(z) is given by

$$\rho \left(z\right)=\frac{1}{\pi }\frac{\partial }{\partial \stackrel{‒}{z}}\mathrm{tr}\left[\frac{{\left(RL\right)}^{-1}{M}_z^{†}}{M_z{M}_z^{†}+g{\left(z\right)}^2}\right],$$ (2.8)

where g(z) is a real, scalar function found by solving

$$\mathrm{tr}\left[\frac{1}{M_z{M}_z^{†}+g^2}\right]=1,$$ (2.9)

for g for each z. As a first example, for the well-known case of M = 0, L = 1, and R = σ1, we have M_z = z/σ and the circular law follows immediately from Eq. (2.5), which yields σ²/|z|² ≥ 1 or |z| ≤ σ for the support, and from Eqs. (2.8)–(2.9) which yield the uniform ρ(z) = 1/(πσ²) within that support. As we noted in the introduction, formulae (2.5)–(2.9) generalize the results of Refs. [31] and [32] for the special case L ∝ R ∝ 1 to arbitrary invertible L and R (the eigenvalue density for the case L ∝ R ∝ 1 and a normal M was also calculated in Ref. [24]).

It is possible and illuminating to express Eqs. (2.7)–(2.9) exclusively in terms of the singular values of M_z, which we denote by s_i(z) (we include possibly vanishing singular values among s_i(z), so that we always have N of them). First, noting that the squared singular values of M_z are the eigenvalues of the Hermitian $M_z{M}_z^{†}$, we can evaluate the trace in Eq. (2.9) in the eigen-basis of the latter matrix, and rewrite this equation as

$$\frac{1}{N}\sum _{i=1}^N\frac{1}{S_i{\left(z\right)}^2+g^2}=1.$$ (2.10)

Similarly, Eq. (2.5) can be equivalently rewritten as

$$\frac{1}{N}\sum _{i=1}^N{S}_i{\left(z\right)}^{-2}\ge 1.$$ (2.11)

As we prove at the end of Sec. III, Eq. (2.8) can also be written in a form that makes it explicit that the dependence of ρ(z) on M, L and R is only through the singular values of M_z and their derivatives with respect to z and $\stackrel{‒}{z}$. We have

$$\rho \left(z\right)=\frac{1}{\pi }{\partial }_{\stackrel{‒}{z}}\left[\frac{1}{N}\sum _{i=1}^N\frac{{\partial }_z\left(s_i{\left(z\right)}^2\right)}{s_i{\left(z\right)}^2+g{\left(z\right)}^2}\right].$$ (2.12)

For the special case of M = 0 and general L and R, our formulas can be simplified considerably. The spectrum is isotropic around the origin in this case, i.e. ρ(z) depends only on r ≡ |z|, and its support is a disk centered at the origin with radius

$$r_0=\parallel RL{\parallel }_{\mathrm{F}}={\left[\frac{1}{N}\sum _{i=1}^N{\sigma }_i^2\right]}^{1∕2},$$ (2.13)

where σ_i are the singular values of RL (this follows from Eq. (2.11) by noting that for M = 0, the singular values of M_z = z(RL)⁻¹ are s_i(z) = |z|/σ_i). Within this support the spectral density is given by

$$\rho \left(r\right)=-\frac{1}{2\pi r}{\partial }_r\left(g{\left(r\right)}^2\right),$$ (2.14)

where g(r)² > 0 is found by solving

$$1=\frac{1}{N}\sum _{i=1}^N\frac{1}{{\sigma }_i^{-2}{r}^2+g{\left(r\right)}^2}.$$ (2.15)

Integrating Eq. (2.14), we see that the proportion of eigenvalues lying a distance larger than r from the origin is, in this case, given by

$$n_{>}\left(r\right)=\{\begin{array}{cc}g{\left(r\right)}^2\hfill & (r<r_0)\hfill \\ 0\hfill & (r\ge r_0).\hfill \end{array}\phantom{\}}$$ (2.16)

In Sec. III we prove that the eigenvalue density, given by Eqs. (2.14)–(2.15), is always a decreasing function of r = |z|, i.e. for r > 0 its derivative with respect to r is strictly negative, as long as the limit distribution of the {σ_i} as N → ∞ has nonzero variance (otherwise ρ(z) is given by the circular law with radius Eq. (2.13)). The values of spectral density at r = 0 and r = r₀ can be calculated explicitly for general L and R:

$$\rho (r=0)=\frac{1}{\pi }\frac{1}{N}\sum _{i=1}^N{\sigma }_i^{-2}$$ (2.17)

$$\rho (r=r_0)=\frac{1}{\pi }{r}_0^2{\left[\frac{1}{N}\sum _{i=1}^N{\sigma }_i^4\right]}^{-1}\le \rho (r=0).$$ (2.18)

As noted above, certain cautions apply in using the above formulae for the eigenvalue density and its boundary ((2.5)–(2.9), or equivalently Eqs. (2.10)–(2.12), and for M = 0, Eqs. (2.14)–(2.15)). We have written these formulas for finite N (assuming it is large). However, the non-crossing approximation used in deriving these formulas is only guaranteed to yield the correct result for the eigenvalue density in the limit, i.e. lim_{N →∞} ρ(z) (see Appendix A); finite-size corrections obtained from Eqs. (2.5)–(2.9) are not in general correct, and o(1) contributions to g(z)² or ρ(z) obtained from Eqs. (2.9) and (2.8) should be discarded.

Furthermore, in general, the correct way of finding the support of lim_{N →∞} ρ(z) using Eq. (2.5) is by setting the left side of the inequality (2.5) to lim g² → 0⁺ lim N → ∞ of the left side of Eq. (2.9), as discussed in Sec. III and Appendix A. However, in writing Eq. (2.5) we have simply set g = 0 in Eq. 2.9, and thus implicitly taken the limit g² → 0⁺ before the N → ∞ limit. To correctly express the support, we must first define the function

$$\begin{array}{cc}\hfill K(g,z)& \equiv \underset{N\to \infty }{\lim }\mathrm{tr}\left[\frac{1}{M_z{M}_z^{†}+g^2}\right]\hfill \\ \hfill & =\underset{N\to \infty }{\lim }\frac{1}{N}\sum _{i=1}^N\left[\frac{1}{s_i{\left(z\right)}^2+g^2}\right]\hfill \end{array}$$ (2.19)

for fixed, strictly positive g, which serves to regularize the denominators in Eq. (2.19) for s_i(z) which are zero or vanishing in the limit N → ∞. The generally correct way of expressing Eq. (2.5) or Eq. (2.11) is then

$$K(0^{+},z)\equiv \underset{g\to 0^{+}}{\lim }K(g,z)\ge 1.$$ (2.20)

Let us denote the the support of lim_{N →∞} ρ(z), given by Eq. (2.20), by S_0⁺ and the region specified by the limit N → ∞ of Eq. (2.5) or Eq. (2.11) by S₀. For many examples of M, L and R, the limits N → ∞ and g → 0⁺ commute everywhere and hence S_0⁺ = S₀. However, if there are z’s at which some of the smallest s_i(z) are either zero or vanish in the limit N → ∞, the two limits may fail to commute, and the naive use of Eq. (2.5) can yield a region, S₀, strictly larger than and containing S_0⁺, the correct support of lim_{N →∞} ρ(z). For example, at z’s for which a Θ(1) number of s_i(z) are zero or o(1), these singular values do not make a contribution to K(g, z) for g > 0 (their contribution to the sum in Eq. (2.19) is O(N ⁻¹)) and hence to K(0⁺, z), but if they vanish sufficiently fast as N → ∞ they can make a nonzero contribution to the left side of Eq. (2.11); such z may fall within S₀, but not within S_0⁺. For finite N, the s_i(z) can vanish exactly when z coincides with an eigenvalue of M ; thus the above situation can, e.g., arise close to eigenvalues of M that are isolated and far from the rest of M ’s spectrum, so that they fall outside the support of lim_{N →∞} ρ(z). In such cases, the spectrum of M + LJR will nonetheless typically also contain isolated eigenvalues (which do not contribute to lim_{N →∞} ρ(z)) with effectively deterministic location, i.e. within o(1) distance of corresponding isolated eigenvalues of M ; examples of this phenomenon, for which S₀ − S_0⁺ is not empty but has zero measure, have been studied in Refs. [36, 37] (for symmetric matrices, outlier eigenvalues corresponding to eigenvalues of the mean matrix were first studied in Ref. [38]). For some choices of M, L and R, however, a more interesting case can arise such that for z in a certain region of the complex plane with nonzero measure, all s_i(z) are nonzero at finite N (hence M has no eigenvalue there), but a few s_i(z) are o(1) and vanish sufficiently fast as N → ∞; in particular when L ∝ R ∝ 1, this can occur for certain highly nonnormal M [60]. In such cases the non-commutation of the two limits can lead to a difference S₀ − S_0⁺ with nonzero measure. In cases we have examined this signifies that there exists a finite, non-vanishing region outside the support of lim_{N →∞} ρ(z) (typically surrounding it) where, although ρ(z) is o(1), it nonetheless converges to zero sufficiently slowly that a Θ(1) number of “outlier” eigenvalues lie there (note that the vast majority of eigenvalues, i.e. (1 − o(1))N of them, lie within the support of the limit density). We will discuss examples of this phenomenon in Sec. II C below; in one of the examples (discussed in Sec. II C 2), the existence of such outlier eigenvalues was first noted in Ref. [39], and their distribution was quantitatively characterized in Ref. [36]. However, the connection between such outlier eigenvalues and nonzero but o(1) singular values of M_z, which arise, e.g., for highly nonnormal M, were not noted before to the best of our knowledge. We have observed in simulations (and also supported by [36]) that the distribution of these outliers remains random as N → ∞, is in general less universal than lim_{N →∞} ρ(z) (e.g. it could depend on the choice of real vs. complex ensembles for J), and its average behavior may not be correctly given by the non-crossing approximation.

2. Relationship to pseudospectra

Finally, we note a remarkable connection between our general result for the support of the spectrum Eq. (2.5) and the notion of pseudospectra, in the case in which the limits g² → 0⁺ and N → ∞ commute (so that Eq. (2.5) correctly describes the support). Pseudospectra are generalizations of eigenvalue spectra, which are particularly useful in the case of nonnormal matrices (see Ref. [17] for a review). The eigenvalue spectrum of matrix M can be thought of as the set of points, z, in the complex plane where (z − M)⁻¹ is singular, i.e. it has infinite norm. Given a fixed choice of matrix norm, ∥·∥, the pseudospectrum of M at level σ, or its “σ-pseudospectrum” in the given norm, is the set of points z for which ∥(z − M)⁻¹∥ ≥ σ⁻¹ (thus as σ → 0 we recover the spectrum). For the specific choice of the operator norm (i.e. when ∥A∥ is taken to be the maximum singular value of A), the σ-pseudospectrum can equivalently be characterized as the set of points, z, for which there exists a matrix perturbation ΔM, with ∥ΔM∥ ≤ σ, such that z is in the eigenvalue spectrum of M + ΔM [17][61]. In words, in the operator norm, the σ-pseudospectrum of M is the set to which its spectrum can be perturbed by adding to it arbitrary perturbations of size σ or smaller.

In our setting we can think of LJR as a perturbation of M. Let us focus on the case where L and R are proportional to the identity, i.e., we have ΔM = σJ, with a positive scalar σ. Our result Eq. (2.7), in this case reads ∥σ(z − M)⁻¹∥_F ≥ 1 or ∥(z − M)⁻¹∥_F ≥ σ⁻¹. In other words, as N → ∞, the spectrum of M + σJ, for an iid random J with zero mean and variance 1/N, is the σ-pseudospectrum of M in the normalized Frobenius norm defined by Eq. 2.3. Interestingly, the perturbation, ΔM = σJ, has normalized Frobenius norm σ as N → ∞: this norm is $\sigma \sqrt{{\sum }_{ij}{J}_{ij}^2∕N}$, which, by the law of large numbers, converges to σ for large N. That is, as N → ∞, the spectrum in response to the random perturbation σJ, which has size σ (in normalized Frobenius norm), is the σ-pseudospectrum of M in the normalized Frobenius norm.

This result sounds similar to the equivalence of the two definitions of pseudospectra for the operator norm which we noted above (one based on the norm of (z − M)⁻¹, and one based on the spectra of bounded perturbations), but it differs in two key respects. First, unlike in the case of the operator norm, the general equivalence of the two notions of pseudospectra noted above does not hold for the normalized Frobenius norm. Second, for the operator norm, it is not in general the case that the σ-pseudospectrum of M is equivalent to the spectrum obtained from a single random perturbation of M of size σ, even in the limit N → ∞ (although the spectra arising from such random perturbations are sometimes used as a “poor man’s version” or approximation of the pseudospectra [17]). This can be seen as follows. The operator norm of the random iid perturbation, σJ, i.e. its maximum singular value, converges almost surely to 2σ as N → ∞ [40]. Condition 2.7 for z to be in the spectrum under this random perturbation is ∥(z − M)⁻¹∥_F ≥ σ⁻¹, or rms({s_i(z)⁻¹}) ≥ 1 where the s_i(z) are the singular values of $\frac{z-M}{\sigma }$ and rms({x_i}) represents the root-mean-square of the set of values {x_i}. This is not equivalent to the condition that z be in the 2σ-pseudospectrum of M in the operator norm, i.e. that $\parallel {(z-M)}^{-1}\parallel \ge {\left(2\sigma \right)}^{-1}$ or s_min (z)⁻¹ ≥ ¹, where s_min(z) is the minimum of the s_i(z); in fact, noting that s_min(z)⁻¹ ≥ rms({s_i(z)⁻¹}), it is easy to see that the spectrum under random iid perturbations with operator norm ∥σJ∥ = 2σ is strictly a proper subset of the 2σ-pseudospectrum in the operator norm. For example, for M = 0, the “poor man’s 2σ-pseudospectrum” in the limit N → ∞ is a ball of radius σ about the origin (the circular law), while the true 2σ-pseudospectra of the zero matrix is the ball of radius 2σ about the origin.

In sum, in the operator norm, the σ-pseudospectrum of M for any N is equivalent to the set of points z for which some perturbation ΔM with ∥ΔM∥ ≤ σ can be found such that z is in the spectrum of M +ΔM [17]. In the normalized Frobenius norm in the limit N → ∞, however, the σ-pseudospectrum of M is equivalent to the spectrum of M + ΔM where ΔM is any random perturbation with zero-mean iid elements with ∥ΔM∥_F = σ. This statement for the normalized Frobenius norm holds when the two limits N → ∞ and g → 0⁺ commute; when the two limits do not commute, the support of the spectral distribution of M + ΔM is a subset of the σ-pseudospectrum of M in the normalized Frobenius norm.

B. Average norm squared and power spectrum

As we mentioned in the introduction, an important phenomenon encountered in dynamics governed by nonnormal matrices, as described by Eq. (2.2) with I(t) = 0, is transient amplification in asymptotically stable systems. In any stable system, the size of the response to an initial perturbation eventually decays to zero, with an asymptotic rate set by the system’s eigenvalues. In stable nonnormal systems, however, after an initial perturbation, the size of the network activity, as measured, e.g., by its norm squared ∥x(t)∥ = x(t)^T x(t), can nonetheless exhibit transient, yet possibly large and long-lasting growth, before it eventually decays to zero. By contrast, in stable normal systems, ∥x(t)∥² can only decrease with time. The strength and even the time scale of transient amplification are set by properties of the matrix A beyond its eigenvalues; they depend on the degree of non-normality of the matrix, as measured, e.g., by the degree of non-orthogonality of its eigenvectors, or alternatively by its hidden feedforward structure (see Eq. (2.34) for the latter’s definition).

Nonnormal systems can also exhibit pseudo-resonances at frequencies that could be very different from their natural frequencies as determined by their eigenvalues; such pseudo-resonances will be manifested in the frequency power spectrum of the response of the system to time dependent inputs. ∥x(t)∥² and the power spectrum of response are examples of quantities that depend not only on the eigenvalues of M + LJR but also on its eigenvectors.

Here, we present a few closely related formulas for general M, L and R. These include a formula for 〈∥x(t)∥²〉_J, i.e. the ensemble average of the norm squared of the state vector, x(t), as it evolves under Eq. (2.2) with I(t) = 0, as well as a formula for the ensemble average of the power spectrum of the response of the network to time-varying inputs. The results of this section are valid, and in the case of the power spectrum meaningful, when the system Eq. (2.2) is asymptotically stable. As we mentioned after Eq. (2.2), this means that M, L, R and γ must be chosen such that for any typical realization of J, all eigenvalues of −γ1+M +LJR have negative real part. In particular, the entire support of the eigenvalue density of M + LJR, as determined by Eq. (2.5), must fall to the left of the vertical line of z’s with real part γ; this is a necessary condition, but may not be sufficient either at finite N or in cases where an O(1) number of eigenvalues remain outside this region of support even as N → ∞.

First, we consider the time evolution of the squared norm, ∥x(t)∥², of the response of the system to an impulse input, I(t) = x₀δ(t), at t = 0, before which we assume the system was in its stable fixed point x = 0 (for t > 0 this is equivalent to the squared norm of the activity as it evolves according to Eq. (2.2) with I(t) = 0, starting from the initial condition x(0) = x₀). We provide a formula for the ensemble average of the more general quadratic function, x(t)^T Bx(t), where B is any N × N symmetric matrix; the norm squared corresponds to B = 1. The result for general B, M, L and R is given as a double inverse Fourier transform

$${\langle \mathbf{x}{\left(t\right)}^{\mathrm{T}}B\mathbf{x}\left(t\right)\rangle }_J=\int \int \frac{d{\omega }_1}{2\pi }\frac{d{\omega }_2}{2\pi }{e}^{it({\omega }_1-{\omega }_2)}\mathrm{Tr}\left[B{C}^{\mathbf{x}}({\omega }_1,{\omega }_2;{\mathbf{x}}_0{\mathbf{x}}_0^{\mathrm{T}})\right],$$ (2.21)

in terms of the N × N Fourier-domain “covariance matrix,” ${\mathit{C}}^{\mathbf{x}}({\omega }_1,{\omega }_2;{\mathbf{x}}_0{\mathbf{x}}_0^{\mathrm{T}})\equiv {\langle \tilde{x}\left({\omega }_1\right)\tilde{x}{\left({\omega }_2\right)}^{†}\rangle }_J$ (where $\tilde{\mathbf{x}}\left(\omega \right)$ is the Fourier transform of x(t)). The expression for the latter is given by

$$C^{\mathbf{x}}({\omega }_1,{\omega }_2;C^{\mathbf{I}})=C_0^{\mathbf{x}}({\omega }_1,{\omega }_2;C^{\mathbf{I}})+\Delta C^{\mathbf{x}}({\omega }_1,{\omega }_2;C^{\mathbf{I}})$$ (2.22)

where

$$C_0^{\mathbf{x}}({\omega }_1,{\omega }_2;C^{\mathbf{I}})\equiv \frac{1}{\gamma +i{\omega }_1-M}{C}^{\mathbf{I}}\frac{1}{\gamma -i{\omega }_2-M^{†}},$$ (2.23)

yields the result obtained by ignoring the randomness in the connectivity (i.e. by setting A = M), and

$$\Delta C^{\mathbf{x}}({\omega }_1,{\omega }_2;C^{\mathbf{I}})\equiv \frac{1}{\gamma +i{\omega }_1-M}L{L}^{†}\frac{1}{\gamma -i{\omega }_2-M^{†}}\times \frac{\mathrm{tr}\left(R^{†}R\frac{1}{\gamma +i{\omega }_1-M}{C}^{\mathbf{I}}\frac{1}{\gamma -i{\omega }_2{M}^{†}}\right)}{1-\mathrm{tr}\left(R^{†}R\frac{1}{\gamma +i{\omega }_1-M}L{L}^{†}\frac{1}{\gamma -i{\omega }_2-M^{†}}\right)},$$ (2.24)

is the contribution of the random part of connectivity LJR. For later use, we have provided these expressions for a general third argument in C^x(·, ·; ·); for use in Eq. (2.21) C^I must be substituted with ${\mathbf{x}}_0{\mathbf{x}}_0^{\mathrm{T}}$. In the special case of 〈∥x(t)∥²〉_J corresponding to B = 1, and iid disorder (L = 1, R = σ1), the contributions from Eqs. (2.23)–(2.24) can be more compactly combined into

$${\langle \parallel \mathbf{x}\left(t\right){\parallel }^2\rangle }_J=\int \int \frac{d{\omega }_1}{2\pi }\frac{d{\omega }_2}{2\pi }{e}^{it({\omega }_1-{\omega }_2)}\frac{{\mathbf{x}}_0^{\mathrm{T}}\frac{1}{\gamma -i{\omega }_2-M^{†}}\frac{1}{\gamma +i{\omega }_1-M}{\mathbf{x}}_0}{1-{\sigma }^2\mathrm{tr}\left(\frac{1}{\gamma +i{\omega }_1-M}\frac{1}{\gamma -i{\omega }_2-M^{†}}\right)}$$ (2.25)

(we used $\mathrm{Tr}\left(\frac{1}{z_1-M}{\mathbf{x}}_0{\mathbf{x}}_0^{\mathrm{T}}\frac{1}{z_2-M^{†}}\right)={\mathbf{x}}_0^{\mathrm{T}}\frac{1}{z_2-M^{†}}\frac{1}{z_1-M}{\mathbf{x}}_0$ write the numerator in Eq. (2.25)).

Next, we look at the power spectrum of the response of the system to a noisy input, I(t), that is temporally white, with zero mean and covariance

$$\overline{{\mathrm{I}}_i\left(t_1\right)I_j\left(t_2\right)}=\delta (t_1-t_2)C_{ij}^{\mathbf{I}}.$$ (2.26)

Here the bar indicates averaging over the input noise (or by ergodicity, over a long enough time). Our general result for the ensemble average of the matrix power spectrum of the response, which by definition is the Fourier transform of the steady-state response covariance,

$$C_{ij}^{\mathbf{x}}\left(\omega \right)\equiv \int d\tau e^{-i\omega \tau }\overline{{\mathbf{x}}_i(t+\tau ){\mathbf{x}}_j\left(t\right)},$$ (2.27)

is given by

$${\langle C^{\mathbf{x}}\left(\omega \right)\rangle }_J=C_0^{\mathbf{x}}\left(\omega \right)+\Delta C^{\mathbf{x}}\left(\omega \right).$$ (2.28)

Here we defined

$$C_0^{\mathbf{x}}\left(\omega \right)\equiv C_0^{\mathbf{x}}(\omega ,\omega ;C^{\mathbf{I}})$$ (2.29)

and

$$\Delta C^{\mathbf{x}}\left(\omega \right)\equiv \Delta C^{\mathbf{x}}(\omega ,\omega ;C^{\mathbf{I}})$$ (2.30)

are the power spectrum matrices obtained by ignoring the randomness in connectivity (i.e. by setting A = M), and the contribution of quenched randomness LJR, respectively.

A closely related quantity is the total power of the steady-state response of the system to a sinusoidal input $\mathbf{I}\left(t\right)={\mathbf{I}}_0\sqrt{2}\cos \omega t$ (the $\sqrt{2}$ serves to normalize the average power of $\sqrt{2}$ cos ωt to unity, so that the total power in the input is ∥I₀∥²). For such an input, the steady-state activity, which we denote by x_ω (t), is also sinusoidal (with a possible phase shift). By total power of the steady-state response we mean the time average of the squared norm of the activity, $\overline{\parallel {\mathbf{x}}_{\omega }\left(t\right){\parallel }^2}$, where now the bar indicates temporal averaging (we call this total power, because the squared norm sums the power in all components of x_ω (t)). As in Eqs. (2.21)–(2.24), we present a formula for the ensemble average of the more general quantity $\overline{{\mathbf{x}}_{\omega }^{\mathrm{T}}B{\mathbf{x}}_{\omega }}$. We have

$${\langle \overline{{\mathbf{x}}_{\omega }^{\mathrm{T}}B{\mathbf{x}}_{\omega }}\rangle }_J=\mathrm{Tr}\left(B{\langle C^{\mathbf{x}}\left(\omega \right)\rangle }_J\right),$$ (2.31)

where 〈C^x(ω)〉_J is given by Eqs. (2.28)–(2.30) with C^I replaced by ${\mathbf{I}}_0{\mathbf{I}}_0^{\mathrm{T}}$. For the special case of B = 1, corresponding to the total power of the response at frequency ω, using Eq. (2.23)–(2.24) with ω₁ = ω₂ = ω, this formula can be simplified into

$${\langle \overline{\parallel {\mathbf{X}}_{\omega }{\parallel }^2}\rangle }_J=\parallel \frac{1}{z-M}{\mathbf{I}}_0{\parallel }^2+\frac{\parallel \frac{1}{z-M}L{\parallel }_{\mathrm{F}}^2\parallel R\frac{1}{z-M}{\mathbf{I}}_0{\parallel }^2}{1-\parallel R\frac{1}{z-M}L{\parallel }_{\mathrm{F}}^2},$$ (2.32)

where z = γ + iω, ∥ · ∥ denotes the vector norm, and ∥ · ∥_F denotes the Frobenius norm defined in Eq. (2.3). Finally, for the case that the random part of the matrix is iid, i.e. L = σ1 and R = 1, we can further simplify Eq. (2.32) into

$${\langle \overline{\parallel {\mathbf{x}}_{\omega }{\parallel }^2}\rangle }_J=\frac{\parallel {(\gamma +i\omega -M)}^{-1}{\mathbf{I}}_0{\parallel }^2}{1-{\sigma }^2\parallel {(\gamma +i\omega -M)}^{-1}{\parallel }_{\mathrm{F}}^2}.$$ (2.33)

The stability of the x = 0 fixed point guarantees the positivity of the expressions Eqs. (2.32)–(2.33) for the power spectrum. This is true because, as we noted above, stability requires that the support of the eigenvalue density of A is entirely to the left of the vertical line Re(z) = γ. By our result Eq. (2.7) for that support, this can only be true if the denominators of the last terms in Eq. (2.32)–(2.33) are positive, which guarantees the positivity of the full expressions.

Note that the first term in Eq. (2.32) and the numerator in Eq. (2.33) represent the power spectrum in the absence of randomness, i.e. if A, in Eq. (2.2) is replaced with M. Thus, formulae (2.32)–(2.33) show that the correct average power spectrum is always strictly larger than the naive power spectrum obtained by assuming that random effects will “average out”. Furthermore, due to the denominators of the last terms in Eqs. (2.32)–(2.33), the power spectrum will be larger for frequencies where the support of the eigenvalue density, Eq. (2.7), is closer to the vertical line with Re(z) = γ. Similar, but less precise statements can also be made about the strength of transient amplification using formulae (2.21)–(2.25) for the squared norm of the impulse response. One measure of the strength of transient amplification up to time T is ${\int }_0^T\parallel \mathbf{x}\left(t\right){\parallel }^{2}dt$. Integrating formulae Eq. (2.21) (with B = 1) or Eq. (2.25) over t, one obtains formulae for ${\int }_0^T\parallel \mathbf{x}\left(t\right){\parallel }^{2}dt$ that are the same as Eqs. (2.21)–(2.25), except for the factor e^{it(ω₁ −ω₂}) in the integrands of Eqs. (2.21) and (2.25) being replaced by $\frac{i[1-e^{iT({\omega }_1-{\omega }_2+iϵ)}]}{{\omega }_1-{\omega }_2+iϵ}$ (with ϵ → 0⁺). Due to the denominator in this factor (for T sufficiently large the numerator is constant), the main contribution to the integrals over ω₁ and ω₂ should typically arise for ω₁ ≈ ω₂. On the other hand, note that for ω₁ = ω₂ the denominators in Eqs. (2.24)–(2.25) reduce to the those in Eqs. (2.32)–(2.33), with the connection to the support of the spectral density noted above. Thus this dominant contribution to ${\int }_0^T\parallel \mathbf{x}\left(t\right){\parallel }^{2}dt$ must be larger, the closer the support of the eigenvalue density, Eq. (2.7), is to the vertical line with Re(z) = γ. This also suggests that, as in the case of the power spectrum, the strength of transient amplification would typically be underestimated if randomness of connectivity is ignored and only its ensemble average M is taken into account in solving Eq. (2.2).

Numerical simulations indicate that the quantities $\parallel \mathbf{x}\left(t\right){\parallel }^2$ and $\overline{\parallel {\mathbf{x}}_{\omega }{\parallel }^2}$ are self-averaging in the large N limit; that is, for large $\parallel \mathbf{x}\left(t\right){\parallel }^2$ or $\overline{\parallel {\mathbf{x}}_{\omega }{\parallel }^2}$ for any typical random realization of J will be very close to their ensemble averages, given by Eq. (2.25) and Eq. (2.32) respectively, with the random deviations from these averages approaching zero as N goes to infinity (see Figs. 1, 3 and 8, below). This conclusion is also corroborated by rough estimations (not shown) based on Feynman diagrams (the diagrammatic method is introduced in Secs. III and IV) of the variance of fluctuations of these quantities for different realizations of J.

FIG. 1.

(Color online) Top panel: the total power spectrum of steady state response $\overline{\parallel {\mathbf{x}}_{\omega }{\parallel }^2}$ as a function of input frequency ω, Eq. (2.33), for the system Eq. (2.2) with A = M + σJ, and M given by Eq. (2.35) with w = 1 and λ_n = ±i (with +i and −i alternating), respectively. Here, N = 700, σ = 0.5, and γ = 0.8. The input was fed into the last component of x (the beginning of the feedforward chain characterized by Eq. (2.35)), which for the matrix M has natural frequency -1. That is, the input was ${\mathbf{I}}_0\sqrt{2}\cos \omega t$ where I₀ was 1 for the last component and 0 for all other components. The green (thick dashed) curve is the ensemble average of the total power spectrum, ${\langle \overline{\parallel {\mathbf{x}}_w{\parallel }^2}\rangle }_J$, calculated numerically using the general formula Eq. (2.33), which is compared with an empirical average over 100 realizations of real Gaussian J (solid red line, mostly covered by the dashed green line). The pink (light gray) area shows the standard deviation among these 100 realizations around this average. The blue (thin) line shows the result when disorder, σJ, is ignored, i.e. A is replaced by its ensemble average M. Bottom panel: the eigenvalue spectrum of M + σJ (black dots). Red big dots at ±i show the eigenvalues of M. The red curve is the outer boundary of the eigenvalue spectrum of A as computed numerically using Eq. (2.5). The real and imaginary axes of the complex plane are interchanged, so that the frequency axis in the top panel can be matched with the imaginary part of the eigenvalues, i.e. the natural frequencies of Eq. (2.2).

Finally, we note that the general formulae presented in this section are valid only for cases where the initial condition, x₀, or the input structure, I₀ or C^I, are chosen independently of the particular realization of the random matrix J (e.g., cases where x₀ is itself random but independent of J, or when x₀ is chosen based on properties of M, L or R). In particular, our results do not apply to cases in which the initial condition or the input is tailored or optimized for the particular realization of the quenched randomness, J, in which case the true result could be significantly different from those given by the formulae of this section.

C. Some specific examples of M, R and L

In this section we present the results of explicit calculations of the eigenvalue density Eq. (2.8), the average squared norm of response to impulse Eqs. (2.21) and (2.25), and the total power in response to sinusoidal input Eq. (2.33), for specific examples of M, L and R (the details of the calculations for the results presented here can be found in Sec. V). For many of the examples presented here, L and R are both proportional to the identity matrix; thus in these examples the full matrix is of the form M + σJ where σ > 0 determines the strength of disorder in the matrix. In Secs. II C 2 and II C 3, we also present examples with nontrivial L and/or R.

Any matrix, M, can be turned into an upper-triangular form by a unitary transformation, i.e.

$$M=UT{U}^{†},$$ (2.34)

where U is unitary and T is upper-triangular (i.e. T_ij = 0 if i > j) with its main diagonal consisting of the eigenvalues of M. The difference between nonnormal and normal matrices is that for the latter, T can be taken to be strictly diagonal. Equation (2.34) is referred to as a Schur decomposition of M [41], and we refer to the orthogonal modes of activity represented by the columns of U as Schur modes. The Schur decomposition provides an intuitive way of characterizing the dynamical system Eq. (2.2). Rewriting Eq. (2.2), with J and I(t) set to zero, in the Schur basis by defining y = U ^†x (i.e. y_i is the activity in the i-th Schur mode), we obtain $\frac{d\mathbf{y}}{dt}$ = −γy+T y. We see that activity in the j-th Schur mode provides an input to the equation for the i-th mode only when i ≤ j (as T_ij = 0 for i > j). Thus the coupling between modes is feedforward, going only from higher modes to lower ones, without any feedback. We refer to T_ij ’s for j > i as feedforward weights. As these vanish for normal matrices, we can say a matrix is more nonnormal the stronger its feedforward weights are.

Due to the invariance of the trace, the norm, and the adjoint operation under unitary transforms, our general formulae for the spectral density Eq. (2.8) and the average squared norm in time and frequency space, Eqs. (2.25) and (2.33), take the same form in any basis, so in particular we can work in the Schur basis of M. Hence M can be replaced by T, provided L and R are also expressed in M ’s Schur basis and x₀ or I₀ are replaced by U ^†x₀ or U ^†I₀, respectively [62]. Thus we use the feed-forward structure of the Schur decomposition to characterize the different examples we consider below. Our examples are chosen to demonstrate interesting features of nonnormal matrices in the simplest possible settings.

1. Single feedforward chain of length N

In the first example, each and every Schur mode is only connected to its lower adjacent mode, forming a long feedforward chain of length N. For simplicity, we take all feedforward weights in this chain to have the same value w, so that

$$M=T=\left(\begin{array}{cccc}\hfill {\lambda }_1\hfill & \hfill w\hfill & \hfill 0\hfill & \hfill \cdots \hfill \\ \hfill 0\hfill & \hfill {\lambda }_2\hfill & \hfill w\hfill & \hfill \cdots \hfill \\ \hfill \cdots \hfill & \hfill \cdots \hfill & \hfill \cdots \hfill & \hfill \ddots \hfill \end{array}\right)$$ (2.35)

or more succinctly M_nm = w δ_n+1,m + λ_nδ_nm.

Figure 1 shows the power spectrum of response (top panel) and the eigenvalue distribution (bottom panel) of A = M + σJ for an example M of the form Eq. (2.35) with alternating imaginary eigenvalues, λ_n = (−1)ⁿ⁺¹i. The black dots in the bottom panel of Fig. 1 show the eigenvalues of A for one realization of J, scattered around the highly degenerate spectrum of M at ±i (red dots). The top panel shows the ensemble average of the total power spectrum of response, ${\langle \overline{\parallel {\mathbf{x}}_w{\parallel }^2}\rangle }_J$, of the system Eq. (2.2) to sinusoidal stimuli as given by our general formula Eq. (2.33) (green curve), showing that it perfectly matches the empirical average (red curve) over a set of 100 realizations of J (the latter was obtained by generating 100 realizations of J, calculating $\overline{\parallel {\mathbf{x}}_{\omega }{\parallel }^2}$ for each realization, which is given by the numerator of Eq. (2.33) with M replaced by M + σJ, and then averaging the results over the 100 realizations). The pink (light gray) shading shows the standard deviation of the power spectrum over these 100 realizations. This will shrink to zero as N goes to infinity, so that for large N the power spectrum of any single realization of A = M + σJ will lie very close to the ensemble average. The system (2.2) in the zero disorder case, σ = 0, has two highly degenerate resonant frequencies (imaginary parts of the eigenvalues of ${\omega }_0^{\pm }=\pm 1$, leading to possible peaks in the power spectrum at these frequencies. The smaller the decay of these modes (in this case given by γ) is, i.e. the closer the eigenvalues of the combined matrix −γ + M are to the imaginary axis, the sharper and stronger are the resonances. Comparing the zero disorder power spectrum (blue curve) with that for A = M + σJ, we see that the disorder has led to strong but unequal amplification of the two resonances relative to the case without disorder. This is partly due to the disorder scattering some of the eigenvalues of −γ + A much closer to the imaginary axis, creating larger resonances.

For M of the form (2.35) with all eigenvalues zero we have analytically calculated the eigenvalue density, Eq. (2.8), the magnitude of response to impulse Eq. (2.25), and the power-spectrum Eq. (2.33). In this case, using Eq. (2.5) naively yields $\mid z\mid \le \sqrt{\mid w{\mid }^2+{\sigma }^2}$ for the support of the eigenvalue density. However, using the correct procedure, Eqs. (2.19)–(2.20), we find that this formula is only correct for σ ≥ |w|, while for σ < |w|, the true support of the eigenvalue density in the limit N → ∞ is the annulus

$$\sqrt{\mid w{\mid }^2-{\sigma }^2}\le \mid z\mid \le \sqrt{\mid w{\mid }^2+{\sigma }_2},$$ (2.36)

(this result was obtained in Ref. [31]). Within this support the eigenvalue density in either case is

$$\rho \left(z\right)=\frac{1}{\pi {\sigma }^2}\left[1-\frac{\mid w{\mid }^2}{\sqrt{4\mid w{\mid }^2\mid z{\mid }^2+{\sigma }^4}}\right].$$ (2.37)

Figure 2 demonstrates the close agreement of Eqs. (2.36)–(2.37) with the empirical spectrum of M + σJ for a single realization of J, for N = 2000 and two different values of σ. The discrepancy between the results obtained by the naive use of Eq. (2.5) and Eq. (2.36) is due to the fact that for |z| < |w|, M_z = (z − M)/σ has an exponentially small, O(e^−cN), singular value (see next paragraph) which makes the result of Eqs. (2.19)–(2.20) dependent on the order of the two limits N → ∞ and g → 0⁺. As we discussed after Eq. (2.12), such a discrepancy can signify the existence of an O(1) number of outlier eigenvalues outside the support of lim_{N →∞} ρ(z). Simulations show that this is the case for $\mid z\mid <\sqrt{\mid w{\mid }^2-{\sigma }^2}$ (see Fig. 2).

FIG. 2.

(Color online) The eigenvalue spectra of A = M + σJ for N = 2000 and M given by Eq. (2.35) with λ_n = 0, w = 1 for single realizations of real Gaussian J. σ = 0.95 and 0.5 in the left and rights panels, respectively. The red circles mark the circular boundaries of the spectral support given by Eq. (2.36). The insets show a comparison of the analytic formula Eq. (2.37) for the spectral density (black smooth trace) and histograms corresponding to the particular realization shown in the main plot (red jagged trace).

The most striking aspect of these results is revealed in the limit σ → 0. For σ = 0, the spectrum is that of M, which is concentrated at the origin. Remarkably, however, as seen from Eqs. (2.36)–(2.37), for very small but nonzero σ the bulk of the eigenvalues are concentrated in the narrow ring with modulus |z| ≈ |w|. Thus in the limit N → ∞ the spectrum has a discontinuous jump at σ = 0. This is a consequence of the extreme non-normality of M, which manifests itself in the extreme sensitivity of its spectrum to small perturbations, which is well-known (see Ref. [17], Ch. 7). The notion of pseudospectra quantifies this sensitivity: the (operator norm) ϵ-pseudospectrum of M is the region of complex plane to which its spectrum can be perturbed by adding to M a matrix of operator norm no larger than ϵ. As we mentioned in Sec. II A, this is precisely the set of complex values z for which ∥(z − M)⁻¹∥ > ϵ⁻¹ [17], and therefore by the definition of the operator norm ∥ · ∥, the region in which ∥(z − M)⁻¹∥⁻¹ = s_min(z − M) < ϵ, where s_min(z − M) is the least singular value of z − M. As noted above, for |z| < |w|, s_min(z − M) is exponentially small: $s_{min}(z-M)\le \mid w\parallel \frac{z}{w}{\mid }^N$ (for a proof see after Eq. (5.15) in Sec. V A). Thus the ϵ-pseudospectrum of M contains the set of points z satisfying $\mid w\mid \mid \frac{z}{w}{\mid }^N<ϵ$, i.e. the centered disk with radius $\mid w\mid {\left(\frac{ϵ}{\mid w\mid }\right)}^{1∕N}$ which approaches |w| as N → ∞. In other words, for large enough N, any point |z| < |w| is in the ϵ-pseudospectrum for any fixed ϵ, no matter how small. It has been stated [17] that dense random perturbations, of the form σJ considered here, tend to trace out the entire ϵ-psuedospectrum (where ϵ = σ∥J∥ ≈ 2σ). Our result shows that, for ϵ, σ « |w|, the spectrum of such perturbations traces out the ϵ-psuedospectrum in quite an uneven fashion; the vast majority (Θ(N)) of the perturbed eigenvalues only trace out the boundary of the pseudospectrum, |z| ≈ |w|, while only a few (O(1)) eigenvalues lie in its interior. Thus, dense random perturbations can fail as a way of visualizing (operator norm based) pseudospectra.

We now turn to the dynamics. We have explicitly calculated the average evolution of the magnitude of x(t), Eq. (2.25), and the total power spectrum of steadystate response, Eq. (2.33), for the case where the initial condition is (or the input is fed into) the last Schur mode, i.e. the beginning of the feedforward chain: x₀ = (0, · · · , 0, 1)^T ( or I₀ ∝ (0, · · · , 0, 1)^T). For the evolution of the average norm squared, with the initial condition x₀ = (0, … , 0, 1)^T, we obtain

$${\langle \parallel \mathbf{x}\left(t\right){\parallel }^2\rangle }_J=e^{-2\gamma t}{I}_0\left(2t\sqrt{\mid w{\mid }^2+{\sigma }^2}\right),(t\ge 0)$$ (2.38)

where I_ν (x) is the ν-th modified Bessel function. Figure 3 plots the function Eq. (2.38) and compares it with the result obtained by ignoring the disorder (corresponding to σ = 0). The main difference between the two curves is the slower asymptotic decay of the σ ≠ 0 result (green) compared with the zero-disorder case (purple). This is the result of the disorder spreading some of the eigenvalues of −γ + A closer to the imaginary axis, creating modes with smaller decay. Importantly, in neither case do we see transient amplification. By contrast, in the σ = 0 and for small enough decay, i.e. for γ < |w|, the system Eq. (2.2) exhibits very strong transient amplification. In this case, starting from the initial condition x₀ = (0, · · · , 0, 1)^T, the solution for the (N − n)-th Schur component is $x_{N-n}\left(t\right)=\frac{{\left(wt\right)}^n}{n!}{e}^{-\gamma t}$ (for 0 ≤ n ≤ N − 1), which is maximized at t = n/γ with a value $\max \mid x_{N-n}\mid ~{\left(\frac{\mid w\mid }{r}\right)}^n$ for n » 1. Thus up to time t ~ N/γ the norm of the activity grows exponentially; $\parallel \mathbf{x}\left(t\right){\parallel }^2\gtrsim {\left(\frac{\mid w\mid }{\gamma }\right)}^{2\gamma t}\mathrm{for}t\lesssim N∕\gamma$ For larger times the activity reaches the end of the N -long feedforward chain and starts decaying to zero; asymptotically $\parallel \mathbf{x}\left(t\right){\parallel }^2~e^{-2\gamma t}$ for t » N/γ. However, as we have seen, the spectrum of M is extremely sensitive to perturbations; even for very small but nonzero σ, the spectrum of −γ1+A has eigenvalues with real part as large as |w|−γ. Therefore, in the limit N → ∞, the system Eq. (2.2) is unstable for |w| > γ, as soon as σ ≠ 0. Conversely, in the presence of disorder (even infinitesimally small disorder in the N → ∞ limit), as long as the system is stable (which from Eq. (5.15) requires $\gamma >\sqrt{\mid w{\mid }^2+{\sigma }^2}$ it exhibits no transient amplification for the initial condition along the last Schur mode. Let us note, however, that as we mentioned after Eq. (2.33), Eq. (2.25) and hence Eq. (2.38) do not yield the correct answer when the direction of the impulse is optimized for the specific realization of the quenched disorder J ; such disorder-tuned initial conditions can yield significant transient amplification even for the stable σ ≠ 0 system.

FIG. 3.

(Color online) The norm squared of the response to impulse, ∥x(t)∥², of the system Eq. (2.2), for A = M + σJ, with binary J, and M given by Eq. (2.35) (with λ_n = 0) describing a N-long feedforward chain with uniform weights w. Here, w = 1, σ = 0.5, $\gamma =1.005\sqrt{{\sigma }^2+w^2}\simeq 1.124$, and N = 700. The green (thick dashed) curve shows our result, Eq. (2.38), for the average squared impulse response, 〈∥x(t)∥²>_J, which lies on top of the red (thick solid) curve showing the empirical average of ∥x(t)∥² over 100 realizations of binary J. The five thin dashed black curves show the result for five particular realizations of J, and the pink (light gray) area shows the standard deviation among the 100 realizations. The standard deviation shrinks to zero as N → ∞, and ∥x(t)∥² for any realization lies close to its average for large N. For comparison the purple (thin, lowest) curve shows ∥x(t)∥² obtained by ignoring the effect of quenched disorder, i.e. by setting A = M.

Incidentally, we can also read the result for M = 0 from Eq. (2.38), by setting w = 0, obtaining 〈∥x(t)∥²〉_J = e_−2γtI₀(2σt). Since in this case, all directions are equivalent, this is the answer for the (normalized) initial condition along any direction, again as long as the direction is chosen independently of the specific realization of J.

Finally, the total power of response to a sinusoidal input with amplitude I₀ = (0, · · · , 0, I₀)^T is given by

$${\langle \overline{\parallel {\mathbf{x}}_{\omega }{\parallel }^2}\rangle }_J=\frac{\parallel {\mathbf{I}}_0{\parallel }^2}{{\omega }^2+{\gamma }^2-\mid w{\mid }^2-{\sigma }^2}.$$ (2.39)

The main effect of the disorder is to reduce the width of the resonance (the peak of ${\langle \overline{\parallel {\mathbf{x}}_w{\parallel }^2}\rangle }_J$ at ω = 0) and increase its height. This is partly a consequence of the scattering of the eigenvalues of −γ +A closer to the imaginary line by the disorder, creating modes with smaller decay.

2. Examples motivated by Dale’s law: 1 or N/2 feedforward chains of length 2

In this section we consider examples motivated by Dale’s law [4–6] in neurobiology. Dale’s law is the observation (which holds generally but with some exceptions [42, 43]) that individual neurons release the same neurotransmitter at all of their synapses. In the context of many theoretical papers including this one, it refers more specifically to the fact that an individual neuron either makes only excitatory synapses or only inhibitory synapses; that is, each column of the synaptic connectivity matrix has a fixed sign, positive for excitatory neurons and negative for inhibitory ones. We will first consider two examples of connectivity matrices respecting Dale’s law which take the form Eq. (2.1) with L = σ⁻¹R = 1, and a scalar σ. At the end of this subsection we consider an example with nontrivial L and R.

In the first example, we consider a matrix M, which as we will show, has a Schur form T that is composed of N/2 disjoint feedforward chains, each connecting only two modes (we assume N is even). For simplicity we will focus on the case where all eigenvalues are zero. Thus in the Schur basis we have

$$T=\left(\begin{array}{ccccc}\hfill 0\hfill & \hfill w_1\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill \cdots \hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill \cdots \hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill w_2\hfill & \hfill \cdots \hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill & \hfill \cdots \hfill \\ \hfill \cdots \hfill & \hfill \cdots \hfill & \hfill \cdots \hfill & \hfill \cdots \hfill & \hfill \ddots \hfill \end{array}\right)=W\otimes \left(\begin{array}{cc}\hfill 0\hfill & \hfill 1\hfill \\ \hfill 0\hfill & \hfill 0\hfill \end{array}\right)$$ (2.40)

where we defined W to be the N/2 × N/2 diagonal matrix of Schur weights W = diag(w₁, w₂, …, w_N/2). T in Eq. (2.40) arises as the Schur form of a mean matrix of the form

$$M=\frac{1}{2}=\left(\begin{array}{cc}\hfill K\hfill & \hfill -K\hfill \\ \hfill K\hfill & \hfill -K\hfill \end{array}\right)=\frac{1}{2}\left(\begin{array}{cc}\hfill 1\hfill & \hfill -1\hfill \\ \hfill 1\hfill & \hfill -1\hfill \end{array}\right)\otimes K$$ (2.41)

where K is a normal (but otherwise arbitrary) N/2×N/2 matrix (note that M is nonetheless nonnormal). The feedforward weights in Eq. (2.40) are then the eigenvalue of K. When K has only positive entries, matrices of the form Eq. (2.41) satisfy Dale’s principle, and were studied in Ref. [7], in the context of networks of excitatory and inhibitory neurons. We imagine a grid of N/2 spatial positions, with an excitatory and an inhibitory neuron at each position. ½K, a matrix with positive entries, describes the mean connectivity strength between spatial positions, which is taken to be identical regardless of whether the projecting, or receiving, neuron is excitatory or inhibitory. The sign of the weight, on the other hand, depends on the excitatory or the inhibitory nature of the projecting or presynaptic neuron; the first (last) N/2 columns of M represent the projections of the excitatory (inhibitory) neurons and are positive (negative). Since K is normal it can be diagonalized by a unitary transform: K = EW E^†, where W is as above, and E = (e₁, e₂, …) is the matrix of the orthonormal eigenvectors e_b of K, b = 1, …, N/2 (with eigenvalues w_b). Then transforming to the basis $\left\{\left(\begin{array}{c}\hfill {\mathbf{e}}_1\hfill \\ \hfill 0\hfill \end{array}\right),\left(\begin{array}{c}\hfill 0\hfill \\ \hfill {\mathbf{e}}_1\hfill \end{array}\right),\left(\begin{array}{c}\hfill {\mathbf{e}}_2\hfill \\ \hfill 0\hfill \end{array}\right),\left(\begin{array}{c}\hfill 0\hfill \\ \hfill {\mathbf{e}}_2\hfill \end{array}\right),\cdots ,\left(\begin{array}{c}\hfill {\mathbf{e}}_{N∕2}\hfill \\ \hfill 0\hfill \end{array}\right),\left(\begin{array}{c}\hfill 0\hfill \\ \hfill {\mathbf{e}}_{N∕2}\hfill \end{array}\right)\right\}$ (where 0 represents the N/2-dimensional vector of 0’s) transforms the matrix to being block-diagonal with the 2 × 2 matrices $\frac{1}{2}\left(\begin{array}{c}\hfill w_b-w_b\hfill \\ \hfill w_b-w_b\hfill \end{array}\right)$ , b = 1, …, N/2, along the diagonal. The b^th block becomes $\left(\begin{array}{cc}\hfill 0\hfill & \hfill w_b\hfill \\ \hfill 0\hfill & \hfill 0\hfill \end{array}\right)$ in its Schur basis $\left\{\frac{1}{\sqrt{2}}\left(\begin{array}{c}\hfill {\mathbf{e}}_b\hfill \\ \hfill {\mathbf{e}}_b\hfill \end{array}\right),\frac{1}{\sqrt{2}}\left(\begin{array}{c}\hfill {\mathbf{e}}_b\hfill \\ \hfill -{\mathbf{e}}_b\hfill \end{array}\right)\right\}$ , so the full matrix takes the form Eq. (2.40). Thus, the b-th difference mode $\frac{1}{\sqrt{2}}\left(\begin{array}{c}\hfill {\mathbf{e}}_b\hfill \\ \hfill -{\mathbf{e}}_b\hfill \end{array}\right)$ feeds forward to the b-th sum mode $\frac{1}{\sqrt{2}}\left(\begin{array}{c}\hfill {\mathbf{e}}_b\hfill \\ \hfill {\mathbf{e}}_b\hfill \end{array}\right)$ with weight w_b. This feedforward structure leads to a specific form of nonnormal transient amplification, which the authors of Ref. [7] dubbed “balanced amplification”; small differences in the activity of excitatory and inhibitory modes feedforward to and cause possibly large transients in modes in which the excitatory and inhibitory activities are balanced.

Another interesting example of Dale’s law is that in which M simply captures the differences between the mean inhibitory and mean excitatory synaptic strengths and between the numbers of excitatory and inhibitory neurons, with no other structure assumed (uniform mean connectivity), as studied in Ref. [39]. Thus, all excitatory projections have the same mean ${\mu }_E∕\sqrt{N}$, and all inhibitory ones have the mean $-{\mu }_I∕\sqrt{N}$. If we assume a fraction f of all neurons are excitatory, then we can write M as

$$M={\mathbf{uv}}^{\mathrm{T}}$$ (2.42)

where u = N ^−1/2(1, …, 1)^T is a unit vector, and the vector v has components v_i = µ_E or v_i = −µ_I for i ≤ f N and i > f N, respectively (for f = 1/2 and µ_E = µ_I, Eq. (2.42) is a special case of Eq. (2.41)). The singlerank matrix M has only one non-zero eigenvalue given by $\mathbf{v}\cdot \mathbf{u}=\frac{1}{\sqrt{N}}{\sum }_i{v}_i$, with eigenvector u. The case in which the excitatory and inhibitory weights are balanced on average, in the sense that ${\sum }_i{v}_i=0$, is of particular interest; mathematically it is in a sense the least symmetric and most nonnormal case as v · u = 0. In this case all eigenvalues of M are equal to zero. Furthermore, since in this case u and v are orthogonal, we can readily read off the Schur decomposition of M from Eq. (2.42). The normalized Schur modes are given by u, v/∥v∥ and N −2 other unit vectors spanning the subspace orthogonal to both u and v. All feedforward Schur weights are zero, except for one very large weight, equal to $\parallel \mathrm{v}\parallel \propto \sqrt{N}$, which feeds from v/∥v∥ to u. Thus the Schur representation of M has the form Eq. (2.40) with $w_1=\parallel \mathrm{v}\parallel \equiv \mu \sqrt{N}$ and w_b≠1 = 0, where we defined

$${\mu }^2\equiv \mathrm{tr}\left(M^{†}M\right)=\parallel \mathbf{v}{\parallel }^2∕N=f{\mu }_E^2+(1-f){\mu }_I^2.$$ (2.43)

Note that this is again a case of balanced amplification: differences between excitatory and inhibitory activity, represented by v, feed forward to balanced excitatory and inhibitory activity, represented by u, with a very large weight. In the following we present results only for this balanced case of Eq. (2.42), which as just noted is a special case of Eqs. (2.40).

We start by presenting the results for the eigenvalue density. For general diagonal W in Eq. (2.40) (or equivalently, for general normal K in Eq. (2.41)), the eigenvalue density, ρ(z), of A = M +σJ is isotropic around the origin z = 0, and depends only on r = |z|. The spectral support is a disk centered at the origin. In cases in which all the weights w_b are O(1), the radius of this disk can be found directly from Eq. (2.5), which yields

$$r_0=\sigma {\left[\frac{1}{2}+\sqrt{\frac{1}{4}+\frac{{\langle \mid w_b{\mid }^2\rangle }_b}{2{\sigma }^2}}\right]}^{1∕2}.$$ (2.44)

Here, 〈|w_b|²〉_b is the average of the squared feedforward weights over all blocks of Eq. (2.40); equivalently, (|w_b|²)_b = 2 tr(M ^†M) ≡ 2µ². As long as some w_b are nonzero, r₀ is larger than the radius of the circular law, σ, with the difference an increasing function of (|w_b|²)_b; thus the spreading of the spectrum of M (originally concentrated at the origin) after the random perturbation by σJ, is larger the more nonnormal M is. In cases in which the feedforward weights of some of the 2 × 2 blocks of Eq. (2.40) grow without bound as N → ∞, there is a corresponding singular value of M_z ∝ z − M for every such block which is nonzero for z ≠ 0 but vanishes in the limit, scaling like $~\frac{\mid z{\mid }^2}{\mid w_b\mid }$ where w_b is the unbounded weight of that block (see Eq. (5.37) and its preceding paragraph). (Note that as stated after Eq. (2.3) we assume $\parallel M{\parallel }_{\mathrm{F}}=\mu =\sqrt{\langle \mid w_b{\mid }^2\rangle ∕2}$, so that at most o(N) number of weights can be unbounded, and each can at most scale like O(N ^1/2).) In line with the general discussion after Eq. (2.12), in such cases the naive use of Eq. (2.5) may yield an area larger than the true support of lim_{N →∞} ρ(z); the correct support must be found by using Eqs. (2.19)–(2.20), which in this case can yield a support radius strictly smaller than Eq. (2.44). We have calculated the explicit results for lim_{N →∞} ρ(z) for two specific examples of M with the Schur form Eq. (2.40). The first example belongs to the first case (bounded w_b’s) where lim_{N →∞} ρ(z) is Θ(1) within the entire disk r ≤ r₀, while the second belongs to the second case (unbounded w_b’s) where the limit density is only nonzero in a proper subset of that disk.

In the first example, we take all the Schur weights in Eq. (2.40) to have the same value, which we denote by w. In this case, the eigenvalue density is given by $\rho \left(r\right)=\frac{1}{2\pi r}\frac{\partial n_{<}\left(r\right)}{\partial r}$, where n_<(r) is the proportion of eigenvalues within a distance r from the origin and is given by

$$n_{<}\left(r\right)=\frac{r^2}{{\sigma }^2}\left[1-\frac{\mid w{\mid }^2}{{\sigma }^2+\sqrt{{\sigma }^4+\mid w{\mid }^4+4\mid w{\mid }^2{r}^2}}\right].$$ (2.45)

n_< (r) reaches unity exactly at r = r₀ given by Eq. (2.44), and ρ(r) is Θ(1) for any smaller r. Figure 4 shows the close agreement of Eq. (2.45) with empirical results based on single binary realizations of J, for N as low as 60.

FIG. 4.

(Color online) The eigenvalue spectra of A = M + σJ for a binary J with σ = 0.1 and M given by Eq. (2.41) with K = 1 (corresponding to w_b = 1 for all the diagonal 2 × 2 blocks in Eq. (2.40)). The main panels show the eigenvalues for single realizations of J, with N = 600 (left) and N = 60 (right). The red circles mark the boundaries of the spectral support, Eq. (2.44). Since A is real in this case, its eigenvalues are either exactly real, or come in complex conjugate pairs; the spectrum is symmetric under reflections about the real axis. However, such signatures of the reality of the matrix appear only as subleading corrections to the spectral density ρ(z); they are finite size effects which vanish as N → ∞. The insets show a comparison of the analytic formula Eq. (2.45) (black curve) and the empirical result, based on the eigenvalues of the realizations in the main panels, for the proportion, n_< (r), of eigenvalues lying within a radius r of the origin (red dots). The random fluctuations and the average bias of the empirical n_< (r) are both already small for N = 60, and negligible for N = 600.

The second example is that of the balanced Eq. (2.42) with u · v = 0. As we saw, all w_b are zero in this case except for one very large, unbounded weight $w_1=\mu \sqrt{N}$. As discussed above, in this case M_z ∝ z − M has an o(1) smallest singular value, approximately given by $\frac{\mid z{\mid }^2}{\mu \sqrt{N}}$. Using Eqs. (2.19)–(2.20), we find that the support of lim_{N →∞} ρ(z) is the disc with radius σ (within the annulus σ < |z| ≤ r₀ the eigenvalue density is o(1)), and solving Eqs. (2.8)–(2.9) for |z| ≤ σ, we find that the spectral density is in fact identical with the circular law (the eigenvalue density for the M = 0 case), i.e.

$$\rho \left(r\right)=\{\begin{array}{cc}\frac{1}{\pi {\sigma }^2}+o\left(1\right),\hfill & (r<\sigma )\hfill \\ o\left(1\right)\hfill & (r>\sigma ).\hfill \end{array}\phantom{\}}$$ (2.46)

It was shown in Refs. [36, 44] that more generally, for any M of rank o(N) and bounded ∥M∥_F, the eigenvalue density of A = M + σJ is given by the circular law in the limit N → ∞. For single rank M (as in the present case) and a diagonal R, it was shown in Ref. [45] that the eigenvalue density of M + J R agrees with that of J R as N → ∞. In the present example, it was observed in Ref. [39] that even though the majority of the eigenvalues are distributed according to the circular law, there also exist a number of “outlier” eigenvalues spread outside the circle |z| = σ, which unlike in the M = 0 case, may lie at a significant distance away from it (see Fig. 5). As we mentioned in Sec. II A, the non-crossing approximation cannot be trusted to correctly yield the o(1) contributions to ρ(z) by these outliers for |z| > σ. However, we found that if we ignore this warning and use Eqs. (2.8)–(2.9), keeping track of finite-size, o(1) contributions, we obtain results that agree surprisingly well (though not completely) with simulations. First, for the total number of outlier eigenvalues lying outside the circle |z| = σ we obtain

FIG. 5.

(Color online) The eigenvalue spectra of A = M + σJ for the M given by Eq. (2.42) in the balanced case, v^T u = 0. Here, N = 800, σ = 1 and µ = 12 (see equation Eq. (2.43)). The black dots are the superimposed eigenvalues of A for 20 different realizations of complex Gaussian J. The small red circle enclosing the vast majority of the eigenvalues has radius σ = 1, corresponding to the standard circular law Eq. (2.46). A Θ(N) number of eigenvalues lie within this circle. $Aϴ\left(\sqrt{N}\right)$ number lie just outside of this circle in a thin boundary layer which shrinks to zero as N → ∞. Finally, a Θ(1) number of eigenvalues lie at macroscopic distances outside the unit circle. ^{The dashed blue circle shows radius r}0 ^{given by Eq. (2.44);} outliers can even lie outside this boundary.

$$N_{>}\left(\sigma \right)\equiv N{n}_{>}\left(\sigma \right)=\sqrt{N}+O\left(1\right)$$ (2.47)

(here we defined n_> (r) = 1 − n_< (r) to be the proportion of eigenvalues lying outside the radius r); see Fig. 6 for a comparison of Eq. (2.47) with simulations. The vast majority of the outlier eigenvalues counted in Eq. (2.47) lie in a narrow boundary layer immediately outside the circle |z| = σ, the width of which shrinks with growing N. In addition to these, however, there are a Θ(1) number of eigenvalues lying at macroscopic, Θ(1) distances outside the circle |z| = σ. Using Eqs. (2.8)–(2.9) we have calculated N_> (r), the number of outlier eigenvalues lying outside radius r for r > σ. Figure 7 shows a plot of N_> (r) and compares it with the results of simulations for different N. For roughly the inner half of the annulus σ < |z| <r₀, N_> (r) agrees well with simulations, but as r increases it deviates significantly from the empirical averages. In particular, N_> (r) calculated from Eqs. (2.8)–(2.9) vanishes at r₀ given by Eq. (2.44), while the empirical average of the number of outliers is nonzero well beyond r₀. Finally, we note that the distribution of these eigenvalues is not self-averaging, and depends on the real vs. complex nature of the random matrix J [36]. In the real case, their distribution has been recently characterized as that of the inverse roots of a certain random power series with iid standard real Gaussian coefficients [36].

FIG. 6.

(Color online) The number of eigenvalues of M + σJ, for the M given by Eq. (2.42), lying outside the circle of radius σ vs. N (red line). Here, σ = 1, µ = 12 and v^T u = 0. The numbers (red points connected by solid red lines) are obtained by numerically calculating the eigenvalues and counting the outliers for 200 realizations of J, and taking the average of the counts over all realizations, for N = 100, 200, 400, 800, 1600 (error bars show standard error of mean). The black dashed line plots $\sqrt{N}$ for comparison with our theoretical result Eq. (2.47); the (dashed) blue line which includes subleading corrections to $\sqrt{N}$, is obtained by numerically solving Eq. (5.42) and substituting the result in Eq. (5.43) (these formulae are in turn obtained from Eqs. (2.8)–(2.9) in Sec. V B).

FIG. 7.

(Color online) The number, N_> (r), of outlier eigenvalues of A = M + σJ, for the M given by Eq. (2.42), lying farther from the origin than r, as a function of r. Here, σ = 1, µ = 12 and v^T u = 0. The vertical line marks |z| = r₀ ≃ 3.54 where r₀ is given by Eq. (2.44). The colored (shades of gray) connected points are N_> (r) for realizations of A, based on 200 samples of J, each color for a different N, for N = 100, 200, 400, 800, 1600 and 3200 (error bars show standard error of sample mean). Note the lack of scaling of N_> (r) with N.

As for the dynamics, we have analytically calculated the magnitude of impulse response, Eq. (2.25), as well as the power-spectrum of steady-state response Eq. (2.33), for A = M + σJ with M given by Eqs. (2.40)–(2.41) with general w_b, when the (impulse or sinusoidal) input feeds into the second Schur mode in one of the N/2 chains/blocks of Eq. (2.40); we denote the index for this block by a. For the average magnitude of impulse response we find

$${\langle \parallel \mathbf{x}\left(t\right){\parallel }^2\rangle }_J=\left[\frac{1+C_a}{2}{I}_0\left(2r_{0}t\right)+\frac{1-C_a}{2}{J}_0\left(2r_{1}t\right)\right]e^{-2\gamma t}$$ (2.48)

where J₀(x) (I₀(x)) is the (modified) Bessel function, r₀

is given by Eq. (2.44), we defined $r_1^2=r_0^2-{\sigma }^2$, and

$$C_a\equiv \frac{1+2\mid w_a{\mid }^2∕{\sigma }^2}{\sqrt{1+2{\langle \mid w_b{\mid }^2\rangle }_b∕{\sigma }^2}},$$ (2.49)

with 〈|w_b|²〉_b = 2 tr(M ^†M) denoting the average squared feedforward weight among all the blocks of Eq. (2.40). In Fig. 8 we plot Eq. (2.48) and compare it with the result obtained by ignoring the disorder (i.e. by setting σ = 0); in the latter case, the block a is decoupled from the rest of the network, and solving the 2 × 2 linear system governed by the matrix $\left(\begin{array}{cc}\hfill -\gamma \hfill & \hfill w_a\hfill \\ \hfill 0\hfill & \hfill -\gamma \hfill \end{array}\right)$ , we obtain $\parallel \mathbf{x}\left(t\right){\parallel }^2=(1+w_a^2{t}^2)e^{-2\gamma t}$. From the figure, we see that the σ ≠ 0 result (green) has a slower asymptotic decay compared with the zero-disorder case (purple); this is due to the disorder having spread some eigenvalues closer to the imaginary axis, creating modes with smaller decay, along with the fact that the coupling between the 2 × 2 blocks induced by the disorder insures that these more slowly decaying modes will be activated. Indeed, for large t, ∥x(t)∥ decays like e^−γt when σ = 0, while in the σ > 0 case, based on Eq. (2.44) it must decay like e^{−(γ−r₀)t}, i.e. by a rate set by the largest real part of the spectrum shifted by −γ (this is indeed what we obtain from Eq. (2.48) using the asymptotics of Bessel functions). In addition, both curves exhibit transient amplification where the magnitude of activity initially grows to a maximum, before it decays asymptotically to zero. The σ ≠ 0 curve shows larger and longer transient amplification, which is most likely attributable both to the eigenvalues being closer to the Re(z) = γ line and to augmented nonnormal effects (e.g. larger effective feed-forward weights, or longer chains). We also mention that, as in our previous examples, if the input direction is optimized for the particular realization of J, significantly larger transient amplification may be achieved.

FIG. 8.

(Color online) The squared norm of response to impulse, x(t) ², of the system Eq. (2.2), for A = M + σJ, with log-normal J, and M given by Eq. (2.40) describing N/2 doublet feedforward chains weights w_b. Here, w_a = |w_b|^{2 b} = 3, σ = 0.4, γ = 1, and N = 1400. The green (thick dashed) curve shows our result, Eqs. (2.48)–(2.49), for the average norm squared which, except for a small window around its peak, lies on top of the red (thick solid) curve showing the empirical average of x(t) ² over 100 realizations of binary J. The five thin dashed black curves show the result for five particular realizations of J, and the pink (light gray) area shows the standard deviation among the 100 realizations. The standard deviation shrinks to zero as N → ∞ and x(t) ² for any realization lies close to its average for large N. For comparison the purple (thin, lowest) curve shows x(t) ² obtained by ignoring the effect of quenched disorder, i.e. by setting A = M.

Finally, the total power spectrum of response to a sinusoidal input, Eq. (2.32), is given by the explicit formula

$${\langle \overline{\parallel {\mathbf{x}}_{\omega }{\parallel }^2}\rangle }_J=\frac{{\omega }^2+{\gamma }^2+\mid w_a{\mid }^2}{{({\omega }^2+{\gamma }^2)}^2-\sigma ({\omega }^2+{\gamma }^2+{\mu }^2)}\parallel {\mathbf{I}}_0{\parallel }^2.$$ (2.50)

where µ² ≡ tr(M ^†M) = (|w_b|²)_b/2 and, as noted above, the direction of I₀ is that of the second Schur mode in block a.

The example Eq. (2.42) motivated by Dale’s law with neurons of either excitatory or inhibitory types, can be generalized to a network of neurons belonging to one of C different types (these could be subtypes of excitatory or inhibitory neurons), in which not only the mean but also the variance of connection strengths depends on the pre- and post-synaptic types. When this dependence is factorizable, in a way we will now describe, the connectivity matrix of such a network will be of the form Eq. (2.1) with non-trivial L and R. Let c(i) ∈ {1, …, C} denote the type of neuron i, and let f_c denote the fraction of neurons of type $c(\mathrm{so}{\sum }_{c=1}^C{f}_c=1)$; we assume C and f_c are all Θ(1). Assume further that each synaptic weight is a product of a pre- and a post-synaptic factor, and that in each synapse these factors are chosen independently from the same distribution, except for a deterministic sign and overall scale that depend only on the type of the pre and post-synaptic neurons, respectively. Thus if A_ij denotes the weight of the synaptic projection from neuron j to neuron i, we have

$$A_{ij}=\frac{1}{\sqrt{N}}\left(l_{c\left(i\right)}{x}_{ij}\right)\left(r_{c\left(j\right)}{y}_{ij}\right)$$ (2.51)

where x_ij’s and y_ij are positive random variables chosen iid from the distributions P_x(x) and P_y (y), respectively. Here, l_c and r_c determine the sign and the scale (apart from the overall $\frac{1}{\sqrt{N}}$ of the pre and post-synaptic factors N of the neurons in cluster c, respectively. Note that when all l_c are positive, A_ij satisfies Dale’s law. By absorbing appropriate constants into l_c’s and r_c’s we can assume that Var[xy] = 〈x²〉 〈y²〉 − 〈x〉² 〈y〉² = 1. Then it is easy to see that A can be cast in the form Eq. (2.1) with

$$L_{ij}=l_{c\left(i\right)}{\delta }_{ij}$$ (2.52)

$$R_{ij}=r_{c\left(i\right)}{\delta }_{ij}$$ (2.53)

$$J_{ij}=\frac{1}{\sqrt{N}}(x_{ij}{y}_{ij}-\xi )$$ (2.54)

$$M=sL{\mathbf{uu}}^{\mathrm{T}}R$$ (2.55)

where u is the unit vector $\frac{1}{\sqrt{N}}{(1,\dots ,1)}^{\mathrm{T}}$,

$$s\equiv \xi \sqrt{N},$$ (2.56)

and ξ ≡ 〈x〉〈y〉 is dimensionless and Θ(1) (note that J, given by Eq. (2.54), indeed has iid elements with zero mean and variance N ⁻¹). Being single-rank, M has N −1 zero eigenvalues; its only (potentially) non-null eigenvector is Lu, with a generically large eigenvalue

$${\lambda }_M=s{\mathbf{u}}^{\mathrm{T}}RL\mathbf{u}=s\frac{1}{N}\sum _{i=1}^N{r}_{c\left(i\right)}{l}_{c\left(i\right)}=\xi \sqrt{N}{\langle {\sigma }_c\rangle }_c$$ (2.57)

where we defined

$${\sigma }_c\equiv l_c{r}_c,$$ (2.58)

$${\langle X_c\rangle }_c\equiv \sum _{c=1}^C{f}_c{X}_c.$$ (2.59)

As for the example Eq. (2.42), we will focus on the balanced case in which λ_M ∝ 〈σ_c〉_c = 0. From Eq. (2.55), $M=\tilde{\mathbf{u}}{\tilde{\mathbf{v}}}^{\mathrm{T}}$ with $\tilde{\mathbf{u}}=L\mathbf{u}$ and $\tilde{v}=sR\mathbf{u}$. The balanced condition is equivalent to $\tilde{\mathbf{u}}\cdot \tilde{\mathbf{v}}=0$ (see Eq. (2.57)). Thus, similar to Eq. (2.42), the Schur representation of M has the form (2.40) with $w_1=\parallel \tilde{\mathbf{u}}\parallel \parallel \tilde{\mathbf{v}}\parallel$ and w_b = 0 for b > 1.

In Sec. V C we prove that, as for Eq. (2.42), for the ensemble Eqs. (2.52)–(2.55) the limit of the eigenvalue distribution, lim_{N →∞} ρ(z), is also not affected by the nonzero mean matrix Eq. (2.55); hence we can obtain lim_{N →∞} ρ(z) for that example by safely setting M to zero, and using formulae Eqs. (2.13)–(2.16) with L and R given by Eqs. (2.52)–(2.53). Thus lim_{N →∞} ρ(z) is isotropic and its support is the disk with radius

$$r_0=\parallel RL{\parallel }_F=\sqrt{{\langle {\sigma }_c^2\rangle }_c}.$$ (2.60)

As in the previous example, when the balance condition 〈σ_c〉_c = 0 holds, use of the naive formula Eq. (2.5) with $M=\tilde{\mathbf{u}}{\tilde{\mathbf{v}}}^{\mathrm{T}}$ would have yielded

$${\tilde{r}}_0=r_0{\left[\frac{1}{2}+\sqrt{\frac{1}{4}+{\xi }^2}\right]}^{1∕2},$$ (2.61)

which is larger than the correct result Eq. (2.60). As discussed above, this result is not correct, but it indicates the existence of Θ(1) number of outlier eigenvalues lying outside the boundary of lim_{N →∞} ρ(z) given by Eq. (2.60). For r < r₀, the N → ∞ limit of the proportion, n_>(r), of eigenvalues lying farther than distance r of the origin is given by g²(r) which is found by solving Eq. (2.15), or equivalently

$${\langle \frac{1}{g^2+{\sigma }_c^{-2}{r}^2}\rangle }_c=1.$$ (2.62)

The results Eqs. (2.17)–(2.18) also hold, wherein the normalized sums over i can be replaced with appropriate averages 〈·〉_c. In the case of two neuronal types a closed solution can be obtained for n_>(r) and ρ(r). Identifying the two types with excitatory and inhibitory neurons, and assuming that l_c = 1, σ_E ≡ σ₁ > 0 and σ_I ≡ σ₂ < 0 (we will use E and I as indices instead of c = 1 and 2, respectively) the ensemble Eqs. (2.52)–(2.55) describes a synaptic connectivity matrix in which all excitatory (inhibitory) connections are iid with mean $\xi {\sigma }_E{N}^{-\frac{1}{2}}(-\xi \mid {\sigma }_I\mid N^{-\frac{1}{2}})$ and variance ${\sigma }_E^2{N}^{-1}\left({\sigma }_I^2{N}^{-1}\right)$. In this case, Eq. (2.62) yields a quadratic equation. Differentiating the solution of that equation with respect to r² we obtain the explicit result

$$\rho \left(r\right)=\frac{{\sigma }_E^{-2}+{\sigma }_I^{-2}}{2\pi }\left[1-\frac{\frac{({\sigma }_E^{-2}+{\sigma }_I^{-2})r^2-1}{2}+\frac{r_0^2-2r^2}{{\sigma }_E^2+{\sigma }_1^2}}{\sqrt{\frac{{(({\sigma }_E^{-2}+{\sigma }_I^{-2})r^2-1)}^2}{4}+\frac{r^2(r_0^2-r^2)}{{\left({\sigma }_E{\sigma }_I\right)}^2}}}\right]$$ (2.63)

This result was first obtained (in a less simplified form) in Ref. [39]. Figure 9 shows two examples of spectra for single realizations of matrices of the form Eq. (2.51), with three neural types (C = 3), where x_ij and y_ij, and hence J_ij, have log-normal distributions. The insets compare n_>(r) based on the numerically calculated eigenvalues, with those found by solving Eq. (2.62). In the right panel, the normally distributed log J_ij have a higher standard deviation, and hence the distribution of J_ij has a heavier tail. The right panel’s inset demonstrates that the convergence to the universal, N → ∞ limit can be considerably slow when the distribution of J_ij is heavy-tailed.

FIG. 9.

(Color online) The eigenvalue spectra of A = M + LJR with M, L and R given by Eqs. (2.52)–(2.55) with neurons belonging to one of three different types (C = 3). The main panels show the eigenvalues for two particular realizations of J. In both panels, N = 2000, f₁ = 0.6, f₂ = f₃ = 0.2, l_c = 1, σ₁ = r₁ = 0.76, σ₂ = r₂ = −0.57, σ₃ = r₃ = −1.71 (so 〈σ_c〉_c = 0 and $r_0^2={\langle {\sigma }_c^2\rangle }_c$ = 1), and J_ij had real entries with log-normal distribution; in the left (right) panel, the normally distributed log₁₀ J_ij had standard deviation 0.5 (0.75). The solid red circles mark the boundaries of the spectral support as given by Eq. (2.60), and the dashed blue circles show the radii given by Eq. (2.61). The insets compare n_>(r) based on the numerically calculated eigenvalues shown in the main panels (connected red dots), with that found by solving Eq. (2.62) (black curve). In the right panel’s inset we have also plotted (green connected dots lying slightly above the red connected circles) the empirically calculated n_>(r) for a single realization with the same ensemble parameters, but with N = 8000; the convergence to the universal limit at N → ∞ is significantly slower in the right panel in which the distribution of J_ij had a considerably heavier tail.

3. Linearizations of nonlinear neural and ecological networks

In neuroscience applications, Eq. (2.2) can arise as a linearization of nonlinear firing rate equations for a recurrent neural network of N neurons, around some stationary background. The nonlinear dynamical equations for the evolution of the network activity typically take the form [63]

$$T\frac{dv\left(t\right)}{dt}=-v\left(t\right)+Wf\left(v\left(t\right)\right)+{\mathbf{I}}^v\left(t\right).$$ (2.64)

Here v(t) is the vector of state variables of all neurons at time t; its i-th component, v_i(t), is commonly thought of as the voltage of the i-th neuron, or the total synaptic input it receives. f (·) is the neuronal nonlinear input-output function, which is imposed element-by-element on its vector argument, with f (v)_i ≡ f (v_i) giving the output, i.e. the firing rate, of neuron i; I^v (t) is the external input vector; T = diag(τ₁, τ₂, · · · , τ_N) is a N × N diagonal matrix whose diagonal elements are the positive time-constants of the neurons (hence T is invertible); and W is the N × N synaptic connectivity matrix.

Suppose that for a constant external input, ${\mathbf{I}}_{\ast }^v$, Eq. (2.64) has a fixed point v_∗. Then, given a small perturbation in the input, ${\mathbf{I}}^v\left(t\right)={\mathbf{I}}_{\ast }^v+\sigma {\mathbf{I}}^v\left(t\right)$, we can write v(t) = v_∗ + x(t), and linearize the dynamics around the fixed point by expanding Eq. (2.64) to first order in x(t) and δI^v (t). This yields the set of linear differential equations

$$T\frac{d\mathbf{x}\left(t\right)}{dt}=-\mathbf{x}\left(t\right)+W\Phi \mathbf{x}\left(t\right)+\delta {\mathbf{I}}^v\left(t\right),$$ (2.65)

for the (small) deviations, where we defined the diagonal Jacobian

$$\Phi =\text{diag}\left(f^{\prime }\left(v_{\ast }\right)\right).$$ (2.66)

Now suppose that the original connectivity matrix can be written as W = 〈W〉 + δW, with a quenched disorder part that is an iid random matrix: δW = σJ. Then multiplying Eq. (2.65) by T ⁻¹, we can convert Eq. (2.65) into the form Eq. (2.2) with γ = 0 and A = M + LJR with

$$M=T^{-1}(-1+\langle W\rangle \Phi )$$ (2.67)

$$L=T^{-1}$$ (2.68)

$$R=\sigma \Phi$$ (2.69)

and input

$$\mathbf{I}\left(t\right)=T^{-1}\delta {\mathbf{I}}^v\left(t\right).$$ (2.70)

This observation is not limited to neuroscience applications, and can also apply to many other frameworks, e.g. those used in mathematical biology. Generalized Lotka-Volterra (GLV) equations [47] used in modeling the dynamics of food webs provide an example. Let n(t) = (n₁(t), …, n_N (t))^T denote the vector of population sizes of N species. The GLV equations take the form $\frac{d{n}_i}{dt}=n_i(r_i+{\sum }_j{W}_{ij}{n}_j)$ or

$$\frac{d\mathit{n}}{dt}=\text{diag}{(\mathit{r}+W_{\mathit{n}})}_{\mathit{n}}$$ (2.71)

where r_i > 0 are the species’ intrinsic growth rates and W is the interaction matrix. Linearizing Eq. (2.71) around a fixed point, n_∗, yields again a linear system of the form Eq. (2.2) with γ = I(t) = 0. Starting with the same simple model W = 〈W〉 + σJ, we find that A can be written in the form Eq. (2.1) with

$$R={\sigma }_1,L=\text{diag}\left({\mathit{n}}_{\ast }\right),$$ (2.72)

$$M=\text{diag}(r+W_{\mathit{n}\ast })+L\langle W\rangle .$$ (2.73)

Note that if no species is extinct in the fixed point, i.e. if all n_i∗ > 0, then M = L(W).

Assuming the linear systems thus obtained, i.e. the fixed points v_* or n_*, are stable, we can therefore think of our results for ∥x(t)∥² and ∥x_ω∥² as characterizing the temporal evolution and the spectral properties of the linear response of the nonlinear system Eq. (2.64) (Eq. (2.71)) in its fixed point v_∗ (n_∗) to perturbations.

The necessary and sufficient condition for the stability of a fixed point (without any change in the external input) is that all eigenvalues of the corresponding A have negative real parts. Our formula for the boundary of the eigenvalue distribution, Eq. (2.5), can be applied in these cases to map out the region in parameter space (parameters here mean the time constants or intrinsic growth rates in T or r, or the connectivity parameters determining the random ensemble for W, i.e. σ and the parameters of 〈W〉) in which a particular fixed point is stable. Recently our general formula Eq. (2.5) was used in this way by colleagues [48] to determine the phase diagram of a clustered network of neurons, in which intra-cluster connectivity is large, but inter-cluster connectivity is random and weak. Because of the strong intra-cluster connectivity, each cluster behaves as a unit with a single self-coupling a. Letting the random inter-cluster couplings between N clusters have zero mean and variance g²/N, their analysis starts from the equation

$$\frac{d\mathit{v}\left(t\right)}{dt}=-\mathit{v}\left(t\right)+a\text{tanh}\left(\mathbf{v}\left(t\right)\right)+gJ\text{tanh}\left(\mathbf{v}\left(t\right)\right)$$ (2.74)

where J is an iid random matrix as above. Here, v is a vector whose i-th component is the mean voltage of cluster i, while the nonlinear function tanh(v(t)) (with the hyperbolic tangent acting component-wise) represents the vector of mean firing rates of the clusters. The analysis of Ref. [48] shows that there is a region of the phase plane (a, g) where the self-connectivity, a, is excitatory and sufficiently strong, in which the system eventually relaxes to non-zero random attractor fixed points v_∗; for smaller values of a, the dynamics is chaotic (chaos in the a = 0 case was established in Ref. [49]). The form of these fixed points (the distribution of the elements of v_∗ as N → ∞ for a given (a, g)) can be obtained using mean-field theory, and the linearization about v_∗ leads to an equation in the form of Eq. (2.2), with A = M + J R, where M and R are the diagonal matrices

$$M=\text{diag}(-1+a{\text{tanh}}^{\prime }\left({\mathit{v}}_{\ast }\right))$$ (2.75)

$$R=\text{diag}\left(g{\text{tanh}}^{\prime }\left({\mathit{v}}_{\ast }\right)\right).$$ (2.76)

Given this form, it can be shown that the fixed point v_∗ is stable if z = 0 is outside and to the right of the spectrum of the Jacobian matrix of the linearization, A. The mean field solution for v_∗ determines the statistics of the elements of R²M ⁻² for a given (a, g). From these it can be determined if z = 0 is outside the spectrum using our formula for the boundary of spectrum Eq. (2.5), which yields the requirement $\mathrm{tr}\left(\frac{R^2}{M^2}\right)<1$. In this way, the region of stability of the fixed points in the (a, g) plane can be mapped (see Ref. [48] for the results, and a complete discussion of the analysis outlined here). Figure 10 shows a numerical example of the eigenvalue distribution for A for a given (a, g) and the superimposed boundary calculated using Eq. (2.5).

FIG. 10.

(Color online) The eigenvalues (black dots) of A = M + J R, with M and R given by Eqs. (2.75)–(2.76) with g = 0.01, a = 1.02 and N = 2000. This matrix governs the dynamics of small perturbations away from a non-trivial random fixed point in a clustered network of neurons (see Eq. (2.74)), studied in Ref. [48]. The cyan dots on the real line are the eigenvalues of M, and the red curve is the boundary of support of the eigenvalue distribution, as calculated numerically from Eq. (2.5).

In closing we note a potential caveat in the applicability of our formulae to the linearization analysis of systems like Eq. (2.65) and Eq. (2.71). We have derived the general formulae of Secs. II A–II B assuming that M, L and R are independent of J. However, M and R as given by Eqs. (2.67) and (2.69) (or M and L in Eqs. (2.72)–(2.73)) depend on J via their dependence on v_∗ (n_∗). However, in our experience this dependence is often too weak and indirect to render our formulae inapplicable; an example is provided by the excellent agreement of the empirical spectrum and the red boundary given by our formula in Fig. 10, which also held for other parameter choices of the model of Ref. [48].

III. DERIVATION OF THE FORMULA FOR THE SPECTRAL DENSITY

In this section we will derive the formulae Eqs. (2.5)–(2.8) for the average spectral density, ρ(z), of random matrices of the form A = M +LJR where M, L and R are deterministic matrices, and J is random with iid elements of zero mean and variance 1/N. We will use the Hermitianized diagrammatic method developed in Refs. [24, 25] (and reviewed in Ref. [50]), which we will recapitulate here for completeness. As mentioned in Sec. II, the spectral density is self-averaging for large N. Furthermore, as established in Ref. [34], it is also universal in the large N limit, in the sense that it is independent of the details of the distribution of the elements of J as long its mean and variance are as stated. The same universality theorem also ensures that the real or complex nature of J does not by itself affect ρ(z) to leading order. Therefore, for simplicity we consider the case where J is a zero-mean complex Gaussian random matrix with 〈J_abJ_cd〉 = 0, and

$$\langle J_{ab}{J}_{cd}^{\ast }\rangle =\frac{1}{N}{\delta }_{ac}{\delta }_{bd}.$$ (3.1)

Thus $\langle \mid J_{ab}{\mid }^2\rangle =\frac{1}{N}$, and all other first and second moments of J (including $\langle J_{ab}^2\rangle$) vanish. The measure on J can be written as

$$d\mu \left(J\right)\propto e^{-N\text{Tr}\left(J{J}^{†}\right)}\prod _{ab}d\text{Im}{J}_{ab}d\text{Re}{J}_{ab}.$$ (3.2)

In this form, and by the invariance of the trace, it is clear that the measure is symmetric with respect to the group U (N) ⊗ U (N), acting on J by J ↦ UJV^† where U and V are arbitrary N × N unitary matrices.

For a particular realization of J, we define the “Green’s function” G(z; J) by

$$G(z;J)\equiv \frac{1}{M_z-J},$$ (3.3)

where M_z = L⁻¹(z − M)R⁻¹ (Eq. (2.6)). In the case L, R ∝ 1, G(z; J) will be proportional to the resolvent of A, $\frac{1}{z-A}$. More generally we have

$$\frac{1}{z-A}=R^{-1}G(z;J)L^{-1}.$$ (3.4)

Following Ref. [24], we will use the identity

$${\delta }^2\left(z\right)=\frac{1}{\pi }{\partial }_{\stackrel{‒}{z}}{\partial }_z\text{ln}\mid z{\mid }^2=\frac{1}{\pi }{\partial }_{\stackrel{‒}{z}}\left(\frac{1}{z}\right)$$ (3.5)

where the first identity follows by noting that $4{\partial }_{\stackrel{‒}{z}}{\partial }_z={\nabla }^2$, where ∇² is the 2-D Laplacian, and recalling from electrostatics that the solution of Poisson’s equation for a point charge at origin, i.e. ∇²φ(z) = 4πδ²(z), in 2-D is given by the potential field φ(z) = ln |z|²; the second identity follows from ${\partial }_z\ln \mid z{\mid }^2={\partial }_z(\ln z+\ln \stackrel{‒}{z})=\frac{1}{z}+0$. Using Eq. (3.5) we can write the empirical spectral density, defined in Eq. (2.4), as

$${\rho }_J\left(z\right)=\frac{1}{\pi }{\partial }_{\stackrel{‒}{z}}\frac{1}{N}\sum _{\alpha }\frac{1}{z-{\lambda }_{\alpha }}=\frac{1}{\pi }{\partial }_{\stackrel{‒}{z}}\text{tr}\frac{1}{z-A}.$$ (3.6)

Performing the ensemble average we obtain

$$\rho \left(z\right)\equiv {\langle {\rho }_J\left(z\right)\rangle }_J=\frac{1}{\pi }{\partial }_{\stackrel{‒}{z}}\text{tr}\left[{\left(RL\right)}^{-1}{\langle G(z;J)\rangle }_J\right],$$ (3.7)

where we used Eq. (3.4), and the linearity and cyclicity of the trace. Thus, to calculate ρ(z), our task boils down to calculating 〈G(z; J)〉_J.

The diagrammatic technique provides a method for calculating averages of products of G(z; J)’s. However, this method in its standard form relies on A being a Hermitian matrix. It starts by an expansion of G(z; J) in powers of J, which is only valid when z is far enough from the spectrum of A, i.e. away from the points we are most interested in. For Hermitian matrices, this is no problem as the spectrum is confined to the real line, and therefore G(z; J) and 〈G(z; J)〉_J will be analytic outside the real line. Thus one can use the expansion for z far away outside the real line, perform the averaging over J, and sum up the most dominant contributions to obtain a result analytic in z. This result can then be analytically continued to z arbitrarily close to the spectrum on the real line, yielding information about the spectrum. All this would seemingly fail in the case of a nonnormal (and in particular non-Hermitian) A, with eigenvalues that in general cover a two dimensional region in the complex plane. However, using a trick introduced by Ref. [24], we can turn this problem to an auxiliary problem of averaging the Green’s functions for a Hermitian matrix. By doubling the degrees of freedom, one defines a z-dependent, 2N × 2N Hermitian “Hamiltonian”

$$H\left(z\right)\equiv \left(\begin{array}{cc}\hfill 0\hfill & \hfill M_z-J\hfill \\ \hfill M_z^{†}-J^{†}\hfill & \hfill 0\hfill \end{array}\right),$$ (3.8)

and the corresponding 2N × 2N resolvent matrix or Green’s function depending on a new complex variable η:

$$\mathbf{G}=(\eta ,z;J)\equiv {(\eta -H\left(z\right))}^{-1}=-\left(\begin{array}{cc}\hfill \frac{\eta }{(M_z-J){(M_z-J)}^{†}-{\eta }^2}\hfill & \hfill \frac{M_z-J}{{(M_z-J)}^{†}(M_z-J)-{\eta }^2}\hfill \\ \hfill \frac{{(M_z-J)}^{†}}{(M_z-J){(M_z-J)}^{†}-{\eta }^2}\hfill & \hfill \frac{\eta }{{(M_z-J)}^{†}(M_z-J)-{\eta }^2}\hfill \end{array}\right).$$ (3.9)

For η → i0, we see that

$$\mathit{G}(0,z;J)=-\left(\begin{array}{cc}\hfill 0\hfill & \hfill {(M_z-J)}^{-†}\hfill \\ \hfill {(M_z-J)}^{-1}\hfill & \hfill 0\hfill \end{array}\right).$$ (3.10)

and thus from Eq. (3.3), for any realization of J

$$G(z;J)=-\underset{\eta \to i0}{\text{lim}}{\mathit{G}}^{21}(\eta ,z;J)$$ (3.11)

Here, we have used the notation

$$G(\eta ,z;J)=\left(\begin{array}{cc}\hfill G^{11}(\eta ,z;J)\hfill & \hfill G^{12}(\eta ,z;J)\hfill \\ \hfill G^{21}(\eta ,z;J)\hfill & \hfill G^{22}(\eta ,z;J)\hfill \end{array}\right),$$ (3.12)

where G^αβ (with α, β ∈ {1, 2}) are N ×N matrices, forming the four blocks of G. We have written the limit in Eq. (3.11) as η → i0 to emphasize that until the end of our calculations η is to retain a nonzero imaginary part, which serves to regularize the denominators in Eq. (3.9); c.f. the discussion after Eq. (3.35). We will be carrying out a perturbation expansion in powers of J, so we decompose the Hamiltonian according to

$$H\left(z\right)=H_0\left(z\right)-J,$$ (3.13)

$$J=\left(\begin{array}{cc}\hfill 0\hfill & \hfill J\hfill \\ \hfill J^{†}\hfill & \hfill 0\hfill \end{array}\right),H_0\left(z\right)\equiv \left(\begin{array}{cc}\hfill 0\hfill & \hfill M_z\hfill \\ \hfill M_z^{†}\hfill & \hfill 0\hfill \end{array}\right).$$ (3.14)

We will sometimes use a tensor product notation to denote matrices in this doubled up space, e.g. writing J = σ⁺ ⊗ J + σ⁻ ⊗ J ^†, where we defined the 2 × 2 matrices

$${\sigma }^{+}=\left(\begin{array}{cc}\hfill 0\hfill & \hfill 1\hfill \\ \hfill 0\hfill & \hfill 0\hfill \end{array}\right){\sigma }^{-}=\left(\begin{array}{cc}\hfill 0\hfill & \hfill 0\hfill \\ \hfill 1\hfill & \hfill 0\hfill \end{array}\right).$$ (3.15)

By a slight abuse of notation we also denote 2N × 2N matrices σ^± ⊗ 1_N×N by σ^±, and we will denote the identity matrix in any space by 1. From Eqs. (3.11) we obtain and tr[(RL)^-1 G(z;J)] = - tr [(σ⁺ ⊗(RL)^-1)G(i0⁺, z;J)], and from Eq. (3.7)

$$\rho \left(z\right)=-\underset{\eta \to i0}{\text{lim}}\frac{1}{\pi }{\partial }_{\stackrel{‒}{z}}\mathrm{tr}\left(({\sigma }^{+}\otimes {\left(RL\right)}^{-1})G(\eta ,z)\right),$$ (3.16)

$$=-\underset{\eta \to i0}{\text{lim}}\frac{1}{\pi }{\partial }_{\stackrel{‒}{z}}\mathrm{tr}\left(\left({\left(RL\right)}^{-1}\right)G^{21}(\eta ,z)\right),$$ (3.17)

where we defined

$$\mathit{G}(\eta ,z)\equiv {\langle \mathit{G}(\eta ,z;J)\rangle }_J.$$ (3.18)

Having expressed ρ(z) in terms of the ensemble average of the Green’s function for a Hermitian matrix, we now develop the diagrammatic method for calculating ensemble averages of products of G(η, z; J) (including G(η, z)). Note that, being the Green’s function of a Hermitian matrix, G(η, z; J) and hence G(η, z) = 〈G(η, z; J)〉_J are analytic functions of η for η outside the real line, and therefore analytic continuation can be used to take the limit η → i0 after obtaining the average over J for η sufficiently away from the real line.

We will denote the elements of a generic 2N × 2N matrix A by $A_{ab}^{\alpha \beta }$, where the Greek indices range in {1, 2} and the Latin indices range in {1, …, N }. Using this notation, the definition Eq. (3.14), and Eq. (3.1), we can write the covariance for the components of J as

$${\langle J_{ab}^{\alpha \beta }{J}_{cd}^{\gamma \delta }\rangle }_J=\frac{1}{N}{\delta }_{ad}{\delta }_{bc}\left({\sigma }_{\alpha \beta }^{+}{\sigma }_{\gamma \delta }^{-}+{\sigma }_{\alpha \beta }^{-}{\sigma }_{\gamma \delta }^{+}\right)$$ (3.19)

(the terms proportional to σ⁺σ⁺ and σ⁻σ⁻ involve 〈J_abJ_cd〉, or its complex conjugate, which vanish for the complex Gaussian ensemble). It will be more handy to rewrite the parenthesis on the right side of Eq. (3.19) as ${\pi }_{\alpha \delta }^1{\pi }_{\gamma \beta }^2+{\pi }_{\alpha \delta }^2{\pi }_{\gamma \beta }^1$, where

$${\pi }^1\equiv \left(\begin{array}{cc}\hfill 1\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 0\hfill \end{array}\right){\pi }^2\equiv \left(\begin{array}{cc}\hfill 0\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 1\hfill \end{array}\right),$$ (3.20)

yielding

$${\langle J_{ab}^{\alpha \beta }{J}_{cd}^{\gamma \delta }\rangle }_J=\frac{1}{N}\sum _{r=1}^2\left({\pi }_{\alpha \beta }^r{\delta }_{ad}\right)\left({\pi }_{\gamma \beta }^{-r}{\delta }_{cb}\right).$$ (3.20)

Also, since J_ab have zero mean, we have 〈J〉_J = 0.

The starting point of the diagrammatic method is the perturbation expansion of G(η, z; J) = (η −H₀(z)−J)⁻¹ in powers of J

$$\mathit{G}(\eta ,z;J)=\mathit{G}(\eta ,z;0){\left[J\mathit{G}(\eta ,z;0)\right]}^n$$ (3.22)

where G(η, z; 0) is given by Eq. (3.9) with the J ’s set to zero. This equation is represented diagrammatically in the third line of Fig. 11; the thin arrows defined in the first line of the figure represents G(η, z; 0), and the dashed lines represent a power of J before ensemble averaging. To obtain the average resolvent, G(η, z), we then average Eq. (3.22), term by term, with respect to the ensemble Eq. (3.2). Since the measure is Gaussian with zero mean, according to Wick’s formula, the average of each term of Eq. (3.22) involving n factors of J is given by a sum over the contributions of all possible complete pairings of the J ’s in that term (in particular, since 〈J〉_J = 0, terms in Eq. (3.22) with odd powers of J vanish after averaging). Each pairing can be represented as a Feynman diagram, as shown in Fig. 11, the first two lines of which define the diagram elements. For example, the last diagram in the fourth line of Fig. 11 shows one possible pairing of the term in Eq. (3.22) corresponding to n = 6. The contribution of each pairing diagram is given by a product of factors, one per each pair, given by Eq. (3.21) (represented by wavy lines) with the right indices for that pair, as well as the factors of G(η, z; 0) (represented by thin arrows), with all the intervening Greek and Latin matrix indices summed over their proper ranges. We show in Appendix A that for Im η ≠ 0, and so long as ∥(RL)⁻¹∥ remains bounded as N → ∞, only non-crossing pairings need to be retained in the large N limit, as crossing pairings are suppressed by inverse powers of N and do not contribute in the limit (a pairing diagram is non-crossing if it can be drawn on a plane, with the wavy lines drawn only on the half-plane above the straight arrow line, without any wavy lines crossing). As the last two lines of Fig. 11 demonstrate, all non-crossing diagrams can be generated by iterating the equation

FIG. 11.

The first two lines define different elements of Feynman diagrams: the Green’s function for J = 0 (zero disorder), ${\mathit{G}}_{ab}^{\alpha \beta }(\eta ,z;0)$, the covariance of two J elements, the ensemble averaged Green’s function, G(η, z) ≡ G(η, z; J) _J, and the self-energy Σ(η, z), Eq. (3.24) (the matrix indices for G(η, z) and Σ(η, z) are arranged as for ${\mathit{G}}_{ab}^{\alpha \beta }(\eta ,z;0)$). The third line is the diagrammatic representation of the expansion Eq. (3.22) of G(η, z; J) before averaging over J, where the J’s are represented by dashed lines. Averaging over Eq. (3.2) is performed by pairing all J ’s and connecting them with the wavy lines representing 〈J J〉. In the large N limit, the contribution of crossing pairings is suppressed by negative powers of N ; the sum of all non-crossing diagrams, shown on the fourth line, yields the leading contribution to G(η, z) for large N. The last line shows the diagrammatic representation of Eq. (3.23), which if iterated generates all the non-crossing diagrams. Alternatively, G(η, z) can be found by solving this self-consistent equation directly.

$$G(\eta ,z)=\mathit{G}(\eta ,z;0)+\mathit{G}(\eta ,z;0)\sum (\eta ,z)G(\eta ,z),$$ (3.23)

starting from G₍₀₎(η, z) = G(η, z; 0). This equation is represented diagrammatically in the last line of Fig. 11, with the “self-energy” matrix, Σ(η, z), defined by the diagram in the second line of that figure, i.e.

$$\sum (\eta ,z)\equiv {\langle JG(\eta ,z)J\rangle }_J.$$ (3.24)

Using Eq. (3.21) we obtain

$$\sum _{ad}^{\alpha \delta }(\eta ,z)={\delta }_{ad}\sum _{r=1}^2{\pi }_{\alpha \delta }^r\frac{1}{N}\mathrm{Tr}\left({\pi }^{3-r}\mathit{G}(\eta ,z)\right),$$ (3.25)

which using Eq. (3.20) we can write as

$$\sum (\eta ,z)=\left(\begin{array}{cc}\hfill -i{g}_2(\eta ,z)1\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill -i{g}_1(\eta ,z)1\hfill \end{array}\right),$$ (3.26)

where we defined the scalar functions

$$g_{\alpha }(\eta ,z)\equiv i\mathrm{tr}{\mathit{G}}^{\alpha \alpha }(\eta ,z).$$ (3.27)

Using Eq. (3.26) we can solve Eqs. (3.23)–(3.26) for G(η, z) at once, in terms of g_α(η, z), and then use Eq. (3.27) to obtain a self-consistency equation, which can be solved for g_α(η, z). To this end, we multiply Eq. (3.23) by G⁻¹(η, z; 0) on the left, and by G⁻¹(η, z) on the right, to obtain

$$\begin{array}{cc}\hfill G(\eta ,z)& ={[{\mathit{G}}^{-1}(\eta ,z;0)-\sum (\eta ,z)]}^{-1}\hfill \\ \hfill & ={[\eta -H_0\left(z\right)-\sum (\eta ,z)]}^{-1}.\hfill \end{array}$$ (3.28)

Using this expression with Eqs. (3.14) and (3.26), it can be easily checked that

$$\mathit{G}(\eta ,z)=-\left(\begin{array}{cc}\hfill (\eta +i{g}_1)K_1^{-1}\hfill & \hfill (z-M)K_2^{-1}\hfill \\ \hfill (\stackrel{‒}{z}-M^{†})K_1^{-1}\hfill & \hfill (\eta +i{g}_2)K_2^{-1}\hfill \end{array}\right),$$ (3.29)

where $K_1\equiv M_z{M}_z^{†}+(g_1-i\eta )(g_2-i\eta )$ and $K_2\equiv M_z^{†}{M}_z+(g_1-i\eta )(g_2-i\eta )$, and we dropped the argu(η, z) for succinctness. Imposing Eq. (3.27) we obtain the self-consistency equations

$$g_1=(g_1-i\eta )\mathrm{tr}\left(K_1^{-1}\right),$$ (3.30)

$$g_2=(g_2-i\eta )\mathrm{tr}\left(K_2^{-1}\right).$$ (3.31)

Before solving these equations for g₁ and g₂, we first show that tr (K₁^-1) = tr(K₂^-1) One way to see this is to use the singular value decomposition (SVD) of M_z in the form

$$M_z=U_z{S}_z{V}_z^{†},$$ (3.32)

where S_z is a nonnegative diagonal matrix with the singular values of M, s_i(z) (i = 1, · · · , N), on the diagonal, and U_z and V_z are unitary matrices (as in Sec. II A we include possibly vanishing singular values among s_i(z), so that S_z, U_z and V_z are always N × N matrices). Using the invariance of trace under similarity transforms, we obtain $\mathrm{tr}\left(K_1^{-1}\right)=\mathrm{tr}\left(K_2^{-1}\right)\mathrm{tr}{(S_z^2+(g_1-i\eta )(g_2-i\eta ))}^{-1}$. Given this equality, it is not hard to see that Eqs. (3.30) cannot be simultaneously satisfied unless g₁(η, z) = g₂(η, z) ≡ g(η, z), with g(η, z) satisfying

$$g=(g-i\eta )\mathrm{tr}\left[\frac{1}{S_z^2+{(g-i\eta )}^2}\right],$$ (3.33)

or as written in the original basis

$$g=(g-i\eta )\mathrm{tr}\left[\frac{1}{M_z{M}_z^{†}+{(g-i\eta )}^2}\right],$$ (3.34)

Noting from Eqs. (3.26), that the self-energy is thus proportional to the 2N × 2N identity matrix, from Eqs. (3.28) and (3.9) (for J = 0) we obtain

$$\begin{array}{cc}\hfill G(\eta ,z)& =\mathit{G}(\eta +ig(\eta ,z),z;0)\hfill \\ \hfill & =-\left(\begin{array}{cc}\hfill \frac{i\gamma }{M_z{M}_z^{†}+{\gamma }^2}\hfill & \hfill \frac{M_z}{M_z^{†}{M}_z+{\gamma }^2}\hfill \\ \hfill \frac{M_z^{†}}{M_z{M}_z^{†}+{\gamma }^2}\hfill & \hfill \frac{i\gamma }{M_z^{†}{M}_z+{\gamma }^2}\hfill \end{array}\right)\hfill \end{array}$$ (3.35)

where γ ≡ g(η, z) − iη.

According to Eq. (3.11), for our case of interest we must solve Eq. (3.34) in the limit η → i0. Note, however, that as shown in Appendix A, the non-crossing approximation is in general guaranteed to work only for Im η ≠ 0; hence the limit η → i0 must be taken after the limit N → ∞ (as already pointed out in Sec. II, taking the limits in this order is important in cases where some of the singular values in S_z vanish in the limit N → ∞). For our purposes, it suffices to let η = iE for some real positive ϵ, and take the limit ϵ → 0⁺ at the end. In this case one must seek a positive solution for g(iE, z) in Eq. (3.34); this is because by definition, g(η, z) = itr G₁₁(η, z) = (tr iG₁₁(η, z; J))_J and from Eq. (3.9) we obtain $g(iϵ,z)={\langle \mathrm{tr}\frac{ϵ}{(M_z-J){(M_z-J)}^{†}+{ϵ}^2}\rangle }_J$ which for E > 0, is the ensemble average of the trace of a positive definite matrix and hence positive. Taking the limit N → ∞ while keeping ϵ (and hence ϵ + g) positive and nonzero, we define

$$K(\gamma ,z)\equiv \underset{N\to \infty }{\text{lim}}\mathrm{tr}\left[\frac{1}{M_z{M}_z^{†}+{\gamma }^2}\right]=\underset{N\to \infty }{\text{lim}}\mathrm{tr}\left[\frac{1}{S_z^2+{\gamma }^2}\right]$$ (3.36)

for γ = g + E > 0. We can then rewrite Eq. (3.34) as

$$\gamma (1-K(\gamma ,z))=ϵ,$$ (3.37)

with γ = g + ϵ. Since ϵ and γ = g + ϵ are positive, it follows that 1−K(γ, z) must also be positive. In the limit ϵ → 0⁺ there are two possible situations: 1) g, γ → 0⁺, in which case we must have

$$\underset{\gamma \to 0^{+}}{\text{lim}}K(\gamma ,z)<1,$$ (3.38)

or lim_γ→0 + K(γ, z) = 1, or 2) the solution for g stays finite and positive in the limit, while K(γ, z) → 1⁻ as γ → g⁺. Thus in the second case g(z) ≡ lim_ϵ→0 + g(ϵ, z) must satisfy K(g(z), z) = 1, i.e.

$$1=\underset{N\to \infty }{\text{lim}}\mathrm{tr}\left[\frac{1}{S_z^2+g{\left(z\right)}^2}\right].$$ (3.39)

Note further that since K(γ, z) is a decreasing function of γ, in the second case we have K(0⁺, z) ≥ K(g(z), z) = 1, i.e.

$$\underset{\gamma \to 0^{+}}{\text{lim}}K(\gamma ,z)\ge 1.$$ (3.40)

Thus the two possible solutions are realized in complimentary regions (with a shared boundary) of the complex plane for z, respectively given by Eqs. (3.38) and (3.40).

Let us substitute the g(z) = 0 solution for the case (3.38) in Eq. (3.35), and naively set η = iE (and thus γ) to zero, to obtain

$$G(\eta =i{0}^{+},z)=\mathit{G}(\eta =i{0}^{+},z;0).$$ (3.41)

From Eqs. (3.10)–(3.11), this solution yields ${\langle G(z;J)\rangle }_J=-G^{21}(\eta =i{0}^{+},z)=M_z^{-1}=R{(z-M)}^{-1}L$ which is analytic outside the spectrum of M. Hence from Eq. (3.7), it yields ρ(z) = 0, at least outside the spectrum of M; a more careful analysis presented in Appendix B, in which we correctly take the limit N → ∞ in Eq. (3.17) before taking ϵ → 0⁺, confirms that in the region Eq. (3.38), lim_{N →∞} ρ(z) always vanishes. We conclude that the support of lim_{N →∞} ρ(z) is where Eq. (3.40) holds (which is Eq. (2.20) of Sec. II); here g(z) is to be found by solving Eq. (3.39), or equivalently Eq. (2.9) or Eq. (2.10). In this region, we obtain ρ(z) by substituting Eq. (3.35), with the solution of Eq. (3.39), into Eqs. (3.17). This yields Eq. (2.8), which we rewrite here as

$$\rho \left(z\right)=\frac{1}{\pi }{\partial }_{\stackrel{‒}{z}}E\left(z\right)$$ (3.42)

$$E\left(z\right)\equiv \mathrm{tr}\left[\frac{{\left(RL\right)}^{-1}{M}_z^{†}}{M_z{M}_z^{†}+g{\left(z\right)}^2}\right],$$ (3.43)

with g(z) given by Eq. (3.39), or equivalently Eq. (2.9).

We will now obtain an alternative expression for ρ(z), equivalent to Eqs. (3.42)–(3.43), which explicitly shows that it depends only on the singular values of M_z. Noting that, from Eq. (2.6), $\partial \left(M_z{M}_z^{†}\right)={\left(RL\right)}^{-1}{M}_z^{†}$, we can write Eq. (3.43) as

$$E\left(z\right)=\mathrm{tr}\left[\frac{{\partial }_z\left(M_z{M}_z^{†}\right)}{M_z{M}_z^{†}+g{\left(z\right)}^2}\right].$$ (3.44)

On the other hand, we have

$$\begin{array}{cc}\hfill {\partial }_z\mathrm{tr}\ln [M_z{M}_z^{†}+g{\left(z\right)}^2]& =\mathrm{tr}\left[\frac{{\partial }_z(M_z{M}_z^{†}+g^2\left(z\right))}{M_z{M}_z^{†}+g{\left(z\right)}^2}\right],\hfill \\ \hfill & =E\left(z\right)+{\partial }_z\left(g^2\left(z\right)\right),\hfill \end{array}$$ (3.45)

where to write the last term we used Eq. (2.9). Thus we obtain

$$E\left(z\right)={\partial }_z\psi \left(z\right),$$ (3.46)

$$\psi \left(z\right)\equiv -g^2\left(z\right)+\text{tr ln}[M_z{M}_z^{†}+g{\left(z\right)}^2].$$ (3.47)

or using the SVD, Eq. (3.32),

$$\psi \left(z\right)=-g^2\left(z\right)+\text{tr ln}[S_z^2+g{\left(z\right)}^2],$$ (3.48)

$$=-g{\left(z\right)}^2+\frac{1}{N}\sum _{i=1}^N\text{ln}[s_i{\left(z\right)}^2+g{\left(z\right)}^2].$$ (3.49)

Finally, substituting Eq. (3.49) in Eq. (3.46), and using Eq. (2.10), we obtain Eq. (2.12).

For the special case of M = 0, we have M_z = z(RL)⁻¹. If we let σ_i to be the singular values of RL, then the singular values of M_z will be given by s_i(z) = |z| ${\sigma }_i^{-1}$. Substituting this in Eq. (2.10) and multiplying both sides by r² = |z|², we obtain Eq. (2.15). We see immediately that g(z), ϕ(z) and ρ(z) depend only on the radius r = |z|. Similarly we can rewrite Eq. (3.49) as

$$\psi \left(r\right)=-g{\left(r\right)}^2+\frac{1}{N}\sum _{i=1}^N\ln [r^2{\sigma }_i^{-2}+g{\left(r\right)}^2].$$ (3.50)

To find the spectral radius (boundary of the spectrum) r₀ we have to solve Eq. (2.15) for r, setting g(r) = 0. This yields $r_0^2=\frac{1}{N}{\sum }_{i=1}^N{\sigma }_i^2=\parallel RL{\parallel }_{\mathrm{F}}^2$, yielding Eq. (2.13). Let us define the proportion of eigenvalues lying outside a radius r from the origin by n_>(r). To obtain Eqs. (2.14) and (2.16), first note that

$$\rho \left(r\right)=\frac{1}{\pi }{\partial }_{\stackrel{‒}{z}}{\partial }_z\psi \left(z\right)=\frac{1}{4\pi }{\nabla }^2\psi \left(z\right)=\frac{1}{4\pi r}\frac{\partial }{\partial r}\left(r{\partial }_r\psi \left(r\right)\right),$$ (3.51)

where we used the expression of Laplacian, ${\nabla }^2={\partial }_x^2+{\partial }_y^2$, in 2-D polar coordinates in the last equality. Using this with the definition $n_{>}\left(r\right)=2\pi {\int }_r^{\infty }\rho \left(r\right)rdr$, we obtain $n_{>}\left(r\right)={\left[\frac{r}{2}{\partial }_r\psi \left(r\right)\right]}_r^{\infty }$. For the limit at r → ∞, note that for r > r₀, g(r) = 0 and we have $\psi \left(r\right)=\frac{1}{N}{\sum }_{i=1}^N\ln \left(r^2{\sigma }_i^{-2}\right)=2\ln r-\frac{2}{N}\ln \det \left(RL\right)$, and hence $\frac{r}{2}{\partial }_r\psi \left(r\right)\to 1$ as r → ∞. Thus we obtain

$$n_{>}\left(r\right)=1-\frac{r}{2}{\partial }_r\psi \left(r\right).$$ (3.52)

Differentiating Eq. (3.50) and using Eq. (2.15) we obtain

$${\partial }_r\psi \left(r\right)=2r\frac{1}{N}\sum _{i=1}^N\frac{1}{r^2+{\sigma }_i^{2}g{\left(r\right)}^2},$$ (3.53)

and

$$n_{>}\left(r\right)=1-r^2\frac{1}{N}\sum _{i=1}^N\frac{1}{r^2+{\sigma }_i^{2}g{\left(r\right)}^2},$$ (3.54)

$$=g{\left(r\right)}^2\frac{1}{N}\sum _{i=1}^N\frac{{\sigma }_i^2}{r^2+{\sigma }_i^{2}g{\left(r\right)}^2}.$$ (3.55)

Using Eq. (2.15) once again we obtain Eq. (2.16). Finally, using the latter together with Eqs. (3.51)–(3.52) yields Eq. (2.14).

We will prove further general properties for the eigenvalue density for M = 0. Let us first define

$$I_{n,k}(g,r)\equiv {\langle \frac{{\sigma }^{-k}}{{(g^2+{\sigma }^{-2}{r}^2)}^n}\rangle }_{\sigma }$$ (3.56)

and

$${\langle f\left(\sigma \right)\rangle }_{\sigma }\equiv \underset{N\to \infty }{\text{lim}}\frac{1}{N}\sum _{i=1}^{N}f\left({\sigma }_i\right).$$ (3.57)

(We assume σ_i have a limit density, ρ_σ (σ), such that ${\langle f\left(\sigma \right)\rangle }_{\sigma }={\int }_0^{\infty }f\left(\sigma \right){\rho }_{\sigma }\left(\sigma \right)d\sigma$ is well-defined for f(σ) with sufficiently fast decay at infinity. Note that since we assumed that ∥(RL)⁻¹∥ = (min_i σ_i)⁻¹ = O(1), this density has no measure at σ = 0 and hence the averages in Eq. (3.56) are non-singular for n, k ≥ 0. Also (f (σ))_σ is finite as long as f (σ) = O(σ²) as σ → ∞, as we are assuming that the ∥RL∥_F = O(1) and ${\lim }_{N\to \infty }\parallel RL{\parallel }_F^2={\langle {\sigma }^2\rangle }_{\sigma }$.) First, we obtain general expressions for ρ(r = 0) and ρ(r = r₀), with r₀ given by Eq. (2.13). From Eq. (2.14), $\pi \rho \left(r\right)=-\frac{\partial n_{>}\left(r\right)}{\partial \left(r^2\right)}$, which using Eq. (2.15), re-expressed as I_1,0(g, r) = 1, we can write as

$$\pi \rho \left(r\right)=\frac{\frac{\partial I_{1,0}}{\partial \left(r^2\right)}}{\frac{\partial I_{1,0}}{\partial \left(g^2\right)}}=\frac{I_{2,2}(g,r)}{I_{2,0}(g,r)}.$$ (3.58)

Using the facts that at r = 0, g = 1, and at r = r₀, g = 0, we obtain

$$\rho (r=0)=\frac{1}{\pi }\frac{I_{2,2}(1,0)}{I_{2,0}(1,0)}=\frac{1}{\pi }{\langle {\sigma }^{-2}\rangle }_{\sigma }$$ (3.59)

$$\rho (r=r_0)=\frac{1}{\pi }\frac{I_{2,2}(0,r_0)}{I_{2,0}(0,r_0)}=\frac{1}{\pi }\frac{{\langle {\sigma }^2\rangle }_{\sigma }}{{\langle {\sigma }^4\rangle }_{\sigma }}.$$ (3.60)

Using the fact that σ⁴ and σ⁻² are anti-correlated and that σ² = σ⁴σ⁻², we see that ${\langle {\sigma }^2\rangle }_{\sigma }\le {\langle {\sigma }^4\rangle }_{\sigma }{\langle {\sigma }^{-2}\rangle }_{\sigma }$ or

$$\rho (r=r_0)\le \rho (r=0),$$ (3.61)

with equality if and only if ρ(σ) is deterministic, i.e., a delta-function. This can happen if all but an o(1) fraction of the σ_i’s have the same limit as N → ∞; in that case the eigenvalue distribution is given by the circular law. More generally, we can prove that ρ(r) is a decreasing function of r for any choice of L and R (with M = 0). Using $\frac{d\rho \left(r\right)}{dr}=2r\frac{d\rho \left(r\right)}{d\left(r^2\right)}$, and Eq. (3.58) we obtain

$$\frac{d\rho \left(r\right)}{dr}=2r\frac{\frac{d{I}_{2,2}}{d\left(r^2\right)}{I}_{2,0}-I_{2,2}\frac{d{I}_{2,0}}{d\left(r^2\right)}}{I_{2,0}^2},$$ (3.62)

and using $\frac{d}{d\left(r^2\right)}=\frac{\partial }{\partial \left(r^2\right)}+\frac{\partial \left(g^2\right)}{\partial \left(r^2\right)}\frac{\partial }{\partial \left(g^2\right)}=\frac{\partial }{\partial \left(r^2\right)}-\partial \rho \left(r\right)\frac{\partial }{\partial \left(g^2\right)}$ and $\frac{\partial I_{n,k}}{\partial \left(g^2\right)}=-n{I}_{n+1,k}$ and $\frac{\partial I_{n,k}}{\partial \left(r^2\right)}=-n{I}_{n+1,k+2}$ (we will drop the explicit (g, r) dependence of I_n,k ’s when convenient) we find

$$\frac{d\rho \left(r\right)}{dr}=-4r\frac{I_{2,0}^2{I}_{3,4}-2I_{2,2}{I}_{2,0}{I}_{3,2}+I_{2,2}^2{I}_{3,0}}{I_{2,0}^3}$$ (3.63)

Defining

$${\langle f\left(\sigma \right)\rangle }_{\sigma }^{\prime }\equiv \frac{{\langle \frac{f\left(\sigma \right)}{{(g^2+{\sigma }^{-2}{r}^2)}^2}\rangle }_{\sigma }}{{\langle \frac{1}{{(g^2+{\sigma }^{-2}{r}^2)}^2}\rangle }_{\sigma }}$$ (3.64)

${(\langle f\left(\sigma \right))\rangle }_{\sigma }^{\prime }$ is a bonafide expectation operator) we can write

$$\frac{d\rho \left(r\right)}{dr}=-4r\left[{\langle \frac{{\sigma }^{-4}}{g^2+{\sigma }^{-2}{r}^2}\rangle }_{\sigma }^{\prime }-2{\langle \frac{{\sigma }^{-2}}{g^2+{\sigma }^{-2}{r}^2}\rangle }_{\sigma }^{\prime }{\langle {\sigma }^{-2}\rangle }_{\sigma }^{\prime }+{\langle \frac{1}{g^2+{\sigma }^{-2}{r}^2}\rangle }_{\sigma }^{\prime }{\langle {\sigma }^{-2}\rangle }_{\sigma }^{\prime 2}\right]$$ (3.65)

or

$$\frac{d\rho \left(r\right)}{dr}=-4r\left[{\mathrm{Cov}}^{\prime }[\frac{{\sigma }^{-2}}{g^2+{\sigma }^{-2}{r}^2},{\sigma }^{-2}]-{\langle {\sigma }^{-2}\rangle }_{\sigma }^{\prime }{\mathrm{Cov}}^{\prime }[\frac{1}{g^2+{\sigma }^{-2}{r}^2},{\sigma }^{-2}]\right]$$ (3.66)

where ${\mathrm{Cov}}^{\prime }[f,g]\equiv {\langle fg\rangle }_{\sigma }^{\prime }-{\langle f\rangle }_{\sigma }^{\prime }{\langle g\rangle }_{\sigma }^{\prime }$ is the covariance under ${\langle \cdot \rangle }_{\sigma }^{\prime }$. Now since $\frac{{\sigma }^{-2}}{g^2+{\sigma }^{-2}{r}^2}$ and σ⁻² are both strictly decreasing functions of σ (since g > 0 for r < r₀), while $\frac{1}{g^2+{\sigma }^{-2}{r}^2}$ is a strictly increasing function of σ (for r > 0), the first covariance on the right hand side of Eq. (3.66) is positive, while the second one is negative, and therefore

$$\frac{d\rho \left(r\right)}{dr}\le 0.$$ (3.67)

This slope is zero at r = 0 and strictly negative for r > 0 as long as Var[σ] > 0 (again when Var[σ] = 0 we obtain the circular law). At r = r₀ we obtain

$$\begin{array}{cc}\hfill {\rho }^{\prime }\left(r_0\right)& =-\frac{4}{r_0}{\langle {\sigma }^{-2}\rangle }_{\sigma }^{\prime }\left({\langle {\sigma }^{-2}\rangle }_{\sigma }^{\prime }{\langle {\sigma }^2\rangle }_{\sigma }^{\prime }-1\right)\hfill \\ \hfill & =-\frac{4}{r_0}\frac{{\langle {\sigma }^2\rangle }_{\sigma }}{{\langle {\sigma }^4\rangle }_{\sigma }^3}\left({\langle {\sigma }^2\rangle }_{\sigma }{\langle {\sigma }^6\rangle }_{\sigma }-{\langle {\sigma }^4\rangle }_{\sigma }^2\right)\hfill \end{array}$$ (3.68)

The curvature of ρ(r) at zero can also be evaluated by taking the limit r → 0 of the bracket in Eq. (3.66), noting that g → 1 as r → 0. We obtain

$${\rho }^{\prime \prime }(r=0)=-4{\mathrm{Var}}^{\prime }\left[{\sigma }^{-2}\right]=-4\frac{\mathrm{Var}\left[{\sigma }^2\right]}{{\langle {\sigma }^4\rangle }_{\sigma }^2}\le 0.$$ (3.69)

IV. DERIVATION OF THE FORMULA FOR THE AVERAGE NORM SQUARED

In this section, we focus on the dynamics governed by the matrix A = M + LJR, according to Eq. (2.2), and derive the general formulae presented in Sec. II B. We will first consider the system’s response to an impulse input, I(t) = x₀δ(t), at t = 0, before which we assume the system was at rest in its fixed point x = 0. We assume x = 0 is a stable fixed point, i.e. all eigenvalues of −γ1+A have negative real parts, or equivalently, all eigenvalues of A have real parts less than γ (more precisely, we assume that as N → ∞, this will be the case almost surely, i.e. for any typical realization of J ; in particular, the vertical line of z’s with real part γ must be to the right of the support of ρ(z), the average eigenvalue density for A, as found by solving Eq. (2.5)). This means that x(t) decays exponentially as t → ∞, and therefore its Fourier transform, $\tilde{\mathbf{x}}\left(\omega \right)\equiv {\int }_{-\infty }^{\infty }{e}^{i\omega t}\mathbf{x}\left(t\right)dt={\int }_0^{\infty }{e}^{-iwt}\mathbf{x}\left(t\right)dt$ is well-defined. Fourier transformation of Eq. (2.2) with I(t) = x₀δ(t) yields $i\omega \tilde{\mathbf{x}}\left(\omega \right)=(-\gamma +A)\tilde{x}\left(\omega \right)+{\mathbf{x}}_0$. Solving algebraically for $\tilde{\mathbf{x}}\left(\omega \right)$, we obtain $\tilde{\mathbf{x}}\left(\omega \right)={(\gamma +i\omega -A)}^{-1}{\mathbf{x}}_0$, or using Eqs. (3.3)–(3.4), $\tilde{\mathbf{x}}\left(\omega \right)=R^{-1}G(\gamma +i\omega ;J)L^{-1}{\mathbf{x}}_0$. The inverse Fourier transform, $\mathbf{x}\left(t\right)={\int }_{-\infty }^{\infty }{e}^{it\omega }\tilde{x}\left(\omega \right)\frac{d\omega }{2\pi }$ yields

$$x\left(t\right)={\int }_{-\infty }^{\infty }\frac{d\omega }{2\pi }{e}^{it\omega }{R}^{-1}G(\gamma +i\omega ;J)L^{-1}{\mathbf{x}}_0.$$ (4.1)

Our goal is to study the statistics of x(t) (e.g., its moments) under the distribution Eq. (3.2). Equation (4.1) allows us to reduce this task to the calculation of various moments of G(z; J) and its adjoint, and these can be found using the diagrammatic technique. Note that, σ g² +σ⁻² r² in general, these moments involve not only the statistics of the eigenvalues, but also that of the eigenvectors of A = M + LJR; this can be seen from the spectral representation R⁻¹G(z; J)L⁻¹ = (z − A)⁻¹ = V (z − Λ)⁻¹V ⁻¹ where Λ is a diagonal matrix of the eigenvalues of A, and V is the matrix whose columns are the eigenvectors of A. Here we will look at the simplest interesting statistic involving the eigenvectors: the average square norm of the state vector, namely, 〈∥x(t)∥²_J〉. As we discussed in Sec. II B, its study is also motivated by the fact that transient amplification due to nonnormality of A manifests itself in the transient growth of ∥x(t)∥² = x(t)^T x(t). With a slight generalization, we derive a formula for the average of a general quadratic function x(t)^T Bx(t) where B is any symmetric matrix; the norm squared corresponds to B = 1. Using, x(t)^T = x(t)^† (x(t) is real), the identity x^†Bx = Tr (Bxx^†), and Eq. (4.1), we obtain

$$\begin{array}{c}\hfill \mathbf{x}{\left(t\right)}^{\mathrm{T}}B\mathbf{x}\left(t\right)=\int \int \frac{d{\omega }_1}{2\pi }\frac{d{\omega }_2}{2\pi }{e}^{it({\omega }_1-{\omega }_2)}\hfill \\ \hfill \mathrm{Tr}\left(B_{R}G(\gamma +i{\omega }_1;J)C_L{G}^{†}(\gamma +i{\omega }_2;J)\right),\hfill \end{array}$$ (4.2)

where we defined $C_L\equiv L^{-1}{\mathbf{x}}_0{\mathbf{x}}_0^{\mathrm{T}}{L}^{-†}$ and B_R ≡ R^−†BR−1. Using Eq. (3.11) and G^†(z; J) = − lim_η→i0+ G (η, z; J), and the 2 × 2 matrices π^r defined in Eq. (3.20), we can rewrite the trace in Eq. (4.2) as Tr(π² ⊗B_RG(0, z₁; J) π¹ ⊗C_LG(0, z₂; J)), with z_i = γ + iω_i, where now the trace is performed over 2N × 2N matrices. Averaging over J we then obtain

$${\langle \mathbf{x}{\left(t\right)}^{\mathrm{T}}B\mathbf{x}\left(t\right)\rangle }_J=\int \int \frac{d{\omega }_1}{2\pi }\frac{d{\omega }_2}{2\pi }{e}^{it({\omega }_1-{\omega }_2)}F(\gamma +i{\omega }_1,\gamma +i{\omega }_2;B,{\mathbf{x}}_0{\mathbf{x}}_0^{\mathrm{T}}),$$ (4.3)

where, for general matrix arguments B and C, we define

$$F(z_1,z_2;B,C)\equiv {\langle \mathrm{Tr}\left(\mathbf{BG}(0,z_1;J)CG(0,z_2;J)\right)\rangle }_J.$$ (4.4)

with

$$\mathbf{B}\equiv {\pi }^2\otimes B_R,B_R\equiv R^{-†}B{R}^{-1},$$ (4.5)

$$\mathbf{C}\equiv {\pi }^1\otimes C_L,C_L\equiv L^{-1}C{L}^{-†}.$$ (4.6)

Before proceeding to the calculation of F(z₁, z₂; B, C) using the diagrammatic technique, we will also express the other quantities presented in Sec. II B in terms of F(γ + iω, γ + iω; B, C), with appropriate B’s and C’s. First, we obtain the desired expression for the matrix power spectrum, Eq. (2.27), of the steady-state response to a temporally white noisy input, I(t), with covariance Eq. (2.26). Using the Fourier transform of Eq. (2.2), and following similar steps to those leading to Eq. (4.1), we can write the steady-state solution for x(t) as in Eq. (4.1) with x₀ replaced by the Fourier transform of the input, $\tilde{\mathbf{I}}\left(\omega \right)$. Using this and exploiting ${\mathbf{x}}_j\left(t_2\right)={\mathbf{x}}_j^{\ast }\left(t_2\right)$ we can write (after averaging over the input noise)

$$\overline{{\mathbf{x}}_i\left(t_1\right){\mathbf{x}}_j\left(t_2\right)}=\int \int \frac{d{\omega }_1}{2\pi }\frac{d{\omega }_2}{2\pi }{e}^{i{t}_1{\omega }_1-i{t}_2{\omega }_2}{K}_{ij}({\omega }_1,{\omega }_2)$$ (4.7)

where the Fourier-domain covariance matrix, $K({\omega }_1,{\omega }_2)\equiv \overline{\tilde{\mathbf{x}}\left({\omega }_1\right)\tilde{\mathbf{x}}{\left({\omega }_2\right)}^{†}}$, is given by

$$K({\omega }_1,{\omega }_2)\equiv R^{-1}G(\gamma +i{\omega }_1;J)L^{-1}{C}^{\mathbf{I}}({\omega }_1,{\omega }_2)L^{-†}{G}^{†}(\gamma +i{\omega }_2;J)R^{-†}.$$ (4.8)

Here, the bars indicate averaging over the input noise distribution, and we defined $C^{\mathbf{I}}({\omega }_1,{\omega }_2)\equiv \overline{\tilde{\mathbf{I}}\left({\omega }_1\right)\tilde{\mathbf{I}}{\left({\omega }_2\right)}^{†}}$. On the other hand, the Fourier transform of Eq. (2.26) yields

$$C^{\mathbf{I}}({\omega }_1,{\omega }_2)\equiv \overline{\tilde{I}\left({\omega }_1\right)\tilde{\mathbf{I}}{\left({\omega }_2\right)}^{†}}=2\pi \delta ({\omega }_1-{\omega }_2)C^{\mathbf{I}},$$ (4.9)

where we also exploited ${\tilde{\mathrm{I}}}_j^{\ast }\left(\omega \right)={\tilde{\mathrm{I}}}_j(-\omega )$ for a real I(t). Substituting Eq. (4.9) into Eqs. (4.7)–(4.8) we obtain

$$\overline{{\mathbf{x}}_i\left(t_1\right){\mathbf{x}}_j\left(t_2\right)}=\int \frac{d\omega }{2\pi }{e}^{i\omega (t_1-t_2)}{C}_{ij}^{\mathbf{x}}\left(\omega \right),$$ (4.10)

where

$$C^{\mathbf{x}}\left(\omega \right)=R^{-1}G(\gamma +i\omega ;J)L^{-1}{C}^{\mathbf{I}}{L}^{-†}{G}^{†}(\gamma +i\omega ;J)R^{-†}.$$ (4.11)

Noting that Eq. (4.10) expresses the covariance of the response as an inverse Fourier transform, we see that C^x(ω) is indeed the power spectrum of the response, as defined in Eq. (2.27). Finally note that the element, C_ij, of any matrix can be expressed as Tr $\left({\mathbf{e}}_j{\mathbf{e}}_i^{\mathrm{T}}C\right)$, where e_i are the unit basis vectors (i.e. vectors whose a-th component is δ_ia). Using this trick with Eq. (4.11), and following the steps leading from Eq. (4.2) to Eq. (4.3), we see that after ensemble averaging, ${\langle C_{ij}^{\mathbf{x}}\left(\omega \right)\rangle }_J$ can be written in the form

$${\langle C_{ij}^{\mathbf{x}}\left(\omega \right)\rangle }_J=F(\gamma +i\omega ,\gamma +i\omega ;{\mathbf{e}}_j{\mathbf{e}}_i^{\mathrm{T}},C^{\mathbf{I}})$$ (4.12)

where F was defined by Eqs. (4.4)–(4.6).

Next, consider the system Eq. (2.2) being driven by a sinusoidal input $\mathbf{I}\left(t\right)={\mathbf{I}}_0\sqrt{2}\cos \omega t$ (the factor of $\sqrt{2}$ serves to normalize the time average of ${\left(\sqrt{2}\cos \omega t\right)}^2$ to one), and consider the steady state response, which will also oscillate at frequency ω. Decomposing the input, I(t), and the steady-state response, x_ω (t), into their positive and negative frequency components (proportional to e^iωt and e^−iωt, respectively), from Eq. (2.2) we obtain

$${\mathbf{x}}_{\omega }\left(t\right)=\sqrt{2}{R}^{-1}\mathrm{Re}\left[e^{i\omega t}G(\gamma +i\omega ;J)\right]L^{-1}{\mathbf{I}}_0.$$ (4.13)

Thus the norm squared of the steady state response, ∥x(t)∥ = x(t)^†x(t), will have a zero frequency component, plus components oscillating at ±2ω. Averaging over time kills the latter, leaving the zero frequency component intact, yielding

$$\begin{array}{cc}\hfill \overline{{\mathbf{x}}_{\omega }{\left(t\right)}^{\mathrm{T}}{\mathbf{x}}_{\omega }\left(t\right)}& ={\mathbf{I}}_0^{\mathrm{T}}{L}^{-1}{G}^{†}(z;J)R^{-†}{R}^{-1}G(z;J)L^{-1}{\mathbf{I}}_0\hfill \\ \hfill & =\mathrm{Tr}\left(R^{-†}{R}^{-1}G(z;J){\rho }_{\mathbf{I}}{G}^{†}(z;J)\right)\hfill \end{array}$$ (4.14)

where z = γ + iω, the bar indicates temporal averaging, and we defined $\rho \mathbf{I}\equiv L^{-1}{\mathbf{I}}_0{\mathbf{I}}_0^{\mathrm{T}}{L}^{-†}$. Generalizing to $\overline{{\mathbf{x}}_{\omega }{\left(t\right)}^{\mathrm{T}}B{\mathbf{x}}_{\omega }\left(t\right)}$, averaging over the ensemble, and following the steps leading from Eq. (4.2) to Eq. (4.3), we obtain

$${\langle \overline{{\mathbf{x}}_{\omega }{\left(t\right)}^{\mathrm{T}}B{\mathbf{x}}_w\left(t\right)}\rangle }_J=F(\gamma +i\omega ,\gamma +i\omega ;{\mathbf{I}}_0{\mathbf{I}}_0^{\mathrm{T}}),$$ (4.15)

where F is given by Eqs. (4.4)–(4.6). Comparing Eq. (4.15) with Eq. (4.12), we also obtain

$${\langle \overline{{\mathbf{x}}_{\omega }{\left(t\right)}^{\mathrm{T}}B{\mathbf{x}}_{\omega }\left(t\right)}\rangle }_j=\mathrm{Tr}\left(B{\langle C^{\mathbf{x}}\left(\omega \right)\rangle }_J\right)$$ (4.16)

which is Eq. (2.31) of Sec. II, it being understood that C^I in Eq. (4.12) is replaced by ${\mathbf{I}}_0{\mathbf{I}}_0^{\mathrm{T}}$ as in Eq. (4.15).

Now that we have expressed all our quantities of interest in terms of the kernel F as defined in Eq. (4.4), our task boils down to performing the average over J in Eq. (4.4) to obtain a closed formula for F with general arguments B and C. To this end, we now proceed to calculate the more general object

$$F_{{\mu }_1{\nu }_2;{\mu }_2{\nu }_1}(1;2)\equiv {\langle {\mathit{G}}_{{\mu }_1{\nu }_1}(1;J){\mathit{G}}_{{\mu }_2{\nu }_2}(2;J)\rangle }_J,$$ (4.17)

using the diagrammatic technique. Here, we adopted the abbreviated notation (1) ≡ (η₁, z₁) and (2) ≡ (η₂, z₂) for the function arguments, and µ_i = (α_i, a_i) (similarly for ν_i) for indices in the 2N dimensional space (as in Sec. III, α, β, …, and a, b, … denote indices in the 2 and N dimensional spaces, respectively). Once we have calculated F_µ^{1 ν2 ;µ2 ν1} (1; 2), we can obtain F (z₁, z₂; B, C), with the appropriate B and C, via

$$F(z_1,z_2;B,C)={\mathbf{B}}_{{\nu }_2{\mu }_1}{F}_{{\mu }_1{\nu }_2;{\mu }_2{\nu }_1}(0,z_1;0,z_2){\mathbf{C}}_{{\nu }_1{\mu }_2},$$ (4.18)

where all indices are summed over, and B and C were defined in Eqs. (4.5)–(4.6).

As before, we start by using the expansion Eq. (3.22) for the two Green’s functions in Eq. (4.17). This is shown diagrammatically in the first line of Fig. 12, for the contribution of m-th and n-th terms in the expansion of the first and the second Green’s function, respectively. As before, for large N, averaging over J entails summing the contribution of all non-crossing pairings. This is indicated in the second line of Fig. 12. Finally, the third line of Fig. 12 shows that summing over all m’s, and n’s and all non-crossing pairings, is equivalent to replacing all solid lines with thick solid lines representing the average Green’s function in the non-crossing approximation, G(η_i, z_i) (defined diagrammatically in the third line of Fig. (11), and given by Eq. (3.35) as we found in the previous section), and summing over all non-crossing pairings with every pairing connecting the thick arrow lines on top and bottom (and not each to itself). This procedure yields a sum over all ladder diagrams with different number of rungs, as shown in the third line of Fig. 12.

FIG. 12.

Contribtutions to Eq. (4.17) in the non-crossing approximation. The first line shows Eq. (4.17) written using the expansion Eq. (3.22). The diagram shows the contribution of the m-th and n-th terms in the expansion for two Green’s functions, respectively. Thus the top (bottom) solid line contains m (n) factors of J, shown by dashed lines. In the large N limit, averaging each summand over J boils down to summing all non-crossing pairings (NCP) of the dashed lines. The second row shows a specific non-crossing pairing for the diagram shown in the first line. Finally, summing over all m, and n and all NCP’s, is equivalent to replacing all solid lines (representing G(η_i, z_i; J = 0)) with thick solid lines representing the non-crossing average Green’s function, G(η_i, z_i) (calculated according to Eqs. (3.28)–(3.26)), and summing over all NCP’s with every pairing connecting the straight lines on top and bottom (and not each to itself). This procedure yields the ladder diagrams, the sum over which is shown in the third line.

As shown in the first row of Fig. 13, the sum of all ladder diagrams can be written as a sum

FIG. 13.

The first row is the diagrammatic representation of Eqs. (4.19)–(4.21). In the last term, ρ’s and λ’s are summed over. It shows the sum of all ladder diagram contributing to Eq. (4.17) (i.e. the last line of Fig. 12) in terms of D, which is defined in the second row. The first term on the right side of the first row equation (the ladder with zero rungs) is the disconnected average Eq. (4.20); it corresponds to taking the average of each Green’s function in Eq. (4.17) separately and then multiplying. The last row shows an iterative form of the equation in the second row, which can be solved to give the expression Eqs. (4.23) and (4.26) for D.

$$F=F^0+F^D,$$ (4.19)

where

$$F_{{\mu }_1{\nu }_2;{\mu }_2{\nu }_1}^0(1;2)\equiv G_{{\mu }_1{\nu }_1}\left(1\right)G_{{\mu }_2{\nu }_2}\left(2\right),$$ (4.20)

is the disconnected average of the two Green’s functions, and $F_{{\mu }_1{\nu }_2;{\mu }_2{\nu }_1}^D$ is the sum of ladder diagrams in which the two Green’s function are connected by at least one wavy line. The latter can be written in the form

$$F_{{\mu }_1{\nu }_2;{\mu }_2{\nu }_1}^D(1;2)\equiv G_{{\lambda }_2{\nu }_2}\left(2\right)G_{{\mu }_1{\rho }_1}\left(1\right)D_{{\rho }_1{\lambda }_2;{\rho }_2{\lambda }_1}(1;2)G_{{\lambda }_1{\nu }_1}\left(1\right)G_{{\mu }_2{\rho }_2}\left(2\right),$$ (4.21)

where all repeated indices are summed over, and the “diffuson”, D, is given by the sum of all diagrams in the second row of Fig. 13.

To calculate D, it helps to first rewrite Eq. (3.21) as

$${\langle J_{ab}^{\alpha \beta }{J}_{cd}^{\gamma \delta }\rangle }_J=\frac{1}{N}\sum _{r,s=1}^2\left({\pi }_{\alpha \delta }^r{\delta }_{ad}\right){\sigma }_{rs}^1\left({\pi }_{\gamma \beta }^s{\delta }_{cb}\right),$$ (4.22)

where ${\sigma }^1=\left(\begin{array}{cc}\hfill 0\hfill & \hfill 1\hfill \\ \hfill 1\hfill & \hfill 0\hfill \end{array}\right)$ is the first Pauli matrix. This helps us because in the expansion of Fig. 13, the two factors in Eq. (3.21) involving π^r and π^s decouple and get absorbed in adjacent loops, or contribute to form factors in the left or right ends of the ladder diagrams. This is demonstrated in Fig. 14 for the second term in the series expansion of D shown in the second line of Fig. 13. Extending this similarly to all the terms in that expansion, we obtain

FIG. 14.

The contribution to $D_{ad;cb}^{\alpha \delta ;\gamma \beta }$ (1; 2) from the second term in the series shown in the second row of Fig. 13, in more detail. The covariance of J in the form Eq. (4.22) is used to write this expression in a more manageable form. The repeated indices, r, t, u, s, are summed over 1 and 2. The matrices inside the loop multiply each other in cyclic order, giving rise to the trace Tr (G(2)π^tG(1)π^u). The whole diagram gives $\frac{1}{N}{\sum }_{rs}{({\pi }^r\otimes 1)}_{ad}^{\alpha \delta }{\left[{\sigma }^1{\Pi }^D{\sigma }^1\right]}_{rs}{({\pi }^s\otimes 1)}_{cb}^{\gamma \beta }$ where the “polarization matrix” ${\Pi }_{tu}^D$ was defined in Eq. (4.25).

$$D_{\mu \rho ;\lambda \nu }(1;2)=D_{ad;cb}^{\alpha \delta ;\gamma \beta }(1;2)=\frac{1}{N}\sum _{r,s=1}^2\left({\pi }_{\alpha \delta }^r{\delta }_{ad}\right)D_{rs}(1;2)\left({\pi }_{\gamma \beta }^s{\delta }_{cb}\right),$$ (4.23)

where µ = (α, a), ν = (β, b), λ = (γ, c), ρ = (δ, d) and we defined the 2 × 2 matrices

$$D(1;2)\equiv {\sigma }^1+{\sigma }^1{\Pi }^D{\sigma }^1+\cdots ={\sigma }^1\sum _{n=0}^{\infty }{\left({\Pi }^D{\sigma }^1\right)}^n,$$ (4.24)

and the “polarization matrix” for the diffuson

$${\Pi }_{rs}^D(1;2)\equiv \mathrm{tr}\left({\pi }^{r}G\left(1\right){\pi }^{s}G\left(2\right)\right)=\mathrm{tr}\left(G^{rs}\left(1\right)G^{sr}\left(2\right)\right).$$ (4.25)

Here, as before, with the trace performed over the 2N dimensional space, and we used Eq. (3.20) to write the last form of Π^D. Summing the geometric series in Eq. (4.24) we obtain

$$D(1;2)={\sigma }^1{(1_{2\times 2}-{\Pi }^D(1;2){\sigma }^1)}^{-1}.$$ (4.26)

The 2 × 2 matrix inversion yields

$$D(1;2)=\frac{1}{(1-{\Pi }_{12}^D)(1-{\Pi }_{21}^D)-{\Pi }_{11}^D{\Pi }_{22}^D}\left(\begin{array}{cc}\hfill {\Pi }_{22}^D\hfill & \hfill 1-{\Pi }_{12}^D\hfill \\ \hfill 1-{\Pi }_{21}^D\hfill & \hfill {\Pi }_{11}^D\hfill \end{array}\right),$$ (4.27)

where all Π^D ’s have arguments (1; 2) = (η₁, z₁; η₂, z₂) which were suppressed for clarity.

Going back to Eq. (4.18), we can also break up F (z₁, z₂; B, C) into a disconnected part and a connected part mirroring the decomposition Eqs. (4.19)–(4.21):

$$F(z_1,z_2;B,C)=F^0(z_1,z_2;B,C)+\Delta F(z_1,z_2;B,C),$$ (4.28)

where F ⁰(z₁, z₂; B, C) and ΔF (z₁, z₂; B, C) are defined as in Eq. (4.18), but with F_µ^{1 ν2 ;µ2 ν1} on the right side replaced by $F_{{\mu }_1{\nu }_2;{\mu }_2{\nu }_1}^0$ and $F_{{\mu }_1{\nu }_2;{\mu }_2{\nu }_1}^D$, respectively. Using Eqs. (4.20)–(4.21) and (4.23), we then obtain

$$\begin{array}{cc}\hfill F^0(z_1,z_2;B,C)& =\mathrm{Tr}\left(\mathbf{BG}(0,z_1)CG(0,z_2)\right)\hfill \\ \hfill & =\mathrm{Tr}\left(B_R{G}^{21}(0,z_1)C_L{G}^{12}(0,z_2)\right),\hfill \end{array}$$ (4.29)

and

$$\Delta F(z_1,z_2;B,C)=\frac{1}{N}\sum _{r,s}\mathrm{Tr}\left(B_R{G}^{2r}(0,z_1)G^{r2}(0,z_2)\right)\times D_{rs}(0,z_1;0,z_2)\mathrm{Tr}\left(G^{s1}(0,z_1)C_L{G}^{1s}(0,z_2)\right),$$ (4.30)

where r and s are summed over {1, 2}.

According to Eq. (4.3) we are interested in z_i = γ + iω_i (i = 1,2) for arbitrary real ω_i. As we mentioned before Eq. (4.1), these trace a vertical line in the complex plane that is entirely to the right of the support of the average eigenvalue density, ρ(z), of A, i.e. they are in the region where the the valid solution of Eq. (3.34) is the trivial g(0, z) = 0. In this case, we have Eq. (3.41), and for η → i0⁺, from Eq. (3.10) (replacing A with M, corresponding to J = 0) we have

$$G(0,z_i)=-\left(\begin{array}{cc}\hfill 0\hfill & \hfill M_{z_i}^{-†}\hfill \\ \hfill M_{z_i}^{-1}\hfill & \hfill 0\hfill \end{array}\right).$$ (4.31)

Using this in Eqs. (4.29)–(4.30) we obtain

$$F^0(z_1,z_2;B,C)=\mathrm{Tr}\left(B_R{M}_{z_1}^{-1}{C}_L{M}_{z_2}^{-†}\right),$$ (4.32)

and

$$\Delta F(z_1,z_2;B,C)=\mathrm{tr}\left(B_r{G}^{21}(0,z_1)G^{12}(0,z_2)\right)\times D_{12}(0,z_1;0,z_2)\mathrm{Tr}\left(G^{21}(0,z_1)C_L{G}^{12}(0,z_2)\right)$$ (4.33)

Using the definitions Eq. (2.6) and Eqs. (4.5)–(4.6) we can simplify Eq. (4.32) to

$$F^0(z_1,z_2;B,C)=\mathrm{Tr}\left(B\frac{1}{z_1-M}C\frac{1}{{\stackrel{‒}{z}}_2-M^{†}}\right).$$ (4.34)

From Eqs. (4.25) and (4.31) we see that (for z_i of interest and for η_i going to zero) Π^rr = 0 and Π¹² = Π²¹, and from Eq. (4.27) we obtain

$$\begin{array}{cc}\hfill D_{12}(0,z_1;0,z_2)& =\frac{1}{1-{\Pi }_{21}^D(0,z_1;0,z_2)}\hfill \\ \hfill & =\frac{1}{1-\mathrm{tr}\left(G^{21}(0,z_{21})G^{12}(0,z_2)\right)}.\hfill \end{array}$$ (4.35)

Substituting this in Eq. (4.33) and using Eq. (4.31) once again, we finally obtain

$$\Delta F_{z_1,z_2;B,C}=\frac{\mathrm{tr}\left(B_R{M}_{z_1}^{-1}{M}_{z_2}^{-†}\right)\mathrm{Tr}\left(M_{z_1}^{-1}{C}_L{M}_{z_2}^{-†}\right)}{1-\mathrm{tr}\left(M_{z_1}^{-1}{M}_{z_2}^{-†}\right)}$$

and after simplification using Eqs. (2.6) and (4.5)–(4.6),

$$\Delta F(z_1,z_2;B,C)=\frac{\mathrm{tr}\left(B\frac{1}{z_1-M}L{L}^{†}\frac{1}{{\stackrel{‒}{z}}_2-M^{†}}\right)\mathrm{Tr}\left(R^{†}R\frac{1}{z_1-M}C\frac{1}{{\stackrel{‒}{z}}_2-M^{†}}\right)}{1-\mathrm{tr}\left(R^{†}R\frac{1}{z_1-M}L{L}^{†}\frac{1}{{\stackrel{‒}{z}}_2-M^{†}}\right)}.$$ (4.36)

The general formulae of Sec. II B readily follow. Equations (2.21)–(2.24), with C^I replaced by x₀x^T, for the case of response to an impulse input follow from Eqs. (4.3), (4.28), (4.34) and (4.36), respectively. Equations (2.28)–(2.30) (with C^x and ΔC^x defined in Eqs. (2.23)–(2.24)) for the power spectrum of the response to a temporally white noisy input, are similarly obtained from Eq. (4.12) by using Eqs. (4.28), (4.34) and Eq. (4.36), after setting $B={\mathbf{e}}_j{\mathbf{e}}_i^{\mathrm{T}}$, C = C^I and z₁ = z₂ = γ + iω (with the traces involving $B={\mathbf{e}}_j{\mathbf{e}}_i^{\mathrm{T}}$ turned into matrices in Eqs. (2.28)–(2.30), using $\mathrm{Tr}\left({\mathbf{e}}_j{\mathbf{e}}_i^{\mathrm{T}}X\right)=X_{ij}$). The result Eq. (2.31) for the steady state response to a sinusoidal input was already derived in Eq. (4.16).

We see that according to Eqs. (4.3) and (4.28)

$${\langle \mathbf{x}{\left(t\right)}^{\mathrm{T}}B\mathbf{X}\left(t\right)\rangle }_J={\left[\mathbf{x}{\left(t\right)}^{\mathrm{T}}B\mathbf{x}\left(t\right)\right]}_{J=0}+\Delta f_B\left(t\right),$$ (4.37)

where the two terms on the right hand side are obtained by replacing F (·, ·; B) in Eq. (4.3) with Eq. (4.34) and Eq. (4.36), respectively. The integrals over ω₁ and ω₂ decouple for the first term yielding the expected result for J = 0,

$$\begin{array}{cc}\hfill {\left[\mathbf{x}{\left(t\right)}^{\mathrm{T}}B\mathbf{x}\left(t\right)\right]}_{J=0}& =e^{-2\gamma t}\mathrm{Tr}\left(B{e}^{tM}{\mathbf{x}}_0{\mathbf{x}}_0^{\mathrm{T}}{e}^{t{M}^{†}}\right),\hfill \\ \hfill & ={\mathbf{x}}_0^{\mathrm{T}}{e}^{t{(-\gamma +M)}^{†}}B{e}^{t(-\gamma +M)}{\mathbf{x}}_0.\hfill \end{array}$$ (4.38)

Unlike the J = 0 contribution, it is not possible to perform the double Fourier transform, Eq. (4.3), needed for obtaining Δf_B (t) for arbitrary M, L and R. In the next section, we will analytically calculate this for some special examples of M, with L and R proportional to the identity matrix (i.e. for iid quenched randomness).

V. CALCULATIONS FOR SPECIFIC EXAMPLES OF M

In this section we give the detailed calculations of the explicit expressions for the spectral density Eq. (2.8), the power spectrum Eq. (2.31), and the average squared norm Eqs. (2.21) and (2.25), for the specific examples of M, L and R presented in Sec. II C.

In the examples worked out in the subsections V A and V B, both R and L are proportional to the identity matrix; we take L = 1 and R = σ1. Furthermore, for such examples we will do the calculations by choosing the unit of time such that σ = 1 (notice that given Eq. (2.2), the elements of A and M have dimensions of frequency); then at the end of our calculations using the replacements t → tσ, z → z/σ, γ → γ/σ, M → M/σ, and ρ → σ²ρ (with the latter applying to both the eigenvalue density and the power spectral density), we obtain the result for general σ. The eigenvalue density and the norm squared ∥x∥² are invariant with respect to unitary transforms, and, for L and R proportional to the identity, so is the distribution of the random part of A, Eq. (3.2). Thus by effecting a unitary transform M → U ^†M U, we can assume M is already in its Schur form Eq. (2.40) without loss of generality.

A. Single feedforward chain of length N : M_ij = w δ_i+1,j − γδ_ij

We start with the example in Sec. II C 1, where M is

$$M=T=\left(\begin{array}{cccc}\hfill 0\hfill & \hfill w\hfill & \hfill 0\hfill & \hfill \cdots \hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill w\hfill & \hfill \cdots \hfill \\ \hfill ⋮\hfill & \hfill ⋮\hfill & \hfill ⋮\hfill & \hfill \ddots \hfill \end{array}\right)$$ (5.1)

or M_ij = w δ_i+1,j. First we calculate the eigenvalue density. According to Eqs. (2.8)–(2.9), in order to calculate the spectral density, we need to calculate first the inverse of M_z M ^† + g² = (z − M)(z − M)^† + g² (remember that we have set σ = 1, as we explained in the beginning of the section). To this end, notice that K_ij ≡ ^r(z − M)(z − M)^†lij = Q_ij − |w| δ_iN δ_jN where

$$Q_{ij}\equiv (\mid z{\mid }^2+\mid w{\mid }^2){\delta }_{ij}-w\stackrel{‒}{z}{\delta }_{i+1,j}-\stackrel{‒}{w}z{\delta }_{i,j+1}.$$ (5.2)

As the difference (z − M)(z − M)^† − Q = −|w|²e ^T is single rank, we can use the Woodbury formula for matrix inversion to write

$$\frac{1}{K+g^2}=\frac{1}{Q+g^2}+\frac{1}{Q+g^2}{\mathbf{e}}_N{\mathbf{e}}_N^{\mathrm{T}}\frac{1}{Q+g^2}\left(\frac{1}{\mid w{\mid }^{-2}-{\mathbf{e}}_N^{\mathrm{T}}{(Q+g^2)}^{-1}{\mathbf{e}}_N}\right),$$ (5.3)

where ${\mathbf{e}}_N^{\mathrm{T}}=(0,\cdots ,0,1)$. (The only conditions for the validity of Eq. (5.3) is that the factor in parenthesis is not singular, i.e. ${\mathbf{e}}_N^{\mathrm{T}}{(Q+g^2)}^{-1}{\mathbf{e}}_N\ne \mid w{\mid }^{-2}$ we will consider the validity of this condition below.) Since Q is Toeplitz and Hermitian, it can be diagonalized easily. Using standard methods, we find that the eigenvalues and eigenvectors of Q, satisfying Qv_n = λ_nv_n, are given by

$${\lambda }_n={\mid \mid z\mid -\mid w\mid e^{i{\varphi }_n}\mid }^2,{\varphi }_n\equiv \frac{\pi n}{N+1}$$ (5.4)

$$v_n^j=\sqrt{\frac{2}{N+1}}{\left(\frac{\stackrel{‒}{w}z}{w\stackrel{‒}{z}}\right)}^{j∕2}\sin {\varphi }_{n}J$$ (5.5)

for n = 1, …, N. The eigenvectors are orthonormal ${\mathbf{v}}_n^{†}{v}_m={\delta }_{nm}$, and we have the spectral representation

$$\frac{1}{Q+g^2}=\sum _{n=1}^N{\mathbf{v}}_n\frac{1}{{\lambda }_n+g^2}{\mathbf{v}}_n^{†}.$$ (5.6)

Using Eqs. (5.4)–(5.6) in Eq. (5.3) we obtain

$$\mathrm{tr}\frac{1}{K+g^2}=I_1(g,z)+\frac{1}{N}\frac{\frac{-\partial I_2(g,z)}{\partial g^2}}{\mid w{\mid }^{-2}-I_2(g,z)},$$ (5.7)

where we defined

$$I_1(g,z)\equiv \mathrm{tr}\frac{1}{Q+g^2}=\frac{1}{N}\sum _{n=1}^N\frac{1}{{\lambda }_n+g^2}$$ (5.8)

$$I_2(g,z)\equiv {\mathbf{e}}_N^{\mathrm{T}}\frac{1}{Q+g^2}{\mathbf{e}}_N=\frac{1}{N+1}\sum _{n=1}^N\frac{2{\sin }^2{\varphi }_n}{{\lambda }_n+g^2}.$$ (5.9)

In writing the numerator of the last term in Eq. (5.7), we used Eqs. (5.5)–(5.6) to write $\mathrm{Tr}\left(\frac{1}{Q+g^2}{\mathbf{e}}_N{\mathbf{e}}_N^{\mathrm{T}}\frac{1}{Q+g^2}\right)=\parallel \frac{1}{Q+g^2}{\mathbf{e}}_N{\parallel }^2=\frac{1}{N+1}{\sum }_{n=1}^N\frac{2{\sin }^2{\varphi }_n}{{({\lambda }_n+g^2)}^2}=-\frac{\partial I_2(g,z)}{\partial g^2}$. In the N → ∞ limit, the sums in Eqs. (5.8)–(5.9) can be approximated by the integrals

$$I_1(g,z)={\int }_0^{2\pi }\frac{1}{{\mid \mid z\mid -\mid w\mid e^{i\varphi }\mid }^2+g^2}\frac{d\varphi }{2\pi },$$ (5.10)

$$I_2(g,z)={\int }_0^{2\pi }\frac{2{\sin }^2\varphi }{{\mid \mid z\mid -\mid w\mid e^{i\varphi }\mid }^2+g^2}\frac{d\varphi }{2\pi }.$$ (5.11)

Some elementary contour integration then yields

$$I_1(g,z)={[{(\mid z{\mid }^2+\mid w{\mid }^2+g^2)}^2-4\mid w{\mid }^2\mid z{\mid }^2]}^{-1∕2},$$ (5.12)

$$I_2(g,z)=\frac{\mid z{\mid }^2+\mid w{\mid }^2+g^2-I_1{(g,z)}^{-1}}{2\mid w{\mid }^2\mid z{\mid }^2}.$$ (5.13)

In particular, we see that I₂(0, z) = min(|w|⁻², |z|⁻²), so that the condition for the validity of Eq. (5.3) would be violated for |z| < |w|, if g turns out to be zero. However, note that I₂(g, z) is a decreasing function of g², so for finite g² > 0, the denominator in Eq. (5.7) is always positive (as is its numerator, for the same reason). Thus if we follow the correct procedure of Eq. (2.19)–(2.20), taking the N → ∞ limit before sending g² to zero, we are justified in using Eqs. (5.3) and (5.7). Furthermore, for g² > 0 the second term in Eq. (5.7) is O(N ⁻¹), and should be neglected. Solving Eq. (2.9) (with left hand side correctly interpreted as Eq. (2.19)), which now takes the form I₁(g, z) = 1, yields

$$g{\left(z\right)}^2=-\mid z{\mid }^2-\mid w{\mid }^2+\sqrt{4\mid w{\mid }^2\mid z{\mid }^2-1}.$$ (5.14)

This is positive if and only if

$$\sqrt{\mid w{\mid }^2-1}\le \mid z\mid \le \sqrt{\mid w{\mid }^2+1},$$ (5.15)

which after the proper rescaling yields Eq. (2.36) for general σ. Note that Eq. (5.15) is precisely the region given by Eq. (2.20), which in the present case reads I₁(0, z) ≥ 1. It is instructive to compare this result with what we would obtain by naively using Eq. (2.5), i.e. tr (K⁻¹) ≥ 1, wherein g is set to zero before taking the N → ∞ limit; as we now show, that only yields the right inequality in Eq. (5.15). To see this, first note that for |w| > |z|, we can use Eq. (5.7) even for g² = 0 (since the denominator of the last term does not vanish), which yields $\left[{\left(M_z{M}_z^{†}\right)}^{-1}\right]=\mathrm{tr}\left(K^{-1}\right)I_1(0,z)+o\left(1\right)$, and by Eq. (2.5), the right inequality in Eq. (5.15). For |z| < |w|, however, we cannot set g = 0 in Eq. (5.7). In fact, when |z| < |w|, the matrix z − M has an exponentially small singular value; to see this, note that the vector u with components $u_i={\left(\frac{z}{w}\right)}^{i-1}$ satisfies $(z-M)\mathbf{u}=w{\left(\frac{z}{w}\right)}^N{\mathbf{e}}_N$, so that $\parallel (z-M)\mathbf{u}\parallel =\mid w\mid \mid \frac{z}{w}{\mid }^N$ and since $s_{\min }(z-M)\le \frac{\parallel (z-M)\mathbf{u}\parallel }{\parallel \mathbf{u}\parallel }$, it follows that $s_{\min }(z-M)\le \mid w\mid \mid \frac{z}{w}{\mid }^N$, which is O(e^−cN) for |z| < |w|. For large enough N, this singular value alone suffices to make Eq. (2.11) (equivalent to Eq. (2.5)) hold for any |z| < |w|, as $\frac{1}{N}{s}_{\min }{\left(z\right)}^{-2}$ diverges despite its $\frac{1}{N}$ prefactor.

Let us now calculate the eigenvalue density in the annulus Eq. (5.15). In order to use Eq. (2.8), we will first calculate

$$\mathrm{tr}\frac{\stackrel{‒}{z}-M^{†}}{K+g{\left(z\right)}^2}=\stackrel{‒}{z}-\mathrm{tr}\frac{M^{†}}{K+g{\left(z\right)}^2}$$ (5.16)

where we used Eq. (2.9) to write the last expression. To obtain $\mathrm{tr}\frac{M^{†}}{K+g{\left(z\right)}^2}$, we will again use Eq. (5.3). In the region Eq. (5.15), the contribution of the second term in Eq. (5.3) is again suppressed by 1/N, and from Eq. (5.1) and (5.6) we have $\mathrm{tr}{M}^{†}{Q+g^2}^{-1}=\stackrel{‒}{w}\frac{1}{N}{\sum }_{n=1}^N\left({\sum }_{j=1}^{N-1}{v}_n^i{v}_n^{j+1}\right)\frac{1}{{\lambda }_n+g^2}$. A straightforward calculation using Eq. (5.5) (and the orthonormality of v_n) yields ${\sum }_{j=1}^{N-1}{v}_n^j{v}_n^{j+1}={\left(\frac{w\stackrel{‒}{z}}{\stackrel{‒}{w}z}\right)}^{1∕2}\cos {\varphi }_n$. Using this and approximating the sum over n with an integral, we obtain

$$\begin{array}{cc}\hfill & \mathrm{tr}\left[\frac{M^{†}}{Q+g{\left(z\right)}^2}\right]\approx \frac{\mid w\mid \mid z\mid }{z}{\int }_0^{2\pi }\frac{\cos \varphi }{{\mid \mid z\mid -\mid w\mid e^{i\varphi }\mid }^2+g{\left(z\right)}^2}\frac{d\varphi }{2\pi },\hfill \\ \hfill & =\frac{1}{2z}[(\mid z{\mid }^2+\mid w{\mid }^2+g{\left(z\right)}^2)I_{1}g(\left(z\right),z)-1].\hfill \end{array}$$ (5.17)

Using Eq. (5.17) with I₁(g(z), z) = 1 (true in the region Eq. (5.15)), differentiating Eq. (5.16) with respect to $\stackrel{‒}{z}$, and substituting in Eq. (2.8), we finally obtain

$$\rho \left(z\right)=\frac{1}{\pi }\left[1-\frac{\mid w{\mid }^2}{\sqrt{4\mid w{\mid }^2\mid z{\mid }^2+1}}\right],$$ (5.18)

for z in the region Eq. (5.15). After the proper rescaling this yields Eq. (2.37).

We now turn to the calculation of 〈∥x(t)∥²〉_J, using (2.25). To calculate the trace in the denominator of Eq. (2.25), first note that for Eq. (5.1) the expansion ${(z-M)}^{-1}={\sum }_{n=0}^{N-1}\frac{M^n}{z^{n+1}}$ terminates and is exact, yielding.

$${\left[\frac{1}{z-M}\right]}_{i,j}=\frac{1}{z}{\left(\frac{w}{z}\right)}^{j-i},$$ (5.19)

for j ≥ i, and zero otherwise. Turning the sums in the trace into a sum over the nonzero diagonals of Eq. (5.19) we obtain

$$\mathrm{tr}\left(\frac{1}{{\stackrel{‒}{z}}_2-M^{†}}\frac{1}{z_1-M}\right)=\frac{1}{{\stackrel{‒}{z}}_2{z}_1}\sum _{n=0}^{N-1}(1-\frac{n}{N})q^n$$ (5.20)

where $q\equiv \mid w{\mid }^2∕\left({\stackrel{‒}{z}}_2{z}_1\right)$ and z_i = γ + iω_i. The condition of stability of Eq. (2.2) requires the entire spectrum of −γ1+A = −γ1+M +J to be to the left of the imaginary axis. By Eq. (5.15), this requires $\gamma >\sqrt{\mid w{\mid }^2+1}>\mid w\mid$. It follows that |q| < 1, and therefore the geometric series Eq. (5.20) converges as N → ∞. Summing the series and retaining terms of leading order as N → ∞, we obtain

$$\mathrm{tr}\left(\frac{1}{{\stackrel{‒}{z}}_2-M^{†}}\frac{1}{z_1-M}\right)=\frac{1}{{\stackrel{‒}{z}}_2{z}_1-\mid w{\mid }^2}.$$ (5.21)

If we set the initial condition x₀ in Eqs. (2.25) (or the input amplitude I₀ in Eq. (2.31)) to e_N = (0, · · · , 0, 1), and use Eq. (5.19), we find that the numerator in Eq. (2.25) is also given by the right hand side of Eq. (5.21). Using this and Eqs. (5.21), we obtain

$$F(z_1,z_2)=\frac{1}{{\stackrel{‒}{z}}_2{z}_1-\mid w{\mid }^2-1},$$ (5.22)

for the integrand of Eq. (2.25) which we denoted by F (z₁, z₂), with z_i = γ + iω_i, (i = 1, 2). By comparing the integrand of Eq. (2.25) with Eq. (2.33), we see that to obtain the total power spectrum for the input amplitude I₀ = I₀(0, · · · , 0, 1)^T, we need to multiply Eq. (5.22) by $I_0^2\parallel {\mathbf{I}}_0{\parallel }^2$, and substitute z₁ = z₂ = γ +iω. With the proper rescaling, this yields Eq. (2.39) for general σ. To obtain the formula for 〈∥x(t)∥²_J, we substitute Eq. (5.22) with z_i = γ + iω_i for the integrand of Eq. (2.25). Changing the integration variables by ω₁ = Ω + ω/2 and ω₂ = Ω − ω/2, we obtain

$$\begin{array}{cc}\hfill {\langle \parallel \mathbf{x}\left(t\right){\parallel }^2\rangle }_J& =\int \frac{d\omega }{2\pi }{e}^{it\omega }\int \frac{d\Omega }{2\pi }\frac{1}{{\Omega }^2+{(\gamma +i\omega ∕2)}^2-\mid w{\mid }^2-1},\hfill \\ \hfill & =\frac{1}{2}\int \frac{d\omega }{2\pi }\frac{e^{it\omega }}{\sqrt{{(\gamma +i\omega ∕2)}^2}-\mid w{\mid }^2-1}.\hfill \end{array}$$ (5.23)

Finally consulting a table of Laplace transforms [51], we obtain

$${\langle \parallel \mathbf{x}\left(t\right){\parallel }^2\rangle }_J=e^{-2\gamma t}{I}_0(2t\sqrt{\mid w{\mid }^2}+1),(t\ge 0)$$ (5.24)

where I₀(x) is the 0-th modified Bessel function. Implementing the rescalings t → tσ, γ → γ/σ and w → w/σ, we obtain Eq. (2.38).

B. N/2 feedforward chains of length 2

Here we carry out the explicit calculations for the example of Sec. II C 2 where M is given by Eq. (2.40) (without loss of generality, we assume M has its Schur form), using formulae (2.8)–(2.9) for the spectral density and Eq. (2.25) for 〈∥x(t)∥²〉_J. First we will calculate the eigenvalue density. From Eq. (2.40), $K\equiv M_z{M}_z^{†}=(z-M){(z-M)}^{†}$ (we are setting L = R = 1 in Eq. (2.6); see the comments at the beginning of this section) is a block-diagonal matrix with 2 × 2 diagonal blocks, with the b-th block (b = 1, …, N/2) given by

$$\left(\begin{array}{cc}\hfill z\hfill & \hfill -w_b\hfill \\ \hfill 0\hfill & \hfill z\hfill \end{array}\right)\left(\begin{array}{cc}\hfill \stackrel{‒}{z}\hfill & \hfill 0\hfill \\ \hfill -{\stackrel{‒}{w}}_b\hfill & \hfill \stackrel{‒}{z}\hfill \end{array}\right)=\left(\begin{array}{cc}\hfill \mid z{\mid }^2+\mid w_b{\mid }^2\hfill & \hfill -w_b\stackrel{‒}{z}\hfill \\ \hfill -{\stackrel{‒}{w}}_{b}z\hfill & \hfill \mid z{\mid }^2\hfill \end{array}\right),$$ (5.25)

where w_b is the corresponding Schur weight in Eq. (2.40). Likewise, (K + g²)⁻¹ whose trace appears in Eqs. (2.9) is given by a block-diagonal matrix with diagonal blocks $\frac{1}{{(\mid z{\mid }^2+g^2)}^2+\mid w_b{\mid }^2{g}^2}\left(\begin{array}{cc}\hfill \mid z{\mid }^2+g^2\hfill & \hfill w_b\stackrel{‒}{z}\hfill \\ \hfill {\stackrel{‒}{w}}_{b}z\hfill & \hfill \mid z{\mid }^2+g^2+\mid w_b{\mid }^2\hfill \end{array}\right)$ . Taking the normalized trace we thus obtain

$$\mathrm{tr}{(K+g^2)}^{-1}={\langle \frac{\mid z{\mid }^2+g^2+\frac{1}{2}\mid w_b{\mid }^2}{{(\mid z{\mid }^2+g^2)}^2+\mid w_b{\mid }^2{g}^2}\rangle }_b,$$ (5.26)

where (·)_b means averaging over the N/2 blocks, i.e. ${\langle f\left(w_b\right)\rangle }_b\equiv \frac{1}{N∕2}{\sum }_{b=1}^{N∕2}f\left(w_b\right)$.

Let us first calculate the support boundary of ρ(z). As discussed in Sec. II A, when (for |z| ≠ 0) all singular values of M_z = z − M are bounded from below as N → ∞, the support is correctly given by Eq. (2.5) (we will discuss cases in which some s_i(z) are o(1) further below). Setting g = 0 in Eq. (5.26), and substituting in Eq. (2.5), this yields

$$1\le \mathrm{tr}{K}^{-1}=\mid z{\mid }^{-2}+{\mu }^2\mid z{\mid }^{-4},$$ (5.27)

where we defined ${\mu }^2=\frac{1}{2}{\langle \mid w_b{\mid }^2\rangle }_b=\mathrm{tr}\left(M^{†}M\right)$. It follows that the support is the disk |z| ≤ r₀, where

$$r_0^2=\frac{1}{2}+\sqrt{\frac{1}{4}+{\mu }^2}.$$ (5.28)

The replacements µ → µ/σ and r₀ → r₀/σ then yield Eq. (2.44).

From Eqs. (2.9) and (5.26), within the support, g²(z) is found by solving the equation

$$\mathrm{tr}\frac{1}{K+g^2}={\langle \frac{\mid z{\mid }^2+g^2+\frac{1}{2}\mid w_b{\mid }^2}{{(\mid z{\mid }^2+g^2)}^2+\mid w_b{\mid }^2{g}^2}\rangle }_b=1,$$ (5.29)

while for |z| > r₀ we have g(z) = 0. It is clear from Eq. (5.29) that g²(z) depends on z and $\stackrel{‒}{z}$ only through |z| ≡ r. From Eq. (2.8), within its support the eigenvalue density is given by

$$\begin{array}{cc}\hfill \pi \rho \left(z\right)& =\frac{\partial }{\partial \stackrel{‒}{z}}\mathrm{tr}\left[M_z^{†}{(K+g^2)}^{-1}\right]\hfill \\ \hfill & =1-\frac{\partial }{\partial \stackrel{‒}{z}}\mathrm{tr}\left[M^{†}{(K+g^2)}^{-1}\right],\hfill \end{array}$$ (5.30)

where we are now using the short-hand g² = g²(|z|) (the solution of Eq. (5.29)), and in writing the second line we used $M_z^{†}=\stackrel{‒}{z}-M^{†}$ and Eq. (5.29). From Eqs. (2.40) and (5.26) we see that $M^{†}{(K+g^2)}^{-1}$ has the same block-diagonal structure as Eq. (2.40), and a short calculation shows that $\mathrm{tr}\left[M^{†}{(K+g^2)}^{-1}\right]=\stackrel{‒}{z}$, where we defined

$$I_3\left(r\right)\equiv {\langle \frac{\frac{1}{2}\mid w_b{\mid }^2}{{(r^2+g{\left(r\right)}^2)}^2+\mid w_b{\mid }^{2}g{\left(r\right)}^2}\rangle }_b.$$ (5.31)

I₃(r) is manifestly positive (assuming some w_b are nonzero), while when g² > 0, from Eq. (5.29) we have I₃(r) ≤ 1, and thus

$$0<I_3\left(r\right)\le 1.$$ (5.32)

Replacing this in Eq. (5.30), and using $\stackrel{‒}{z}\frac{\partial f(\mid z\mid )}{\partial \stackrel{‒}{z}}=2r\frac{\partial f\left(r\right)}{\partial r}{\mid }_{r=\mid z\mid }$, we obtain

$$\pi \rho \left(z\right)=\frac{1}{2r}\frac{\partial }{\partial r}[r^2-r^2{I}_3\left(r\right)],$$ (5.33)

for r = |z| ≤ r₀, and zero otherwise; the spectral density is rotationally symmetric and depends only on r = |z|. The advantage of writing the density as a complete derivative is that it can be immediately integrated to yield n_< (r), the proportion of eigenvalues with modulus smaller than some radius r. We have $n_{<}\left(r\right)=2\pi {\int }_0^r\rho \left(r^{\prime }\right)r^{\prime }d{r}^{\prime }$, which upon substitution of Eq. (5.33), yields

$$n_{<}\left(r\right)=r^2(1-I_3\left(r\right))(r\le r_0).$$ (5.34)

Likewise, we define n_> (r) ≡ 1 − n_< (r) to be the proportion of eigenvalues with modulus larger than r. From these definitions we have

$$\rho \left(r\right)=\frac{1}{2\pi r}\frac{\partial n_{<}\left(r\right)}{\partial r}=-\frac{1}{2\pi r}\frac{\partial n_{>}\left(r\right)}{\partial r},$$ (5.35)

and from Eqs. (5.34) and n_> (r) = 1 − n_< (r), after some manipulation exploiting Eq. (5.29), we obtain

$$n_{>}\left(r\right)=g{\left(r\right)}^2(1+I_3\left(r\right)).$$ (5.36)

We see that beyond the radius r at which g² vanishes (which when all w_b’s are bounded is r = r₀), n_> (r) and $\rho \left(r\right)=-\frac{1}{2\pi r}\frac{{{\partial }_n}_{>}}{\partial r}$ vanish identically, while for smaller r they are positive.

In cases in which some w_b grow without bound as N → ∞, some singular values, s_i(z), of M_z = z − M are o(1), and more care is needed. First, to see this, note that by definition s_i(z)² are the eigenvalues of the block-diagonal $K=M_z{M}_z^{†}$; thus they come in pairs composed of the eigenvalues of K’s 2 × 2 blocks, given by Eq. (5.25). We denote the pair of eigenvalues corresponding to block b by s_b, (z)², with the plus and minus subscripts denoting the larger and smaller singular value, respectively. The sum s_b+(z)² + s_b (z)² and the product s_b+ (z)²s_b (z)² are given by the trace and determinant of Eq. (5.25), i.e. by |w_b|² + 2|z|² and |z|⁴, respectively. It follows that for blocks where the feedforward weight w_b is O(1), both s_b,±(z) will be Θ(1) for |z| ≠ 0, while for blocks in which as N, we have

$$s_{b+}^2\left(z\right)=\mid w_b{\mid }^2+O\left(1\right)\to \infty$$ (5.37)

$$s_{b-}^2\left(z\right)\approx \frac{\mid z{\mid }^4}{\mid w_b{\mid }^2}=o\left(1\right).$$ (5.38)

(Note that as stated after Eq. (2.3) we assume $\parallel M{\parallel }_{\mathrm{F}}^2={\mu }^2={\langle \mid w_b{\mid }^2\rangle }_b∕2$ is O(1), so that at most o(N) number of weights can be unbounded, and each such w_b can at most be $O\left(\sqrt{N}\right)$.) If all the w_b are O(1), and hence all singular values are Θ(1) (for |z| ≠ 0), Eq. (5.28) yields the correct support radius as noted above, and for r ≤ r₀, Eq. (5.29) yields a Θ(1) solution for g(r), which leads to a Θ(1) solution for n_> (r) and ρ(r) via Eqs. (5.36)–(5.35). In cases in which some w_b are unbounded, however, Eq. (5.28) (derived from Eq. (2.5)) may not yield the correct support boundary. Such cases are examples of the highly nonnormal cases mentioned in the general discussion after Eq. (2.12), for which the support of lim_{N →∞} ρ(z) must be found by using Eqs. (2.19)–(2.20). This is equivalent to solving Eq. (5.29) after the limit N → ∞ is taken (assuming g² > 0), and then finding where the solution for g²(|z|) vanishes, which yields the correct support radius. From Eq. (5.36) this is indeed the radius at which lim_{N →∞} n_>(r) and hence lim_{N →∞} ρ(r) vanish as well. This radius is in general smaller than r₀ as given by Eq. (5.28).

We now calculate ρ(z) for two specific examples of M from each group. The first example is that of equal and O(1) feedforward weights in all blocks, which we denote by w (in terms of Eq. (2.41), this case corresponds to K = w1). Here we can drop the block averages in Eqs. (5.29) and (5.31), replacing w_b with w. Solving Eqs. (5.29) for g²(|z| = r) we find

$$g^2\left(r\right)=\frac{1}{2}-\frac{w}{2}-r^2+\frac{1}{2}\sqrt{1+w^4+4w^2{r}^2}.$$ (5.39)

Substituting this into Eq. (5.31) and Eq. (5.34) yields

$$n_{<}\left(r\right)=r^2-\frac{w^2{r}^2}{1+\sqrt{1+w^4+4w^2{r}^2}}.$$ (5.40)

The replacements w → w/σ and r → r/σ then yield Eq. (2.45) for general σ, and ρ(r) can be caclulated using Eq. (5.35).

The second case is that of Eq. (2.42). In this case only one of the blocks has a nonzero Schur weight given by |w₁|² = Tr (M ^†M ) = N µ² = O(N ), where µ = O(1) is given by Eq. (2.43). Equation (5.29) now yields

$$1=\frac{1}{r^2+g^2}+\frac{{\mu }^2}{{(r^2+g^2)}^2+N{\mu }^2{g}^2}\cdot \frac{r^2-g^2}{r^2+g^2},$$ (5.41)

or

$$\frac{r^2+g^2-1}{r^2-g^2}=\frac{{\mu }^2}{{(r^2+g^2)}^2+N{\mu }^2{g}^2}.$$ (5.42)

The right hand side of this last equation is I₃(r), as follows from Eq. (5.31); thus using Eq. (5.42) we can rewrite Eq. (5.36) as

$$n_{>}\left(r\right)=g^2\left(r\right)\frac{2r^2-1}{r^2-g^2\left(r\right)}.$$ (5.43)

Let us now solve Eq. (5.42) to find g(r)². As noted above, and in accordance with the general prescription given after Eq. (2.12), for the purpose of obtaining lim_{N →∞} ρ(z) we have to first take the N → ∞ limit in Eq. (5.41), keeping g² > 0 fixed, and only then solve for g². Doing so makes the last term in Eq. (5.41) vanish, and we obtain g²(r) = 1 − r². This is positive for r ≤ 1 and vanishes at r = 1, the correct support radius of lim_{N →∞} ρ(z), which is strictly smaller than r₀ given by Eq. (5.28). From Eq. (5.43) we obtain n_> (r) = g (r) = 1 − r. It then follows from Eq. (5.35) that the N → ∞ limit of the eigenvalue density is identical with the circular law (the result for the M = 0), i.e. ${\lim }_{N\to \infty }\rho \left(r\right)=\frac{1}{\pi }$ for r ≤ 1 and zero otherwise. With the correct scaling, this yields Eq. (2.46).

Contrary to the general prescription given after Eq. (2.12), we will now solve equations Eqs. (5.29), (5.31) and Eq. (5.36) for r > 1, without taking the limit N → ∞ first. As we will see, the obtained solution for g(r)², and by Eqs. (5.32) and (5.36) therefore the solutions for n_>(r) and ρ(r), will be nonzero but o(1) for 1 < r ≤ r₀. As discussed in Sec. II C 2, these finite-size corrections, which in general are not trustworthy, in the present case are in surprisingly good agreement with simulations for some range of r’s beyond r_Θ(1), but deviate from the true n_>(r) for larger r (see Fig. 7). At finite N, it can indeed be checked that Eq. (5.29) has a positive solution for g² if and only if r < r₀, with r₀ given by Eq. (5.28). Simplifying Eq. (5.42) yields a cubic equation in g². However, it turns out that ignoring the cubic term in g² is harmless for large N ; the quadratic approximation has the positive solution

$$g^2\left(r\right)=\sqrt{{\left[\frac{1-r^2}{2}\right]}^2+\frac{r^2(r^2+{\mu }^2)r^6}{{\mu }^{2}N}}+\frac{1-r^2}{2},$$ (5.44)

and for all r < r₀, corrections to Eq. (5.44) when the cubic term is reinstated decay faster than the leading contribution from Eq. (5.44) (nevertheless we numerically solved the full cubic equation (5.29) to obtain the black curve in Fig. 7, and the blue trace in Fig. 6). First, analyzing Eq. (5.44) we see that g²(r) is indeed Θ(1) only for r < 1, where as we already found g²(r) = 1 − r² + o(1). Furthermore, for a fixed r > 1 (such that r − 1 does not vanish as N → ∞), the solution for g(r) is O(N ⁻¹). Thus from Eq. (5.43) we thus see that N n_> (r), i.e. the total number of eigenvalues with modulus larger than r, for 1 < r < r₀ (and r − 1 = Θ(1)) is only O(1); the solution for N n_> (r) is shown in Fig. 7. Correspondingly, from Eqs. (5.43) and (5.35) we see that ρ(r) is o(1) in this region and vanishes in the limit N → ∞, as already found.

Now let us calculate the total number of eigenvalues lying outside the circle |z| = 1. This is given by N n_> (1). From Eq. (5.44) we find $g^2\left(1\right)=\frac{1}{\sqrt{N}}$, and substituting in Eq. (5.43) we obtain

$$N_{>}\left(1\right)\equiv N{n}_{>}\left(1\right)=\sqrt{N}+O\left(1\right).$$ (5.45)

With the proper rescaling this yields Eq. (2.47) for general σ. Note that, according to Eq. (5.44), g(r) (and hence n_> (r)) remains Θ(N ^{− 1/2}) (as opposed to O(N ⁻¹)) in a thin boundary layer outside of width Θ(N ^−1/2) just outside of the circle |z| = 1.

We will now work out the formula for 〈∥x(t)∥²〉_J, Eqs. (2.25)–(2.21), when the initial condition x₀ is the second Schur-vector in block b = a, which we denote by e_a2; in the Schur representation, Eq. (2.40), we have e_a2 = (0, 1)^T (we only write the components of e_a2 in block a). To calculate the numerator in Eq. (2.25) we first calculate (z − T_a2) where $T_a=\left(\begin{array}{cc}\hfill 0\hfill & \hfill w_a\hfill \\ \hfill 0\hfill & \hfill 0\hfill \end{array}\right)$ denotes the a-th diagonal 2 × 2 block of Eq. (2.40). Since $T_a^2=0$, we have ${(z-T_a)}^{-1}=z^{-1}+z^{-2}{T}_a$, which yields v_a(z) ≡ (z − T_a)⁻¹e_a2 = (w_az⁻², z⁻¹)^T. We thus obtain

$${\mathbf{x}}_0^{\mathrm{T}}\frac{1}{{\stackrel{‒}{z}}_2-M^{†}}\frac{1}{z_1-M}{\mathbf{x}}_0={\mathbf{v}}_a{\left(z_2\right)}^{†}{\mathbf{v}}_a\left(z_1\right)=\frac{1}{z_1{\stackrel{‒}{z}}_2}+\frac{\mid w_a{\mid }^2}{z_1^2{\stackrel{‒}{z}}_2^2}.$$ (5.46)

On the other hand, we have

$$\mathrm{tr}\frac{1}{{\stackrel{‒}{z}}_2-M^{†}}\frac{1}{z_1-M}={\langle \frac{1}{2}{\mathrm{Tr}}_{2\times 2}{({\stackrel{‒}{z}}_2-T_b^{†})}^{-1}{(z_1-T_b)}^{-1}\rangle }_b=\frac{1}{z_1{\stackrel{‒}{z}}_2}+\frac{{\langle \mid w_b{\mid }^2\rangle }_b∕2}{z_1^2{\stackrel{‒}{z}}_2^2}.$$ (5.47)

Substituting Eqs. (5.46)–(5.47) in Eq. (2.25) we obtain

$$F(z_1,z_2)=\frac{z_1{\stackrel{‒}{z}}_2+\mid w_a{\mid }^2}{{\left(z_1{\stackrel{‒}{z}}_2\right)}^2-(z_1{\stackrel{‒}{z}}_2+{\mu }^2)}.$$ (5.48)

where we used µ² = (|w_b|²)_b//2, and we denoted the integrand of Eq. (2.25) by F (z₁, z₂) with z_i = γ + iω_i, (i = 1, 2). By comparing the integrand of Eq. (2.25) with Eq. (2.33), we see that substituting z₁ = z₂ = γ + iω into Eq. (5.48) yields the total power spectrum, ${\langle \overline{\parallel {\mathbf{x}}_w{\parallel }^2}\rangle }_J$. After the proper rescalings, this yields Eq. (2.50) for general σ. To obtain ${\langle \parallel \mathbf{x}\left(t\right){\parallel }^2\rangle }_J$, on the other hand, we should substitute Eq. (5.48) into Eq. (2.21) with z_i = γ+iω_i. Let us use the change of variables ω₁ = Ω+ω and ω₂ = Ω − ω. Then we have $z_1{\stackrel{‒}{z}}_2=\Omega +{(\gamma +i\omega )}^2$ and from Eq. (2.21) we obtain

$${\langle \parallel \mathbf{x}\left(t\right){\parallel }^2\rangle }_J=\int \frac{d\omega }{2\pi }{e}^{2it\omega }{f}_a(\gamma +i\omega )$$ (5.49)

where we defined

$$f_a\left(u\right)\equiv 2\int \frac{d\Omega }{2\pi }\frac{{\Omega }^2+u^2+\mid w_a{\mid }^2}{{({\Omega }^2+u^2)}^2-({\Omega }^2+u^2+{\mu }^2)}.$$ (5.50)

Let us rewrite the integrand in Eq. (5.50) as

$$\begin{array}{c}\hfill \frac{{\Omega }^2+u^2+\mid w_a{\mid }^2}{({\Omega }^2+u^2-r_0^2)({\Omega }^2+u^2+r_1^2)}=\frac{{\Omega }^2+u^2+\mid w_a{\mid }^2}{r_0^2+r_1^2}\times \hfill \\ \hfill \left[\frac{1}{{\Omega }^2+u^2-r_0^2}-\frac{1}{{\Omega }^2+u^2+r_1^2}\right],\hfill \end{array}$$ (5.51)

where $r_0^2$ was defined in Eq. (5.28) and

$$r_1^2\equiv r_0^2-1\ge 0.$$ (5.52)

One can calculate the integral over Ω in Eq. (5.50) by contour integration, closing the contour, say, in the upper half of complex plane. The poles of the first and the second terms on the second line of Eq. (5.51) are located at ${\Omega }_{0,\pm }=\pm i\sqrt{u^2-r_0^2}$ and ${\Omega }_{1,\pm }=\pm i\sqrt{u^2+r_1^2}$, respectively. For u = γ + iω (γ > 0) the roots falling in the upper half plane are Ω_0,+ and Ω_1,+, independently of ω. From their residues we obtain

$$f_a\left(u\right)=\frac{1}{r_0^2+r_1^2}\left[\frac{r_0^2+\mid w_a{\mid }^2}{\sqrt{u^2-r_0^2}}+\frac{r_1^2-\mid w_a{\mid }^2}{\sqrt{u^2+r_1^2}}\right].$$ (5.53)

The integral of Eq. (5.53) in Eq. (5.49) is essentially the inverse Laplace transform of Eq. (5.53). Consulting a table of Laplace transforms [51] yields

$$\begin{array}{cc}\hfill & {\langle \parallel \mathbf{x}\left(t\right){\parallel }^2\rangle }_J=\hfill \\ \hfill & e^{-2\gamma t}\left[\frac{r_0^2+\mid w_a{\mid }^2}{r_0^2+r_1^2}{I}_0\left(2r_{0}t\right)+\frac{r_1^2-\mid w_a{\mid }^2}{r_0^2+r_1^2}{J}_0\left(2r_{1}t\right)\right].\hfill \end{array}$$ (5.54)

where J₀(x) (I₀(x)) is the 0-th Bessel function (modified Bessel function). From Eqs. (5.28) and (5.52) it follows that $r_0^2+r_1^2=\sqrt{1+4{\mu }^2}$, and using ${\mu }^2{\langle \mid w_b{\mid }^2\rangle }_b∕2$ once again, we obtain

$${\langle \parallel \mathbf{x}\left(t\right){\parallel }^2\rangle }_J=\left[\frac{1+C_a}{2}{I}_0\left(2r_{0}t\right)+\frac{1-C_a}{2}{J}_0\left(2r_{1}t\right)\right]e^{-2\gamma t}$$ (5.55)

where we defined

$$C_a\equiv \frac{1+2\mid w_a{\mid }^2}{\sqrt{1+2{\langle \mid w_a{\mid }^2\rangle }_b}}.$$ (5.56)

Effecting the proper rescalings we obtain the result for general σ, Eqs. (2.48)–(2.49).

C. Network with different neural types and independent, factorizable weights

Here we carry out the explicit calculations for the network with C neural types presented Sec. II C 2, with M, L and R are given by Eqs. (2.52)–(2.55). From Eqs. (2.6) and (2.52)–(2.55) we obtain M_z = z(RL)⁻¹ − suu^T , and

$$M_z{M}_z^{†}=\mid z{\mid }^2{\left(RL\right)}^{-2}-z\mathit{s}{\mathbf{vu}}^{\mathrm{T}}-\stackrel{‒}{z}\mathit{s}{\mathbf{vu}}^{\mathrm{T}}+{\mathit{s}}^2{\mathbf{uu}}^{\mathrm{T}},$$ (5.57)

where we defined v ≡ (RL)⁻¹u. Using the Woodbury matrix identity we can write

$$\frac{1}{g^2+M_z{M}_z^{†}}=Q-QU\frac{1}{D^{-1}+U^{†}QU}{U}^{†}Q$$ (5.58)

where we defined the N × 2 matrix U = ⁽u , v⁾, and

$$Q\equiv \frac{1}{g^2+\mid z{\mid }^2{\left(RL\right)}^{-2}}$$ (5.59)

$$D\equiv \left(\begin{array}{cc}\hfill s^2\hfill & \hfill -\stackrel{‒}{z}s\hfill \\ \hfill -zs\hfill & \hfill 0\hfill \end{array}\right).$$ (5.60)

We will argue that for $g>0,\mathrm{tr}\left(g^2\right)+M_z{M_z^{†}}^{-1}=\mathrm{tr}Q$ up to o(1) corrections. From Eq. (5.58), for the remainder $\Delta (g,z)\equiv {(g^2+M_z{M}_z^{†})}^{-1}-\mathrm{tr}Q$, we obtain

$$\Delta (g,z)=-\frac{1}{N}\text{Tr}\left[\frac{U^{†}{Q}^{2}U}{D^{-1}+U^{†}QU}\right]$$ (5.61)

where the trace is now over 2 × 2 matrices. We have

$$D^{-1}=-\left(\begin{array}{cc}\hfill 0\hfill & \hfill {\left(zs\right)}^{-1}\hfill \\ \hfill {\left(\stackrel{‒}{z}s\right)}^{-1}\hfill & \hfill \mid z{\mid }^{-2}\hfill \end{array}\right)$$ (5.62)

and for n = 1, 2 we obtain

$$U^{†}{Q}^{n}U=\left(\begin{array}{cc}\hfill {\mathbf{u}}^{†}{Q}^n\mathbf{u}\hfill & \hfill {\mathbf{u}}^{†}{Q}^n\mathbf{v}\hfill \\ \hfill {\mathbf{u}}^{†}{Q}^n\mathbf{v}\hfill & \hfill {\mathbf{v}}^{†}{Q}^n\mathbf{v}\hfill \end{array}\right)=\left(\begin{array}{cc}\hfill I_{n,0}\hfill & \hfill I_{n,1}\hfill \\ \hfill I_{n,1}\hfill & \hfill I_{n,2}\hfill \end{array}\right),$$ (5.63)

where

$$I_{n,k}(g,z)\equiv \frac{1}{N}\sum _{i=1}^N\frac{{\left(l_{c\left(i\right)}{r}_{c\left(i\right)}\right)}^{-k}}{{[g^2+\mid z{\mid }^2{\left(l_{c\left(i\right)}{r}_{c\left(i\right)}\right)}^{-2}]}^n}$$ (5.64)

$$={\langle \frac{{\sigma }_c^{-k}}{{(g^2+{\sigma }_c^{-2}\mid z{\mid }^2)}^n}\rangle }_c$$ (5.65)

and we are using the notation Eq. (2.59) (we will drop the explicit g and z dependence of I_n,k when convenient). Note that all I_n,k (g, z) are O(1) and for even k are positive. Inverting D⁻¹ + U ^†QU we obtain

$$\Delta (g,z)=\frac{1}{N}\frac{T(g,z)}{-\text{det}(D^{-1}+U^{†}QU)}$$ (5.66)

where

$$\begin{array}{cc}\hfill & T(g,z)\equiv \text{Tr}\left[\left(\begin{array}{cc}\hfill I_{2,0}\hfill & \hfill I_{2,1}\hfill \\ \hfill I_{2,1}\hfill & \hfill I_{2,2}\hfill \end{array}\right)\left(\begin{array}{cc}\hfill I_{1,2}-\mid z{\mid }^{-2}\hfill & \hfill {\left(zs\right)}^{-1}-I_{1,1}\hfill \\ \hfill {\left(\stackrel{‒}{z}s\right)}^{-1}-I_{1,1}\hfill & \hfill I_{1,0}\hfill \end{array}\right)\right]\hfill \\ \hfill & I_{2,2}{I}_{1,0}+I_{2,0}\left(I_{1,2}-\frac{1}{\mid z{\mid }^2}\right)-2I_{2,1}\left(I_{1,1}-\frac{1}{s\mathrm{Re}z}\right)\hfill \end{array}$$

and

$$\begin{array}{cc}\hfill -\text{det}(D^{-1}+U^{†}QU)& =I_{1,0}\left(\frac{1}{\mid z{\mid }^2}-I_{1,2}\right)+{\mid I_{1,1}-\frac{1}{sz}\mid }^2\hfill \\ \hfill & =\frac{g^2}{\mid z{\mid }^2}{I}_{1,0}^2+{\mid I_{1,1}-\frac{1}{sz}\mid }^2.\hfill \end{array}$$ (5.67)

We see that both T (g, z) and −det(D⁻¹ + U ^†QU ) are O(1) (to obtain their limits as N → ∞ we can set s⁻¹ = O(N ^−1/2) equal to zero) and since $-\det \left(D^{-1}\right)(D^{-1}+U^{†}QU)\ge \frac{g^2}{\mid z{\mid }^2}{I}_{1,0}^2$ and $I_{1,0}^2>0$ we see that for g > 0, the denominator in Eq. (5.66) is bounded away from zero, and hence Δ(g, z) = O(N ⁻¹) and can be safely ignored for g > 0.

We will thus use $\mathrm{tr}{(g^2+M_z{M}_z^{†})}^{-1}=\mathrm{tr}Q+o\left(1\right)$. From Eq. (5.59) we obtain tr Q = I_1,0(g, z) and hence from Eqs. (2.19),

$$K(g,z)=\underset{N\to \infty }{\text{lim}}\text{tr}Q={\langle \frac{1}{g^2+{\sigma }_c^{-2}{r}^2}\rangle }_c$$ (5.68)

where r ≡ |z|. Note that the approximation $\mathrm{tr}\left(g^2\right)+M_z{M_z^{†}}^{-1}=\mathrm{tr}Q$ is equivalent to using $M_z{M}_z^{†}=\mid z{\mid }^2{\left(RL\right)}^{-2}$ instead of the full expression Eq. (5.57) and hence to setting M = 0. Accordingly, the support of the eigenvalue distribution is given by Eq. (2.13), or equivalently by Eq. (2.60), and within this support, g² is depends only on |z| = r and is found by solving Eq. (2.15), or equivalently Eq. (2.62). Similar considerations show that in using Eq. (2.8) to obtain lim_{N →∞} ρ(z) we can set M = 0, yielding an isotropic eigenvalue density. From Eqs. (2.14)–(2.16), the proportion, n_>, of eigenvalues lying a distance larger than r is equal to g²(r), which is found by solving Eq. (2.62). The results Eqs. (2.17)–(2.18) also hold, wherein the normalized sums over i can be replaced with appropriate averages (·)_c.

Let us now go back to the expression for Δ(g, z), and consider the case g = 0. In this case

$$I_{n,k}(0,z)=\mid z{\mid }^{-2n}{\langle {\sigma }_c^{2n-k}\rangle }_c,$$ (5.69)

and we obtain

$$T(g,z)=\mid z{\mid }^{-6}({\langle {\sigma }_c^2\rangle }_c^2-2{\langle {\sigma }_c^3\rangle }_c{\langle {\sigma }_c\rangle }_c)+2{\langle {\sigma }_c^3\rangle }_c\frac{s^{-1}}{\mid z{\mid }^4\text{Re}z}$$ (5.70)

and

$$-\text{det}(D^{-1}+U^{†}QU)={\mid \mid z{\mid }^{-2}{\langle {\sigma }_c\rangle }_c-\frac{s^{-1}}{z}\mid }^2.$$ (5.71)

In the special case in which (σ_c)_c = 0 (this corresponds to the special case of the example Eq. (2.42) with f µ_E − (1 − f )µ_I ∝ u · v = 0, which we considered above), the determinant will have a vanishing limit as N → ∞ (or s⁻¹ → 0). This leads to a finite limit for Δ(0, z) and we obtain

$$\Delta (0,z)=\frac{s^2{\langle {\sigma }_c^2\rangle }^2}{N\mid z{\mid }^4}=\frac{{\xi }^2{\langle {\sigma }_c^2\rangle }^2}{\mid z{\mid }^4},({\langle {\sigma }_c\rangle }_c=0).$$ (5.72)

Adding this to tr Q in the right side of Eq. (5.68), and using the naive formula Eq. (2.5) or K(0, z) = 1 for the spectral boundary, we would have obtained the equation

$$1=\frac{{\langle {\sigma }_c^2\rangle }_c}{r^2}+\frac{{\xi }^2{\langle {\sigma }_c^2\rangle }^2}{r^4}.$$ (5.73)

This in turn yields the radius Eq. (2.61) which is larger than the true boundary of the support of lim_{N →∞} ρ(z) given by Eq. (2.60).

VI. CONCLUSIONS

We have provided a general formula for the eigenvalue density of partly random matrices, i.e. matrices with general mean and non-trivial covariance structure. General formulae have also been derived for the magnitude of impulse response and frequency power spectrum in an N -dimensional linear dynamical system with a coupling given by such partly random matrices. Our theory makes no requirement on the normality of matrices; its applications include therefore the stability and linear response analysis of neural circuits, whose linearized dynamics is always nonnormal. We have demonstrated our theory by tackling analytically two specific neural circuits: a feedforward chain of length N, and a set of randomly coupled feedforward subchains of length 2. A connection has also been revealed between the eigenvalue spectra of dense random matrix perturbations, and the theory of pseudospectra.

The non-crossing diagrammatic method can be used to calculate other quantities of interest for matrix ensembles of the form A = M + LJR, considered here as well; possible examples are direct statistics of eigenvectors [52], or the correlations of the random fluctuations of the eigenvalue density (δρ_J (z)δρ_J (z + w))_J for macroscopic w (i.e. for |w| = Θ(1)). On the other hand, quantities such as the microscopic structure of (δρ_J (z)δρ_J (z + w))_J, e.g. for |w| = Θ(N ^−1/2) with z inside the support, which could be of interest in the study of eigenvalue repulsion are not accessible to the non-crossing approximation. This is also the case, in general, for the statistics of the “outlier” eigenvalues that we discussed after Eq. (2.12) and in the examples of Sec. II C 1 and Sec. II C 2, which may be of importance in practical applications. The calculation of such quantities is possible, for example, by using the replica technique (see e.g. Ref. [53]).

Finally, there are important forms of disorder which are not covered by the general ensemble A = M + LJR with iid, and hence dense, J. Examples of relevance to neuroscientific applications include sparse A [54–56] (note that, e.g., binary matrices with probability of a nonzero weight, p, which is small but Θ(1) as N → ∞ are covered by our formulae; by “sparse” disorder we refer, e.g. to the case p = o(1)), or more general structure of correlations between the elements of A (in the ensemble considered in this article, and for real J, the covariance ${\langle A_{ij}\delta A_{i^{\prime }{j}^{\prime }}\rangle }_J={\left(L{L}^{\mathrm{T}}\right)}_{i{i}^{\prime }}{\left(R^{\mathrm{T}}R\right)}_{j{j}^{\prime }}$ is single rank); the latter is of importance in considering networks with local topologies where, e.g., the matrix A has a banded structure. Generalization to other forms of random disorder is thus an important direction for future research.

Appendix A. Validity of the non-crossing approximation

In this appendix we will give the justification for the non-crossing approximation used in Sec. (III) and (IV). That is, we will show that the only diagrams not suppressed by inverse powers of N are the non-crossing diagrams. We will limit our discussion to the case of the eigenvalue density considered in Sec. (III), but the generalization to the quantities calculated in (IV) is straightforward. As explained after Eq. (3.22), averaging of G(η, z; J ) over the disorder J involves summing over all complete pairings of the factors of J in every term of the expansion Eq. (3.22), with each pairing of each term represented by a diagram as shown in Fig. 11. Each such diagram is composed of a solid directed line (each segment of which represents a factor of ${\mathit{G}}_{ab}^{\alpha \beta }(\eta ,z;0)$, with a number of wavy lines (each representing the expression Eq. (3.21), with different indices) connecting different points on the solid arrow line, and all the internal matrix indices summed over. For the purpose of calculating the eigenvalue density, according to Eqs. (3.16)–(3.18), what we need to calculate is actually tr⁽σ⁺(RL)⁻¹ (G(η, z; J ))^{J )}; thus we can imagine the solid arrow making a loop by closing-in on itself sandwiching σ⁺ ⊗ (RL)⁻¹ (see Fig. 15).

FIG. 15.

(Color online) The orbits (shown by thin red paths) for two diagrams for the spectral density in a complex J ensemble. The non-crossing diagram on top has three orbits: orbit (1) is the external orbit connecting the two ends of the Green’s function, while orbits (2) and (3) are the internal orbits. As in Eqs. (A1) and (A2), they contribute $\mathrm{tr}\left({\sigma }^{+}\mathit{G}(\eta ,z;0){\pi }^{r_1}\mathit{G}(\eta ,z;0)\right),\mathrm{Tr}\left({\pi }^{3-r_1}G(\eta ,z;0){\pi }^{r_2}\mathit{G}(\eta ,z;0)\right)$ and $\mathrm{Tr}\left({\pi }^{3-r_2}\mathit{G}(\eta ,z;0)\right)$ respectively, with r₁ and r₂ summed over 1 and 2 (cf. Eq. (3.21)). The trace contributed by each of the three orbits is O(N ), which when combined with the three factors of 1/N accounting for the two wavy lines and the normalization of the external orbit’s trace, yield an O(1) expression for this diagram. By contrast, the crossing diagram on the right has no internal orbits. Its only external orbit contributes $\mathrm{Tr}\left({\sigma }^{+}\mathit{G}(\eta ,z;0){\pi }^{r_2}\mathit{G}(\eta ,z;0){\pi }^{3-r_1}\mathit{G}(\eta ,z;0){\pi }^{3-r_2}\mathit{G}(\eta ,z;0){\pi }^{r_1}\mathit{G}(\eta ,z;0)\right)$ which after normalization is O(1). Accounting for two factors of 1/N coming from the wavy lines, we then see that this crossing diagram is O(N ⁻²) and hence is suppressed as N → ∞.

Given the structure of the Kronecker deltas in Eq. (3.21), it is more convenient for our purpose here, however, to think of each diagram as a number of “orbits,” each formed by starting somewhere on the solid line and moving on it always along its arrow until the next wavy line is encountered, whereby we leave the solid line, continuing on the wavy line without crossing it (because Eq. (3.21) is composed of two Kronecker deltas, one for each side of the wavy line, enforcing index identification at the corresponding ends on each side [64]) and return somewhere else on the solid line, continuing as before until we reach the initial point (see Fig. 15). As we go around this orbit, for each solid line traversed we write down, from right to left, a G(η, z; 0) and for each wavy line a π^ri (see Eqs. (3.20)) where i is the index of the wavy line. Because all matrix indices are summed over, such adjacent factors multiply like matrices, and since the orbit forms a loop, in the end we obtain the trace of the matrix product thus obtained. (This recipe for assigning the contribution of each orbit accounts for the Kronecker deltas and π^r ’s in Eq. (3.21), but not for the factor $\frac{1}{\sqrt{N}}$ and the sum over r’s; we will account for these, at the end, after Eq. (A2).) A generic orbit, which we refer to as internal, closes on itself after traversing, say, m wavy lines sanwiching m Green’s functions (e.g. the orbits labeled 2 and 3 in panel (a) of Fig. 15), and thus contributes a trace of the form

$$I_{m,\mathit{r}}\equiv \text{Tr}(\mathit{G}(\eta ,z;0){\pi }^{r_{i_1}}\cdots \mathit{G}(\eta ,z;0){\pi }^{r_{i_m}}),$$ (A1)

where r is short-hand for {r_i1 , …, r^im }, and i_k are the indices of the wavy lines traversed in the orbit. In every diagram, there is also exactly one orbit (e.g. the orbits labeled 1 in both panels of Fig. 15) which in addition includes the factor σ⁺(RL)⁻¹ sandwhiched between the two external Green’s functions. This orbit, which we call the external orbit, contributes a trace of the form

$$E_{m,\tilde{\mathit{r}}}\equiv \text{Tr}({\sigma }^{+}{\left(RL\right)}^{-1}\mathit{G}(\eta ,z;0){\pi }^{r_{j_1}}\cdots {\pi }^{r_{j_n}}\mathit{G}(\eta ,z;0))$$ (A2)

where n is the number wavy lines the orbit traverses and $\tilde{\mathbf{r}}$ is short for {r_j1 , …, r_jn }, and j_k are the indices of the wavy lines traversed in this orbit (in writing Eq. (A2) we dropped the $\frac{1}{N}$ that normalizes the trace in Eqs. (3.16), but we will account for it below). For succinctness, in Eqs. (A1)–(A2) we suppressed the arguments (η, z) for I_m,r and $E_{n,\tilde{r}}$ on which they depend. The full expression for the diagram is obtained by multiplying all such trace factors contributed by every orbit in the diagram, as well as a factor of N ^−w−1 where w is the number of wavy lines in the diagrams, to account for the N ⁻¹ in Eq. (3.21) for each wavy line, as well as the extra N ⁻¹ which normalizes the trace in the external orbit Eq. (A2) as dictated by Eq. (3.16). The obtained expression is finally summed over all the r-indices corresponding to each wavy line, as required by Eq. (3.21).

The justification for the non-crossing approximation is based on the claim that each trace contributed by a orbit (external or internal) as in Eqs. (A1)–(A2) is O(N ), irrespective of η, z, m or r. We will provide justification for this claim below. However, accepting it as true, we see that any diagram’s scaling with N solely depends on the number of orbits and wavy lines it contains. A wellknown topological argument then shows that the contributions of crossing diagrams are suppressed by inverse powers of N [57, 58]; for completeness we will summarize this argument here. First note that, assuming the claim, any diagram will yield an expression that is O(N ^α) with

$$\alpha \equiv f-w-1,$$ (A3)

where f is the number of orbits in the diagram (the sum over at most 2^w possible configurations of r_i does not contribute to the scaling with N ). Let V denote the total number of vertices in the diagram (i.e. the number of intersections of wavy lines and the solid line, plus an extra one representing the insertion of σ⁺(RL)⁻¹ in the solid line loop) and let E denote its total number of edges, i.e. E ≡ w + s, where s is the number of solid line segments (s = 5 in both panels of Fig. 15). It is easy to see that V = s. Thus we have E − V = w. Formally defining the number of “faces” in the diagram by F ≡ f + 1, and its “Euler characteristic” by

$$\chi \equiv F-E+V,$$ (A4)

we then find that χ = F − (E − V ) = f + 1 − w. From Eq. (A3) we then obtain

$$\alpha =\chi -2.$$ (A5)

Thus the contribution of a diagram is O(N ^α), with α determined solely by the diagram’s formal “Euler characteristic” via Eq. (A5). It can be shown that a diagram with F formal “faces” and a formal “Euler characteristic” χ as defined above, can be drawn on (embedded in) a two-dimensional oriented surface with Euler characteristic χ, such that no edges (solid or wavy) cross to create new vertices, and each face created on the surface by its partitioning by the drawn diagram, a) is topologically a disk, and b) has a one-to-one correspondence with and is encircled by an orbit in the diagram, where we now count among the orbits, also the loop formed by the solid arrow line. Thus the number of faces on the surface is indeed F = f + 1, and the χ, as defined above for the diagram, indeed agrees with the Euler characteristic of the surface, as conventionally defined. Topologically, such a surface is a generalized torus with g holes, satisfying χ = 2 − 2g; the surface with zero holes is the sphere, or after decompactification, the plane (e.g. the diagram in panel (b) of Fig. 15 can be drawn in this manner on a torus). We thus see that

$$\alpha =-2g,$$ (A6)

and therefore the only diagrams that are not suppressed by inverse powers of N are those that can be drawn, as described above, on the plane. Since we took the area enclosed by the solid arrow line loop as a face by itself, this means that the diagram should be drawable with the wavy lines remaining outside this area (in order not to partition it into several faces) without crossing each other; this is the precise definition of the diagram being non-crossing [65].

Let us now go back to justifying the claim that the traces contributed by the orbits as in Eqs. (A1)–(A2) are O(N ). For this purpose we will make use of the singular value decomposition of M_z introduced in Eq. (3.32). Defining the unitary matrix

$$U_z\equiv \left(\begin{array}{cc}\hfill U_z\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill V_z\hfill \end{array}\right),$$ (A7)

and using Eq. (3.32), we can write H₀(z), defined in Eq. (3.14), as

$$H_0\left(z\right)=U_z{\tilde{H}}_0\left(z\right)U_z^{†},$$ (A8)

where

$${\tilde{H}}_0\left(z\right)\equiv \left(\begin{array}{cc}\hfill 0\hfill & \hfill S_z\hfill \\ \hfill S_z\hfill & \hfill 0\hfill \end{array}\right).$$ (A9)

Let us also define $\tilde{\mathit{G}}(\eta ,z;0)\equiv U_z^{†}\mathit{G}(\eta ,z;0)U_z$, such that

$$\mathit{G}(\eta ,z;0)=U_z\tilde{\mathit{G}}(\eta ,z;0)U_z^{†}.$$ (A10)

Then using the definition G(η, z; 0) = (η − H₀(z))⁻¹, we see that

$$\tilde{\mathit{G}}(\eta ,z;0)=\frac{1}{\eta -{\tilde{H}}_0\left(z\right)}=\left(\begin{array}{cc}\hfill \frac{\eta }{{\eta }^2-S_z^2}\hfill & \hfill \frac{S_z}{{\eta }^2-S_z^2}\hfill \\ \hfill \frac{S_z}{{\eta }^2-S_z^2}\hfill & \hfill \frac{\eta }{{\eta }^2-S_z^2}\hfill \end{array}\right),$$ (A11)

where we used Eq. (A9) to write the last equality. Given the block-diagonal nature of Eq. (A7) and the definitons Eq. (3.20), we also have

$${\pi }^r=U_z{\pi }^r{U}_z^{†}.$$ (A12)

We now substitute G(η, z; 0) and π^ri in Eqs. (A1)–(A2) with the right hand sides of Eqs. (A10) and (A12), respectively. After canceling the U_z ’s we obtain

$$I_{m,\mathit{r}}=\text{Tr}\left(\tilde{\mathit{G}}(\eta ,z;0){\pi }^{r_{i_1}}\cdots \tilde{\mathit{G}}(\eta ,z;0){\pi }^{r_{i_m}}\right),$$ (A13)

$$E_{n,\tilde{\mathit{r}}}=\text{Tr}\left({\sigma }^{+}A\left(z\right)\tilde{\mathit{G}}(\eta ,z;0){\pi }^{r_{j_1}}\cdots {\pi }^{r_{j_n}}\tilde{\mathit{G}}(\eta ,z;0)\right)$$ (A13)

where we defined

$$A\left(z\right)\equiv U_z^{†}{\left(RL\right)}^{-1}{V}_z,$$ (A15)

such that $U_z^{†}[{\sigma }^{+}\otimes {\left(RL\right)}^{-1}]U_z={\sigma }^{+\otimes }A\left(z\right)\equiv {\sigma }^{+}A\left(z\right)$. For the internal orbits, we see from Eq. (A11) that each $\tilde{\mathit{G}}(\eta ,z;0)$, depending on whether it is sandwiched between the same projectors π^r, or between two opposite projectors, π^r and π^3−r, contributes a diagonal factor equal to $\eta ∕({\eta }^2-S_z^2)\mathrm{or}{S}_z∕({\eta }^2-S_z^2)$, respectively. Thus, for any configuration of r_i’s, if the number of Green’s functions sandwiched the second way is k (1 ≤ k ≤ m), we obtain

$$I_{m,\mathit{r}}(\eta ,z)=\sum _{i=1}^N\frac{{\eta }^{m-k}{s}_i{\left(z\right)}^k}{{({\eta }^2-s_i{\left(z\right)}^2)}^m}(1\le k\le m)$$ (A16)

for the internal orbits (in particular, we see that the sole dependence of I_m,r (η, z) on r is via the number k). We therefore have

$$\mid I_{m,\mathit{r}}(\eta ,z)\mid \le N\underset{i}{\text{max}}\mid \frac{{\eta }^{m-k}{s}_i{\left(z\right)}^k}{{({\eta }^2-s_i{\left(z\right)}^2)}^m}\mid .$$ (A17)

When the imaginary part of η is nonzero, the denominator in the right hand side of Eq. (A17) cannot vanish for any value of s_i(z) (while as η → i0, which is the limit we have to take after summing up the relevant diagrammatic series, s_i(z) that approach zero as N grows can make this expression unbounded as N → ∞). Assuming Im η > 0, it will be sufficient for our purposes to substitute Eq. (A17) with the weaker bound

$$\mid I_{m,\mathit{r}}(\eta ,z)\mid \le N\underset{s}{\text{max}}\mid \frac{{\eta }^{m-k}{s}_k}{{({\eta }^2-s^2)}^m}\mid ,(\text{Im}\eta >0)$$ (A18)

where now the maximum is taken for s ranging over the whole [0, ∞). Since Im η > 0 the expression has no singularities at finite real s, and since 2m > k, it cannot diverge as s → ∞ either; thus it has a finite maximum independent of N. More precisely, it is easy to show that ${\max }_s\mid \frac{{\eta }^{m-k}{s}^k}{{({\eta }^2-s^2)}^m}\mid \le \left[\frac{\sqrt{2}}{\mid \mathrm{Im}\eta \mid }\right]$, irrespective of k as long as 1 ≤ k ≤ m, yielding

$$\mid I_{m,\mathit{r}}(\eta ,z)\mid \le N{\left[\frac{\sqrt{2}}{\mid \text{Im}\eta \mid }\right]}^m,(\text{Im}\eta >0).$$ (A19)

Similarly, the trace for the external orbit can be written in the new basis Eq. (A11) as

$$E_{n,\tilde{\mathit{r}}}(\eta ,z)=\sum _{i=1}^N{A}_{ii}\left(z\right)\frac{{\eta }^{n-\tilde{k}}{s}_i{\left(z\right)}^{\tilde{k}}}{{({\eta }^2-s_i{\left(z\right)}^2)}^n},(1\le \tilde{k}\le n),$$ (A20)

where $\tilde{k}$ is the number of Green’s functions in Eq. (A2) sandwiched between two π^r ’s with different superscripts; this convention works correctly for the external orbit as well, if we account for the presence of σ⁺ by imagining a π² (π¹) to the left (right) of the leftmost (rightmost) Green’s function. From Eq. (A15), we can write A_ii(z) = u_i(z)^†(RL)⁻¹v_i(z), where we defined the vectors u_i(z) and v_i(z) to be the i-th column of U_z and V_z, respectively. By the Cauchy-Schwartz inequality we then have

$$\begin{array}{cc}\hfill \mid A_{ii}\left(z\right)\mid & \le \parallel {\mathbf{u}}_i\left(z\right)\parallel \parallel {\left(RL\right)}^{-1}{\mathbf{v}}_i\left(z\right)\parallel \hfill \\ \hfill & \le \parallel {\mathbf{u}}_i\left(z\right)\parallel \parallel {\mathbf{v}}_i\left(z\right)\parallel \parallel {\left(RL\right)}^{-1}\parallel ,\hfill \end{array}$$ (A21)

where ∥(RL)⁻¹∥ is the operator norm, or the maximum singular value, of (RL)⁻¹. But since U_z and V_z are unitary matrices, u_i(z) and v_i(z) are unit vectors, and we obtain

$$\mid A_{ii}\left(z\right)\mid \le \parallel {\left(RL\right)}^{-1}\parallel .$$ (A22)

Going back to Eq. (A20), this yields the bound

$$\mid E_{n,\tilde{\mathit{r}}}(\eta ,z)\mid \le N\parallel {\left(RL\right)}^{-1}\parallel \underset{i}{\text{max}}\mid \frac{{\eta }^{n-\tilde{k}}{s}_i{\left(z\right)}^{\tilde{k}}}{{({\eta }^2-s_i{\left(z\right)}^2)}^n}\mid .$$ (A23)

The only difference with the inequality for I_m,r is the factor ∥(RL)⁻¹∥. Repeating the same argument as for the internal traces, we therefore see that

$$\mid E_{n,\tilde{\mathit{r}}}(\eta ,z)\mid \le N{\left[\frac{\sqrt{2}}{\mid \text{Im}\eta \mid }\right]}^n\parallel {\left(RL\right)}^{-1}\parallel ,(\text{Im}\eta >0),$$ (A24)

and thus a sufficient condition for $E_{n,\tilde{r}}$ to be O(N ) for Im η > 0, is that ∥(RL)⁻¹∥ remains bounded as N → ∞, i.e.

$$\parallel {\left(RL\right)}^{-1}\parallel =O\left(1\right).$$ (A25)

Combining Eqs. (A19) and (A24), and given the prescription after Eq. (A2), we can bound the absolute value of the contribution of a diagram with genus g (or g crossings), w wavy lines, and s solid lines, by $2^w{\left[\frac{\sqrt{2}}{\mid \mathrm{Im}\mid \eta }\right]}^s\parallel {\left(RL\right)}^{-1}\parallel N^{-2g}$ (the power of s is obtained by noting that the powers of m and n in the bounds Eqs. (A19) and (A24), when summed over all orbits must equal s, since every Green’s function or solid line appears in exactly one orbit). Hence for a fixed, nonzero Im η, the contribution of crossing diagrams (i.e. those with g ≥ 1) goes to zero as N → ∞. Thus if we take the limit N → ∞ before the limit η → i0⁺, ignoring the crossing diagrams is safe, and the expression for ρ(z) obtained from Eq. (3.16) after analytic continuation of tr⁽σ⁺(RL)⁻¹G(η, z)⁾ to η = i0, with G(η, z) given by the contribtiuon of non-crossing diagrams to (G(η, z; J ))_J, gives the correct result for lim_{N →∞} ρ(z). We mention that when the smallest singular value s_i(z) remains bounded away from zero as N → ∞, even at η = 0 the traces Eqs. (A16) and (A20) are O(N ), as is not hard to check, justifying the non-crossing approximation at η = 0. Thus it is only when some s_i(z) are o(1) that it becomes important to send η to i0⁺ only after the limit N → ∞ has been taken. In particular, in such cases, applying the limit η → i0⁺ to the results obtained using the non-crossing approximation before taking the limit N → ∞, may yield finite-size contributions to lim_{N →∞} ρ(z), which in general may yield incorrect subleading corrections.

Appendix B. ρ(z) vanishes in the region Eq. (3.38)

In this appendix we prove more rigorously that in the region Eq. (3.38), the eigenvalue density vanishes. More precisely, we prove that ρ(z) ≡ lim_ϵ→0+ lim_{N →∞} ρ_N (z, E) = 0, where

$${\rho }_N(z,ϵ)\equiv \frac{1}{\pi }\frac{\partial }{\partial \stackrel{‒}{z}}\text{tr}\left[\frac{{\left(RL\right)}^{-1}{M}_z^{†}}{M_z{M}_z^{†}+{\gamma }^2}\right],$$ (B1)

is obtained by substituting Eq. (3.35) into Eqs. (3.17). Here, γ = g(z, E) + E, is the solution of Eq. (3.37), which as we argued in Sec. III, vanishes as ϵ → 0⁺ when z is in the region Eq. (3.38) (note that since Eq. (3.37) is defined in the limit N → ∞, γ has no dependence on N ). Recall that for E > 0, g(z, E) is positive and therefore γ > E > 0. Expanding the derivative in Eq. (B1) we obtain

$$\begin{array}{cc}\hfill & \pi {\rho }_N(z,ϵ)=\text{tr}\left[\frac{{\left(RL\right)}^{-1}{\left(RL\right)}^{-†}}{M_z{M}_z^{†}+{\gamma }^2}\right]\hfill \\ \hfill & -\text{tr}\left[{\left(RL\right)}^{-1}{M}_z^{†}\frac{1}{M_z{M}_z^{†}+{\gamma }^2}{M}_z{\left(RL\right)}^{-†}\frac{1}{M_z{M}_z^{†}+{\gamma }^2}\right]\hfill \\ \hfill & -\text{tr}\left[\frac{{\left(RL\right)}^{-1}{M}_z^{†}}{M_z{M}_z^{†}+{\gamma }^2}\frac{1}{M_z{M}_z^{†}+{\gamma }^2}\right]{\partial }_{\stackrel{‒}{z}}\left({\gamma }^2\right)\hfill \\ \hfill & =\text{tr}\left[\frac{{\left(RL\right)}^{-1}Q{\left(RL\right)}^{-†}}{M_z{M}_z^{†}+{\gamma }^2}\right]\hfill \\ \hfill & -\text{tr}\left[\frac{{\left(RL\right)}^{-1}{M}_z^{†}}{M_z{M}_z^{†}+{\gamma }^2}\frac{1}{M_z{M}_z^{†}+{\gamma }^2}\right]{\partial }_{\stackrel{‒}{z}}\left({\gamma }^2\right),\hfill \end{array}$$ (B2)

where we defined $Q=1-M_z^{†}\frac{1}{M_z{M}_z^{†}+{\gamma }^2}{M}_z$ (we suppress the explicit dependence of γ on z for simplicity). By the Woodbury matrix identity $Q=\frac{1}{M_z^{†}{M}_z+{\gamma }^2}$ which upon substitution in Eq. (B2) yields

$$\begin{array}{cc}\hfill \pi {\rho }_N(z,ϵ)=& \text{tr}\left[\frac{{\left(RL\right)}^{-1}}{M_z{M}_z^{†}+{\gamma }^2}\frac{{\left(RL\right)}^{-†}}{M_z{M}_z^{†}+{\gamma }^2}\right]{\gamma }^2\hfill \\ \hfill & -\text{tr}\left[\frac{{\left(RL\right)}^{-1}{M}_z^{†}}{M_z{M}_z^{†}+{\gamma }^2}\frac{1}{M_z{M}_z^{†}+{\gamma }^2}\right]{\partial }_{\stackrel{‒}{z}}\left({\gamma }^2\right).\hfill \end{array}$$ (B3)

Differentiating Eq. (3.37) with respect to $\stackrel{‒}{z}$ yields

$$-{\partial }_{\stackrel{‒}{z}}\left({\gamma }^2\right)=\frac{-2{\gamma }^2{\partial }_{\stackrel{‒}{z}}K}{1-K-2{\gamma }^2{\partial }_{{\gamma }^2}K},$$ (B4)

with the partial derivatives of K(γ, z) given by

$$\begin{array}{cc}\hfill & -{\partial }_{{\gamma }^2}K=T_{\infty }^{>}\left(\gamma \right)\equiv \underset{N\to \infty }{\text{lim}}{T}_N^{>}\left(\gamma \right)\hfill \\ \hfill & -{\partial }_{\stackrel{‒}{z}}K=V_{\infty }\left(\gamma \right)\equiv \underset{N\to \infty }{\text{lim}}{V}_N\left(\gamma \right),\hfill \end{array}$$ (B5)

where we defined

$$T_N^{>}\left(\gamma \right)\equiv \text{tr}\left[\frac{1}{{(M_z{M}_z^{†}+{\gamma }^2)}^2}\right]$$ (B6)

$$V_N\left(\gamma \right)\equiv \text{tr}\left[\frac{1}{M_z{M}_z^{†}+{\gamma }^2}\frac{M_z{\left(RL\right)}^{-†}}{M_z{M}_z^{†}+{\gamma }^2}\right].$$ (B7)

We thus obtain

$$\pi {\rho }_N(z,ϵ)={\gamma }^2{T}_N\left(\gamma \right)+\frac{2{\gamma }^2{V}_{\infty }\left(\gamma \right)V_N{\left(\gamma \right)}^{\ast }}{1-K\left(\gamma \right)+2{\gamma }^2{T}_{\infty }^{>}\left(\gamma \right)}$$ (B8)

where we defined

$$T_N\left(\gamma \right)\equiv \text{tr}\left[\frac{{\left(RL\right)}^{-1}}{M_z^{†}{M}_z+{\gamma }^2}\frac{{\left(RL\right)}^{-†}}{M_z{M}_z^{†}+{\gamma }^2}\right].$$ (B9)

Having eliminated derivatives of γ, we now simply need to show that lim_γ→0+ lim_{N →∞} of the right side of Eq. (B8) vanishes for z is in the region Eq. (3.38) (where γ = 0⁺ is the solution of Eq. (3.37) as ϵ → 0⁺).

We will start by bounding the traces T_N (γ) and V_N (γ) in Eq. (B8). For V_N (γ) we use the singular value decomposition Eq. (3.32):

$$\begin{array}{cc}\hfill \mid V_N\left(\gamma \right)& =\mid \text{tr}\left[\frac{{\left(RL\right)}^{-1}{M}_z^{†}}{M_z{M}_z^{†}+{\gamma }^2}\frac{1}{M_z{M}_z^{†}+{\gamma }^2}\right]\mid \hfill \\ \hfill & =\text{tr}\left[U_z^{†}{\left(RL\right)}^{-1}{V}_z\frac{S_z}{{(S_z^2+{\gamma }^2)}^2}\right]\hfill \\ \hfill & \le \parallel {\left(RL\right)}^{-1}\parallel \text{tr}\left[\frac{S_z}{{(S_z^2+{\gamma }^2)}^2}\right]\hfill \end{array}$$ (B10)

(where in the last line we used Eq. (A22)), i.e.

$$\mid V_N\left(\gamma \right)\mid \le \parallel {\left(RL\right)}^{-1}\parallel V_N^{>}\left(\gamma \right)$$ (B11)

where we defined

$$V_N^{>}\left(\gamma \right)\equiv \frac{1}{N}\sum _{i=0}^N\frac{s_i\left(z\right)}{{(s_i{\left(z\right)}^2+{\gamma }^2)}^2}.$$ (B12)

Taking the limit N → ∞, we obtain from Eq. (B11)

$$\mid V_{\infty }\left(\gamma \right)\mid \le C{V}_{\infty }^{>}\left(\gamma \right)$$ (B13)

where $V_{\infty }^{>}\left(\gamma \right)\equiv {\lim }_{N\to \infty }{V}_N^{>}\left(\gamma \right)$, and C is an upper bound on ∥(LR)⁻¹∥ (which we have assumed is O(1) as N → ∞). To bound T_N (γ), we use the inequality

$$\mid \text{tr}\left(ABCD\right)\mid \le \parallel A\parallel \parallel C\parallel \text{tr}{\left(B{B}^{†}\right)}^{\frac{1}{2}}\text{tr}{\left(D{D}^{†}\right)}^{\frac{1}{2}}.$$ (B14)

This can be derived by first using the Cauchy-Schwartz inequality, |tr (AB)|² ≤ tr (AA^†)tr (BB^†), and then using the inequality |tr (AB)| ≤ ∥B∥tr (A), valid for positive semi-definite A (which in turn follows from the definition of ∥B∥ after unitary diagonalization of A). Using (B14) we obtain

$$\begin{array}{cc}\hfill & \mid \text{tr}\left[\frac{{\left(RL\right)}^{-1}}{M_z^{†}{M}_z+{\gamma }^2}\frac{{\left(RL\right)}^{-†}}{M_z{M}_z^{†}+{\gamma }^2}\right]\mid \le \hfill \\ \hfill & \le \parallel {\left(RL\right)}^{-1}{\parallel }^2\text{tr}\left[{\left(\frac{1}{M_z{M}_z^{†}+{\gamma }^2}\right)}^2\right]\hfill \end{array}$$ (B15)

or

$$\mid T_N\left(\gamma \right)\mid \le \parallel {\left(RL\right)}^{-1}{\parallel }^2{T}_N^{>}\left(\gamma \right).$$ (B16)

Using the inequalities, (B11), (B13) and (B16) in Eq. (B8) we obtain

$$\pi \mid {\rho }_N(z,ϵ)\mid \le C^2\left[{\gamma }^2{T}_N^{>}\left(\gamma \right)+\frac{2{\gamma }^2{V}_{\infty }^{>}\left(\gamma \right)V_N^{>}\left(\gamma \right)}{1-K\left(\gamma \right)+2{\gamma }^2{T}_{\infty }^{>}\left(\gamma \right)}\right]$$ (B17)

Taking the N → ∞ limit (while keeping γ finite), and defining ρ(z, E) ≡ lim_{N →∞} ρ_N (z, E), we obtain

$$\pi \rho (z,ϵ)\le C^2\left[{\gamma }^2{T}_{\infty }^{>}\left(\gamma \right)+\frac{2{\left(\gamma V_{\infty }^{>}\left(\gamma \right)\right)}^2}{1-K\left(\gamma \right)+2{\gamma }^2{T}_{\infty }^{>}\left(\gamma \right)}\right]$$ (B18)

where we defined

$$T_{\infty }^{>}\left(\gamma \right)\equiv \underset{N\to \infty }{\text{lim}}\frac{1}{N}\sum _{i=0}^N\frac{1}{{(s_i{\left(z\right)}^2+{\gamma }^2)}^2}$$ (B19)

$$V_{\infty }^{>}\left(\gamma \right)\equiv \underset{N\to \infty }{\text{lim}}\frac{1}{N}\sum _{i=0}^N\frac{s_i\left(z\right)}{{(s_i{\left(z\right)}^2+{\gamma }^2)}^2}.$$ (B20)

Thus, to show that lim_ϵ→0+ ρ(z, E) = 0, it suffices to show that ${\gamma }^2{T}_{\infty }^{>}\left(\gamma \right)$ and $\gamma V_{\infty }^{>}\left(\gamma \right)$ vanish as γ → 0⁺ (since z is in the region Eq. (3.38), 1−K(γ) and hence the denominator in the last term in Eq. (B18) remains positive as γ → 0⁺). Let us rewrite Eq. (B19) as

$$T_{\infty }^{>}\left(\gamma \right)={\int }_0^{\infty }\frac{{\rho }_{\mathrm{S}}(s;z)ds}{{(s^2+{\gamma }^2)}^2}$$ (B21)

$$V_{\infty }^{>}\left(\gamma \right)={\int }_0^{\infty }\frac{s{\rho }_{\mathrm{S}}(s;z)ds}{{(s^2+{\gamma }^2)}^2}$$ (B22)

where we defined

$${\rho }_{\mathrm{S}}(s;z)=\underset{N\to \infty }{\text{lim}}\frac{1}{N}\sum _{i=0}^N\delta (s-s_i\left(z\right))$$ (B23)

as the limit of the density of the singular values of M_z [66]. Note that contributions to $T_{\infty }^{>}\left(\gamma \right)$ and $V_{\infty }^{>}\left(\gamma \right)$ from integration on [s₀, ∞) for any fixed, nonzero s₀ remain finite as γ → 0⁺; only singular contributions arising from the region s = O(γ) « 1 can contribute to ${\gamma }^2{T}_{\infty }^{>}\left(\gamma \right)$ and ${\gamma }^2{V}_{\infty }^{>}\left(\gamma \right)$ as γ → 0⁺. Thus we only need concern ourselves with the portion of integrals from 0 to some arbitrary small, but fixed s₀, and show that ${\gamma }^2{\int }_0^{s_0}\frac{{\rho }_S(s;z)ds}{{(s^2+{\gamma }^2)}^2}$ and $\gamma {\int }_0^{s_0}\frac{s{\rho }_S(s;z)ds}{{(s^2+{\gamma }^2)}^2}$ vanish as γ → 0. Let us first consider the situation similar to that in the two examples Eqs. (5.1) and (2.42). For those examples, there is a region of z outside Eq. (3.38), where a single (more generally O(1)) singular value s_i(z) vanishes as N → ∞, while all the other s_i(z) remain bounded from below. But an O(1) set of (vanishing) singular values does not contribute to the density Eq. (B23) and since the other s_i(z) are bounded from below, there is an s₀ below which ρ_S (s; z) identically vanishes. So the claim is clearly true for such cases. More generally, we exploit the fact that z is in the region Eq. (3.38), so that

$$\underset{\gamma \to 0^{+}}{\text{lim}}{\int }_0^{\infty }\frac{{\rho }_{\mathrm{S}}(s;z)ds}{s^2+{\gamma }^2}<1.$$ (B24)

We conclude that as s → 0⁺ the density, ρ_S (s; z), must vanish at least as fast as s^α, i.e. it must be O(s^α), for some α > 1; otherwise the integral in Eq. (B24) diverges in the limit. Let us therefore choose s₀ to be small enough such that for s ≤ s₀, ρ(s; z) < cs^α for some constant c and α > 1. It is then an elementary exercise to show that ${\gamma }^2{\int }_0^{s_0}\frac{s^{\alpha }ds}{{(s^2+{\gamma }^2)}^2}$ and ${\gamma }^2{\int }_0^{s_0}\frac{s^{\alpha +1}ds}{{(s^2+{\gamma }^2)}^2}$ are $O\left({\gamma }^{\min (2,\alpha -1)}\right)$ and $O\left({\gamma }^{\min (1,\alpha -1)}\right)$, respectively, as γ → 0⁺, and since α > 1, they both vanish in the limit, proving the claim.