Multiple bumps can enhance robustness to noise in continuous attractor networks

Abstract

A central function of continuous attractor networks is encoding coordinates and accurately updating their values through path integration. To do so, these networks produce localized bumps of activity that move coherently in response to velocity inputs. In the brain, continuous attractors are believed to underlie grid cells and head direction cells, which maintain periodic representations of position and orientation, respectively. These representations can be achieved with any number of activity bumps, and the consequences of having more or fewer bumps are unclear. We address this knowledge gap by constructing 1D ring attractor networks with different bump numbers and characterizing their responses to three types of noise: fluctuating inputs, spiking noise, and deviations in connectivity away from ideal attractor configurations. Across all three types, networks with more bumps experience less noise-driven deviations in bump motion. This translates to more robust encodings of linear coordinates, like position, assuming that each neuron represents a fixed length no matter the bump number. Alternatively, we consider encoding a circular coordinate, like orientation, such that the network distance between adjacent bumps always maps onto 360 degrees. Under this mapping, bump number does not significantly affect the amount of error in the coordinate readout. Our simulation results are intuitively explained and quantitatively matched by a unified theory for path integration and noise in multi-bump networks. Thus, to suppress the effects of biologically relevant noise, continuous attractor networks can employ more bumps when encoding linear coordinates; this advantage disappears when encoding circular coordinates. Our findings provide motivation for multiple bumps in the mammalian grid network.

Author summary

Our brains maintain an internal sense of location and direction so we can, for example, find our way to the door if the lights go off. A class of neural circuits called continuous attractor networks is believed to be responsible for this ability. These circuits must be resilient against the myriad forms of imperfections and random fluctuations present in the brain, which can degrade the accuracy of their encoded information. We have discovered a new way in which continuous attractor networks can improve their robustness to noise: they should distribute their activity among multiple regions in the network, called bumps, instead of concentrating it in a single bump. Bump number is a fundamental feature of continuous attractor networks, but its connection to error suppression has never been appreciated. A recent experiment in rodents suggests that one such network indeed contains multiple regions of activity; our finding provides motivation for why such a configuration may have been evolved.

Introduction

Continuous attractor networks (CANs) sustain a set of activity patterns that can be smoothly morphed from one to another along a low-dimensional manifold [1–3]. Network activity is typically localized into attractor bumps, whose positions along the manifold can represent the value of a continuous variable. These positions can be set by external stimuli, and their persistence serves as a memory of the stimulus value. Certain CAN architectures are also capable of a feature called path integration. Instead of receiving the stimulus value directly, the network receives its changes and integrates over them by synchronously moving the attractor bump [4–6]. Path integration allows systems to estimate an external state based on internally perceived changes, which is useful in the absence of ground truth.

Path-integrating CANs have been proposed as a mechanism through which brains encode various physical coordinates. Head direction cells in mammals and compass neurons in insects encode spatial orientation by preferentially firing when the animal faces a particular direction relative to landmarks (Fig 1A, top; Refs [7] and [8]). They achieve this as members of 1D CANs whose attractor manifolds have ring topologies [9, 10]. For the case of compass neurons, a ring structure also exists anatomically, and its demonstration of continuous attractor dynamics is well-established [8, 11–13]. Grid cells in mammals encode position by preferentially firing at locations that form a triangular lattice in 2D space (1D analogue in Fig 1A, bottom; Ref [14]). They are thought to form a 2D CAN with toroidal topology [15–18], and mounting experimental evidence supports this theory [19–22]. The ability for head direction cells, compass neurons, and grid cells to maintain their tunings in darkness without external cues demonstrates that these CANs can path integrate [8, 14, 23].

Fig 1. Continuous attractor networks with any number of bumps can produce head direction cells and grid cells.

(A) Desired tuning curves of a head direction cell and a 1D grid cell. (B) Orientation and position coordinates whose changes drive bump motion. (C) One- and two-bump ring attractor networks. Each black neuron produces the desired tuning curves in A. In the two-bump network, the coupling to coordinate changes is half as strong, and the second bump is labeled for clarity.

CANs also appear in studies of other brain regions and neural populations. Signatures of continuous attractor dynamics have been detected in the prefrontal cortex during spatial working memory tasks [24–26]. Theorists have further invoked CANs to explain place cells [27, 28], hippocampal view cells [29], eye tracking [4, 6], visual orientation tuning [30, 31], and perceptual decision making [32, 33]. Thus, CANs are a crucial circuit motif throughout the brain, and better understanding their performance would provide meaningful insights into neural computation.

One factor that strongly affects the performance of CANs in path integration is biological noise. To accurately represent physical coordinates, attractor bumps must move in precise synchrony with the animal’s trajectory. Hence, the bump velocity must remain proportional to the driving input that represents coordinate changes [18]. Different sources of noise produce different types of deviations from this exact relationship, all of which lead to path integration errors. While noisy path-integrating CANs have been previously studied [10, 18, 34, 35], these works did not investigate of role of bump number. CANs with different connectivities can produce different numbers of attractor bumps, which are equally spaced throughout the network and perform path integration by moving in unison [16, 18, 36]. Two networks with different bump numbers have the same representational capability (Fig 1). They can share the same attractor manifold and produce neurons with identical tuning curves, as long as the coupling strength between bump motion and driving input scales appropriately. The computational advantages of having more or fewer bumps are unknown.

Our aim is to elucidate the relationship between bump number and robustness to noise. We first develop a rigorous theoretical framework for studying 1D CANs that path integrate and contain multiple bumps. Our theory predicts the number, shape, and speed of bumps. We then introduce three forms of noise. The first is Gaussian noise added to the total synaptic input, which can represent fluctuations in a broad range of cellular processes occurring at short timescales. The second is Poisson spiking noise. The third is noise in synaptic connectivity strengths; the ability for bumps to respond readily to driving inputs is generally conferred by a precise network architecture. We add Gaussian noise to the ideal connectivity and evaluate path integration in this setting. The first two forms of noise are independent over time and neurons, in contrast to the third. We find that networks with more bumps can better resist all three forms of noise under certain encoding assumptions. These observations are explained by our theoretical framework with simple scaling arguments. The following Results section presents all simulation findings and major theoretical conclusions; complete theoretical derivations are found in the Theoretical model section.

Results

Bump formation in a ring attractor network

We study a 1D ring attractor network that extends the model of Ref [37] to allow for multiple attractor bumps. It contains two neural populations α ∈ {L, R} at each network position x, with N neurons in each population (Fig 2A). Each neuron is described by its total synaptic input g that obeys the following dynamics:

$$\begin{array}{c}\hfill \tau \frac{\mathrm{d}{g}_{\alpha }(x,t)}{\mathrm{d}t}+g_{\alpha }(x,t)=\sum _{\beta }\int \mathrm{d}y{W}_{\beta }(x,y)s_{\beta }(y,t)+A{\pm }_{\alpha }\gamma b\left(t\right)+{\zeta }_{\alpha }(x,t),\end{array}$$ (1)

where ±_L means − and ±_R means +. Aside from spiking simulations, firing rates s are given by

$$\begin{array}{c}\hfill s_{\alpha }(x,t)=\varphi \left[g_{\alpha }(x,t)\right],\end{array}$$ (2)

where ϕ is a nonlinear activation function. For all simulations in this Results section, we take ϕ to be the rectified linear unit (ReLU) activation function (Eq 35). Our theoretical formulas for diffusion coefficients and velocities in this section also assume a ReLU ϕ. In S1 Text, we consider a logistic ϕ instead and find that all major conclusions are preserved (Fig A in S1 Text), and in the Theoretical methods section, we derive most expressions for general ϕ. W is the synaptic connectivity and only depends on the presynaptic population β. It obeys a standard continuous attractor architecture based on local inhibition that is strongest at an inhibition distance l. Each population has its synaptic outputs shifted by a small distance ξ ≪ l in opposite directions. We use the connectivity profile described in Fig 2B and Eq 38 for all simulations, but all theoretical expressions in this Results section are valid for any W. A is the resting input to all neurons. The driving input, or drive, b is proportional to changes in the coordinate encoded by the network; for the physical coordinates in Fig 1B, it represents the animal’s velocity obtained from self-motion cues. In our results, b is constant in time. It is coupled to the network with strength γ. We will consider various forms of noise ζ. Finally, τ is the neural time constant.

Fig 2. Bump formation in a ring attractor network.

(A) Network schematic with populations L and R and locally inhibitory connectivity W. (B and C) Networks with 200 neurons and 3 bumps. (B) Connectivity weights for a neuron at the origin. The inhibition distance is l = 29 and the connectivity shift is ξ = 2. (C) Steady-state synaptic inputs. Curves for both populations lie over each other. With a ReLU activation function, the firing rates follow the solid portions of the colored lines and are 0 over the dashed portions. The bump distance is λ = 200/3. Thick gray line indicates Eq 4. (D and E) Networks with 500 neurons. (D) More bumps and shorter bump distances are produced by smaller inhibition distances. Points indicate data from 10 replicate simulations. Line indicates Eq 5. (E) The inhibition distance l = 55 corresponds to the black point in D with λ = 125 and M = 4. These values also minimize the Lyapunov functional (Eq 6), which varies smoothly across λ for infinite networks (line) and takes discrete values for finite networks (points). (F) The scaled bump shape remains invariant across network sizes and bump numbers, accomplished by rescaling connectivity strengths according to Eq 7. Curves for different parameters lie over one another.

With no drive b = 0 and no noise ζ = 0, the network dynamics in Eqs 1 and 2 can be simplified to

$$\begin{array}{c}\hfill \tau \frac{\mathrm{d}g(x,t)}{\mathrm{d}t}+g(x,t)=2\int \mathrm{d}yW(x-y)\varphi \left[g(y,t)\right]+A,\end{array}$$ (3)

where 2W(x − y) = ∑_β W_β(x, y) and the synaptic inputs g are equal between the two populations. This baseline equation evolves towards a periodic steady-state g with approximate form (see also Ref [38]).

$$\begin{array}{c}\hfill g\left(x\right)=a\text{cos}\frac{2\pi (x-x_0)}{\lambda }+d.\end{array}$$ (4)

Expressions for a and d are given in the Theoretical model section (Eq 60). The firing rates s(x) = ϕ[g(x)] exhibit attractor bumps with periodicity λ, a free parameter that we call the bump distance (Fig 2C). x₀ is the arbitrary position of one of the bumps. It parameterizes the attractor manifold with each value corresponding to a different attractor state up to λ.

The bump number M = N/λ is determined through λ. It can be predicted by the fastest-growing mode in a linearized version of the dynamics (Eq 43; Refs [39] and [40]). The mode with wavenumber q and corresponding wavelength 2π/q grows at rate $(2\tilde{W}\left(q\right)-1)/\tau$, where $\tilde{W}\left(q\right)$ is the Fourier transform of W(x). Thus,

$$\begin{array}{c}\hfill \frac{2\pi }{\lambda }=\underset{q}{\text{argmax}}\tilde{W}\left(q\right).\end{array}$$ (5)

Fig 2D shows that simulations follow the predicted λ and M over various inhibition distances l. Occasionally for small l, a different mode with a slightly different wavelength will grow quickly enough to dominate the network. A periodic network enforces an integer bump number, which discretizes the allowable wavelengths and prevents changes in λ and M once they are established. In an aperiodic or infinite system, the wavelength can smoothly vary from an initial value to a preferred length over the course of a simulation [18, 41]. To determine this preferred λ theoretically, we notice that the nonlinear dynamics in Eq 3 obey the Lyapunov functional

$$\begin{array}{c}\hfill L=-\int \int \mathrm{d}x\mathrm{d}yW(x-y)s\left(x\right)s\left(y\right)+\int \mathrm{d}x{\int }_0^{s\left(x\right)}\mathrm{d}\rho {\varphi }^{-1}\left[\rho \right]-A\int \mathrm{d}xs\left(x\right).\end{array}$$ (6)

In the Theoretical model section, we find for ReLU ϕ that L is minimized when q = 2π/λ maximizes $\tilde{W}\left(q\right)$ (Eq 66). This is the same condition as for the fastest-growing mode in Eq 5 (Fig 2E). In other words, the wavelength λ most likely to be established in a periodic network is the preferred bump distance in an aperiodic or infinite system, up to a difference of one fewer or extra bump due to discretization.

We now understand how to produce different bump numbers M in networks of different sizes N by adjusting the inhibition distance l. To compare networks across different values of M and N, we scale the connectivity strength W according to

$$\begin{array}{c}\hfill W_{\beta }(x,y)\propto \frac{M}{N}.\end{array}$$ (7)

This keeps the total connectivity strength per neuron ∫dxW_β(x, y) constant over M and N. In doing so, the shape of each attractor bump as a function of scaled network position x/λ remains invariant (Fig 2F). Thus, Eq 7 removes additional variations in bump shape and helps to isolate our comparisons across M and N to those variables themselves. In S1 Text, we consider the alternative without this scaling and find that many major results are preserved (Fig B in S1 Text).

Bump dynamics: Path integration and diffusion

The drive b produces coherent bump motion by creating an imbalance between the two neural populations. A positive b increases input to the R population and decreases input to the L population (Fig 3A). Because the synaptic outputs of the former are shifted to the right, the bump moves in that direction. Similarly, a negative b produces leftward bump motion. The bump velocity v_drive can be calculated in terms of the baseline firing rates s(x) obtained without drive and noise (see also Refs [37] and [42]):

$$\begin{array}{c}\hfill v_{\text{drive}}=-\frac{{\displaystyle \gamma b\xi \int \mathrm{d}x\frac{{\mathrm{d}}^{2}s}{\mathrm{d}{x}^2}}}{{\displaystyle \tau \int \mathrm{d}x{\left(\frac{\mathrm{d}s}{\mathrm{d}x}\right)}^2}}.\end{array}$$ (8)

Fig 3. Dynamics in a ring attractor network.

(A–C) Networks with 200 neurons and 3 bumps. (A) Synaptic inputs for populations L and R under drive b = 2. Snapshots taken at 150 ms intervals demonstrate rightward motion. (B) Bump velocity is proportional to drive. The connectivity shift is ξ = 2. (C) Bump velocity is largely proportional to connectivity shift. The drive is b = 0.5. (D–H) Networks with synaptic input noise. (D) Bump displacements for 48 replicate simulations demonstrating diffusion with respect to coherent motion. Networks with 200 neurons and 1 bump. (E and F) Mean bump velocity is proportional to drive and remains largely independent of network size, bump number, and noise magnitude. (G and H) Bump diffusion coefficient scales quadratically with noise magnitude, remains largely independent of drive, and varies with network size and bump number. The noise magnitude is σ = 0.5 in D, E, and G, and the drive is b = 0.5 in D, F, and H. Values for both bumps in two-bump networks lie over each other. Points indicate data from 48 replicate simulations and bars indicate bootstrapped standard deviations. Dotted gray lines indicate Eqs 8 and 10.

As a note, these integrals, as well as subsequent ones, do not include the singular points at the edges of attractor bumps. Eq 8 states that bump velocity is proportional to drive b and connectivity shift ξ, which is reflected in our simulations, with some deviation at larger ξ (Fig 3B and 3C). The strict proportionality between v and b is crucial because it implies faithful path integration [18]. If b(t) represents coordinate changes (such as angular or linear velocity in Fig 1B), then the bump position θ(t) will accurately track the coordinate itself (orientation or position).

In contrast to drive, uncorrelated noise ζ produces bump diffusion. To illustrate this effect, we introduce one form of ζ that we call synaptic input noise. Suppose ζ is independently sampled for each neuron at each simulation timestep from a Gaussian distribution with mean 0 and variance σ². Loosely, it can arise from applying the central limit theorem to the multitude of noisy synaptic and cellular processes occurring at short timescales. Then,

$$\begin{array}{c}\hfill ⟨{\zeta }_{\alpha }(x,t)⟩=0,⟨{\zeta }_{\alpha }(x,t){\zeta }_{\beta }(y,t^{\prime })⟩={\sigma }^2\Delta t\delta (t-t^{\prime }){\delta }_{\alpha \beta }\delta (x-y),\end{array}$$ (9)

where the timestep Δt sets the resampling rate of ζ, and angle brackets indicate averaging over an ensemble of replicate simulations. Input noise causes bumps to diffuse away from the coherent driven motion (Fig 3D). The mean velocity 〈v〉 remains proportional to drive b, which means that the network still path integrates on average (Fig 3E). Since 〈v〉 is largely independent of noise magnitude σ, and the bump diffusion coefficient D is largely independent of b, drive and input noise do not significantly interact within the explored parameter range (Fig 3F and 3G). D can be calculated in terms of the baseline firing rates (see also Refs [35] and [43]):

$$\begin{array}{c}\hfill D_{\text{input}}=\frac{{\displaystyle {\sigma }^2\Delta t}}{{\displaystyle 4{\tau }^2\int \mathrm{d}x{\left(\frac{\mathrm{d}s}{\mathrm{d}x}\right)}^2}}.\end{array}$$ (10)

The quadratic dependence of D on σ is confirmed by simulation (Fig 3H).

We now turn our attention to bump number M and network size N. The mean bump velocity 〈v〉 is independent of these parameters (Fig 3E and 3F), which can be understood theoretically. Bump shapes across M and N are simple rescalings of one another (Fig 2F), so derivatives of s with respect to x are simply proportional to M (more bumps imply faster changes) and inversely proportional to N (larger networks imply slower changes). Similarly, integrals of expressions containing s over x are simply proportional to N. In summary,

$$\begin{array}{c}\hfill \frac{\mathrm{d}s}{\mathrm{d}x}\propto \frac{M}{N},\frac{{\mathrm{d}}^{2}s}{\mathrm{d}{x}^2}\propto \frac{M^2}{N^2},\int \mathrm{d}x\propto N.\end{array}$$ (11)

Applying these scalings to Eq 8, we indeed expect v_drive to be independent of M and N. In contrast, Fig 3G and 3H reveal that the diffusion coefficient D varies with these parameters. When a one-bump network is increased in size from 200 to 400 neurons, D increases as well, which implies greater path integration errors. This undesired effect can be counteracted by increasing the bump number from 1 to 2, which lowers D below that of the one-bump network with 200 neurons. These initial results suggest that bump number and network size are important factors in determining a CAN’s resilience to noise. We will explore this idea in greater detail.

Mapping network coordinates onto physical coordinates

Before further comparing networks with different bump numbers M and sizes N, we should scrutinize the relationship between bump motion and the physical coordinate encoded by the network. After all, the latter is typically more important in biological settings. First, we consider the trivial case in which each neuron represents a fixed physical interval across all M and N; this is equivalent to using network coordinates without a physical mapping (Fig 4A). It is suited for encoding linear variables like position that lack intrinsic periodicity, so larger networks can encode wider coordinate ranges. However, with more bumps or fewer neurons, the range over which the network can uniquely encode different coordinates is shortened. We assume that ambiguity among coordinates encoded by each bump can be resolved by additional cues, such as local features, that identify the true value among the possibilities [44–46]; this process will be examined in detail below. We leave quantities with dimensions of network distance in natural units of neurons.

Fig 4. Possible mappings between network coordinates and two types of physical coordinates.

(A) In networks encoding linear coordinates such as position, one neuron always represents a fixed physical interval. This mapping is trivial and identical to using network coordinates. (B) In networks encoding circular coordinates such as orientation, the bump distance always represents 360°.

Multi-bump networks are intrinsically periodic, especially those with a ring architecture. A natural way for them to encode a circular coordinate like orientation would be to match network and physical periodicities. For example, the bump distance may always represent 360°across different M and N so that neurons always exhibit unimodal tuning (Fig 4B). This relationship implies that quantities with dimensions of network distance should be multiplied by powers of the conversion factor

$$\begin{array}{c}\hfill \frac{360°·M}{N},\end{array}$$ (12)

which converts units of neurons to degrees. In other words, circular mapping implies normalizing network distances by the bump distance λ = N/M.

For circular mapping, we must also ensure that networks with different bump numbers M and sizes N path integrate consistently with one another. The same drive b should produce the same bump velocity v in units of degree/s. To do so, we rescale the coupling strength γ only under circular mapping:

$$\begin{array}{c}\hfill \gamma \propto \frac{N}{M}.\end{array}$$ (13)

This effectively compensates for the factor of M/N in Eq 12. To see this explicitly, recall that v_drive does not depend on M and N in units of neuron/s, as shown in Fig 3E and 3F and previously explained through scaling arguments. Under circular mapping, v_drive would be multiplied by one power of the conversion factor in Eq 12. Since its formula contains γ in the numerator (Eq 8), v_drive receives an additional power of the rescaling factor in Eq 13. The two factors cancel each other, so v_drive does not depend on M and N under either mapping:

$$\begin{array}{c}\hfill v_{\text{drive}}\propto 1\text{linear},v_{\text{drive}}\propto 1\text{circular}.\end{array}$$ (14)

Thus, a consistent relationship between b and v_drive is preserved in units of both neurons/s and degrees/s.

Of course, there are other possible mappings between network and physical coordinates across bump numbers and network sizes. For example, intermediate scalings can be achieved with the conversion factor (M/N)^μ for 0 < μ < 1 instead of Eq 12, with the corresponding γ ∝ (N/M)^μ instead of Eq 13. But for the rest of our paper, we will consider the linear and circular cases, which correspond to μ = 0 and μ = 1, respectively. To be clear, networks with the same ring architecture are used for both mappings. We will see how noise affects encoding quality in either case.

More bumps improve robustness to input and spiking noise under linear mapping

We now revisit the effect of input noise on bump diffusion, as initially explored in Fig 3D–3H. We measure how the diffusion coefficient D varies with bump number M and network size N under linear and circular mappings. Under linear mapping, D decreases as a function of M but increases as a function of N (Fig 5A and 5B). Thus, more bumps attenuate diffusion produced by input noise, which is especially prominent in large networks. However, for circular coordinates, D remains largely constant with respect to M and decreases with respect to N (Fig 5A and 5B). Increasing the number of bumps provides no benefit. These results can be understood through Eqs 10, 11 and 12, which predict

$$\begin{array}{c}\hfill D_{\text{input}}\propto \frac{N}{M^2}\text{linear},D_{\text{input}}\propto \frac{1}{N}\text{circular}.\end{array}$$ (15)

Fig 5. Bump diffusion due to input and spiking noise.

(A, B) Networks with synaptic input noise of magnitude σ = 0.5 and drive b = 0.5. Dotted gray lines indicate Eq 10. (A) Diffusion decreases with bump number under linear mapping and remains largely constant under circular mapping. Networks with 600 neurons. (B) Diffusion increases with network size under linear mapping and decreases under circular mapping. Networks with 3 bumps. (C and D) Same as A and B, but for networks with Poisson spiking noise instead of input noise. Dotted gray lines indicate Eq 20. Points indicate data from 48 replicate simulations and bars indicate bootstrapped standard deviations.

Two powers of the conversion factor in Eq 12 account for the differences between the two mappings.

Next, we investigate networks with spiking noise instead of input noise. To do so, we replace the deterministic formula for firing rate in Eq 2 with

$$\begin{array}{c}\hfill s_{\alpha }(x,t)=\frac{c_{\alpha }(x,t)}{\Delta t}.\end{array}$$ (16)

Here, s is a stochastic, instantaneous firing rate given by the number of spikes c emitted in a simulation timestep divided by the timestep duration Δt. We take the c’s to be independent Poisson random variables driven by the deterministic firing rate:

$$\begin{array}{c}\hfill c_{\alpha }(x,t)\sim \text{Pois}\left[\varphi \right[g_{\alpha }(x,t)]\Delta t].\end{array}$$ (17)

As fully explained in the Theoretical model section (Eq 99), we can approximate this spiking process by the rate-based dynamics in Eqs 1 and 2 with the noise term

$$\begin{array}{c}\hfill {\zeta }_{\alpha }(x,t)=\sum _{\beta }\int \mathrm{d}y{W}_{\beta }(x,y)\sqrt{\frac{}{}}\varphi \left[g_{\beta }(y,t)\right]\Delta t{\eta }_{\beta }(y,t).\end{array}$$ (18)

The η’s are independent random variables with zero mean and unit variance:

$$\begin{array}{c}\hfill ⟨{\eta }_{\alpha }(x,t)⟩=0,⟨{\eta }_{\alpha }(x,t){\eta }_{\beta }(y,t^{\prime })⟩=\Delta t\delta (t-t^{\prime }){\delta }_{\alpha \beta }\delta (x-y).\end{array}$$ (19)

As for Eq 9, the simulation timestep Δt sets the rate at which η is resampled. This spiking noise produces bump diffusion with coefficient (see also Ref [43])

$$\begin{array}{c}\hfill D_{\text{spike}}=\frac{{\displaystyle \int \mathrm{d}xs\left(x\right){\left(\frac{\mathrm{d}s}{\mathrm{d}x}\right)}^2}}{{\displaystyle 4{\tau }^2{[\int \mathrm{d}x{\left(\frac{\mathrm{d}s}{\mathrm{d}x}\right)}^2]}^2}}.\end{array}$$ (20)

As before, s is the baseline firing rate configuration without noise and drive. Through the relationships in Eqs 11 and 12, D_spike scales with M and N in the same way as D_input does:

$$\begin{array}{c}\hfill D_{\text{spike}}\propto \frac{N}{M^2}\text{linear},D_{\text{spike}}\propto \frac{1}{N}\text{circular}.\end{array}$$ (21)

These findings are presented in Fig 5C and 5D along with simulation results that confirm our theory. Spiking noise behaves similarly to input noise. Increasing bump number improves robustness to noise under linear mapping but has almost no effect under circular mapping. Bump diffusion in larger networks is exacerbated under linear mapping but suppressed under circular mapping. For both input noise and spiking noise, the conversion factor in Eq 12 produces the differences between the two mappings. Coupling strength rescaling in Eq 13 does not play a role because γ does not appear in Eqs 10 and 20. In S1 Text, we consider splitting a large network with many bumps into smaller networks, each with fewer bumps; the intact network and the combined readout of the split networks exhibit similar diffusion properties.

To evaluate noise robustness in a complementary way, we perform mutual information analysis of networks with input noise. Mutual information describes how knowledge of one random variable can reduce the uncertainty in another, and it serves as a general metric for encoding quality. Before proceeding, we mention a related quantity called Fisher information, which is directly related to mutual information [47, 48] and inversely related to bump diffusion in CANs [43]. Thus, we expect that networks with less diffusion in Fig 5 should generally contain more mutual information about their encoded coordinate. Fisher information also permits a more intuitive explanation for our diffusion scalings. It is proportional to network size N [43], because monitoring a larger number of noisy neurons tells us more about the encoded coordinate. Otherwise, it only depends on the tuning curves of neurons within the network; in particular, steeper tuning curves convey quadratically more information [43]. For our networks, across M and N, linear tuning curves are simple rescalings of each other with derivative inversely proportional to λ = N/M, while circular tuning curves remain unimodal and identical (Figs 2F and 4). Thus, Fisher information should be proportional to N ⋅ (M/N)² and N for linear and circular coordinates, respectively. Applying the inverse proportionality between Fisher information and diffusion coefficients discovered by Ref [43], we arrive at Eqs 15 and 21.

Using simulations, we investigate the mutual information I between the physical coordinate encoded by the noisy network, represented by the random variable U with discretized sample space $\mathcal{U}$, and the activity of a single neuron, represented by the random variable S with discretized sample space $\mathcal{S}$ (see Simulation methods):

$$\begin{array}{c}\hfill I[S;U]=\sum _{s\in \mathcal{S},u\in \mathcal{U}}p\left(s\right|u)p\left(u\right)\text{log}\frac{p\left(s\right|u)}{p\left(s\right)}.\end{array}$$ (22)

We then average I across neurons. Larger mean mutual information implies more precise coordinate encoding and greater robustness to noise. Note that the joint activities of all the neurons confer much more coordinate information than single-neuron activities do, but since estimating high-dimensional probability distributions over the former is computationally very costly, we use the latter as a metric for network performance.

The physical coordinate U is either position or orientation and obeys the mappings described in Fig 4 across bump numbers M and network sizes N. To obtain the probability distributions in Eq 22 required to compute I, we initialize multiple replicate simulations at evenly spaced coordinate values u (Fig 6A). We do not apply a driving input, so the networks should still encode their initialized coordinates at the end of the simulation. However, they contain input noise that degrades their encoding. Collecting the final firing rates produces p(s|u) for each neuron. For both position and orientation, we consider narrow and wide coordinate ranges to assess network performance in both regimes.

Fig 6. Mutual information between neural activity and physical coordinates with input noise of magnitude σ = 0.5.

(A) To compute mutual information, we initialize replicate simulations without input drive at different coordinate values (thick black lines) and record the final neural activities (thin colored lines). The physical coordinate can be linear or circular and its range can be narrow or wide; here, we illustrate two possibilities for networks with 600 neurons and 3 bumps. (B and C) Mutual information between physical coordinate and single-neuron activity under narrow coordinate ranges. (B) Information increases with bump number for linear coordinates and remains largely constant for circular coordinates. Networks with 600 neurons. (C) Information decreases with network size for linear coordinates and increases for circular coordinates. Networks with 3 bumps. (D and E) Mutual information between physical coordinate and single-neuron activity under wide coordinate ranges. The trends in B and C are preserved for circular coordinates. They are also preserved for linear coordinates, except for the shaded regions in which the coordinate range exceeds the bump distance. (F) Coarse local cues are active over different quadrants of the wide coordinate ranges. (G and H) Mutual information between physical coordinate and the joint activities of a single neuron with the four cues in F under wide coordinate ranges. The trends in B and C are preserved for both linear and circular coordinates. Points indicate data from 96 replicate simulations at each coordinate value averaged over neurons and bars indicate bootstrapped standard errors of the mean. Cue icons adapted from Streamline Freemoji (CC BY license, https://www.streamlinehq.com/emojis/freebies-freemojis).

We first consider narrow coordinate ranges. For linear coordinates, information increases as a function of M but decreases as a function of N; for circular coordinates, it does not strongly depend on M and increases as a function of N (Fig 6B and 6C). These results exactly corroborate those in Fig 5A and 5B obtained for bump diffusion, since we expect information and diffusion to be inversely related.

We next consider wide coordinate ranges. Our ring networks can only uniquely represent coordinate ranges up to their bump distances (converted to physical distances by Fig 4). Beyond these values, two physical coordinates separated by the converted bump distance cannot be distinguished by the network. Our mutual information analysis captures this phenomenon; for linear coordinates, the increase in information with larger M or smaller N as observed in Fig 6B and 6C disappears once the converted bump distance drops below the physical range of 200 cm (green shaded regions of Fig 6D and 6E). In this regime, the benefits of more bumps and smaller networks toward decreasing diffusion are counteracted by bump ambiguity. In contrast, the circular mapping in Fig 4 lacks bump ambiguity since the bump distance is always converted to the maximum physical range of 360°, so the same qualitative trends in mutual information are observed for any coordinate range (Fig 6D and 6E).

For linear coordinates with wide ranges, the advantages of increasing bump number can be restored by coarse local cues. We illustrate this process by introducing four cues, each of which is active over a different quadrant and is otherwise inactive (Fig 6F). They can be conceptualized as two-state sensory neurons or neural populations that fire in the presence of a local stimulus. By themselves, the cues do not encode precise coordinate values. Mutual information computed with the joint activity of each neuron with these cues recovers the behavior observed for narrow ranges across all M and N (Fig 6G and 6H). Ring attractor neurons provide information beyond the 2 bits conveyed by the cues alone, and for position, this additional information increases with more bumps and fewer neurons without saturating.

In summary, our conclusions about robustness to input noise obtained by diffusion analysis are also supported by mutual information analysis. Moreover, the latter explicitly reveals how networks encoding wide, linear coordinate ranges can leverage coarse local cues to address ambiguities and preserve the advantages of multiple bumps. In S1 Text, we calculate mutual information for a few additional regimes (Fig C of S1 Text; see also Refs [49] and [50], which investigated the Fisher information conveyed by multi-bump tuning curves).

More bumps improve robustness to connectivity noise under linear mapping

Another source of noise in biological CANs is inhomogeneity in the connectivity W. Perfect continuous attractor dynamics requires W to be invariant to translations along the network [9, 10, 16, 18, 28], a concept related to Goldstone’s theorem in physics [51, 52]. We consider the effect of replacing W → W + V, where V is a noisy connectivity matrix whose entries are independently drawn from a zero-mean Gaussian distribution. V disrupts the symmetries of W. This noise is quenched and does not change over the course of the simulation, in contrast to input and spiking noise, which are independently sampled in time. It contributes a noise term

$$\begin{array}{c}\hfill {\zeta }_{\alpha }(x,t)=\sum _{\beta }\int \mathrm{d}y{V}_{\alpha \beta }(x,y)s_{\beta }(y,t).\end{array}$$ (23)

This formula implies that V produces correlated ζ’s across neurons, which also differs from input and spiking noise. Because of these distinctions, the dominant effect of connectivity noise is not diffusion, but drift. V induces bumps to move with velocity v_conn(θ), even without drive b:

$$\begin{array}{c}\hfill v_{\text{conn}}\left(\theta \right)=-\frac{{\displaystyle \sum _{\alpha \beta }\int \int \mathrm{d}x\mathrm{d}y{V}_{\alpha \beta }(x,y)\frac{\mathrm{d}s(x-\theta )}{\mathrm{d}x}s(y-\theta )}}{{\displaystyle 2\tau \int \mathrm{d}x{\left(\frac{\mathrm{d}s}{\mathrm{d}x}\right)}^2}}.\end{array}$$ (24)

The movement is coherent but irregular, as it depends on the bump position θ (Fig 7A). Refs [53] and [54] refer to v_conn(θ) as the drift velocity.

Fig 7. Bump trapping due to connectivity noise at low drive.

(A–C) Networks with 600 neurons, 1 bump, and the same realization of connectivity noise of magnitude 0.002. (A) Theoretical values for drift velocity as a function of bump position using Eq 24. (B) Bumps drift towards trapped positions over time. The drive is b = 0. Arrows indicate predictions from v_conn(θ) crossing 0 with negative slope in A. Lines indicate simulations with different starting positions. (C) Bump trajectories with smallest positive and negative drive required to travel through the entire network. Respectively, b = 0.75 and b = −0.52. The larger of the two in magnitude is the escape drive b₀ = 0.75. Note that positions with low bump speed exhibit large velocities in the opposite direction in A. (D and E) Networks with multiple realizations of connectivity noise of magnitude 0.002. (D) Escape drive decreases with bump number under linear mapping and remains largely constant under circular mapping. Networks with 600 neurons. (E) Escape drive increases with network size under linear mapping and remains largely constant under circular mapping. Networks with 3 bumps. Points indicate simulation means over 48 realizations and bars indicate standard deviations. Dotted gray lines indicate Eq 26 averaged over 96 realizations.

Connectivity noise traps bumps at low drive b. We first consider b = 0, so bump motion is governed solely by drift according to dθ/dt = v_conn(θ). The bump position θ has stable fixed points wherever v_conn(θ) crosses 0 with negative slope [53, 54]. Simulations confirm that bumps drift towards these points (Fig 7B). The introduction of b adds a constant v_drive that moves the curve in Fig 7A up for positive b or down for negative b:

$$\begin{array}{c}\hfill v_{\text{total}}\left(\theta \right)=v_{\text{drive}}+v_{\text{conn}}\left(\theta \right).\end{array}$$ (25)

If v_total(θ) still crosses 0, bumps would still be trapped. The absence of bump motion in response to coordinate changes encoded by b would be a catastrophic failure of path integration. To permit bump motion through the entire network, the drive must be strong enough to eliminate all zero-crossings. Fig 7C shows bump motion at this drive for both directions of motion. The positive b is just large enough for the bump to pass through the region with the most negative v_conn(θ) in Fig 7A; likewise for negative b and positive v_conn(θ). We call the larger absolute value of these two drives the escape drive b₀. Simulations show that b₀ decreases with bump number M and increases with network size N under linear mapping (Fig 7D and 7E). A smaller b₀ implies weaker trapping, so smaller networks with more bumps are more resilient against this phenomenon. Under circular mapping, however, b₀ demonstrates no significant dependence on M or N. We can predict b₀ by inverting the relationship in Eq 8 between b and v:

$$\begin{array}{c}\hfill b_0=-\frac{{\displaystyle \underset{\theta }{\text{max}}\left|v_{\text{conn}}\left(\theta \right)\right|·\tau \int \mathrm{d}x{\left(\frac{\mathrm{d}s}{\mathrm{d}x}\right)}^2}}{{\displaystyle \gamma \xi \int \mathrm{d}x\frac{{\mathrm{d}}^{2}s}{\mathrm{d}{x}^2}}}.\end{array}$$ (26)

This theoretical result agrees well with values obtained by simulation (Fig 7D and 7E). In the Theoretical model section, we present a heuristic argument (Eq 124) that leads to the observed scaling of escape drive on M and N:

$$\begin{array}{c}\hfill b_0\propto \frac{N}{M}\text{linear},b_0\propto 1\text{circular}.\end{array}$$ (27)

At high drive |b| > b₀, attractor bumps are no longer trapped by the drift velocity v_conn(θ). Instead, the drift term produces irregularities in the total velocity v_total(θ) (Fig 8A). They can be decomposed into two components: irregularities between directions of motion and irregularities within each direction. Both imply errors in path integration because v and b are not strictly proportional. To quantify these components, we call |v₊(θ)| and |v₋(θ)| the observed bump speeds under positive and negative b. We define speed difference as the unsigned difference between mean speeds in either direction, normalized by the overall mean speed:

$$\begin{array}{c}\hfill \text{speed}\text{difference}=\frac{{\displaystyle 2\left|\underset{\theta }{\text{mean}}|v_{+}\left(\theta \right)|-\underset{\theta }{\text{mean}}\left|v_{-}\left(\theta \right)\right|\right|}}{{\displaystyle \underset{\theta }{\text{mean}}|v_{+}\left(\theta \right)|+\underset{\theta }{\text{mean}}\left|v_{-}\left(\theta \right)\right|}}.\end{array}$$ (28)

Fig 8. Bump speed irregularity due to connectivity noise at high drive.

(A) Bump speed as a function of bump position with connectivity noise of magnitude 0.002 and drive b = 1.5. Network with 600 neurons, 1 bump, and the same realization of connectivity noise as in Fig 7A–7C. Thick gray lines indicate Eq 25. (B–E) Networks with multiple realizations of connectivity noise of magnitude 0.002 and drive b = 1.5. (B) Speed difference between directions decreases with bump number under linear mapping and remains largely constant under circular mapping. Networks with 600 neurons. (C) Speed difference increases with network size under linear mapping and remains largely constant under circular mapping. Networks with 3 bumps. (D and E) Same as B and C, but for speed variability within each direction. Points indicate simulation means over 48 realizations and bars indicate standard deviations. Dotted gray lines indicate Eqs 30 and 31 averaged over 96 realizations.

We then define speed variability as the standard deviation of speeds within each direction, averaged over both directions and normalized by the overall mean speed:

$$\begin{array}{c}\hfill \text{speed}\text{variability}=\frac{{\displaystyle \underset{\theta }{\text{std}}|v_{+}\left(\theta \right)|+\underset{\theta }{\text{std}}\left|v_{-}\left(\theta \right)\right|}}{{\displaystyle \underset{\theta }{\text{mean}}|v_{+}\left(\theta \right)|+\underset{\theta }{\text{mean}}\left|v_{-}\left(\theta \right)\right|}}.\end{array}$$ (29)

Speed difference and speed variability follow the same trends under changes in bump number M and network size N (Fig 8B–8E). Under linear mapping, they decrease with M and increase with N. Under circular mapping, they do not significantly depend on M and N. These are also the same trends exhibited by the escape drive b₀ (Fig 7D and 7E). In terms of theoretical quantities, the formulas for speed difference and variability become

$$\begin{array}{c}\hfill \text{speed}\text{difference}=\frac{{\displaystyle 2\left|\underset{\theta }{\text{mean}}{v}_{\text{conn}}\left(\theta \right)\right|}}{|v_{\text{drive}}|}\end{array}$$ (30)

and

$$\begin{array}{c}\hfill \text{speed}\text{variability}=\frac{{\displaystyle \underset{\theta }{\text{std}}{v}_{\text{conn}}\left(\theta \right)}}{|v_{\text{drive}}|}.\end{array}$$ (31)

These expressions match the observed values well (Fig 8B–8E). In the Theoretical methods section, we calculate the observed dependence of speed difference (Eq 113) and variability (Eq 120) on M and N:

$$\begin{array}{cccc}\hfill \text{speed}\text{difference}\text{and}\text{variability}& \propto \frac{N}{M}\hfill & & \text{linear},\hfill \\ \hfill \text{speed}\text{difference}\text{and}\text{variability}& \propto 1& & \text{circular}.\end{array}$$ (32)

For all results related to connectivity noise, the coupling strength rescaling in Eq 13 produces the differences between the two mappings via the γ in Eq 8. The conversion factor in Eq 12 does not play a role because escape drive, speed difference, and speed variability do not have dimensions of network distance.

To summarize, CANs with imperfect connectivity benefit from more attractor bumps when encoding linear coordinates. This advantage is present at all driving inputs and may be more crucial for larger networks. On the other hand, connectivity noise has largely the same consequences for networks of all bump numbers and sizes when encoding circular coordinates.

Discussion

We demonstrated how CANs capable of path integration respond to three types of noise. Additive synaptic input noise and Poisson spiking noise cause bumps to diffuse away from the coherent motion responsible for path integration (Figs 3 and 5). This diffusion is accompanied by a decrease in mutual information between neural activity and encoded coordinate (Fig 6). Connectivity noise produces a drift velocity field that also impairs path integration by trapping bumps at low drive and perturbing bump motion at high drive (Figs 7 and 8).

For all three types of noise, CANs with more attractor bumps exhibit less deviation in bump motion in network units. This is observed across network parameters (Figs A and B in S1 Text). Thus, they can more robustly encode linear variables whose mapping inherits network units and does not rescale with bump number (Fig 4A). If grid cell networks were to encode spatial position in this manner, then multiple attractor bumps would be preferred over a single bump. Ref [20] reports experimental evidence supporting multi-bump grid networks obtained by calcium imaging of mouse medial entorhinal cortex. Our work implies that the evolution of such networks may have been partially encouraged by biological noise. Additional bumps introduce greater ambiguity among positions encoded by each bump, but this can be resolved by a rough estimate of position from additional cues, such as local landmarks [44, 55–57], another grid module with different periodicity [15, 40, 41, 44, 46, 58], or a Bayesian prior based on recent memory [45]. In this way, grid modules with finer resolution and more attractor bumps could maintain a precise egocentric encoding of position, while coarser modules and occasional allocentric cues would identify the true position out of the few possibilities represented. We explicitly explored one realization of this concept and observed how cues enable networks to improve their information content by increasing bump number, despite a concomitant increase in bump ambiguity (Fig 6F–6H).

In contrast, CANs encoding circular variables may rescale under different bump numbers to match periodicities (Fig 4B), which eliminates any influence of bump number on encoding accuracy for all three types of noise. If head direction networks were to encode orientation in this manner, then they would face less selective pressure to evolve beyond the single-bump configuration observed in Drosophila [8]. Moreover, without the assumption of bump shape invariance accomplished by Eq 7, robustness to all three types of noise decreases with bump number, which actively favors single-bump orientation networks (Fig B in S1 Text). Further experimental characterization of bump number in biological CANs, perhaps through techniques proposed by Ref [59], would test the degree to which the brain can leverage the theoretical advantages identified in this work.

Under linear mapping, larger CANs exhibit more errors in path integration from all three types of noise. The immediate biological implication is that larger brains face a dramatic degradation of CAN performance, accentuating the importance of suppressing error with multi-bump configurations. However, the simple rule that one neuron always represents a fixed physical interval does not need to be followed, and larger animals may tolerate greater absolute errors in path integration because they interact with their environments over larger scales. Nevertheless, our results highlight the importance of considering network size when studying the performance of noisy CANs. Under circular mapping, bump diffusion decreases with network size for input and spiking noise, and the magnitude of errors due to connectivity noise is independent of network size. This implies that head direction networks can benefit from incorporating more neurons; the observed interactions among such networks across different mammalian brain regions may act in this manner to suppress noise [60].

The computational advantages of periodic over nonperiodic encodings has been extensively studied in the context of grid cells [45, 46, 49, 61–64]. Our results extend these findings by demonstrating that some kinds of periodic encodings can perform better than others. Our results also contribute to a rich literature on noisy CANs. Previous studies have investigated additive input noise [35, 43, 54, 65, 66], multiplicative input noise [67], spiking noise [18, 43, 54, 62, 63, 68], and quenched noise due to connectivity or input inhomogeneities [10, 53, 54, 66, 69]. Among these works, the relationship between bump number and noise has only been considered in the context of multiple-item working memory, in which a single network can be dynamically loaded with different numbers of bumps [62, 63, 67, 68]. Interestingly, they find that robustness to noise decreases with bump number, which is opposite to our results (cf. Ref [63], which reports no dependence between bump number and encoding accuracy under certain conditions). It appears that CANs designed for path integration with fixed bump number and CANs designed for multiple-item working memory with variable bump number differ crucially in their responses to noise. Further lines of investigation that compare these two classes would greatly contribute to our general understanding of CANs.

Beyond our concrete results on CAN performance, our work offers a comprehensive theoretical framework for studying path-integrating CANs. We derive a formula for the multi-bump attractor state and a Lyapunov functional that governs its formation. For a ReLU activation function, we calculate all key dynamical quantities such as velocities and diffusion coefficients in terms of firing rates. Our formulas yield scaling relationships that facilitate an intuitive understanding for their dependence on bump number and network size. Much of our theoretical development does not assume a specific connectivity matrix or nonlinear activation function, which allows our results to have wide significance. For example, we expect them to hold for path-integrating networks that contain excitatory synapses. Other theories have been developed for bump shape [37, 38, 53, 66, 67, 70], path integration velocity [37, 42], diffusion coefficients [35, 43, 54, 66, 67, 71], and drift velocity [10, 53, 54]. Our work unifies these studies through a simple framework that features path integration, multiple bumps, and a noise term that can represent a wide range of sources. It can be easily extended to include other components of theoretical or biological significance, such as slowly-varying inputs [27, 66, 72], synaptic plasticity [34, 73], neural oscillations [74–76], and higher-dimensional attractor manifolds [2, 28].

Theoretical model

Architecture

We investigate CAN dynamics through a 1D ring attractor network. This class of network has been analyzed in previous theoretical works, and at various points, our calculations will parallel those in Refs [37, 38, 43, 53, 54, 66], and [42].

There are two neurons at each position i = 0, …, N − 1 with population indexed by α ∈ {L, R} (Fig 1A). For convenient calculation, we unwrap the ring and connect copies end-to-end, forming a linear network with continuous positions x ∈ (−∞, ∞). Unless otherwise specified, integrals are performed over the entire range. To map back onto the finite-sized ring network, we enforce our results to have a periodicity λ that divides N. For example, λ = N corresponds to a single-bump configuration. Integrals would then be performed over [0, N), with positions outside this range corresponding to their equivalents within this range.

The network obeys the following dynamics for synaptic inputs g:

$$\begin{array}{c}\hfill \tau \frac{\mathrm{d}{g}_{\alpha }(x,t)}{\mathrm{d}t}+g_{\alpha }(x,t)=\sum _{\beta }\int \mathrm{d}y{W}_{\beta }(x,y)s_{\beta }(y,t)+A{\pm }_{\alpha }\gamma b\left(t\right)+{\zeta }_{\alpha }(x,t),\end{array}$$ (33)

where ±_L means − and ±_R means +, and the opposite for ∓_α. τ is the neural time constant, W is the synaptic connectivity, and A is the resting input. The nonlinear activation function ϕ converts synaptic inputs to firing rates:

$$\begin{array}{c}\hfill s_{\alpha }(x,t)=\varphi \left[g_{\alpha }(x,t)\right].\end{array}$$ (34)

Most of our results will apply to general ϕ, but we also consider a ReLU activation function specifically:

$$\begin{array}{c}\hfill \varphi \left[g\right]=\{\begin{array}{cc}g\hfill & g>0\hfill \\ 0\hfill & g\le 0.\hfill \end{array}\end{array}$$ (35)

In this section, we will explicitly mention when we specifically consider the ReLU case, and we will always simplify the function away. Thus, if an expression contains the symbol ϕ, then it holds for general ϕ. In the Results section, formulas for D_input, D_spike, v_drive, and v_conn(θ) as well as all simulation results invoke Eq 35. We will also use this form in the Bump shape g subsection of this section. On the other hand, scalings for D_input, D_spike, v_drive, and v_conn(θ) in the Results section will hold for general ϕ, as long as the connectivity obeys Eq 7 such that g(x/λ) remains invariant over M and N (Fig 2F). b is the driving input, γ is its coupling strength, and ζ is the noise, which can take different forms. γb and ζ are small compared to the rest of the right-hand side of Eq 33. For notational convenience, we will often suppress dependence on t.

W_β(x, y) obeys a standard continuous attractor architecture based on a symmetric and translation-invariant W:

$$\begin{array}{c}\hfill W_{\beta }(x,y)=W(x-y{\mp }_{\beta }\xi )\text{where}W(-x)=W\left(x\right).\end{array}$$ (36)

Each population β deviates from W by a small shift ξ ≪ N in synaptic outputs. Thus, the following approximation holds:

$$\begin{array}{c}\hfill \sum _{\beta }{W}_{\beta }(x,y)\approx 2W(x-y).\end{array}$$ (37)

We will consider the specific form of W (Fig 1B):

$$\begin{array}{c}\hfill W\left(x\right)=\{\begin{array}{cc}w·{\displaystyle \frac{\text{cos}\pi x/l-1}{2}}\hfill & \left|x\right|<2l\hfill \\ 0\hfill & \left|x\right|\ge 2l\hfill \end{array}=\{\begin{array}{cc}w·{\displaystyle \frac{\text{cos}kx-1}{2}}\hfill & \left|x\right|<2\pi /k\hfill \\ 0\hfill & \left|x\right|\ge 2\pi /k,\hfill \end{array}\end{array}$$ (38)

where k = π/l. We will explicitly mention when we specifically consider this form; in fact, we only do so for Eqs 46, 47, 59 and 60, as well as for our simulation results in the Results section. Otherwise, each expression holds for general W.

Baseline configuration without drive and noise

Linearized dynamics and bump distance λ

First, we consider the case of no drive b = 0 and no noise ζ = 0. The dynamical equation Eq 33 becomes

$$\begin{array}{c}\hfill \tau \frac{\mathrm{d}{g}_{\alpha }\left(x\right)}{\mathrm{d}t}+g_{\alpha }\left(x\right)=\sum _{\beta }\int \mathrm{d}y{W}_{\beta }(x,y)\varphi \left[g_{\beta }\left(y\right)\right]+A.\end{array}$$ (39)

Since the right-hand side no longer depends on α, g must be the same for both populations, and we can use Eq 37 to obtain

$$\begin{array}{c}\hfill \tau \frac{\mathrm{d}g\left(x\right)}{\mathrm{d}t}+g\left(x\right)=2\int \mathrm{d}yW(x-y)\varphi \left[g\left(y\right)\right]+A.\end{array}$$ (40)

We analyze these dynamics using the Fourier transform $\mathcal{F}$. Our chosen convention, applied to the function h, is

$$\begin{array}{c}\tilde{h}\left(q\right)=\mathcal{F}\left[h\right]\left(q\right)=\int \mathrm{d}x{\mathrm{e}}^{-\mathrm{i}qx}h\left(x\right)\\ h\left(x\right)={\mathcal{F}}^{-1}\left[\tilde{h}\right]\left(x\right)=\int \frac{\mathrm{d}q}{2\pi }{\mathrm{e}}^{\mathrm{i}qx}\tilde{h}\left(q\right).\end{array}$$ (41)

Fourier modes $\tilde{h}\left(q\right)$ represent sinusoids with wavenumber q and corresponding wavelength 2π/q. Applying this transform to Eq 40, we obtain

$$\begin{array}{c}\hfill \tau \frac{\mathrm{d}\tilde{g}\left(q\right)}{\mathrm{d}t}+\tilde{g}\left(q\right)=2\tilde{W}\left(q\right)\mathcal{F}\left[\varphi \right[g\left]\right]\left(q\right)+2\pi A\delta \left(q\right).\end{array}$$ (42)

In this subsection, we consider the case of small deviations, such that g(x) ≈ g₀ and ϕ[g(x)] ≈ ϕ[g₀] + ϕ′[g₀](g(x) − g₀). Then, Eq 42 becomes

$$\begin{array}{c}\hfill \tau \frac{\mathrm{d}\tilde{g}\left(q\right)}{\mathrm{d}t}+\tilde{g}\left(q\right)=2{\varphi }^{\prime }\left[g_0\right]\tilde{W}\left(q\right)\tilde{g}\left(q\right)+2\pi A_0\delta \left(q\right),\end{array}$$ (43)

where $A_0=A+2\tilde{W}\left(0\right)(\varphi \left[g_0\right]-{\varphi }^{\prime }\left[g_0\right]g_0)$. The solution to this linearized equation for q ≠ 0 is

$$\begin{array}{c}\hfill \tilde{g}(q,t)=\tilde{g}(q,0){\mathrm{e}}^{r\left(q\right)t}.\end{array}$$ (44)

Each mode grows exponentially with rate

$$\begin{array}{c}\hfill r\left(q\right)=\frac{2{\varphi }^{\prime }\left[g_0\right]\tilde{W}\left(q\right)-1}{\tau },\end{array}$$ (45)

so the fastest-growing component of g is the one that maximizes $\tilde{W}\left(q\right)$, as stated in Eq 5 of the Results section. The wavelength 2π/q of that component predicts the bump distance λ.

For the specific W in Eq 38, its Fourier transform is

$$\begin{array}{c}\hfill \tilde{W}\left(q\right)=-w\frac{k^2\text{sin}\frac{2\pi q}{k}}{k^{2}q-q^3},\end{array}$$ (46)

so

$$\begin{array}{c}\hfill \lambda =\frac{2\pi }{{\displaystyle \underset{q}{\text{argmin}}\frac{k^2\text{sin}\frac{2\pi q}{k}}{k^{2}q-q^3}}}=\frac{2l}{{\displaystyle \underset{\psi }{\text{argmin}}\frac{\text{sin}2\pi \psi }{\psi -{\psi }^3}}}\approx 2.28l.\end{array}$$ (47)

λ is proportional to l, as also noted by Refs [16, 18, 41], and [40].

Bump shape g

We call the steady-state synaptic inputs g without drive and noise the baseline configuration. To calculate its shape, we must account for the nonlinearity of the activation function ϕ and return to Eq 42. We invoke our particular form of ϕ in Eq 35 to calculate $\mathcal{F}\left[\varphi \right[g\left]\right]\left(q\right)$. g must be periodic, and its periodicity is the bump distance λ with wavenumber κ = 2π/λ. Without loss of generality, we take g to have a bump centered at 0. Since W is symmetric, g is an even function. We define z as the position where g crosses 0:

$$\begin{array}{c}\hfill g\left(z\right)=0.\end{array}$$ (48)

If g is approximately sinusoidal, then g(x) > 0 wherever nλ − z < x < nλ + z for any integer n. The ReLU ϕ in Eq 35 implies

$$\begin{array}{c}\hfill \varphi \left[g\left(x\right)\right]=g\left(x\right)\Phi \left(x\right)\text{where}\Phi \left(x\right)=\sum _{n=-\infty }^{\infty }\Theta [x-n\lambda +z]\Theta [-(x-n\lambda -z)].\end{array}$$ (49)

Θ is the Heaviside step function. The Fourier transform for Φ is

$$\begin{array}{c}\hfill \tilde{\Phi }\left(q\right)=2\frac{\text{sin}qz}{q}\sum _{n=-\infty }^{\infty }{\mathrm{e}}^{-2\pi \mathrm{i}nq/\kappa }=2\frac{\text{sin}qz}{q}\sum _{n=-\infty }^{\infty }\delta (n-\frac{q}{\kappa })=2\kappa \frac{\text{sin}qz}{q}\sum _{n=-\infty }^{\infty }\delta (q-n\kappa ),\end{array}$$ (50)

where the second equality comes from the Fourier series for a Dirac comb. Therefore,

$$\begin{array}{c}\hfill \mathcal{F}\left[\varphi \right[g\left]\right]\left(q\right)=\frac{1}{2\pi }\int \mathrm{d}{q}^{\prime }\tilde{\Phi }(q-q^{\prime })\tilde{g}\left(q^{\prime }\right)=\frac{1}{\pi }\sum _{n=-\infty }^{\infty }\frac{\text{sin}n\kappa z}{n}\tilde{g}(q-n\kappa ),\end{array}$$ (51)

so Eq 42 becomes

$$\begin{array}{c}\hfill \tau \frac{\mathrm{d}\tilde{g}\left(q\right)}{\mathrm{d}t}+\tilde{g}\left(q\right)=\frac{2}{\pi }\tilde{W}\left(q\right)\sum _{n=-\infty }^{\infty }\frac{\text{sin}n\kappa z}{n}\tilde{g}(q-n\kappa )+2\pi A\delta \left(q\right).\end{array}$$ (52)

This equation describes the full dynamics of g with a ReLU activation function. It contains couplings between all modes q that are multiples of the wavenumber κ, which corresponds to the bump distance.

To find the baseline g, we set $\mathrm{d}\tilde{g}/\mathrm{d}t=0$. We also simplify $\tilde{g}\left(q\right)$ by only considering the lowest modes that couple to each other: q = 0, ±κ. Due to symmetry, $\tilde{W}(-q)=\tilde{W}\left(q\right)$ and $\tilde{g}(-q)=\tilde{g}\left(q\right)$. Eq 52 gives

$$\begin{array}{cc}\hfill \tilde{g}\left(0\right)& =\frac{2}{\pi }\tilde{W}\left(0\right)[\kappa z\tilde{g}\left(0\right)+2\text{sin}\left(\kappa z\right)\tilde{g}\left(\kappa \right)]+2\pi A\delta \left(0\right)\\ \hfill \tilde{g}\left(\kappa \right)& =\frac{2}{\pi }\tilde{W}\left(\kappa \right)[\text{sin}\left(\kappa z\right)\tilde{g}\left(0\right)+(\kappa z+\frac{\text{sin}2\kappa z}{2}\left)\tilde{g}\left(\kappa \right)\right].\end{array}$$ (53)

Now we need to impose Eq 48: g(z) = 0. To do so, we note that $\tilde{g}\left(0\right)$ and $\tilde{g}\left(\kappa \right)$ are both proportional to δ(0) according to Eq 53. That means $\tilde{g}\left(q\right)$ has the form

$$\begin{array}{c}\hfill \tilde{g}\left(q\right)=G_0\delta \left(q\right)+G\delta (q-\kappa )+G\delta (q+\kappa ),\end{array}$$ (54)

where G₀ and G are the Fourier modes with delta functions separated. This implies

$$\begin{array}{c}\hfill g\left(x\right)=\frac{G_0}{2\pi }+\frac{G}{\pi }\text{cos}\kappa x,\end{array}$$ (55)

and g(z) = 0 implies

$$\begin{array}{c}\hfill G_0=-2\text{cos}\left(\kappa z\right)G.\end{array}$$ (56)

Substituting Eqs 54 and 56 into Eq 53, we obtain

$$\begin{array}{cc}\hfill \frac{G}{\pi }& =-\frac{\pi A}{2\tilde{W}\left(0\right)(\text{sin}\kappa z-\kappa z\text{cos}\kappa z)+\pi \text{cos}\kappa z}\hfill \\ \hfill \kappa z-\text{cos}\kappa z\text{sin}\kappa z& =\frac{\pi }{2\tilde{W}\left(\kappa \right)}.\hfill \end{array}$$ (57)

We can solve the second equation of Eq 57 for κz and then substitute it into the first equation to obtain G. This then gives us g(x), which becomes through Eqs 55 and 56

$$\begin{array}{c}\hfill g\left(x\right)=\frac{G}{\pi }(\text{cos}\kappa x-\text{cos}\kappa z).\end{array}$$ (58)

In particular, let’s use the W defined by Eq 38 with Fourier transform Eq 46. Then,

$$\begin{array}{c}\hfill \tilde{W}\left(0\right)=-\frac{2\pi w}{k}\text{and}\tilde{W}\left(\kappa \right)=-\frac{w}{\kappa }\frac{k^2}{k^2-{\kappa }^2}\text{sin}\frac{2\pi \kappa }{k}.\end{array}$$ (59)

Thus,

$$\begin{array}{cc}\hfill g\left(x\right)& =\frac{G}{\pi }(\text{cos}\kappa x-\text{cos}\kappa z)\hfill \\ \hfill \kappa z-\text{cos}\kappa z\text{sin}\kappa z& =-\pi /\left[\frac{2w}{\kappa }\frac{k^2}{k^2-{\kappa }^2}\text{sin}\frac{2\pi \kappa }{k}\right]\hfill \\ \hfill \frac{G}{\pi }& =A/\left[\frac{4w}{k}\right(\text{sin}\kappa z-\kappa z\text{cos}\kappa z)-\text{cos}\kappa z].\hfill \end{array}$$ (60)

This provides expressions for a and d in Eq 4 of the Results section, where a = G/π and d = −(G/π)cos κz.

Lyapunov functional and bump distance λ

The dynamical equation in Eq 40 admits a Lyapunov functional. In analogy to the continuous Hopfield model [77], we can define a Lyapunov functional in terms of s(x) = ϕ[g(x)]:

$$\begin{array}{c}\hfill L=-\int \int \mathrm{d}x\mathrm{d}yW(x-y)s\left(x\right)s\left(y\right)+\int \mathrm{d}x{\int }_0^{s\left(x\right)}\mathrm{d}\rho {\varphi }^{-1}\left[\rho \right]-A\int \mathrm{d}xs\left(x\right).\end{array}$$ (61)

The nonlinearity ϕ must be invertible in the range (0, s) for any possible firing rate s. For L to be bounded from below for a network of any size N, we need

1. W(x) to be negative-definite, and

2. ${\int }_0^s\mathrm{d}\rho {\varphi }^{-1}\left[\rho \right]-As$ to be bounded from below for any possible firing rate s.

We can check that these hold for our particular functions. Eq 38 immediately shows that the first condition is met. Eq 35 states that ϕ⁻¹[ρ] = ρ when ρ > 0, so ${\int }_0^s\mathrm{d}\rho {\varphi }^{-1}\left[\rho \right]-As=\frac{1}{2}{s}^2-As$, which satisfies the second condition.

Now we take the time derivative and use Eq 40:

$$\begin{array}{cc}\hfill \frac{\mathrm{d}L}{\mathrm{d}t}& =-\int \mathrm{d}x\{2\int \mathrm{d}yW(x-y)s\left(y\right)-{\varphi }^{-1}\left[s\left(x\right)\right]+A\}\frac{\mathrm{d}s\left(x\right)}{\mathrm{d}t}\hfill \\ & =-\tau \int \mathrm{d}x\frac{\mathrm{d}g\left(x\right)}{\mathrm{d}t}\frac{\mathrm{d}s\left(x\right)}{\mathrm{d}t}\hfill \\ & =-\tau \int \mathrm{d}x{\varphi }^{\prime }\left[g\left(x\right)\right]{\left(\frac{\mathrm{d}g\left(x\right)}{\mathrm{d}t}\right)}^2.\hfill \end{array}$$ (62)

As long as ϕ is a monotonically nondecreasing function, dL/dt ≤ 0. Thus, L is a Lyapunov functional.

Now we seek to simplify Eq 61. Suppose we are very close to a steady-state solution, so dg/dt ≈ 0. We substitute Eq 40 into Eq 61 to obtain

$$\begin{array}{cc}\hfill L& =-\frac{1}{2}\int \mathrm{d}x\left[g\right(x)-A]s\left(x\right)+\int \mathrm{d}x{\int }_0^{s\left(x\right)}\mathrm{d}\rho {\varphi }^{-1}\left[\rho \right]-A\int \mathrm{d}xs\left(x\right)\\ & =-\frac{1}{2}\int \mathrm{d}xg\left(x\right)s\left(x\right)+\int \mathrm{d}x{\int }_0^{s\left(x\right)}\mathrm{d}\rho {\varphi }^{-1}\left[\rho \right]-\frac{A}{2}\int \mathrm{d}xs\left(x\right).\end{array}$$ (63)

Now we invoke our ReLU ϕ from Eq 35 to obtain

$$\begin{array}{c}\hfill L=-\frac{1}{2}\int \mathrm{d}x\left[g\right(x)-s(x\left)\right]s\left(x\right)-\frac{A}{2}\int \mathrm{d}xs\left(x\right)=-\frac{A}{2}\int \mathrm{d}xs\left(x\right).\end{array}$$ (64)

The last equality was obtained by noticing that for any x, either s(x) = 0 or g(x) − s(x) = 0 with our ϕ. Therefore, the stable solution that minimizes L is the one that maximizes the total firing rate.

We can apply our sinusoidal g in Eq 58 to perform the integral, recalling that κ = 2π/λ:

$$\begin{array}{c}\hfill L=-\frac{NAG}{2{\pi }^2}(\text{sin}\kappa z-\kappa z\text{cos}\kappa z),\end{array}$$ (65)

where N is the network size. So L depends on G and the quantity κz, which we will rewrite as ψ. We now simplify Eq 65 using Eq 57:

$$\begin{array}{c}\hfill L=-\frac{N{A}^2(\text{sin}\psi -\psi \text{cos}\psi )}{4\tilde{W}\left(0\right)(\text{sin}\psi -\psi \text{cos}\psi )-2\pi \text{cos}\psi }=\frac{N{A}^2}{-4\tilde{W}\left(0\right)+\frac{2\pi }{\text{tan}\psi -\psi }}.\end{array}$$ (66)

Note that since W is negative-definite, $\tilde{W}\left(0\right)=\int \mathrm{d}xW\left(x\right)<0$. Also note that 1/(tanψ − ψ) is a monotonically decreasing function of ψ in the range [0, π]. Thus, to minimize L, we need to minimize ψ. Meanwhile, Eq 57 now reads $\psi -\text{cos}\psi \text{sin}\psi =\pi /2\tilde{W}\left(\kappa \right)$. The left-hand side is a monotonically increasing function of ψ in the range [0, π], so to minimize ψ, we need to maximize $\tilde{W}\left(\kappa \right)$. Thus, the Lyapunov-stable wavelength λ = 2π/κ is the one that maximizes $\tilde{W}\left(\kappa \right)$. This is the same mode that grows fastest for the linearized dynamics in Eq 45.

Bump motion under drive and noise

Dynamics along the attractor manifold

Now that we have determined the baseline configuration g, including the bump shape and bump distance, we investigate its motion under drive b and noise ζ. We introduce θ to label the position of the configuration. It can be defined as the center of mass or the point of maximum activity of one of the bumps. We expand the full time-dependent configuration with respect to the baseline configuration located at θ:

$$\begin{array}{c}\hfill g_{\alpha }(x,t)=g(x-\theta )+\delta g_{\alpha }(x,t).\end{array}$$ (67)

g(x − θ) solves Eq 40 with dg/dt = 0; to facilitate calculations below, we will write the baseline equation in this form:

$$\begin{array}{c}\hfill g(x-\theta )=\sum _{\beta }\int \mathrm{d}y{W}_{\beta }(x,y)\varphi \left[g(y-\theta )\right]+A.\end{array}$$ (68)

Substituting Eq 67 into Eq 33 and invoking Eq 68, we obtain the following linearized dynamics for δg:

$$\begin{array}{c}\hfill \tau \frac{\mathrm{d}\delta g_{\alpha }(x,t)}{\mathrm{d}t}+\delta g_{\alpha }(x,t)=\sum _{\beta }\int \mathrm{d}y{W}_{\beta }(x,y){\varphi }^{\prime }\left[g(y-\theta )\right]\delta g_{\beta }(y,t){\pm }_{\alpha }\gamma b\left(t\right)+{\zeta }_{\alpha }(x,t).\end{array}$$ (69)

We can rewrite this as

$$\begin{array}{c}\hfill \tau \frac{\mathrm{d}\delta g_{\alpha }(x,t)}{\mathrm{d}t}=\sum _{\beta }\int \mathrm{d}y{K}_{\alpha \beta }(x,y;\theta )\delta g_{\beta }(y,t){\pm }_{\alpha }\gamma b\left(t\right)+{\zeta }_{\alpha }(x,t),\end{array}$$ (70)

where

$$\begin{array}{c}\hfill K_{\alpha \beta }(x,y;\theta )=W_{\beta }(x,y){\varphi }^{\prime }\left[g(y-\theta )\right]-{\delta }_{\alpha \beta }\delta (x-y).\end{array}$$ (71)

We will often suppress the argument of derivatives of g. If we consider a configuration located at θ, dg/dx implies dg(x − θ)/dx. We make the argument explicit when necessary.

If we differentiate Eq 68 by θ, we obtain

$$\begin{array}{cc}\hfill \frac{\mathrm{d}g}{\mathrm{d}x}& =\sum _{\beta }\int \mathrm{d}y{W}_{\beta }(x,y){\varphi }^{\prime }\left[g(y-\theta )\right]\frac{\mathrm{d}g}{\mathrm{d}y}\hfill \\ \hfill 0& =\sum _{\beta }\int \mathrm{d}y{K}_{\alpha \beta }(x,y;\theta )\frac{\mathrm{d}g}{\mathrm{d}y},\hfill \end{array}$$ (72)

which indicates that dg/dx is a right eigenvector of K with eigenvalue 0. To be explicit about this, we recover the discrete case by converting continuous functions to vectors and matrices:

$$\begin{array}{c}\hfill g_i=g(i-\theta ),\Delta g_i=\frac{\mathrm{d}g(x-\theta )}{\mathrm{d}x}{|}_{x=i},K_{\alpha \beta ij}=K_{\alpha \beta }(i,j;\theta ).\end{array}$$ (73)

If we concatenate matrices and vectors across populations as

$$\begin{array}{c}\hfill \mathsf{J}=\left(\begin{array}{cc}{\mathsf{K}}_{\mathrm{L}\mathrm{L}}& {\mathsf{K}}_{\mathrm{L}\mathrm{R}}\\ {\mathsf{K}}_{\mathrm{R}\mathrm{L}}& {\mathsf{K}}_{\mathrm{R}\mathrm{R}}\end{array}\right),\mathbf{e}=\left(\begin{array}{c}\Delta \mathbf{g}\\ \Delta \mathbf{g}\end{array}\right),\end{array}$$ (74)

e is the right null eigenvector of $\mathsf{J}:0={\displaystyle {\sum }_j{J}_{ij}}{e}_j$.

Since K is not symmetric, its left and right eigenvectors may be different. To find the left null eigenvector, we again differentiate Eq 68 with respect to θ, but this time interchanging variables x and y:

$$\begin{array}{cc}\hfill \frac{\mathrm{d}g}{\mathrm{d}y}& =\sum _{\beta }\int \mathrm{d}x{W}_{\beta }(y,x){\varphi }^{\prime }\left[g(x-\theta )\right]\frac{\mathrm{d}g}{\mathrm{d}x}\\ & \approx 2\int \mathrm{d}xW(x-y){\varphi }^{\prime }\left[g(x-\theta )\right]\frac{\mathrm{d}g}{\mathrm{d}x}.\end{array}$$ (75)

The second equality is obtained from Eqs 36 and 37. Replacing the position y by y ±_β ξ, where ξ is the connectivity shift, we get

$$\begin{array}{c}\hfill \frac{\mathrm{d}g(y-\theta {\pm }_{\beta }\xi )}{\mathrm{d}y}\approx 2\int \mathrm{d}xW(x-y{\mp }_{\beta }\xi ){\varphi }^{\prime }\left[g(x-\theta )\right]\frac{\mathrm{d}g(x-\theta )}{\mathrm{d}x},\end{array}$$ (76)

where we have made the arguments of g explicit. Let’s define shifted versions of the baseline g for each population α:

$$\begin{array}{c}\hfill {\overline{g}}_{\alpha }\left(x\right)=g\left(x{\pm }_{\alpha }\xi \right).\end{array}$$ (77)

Since ξ is small,

$$\begin{array}{c}\hfill \sum _{\alpha }{\overline{g}}_{\alpha }\left(x\right)\approx 2g\left(x\right).\end{array}$$ (78)

Applying these expressions to Eq 76 and recalling Eq 36,

$$\begin{array}{cc}\hfill \frac{\mathrm{d}{\overline{g}}_{\beta }}{\mathrm{d}y}& \approx 2\int \mathrm{d}x{W}_{\beta }(x,y){\varphi }^{\prime }\left[g(x-\theta )\right]\frac{\mathrm{d}g}{\mathrm{d}x}\hfill \\ & \approx \sum _{\alpha }\int \mathrm{d}x{W}_{\beta }(x,y){\varphi }^{\prime }\left[g(x-\theta )\right]\frac{\mathrm{d}{\overline{g}}_{\alpha }}{\mathrm{d}x}.\hfill \end{array}$$ (79)

Finally, we multiply both sides of the equation by ϕ′[g(y − θ)] to obtain

$$\begin{array}{cc}\hfill {\varphi }^{\prime }\left[g(y-\theta )\right]\frac{\mathrm{d}{\overline{g}}_{\beta }}{\mathrm{d}y}& \approx \sum _{\alpha }\int \mathrm{d}x{W}_{\beta }(x,y){\varphi }^{\prime }\left[g(y-\theta )\right]{\varphi }^{\prime }\left[g(x-\theta )\right]\frac{\mathrm{d}{\overline{g}}_{\alpha }}{\mathrm{d}x}\hfill \\ \hfill 0& =\sum _{\alpha }\int \mathrm{d}x{K}_{\alpha \beta }(x,y;\theta ){\varphi }^{\prime }\left[g(x-\theta )\right]\frac{\mathrm{d}{\overline{g}}_{\alpha }}{\mathrm{d}x}.\hfill \end{array}$$ (82)

Thus ${\varphi }^{\prime }\left[g(x-\theta )\right]\mathrm{d}{\overline{g}}_{\alpha }/\mathrm{d}x$ is the left null eigenvector for K_αβ. Again, to be explicit, the discrete equivalent is

$$\begin{array}{c}\hfill \mathsf{J}=\left(\begin{array}{cc}{\mathsf{K}}_{\mathrm{L}\mathrm{L}}\hfill & {\mathsf{K}}_{\mathrm{L}\mathrm{R}}\\ {\mathsf{K}}_{\mathrm{R}\mathrm{L}}& {\mathsf{K}}_{\mathrm{R}\mathrm{R}}\end{array}\right),\mathbf{f}=\left(\begin{array}{c}{\varphi }^{\prime }\left[\mathbf{g}\right]\odot \Delta {\overline{\mathbf{g}}}_{\mathrm{L}}\\ {\varphi }^{\prime }\left[\mathbf{g}\right]\odot \Delta {\overline{\mathbf{g}}}_{\mathrm{R}}\end{array}\right),\end{array}$$ (81)

where ⊙ represents element-wise (Hadamard) multiplication. Then, f is the left null eigenvector of $\mathsf{J}:0={\displaystyle {\sum }_i{J}_{ij}}{f}_i$.

We now revisit Eq 67 and assume that g changes such that the bumps slowly move along the attractor manifold:

$$\begin{array}{c}{g}_{\alpha }(x,t)\approx g(x-\theta \left(t\right)),\\ \frac{\mathrm{d}\delta g_{\alpha }(x,t)}{\mathrm{d}t}=\frac{\mathrm{d}{g}_{\alpha }(x,t)}{\mathrm{d}t}\approx -\frac{\mathrm{d}g(x-\theta (t\left)\right)}{\mathrm{d}x}\frac{\mathrm{d}\theta }{\mathrm{d}t}.\end{array}$$ (82)

Again for simplicity, we will often suppress arguments of derivatives of g and dependence on t. We return to Eq 70, project it along the left null eigenvector, and apply Eq 82 to obtain

$$\begin{array}{cc}\hfill -\tau \frac{\mathrm{d}\theta }{\mathrm{d}t}\sum _{\alpha }\int \mathrm{d}x{\varphi }^{\prime }\left[g(x-\theta )\right]\frac{\mathrm{d}{\overline{g}}_{\alpha }}{\mathrm{d}x}\frac{\mathrm{d}g}{\mathrm{d}x}=& \gamma b\sum _{\alpha }\int \mathrm{d}x\left({\pm }_{\alpha }1\right)·{\varphi }^{\prime }\left[g(x-\theta )\right]\frac{\mathrm{d}{\overline{g}}_{\alpha }}{\mathrm{d}x}\\ & +\sum _{\alpha }\int \mathrm{d}x{\varphi }^{\prime }\left[g(x-\theta )\right]\frac{\mathrm{d}{\overline{g}}_{\alpha }}{\mathrm{d}x}{\zeta }_{\alpha }\left(x\right).\end{array}$$ (83)

The velocity of bump motion is given by dθ/dt. It is

$$\begin{array}{cc}\hfill \frac{\mathrm{d}\theta }{\mathrm{d}t}\approx & -\frac{{\displaystyle \gamma b\sum _{\alpha }\int \mathrm{d}x\left({\pm }_{\alpha }1\right)·{\varphi }^{\prime }\left[g(x-\theta )\right]\frac{\mathrm{d}{\overline{g}}_{\alpha }(x-\theta )}{\mathrm{d}x}}}{{\displaystyle 2\tau \int \mathrm{d}x{\varphi }^{\prime }\left[g(x-\theta )\right]{\left(\frac{\mathrm{d}g(x-\theta )}{\mathrm{d}x}\right)}^2}}\\ & -\frac{{\displaystyle \sum _{\alpha }\int \mathrm{d}x{\varphi }^{\prime }\left[g(x-\theta )\right]\frac{\mathrm{d}{\overline{g}}_{\alpha }(x-\theta )}{\mathrm{d}x}{\zeta }_{\alpha }\left(x\right)}}{{\displaystyle 2\tau \int \mathrm{d}x{\varphi }^{\prime }\left[g(x-\theta )\right]{\left(\frac{\mathrm{d}g(x-\theta )}{\mathrm{d}x}\right)}^2}},\end{array}$$ (84)

where we have made the arguments of g explicit. This equation encapsulates all aspects of bump motion for our theoretical model. It includes dependence on both drive b and noise ζ, the latter of which is kept in a general form. We will proceed by considering specific cases of this equation.

Path integration velocity v_drive due to driving input b

The noiseless case of Eq 84 with ζ_α(x) = 0 yields the bump velocity due to drive b, which is responsible for path integration:

$$\begin{array}{c}\hfill v_{\text{drive}}=-\frac{{\displaystyle \gamma b\int \mathrm{d}x{\varphi }^{\prime }\left[g(x-\theta )\right](\frac{\mathrm{d}{\overline{g}}_{\mathrm{R}}}{\mathrm{d}x}-\frac{\mathrm{d}{\overline{g}}_{\mathrm{L}}}{\mathrm{d}x})}}{{\displaystyle 2\tau \int \mathrm{d}x{\varphi }^{\prime }\left[g(x-\theta )\right]{\left(\frac{\mathrm{d}g}{\mathrm{d}x}\right)}^2}}.\end{array}$$ (85)

Note that this expression is independent of the position θ. We can explicitly remove θ by shifting the dummy variable x → x + θ:

$$\begin{array}{cc}\hfill v_{\text{drive}}& =-\frac{{\displaystyle \gamma b\int \mathrm{d}x{\varphi }^{\prime }\left[g\left(x\right)\right](\frac{\mathrm{d}g(x+\xi )}{\mathrm{d}x}-\frac{\mathrm{d}g(x-\xi )}{\mathrm{d}x})}}{{\displaystyle 2\tau \int \mathrm{d}x{\varphi }^{\prime }\left[g\left(x\right)\right]{\left(\frac{\mathrm{d}g\left(x\right)}{\mathrm{d}x}\right)}^2}}\hfill \\ & \approx -\frac{{\displaystyle \gamma b\xi \int \mathrm{d}x{\varphi }^{\prime }\left[g\left(x\right)\right]\frac{{\mathrm{d}}^{2}g}{\mathrm{d}{x}^2}}}{{\displaystyle \tau \int \mathrm{d}x{\varphi }^{\prime }\left[g\left(x\right)\right]{\left(\frac{\mathrm{d}g}{\mathrm{d}x}\right)}^2}}.\hfill \end{array}$$ (86)

Now let’s consider the specific ReLU activation function ϕ. Eq 35 implies

$$\begin{array}{c}\hfill {\varphi }^{\prime }\left[g\right]=\{\begin{array}{cc}1\hfill & g>0\hfill \\ 0\hfill & g\le 0,\hfill \end{array}\text{so}{\varphi }^{\prime }{\left[g\right]}^2={\varphi }^{\prime }\left[g\right]\text{and}{\varphi }^{\prime }\left[g\right]·\varphi \left[g\right]=\varphi \left[g\right].\end{array}$$ (87)

These identities, along with the definition for s (Eq 34), give

$$\begin{array}{c}\hfill {\varphi }^{\prime }\left[g\left(x\right)\right]\frac{{\mathrm{d}}^{2}g}{\mathrm{d}{x}^2}=\frac{{\mathrm{d}}^{2}s}{\mathrm{d}{x}^2},{\varphi }^{\prime }\left[g\left(x\right)\right]{\left(\frac{\mathrm{d}g}{\mathrm{d}x}\right)}^2={\left(\frac{\mathrm{d}s}{\mathrm{d}x}\right)}^2,\varphi \left[g\left(x\right)\right]{\left(\frac{\mathrm{d}g}{\mathrm{d}x}\right)}^2=s\left(x\right){\left(\frac{\mathrm{d}s}{\mathrm{d}x}\right)}^2.\end{array}$$ (88)

Applying the first two equalities to Eq 86 produces Eq 8 of the Results section.

Now we reintroduce noise ζ and assume it is independent across neurons and timesteps, with mean 〈ζ〉. If we average Eq 84 over ζ, the numerator of the second term becomes

$$\begin{array}{c}\hfill \sum _{\alpha }\int \mathrm{d}x{\varphi }^{\prime }\left[g(x-\theta )\right]\frac{\mathrm{d}{\overline{g}}_{\alpha }(x-\theta )}{\mathrm{d}x}⟨\zeta ⟩=0.\end{array}$$ (89)

The integral vanishes because g is even and ${\sum }_{\alpha }\mathrm{d}{\overline{g}}_{\alpha }/\mathrm{d}x$ is odd. Thus,

$$\begin{array}{c}\hfill ⟨\frac{\mathrm{d}\theta }{\mathrm{d}t}⟩=v_{\text{drive}},\end{array}$$ (90)

demonstrating that networks with independent noise still path integrate on average.

Diffusion D_input due to input noise

Independent noise ζ produces diffusion, a type of deviation in bump motion away from the average trajectory. It is quantified by the diffusion coefficient D:

$$\begin{array}{c}\hfill ⟨{\left[\theta \right(t)-⟨\theta (t)⟩]}^2⟩=2Dt.\end{array}$$ (91)

In terms of derivatives of θ,

$$\begin{array}{c}\hfill ⟨{\left[\theta \right(t)-⟨\theta (t)⟩]}^2⟩={\int }_0^t{\int }_0^t\mathrm{d}{t}^{\prime }\mathrm{d}{t}^{\prime \prime }⟨(\frac{\mathrm{d}\theta }{\mathrm{d}{t}^{\prime }}-⟨\frac{\mathrm{d}\theta }{\mathrm{d}{t}^{\prime }}⟩)(\frac{\mathrm{d}\theta }{\mathrm{d}{t}^{\prime \prime }}-⟨\frac{\mathrm{d}\theta }{\mathrm{d}{t}^{\prime \prime }}⟩)⟩.\end{array}$$ (92)

Eqs 84 and 90 imply

$$\begin{array}{c}\hfill \frac{\mathrm{d}\theta }{\mathrm{d}t}-⟨\frac{\mathrm{d}\theta }{\mathrm{d}t}⟩=-\frac{{\displaystyle \sum _{\alpha }\int \mathrm{d}x{\varphi }^{\prime }\left[g(x-\theta )\right]\frac{\mathrm{d}{\overline{g}}_{\alpha }}{\mathrm{d}x}{\zeta }_{\alpha }\left(x\right)}}{{\displaystyle 2\tau \int \mathrm{d}x{\varphi }^{\prime }\left[g(x-\theta )\right]{\left(\frac{\mathrm{d}g}{\mathrm{d}x}\right)}^2}}.\end{array}$$ (93)

We then shift the dummy variable x → x + θ(t) and reintroduce explicit dependence on t to obtain

$$\begin{array}{c}\hfill ⟨{\left[\theta \right(t)-⟨\theta (t)⟩]}^2⟩={\int }_0^t{\int }_0^t\mathrm{d}{t}^{\prime }\mathrm{d}{t}^{\prime \prime }\frac{{\displaystyle \sum _{\alpha \beta }\int \int \mathrm{d}x\mathrm{d}y{\varphi }^{\prime }\left[g\left(x\right)\right]{\varphi }^{\prime }\left[g\left(y\right)\right]\frac{\mathrm{d}{\overline{g}}_{\alpha }}{\mathrm{d}x}\frac{\mathrm{d}{\overline{g}}_{\beta }}{\mathrm{d}y}⟨{\zeta }_{\alpha }(x+\theta \left(t^{\prime }\right),t^{\prime }){\zeta }_{\beta }(y+\theta \left(t^{\prime \prime }\right),t^{\prime \prime })⟩}}{{\displaystyle 4{\tau }^2{[\int \mathrm{d}x{\varphi }^{\prime }\left[g\left(x\right)\right]{\left(\frac{\mathrm{d}g}{\mathrm{d}x}\right)}^2]}^2}}.\end{array}$$ (94)

One class of independent ζ is Gaussian noise added to the total synaptic input, which represents neural fluctuations at short timescales. We assume it is independent across neurons and timesteps with zero mean and fixed variance σ²:

$$\begin{array}{c}\hfill ⟨{\zeta }_{\alpha }(x,t)⟩=0,⟨{\zeta }_{\alpha }(x,t){\zeta }_{\beta }(y,t^{\prime })⟩={\sigma }^2\Delta t\delta (t-t^{\prime }){\delta }_{\alpha \beta }\delta (x-y).\end{array}$$ (95)

Δt is the simulation timestep, which defines the rate at which the random noise variable is resampled. Eq 94 then becomes, with the help of Eq 78,

$$\begin{array}{cc}\hfill ⟨{\left[\theta \right(t)-⟨\theta (t)⟩]}^2⟩& ={\int }_0^t\mathrm{d}{t}^{\prime }\frac{{\displaystyle {\sigma }^2\Delta t\sum _{\alpha }\int \mathrm{d}x{\varphi }^{\prime }{\left[g\left(x\right)\right]}^2{\left(\frac{\mathrm{d}{\overline{g}}_{\alpha }}{\mathrm{d}x}\right)}^2}}{{\displaystyle 4{\tau }^2{[\int \mathrm{d}x{\varphi }^{\prime }\left[g\left(x\right)\right]{\left(\frac{\mathrm{d}g}{\mathrm{d}x}\right)}^2]}^2}}\hfill \\ & \approx \frac{{\displaystyle {\sigma }^2\Delta t\int \mathrm{d}x{\varphi }^{\prime }{\left[g\left(x\right)\right]}^2{\left(\frac{\mathrm{d}g}{\mathrm{d}x}\right)}^2}}{{\displaystyle 2{\tau }^2{[\int \mathrm{d}x{\varphi }^{\prime }\left[g\left(x\right)\right]{\left(\frac{\mathrm{d}g}{\mathrm{d}x}\right)}^2]}^2}}·t.\hfill \end{array}$$ (96)

Reconciling this with the definition of the diffusion coefficient D in Eq 91 yields

$$\begin{array}{c}\hfill D_{\text{input}}=\frac{{\displaystyle {\sigma }^2\Delta t\int \mathrm{d}x{\varphi }^{\prime }{\left[g\left(x\right)\right]}^2{\left(\frac{\mathrm{d}g}{\mathrm{d}x}\right)}^2}}{{\displaystyle 4{\tau }^2{[\int \mathrm{d}x{\varphi }^{\prime }\left[g\left(x\right)\right]{\left(\frac{\mathrm{d}g}{\mathrm{d}x}\right)}^2]}^2}}.\end{array}$$ (97)

Applying Eq 88 for a ReLU ϕ gives Eq 10 of the Results section.

Diffusion D_spike due to spiking noise

Instead of input noise, we consider independent noise arising from spiking neurons. In this case, the stochastic firing rate s is no longer the deterministic expression in Eq 34. Instead,

$$\begin{array}{c}\hfill s_{\alpha }(x,t)=\frac{c_{\alpha }(x,t)}{\Delta t},\end{array}$$ (98)

where c is the number of spikes emitted in a simulation timestep of length Δt. We model each c_α(x, t) as an independent Poisson-like random variable driven by the deterministic firing rate ϕ[g_α(x, t)] with Fano factor F. It has mean ϕ[g_α(x, t)]Δt and variance Fϕ[g_α(x, t)]Δt. Therefore,

$$\begin{array}{c}\hfill s_{\alpha }(x,t)=\varphi \left[g_{\alpha }(x,t)\right]+\sqrt{\frac{}{}}F\varphi \left[g_{\alpha }(x,t)\right]\Delta t{\eta }_{\alpha }(x,t),\end{array}$$ (99)

where each η_α(x, t) is an independent random variable with zero mean and unit variance:

$$\begin{array}{c}\hfill ⟨{\eta }_{\alpha }(x,t)⟩=0,⟨{\eta }_{\alpha }(x,t){\eta }_{\beta }(y,t^{\prime })⟩=\Delta t\delta (t-t^{\prime }){\delta }_{\alpha \beta }\delta (x-y).\end{array}$$ (100)

As in Eq 95, the simulation timestep Δt defines the rate at which η is resampled. By substituting Eq 99 into Eq 33, we see that spiking neurons can be described by deterministic firing rate dynamics with the stochastic noise term

$$\begin{array}{c}\hfill {\zeta }_{\alpha }(x,t)=\sum _{\beta }\int \mathrm{d}y{W}_{\beta }(x,y)\sqrt{\frac{}{}}F\varphi \left[g_{\beta }(y,t)\right]\Delta t{\eta }_{\beta }(y,t).\end{array}$$ (101)

Now we calculate the diffusion coefficient produced by this noise. Eq 93 becomes

$$\begin{array}{cc}\hfill \frac{\mathrm{d}\theta }{\mathrm{d}t}-⟨\frac{\mathrm{d}\theta }{\mathrm{d}t}⟩& =-\frac{{\displaystyle \sum _{\alpha \beta }\int \int \mathrm{d}x\mathrm{d}y{W}_{\beta }(x,y){\varphi }^{\prime }\left[g(x-\theta )\right]\frac{\mathrm{d}{\overline{g}}_{\alpha }}{\mathrm{d}x}\sqrt{\frac{}{}}F\varphi \left[g\right(y-\theta \left)\right]\Delta t{\eta }_{\beta }\left(y\right)}}{{\displaystyle 2\tau \int \mathrm{d}x{\varphi }^{\prime }\left[g(x-\theta )\right]{\left(\frac{\mathrm{d}g}{\mathrm{d}x}\right)}^2}}\hfill \\ & =-\frac{{\displaystyle \sum _{\beta }\int \mathrm{d}y\frac{\mathrm{d}{\overline{g}}_{\beta }}{\mathrm{d}y}\sqrt{\frac{}{}}F\varphi \left[g\right(y-\theta \left)\right]\Delta t{\eta }_{\beta }\left(y\right)}}{{\displaystyle 2\tau \int \mathrm{d}x{\varphi }^{\prime }\left[g(x-\theta )\right]{\left(\frac{\mathrm{d}g}{\mathrm{d}x}\right)}^2}}.\hfill \end{array}$$ (102)

We recalled Eqs 67 and 79 to obtain these equalities. We then proceed as for input noise to calculate

$$\begin{array}{c}\hfill ⟨{\left[\theta \right(t)-⟨\theta (t)⟩]}^2⟩={\int }_0^t{\int }_0^t\mathrm{d}{t}^{\prime }\mathrm{d}{t}^{\prime \prime }\frac{{\displaystyle \frac{F}{\Delta t}\sum _{\alpha \beta }\int \int \mathrm{d}x\mathrm{d}y\sqrt{\varphi \left[g\right(x\left)\right]\varphi \left[g\right(y\left)\right]}\frac{\mathrm{d}{\overline{g}}_{\alpha }}{\mathrm{d}x}\frac{\mathrm{d}{\overline{g}}_{\beta }}{\mathrm{d}y}⟨{\eta }_{\alpha }(x+\theta \left(t^{\prime }\right),t^{\prime }){\eta }_{\beta }(y+\theta \left(t^{\prime \prime }\right),t^{\prime \prime })⟩}}{{\displaystyle 4{\tau }^2{[\int \mathrm{d}x{\varphi }^{\prime }\left[g\left(x\right)\right]{\left(\frac{\mathrm{d}g}{\mathrm{d}x}\right)}^2]}^2}},\end{array}$$ (103)

which yields the diffusion coefficient

$$\begin{array}{c}\hfill D_{\text{spike}}=\frac{{\displaystyle F\int \mathrm{d}x\varphi \left[g\left(x\right)\right]{\left(\frac{\mathrm{d}g}{\mathrm{d}x}\right)}^2}}{{\displaystyle 4{\tau }^2{[\int \mathrm{d}x{\varphi }^{\prime }\left[g\left(x\right)\right]{\left(\frac{\mathrm{d}g}{\mathrm{d}x}\right)}^2]}^2}}.\end{array}$$ (104)

After applying Eq 88 for a ReLU ϕ and setting F = 1 for Poisson spiking, we obtain Eq 20 of the Results section.

Drift velocity v_conn(θ) due to quenched connectivity noise

Suppose that we perturb the symmetric, translation-invariant W by a small component V representing deviations away from an ideal attractor architecture:

$$\begin{array}{c}\hfill W_{\beta }(x,y)\to W_{\beta }(x,y)+V_{\alpha \beta }(x,y).\end{array}$$ (105)

By Eq 33, this produces the noise term

$$\begin{array}{c}\hfill {\zeta }_{\alpha }(x,t)=\sum _{\beta }\int \mathrm{d}y{V}_{\alpha \beta }(x,y)\varphi \left[g_{\beta }(y,t)\right].\end{array}$$ (106)

In contrast to input and spiking noise, this noise is correlated across neurons and time, so it cannot be averaged away as in Eqs 89 and 90. Substituting Eq 106 into Eq 84, we obtain

$$\begin{array}{c}\hfill \frac{\mathrm{d}\theta }{\mathrm{d}t}=v_{\text{drive}}+v_{\text{conn}}\left(\theta \right),\end{array}$$ (107)

where the drift velocity is

$$\begin{array}{c}\hfill v_{\text{conn}}\left(\theta \right)=-\frac{{\displaystyle \sum _{\alpha \beta }\int \int \mathrm{d}x\mathrm{d}y{V}_{\alpha \beta }(x,y){\varphi }^{\prime }\left[g(x-\theta )\right]\frac{\mathrm{d}g(x-\theta )}{\mathrm{d}x}\varphi \left[g(y-\theta )\right]}}{{\displaystyle 2\tau \int \mathrm{d}x{\varphi }^{\prime }\left[g(x-\theta )\right]{\left(\frac{\mathrm{d}g(x-\theta )}{\mathrm{d}x}\right)}^2}}.\end{array}$$ (108)

Because V is already small, we ignored ξ in Eq 77 to obtain this expression. We have also made the dependence on bump position θ explicit to illustrate how it influences v_conn(θ). After applying Eq 88 for a ReLU ϕ, we obtain Eq 24 of the Results section.

We now make scaling arguments for speed difference (Eq 30), speed variability (Eq 31), and escape drive b₀ (Eq 26). To do so, we impose a ReLU ϕ and return to discrete variables to be explicit:

$$\begin{array}{c}\hfill v_{\text{conn};\theta }=-\frac{{\displaystyle \sum _{\alpha \beta }\sum _{ij}{V}_{\alpha \beta ij}·\Delta s_{i-\theta }·s_{j-\theta }}}{{\displaystyle 2\tau \sum _i{(\Delta s_{i-\theta })}^2}}.\end{array}$$ (109)

We need to understand how the numerator scales with M and N. It is a weighted sum of 4N² independent Gaussian random variables V_αβij and is thus a Gaussian random variable itself. It has zero mean, but its variance is proportional to N² ⋅ M²/N². The N² comes from the number of terms in the sum and the M²/N² comes from the scaling of ds/dx (Eq 11). In combination with the scaling of the denominator, we conclude that v_conn;θ is a Gaussian random variable with

$$\begin{array}{c}\hfill \mathrm{E}\left[v_{\text{conn};\theta }\right]=0,\text{Var}\left[v_{\text{conn};\theta }\right]\propto \frac{N^2}{M^2}.\end{array}$$ (110)

Eq 109 implies that v_conn;θ is correlated over θ. The weights for the sum over V_αβij are the firing rates and their derivatives for a bump centered at θ. If θ is slightly changed, almost the same entries of V will be summed over with similar weights. The amount of correlation across θ is determined by the degree of overlap in weights, and therefore, by the width and number of bumps. Let’s consider the effects of changing N and M on the covariance matrix Cov[v_conn;θ, v_conn;θ′]. A larger N increases the bump width and the correlation length proportionally, so values of the main diagonal decay proportionally more slowly into the off diagonals. A larger M redistributes values among the diagonals by decreasing the bump width and adding more bumps, but it does not change the total amount of correlation. Thus,

$$\begin{array}{c}\hfill \sum _{\theta ,{\theta }^{\prime }}\text{Cov}[v_{\text{conn};\theta },v_{\text{conn};{\theta }^{\prime }}]\propto N^2·\text{Var}\left[v_{\text{conn};\theta }\right].\end{array}$$ (111)

This allows us to evaluate

$$\begin{array}{c}\hfill \text{Var}\left[\underset{\theta }{\text{mean}}{v}_{\text{conn};\theta }\right]=\text{Var}\left[\frac{1}{N}\sum _{\theta }{v}_{\text{conn};\theta }\right]=\frac{1}{N^2}\sum _{\theta ,{\theta }^{\prime }}\text{Cov}[v_{\text{conn};\theta },v_{\text{conn};{\theta }^{\prime }}]\propto \frac{N^2}{M^2}.\end{array}$$ (112)

As a sum of zero-mean Gaussian random variables, mean_θ v_conn;θ is also a zero-mean Gaussian random variable. That means |mean_θ v_conn;θ| follows a folded normal distribution, which obeys

$$\begin{array}{c}\hfill \mathrm{E}\left[\right|\underset{\theta }{\text{mean}}{v}_{\text{conn};\theta }\left|\right]=\sqrt{\frac{2}{\pi }\text{Var}\left[\underset{\theta }{\text{mean}}{v}_{\text{conn};\theta }\right]}\propto \frac{N}{M}.\end{array}$$ (113)

Combining this with Eqs 12 and 14 produces the scalings for speed difference in Eq 32.

We now study speed variability, which involves the expression

$$\begin{array}{c}\hfill \underset{\theta }{\text{std}}{v}_{\text{conn};\theta }=\sqrt{\frac{1}{N}\sum _{\theta }{v}_{\text{conn};\theta }^2}.\end{array}$$ (114)

Since each v_conn;θ is Gaussian, the sum of their squares follows a generalized chi-square distribution. Its mean is the trace of the covariance matrix Cov[v_conn;θ, v_conn;θ′], which is equal to N times the variance. Thus, by Eq 110,

$$\begin{array}{c}\hfill \mathrm{E}\left[\frac{1}{N}\sum _{\theta }{v}_{\text{conn};\theta }^2\right]=\frac{1}{N}·N·\text{Var}\left[v_{\text{conn};\theta }\right]\propto \frac{N^2}{M^2}.\end{array}$$ (115)

We are interested in the square root of the random variable on the left-hand side, and we anticipate its expected value to scale as the square root of the right-hand side. We can make this argument precise. Suppose H is a random variable with a probability distribution function p(h) that scales with a power of the parameter B. We can write

$$\begin{array}{c}\hfill p\left(h\right)=B^{n}P\left(B^{m}h\right)\end{array}$$ (116)

for exponents n and m, where the rescaled probability distribution function P does not scale with B. Conservation of total probability implies

$$\begin{array}{c}\hfill B^n\int \mathrm{d}hP\left(B^{m}h\right)=B^n{B}^{-m}\int \mathrm{d}{h}^{\prime }P\left(h^{\prime }\right)=1.\end{array}$$ (117)

Thus, m = n. Next, suppose we know that E[H] ∝ B^o:

$$\begin{array}{c}\hfill \mathrm{E}\left[H\right]=B^n\int \mathrm{d}hhP\left(B^{n}h\right)=B^{-n}\int \mathrm{d}{h}^{\prime }{h}^{\prime }P\left(h^{\prime }\right)\propto B^o.\end{array}$$ (118)

Thus, n = −o. We can now conclude that $\mathrm{E}\left[\sqrt{H}\right]\propto \sqrt{\mathrm{E}\left[H\right]}$:

$$\begin{array}{c}\hfill \mathrm{E}\left[\sqrt{H}\right]=B^{-o}\int \mathrm{d}h\sqrt{h}P\left(B^{-o}h\right)=B^{o/2}\int \mathrm{d}{h}^{\prime }\sqrt{h^{\prime }}P\left(h^{\prime }\right)\propto B^{o/2}.\end{array}$$ (119)

Applying this result to Eq 115, we obtain

$$\begin{array}{c}\hfill \mathrm{E}\left[{\displaystyle \underset{\theta }{\text{std}}}{v}_{\text{conn};\theta }\right]=\mathrm{E}\left[\sqrt{\frac{1}{N}\sum _{\theta }{v}_{\text{conn};\theta }^2}\right]\propto \sqrt{\mathrm{E}\left[\frac{1}{N}\sum _{\theta }{v}_{\text{conn};\theta }^2\right]}\propto \frac{N}{M}.\end{array}$$ (120)

Combining this with Eqs 12 and 14 produces the scalings for speed variability in Eq 32.

The escape drive b₀ involves the expression max_θ|v_conn;θ|. Extreme value statistics for correlated random variables is generally poorly understood. We follow Ref [78] and provide a heuristic argument for its scaling. We can partition v_conn;θ across θ into groups that are largely independent from one another based on its correlation structure. As discussed above, v_conn;θ is a weighted sum of independent Gaussian random variables V_αβij (Eq 109). The weights are products between the firing rates s_{j − θ} and their derivatives Δs_{i − θ} for a configuration centered at position θ. If we choose two θ’s such that their bumps do not overlap, the corresponding v_conn;θ’s will sum over different V_αβij’s and will be independent. Thus, λ/z roughly sets the number of independent components, where λ is the bump distance and z is the bump width. This ratio does not change with M or N in our networks (Fig 2F), so the maximum function does not change the scaling of |v_conn;θ|:

$$\begin{array}{c}\hfill \underset{\theta }{\text{max}}|v_{\text{conn};\theta }|\propto |v_{\text{conn};\theta }|.\end{array}$$ (121)

The scaling of E[|v_conn;θ|] can be determined from Var[v_conn;θ] through arguments similar to those made in Eqs 116–119. Suppose we know that Var[H] ∝ B^o and E[H] = 0. Then,

$$\begin{array}{c}\hfill \text{Var}\left[H\right]=B^n\int \mathrm{d}h{h}^{2}P\left(B^{n}h\right)=B^{-2n}\int \mathrm{d}{h}^{\prime }{\left(h^{\prime }\right)}^{2}P\left(h^{\prime }\right)\propto B^o.\end{array}$$ (122)

Thus, n = −o/2. We can now conclude that $\mathrm{E}\left[\right|H\left|\right]\propto \sqrt{\text{Var}\left[H\right]}$:

$$\begin{array}{c}\hfill \mathrm{E}\left[\right|H\left|\right]=B^{-o/2}\int \mathrm{d}h\left|h\right|P\left(B^{-o/2}h\right)=B^{o/2}\int \mathrm{d}{h}^{\prime }\left|h^{\prime }\right|P\left(h^{\prime }\right)\propto B^{o/2}.\end{array}$$ (123)

Applying this result to Eq 121, we obtain

$$\begin{array}{c}\hfill \mathrm{E}\left[\underset{\theta }{\text{max}}\left|v_{\text{conn};\theta }\right|\right]\propto \mathrm{E}\left[|v_{\text{conn};\theta }|\right]\propto \sqrt{\text{Var}\left[v_{\text{conn};\theta }\right]}\propto \frac{N}{M}.\end{array}$$ (124)

Combining this with Eqs 12, 13 and 26 produces the scalings for the escape drive b₀ in Eq 27.

Simulation methods

Dynamics and parameter values

To simulate the dynamics in Eq 33, we discretize the network by replacing neural position x with index i and propagate forward in time with the simple Euler method:

$$\begin{array}{c}\hfill \tau \frac{g_{\alpha i}(t+\Delta t)-g_{\alpha i}\left(t\right)}{\Delta t}+g_{\alpha i}\left(t\right)=\sum _{\beta j}{W}_{\beta ij}{s}_{\beta j}\left(t\right)+A{\pm }_{\alpha }\gamma b\left(t\right)+{\zeta }_{\alpha i}\left(t\right).\end{array}$$ (125)

We use τ = 10 ms. We use Δt = 0.5 ms and A = 1 for all simulations except those with spiking neurons. In the latter case, we use finer timesteps Δt = 0.1 ms and set A = 0.1 ms^-1. Synaptic inputs g and resting inputs A can be dimensionless for rate-based simulations, but they must have units of rate for spiking simulations. We use γ = 0.1 for rate-based simulations and γ = 0.01 ms^-1 for spiking simulations. In all cases, we run the simulation for 1000 timesteps before recording any data to form the bumps. To achieve the relationship in Eq 13 for circular mapping, we rescale γ with network size N and bump number M:

$$\begin{array}{c}\hfill \gamma \to \gamma ·\frac{N}{600}·\frac{3}{M}.\end{array}$$ (126)

The connectivity W takes the form in Eq 38. Unless otherwise specified, we use shift ξ = 2. To produce M bumps in a network of size N, we turn to Eq 47 and set l = 0.44N/M. Note that an alternative to Eqs 13 and 126 would be to rescale ξ proportionally with l ∝ N/M under circular mapping, since bump velocity is also proportional to ξ (Fig 3C). We use w = 8M/N ≈ 3.5/l. For the case of 2l > N/2, which corresponds to a one-bump network, the tails of the cosine function extend beyond the network size. Instead of truncating them, we wrap them around the ring:

$$\begin{array}{c}\hfill W\left(x\right)\to W\left(x\right)+W(x-N)+W(x+N).\end{array}$$ (127)

This procedure, along with the scaling of w with N and M, accomplishes Eq 7 and keeps the total connectivity strength per neuron ∑_i W_i constant across all N and M, where W_i is the discrete form of W(x).

To generate the Poisson-like spike counts c_αi(t) in Eq 98, we rescale Poisson random variables:

$$\begin{array}{c}\hfill c_{\alpha i}\left(t\right)=F·C_{\alpha i}\left(t\right),C_{\alpha i}\left(t\right)\sim \text{Pois}\left[\varphi \right[g_{\alpha i}\left(t\right)]\Delta t/F].\end{array}$$ (128)

These counts will be multiples of the Fano factor F. To produce a c_αi(t) whose domain is the natural numbers, one can follow Ref [18], which takes multiple samples of C_αi(t) during each timestep.

To obtain theoretical values in Figs 3, 5, 7 and 8, we need to substitute the baseline inputs g_i into the appropriate equations. We use noiseless and driveless simulations to generate g_i instead of using Eq 4.

Bump position

We track the position θ of each bump using the firing rate summed across both populations S_i(t) = ∑_α ϕ[g_αi(t)]. We first compute the circular center of mass of S_i(t) with periodicity N/M:

$$\begin{array}{c}\hfill {\theta }_0=\frac{N}{2\pi M}\text{atan2}[\sum _i{S}_i\left(t\right)\text{sin}(2\pi iM/N),\sum _i{S}_i\left(t\right)\text{cos}(2\pi iM/N)].\end{array}$$ (129)

atan2 is the two-argument arctangent, and we choose its branch cut such that its range is [0, 2π). Thus, θ₀ lies between 0 and N/M and represents the bump position averaged periodically across bumps. To track the position of each bump independently, we then partition the network into segments of length ⌊N/M⌋. If N/M is not an integer, we skip one neuron between some segments to have them distributed as evenly as possible throughout the network. We perform a circular shift of S_i(t) such that network position θ₀ is shifted to the middle of the first segment N/2M, after rounding both quantities to integers. The purpose of this process is to approximately center each bump within a segment so that S_i(t) drops to 0 before reaching segment boundaries. We then calculate the center of mass of S_i(t) within each segment. After reversing the circular shift, these centers of masses are taken to be the bump positions.

Path integration velocity and diffusion

To obtain our results in Figs 3 and 5, we run each simulation for T = 5 s. To extract the bump velocity v produced by a constant drive b, we calculate the mean displacement Θ as a function of time offset u:

$$\begin{array}{c}\hfill \Theta \left(u\right)=\frac{\Delta t}{T-u}\sum _t\left[\theta \right(t+u)-\theta (t\left)\right].\end{array}$$ (130)

θ is the bump position. This equation averages over the fiducial starting time t, which ranges from 0 to T − u − Δt in increments of Δt. We vary u between 0 and T/2 in increments of Δt; the maximum is T/2 to ensure enough t’s for accurate averaging. We then fit Θ(u) to a line through the origin to obtain the velocity:

$$\begin{array}{c}\hfill \Theta \left(u\right)\approx vu.\end{array}$$ (131)

We calculate the diffusion coefficient D based on an ensemble of replicate simulations. In this section, angle brackets will indicate averaging over this ensemble. Following the definition of D in Eq 92, we calculate each bump’s position relative to the mean motion of the ensemble:

$$\begin{array}{c}\hfill \omega \left(t\right)=\theta \left(t\right)-⟨\theta \left(t\right)⟩.\end{array}$$ (132)

We compute squared displacements and then average over fiducial starting times to obtain a mean squared displacement for each bump as a function of time offset u:

$$\begin{array}{c}\hfill \Omega \left(u\right)=\frac{\Delta t}{T-u}\sum _t{\left[\omega \right(t+u)-\omega (t\left)\right]}^2.\end{array}$$ (133)

t and u span the same time ranges as they did for Θ. We average Ω(u) over the ensemble and fit it to a line through the origin to obtain the diffusion coefficient:

$$\begin{array}{c}\hfill ⟨\Omega \left(u\right)⟩\approx 2Du.\end{array}$$ (134)

For simulations with M bumps, we arbitrarily assign identity numbers 1, …, M to bumps in each simulation. We perform ensemble averaging over bumps with the same identity numbers; that is, we only average over one bump per simulation. This way, we obtain separate values for each bump in Fig 3E–3H; nevertheless, these values lie on top of each other. In Fig 3B and 3C, each point represents v averaged across bumps. To calculate the mean velocity 〈v〉 in Fig 3E and 3F, we fit 〈Θ(u)〉 to a line through the origin. To estimate standard deviations for Figs 3E–3H and 5, we create 48 bootstrapped ensembles, each of which contains 48 replicate simulations sampled with replacement from the original ensemble. We calculate 〈v〉 or D for each bootstrapped ensemble and record the resulting standard deviation. In Fig 5, each point represents D and its estimated standard deviation averaged across bumps.

Trapping and position-dependent velocity

For simulations with connectivity noise, we determine the escape drive b₀ (Fig 7), the smallest drive that allows the bumps to travel through the entire network, by a binary search over b. We perform 8 rounds of search between the limits 0 and 1.28 and another 8 rounds between 0 and −1.28 to obtain b₀ within an accuracy of 0.01. In each round, we run a simulation with the test b and see whether the bumps travel through the network or get trapped. Traveling through the network means that every position (rounded to the nearest integer) has been visited by a bump, and trapping means that the motion of at least one bump slows below a threshold for a length of time.

To obtain the position-dependent bump velocity v(θ) produced by connectivity noise when |b| > b₀, we run a simulation until the bumps have traveled through the network. At each timestep, we record the positions of the bumps (binned to the nearest integer) and their instantaneous velocities with respect to the previous timestep. We smooth the velocities in time with a Gaussian kernel of width 10 ms, which is the neural time constant τ. We compute the mean and standard deviation of these smoothed velocities for each position bin.

Mutual information

For simulations with input noise, we explore the mutual information between encoded coordinate and single-neuron activity (Fig 6). To do so, we must generate data from which we can compute p(s|u) in Eq 22, for coordinate $u\in \mathcal{U}$ and activity $s\in \mathcal{S}$. We have chosen one set of conditions for performing this analysis, which we detail below.

We first choose to represent either a linear or circular coordinate, which we take to be position or orientation, respectively. We then choose to represent a narrow or wide coordinate range u_max, which is 20 cm or 200 cm for position and 36°or 360°for orientation. We divide the range into 20 equally spaced coordinates such that $\mathcal{U}=\{u_{\text{max}}/20,\dots ,u_{\text{max}}\}$. We convert these coordinates to network positions according to the mappings in Fig 4. For each coordinate value u, we initialize 96 replicate simulations at the corresponding network position by applying additional synaptic input to the desired bump positions during bump formation. We run the simulations for 5 s, record the final firing rates, and bin them using 6 equally spaced bins from 0 to the 99th percentile across all neurons. All rates above the 99th percentile are also added to the 6th bin. These bins define the discrete $\mathcal{S}$, and normalizing the bin counts produces p(s|u). We marginalize over u to obtain p(s), and p(u) is uniform. We can then use Eq 22 to compute the mutual information.

The 4 local cues in Fig 6F–6H correspond to 4 activity states ${\mathcal{S}}_{\text{cue}}$ separate from the 6 activity bins of the CAN neurons, ${\mathcal{S}}_{\text{neuron}}$. The joint sample space of a single neuron with cues is thus $\mathcal{S}={\mathcal{S}}_{\text{neuron}}\times {\mathcal{S}}_{\text{cue}}$ with 6 × 4 = 24 total states. We bin neural activity across these more numerous states, using the coordinate value u to determine the cue state value, to again compute p(s|u) and then the mutual information.

We choose to compute mutual information with single-neuron activities binned into 6 discrete states due to computational tractability. A better indication of encoding quality for the entire network would involve using the joint activity of multiple neurons. However, assuming the same binning process, that would involve estimating probability distributions over 6ⁿ states for n neurons, which would require exponentially more replicate simulations per coordinate value than the 96 we use. Alternatively, one could reduce the dimensionality of the network activity by projecting it onto various attractor configurations, as done by Ref [79].

Supporting information

S1 Text. Contains text on different model parameters, additional information results, and splitting networks, as well as Figs A, B, and C.

(PDF)

Data Availability

Simulation and analysis code is available at https://github.com/louiskang-group/wang-2022.

Funding Statement

RW and LK were funded by RIKEN Center for Brain Science. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.