Stability of discrete memory states to stochastic fluctuations in neuronal systems

Abstract

Noise can degrade memories by causing transitions from one memory state to another. For any biological memory system to be useful, the time scale of such noise-induced transitions must be much longer than the required duration for memory retention. Using biophysically-realistic modeling, we consider two types of memory in the brain: short-term memories maintained by reverberating neuronal activity for a few seconds, and long-term memories maintained by a molecular switch for years. Both systems require persistence of (neuronal or molecular) activity self-sustained by an autocatalytic process and, we argue, that both have limited memory lifetimes because of significant fluctuations. We will first discuss a strongly recurrent cortical network model endowed with feedback loops, for short-term memory. Fluctuations are due to highly irregular spike firing, a salient characteristic of cortical neurons. Then, we will analyze a model for long-term memory, based on an autophosphorylation mechanism of calcium/calmodulin-dependent protein kinase II (CaMKII) molecules. There, fluctuations arise from the fact that there are only a small number of CaMKII molecules at each postsynaptic density (putative synaptic memory unit). Our results are twofold. First, we demonstrate analytically and computationally the exponential dependence of stability on the number of neurons in a self-excitatory network, and on the number of CaMKII proteins in a molecular switch. Second, for each of the two systems, we implement graded memory consisting of a group of bistable switches. For the neuronal network we report interesting ramping temporal dynamics as a result of sequentially switching an increasing number of discrete, bistable, units. The general observation of an exponential increase in memory stability with the system size leads to a trade-off between the robustness of memories (which increases with the size of each bistable unit) and the total amount of information storage (which decreases with increasing unit size), which may be optimized in the brain through biological evolution.

Noise is present on many scales within the cerebral cortex.^1–4 A major source of noise is the stochastic molecular and ionic motion, which is ubiquitous in the physical world, but averages out to produce deterministic behavior in many life-scale physical and chemical systems. In the brain, such thermal noise of microscopic motion is amplified to larger scales by a number of discrete transitions. Local fluctuations in calcium concentration can lead to discrete changes in the number of ions bound to a protein⁵ and discrete changes in its conformational or activation state.⁶ Such discrete, stochastic changes in the states of proteins can cause fluctuations in channel conductances,⁷ or can determine the release of a vesicle containing thousands of molecules of neurotransmitters,⁶ or can lead to the switching of state of a set of proteins maintaining synaptic strength.⁸ Fluctuations of transmitter release and of ionic conductances cause fluctuations of inputs (ion currents) to and outputs (the timing of all-or-none action potentials) from neurons.^9,10 Random variations in synaptic strength between neurons, and fluctuations in times and numbers of action potentials, can be amplified in a network of neurons to produce significantly different patterns of neuronal activity.^9,11 Since it is the activity of networks of neurons that produce our behavior, it appears that our brains are able to amplify the randomness inherent in molecular motion to produce nondeterministic macroscopic action. Hence, noise amplified within our brain allows different behavioral outcomes given identical external circumstances. If an animal’s pattern of behavior is completely predictable, at best a competitor can use the information to gain some advantage in a social setting (a mixed strategy, i.e., randomness is the optimal strategy in a rock-paper-scissors game¹²), while at worst the animal becomes an easy prey. This suggests an evolutionary advantage to nondeterministic behavior.

I. INTRODUCTION

Noise in the brain can also arise from chaotic, deterministic trajectories without the need for external or microscopic fluctuations. Such chaotic randomness arises when neurons are coupled sparsely and receive similar amounts of (regular) excitation and inhibition^13–16 (i.e., they receive balanced inputs). In a chaotic network small differences in initial conditions produce divergent trajectories of neuronal activity, allowing for variations in behavior. Since the steady stream of external inputs and ongoing internal activity ensure a brain is never in exactly the same state twice, one would never expect an exact repetition of a pattern of neuronal activity, even under identical (repeated) external conditions. A chaotic trajectory implies the lack of an exact repetition and can remove any correlation between a neuron’s behavior on one trial and another. An analysis of changes in connection strengths reveals that a chaotic trajectory for learning in a game of matching pennies results in a failure to achieve the optimal, purely random Nash equilibrium.¹⁷ It is presently unknown whether deterministic chaos plays a major role in stochastic neural dynamics in the brain.

In this paper we focus on two particular situations in the brain where noise can be amplified, by causing a random transition between two discrete states of a memory system. Senn and Fusi have shown that a stochastic method for imprinting memories leads to near optimal storage¹⁸ of new memories without erasing old ones. However, for information storage, random fluctuations are detrimental as they degrade the memory. Whether or not memory systems in the brain require stochastic fluctuations, if the size of system devoted to a bit of information storage is too small, then inherent fluctuations can cause a switch from one memory state to another.^19,20,8 This is equivalent to the stability problem arising from small numbers of molecules in genetic switches that have been extensively studied.^21–26 Hence, given a biological mechanism to maintain a particular kind of memory, a tradeoff exists between reducing the resources devoted to each bit of information, allowing for greater information storage in total, and the rate of information loss that increases with reduced system size.

In Sec. II, we consider the stability of discrete stable states of action potential firing in networks of neurons that can maintain short-term memories. The autocatalytic component that maintains the memory in a group of neurons is the reverberating excitation^27–31 that one neuron provides to others in the group. The times of the action potentials providing recurrent excitation fluctuate with approximately Poisson statistics, so the self-excitation contains a considerable amount of noise. In real systems, significant noise correlation between neurons makes it impossible to average out noise with an increasing network size. Without enforcing realistic levels of noise correlation, we investigate how the number of neurons, N, in the group reduces the effect of noise in the reverberatory excitation. Brunel^32,33 has also considered the loss of stability of states in neuronal networks arising from noise that scales inversely with N (a finite-size effect, as in our case). He found the destabilization of an asynchronous, low firing rate state by spontaneous oscillation in a mixed excitatory-inhibitory network,³³ and the loss of temporal coherence of oscillations in a purely inhibitory network.³²

In Sec. III, we consider the stability of a molecular switch, based on the phosphorylation of calcium-dependent protein kinase II (CaMKII). Phosphorylation of CaMKII in the postsynaptic density has been proposed to play a crucial role in long-term memory.³⁴ Since each postsynaptic density contains in the range of 10–30 CaMKII holoenzymes, a key question is for how long can such a small group maintain their persistent phosphorylation through autocatalytic activity.

In both Secs. II and III we discuss how the systems can be analyzed as a transition between two stable states across a barrier using a one-dimensional effective potential. In the CaMKII system, the noise is intrinsic and set by the inevitable stochastic nature of chemical reactions, whereas in the neuronal network, we add noise to neurons, to ensure they fire spikes in the observed, nearly Poisson manner. In each section we consider how multiple discrete states can lead to graded memory. In the neuronal network, we couple the discrete states, since groups of similarly behaving neurons are often connected together, but we assume the protein switches are uncoupled, since they exist on separate spines of neuronal dendrites and each spine is to a first approximation an isolated chemical unit. Such a difference in connectivities, leads to M states for a connected chain of M neuronal groups, where the coupling can reduce the rate of information loss, but up to 2^M states for M uncoupled synapses.

II. BISTABILITY IN NETWORKS OF NEURONS

Working memory, with which we hold information in mind for a few seconds in order to act on it or make a decision, depends on persistent neuronal activity.^35,29 When recording from the prefrontal cortex of monkeys engaged in a working memory task, on some trials the persistent activity switches off during a delay³⁶ between stimulus and response. Such transitions from an ON state of high firing rate to an OFF state of low firing rate correlate with errors in performance of the task, suggesting they are the neuronal correlates of a spontaneous loss of short-term memory.

Here we discuss methods for analyzing the average lifetime of discrete memory states generated by recurrent excitation within groups of neurons. We consider the case where neurons receive enough external noise that they fire spikes in a Poisson manner. Poisson spike trains are typically observed in vivo in the cortex,¹ but are difficult to generate in computational models at high firing rates. The particular difficulty arises when a spontaneous state with low firing rate is required to coexist with a second state with high firing rate; a bistable regime. The low firing rate state (the DOWN state) can be achieved in the balanced regime, where on average, inhibitory currents cancel excitatory currents, so that action potentials are driven purely by noise. However, a second state with higher firing rate (the UP state) requires extra excitatory drive to each cell. The second state is stable if the extra excitation needed to maintain the higher firing rate is produced by feedback current to the neurons through recurrent connections.^37,38 Given sufficient recurrent input, the neuron is excited above its deterministic threshold. The deterministic threshold is the point at which the average current causes the steady-state membrane potential to cross the threshold for spike generation, so the neuron would fire spikes in the absence of noise. Once above threshold, the neuron tends to integrate current, firing in a more regular manner. Such regular firing is not seen in vivo in the cerebral cortex.

Renart has suggested a method for obtaining bistability in the balanced regime, producing irregular spike trains³⁹ in both DOWN and UP states. The key is that noise, rather than the mean excitatory drive, is stronger in the UP state of high firing rate. This method can result in a greater coefficient of variation (CV) of interspike intervals in the UP state than the DOWN state, as has been observed in some data.³ However, in this paper, we produce irregular spiking by providing each neuron with independent, relatively high-conductance, excitatory, and inhibitory Poisson inputs that are assumed to arise from cells outside the local network. The noise is so strong that even the UP state (of 40 Hz) is below the deterministic threshold, so the CV is relatively high (0.9 in the absence of rate fluctuations).

The background noise provided to each neuron has several important effects. First, it smooths out the firing rate curve of each neuron as a function of total excitatory input,⁴⁰ allowing the neuron to fire when the average membrane potential is below the deterministic threshold. In a recurrent network, this allows for a DOWN state that is not silent, in fact with a firing rate as high as 10 Hz in our case.

Second, the barrage of synaptic inputs has a net effect of increasing the conductance across the membrane of a neuron. This reduces the time constant of integration for the membrane potential allowing for more rapid fluctuations, but also prevents integration of small, slow current inputs, which quickly leak away. Such a reduction in response to small changes in steady state current manifests itself as a reduction in gain⁴¹ (i.e., a decreased slope of the firing rate curve).

Finally, since the noise is independent of each neuron, it prevents synchronization, allowing for asynchronous states of activity. Asynchrony is important in stabilizing persistent activity, as independent fluctuations in synaptic current from different presynaptic neurons average out, decreasing the total variance by the number of neurons. In contrast, correlated fluctuations in rate, which are strongest in a synchronized network, destroy the stability of a persistent state.^42,20 After all neurons have fired in a synchronous network, there is a silent interval, which, especially in the absence of NMDA receptors, causes synaptic currents to die away. Asynchronous firing is also important to allow a group of neurons to encode and transmit an analog variable in their firing rate,⁴³ whereas synchrony implies an all-or-none response in the network.

A. Simulations of neuronal activity

We simulated networks of leaky integrate and fire neurons⁴⁴ as one homogeneous population with strong recurrent excitation. Our goal was to produce a network readily amenable to analysis, so we did not include synaptic depression or facilitation which introduce correlations between times of vesicle release. We also wanted to generate an UP state that had a rate under 50 Hz and a CV close to 1. The specific model parameters are given in the Supporting Information, and we summarize the key points here.

Each neuron receives a barrage of independent excitatory and inhibitory inputs through synapses with fast time constants (5 ms for inhibition mediated by GABA receptors and 2 ms for excitation via AMPA receptors). These external inputs cause the membrane of the neurons to fluctuate rapidly and independently. Recurrent excitatory inputs from within the network are mediated by slower NMDA receptors (with 100 ms time constant). Except for the networks with probabilistic synapses, or sparse connectivity, the recurrent excitatory current is identical for all neurons in the population. The membrane potential changes by integrating the synaptic currents, filtered through its time constant (of 10 ms). When the membrane potential reaches its threshold, a spike is emitted, causing a postsynaptic current in neurons to which it is connected.

To aid the analysis, we first simulated a single neuron by adding constant excitatory input conductance, instead of including any input from other neurons within the network (noisy background inputs are included). We obtain the firing rate of the neuron [Fig. 1(a)], and the coefficient of variation (CV) of interspike intervals (ISIs) [Fig. 1(b)] as a function of fixed input conductance (g_ES_in is fixed for each data point). We chose this empirical method, since analytic expressions for rate and CV are not simple functions and are only available to lowest order in synaptic time constants.⁴⁵ We also obtain the effect of such a spike train on a postsynaptic neuron, measuring both the average [Fig. 1(c)] and variance [Fig. 1(d)] in the synaptic variable, s [given by Eq. (1)] that such a set of spikes would provide through NMDA channels.

FIG. 1.

Transduction function for a single neuron. (a) Firing rate as a function of synaptic input, held constant, with standard background noise. Solid black curve: simulated data; dashed gray curve: fitted function used for analysis. (b) Black, upper curve is CV, gray, lower curve is CV2 for the neuron in (a). Spike train is Poisson at low rates, but more regular at higher rates. The rate of the UP state in our simulations is approximately 40 Hz, where the CV is approximately 0.9, and CV2 is approximately 0.8. (c) Average synaptic output increases with firing rate, but saturates. Synaptic output of a neuron is proportional to the synaptic input provided to its postsynaptic partner. Solid black curve: simulated data; dashed gray curve: fitted function used for analysis. Dotted (red) straight line is S_in = Ws_out representing feedback and intersects the curve at three points, which are self-consistent, steady-state solutions. (d) Variance of the synaptic output. Solid black lines are simulated data with P_R = 1 and α̃ = 0.25 (lower curve) and P_R = 0.5 and α̃ = 0.5 (upper curve). Note the average synaptic output is identical for the two sets of parameters. Dashed gray curves are the corresponding analytic results assuming Poisson spike trains.

We assume synaptic transmission through NMDA receptors which have a slow time constant of 100 ms. The slow time constant helps to prevent network synchrony that could destabilize memory activity.^46,47 Coupling to an inhibitory population with a faster time constant can have a similar stabilizing effect.^48,49 The NMDA receptors also saturate, which limits the feedback and allows for a stable, high firing state⁴⁶ instead of runaway excitation that would otherwise occur in our network without depressing synapses.

In a network with a relatively small number of neurons [10 in Fig. 2(a)] the average firing rate of neurons in the network fluctuates significantly, but distinct periods of low or high activity are visible. Increasing the number of neurons demarcates the two stable states [Figs. 2(b) and 3(c)] and increases the lifetime of each state—the time in a state before a spontaneous fluctuation causes a transition. If we measure the average CV of neurons in both the UP and DOWN states, we find the fluctuations in average rate have increased the variance in ISIs. For example, in the network with 100 neurons the CV is 0.97 in the UP state [0.9 is the expected value for a constant mean rate of 40 Hz from Fig. 1(b)] while the CV is 1.07 in the DOWN state (compared to the expected value of 1, at 10 Hz). The smaller network of 10 neurons with greater fluctuations has a CV of 1.28 in the DOWN state and 1.02 in the UP state. Since the rate fluctuations are slow compared to the ISI, we calculate the average CV2 (Ref. 50) across the spike train. CV2 is calculated from consecutive ISIs, (ISI_i, ISI_i+1) as the absolute difference divided by the mean, CV2_i = 2|ISI_i − ISI_i+1|/(ISI_i + ISI_i+1). If consecutive ISIs are random samples of the complete set of ISIs, then CV2 should equal the CV. Any positive correlation between one ISI and the next one reduces CV2, since consecutive ISIs would be more likely to be similar than randomly selected ISIs. We find the average CV2 matches that of a constant firing rate [1 in the DOWN state, 0.8 in the UP state, Fig. 1(b)] and varies little with network size. CV2 is lower than the CV of a neuron with constant firing rate, because even the small time constants of 2 ms and 5 ms for the excitatory and inhibitory external noise produce some correlation between one ISI and the next one. The difference between CV and CV2 increases with rate, as the synaptic time constants become a larger fraction of the average ISI.

FIG. 2.

Stochastic fluctuations in a bistable network of neurons. (a) Network of ten neurons. Top: series of ten spike trains for the 14 s interval between dashed lines. Bottom: population-average firing rate. The variance in firing rate is large, blurring the distinction between a low and high activity state. Random transitions between the two states occur on a time scale of seconds. (b) Network of 100 neurons. Top: Series of 100 spike trains for the 30 s interval between dashed lines. Bottom: Population-average firing rate. With reduced variance, the two states are more distinct. Stability of each state is on the order of tens of seconds.

FIG. 3.

Analytical method for calculating transitions times. (a) Mean field theory gives ds/dt as a function of s for the network with feedback, calculated from the difference between curves and dotted line in Fig. 1(c). Zero crossings are fixed points of the system, and those with negative gradient are stable. Solid line is calculated from the analytic fit to the neurons transduction function; circles are calculated from the numerical data points. (b) Effective potentials generated from the analytic form of ds/dt and the variance of s, for a bistable population of ten neurons (dashed) and 100 neurons (solid). (c) Probability density function for the system of ten neurons (broader and darker curves) and of 100 neurons (more peaked, blue/lighter curves). Dashed lines result from analytic calculation; solid curves are data from numerical simulations. (d) Transition times increase exponentially with system size at large N. Dashed lines are analytic results; open symbols are data from simulations. Upper, darker dashed line and circles are duration of the DOWN state; lower, blue/lighter dashed line and squares are duration of the UP state.

B. Analysis of Poisson spiking with saturating synapses

In order to begin our analysis of the bistable network, we first consider how a Poisson series of spike times, with average rate r, in a presynaptic neuron affects the postsynaptic neuron. Our analysis covers the case of probabilistic release of vesicles (with release probability, P_R), since if a spike train is Poisson, at rate r₁ and if release of vesicles is uncorrelated between spikes, the vesicular release times also follow Poisson statistics with an altered rate, r₂ = P_Rr₁. We ignore here the effects of depression, which reduces CV, and facilitation, which enhances CV.

Each vesicle release causes a fraction of the postsynaptic channels to open, changing the conductance of the postsynaptic neuron. If the synaptic time constant τ_s is relatively slow, so that saturation is important, the synaptic gating variable s (denoting the fraction of open channels) follows

$$\frac{ds}{dt}=-\frac{s}{{\tau }_s}+\alpha (1-s)\sum \delta (t-t_{\text{release}}),$$ (1)

where α is a saturation factor. Writing α̃ = 1 − e^−α the step change in the gating variable following release of a vesicle is

$$\mathrm{\Delta }s=s^{+}-s^{-}=\stackrel{\sim }{\alpha }(1-s^{-}),$$ (2)

where s⁻ and s⁺ are the values of s immediately before and after the time of vesicle release t_release, respectively.

Moments of s can be calculated for a Poisson time series {t_release} at rate r to give (cf. Brunel and Wang⁵¹ with τ_rise = 0)

$$\langle s\rangle =\frac{\stackrel{\sim }{\alpha }r{\tau }_s}{1+\stackrel{\sim }{\alpha }r{\tau }_s},$$ (3)

$$\text{Var}(s)=\frac{{\stackrel{\sim }{\alpha }}^{2}r{\tau }_s}{{(1+\stackrel{\sim }{\alpha }r{\tau }_s)}^2(2+2\stackrel{\sim }{\alpha }r{\tau }_s-{\stackrel{\sim }{\alpha }}^{2}r{\tau }_s)}=\frac{\stackrel{\sim }{\alpha }\langle s\rangle {(1-\langle s\rangle )}^2}{2-\stackrel{\sim }{\alpha }\langle s\rangle }.$$ (4)

The assumption of Poisson spike trains is reasonable, since the coefficient of variation (CV) of interspike intervals (ISIs) is approximately one, like that of a Poisson process, for spike trains in the cortex in vivo¹ and the distribution of ISIs is nearly exponential as expected for a Poisson⁵² process. In the computer simulations, we cause the neurons to fire with a high CV, close to one, even at high rates, by including strong, independent, excitatory, and inhibitory noise inputs to all neurons. These inputs represent activity of cortical neurons that are not included in the local circuit that we simulate. Figure 1(d) shows the variance in s as a function firing rate for the simulated neuron, and the analytic result for a Poisson spike train [Eq. (4)]. At larger rates the increasing regularity causes the variance of the simulated neuron to fall more rapidly than the analytic result.

In order to model the effects of noise on the system, we treat the synaptic gating variable, s, as an Ornstein-Uhlenbeck (OU) process with mean and variance depending on the presynaptic rate through Eqs. (3) and (4). Hence we can match the steady “force” term, A(s) and the noise term, D⁰(s),⁵³ for an OU process with the same mean dynamics and same variance to give

$$A(s)=\frac{-s}{\tau }+\stackrel{\sim }{\alpha }(1-s)r=-\left(\frac{1}{\tau }+\stackrel{\sim }{\alpha }r\right)(s-\langle s\rangle ),$$ (5)

$$D^0(s)=\frac{{\stackrel{\sim }{\alpha }}^{2}r}{(1+\stackrel{\sim }{\alpha }r\tau )(1+\stackrel{\sim }{\alpha }r\tau -{\stackrel{\sim }{\alpha }}^{2}r\tau /2)}.$$ (6)

Note how A(s) for a Poisson process is simply related to Eq. (1) with the series of δ functions for the incoming spike train replaced by the average rate, r.

C. All-to-all connectivity and firing rate model

Given the input-output characteristics for an individual neuron, we can now study the behavior of a population of such neurons coupled in a network. We will show how the mean synaptic variable across the network (s̄), describing the amount of recurrent excitation, can be used as a state variable, whose probability distribution will lead to the prediction of transition times between metastable states.

If all neurons within a population have the same recurrent connections and if synapses are not probabilistic, then the synaptic conductance arising from activity in the network is identical for all neurons in the population. That is to say,

$$S_{\text{in},i}(t)=\sum _j{W}_{j\mapsto i}{s}_{\text{out},j}(t)$$ (7)

is independent of i for all neurons in a population with the same set of connection strengths, W_j↦i. The firing rates r_i(t) are a function, f[S_i(t)] of S_i(t). Based on the analytic function of Abbott and Chance,⁵⁴ we fit the function

$$f(S)=\frac{{\beta }_1(S-{\beta }_2)}{1-exp[-{\beta }_3(S-{\beta }_2)]}$$ (8)

to the simulated results. Since the spiking is not exactly Poisson at higher rates, the mean synaptic gating, 〈s〉 does not follow Eq. (3) precisely—more regular spiking leads to higher 〈s〉 for the same rate. So we fit β₁, β₂, and β₃ to match the analytic expression s_out = α̃τf(S_in)/[1 + α̃τf(S_in)] in Fig. 1(c) (gray, dashed curve). The fitted function f(S_in) also reproduces the firing rate curve well [Fig. 1(a) gray, dashed curve].

In the case of a single population with all-to-all connectivity, we have one connection strength, W = W_i↦i so that S_in = NWs̄, where s̄ is the average synaptic gating variable, s̄ = Σ_is_i/N which follows

$$\frac{d\overline{s}}{dt}=-\frac{\overline{s}}{{\tau }_s}+\stackrel{\sim }{\alpha }f[NW\overline{s}(t)](1-\overline{s}),$$ (9)

where N is the number of neurons in the population. We scale synaptic weights with N such that NW = W₀ is constant. Using Eqs. (8) and (9) we can evaluate ds̄/dt as a function of s̄ and evaluate the fixed points where ds̄/dt = 0 [Fig. 3(a)]. We note that our system is designed so that the final term of Eq. (9) is nearly linear in s̄ [see Fig. 1(c)] between the DOWN and UP states resulting in very small values for ds̄/dt. A neuronal input-output function with greater curvature could result in much greater values (in the range of s̄/τ) for ds̄/dt. The following paragraphs will show that our system with minimal ds̄/dt is one of minimal stability, so our results can be considered as approaching the lower bound of network stability.

The stable (and unstable) fixed points of s̄ are the approximate positions of the minima (and maximum) of an effective potential [Fig. 3(b), Ref. 53] given by

$$\mathrm{\Phi }(\overline{s})=-2{\int }_0^{\overline{s}}\frac{A({\overline{s}}^{\prime })}{D({\overline{s}}^{\prime })}d{\overline{s}}^{\prime }=-2N{\int }_0^{\overline{s}}\frac{A({\overline{s}}^{\prime })}{D^0({\overline{s}}^{\prime })}d{\overline{s}}^{\prime },$$ (10)

where the noise strength, D(s̄) = D⁰(s̄)/N scales as 1/N because s̄ is the average of N independent synapses.

The effective potential, Φ(s̄), describes the probability distribution, P(s̄), of the noisy process through

$$P(\overline{s})=\frac{2}{KD(\overline{s})}exp[-\mathrm{\Phi }(\overline{s})],$$ (11)

where K is a normalization constant. The probability distribution is peaked near the minima of the potential [see Fig. 3(c)] with the peaks sharpening as a function of N because of the reduction in effective noise. The mean transition time is then given between any two values, s₁ and s₂ as ⁵³

$$T(s_1,s_2)=\frac{K}{2}{\int }_{s_1}^{s_2}dsexp[\mathrm{\Phi }(s)]{\int }_0^{s}P(s^{\prime }){ds}^{\prime }.$$ (12)

We calculate T(s₁, s₂) where s₁ is in the vicinity of the first potential minimum and s₂ is the second potential minimum, so the transition from s₁ to s₂ and vice versa corresponds to a change of state.

In the limit of large N, the probability distribution is nonzero only near the potential minima, whereas the “barrier function” proportional to exp[Φ(s)] is only nonzero near the potential maximum. In this case the two integrals above can be separated as

$$\begin{array}{l}T(s_1,s_2)\approx {\int }_{s^{\ast }-\mathrm{\Delta }s}^{s^{\ast }+\mathrm{\Delta }s}\mathit{ds}exp[\mathrm{\Phi }(s)]·2{\int }_{s_0-\mathrm{\Delta }s}^{s_0+\mathrm{\Delta }s}\frac{exp[-\mathrm{\Phi }(s)]}{D(s^{\prime })}{ds}^{\prime }\\ \propto exp[kN],\end{array}$$ (13)

where $k={\int }_{s_0}^{s^{\ast }}2A(s)/D^0(s)ds$ is the height of the effective potential barrier between the stable fixed point, s₀, of the starting state and the unstable fixed point, s^*, for a single synapse. Notably, the average time between transitions increases exponentially with N so appears linear in Fig. 3(d) at large N.

D. Probabilistic synapses

Koulakov²⁰ made a similar estimate of the stability of a self-excited group of neurons. He assumed the predominant noise arose from probabilistic vesicular release in the synapses.⁵⁵ If the probability of release is low, even if the presynaptic neuron fires a regular train of action potentials, the current inputs to the postsynaptic neuron are close to a Poisson process. However, if the spike train is already a Poisson process, adding probabilistic vesicle release does not change the CV, it simply converts the postsynaptic Poisson current barrage to one of a lower rate.

His formula for the lifetime of the UP state yields T_UP ≈ τ exp(kP_RN²), where k is a network-dependent parameter and P_R is the vesicular release probability. The formula for a network with all-to-all connectivity yields a quadratic rather than linear curve on a log-plot of lifetime as a function of N. The different dependence on N arises because the source of noise is within each synapse, as neurons fire with a low CV, and the number of synapses scales as N².

We tested the effect of probabilistic release on our network, by comparing a network with α̃ = 0.5 and P_R = 0.5 with our control having α̃ = 0.25 and P_R = 1 (i.e., deterministic release). We use P_R = 0.5 for illustrative purposes, whereas P_R = 0.2–0.3 would be more realistic. These two networks have exactly the same mean behavior for the synaptic variable, but the noise term is greater (approximately twice as large) for the probabilistic network (since each synapse has fewer kicks, but of larger amplitude). Hence one expects the same form for stability as a function of time approaching exp(kP_RN) at large N. This is verified in Fig. 3(d) where the probabilistic synapses yield a similar result to the deterministic synapses, but with a shift to lower stability from the greater noise [Fig. 1(d)]. Our result differs from Koulakov’s because the predominant noise is from afferent feedforward connections, rather than the probabilistic synapses.

One might think a sparse network with probabilistic connections would yield the same behavior as a fully connected network with probabilistic synapses if the probabilities are the same. For illustrative purposes, we compared a sparse network where, like the probabilistic network described above, each presynaptic spike only has a 50% chance of eliciting a current in any specified postsynaptic neuron (with α̃ = 0.5). In the sparse network, the mean behavior is significantly different. Since the number of available synapses per neuron is smaller, they saturate more rapidly. For example, if twenty presynaptic neurons release ten vesicles to all of the ten available synapses (sparse network) there is significant saturation of postsynaptic receptors, compared to the release of ten vesicles across twenty synapses (probabilistic release).

It is worth noting that all moments of the synaptic variable, s depends on the rate through the combination rτ. Hence a network with Poisson spiking and probabilistic release gives rise to a synaptic variable with the same set of moments as a network where the synapses have a shorter time constant, τ. However, in the case of smaller τ, the time scale of fluctuations in s is shorter, and rapid fluctuations in s produce greater fluctuations in the membrane potential (which essentially filters conductance fluctuations through the membrane time constant). So we find the lifetime of network states with α̃ = 0.5, P_R = 1, and τ = 50 ms are shorter than the network with probabilistic release (α̃ = 0.5, P_R = 0.5, and τ= 100 ms) but the dependence on N is similar.

E. The discrete integrator

Continuous neuronal integrators^56,57 or equivalently, continuous memory elements^58,59 have two shortcomings. First, they require fine tuning of parameters⁵⁷ that are not robust to parameter variability. Second, they integrate noise in the manner of a random walk^58,60,61 so memory is not robust. Koulakov and co-workers⁶² proposed a solution to both problems, by designing a discrete integrator from a set of bistable elements (an element is a small group of neurons). Strong excitation within a group of neurons is combined with weak excitation between all groups and a range of excitabilities across groups, so that as one group switches ON, the extra excitation primes the next most excitable group to be switched ON. We achieve a similar integrator, by including many populations with a similar, high threshold. Each population only receives excitation from one neighboring population [Fig. 4(a)] so again a domino effect is created, as one population switches on it primes its neighbor.

FIG. 4.

The discrete integrator. (a) Network architecture: Recurrent excitation within a population coupled with excitation to a neighboring population. (b) The 1D representation of a discrete integrator is a series of potential wells, known as a washboard potential. Top: Series of stable states with no applied stimulus. Center: A small applied stimulus lowers the barrier for transitions in one direction. Noise causes transitions between states preferentially in the direction of the lower barrier to produce slow, stochastic integration. Bottom: A large applied stimulus lowers the barrier further, so states are no longer stable and integration time is dominated by deterministic processes. (c)–(e) Simulation results for a series of 12 bistable groups, each of N neurons. (c) Three trials with a low stimulus show slow stochastic integration (N = 400, I_app = 0.1). (d) Trial-averaged integration for a low stimulus (I = 0.1) for N = 40 (left, green), N = 160 (middle, red) and N = 400 (right, black). (e) Trial-averaged integration for a large stimulus (I = 0.3) same set of N as (d). Note the greater dependence on N in (d) and the quasilinear nature of the integration in (d) and (e).

The output of the integrator is given by the number of groups switched ON, which once changed can be maintained without drift if the barrier in the effective potential is high enough [Fig. 4(b)]. In the limit of low noise, such an integrator has a hard threshold, and only changes activity when sufficiently strong input is applied to tilt the effective potential to remove any barrier [Fig. 4(b), lower curve]. However, if the noise is significant, an intermediate regime appears [Fig. 4(b), intermediate curve] where a small applied current causes stochastic transitions between states to be strongly biased in one direction [Fig. 4(c)]. This regime of noise-induced integration can have a time scale much slower than any time constant in the system⁶³ because it is determined by the slow transitions across a barrier [Eq. (12)]. The rate of integration depends strongly on the noise level and hence the number of neurons in the population [Fig. 4(d)]. As the input strength increases toward the level of deterministic integration, the effect of noise (and hence system size) diminishes [Fig. 4(e)]. Temporal integration in a similar system (but with bistability in individual neurons) has been suggested to underly linear ramping of neuronal activity during delays.⁶⁴

Okamoto and Fukai^63,65 (see also Ref. 66) have shown that a similar system with no range of thresholds and connections between all bistable units can reproduce Weber’s Law,^67,68 typical of psychophysical data in estimation of time by humans and animals—the standard deviation across trials is proportional to the mean estimated time.⁶⁹ Weber’s Law can arise from barrier hopping, as the distribution of times for hopping over each barrier is exponential, with standard deviation equal to the mean time. We find Weber’s Law to hold in our system with ten switches, where we produce a range of integration times by varying the input current to the system. An applied current reduces the effective height of each barrier by the same amount, −ΔU [Fig. 4(b)]. Hence the time constants and standard deviation for hopping over all barriers are multiplied by a constant factor, exp(−ΔU). For a system of N_S identical switches coupled successively, the mean time for them to all switch on is 〈T〉 = τN_S where τ is the individual mean switching time, while the standard deviation is ${\sigma }_T=\tau \sqrt{N_S}$. The ratio, ${\sigma }_T/\langle T\rangle =1/\sqrt{N_S}$ is independent of τ. We find approximately this ratio (0.35) with 10 switches. Importantly, the ratio can be reduced (and accuracy improved) by increasing the number of switches.

III. STABILITY OF A PROTEIN KINASE-PHOSPHATASE SWITCH

The theme of building memories from multiple bistable units is also important for long-term memory. In this case we require the memory system to be stable over many years, up to a lifetime. Protein synthesis is undoubtedly necessary for maintenance of long-term memories,⁷⁰ so a full stability analysis would require a model of transcription as well as the protein reaction pathways. Here, we just consider a model of bistability of the protein states, where the role of transcription is simply to replace proteins lost by turnover.

Long-term memory is believed to be encoded by changes in the strengths of synapses.⁷¹ In particular, a temporary rise in the level of calcium concentration in the dendritic spine of a postsynaptic neuron can lead to a persistent increase in synaptic strength, known as long-term potentiation (LTP).^72,73 Changes in the strength of individual synapses are discrete^74,75 suggesting the presence of one or more binary switches within each synapse. Strong evidence suggests, at least in the hippocampus, that calcium/calmodulin-dependent protein kinase II (CaMKII) must be activated (that is, phosphorylated) for induction of LTP^76,77 and that after LTP it remains in a highly phosphorylated state.⁷⁸ CaMKII is found mostly in the postsynaptic density (PSD) of dendritic spines,⁷⁹ is activated by calcium,⁸⁰ has the autocatalytic properties necessary for bistability,^81–83 and can phosphorylate the excitatory glutamatergic AMPA receptors.⁷⁹ Hence, the protein is the ideal candidate to be a memory molecule, whose activation determines the strength of a synapse.³⁴ However, at present, persistent activity of CaMKII has not been shown to be essential for the maintenance of long-term memory and bistability in its phosphorylation levels has not been proven in biological conditions.

Lisman and Zhabotinsky^84,85 have formulated a realistic set of biophysical reactions, based on known proteins within the PSD and their rates of reaction. Switchlike behavior is achieved through a combination of autophosphorylation of CaMKII combined with saturating dephosphorylation by the phosphatase, PP1.

The number of CaMKII holoenzymes within the PSD is small, typically in the range of 5–30.⁸⁶ When the number of reactants is so small, the deterministic Law of Mass Action, used for test-tube chemistry, is no longer appropriate.⁸⁷ The concentrations of reactants and products can only vary discretely, and changes occur probabilistically, at discrete moments in time, due to individual reaction events, rather than continuously. Such temporal discreteness adds an inherent shot noise to the system of reactions, which could destroy the stability of a macroscopically stable state.

The stability is further weakened by protein turnover, which occurs randomly with an average lifetime of 30 h for each holoenzyme.⁸⁸ Crick first pointed out⁸⁹ that for a group of proteins to play a role in long-term memory storage, they must have a mechanism for communicating their state to incoming, naive proteins. Such a mechanism would allow the switch to retain its state, even when all of its original constituent proteins are gone. In the reaction scheme of Lisman and Zhabotinsky, it is the level of phosphatase activity that communicates the state of the switch to incoming naive CaMKII holoenzymes. If other holoenzymes are highly phosphorylated, PP1 spends a significant amount of time bound to them and is less available to dephosphorylate the newly inserted kinase.

In an earlier work,⁸ we investigated the stability of the two phosphorylation states (termed UP for high phosphorylation and DOWN for low phosphorylation) of small numbers of CaMKII holoenzymes using numerical methods. We simulated the set of reactions in the CaMKII-PP1 system (Fig. 5) stochastically, using Gillespie’s algorithm.⁹⁰ Spontaneous transitions between states occur in a Poisson-type manner (so the distribution of lifetimes is approximately exponential, with a standard deviation equal to the average lifetime). We estimated the average stability of the UP and DOWN states [Figs. 6(a) and 6(b)] from the time for 400 or more transitions from each state giving 5% (i.e., $1/\sqrt{400}$) or better accuracy. We found a sensitive dependence of stability on parameters and in general an exponential increase of stability with system size as shown in Fig. 6(c). In optimal conditions, a system of 15–20 holoenzymes could persist stably for many decades. Here, we discuss a more analytical approach, which while approximate, permits calculations of average lifetimes in a few seconds on one computer processor. By comparison, the stochastic simulation can take many days on ten or more processors due to the large time simulated (tens to hundreds of years) while many reactions can occur each second.

FIG. 5.

Configurations and reactions of the CaMKII-PP1 switch. (a) The different configurations of phosphorylated subunits of a six-unit ring (half of the holoenzyme) of CaMKII. Filled circles represent phosphorylated subunits. The ring has rotational symmetry but not reflectional symmetry (so states P3b and P3c differ). (b) Phosphorylation is more rapid when catalyzed by a neighboring subunit, so only one Ca²⁺/calmodulin needs to bind rather than two. The fractional activation of calmodulin by calcium, f(Ca) is very low, approximately 1 / 1000 at basal levels of Ca²⁺. (c) Dephosphorylation by PP1 through the Michaelis-Menten scheme. PP1 activity saturates when the majority of molecules of PP1 are in the intermediate state, bound to CaMKII.

FIG. 6.

Stochastic transitions in phosphorylation of CaMKII. (a) Random transitions occur on a time scale on the order of ten years with 12 CaMKII holoenzymes and six PP1 molecules. (b) Log of probability distribution for the total phosphorylation. Black, lower curve for 16 holoenzymes; lighter (red) upper curve for four holoenzymes. Dashed curves are analytic results. Inset: Probability distribution from simulations is bimodal, indicating two distinct states. (c) Average lifetimes of the UP state (plus, red/lighter) and the DOWN state (circles, black) increase exponentially with the number of reacting species. Dashed lines are linear fits, denoting exponential rise in the log-linear plot. Solid lines are analytic results (no fitted parameters). Standard deviation equals the average lifetime. Standard error of estimated average is ±5% for simulations. (d) For a fixed number of CaMKII holoenzymes (N_CaMKII = 8, right set of curves; N_CaMKII = 4, left set of curves) the number of PP1 molecules is varied. Average lifetime of DOWN states (solid lines, circles) increases with N_PP1 while average lifetime of UP states (dashed line, squares) decreases. Symbols are simulation points [same S.E. as (c)] and lines are analytic results.

A. CaMKII-PP1 system

CaMKII in the PSD is a holoenzyme consisting of twelve subunits, arranged as two hexagonal rings.^91,92 Phosphorylation of the first subunit in a ring (at the Thr286/287 site) requires two Ca²⁺/calmodulin molecules to bind, so is slow in basal calcium concentrations. Once one subunit is phosphorylated, it catalyzes phosphorylation on a neighboring subunit⁹³ so that only one Ca²⁺/calmodulin molecule needs to bind. For geometrical considerations,⁹¹ we assume such autocatalytic activity is directional [Fig. 5(b)]. Hence we consider just two possible states of a subunit—phosphorylated or unphosphorylated at the Thr286/287 site. This leads to fourteen distinguishable states for a ring that cannot be mapped into each other by rotation [Fig. 5(a)]. In our model, we consider phosphorylation of rings independently, so the location of two rings in one holoenzyme is only relevant during turnover, when a complete holoenzyme (of two rings) is removed and replaced.

Dephosphorylation is by PP1 through the Michaelis-Menten scheme⁸ [Fig. 5(c)]. Importantly, in the UP state, the concentration of phosphorylated subunits greatly exceeds the Michaelis constant for PP1 and so saturates the phosphatase activity. The full set of reactions are given in the Supporting Information.⁸

B. Reduction to 1D through separation of time scales

Autophosphorylation and dephosphorylation of individual subunits in a CaMKII ring that is partially phosphorylated occur at a much faster rate than the rate of phosphorylation of an unphosphorylated ring or the rate of turnover. Such a separation in time scales suggests a natural method for reducing the set of states to a single variable, namely the total number of rings with any phosphorylated subunits. We label such rings as “on” and all other rings in the totally unphosphorylated state, P0, as “off.”

We assume “on” rings reach their equilibrium distribution of phosphorylation in the time between changes in the total number of “on” rings. The equilibrium distribution is determined by the activity of phosphatase and depends on the total number of phosphorylated subunits, so it must be calculated self-consistently. The equilibrium indicates the fraction of time spent in each state by an “on” ring or, equivalently, the probability of finding an “on” ring in each state. It is evaluated as the eigenvector of the transition rate matrix (see Appendix); a matrix of transition rates between the different states.

Given the fractional time, ρ_Pi spent in each configuration, labeled Pi where i is the number of phosphorylated subunits [Fig. 5(a)] the average phosphorylation per ring that is on becomes P_av = Σ_iiρ_Pi. Hence the total phosphorylation, Sp_tot, when N_on rings have some level of phosphorylation becomes

$${Sp}_{\text{tot}}=\sum _i{iN}_i=\sum _i{iN}_{\text{on}}{\rho }_{Pi}=N_{\text{on}}{P}_{\text{av}}.$$ (14)

The calculation becomes self-consistent since the phosphatase activity per subunit depends on Sp_tot (see Supporting Information) but also determines Sp_tot since it enters the matrix of transition rates that determine ρ_Pi.

Given a self-consistent equilibrium distribution for “on” states and a corresponding saturation level of the phosphatase activity, the rates for individual rings to switch off or on are calculated. The rate to switch on is independent of phosphorylation level and is simply 6k₁f²(Ca) [Fig. 5(b)]. The rate to switch off is the turnover rate ν_T plus the rate to dephosphorylate from the P1 state [Fig. 5(a)] multiplied by the probability of being in that state, ρ_P1.

We repeat the calculation of individual switching rates as the number of rings that are on varies from 0 to 2N_CaMKII (N_CaMKII is the total number of holoenzymes). Hence we obtain a simplified 1D model of the system, with rates to change the number of rings on by ±1 (by phosphorylation or dephosphorylation/turnover) or by −2 (by turnover). The rates for the 1D system allow us to calculate the total time to transition between any two states (see Appendix).

We find good agreement with the simulation results in the range of parameters tested [Figs. 6(c) and 6(d)]. This gives us confidence to calculate transition times rapidly, analytically, for a heterogeneous set of synapses with variation from the optimal parameter set. We consider such a set of synapses as a graded memory store in the next subsection.

C. Multiple synapses in parallel

We calculate the stable lifetimes of UP and DOWN states for a set of CaMKII-PP1 switches with varying parameters. Each switch could represent a single synapse, or some synapses might contain several switches, but the overall effect is to produce graded information storage. We include switches of up to 16 CaMKII holoenzymes and all numbers of PP1 molecules where bistability is still possible at resting calcium (0.1 μm). Figure 6(d) shows how the lifetime of the UP state falls rapidly with increasing numbers of phosphatase, while the DOWN state lifetime drops if too few phosphatase molecules are present. The curves stop where the concentration of PP1 is outside the bistable range. We do not include such monostable systems, beyond the curves’ end points, in the simulations.

With a variety of 264 switches, we calculated the range of time constants for them to switch DOWN/UP following a transient induced potentiation/depotentiation. The large range is evident in Fig. 7(a), where many switches remain UP, even after 100 years when the average synapse only has ten holoenzymes (the range is 2–16). Such slow time constants could lead to long-term, robust memory storage, if those switches with the slowest time scale for decay from the UP state were also the most difficult and slowest to switch to the UP state. However, the reverse is true [Fig. 6(d)].

FIG. 7.

Decay of memory for a heterogeneous population of synapses, with 2≤N_CaMKII ≤16 and N_CaMKII /2≤N_PP1 ≤2N_CaMKII. (a) Synapses are initially configured in one state, and the fraction that remain unchanged decays very slowly over time. The two curves correspond to all synapses initially UP (lower trace) and all synapses initially DOWN (upper trace). (b) Same simulations as (a) but now the fraction of synapses UP is plotted as a function of time. In this case over a few months nearly all information about the original state is lost.

In Fig. 7(b) we compare the total number of switches UP at any time, following two initial conditions. For the upper curve, initially all switches are UP, whereas for the lower curve, initially all switches are DOWN. As the two curves approach each other, any memory of the initial conditions is lost. Such loss is over several weeks [rather than tens of years, Fig. 7(a)] because those switches that stay UP longest following a potentiation are the same ones that switch UP most rapidly, spontaneously from an initial DOWN state. Hence if such a switch is in the UP state, it provides little information about its past history. In a random array, most switches have a preferred state, but only those switches that have similar UP and DOWN lifetimes [at the crossing of curves in Fig. 6(d)] provide optimal memory storage. It would be intriguing to find if the ratio of CaMKII to PP1 is maintained at such an optimal level through structural mechanisms in the PSD.

Shouval has suggested a method for generating a discrete set of stable memory states through clustering of interacting receptors in the PSD.⁹⁴ In his model, consecutive stable states are reached by adding an extra edge to the rectangle, so the states are not independent of each other, but resemble those of the discrete integrator of Sec. II E.

IV. DISCUSSION AND CONCLUSIONS

For analytical reasons, the average lifetime of a state before a spontaneous transition to another state should increase exponentially with N, the number of molecules involved^19,95 [Eq. (13)]. In the two systems we studied here, such an exponential dependence is seen, at least for large N, for two different reasons. In the biochemical switch, the number of reaction steps that must occur in an unfavorable direction (i.e., in a direction where the reverse reaction is faster) increases with N. An exponential dependence on N arises in a similar manner to the increase in number of rolls of a die before getting N sixes in a row. The probability of the latter is p^N (where p = 1/6) so the number of sets of N rolls rises as (1/p)^N = exp[N ln(1/p)].

On the other hand, for the network of neurons, the total conductance received by a neuron is a fluctuating variable whose mean is scaled to be constant with the number of synaptic inputs, but whose variance scales inversely with the number of inputs. It is the reduction in variance of recurrent excitation that leads to exponentially greater stability with N, in the same way that thermal barrier hopping is exponentially reduced by lowering temperature.⁹⁶ However, the spike times of real neurons in a cortical column are not independent. The correlations between spike times of different neurons result in a lower limit in the variance of total synaptic conductance in a postsynaptic neuron, on the order of that arising from ≈100 neurons.^97,98 This places a similar limit on the stability of UP and DOWN network states. Therefore, in practice the exponential rise of lifetimes will saturate with increasing N and the stability can no longer improve for N larger than ≈ 100.

One can make analytic progress to calculate transition times between stable states, by mapping the problem to a 1D model with an effective potential. For both types of memory systems, a separation of time scales allowed us to pick one slow variable to denote the state of the system. We assume the total synaptic current varies slowly compared to the times between individual spikes in a network of neurons, and in the molecular switch, we assume changes in phosphorylation of a holoenzyme with some phosphorylated subunits are fast compared to the rate of initial phosphorylation and rate of return to an unphosphorylated state.

A similar method of projection to 1D through separation of time scales has been used by Chow and White to estimate the effects of channel fluctuations on the membrane potential of a neuron.⁹⁹ In that case, the time scale for changes in the membrane potential is assumed to be slower than the kinetics of particular gating variables of the channels. Transitions over the effective barrier in the membrane potential result in spontaneous action potentials, occurring at a rate that falls off exponentially with the number of channels contributing to the fluctuations.

Since transition rates depend exponentially on the barrier height of the effective potential, the key to a successful 1D projection is to model the correct barrier states, that is, the most likely states that the system passes through when changing from one state to another. When many routes have a similar likelihood for the transition, and especially if the system exhibits hysteresis (in terms of its state by state changes, not with respect to external parameter variation, where hysteresis is an essential component of any switch) then a 1D projection to a single effective potential is not possible. This is relevant in the CaMKII-PP1 system, where a state like P3d (Fig. 5) is much more probable as rings switch off than when they switch on. However, it is possible to model separate effective potentials for each type of transition, if intermediate states are known. Also, while we just calculated the stability analytically for one neuronal population, the methods of dimensional reduction by Mascaro and Amit¹⁰⁰ can be combined with the methods presented here to allow for generalization to many populations.

We have considered patterns of network activity separately from changes in synaptic strength, which form the connections between neurons in a network. Others have considered the interplay of the two, during learning (reviewed in Ref. 101). For instance, spike-timing-dependent plasticity can cause a network to self-organize to produce two-state membrane potential fluctuations,¹⁰² and stochastic learning in synapses can lead to optimal network memory storage.¹⁰¹

In practice the brain may have more than the two time scales considered here for learning and memory. The brain’s robustness, combined with its ability to respond and encode rapidly may rely on a series of interacting memory systems with a range of lifetimes and varying stability to fluctuations.¹⁰³

A. 1D discrete model

Here we adapt the “pedestrian approach” described by Gillespie⁵³ for the average first passage time in a birth-death Markov process in order to calculate the average first passage time for a general 1D discrete set of states. If a state is labeled by n (n is the number of “on” rings of CaMKII in our example) then a birth-death process only permits changes in n by ±1. Since turnover can cause two rings to be lost (since one holoenzyme contains two rings) or up to twelve phosphorylated subunits (the number of phosphorylated subunits is an alternative label for states), then we extend the calculation of first passage time to the case of general changes in n.

The key to the calculation is to relate the average time, t_n, spent in a state, n, to the number of times, V_n, the process visits state n as

$$V_n=t_n\sum _m{r}_n^{\pm m},$$ (15)

where $r_n^{\pm m}$ is the rate for a single step to take the process from state n to n ± m. This can be extended to give the number of specific transitions, from state n to n ± m as

$$V_n^{\pm m}=t_n{r}_n^{\pm m}.$$ (16)

In a birth-death Markov process (where $r_n^{\pm m}=0$ for all m >1) for a passage from state N₁ to state N₂ (we assume here N₂ >N₁) then

$$V_n^{+}=1+V_{n+1}^{-}\text{for}{N}_1\le n<N_2$$ (17)

and

$$V_n^{+}=V_{n+1}^{-}\text{for}n<N_1$$ (18)

which is to say that at all points along the required path an extra step must be taken forward in the desired direction than steps back in the reverse direction in order to complete the path.

Combining Eqs. (17) and (18) with the requirement that only one step need be taken to the end point, during a transition from N₁ to N₂ we have for a birth-death process

$$\begin{array}{l}{t}_n=0\text{for}n\ge N_2,\\ r_n^{+}{t}_n=1+r_{n+1}^{-}{t}_{n+1}\text{for}{N}_1\le n<N_2\\ r_n^{+}{t}_n=r_{n+1}^{-}{t}_{n+1}\text{for}n<N_1.\end{array}$$ (19)

These equations can be solved iteratively, beginning at t_N2 and working back to t₀ to evaluate all {t_n} from which the total transition time is calculated as

$$T(N_1\mapsto N_2)=\sum _n{t}_n.$$ (20)

To begin the generalization, we first note that the number of reaction steps leading to a state n is equal to the number of reaction steps away from that state, except for the initial state n = N₁ and the final state n = N₂. So we can rewrite the above equations as

$$r_{n-1}^{+}{t}_{n-1}-(r_n^{+}+r_n^{-})t_n+r_{n+1}^{-}{t}_{n+1}=\{\begin{array}{c}-1\text{for}n=N_1\\ 1\text{for}n=N_2\\ 0\text{for}n\ne N_1,N_2,\end{array}$$ (21)

which corresponds to a tridiagonal matrix equation (with components from n = 0 up to n = N₂) that is readily solved for {t_n}. The sum of consecutive rows from 0 to n in the matrix equation [Eq. (21)] leads to Eq. (19).

To ease the notation for transitions by more than one in n, we rewrite the transition matrix from n to n′ as R_nn′ such that $r_n^{\pm m}=R_{nn\pm m}$. In this case the tridiagonal matrix equation [Eq. (21)] generalizes to

$$\sum _{m\ne n}{t}_m{R}_{mn}+t_n\sum _l{R}_{nl}=\{\begin{array}{c}-1\text{for}n=N_1\\ 0\text{for}n\ne N_1,n<N_2\end{array}$$ (22)

and to terminate the process for hops beyond N₂ as well as those that reach N₂ precisely, we have for the final row

$$\sum _{m<N_2}{t}_m\sum _{n\ge N_2}{R}_{mn}=1.$$ (23)

A straightforward matrix inversion leads to the set of times in each state, {t_n} which are summed to produce the average transition time.

We calculate the probability distribution of the total phosphorylation [Fig. 6(b)] by first calculating the fraction of time a given number N_on of holoenzymes are on (the sum of t_Non for an upward transition and a downward transition, divided by the sum of total time for an upward transition and a downward transition). We calculate the variance in phosphorylation level of a holoenzyme as Var[P(N_on)] = Σ_ii²ρ_Pi−[Σ_iiρ_Pi]² for each N_on and assume when N_on holoenzymes are on, the total phosphorylation is distributed with mean N_onΣ_iiρ_Pi [see Eq. (14)] and variance N_on Var[P(N_on)].