Synaptic plasticity as Bayesian inference

Abstract

Learning, especially rapid learning, is critical for survival. However, learning is hard: a large number of synaptic weights must be set based on noisy, often ambiguous, sensory information. In such a high-noise regime, keeping track of probability distributions over weights is the optimal strategy. Here we hypothesize that synapses take that strategy; in essence, when they estimate weights, they include error bars. They then use that uncertainty to adjust their learning rates, with more uncertain weights having higher learning rates. We also make a second, independent, hypothesis: synapses communicate their uncertainty by linking it to variability in PSP size, with more uncertainty leading to more variability. These two hypotheses cast synaptic plasticity as a problem of Bayesian inference, and thus provide a normative view of learning. They generalize known learning rules, offer an explanation for the large variability in the size of post-synaptic potentials, and make falsifiable experimental predictions.

1. Introduction

To survive, animals must accurately estimate the state of the world. This estimation problem is plagued by uncertainty: not only is information often extremely limited (e.g., because it is dark) or ambiguous (e.g., a rustle in the bushes could be the wind, or it could be a predator), but sensory receptors, and indeed all neural circuits, are noisy. Historically, models of neural computation ignored this uncertainty, and relied instead on the idea that the nervous system estimates values of quantities in the world, but does not include error bars [1]. However, this does not seem to be what animals do — not only does ignoring uncertainty lead to suboptimal decisions, it is inconsistent with a large body of experimental work [2, 3]. Thus, the current view is that in many, if not most, cases, animals keep track of uncertainty, and use it to guide their decisions [3].

Accurately estimating the state of the world is just one problem faced by animals. They also need to learn, and in particular they need to leverage their past experience. It is believed that learning primarily involves changing synaptic weights. But estimating the correct weights, like estimating the state of the world, is plagued by uncertainty: not only is the information available to synapses often extremely limited (in many cases just pre and post synaptic activity), but that information is highly unreliable. Historically, models of synaptic plasticity ignored this uncertainty, and assumed that synapses do not include error bars when they estimate their weights. However, uncertainty is important for optimal learning — just as it is important for optimal inference of the state of the world.

Motivated by these observations, we propose two hypotheses. The first, Bayesian Plasticity (so named because it is derived using Bayes' rule), states that during learning, synapses do indeed take uncertainty into account. Under this hypothesis, synapses do not just estimate what their weight should be, they also include error bars. This allows synapses to adjust their learning rates on the fly: when uncertainty is high learning rates are turned up, and when uncertainty is low learning rates are turned down. We show that these adjustments allow synapses to learn faster, so there is likely to be considerable evolutionary pressure for such a mechanism. And indeed, the same principle has recently been shown to recover state-of-the-art adaptive optimization algorithms for artificial neural networks [4].

Bayesian Plasticity is a hypothesis about what synapses compute. It does not, however, tell synapses how to set their weights. For that a second hypothesis is needed. Here we propose that weights are sampled from the probability distribution describing the synapse’s degree of uncertainty. Under this hypothesis, which we refer to as Synaptic Sampling, trial to trial variability provides a readout of uncertainty: the larger the trial to trial variability in synaptic strength, the larger the uncertainty. Synaptic Sampling is motivated by the observation that the uncertainty associated with a particular computation should depend on the uncertainty in the weights. Thus, to make optimal decisions, the brain needs to know something about the uncertainty; one way for synapses to communicate that is via variability in the postsynaptic potential (PSP) amplitude (see Supplementary Math Note, Sec. S5, for an extended discussion).

Combined, these two hypotheses make several strong experimental predictions. As we discuss below, one is consistent with re-analysis of existing experimental data; the others, which are feasible in the not so distant future, could falsify one or both hypotheses. We begin by analyzing the first hypothesis, that synapses keep track of their uncertainty (Bayesian Plasticity); after that we discuss our second hypothesis, that synapses sample from the resulting distribution (Synaptic Sampling).

2. Results

Under Bayesian Plasticity, each synapse computes its mean and variance, and updates both based on the pattern of presynaptic spikes. In analogy to classical learning rules, the update rule for the mean pushes it in a direction that reduces a cost function. But in contrast to classical learning rules, the amount the mean changes depends on the uncertainty: the higher the uncertainty, as measured by the variance, the larger the change in the mean. The variance thus sets the learning rate, as shown in Fig. 1. In essence, there is a rule for computing the learning rate of each synapse.

Figure 1.

The delta rule is suboptimal. The error bars denote uncertainty (measured by the standard deviation around the mean) in two synapses' estimates of their target weights, w_tar,1 and w_tar,2. The first is reasonably certain; the second less so. The red arrows denote possible changes in response to a negative feedback signal. The arrow labeled “delta rule” represents an equal decrease in the first and second target weights. In contrast, the arrow labeled “optimal” takes uncertainty into account, so there is a larger change in the second, more uncertain, target weight.

To illustrate these ideas, we consider a model of synaptic integration in which postsynaptic potentials combine linearly,

$$V(t)=\sum _i{w}_i(t)x_i(t)+{\xi }_V(t)$$ (1)

where V(t) is the membrane potential, x_i(t) is the synaptic input from neuron i, w_i(t) is the corresponding PSP amplitude, and ξ_V(t) is the membrane potential noise. For simplicity we work in discrete time, so x_i(t) is either 1 (when there is a spike at time t) or 0 (when there is no spike), and we take the time step to be 10 ms, on the order of the membrane time constant [5].

We assume that the goal of the neuron is to set its weights, w_i, so that it achieves a “target” membrane potential (denoted V_tar) - the membrane potential that minimizes some cost to the animal. In this setting, the weights are found using a neuron-specific feedback signal, denoted f. Critically, this feedback signal contains information about the target weights through its dependence on a true error signal, δ - the difference between the target and actual membrane potential,

$$\delta \equiv V_{\text{tar}}-V.$$ (2)

Our focus is on how to use the feedback signal most efficiently, not on where it comes from (an active area of research [6, 7, 8, 9, 10]). Thus, in most of our analysis we simply assume that the neuron receives a feedback signal, and ask how to optimally update the weights via Bayesian inference.

We consider several learning scenarios. In the first, we simply add noise, denoted ξ_δ, to δ, resulting in the error signal f_lin = δ + ξ_δ (the subscript “lin” indicates that the average feedback is linear in δ). The second corresponds to cerebellar learning, in which a Purkinje cell receives a complex spike if its output is too high, thus triggering long term depression [11]. To mimic the all-or-nothing nature of a complex spike [12], we use a cerebellar-like feedback signal: f_cb = Θ(δ + ξ_δ − θ) where Θ is the Heaviside step function. For this feedback signal, f_cb is likely to be 1 if δ is above a threshold, θ, and likely to be 0 if it is below threshold. The third corresponds to reinforcement learning, in which the feedback represents the reward. The reward provides the magnitude of the error signal, but not its sign, so the feedback signal is f_rl = − |δ + ξ_δ|. In the fourth scenario we move beyond analysis of single neurons, and consider learning the output weights of a recurrent neural network. In this scenario, the error signal is δ, without added noise.

The main idea behind Bayesian plasticity is most easily illustrated in the simplest possible setting, linear feedback, f_lin = δ + ξ_δ. In that case there is a well known learning rule, the delta rule [13, 14],

$$\Delta w_i=\eta x_i{f}_{\text{lin}}.$$ (3)

(This is most easily recognized as the delta rule in the absence of noise, so that f_lin = δ.) The change in the weight is the product of a learning rate, η, a presynaptic term, x_i and a postsynaptic term, f_lin. Importantly, the learning rate, η, is the same for all synapses, so all synapses whose presynaptic cells are active (i.e., for which x_i = 1) change by the same amount (the red arrow labeled “delta rule” in Fig. 1).

In the absence of any other information, the delta rule is perfectly sensible. However, suppose, based on previous information, that synapse 1 is relatively certain about its target weight, whereas synapse 2 is uncertain (error bars in Fig. 1). In that case, new information should have a larger effect on synapse 2 than synapse 1, so synapse 2 should update the estimate of its weight more than synapse 1 (red arrow labeled “optimal” in Fig. 1).

Implementing this scheme leads to several features that are not present in classical learning rules. First, the variance needs to be inferred; second, the change in the weight must depend on the inferred variance; and third, because of uncertainty, the “weight” is in fact the inferred mean weight. In Supplementary Math Note, Sec. S1.3, we derive approximate learning rules that take these features into account (see Methods, Sec. M2, for the exact rules). Using μ_i and ${\sigma }_i^2$ to denote the inferred mean and variance of the distribution over weights, those learning rules are

$$\Delta {\mu }_i\approx \frac{{\sigma }_i^2}{{\sigma }_{\delta }^2}{x}_i{f}_{\text{lin}}-\frac{1}{\tau }\left({\mu }_i-{\mu }_{\text{prior}}\right)$$ (4a)

$$\Delta {\sigma }_i^2\approx -\frac{{\sigma }_i^4}{{\sigma }_{\delta }^2}{x}_i^2-\frac{2}{\tau }\left({\sigma }_i^2-{\sigma }_{\text{prior}}^2\right)$$ (4b)

where ${\sigma }_{\delta }^2$ is the variance of f_lin, and τ, μ_prior, and ${\sigma }_{\text{prior}}^2$ are fixed parameters (described shortly). Note that σ_i corresponds to the length of the error bars in Fig. 1.

The update rule for the mean weight, Eq. (4a), is similar to the delta rule, Eq. (3). There are, however, two important differences. First, the fixed learning rate, η, that appears in Eq. (3) has been replaced by a variable learning rate, ${\sigma }_i^2/{\sigma }_{\delta }^2$, which is proportional to the synapse’s uncertainty, as measured by ${\sigma }_i^2$. Thus, the more uncertain a synapse is about its target weight, the larger the change in its mean weight when new information arrives — exactly what we expect given Fig. 1. Moreover, as the feedback signal gets noisier (as measured by the variance of f_lin), and thus less informative, the learning rate falls. Second, in the absence of information (x_i = 0, meaning no spikes), the inferred mean weight, μ_i, moves toward the prior, μ_prior. That is because we are considering the realistic case in which the target weights drift randomly over time due to changes in the statistics of the world and/or surrounding circuits. See Methods, Sec. M1.1 for a detailed discussion of this point.

Unlike the update rule for the mean, the update rule for the uncertainty, ${\sigma }_i^2$ (Eq. 4b), does not have a counterpart in classical learning rules. It does, however, have a natural interpretation. The first term in Eq. (4b) reduces uncertainty (note the negative sign) whenever the presynaptic cell is active (x_i = 1); that is, whenever the synapse receives information. The second term has the opposite effect: it continually increases uncertainty (up to the prior uncertainty, ${\sigma }_{\text{prior}}^2$), independent of presynaptic spikes. That term arises because random drift slowly reduces knowledge about the target weights.

The learning rules given in Eq. (4) are approximate - their form was optimized for ease of interpretation rather than accuracy. However, the more exact learning rules (Methods, Eqs. (M.41), (M.43) and (M.47), for the three feedback signals) are not that different. In particular, they retain the same flavor: they consist of a presynaptic term (x_i) and a postsynaptic term (a function of f_lin), and the effective learning rate is updated on each timestep. Moreover, the interpretation is the same: the mean is moved, on average, toward its true value, with a rate that scales with uncertainty, and whenever there is a presynaptic spike the uncertainty is reduced.

To determine whether our Bayesian learning rules are able to accurately compute the mean and variance of the weights, we generated a set of target weights, denoted w_tar,i, and used those to construct V_tar,

$$V_{\text{tar}}(t)=\sum _i{w}_{\text{tar},i}(t)x_i(t).$$ (5)

Simulations show that the mean weights track the target weights very effectively (Fig. 2; compare the black and red lines, which correspond to the target weight and its inferred mean, respectively). Just as importantly, the synapse’s estimate of its uncertainty tracks the difference between its estimate and the actual target (the black line should be inside the 95% confidence interval – the red area in Fig. 2 - 95% of the time, and it is very close to that: linear, 96.1%; cerebellar learning, 95.4%; reinforcement learning, 96.8%). Note that the uncertainty is much lower at high presynaptic firing rate than at low rate (the red regions in the top row of Fig. 2 are much narrower than in the bottom row). That is because for low firing rate x_i is mainly zero, and so there is little decrease in ${\sigma }_i^2$; see Eq. (4b).

Figure 2.

Bayesian learning rules track the target weight and estimate uncertainty. The black line is the target weight, the red line is the mean of the inferred distribution, and the red area represents 95% confidence intervals of the inferred distribution. Panels a-c correspond to the highest presynaptic firing rate used in the simulations; panels d-f to the lowest. Consistent with our analysis (see in particular Eq. 9), higher presynaptic firing rate resulted in lower uncertainty. a and d. Linear feedback, f_lin = δ + ξ_δ. b and e. Cerebellar learning, f_cb = Θ (δ + ξ_δ − θ). c and f. Reinforcement learning, f_rl = − |δ +ξ_δ|. See Supplementary Math Note, Sec. S3, for simulation details. Note while the red lilnes are all plotted at the same thickness, the greater variability in the lower plots may make those lines appear thicker.

The critical aspect of the learning rules in Eq. (4) is that the learning rate — the change in mean PSP amplitude, μ_i, per presynaptic spike — increases as the synapse’s uncertainty, ${\sigma }_i^2$, increases. This is a general feature of our learning rules, and not specific to any one of them. Consequently, independent of the learning scenario, we expect performance to be better than for classical learning rules, which do not take uncertainty into account. To check whether this is true, we computed the mean squared error between the actual and target membrane potential, V and V_tar, for classical learning rules, and compared it to the Bayesian learning rules. The results are shown in Fig. 3. The black line in each panel is the mean squared error for the classical learning rules as a function of learning rate; the red line is the mean squared error for the Bayesian learning rules. (The latter is constant because the Bayesian learning rules do not depend on the classical learning rate.) As predicted, the Bayesian learning rules always do better than the classical ones, even if the learning rate is tuned to its optimal value. This result was robust to model mismatch (Supplementary Math Note, Fig. S.6).

Figure 3.

Bayesian learning rules exhibit lower error than classical ones. Red: mean squared error between the target and actual membrane potential for the Bayesian learning rules; black: mean squared error for the classical rules. a. Linear feedback, f_lin = δ + ξ_δ. b. Cerebellar learning, f_cb = Θ (δ + ξ_δ − θ). c. Reinforcement learning, f_rl = −|δ + ξ_δ|. See Supplementary Math Note, Sec. S3, for simulation details.

For the examples so far, we considered a single neuron inferring only its own input weights. We focused on this case primarily to illustrate our method in the simplest possible setting. In reality, however, the brain needs to optimize some cost function based on a feedback signal applied to a recurrent neural network. To investigate Bayesian plasticity in this, more realistic, regime, we trained the output weights of a recurrent neural network to produce a target function, using as a feedback signal the difference between the target function and its network estimate (Fig. 4a). The learning rules are very similar to Eq. (4) (see Methods, Sec. M2.3). However, the target weights are not known, so we cannot compare the inferred weights to the target weights, as we did in Fig. 2. We can, however, compare the mean squared error between the target and actual membrane potential, as in Fig. 3. Bayesian plasticity does indeed outperform classical learning rules (Figs. 4b and c). Moreover, the effect is much larger than in Fig. 3: the mean squared error is about an order of magnitude smaller for the Bayesian than for the classical learning rule (note the log scale in Fig. 4c), a result that was highly robust to model mismatch (Supplementary Math Note, Fig. S.7). These simulations suggest that taking into account weight uncertainty has a much larger effect in networks than in single neurons.

Figure 4.

Recurrent neural network. a. Schematic of the circuit. I(t) is the input (used to initialize activity) and w corresponds to the learned output weights. The feedback weights (black arrows from V to the recurrent network) are fixed, as are the recurrent weights. During learning, the output of the network, V(t), is compared to the target output, V_tar(t), and the error is used to update the output weights, w. At test time, the target output is not fed back to the circuit. b. Learning curves, measured using mean squared error, for Bayesian and classical learning rules (red and blue, respectively, at a range of learning rates for the classical rule). Although the initial improvement in performance for the Bayesian and classical learning rules was about the same, after 100 time steps Bayesian learning became much more efficient. The arrows correspond to the number of time steps used for the comparison in panel c. c. Mean squared error versus the learning rate of the classical rule. Solid lines: classical learning rules; dashed lines: Bayesian learning rules. The mean squared error for the Bayesian learning rule was about an order of magnitude smaller than for the classical one. In panels c and d we plot the median, taken over n = 400 network/target pairs; error bars are 95% confidence intervals, computed using the percentile bootstrap.

Figures 3 and 4 indicate that there is a clear advantage to using uncertainty to adjust learning rates. But does the brain do this? Addressing that question will require a new generation of plasticity experiments. At present, in typical plasticity experiments only changes in weights are measured; to test our hypothesis, it will be necessary to measure changes in learning rates, and at the same time determine how those changes are related to the synapse’s uncertainty. This presents two challenges. First, measuring changes in learning rates is difficult, as weights must be monitored over long periods of time and under natural conditions, preferably in vivo. Second, we cannot measure the synapse’s uncertainty directly. Here we discuss two approaches to overcoming these challenges.

The first approach is indirect: use neural activity measured over long periods in vivo to estimate the uncertainty a synapse should have; then, armed with that estimate, test the prediction that the learning rate increases with uncertainty. To estimate the uncertainty a synapses should have, we take advantage of a general feature of essentially all learning rules: synapses get information only when the presynaptic neuron spikes. Consequently, the synapse’s uncertainty should fall as the presynaptic firing rate increases. In fact, under mild assumptions, we can derive a very specific relationship: the relative change in weight under a plasticity protocol, Δμ_i/μ_i, should scale approximately as $1/\sqrt{{\nu }_i}$ where ν_i is the firing rate of the neuron presynaptic to synapse i,

$$\frac{\Delta {\mu }_i}{{\mu }_i}\propto \frac{1}{\sqrt{{\nu }_i}},$$ (6)

a relationship that holds in our simulations for firing rates above about 1 Hz (see Supplementary Math Note, Fig. S.3, bottom row). In essence, firing rate is a proxy for uncertainty, with higher firing rate indicating lower uncertainty and vice versa. This prediction could be tested by observing neurons in vivo, estimating the presynaptic firing rates, then performing plasticity experiments to determine the relative change in synaptic strength, Δμ_i/μ_i.

The second approach is more direct, but it requires an additional hypothesis. While Bayesian Plasticity tells us how to compute the mean and variance of the weights, it does not tell us what weight to use when a spike arrives. But the Synaptic Sampling hypothesis does: it tells us that the mean and variance of the PSP amplitude should be equal to the mean and variance of the inferred distribution over the target weight,

$$\text{PSP}\text{mean}={\mu }_i$$ (7a)

$$\text{PSP}\text{variance}={\sigma }_i^2.$$ (7b)

Under our learning rules, the change in mean synaptic weight is proportional to the variance, ${\sigma }_i^2$ (Eq. 4a). Consequently, the relative change in weight Δμ_i/μ_i, is proportional to ${\sigma }_i^2/{\mu }_i$; combining this with Eq. (7) gives us

$$\frac{\Delta {\mu }_i}{{\mu }_i}\propto \frac{\text{PSP}\text{variance}}{\text{PSP}\text{mean}}\equiv \begin{array}{c}\text{Normalized}\\ \text{Variability}\end{array},$$ (8)

where we have defined the normalized variability to be the ratio of PSP variance to its mean. We verify that this relationship holds in simulations (see Supplementary Math Note, Fig. S.2).

Equation (8) implies that when the PSP variance is high, learning rates are also high. Testing that experimentally is technically difficult: it requires monitoring the PSP mean and variance for long periods in vivo, and comparing normalized variability to changes in the mean. However, such experiments are likely to be possible in the near future.

A more indirect approach based on this idea, for which we can apply current data, makes use of Eq. (6) to replace the left hand side of Eq. (8), Δμ_i/μ_i, with $1/\sqrt{{\nu }_i}$. This gives us

$$\begin{array}{c}\text{Normalized}\\ \text{Variability}\end{array}\propto \frac{1}{\sqrt{{\nu }_i}}.$$ (9)

This is intuitively sensible: as discussed above, higher presynaptic firing rates means the synapse is more certain, and Synaptic Sampling states that higher certainty should reduce the observed variability. This relationship can be tested by estimating presynaptic firing rates in vivo, and comparing them to the normalized variability measured using paired recordings. Such data can be extracted from experiments by Ko and colleagues [15]. In those experiments, calcium signals in mouse visual cortex were recorded in vivo under a variety of stimulation conditions, which provided an estimate of the firing rate of each imaged neuron; subsequently, whole cell recordings of pairs of identified neurons were made in vitro, and the mean and variance of the PSPs were measured. In Fig. 5 we plot the normalized variability versus the firing rate on a log-log scale; on this scale, our theory predicts a slope of −1/2 (red line). The normalized variability does indeed decrease as the firing rate increases (blue line), (p < 0.003), and the slope is not significantly different from the predicted value of −1/2 (p = 0.57). This pattern is broadly matched by simulations, at least at sufficiently high firing rate (Supplementary Math Note, Fig. S.3, top row).

Figure 5.

Normalized variability (the ratio of the PSP variance to the mean) versus presynaptic firing rate as a diagnostic of our theory; data supplied to us by the authors of [15] (see Supplementary Math Note, Sec. S4.4). The red line, which has a slope of –1/2, is our prediction (the intercept, for which we do not have a prediction, was chosen to give the best fit to the data). The blue line is fit by linear regression(n =136 points), and the gray region represents 2 standard errors. The slope of the blue line, −0.62, is statistically significantly different from 0 (p < 0.003, t-test) and not significantly different from −1/2 (p = 0.57, t-test; assumes normality which was not formally tested). The firing rate was measured by taking the average signal from a spike deconvolution algorithm [45]. Units are arbitrary because the scale factor relating the average signal from the deconvolution algorithm and the firing rate is not exactly one [46]. Data from layer 2/3 of mouse visual cortex [15].

An alternative explanation for this result is that increases in firing rate reduce the normalized variability because of short term effects on release probability. The release probability, denoted p_r, scales the variance of the PSP by a factor of p_r(1 − p_r) and the mean by a factor of p_r, so the normalized variability (the variance divided by the mean) scales as 1 − p_r. Consequently, an increase in release probability with firing rate would explain Fig. 5. Such increases do indeed occur [16]. However, much more common — especially in rodent layer 2/3, where these experiments were performed — is a decrease in release probability with firing rate [17, 18]. Thus, short term synaptic plasticity would typically lead to an increase, not a decrease, in the normalized variability when firing rate increases; the opposite of what we see experimentally.

3. Discussion

We proposed that synapses do not just keep track of point estimates of their weights, as they do in classical learning rules; they also keep track of their uncertainty. They then use that uncertainty to set learning rates: the higher the uncertainty, the higher the learning rate. This allows different synapses to have different learning rates, and leads to learning rules that allow synapses to exploit all locally available information. This in turn leads to better performance, as measured by mean squared error (Figs. 3 and 4b,c). It also leads to faster learning. That is implicit in Fig. 3 (because the target weights drift, fast learning is essential for achieving low mean squared error) and it is explicit in Fig. 4b (compare red to blue curves).

The critical difference between our learning rules and classical ones is that the learning rates themselves undergo plasticity. We derived three rules, based on three different assumptions about the feedback signal received by the neuron, and in all cases the updates for the mean had the flavor of a classical rule: the change in the mean weight was a function of the presynaptic activity and an error signal. Other assumptions about the feedback signal are clearly possible, and our method can generate a broad range of learning rules. Whether or not they can generate all rules that have been observed experimentally is an avenue for future research.

The hypothesis that synapses keep track of uncertainty, which we refer to as the Bayesian Plasticity hypothesis, makes the general prediction that learning rates, not just synaptic strengths, are a function of pre and postsynaptic activity — something that should be testable with the next generation of plasticity experiments. In particular, it makes a specific prediction about learning rates in vivo: learning rates should vary across synapses, being higher for synapses with lower presynaptic firing rates.

We also made a second, independent, hypothesis, Synaptic Sampling. This hypothesis states that the variability in PSP size associated with a particular synapse matches the uncertainty in the strength of that synapse. This allows synapses to communicate their uncertainty to surrounding circuitry — information that is critical if the brain is to monitor the accuracy of its own computations. The same principle has been applied to neural activity, where it is known as the neural sampling hypothesis [19, 20, 21, 22], which posits that variability in neural activity matches uncertainty about the state of the external world. The neural sampling hypothesis meshes well with synaptic sampling: uncertainty in the weights increases uncertainty in the current estimate of the state of the world, and likewise, variability in the weights increase variability in neural activity (see Supplementary Math Note, Sec. S5). While there is some experimental evidence for the neural sampling hypothesis [21, 23, 22, 24, 25], it has not been firmly established. Whether other proposals for encoding probability distributions with neural activity, such as probabilistic population codes [3, 26], can be combined with Synaptic Sampling is an open question.

By combining our two hypotheses, we were able to make additional predictions. These focused on what we call the normalized variability — the ratio of the variance in PSP size to the mean. First, we predicted that plasticity should increase with normalized variability, which remains to be tested. Second, we predicted that normalized variability should decrease with presynaptic firing rate. Reanalyzing data from [14], we provided evidence that this is indeed the case.

In machine learning, the idea that it is advantageous to keep track of the distribution over weights has a long history [27, 28, 29]. Especially relevant is a recent study in which, as in our scheme, learning rates were reduced when certainty was high [30]. However, rather than updating the uncertainty on every time step, as we do, updating occurred only when there was a change in the task. This occurs on the timescale of minutes to hours; not the millisecond timescale on which uncertainty is updated in our model. Nevertheless, this approach worked well in settings in which deep networks had to learn multiple tasks.

In neuroscience, weight uncertainty was first explored in the context of reinforcement learning [31]. In that work, the weights related sensory stimuli to rewards, and weight correlations that developed due to Bayesian learning provided an exceptionally elegant explanation of backward blocking. The idea lay dormant for over a decade, until it was rediscovered with a slightly different focus, one in which knowledge of weight uncertainty is critical for knowledge of computational uncertainty [3]. Several theoretical studies followed. The first of those [32] bore some resemblance to ours, in that weights were sampled from a distribution. However, the timescale for sampling was hours rather than milliseconds; too slow to explain the spike-to-spike variability in PSP size that is ubiquitous in the brain. More recently, Hiratani and Fukai [33] postulated that the multiple synaptic contacts per connection observed in cortex provides a scaffolding for constructing a non-parametric estimate of the probability distribution over synaptic strength. Weight uncertainty has also been applied to drift diffusion models [34], using methods similar to those in [31]; the main difference was that the reward was binary (correct or incorrect) rather than continuous. Finally, recent work proposed that short-term plasticity is also governed by a Bayesian updating process [35]. It will be interesting to determine which combination of these schemes is used by the brain.

If the Bayesian Plasticity hypothesis is correct, synapses would have to keep track of, and store, two variables: the mean, as is standard, but also the variance (or, equivalently, the learning rate), which is not. The complexity of synapses [36, 37, 38], and their ability to use non-trivial learning rules (e.g., synaptic tagging, in which activity at a synapse “tags” it for future long term changes in strength [39, 40, 41], and metaplasticity, in which the learning rate can be modified by synaptic activity without changing the synaptic strength [42, 43, 44]), suggests that representing uncertainty — or learning rate — is quite possible. It will be nontrivial, but important, to work out how.

Our framework has several implications, both for the interpretation of neurophysiological data and for future work. First, under the Synaptic Sampling hypothesis, PSPs are necessarily noisy. Consequently, noise in synapses (e.g., synaptic failures) is a feature, not a bug. We thus provide a normative theory for one of the major mysteries in synaptic physiology: why neurotransmitter release is probabilistic. Second, our approach allows us to derive local, biologically plausible learning rules, no matter what information is available at the synapse, and no matter what the statistics of the synaptic input. Thus, our approach provides the flexibility necessary to connect theoretical approaches based on optimality to complex biological reality.

In neuroscience, Bayes theorem is typically used to analyze high level inference problems, such as decision-making under uncertainty. Here we demonstrated that Bayes’ theorem, being the optimal way to solve any inference problem, big or small, could be implemented in perhaps the smallest computationally relevant element in the brain: the synapse.

Methods

Here we provide a complete description of our model (Sec. M1) and sketch the derivation of the learning rules (Sec. M2).

M1. Description of our model

In the main text we specified how the membrane potential depends on the weights and incoming spikes (Eq. 1) and how the target membrane potential depends on the target weights and incoming spikes (Eq. 5), and we defined the error signal (Eq. 2). In this section we describe how target weights, w_tar,i, the weights, w_i, the and the spikes, x_i, are generated.

M1.1. Target weights

The target weights are the weights that in some sense optimize the performance of the animal. We do not expect these weights to remain constant over time, for two reasons. First, both the state of the world and the organism change over time, thus changing the target weights. Second, we take a local, single neuron view to learning, and define the target weights on a particular neuron to be the optimal weights given the weights on all the other neurons in the network. Consequently, as the weights of surrounding neurons change due to learning, the target weights on our neuron also change. While these changes may be quite systematic, to a single synapse deep in the brain they are likely to appear random.

Motivated by this last observation, in our model we assume that the target weights evolve according to a random process. To ensure that the weights do not change sign, we work in log space, and on each time step we add a small amount of noise to the log of the target weights. And to ensure that the weights do not become too small or too large, we add a small drift toward a prior log weight. Specifically, defining

$${\lambda }_{\text{tar},i}=\text{log}\left|w_{\text{tar},i}\right|$$ (M.10)

(note the absolute value sign, which allows the weights to be either positive or negative), we let λ_tar,i evolve according to

$${\lambda }_{\text{tar},i}(t+1)={\lambda }_{\text{tar},i}(t)-\frac{{\lambda }_{\text{tar},i}(t)-m_{\text{prior}}}{\tau }+\sqrt{\frac{2s_{\text{prior}}^2}{\tau }}{\xi }_{\text{tar},i}$$ (M.11)

where m_prior and $s_{\text{prior}}^2$ are the prior mean and variance of λ_tar,i(t), τ (which is dimensionless) is the characteristic number of steps over which λ_tar,i(t) changes, and ξ_tar,i is a zero mean, unit variance Gaussian random variable.

We chose the noise process described in Eq. (M.11) for three reasons. First, w_tar,i is equal to either +e^λ_tar,i (for excitatory weights) or −e^λ_tar,i (for inhibitory weights), and thus cannot change sign as λ_tar,i changes with learning. Consequently, excitatory weights cannot become inhibitory, and vice versa, so Dale’s law is preserved. Second, spine sizes obey this stochastic process [47], and while synaptic weights are not spine sizes, they are correlated [48]. Third, this noise process gives a log-normal stationary distribution of weights, as is observed experimentally [49].

The parameters that determine how the weights drift, m_prior and $s_{\text{prior}}^2$, were set to the mean and variance of measured log-weights using data from Ref. [49] (Supplementary Math Note, Sec. S4.1). We used a time step of 10 ms, within the range of measured membrane time-constants. For the linear and cerebellar models we set τ to 10⁵; for reinforcement learning we set τ to 5 × 10⁵. These values were chosen so that uncertainty roughly matched observed variability (Supplementary Math Note, Sec. S4.3). For the recurrent network we do not know the target weights, so we do not know the drift rate. Nor do we know the effective drift associated with the fact that the optimal weight on one synapses changes as the surrounding circuit changes. We therefore tried different drifts in our simulations (data not shown). We found that near zero drift was optimal, so we set τ to ∞.

M1.2. Synaptic weights

Our inference algorithm computes a distribution over the target weights. Given that distribution, though, there is nothing in the Bayesian Plasticity hypothesis that tells us how to set the weights when a spike arrives. That is where the Sampling Hypothesis comes in: it tells us to sample the weights, w_i, from the posterior,

$$w_i=e^{m_i+s_i{\xi }_i},$$ (M.12)

where m_i and s_i are the mean and standard deviation of the posterior distribution over the log weights, and ξ_i is a zero mean, unit variance Gaussian random variable. The mean and variance of w_i under Eq. (M.12), for which we use μ_i and ${\sigma }_i^2$, respectively, are the standard expressions for the mean and variance of a log-normal distribution,

$$𝔼\left[w_i\mid {𝒟}_i\right]\equiv {\mu }_i=e^{m_i+s_i^2/2}$$ (M.13a)

$$\text{Var}\left[w_i\mid {𝒟}_i\right]\equiv {\sigma }_i^2={\mu }_i^2\left[e^{s_i^2}-1\right]$$ (M.13b)

where 𝓓_i is the data seen by the synapse so far (see Eq. M.19 below).

In the main text, we compare our Bayesian learning rules to classical ones (see in particular Figs. 3 and 4). For classical rules there is no posterior to sample from, so we can not use Eq. (M.12). Consequently, for the the classical implementation of linear and cerebellar rules we do not sample, and for Bayesian learning we use w_i = μ_i. The reinforcement learning rule, however, requires sampling, for both Bayesian and classical learning (see Eqs. M.47 and M.48). We thus assumed that the variance is proportional to the mean (as is the case for Poisson statistics). To find the constant of proportionality, denoted k, we use data from Ref. [49]; see Supplementary Math Note, Sec. S4.2 for details. A least squares fit to that data gives k = 0.0877. A naive way to implement this is to sample weights using $w_i={\mu }_i+\sqrt{k{\mu }_i}{\xi }_i$ with ξ_i ~ 𝒩 (0,1). However, that allows w_i to change sign, so instead we sample the weights using

$$w_i={\mu }_i{e}^{{\beta }_i+{\gamma }_i{\xi }_i},$$ (M.14)

and choose β_i and γ_i so that the mean and variance of w_i are μ_i and kμ_i, respectively. As is straightforward to show, these conditions are satisfied when β_i and γ_i are given by

$${\beta }_i=-\frac{\text{log}\left(1+k/{\mu }_i\right)}{2}$$ (M.15a)

$${\gamma }_i=\sqrt{\text{log}\left(1+k/{\mu }_i\right)}.$$ (M.15b)

M1.3. Synaptic input

For linear, cerebellar and reinforcement learning, neurons receive input from n presynaptic neurons, all firing at different rates. The firing rates, ν_i (i labels presynaptic neuron), are drawn from a log-normal distribution, using a distribution that is intermediate between the narrow range found by some [50] and the broad range found by others [51]: a log-normal with median at 1 Hz and with 95% of firing rates being between 0.1 Hz and 10 Hz,

$$\text{log}{\nu }_i\sim 𝒩\left(0,{(\text{log}\sqrt{10})}^2\right)$$ (M.16)

with ν_i measured in Hz. On each time step, x_i is drawn from a Bernoulli distribution (so it is either 0 or 1),

$$P(x_i)={({\nu }_i\Delta t)}^{x_i}{(1-{\nu }_i\Delta t)}^{1-x_i}.$$ (M.17)

M2. Learning rules

Here we outline how a synapse can infer a probability distribution over its target weights. This is done using a well-understood class, hidden Markov models, for which we can use a standard, two-step procedure: in the first step the synapse incorporates new data using Bayes theorem; in the second step it take into account random changes in the target weight.

While straightforward in principle, in practice there are two difficulties with this approach. The first is that it results in a joint distribution over all synaptic weights. It is unlikely, however, that synapses could store such a distribution: even with a Gaussian approximation, for n synapses there are about n²/2 parameters. And it is even more unlikely that they could compute it, as that would require communication among synapses on different dendritic branches. We thus assume that each synapse performs probabilistic inference based only on the data available to it. This makes each synapse locally optimal, and it allows us to derive local learning rules. It is potentially the most important theoretical advance of our analysis. And within the Bayesian framework it is straightforward: each synapse simply integrates over the uncertainty in the target weights of all the other synapses. Nonetheless, this is an unusual approach, and further work is necessary to understand its theoretical properties.

The second difficulty is that even with the local approximation, inference is intractable, as it requires pointwise multiplication of probability distributions and a convolution (see Eqs. M.20 and M.21 below). To remedy this, we approximate the true distribution by a simpler one, a log-normal. The log-normal distribution was chosen for two reasons: it prevent synapses from changing sign, so Dale’s law is respected; and it matches the distribution of the target weights, Eq. (M.11), so it produces the correct distribution in the absence of presynaptic spikes.

M2.1. Single neuron learning rules: general formalism

The goal of a synapse is to compute the probability distribution over synaptic strength given data up to the last time step. Here the data - assumed local, as just discussed - consists of the feedback signal, f (shorthand for f_lin, f_cb or f_rl), the presynaptic input, x_i, and the actual weight, w_i. To reduce clutter, we use d_i(t) to denote the data at time t,

$$d_i(t)\equiv \left\{f(t),x_i(t),w_i(t)\right\},$$ (M.18)

and 𝓓_i(t) to denote past data,

$${𝒟}_i(t)\equiv \left\{d_i(t),d_i(t-1),d_i(t-2),\dots \right\}.$$ (M.19)

With this notation, the goal of the synapse is to compute P(λ_tar,i(t + 1)|𝓓_i(t)) in terms of P(λ_tar,i(t)|𝓓_i(t − 1)). To reduce clutter even further, here and in what follows all quantities without an explicitly specified time index are evaluated at time step t; thus, we will derive an update rule for P(λ_tar,i(t + 1)|𝓓_i) in terms of P(λ_tar,i|𝓓_i(t − 1)).

Making, as discussed above, the approximation that synapses perform inference based only on local information, the first step in the derivation of the update rule, incorporating new data using Bayes theorem, gives us

$$P\left({\lambda }_{\text{tar},i}\mid {𝒟}_i\right)=P\left({\lambda }_{\text{tar},i}\mid d_i,{𝒟}_i(t-1)\right)\propto P\left(d_i\mid {\lambda }_{\text{tar},i}\right)P\left({\lambda }_{\text{tar},i}\mid {𝒟}_i(t-1)\right)$$ (M.20)

where we used the Markov property: P(d_i|λ_tar,i, 𝓓_i(t − 1)) = P(d_i|λ_tar,i). (Recall that λ_tar,i is the log of the absolute value of the i^th target weight, w_tar,i; see Eq. M.10.) In the second step, the synapse takes into account random changes in the target weight,

$$P\left({\lambda }_{\text{tar},i}(t+1)\mid {𝒟}_i\right)=\int d{\lambda }_{\text{tar},i}P\left({\lambda }_{\text{tar},i}(t+1)\mid {\lambda }_{\text{tar},i}\right)P\left({\lambda }_{\text{tar},i}\mid {𝒟}_i\right).$$ (M.21)

The conditional distribution, P(λ_tar,i(t + 1)|λ_tar,i), can be extracted from Eq. (M.11). Combining both steps takes us from the distribution at time t, P(λ_tar,i|𝓓_i(t − 1)), to the distribution at the time t + 1, P (λ_tar,i(t + 1) |𝓓_i).

To make progress analytically, we approximate the true distribution by a log-normal one with mean m_i and variance $s_i^2$; that is, we assume that

$${\lambda }_{\text{tar},i}\mid {𝒟}_i(t-1)\sim 𝒩\left(m_i,s_i^2\right).$$ (M.22)

This is the quantity the synapse needs when it sets the actual weight, w_i. (Recall that quantities with no explicit time dependence are to be evaluated at time t; thus, the left hand side is the probability distribution over λ_tar,i(t) given data up to the previous time step.)

Finalizing the calculation requires two steps: 1) insert Eq. (M.22) into (M.20) and compute P(λ_tar,i|𝓓_i); 2) insert that into Eq. (M.21) and compute P(λ_tar,i(t + 1)|𝓓_i). However, Eq. (M.20) takes us out of our log-normal model class. To remedy this we use Assumed Density Filtering [52], for which posteriors are taken to be log-normal with mean and variance chosen to produce the distribution closest to the true one, where “close” is measured by the KL-divergence between the true and log-normal distributions. This can be achieved by matching moments: the mean and variance of the “closest” log-normal distribution are

$$m_i=𝔼\left[{\lambda }_{\text{tar},i}\mid {𝒟}_i(t-1)\right]$$ (M.23a)

$$s_i^2=\text{Var}\left[{\lambda }_{\text{tar},i}\mid {𝒟}_i(t-1)\right].$$ (M.23b)

We will apply this first to Eq. (M.20). Taking the log of both sides of that equation gives

$$\text{log}P\left({\lambda }_{\text{tar},i}\mid {𝒟}_i\right)=L\left({\lambda }_{\text{tar},i}\right)+\text{log}P\left({\lambda }_{\text{tar},i}\mid {𝒟}_i(t-1)\right)+\text{const}$$ (M.24)

where

$$L\left({\lambda }_{\text{tar},i}\right)\equiv \text{log}P\left(d_i\mid {\lambda }_{\text{tar},i}\right)$$ (M.25)

is the log likelihood of the data at time t given the target weight; we suppress the dependence on d_i to avoid clutter. Under our log-normal assumption, the second term on the right hand side of Eq. (M.24) is Gaussian in λ_tar,i. Motivated by the fact that new data does not provide much information, we assume that the likelihood is a slowly varying function of the target weights. This allows us to make a Laplace approximation: we Taylor expand the log likelihood around m_i, the mean of P(λ_tar,i|𝓓_i(t − 1)), and work only to second order in λ_tar,i − m_i. Also using Eq. (M.22), we have

$$\text{log}P\left({\lambda }_{\text{tar},i}\mid {𝒟}_i\right)=L^{\prime }\left(m_i\right)\left({\lambda }_{\text{tar},i}-m_i\right)+L^{″}\left(m_i\right)\frac{{\left({\lambda }_{\text{tar},i}-m_i\right)}^2}{2}-\frac{{\left({\lambda }_{\text{tar},i}-m_i\right)}^2}{2s_i^2}+\text{const}.$$ (M.26)

The right hand side is now quadratic in λ_tar,i. Consequently, P(λ_tar,i|𝓓_i) is Gaussian, with mean and variance given by

$$𝔼\left[{\lambda }_{\text{tar},i}\mid {𝒟}_i\right]=m_i+\frac{s_i^2{L}^{\prime }\left(m_i\right)}{1-s_i^2{L}^{″}\left(m_i\right)}\approx m_i+s_i^2{L}^{\prime }\left(m_i\right)$$ (M.27a)

$$\text{Var}\left[{\lambda }_{\text{tar},i}\mid {𝒟}_i\right]=s_i^2+\frac{s_i^4{L}^{″}\left(m_i\right)}{1-s_i^2{L}^{″}\left(m_i\right)}\approx s_i^2+s_i^4{L}^{″}\left(m_i\right).$$ (M.27b)

To derive the approximation expressions, we assumed $s_i^2\left|L^{″}\left(m_i\right)\right|\ll 1$. This holds in the limit of slowly varying log likelihood, which we assume throughout our analysis.

Equation (M.27) tells us how to incorporate new data; we now need to incorporate random drift, via the integral in Eq. (M.21). From Eq. (M.11), we see that P(λ_tar,i(t + 1)|λ_tar,i) is Gaussian, so the integral is straightforward, and we have

$$m_i(t+1)=\left(1-\frac{1}{\tau }\right)𝔼\left[{\lambda }_{\text{tar},i}\mid {𝒟}_i\right]+\frac{m_{\text{prior}}}{\tau }$$ (M.28a)

$$s_i^2(t+1)={\left(1-\frac{1}{\tau }\right)}^2\text{Var}\left[{\lambda }_{\text{tar},i}\mid {𝒟}_i\right]+\frac{2s_{\text{prior}}^2}{\tau }.$$ (M.28b)

Inserting Eq. (M.27) into (M.28), and working to lowest nonvanishing order in 1/τ, s_iL’(m_i) and $s_i^2{L}^{″}\left(m_i\right)$, we arrive at our final update equations,

$$\Delta m_i=s_i^2{L}^{\prime }\left(m_i\right)-\frac{m_i-m_{\text{prior}}}{\tau }$$ (M.29a)

$$\Delta s_i^2=s_i^4{L}^{″}\left(m_i\right)-\frac{2\left(s_i^2-s_{\text{prior}}^2\right)}{\tau }$$ (M.29b)

where Δm_i ≡ m_i(t + 1) − m_i and $\Delta s_i^2\equiv s_i^2(t+1)-s_i^2$. Thus, to update the mean and variance, all we have to do is compute the log likelihood and take the first and second derivatives. In Sec. M2.2 below we sketch how to do that; additional details are provided in Supplementary Math Note, Sec. S1. Note that equality in these expressions (and many that follow) is shorthand for equality under the assumptions and approximations of our model.

M2.2. Single neuron learning rules for our three models

According to the above analysis (see in particular Eq. M.29), to determine the update rules we just need the log likelihood of the current data, d_i(t), given the error signal (either f_lin, f_cb or f_rl). Computing it is nontrivial, as several approximations are required. However, it is not hard to get an intuitive understanding of its form.

Using Eq. (M.18) for the data, d_i, the likelihood - the probability of the data given w_tar,i - may be written

$$P\left(d_i\mid {\lambda }_{\text{tar},i}\right)=P\left(f\mid x_i,w_i,{\lambda }_{\text{tar},i}\right)P\left(x_i,w_i\mid {\lambda }_{\text{tar},i}\right)\propto P\left(f\mid x_i,w_i,{\lambda }_{\text{tar},i}\right),$$ (M.30)

where we are able to drop the term P(x_i, w_i|λ_tar,i) because without an error signal, x_i and w_i do not provide any information about λ_tar,i. For all of our feedback signals, f is a function of f_lin (see main text); we take advantage of this to write

$$P\left(f\mid x_i,w_i,{\lambda }_{\text{tar},i}\right)=\int d{f}_{\text{lin}}P\left(f\mid f_{\text{lin}}\right)P\left(f_{\text{lin}}\mid x_i,w_i,{\lambda }_{\text{tar},i}\right).$$ (M.31)

We focus here on computing P(f_lin|x_i, w_i, λ_tar,i), and leave the integral for Supplementary Math Note, Sec. S1.1. Using Eqs. (1), (2) and (5) from the main text, we have

$$f_{\text{lin}}=\left(w_{\text{tar},i}-w_i\right)x_i+\sum _{j\ne i}\left(w_{\text{tar},j}-w_j\right)x_j+{\xi }_V+{\xi }_{\delta }.$$ (M.32)

For synapse i, all the terms in the sum over j are unobserved, and so correspond to noise. By the Central Limit Theorem (and the assumed independence of the synapses), that noise is Gaussian; we take the added noise, ξ_V and ξ_δ, to be Gaussian as well, with total variance

$${\sigma }_0^2\equiv \text{Var}\left[{\xi }_{\delta }\right]+\text{Var}\left[{\xi }_V\right].$$ (M.33)

Consequently, we may write

$$f_{\text{lin}}\mid w_i,x_i,{\lambda }_{\text{tar},i}\sim 𝒩\left(\left(w_{\text{tar},i}-w_i\right)x_i,{\sigma }_{\delta ,i}^2\right)$$ (M.34)

where

$${\sigma }_{\delta ,i}^2\equiv \text{Var}\left[\sum _{j\ne i}\left(w_{\text{tar},j}-w_j\right)x_j\mid {𝒟}_i(t-1)\right]+{\sigma }_0^2.$$ (M.35)

The quantity ${\sigma }_{\delta ,i}^2$ depends on synapse, i. However, in the limit that there are a large number of synapses, that dependence is weak. We thus approximate it by including all terms in the sum over j, which we denote ${\sigma }_{\delta }^2$,

$${\sigma }_{\delta ,i}^2\approx {\sigma }_{\delta }^2\equiv \text{Var}\left[\sum _j\left(w_{\text{tar},i}-w_j\right)x_j\mid {𝒟}_i(t-1)\right]+{\sigma }_0^2.$$ (M.36)

Under this approximation,

$$f_{\text{lin}}\mid w_i,x_i,{\lambda }_{\text{tar},i}\sim 𝒩\left(\left(\pm e^{{\lambda }_{\text{tar},i}}-w_i\right)x_i,{\sigma }_{\delta }^2\right).$$ (M.37)

For much of our analysis we use the value of ${\sigma }_{\delta }^2$ under the prior. That quantity, denoted ${\sigma }_{\delta 0}^2$ given by

$${\sigma }_{\delta 0}^2\equiv \left({\sigma }_{\text{prior}}^2+{\sigma }_{w,\text{prior}}^2\right)\sum _j{\nu }_j\Delta t\left(1-{\nu }_j\Delta t\right)+{\sigma }_0^2$$ (M.38)

where the term ν_jΔt(1 − ν_jΔt) comes from the Bernoulli statistics of x_j (see Eq. M.17), and ${\sigma }_{\text{prior}}^2$ and ${\sigma }_{w,\text{prior}}^2$ are the variances of w_tar,i and w_i under the prior. The latter, ${\sigma }_{w,\text{prior}}^2$, depends on whether or not we are sampling,

$${\sigma }_{w,\text{prior}}^2\equiv \{\begin{array}{ll}{\sigma }_{\text{prior}}^2\hfill & \text{Synaptic}\text{Sampling}\hfill \\ k{\mu }_{\text{prior}}\hfill & \text{variance}\text{proportional}\text{to}\text{the}\text{mean}(\text{Sec}.\text{M1}.2).\hfill \end{array}$$ (M.39)

The prior mean and variance of the weights in terms of the log weights (the quantities we have access to; see Table 1 below) are given by Eq. (M.13),

$${\mu }_{\text{prior}}\equiv e^{m_{\text{prior}}+s_{\text{prior}}^2}/2$$ (M.40a)

$${\sigma }_{\text{prior}}^2\equiv {\mu }_{\text{prior}}^2\left[e^{s_{\text{prior}}^2}-1\right].$$ (M.40b)

This analysis tells us that the distribution P(f_lin|w_i, x_i, λ_tar,i) is Gaussian in e^λ_tar,i. To determine the learning rules, all we have to do is insert Eq. (M.37) into Eq. (M.31), perform an integral, take the log, compute the first two derivatives, and evaluate them at m_i (see Eq. M.29). These steps, which are performed in Supplementary Math Note, Sec. S1.1, are not completely straightforward, as various approximations must be made. However, from a conceptual point of view the approximations do not add much. Thus, here we simply give the results.

Linear feedback, f = f_lin

The Bayesian update rules are

$$\Delta m_i=\left(\frac{s_i^2{\mu }_i}{{\sigma }_{\delta 0}^2}\right)x_i{f}_{\text{lin}}-\frac{1}{\tau }\left(m_i-m_{\text{prior}}\right)$$ (M.41a)

$$\Delta s_i^2=-\left(\frac{s_i^4{\mu }_i^2}{{\sigma }_{\delta 0}^2}\right)x_i^2-\frac{2}{\tau }\left(s_i^2-s_{\text{prior}}^2\right).$$ (M.41b)

For classical learning, we use the delta rule (main text, Eq. 3),

$$\Delta w_i=\eta x_i{f}_{\text{lin}}.$$ (M.42)

Note that we are not including weight drift in the classical learning rate (both here and below), as weight drift was derived using Bayesian analysis, and has no classical counterpart.

Cerebellar feedback, f = f_cb = Θ(f_lin − θ)

The Bayesian update rules are

$$\Delta m_i=\left(\frac{s_i^2{\mu }_i}{{\sigma }_{\delta 0}^2}\right)x_i{\sigma }_{\delta 0}\left(2f_{\text{cb}}-1\right)\frac{𝒩\left({\theta }_{\text{cb}}\right)}{\text{Φ}\left({\theta }_{\text{cb}}\right)}-\frac{1}{\tau }\left(m_i-m_{\text{prior}}\right)$$ (M.43a)

$$\Delta s_i^2=-\left(\frac{s_i^4{\mu }_i^2}{{\sigma }_{\delta 0}^2}\right)x_i^2\frac{𝒩\left({\theta }_{\text{cb}}\right)}{\text{Φ}\left({\theta }_{\text{cb}}\right)}\left[{\theta }_{\text{cb}}+\frac{𝒩\left({\theta }_{\text{cb}}\right)}{\text{Φ}\left({\theta }_{\text{cb}}\right)}\right]-\frac{2}{\tau }\left(s_i^2-s_{\text{prior}}^2\right)$$ (M.43b)

where Φ and (in a slight abuse of notation) 𝒩 are the cumulative normal and normal functions, respectively,

$$\text{Φ}(z)\equiv {\int }_{-\infty }^{z}du\frac{e^{-u^2/2}}{{(2\pi )}^{1/2}}$$ (M.44a)

$$𝒩(z)\equiv \frac{e^{-z^2/2}}{{(2\pi )}^{1/2}}$$ (M.44b)

and θ_cb is given in terms of the threshold, θ, as

$${\theta }_{\text{cb}}\equiv \left(1-2f_{\text{cb}}\right)\frac{\theta }{{\sigma }_{\delta 0}}.$$ (M.45)

For classical learning, we absorb most of the prefactor in the above mean update into a fixed learning rate,

$$\Delta w_i=\eta \left(2f_{\text{cb}}-1\right)x_i\frac{𝒩\left({\theta }_{\text{cb}}\right)}{\text{Φ}\left({\theta }_{\text{cb}}\right)}.$$ (M.46)

Reinforcement learning, f = f_rl = − |f_lin|

The Bayesian update rules are

$$\Delta m_i=\left(\frac{s_i^2{\mu }_i}{{\sigma }_{\delta }^2}\right)\left(\frac{f_{\text{rl}}^2}{{\sigma }_{\delta }^2}-1\right)x_i^2\left({\mu }_i-w_i\right)-\frac{1}{\tau }\left(m_i-m_{\text{prior}}\right)$$ (M.47a)

$$\Delta s_i^2=-\left(\frac{s_i^4{\mu }_i^2}{{\sigma }_{\delta }^2}\right)\left(1-\frac{f_{\text{rl}}^2}{{\sigma }_{\delta }^2}\right)x_i^2-\frac{2}{\tau }\left(s_i^2-s_{\text{prior}}^2\right).$$ (M.47b)

This learning rule appears non-local, as it depends on ${\sigma }_{\delta }^2$, which in turn depends on all the synapses (Eq. M.36). However, we make it local by changing the feedback signal to $\left(1-f_{\text{rl}}^2/{\sigma }_{\delta }^2\right)/{\sigma }_{\delta }^2$. For classical learning, we again absorb most of the prefactor into the learning rate,

$$\Delta w_i=\eta x_i\left(f_{\text{rl}}\tanh \left(\left({\mu }_i-w_i\right)x_i{f}_{\text{rl}}/{\sigma }_{\delta }^2\right)-\left({\mu }_i-w_i\right)x_i\right).$$ (M.48)

Note that tanh appears in the classical, but not Bayesian, learning rules. That is because for the Bayesian learning rules we made the approximation tanh $\left(\left({\mu }_i-w_i\right)x_i{f}_{\text{rl}}/{\sigma }_{\delta }^2\right)\approx \left({\mu }_i-\right)\left)w_i\right)x_i{f}_{\text{rl}}/{\sigma }_{\delta }^2$. This approximation, however, made the classical learning rule unstable.

M2.3. Recurrent neural network learning rules

So far we have focused on single neurons. Here we generalize to the more realistic scenario in which the output weights of a recurrent network are trained to produce a time-dependent target function. We will assume that the network, which contains N neurons, evolves according to

$${\tau }_m\frac{d{v}_i}{dt}=-v_i+\sum _{j=1}^N{J}_{ij}{x}_j+A_{i}V(t)+I_i(t)$$ (M.49a)

$$x_j=\tanh \left(v_j\right)$$ (M.49b)

$$V(t)=\sum _{j=1}^N{w}_j{x}_j.$$ (M.49c)

We interpret v_i as the membrane potential and x_j as the firing rate relative to baseline. The recurrent weights, J_ij, and feedback weights, A_i, are fixed. Parameters of the network, and details of the simulations, are given in Supplementary Math Note, Sec. S3.

The goal of the network is to minimize the distance between V(t) and some target function, denoted V_tar(t); that is, to minimize the error, δ(t), defined, as in Eq. (2), to be

$$\delta (t)\equiv V_{\text{tar}}(t)-V(t).$$ (M.50)

As with single neurons, we take a Bayesian approach. There are, however, two important differences. The first is that we do not know the target weights (we do not specify them; instead they must be learned). We assume, though, that target weights exist, which means we can write

$$\delta (t)=\sum _j\left(w_{\text{tar},j}(t)-w_j(t)\right)x_j(t).$$ (M.51)

The second difference is that the feedback signal, δ(t), is a continuous function of time. Consequently, information at times t and t + dt is largely redundant. To deal with this redundancy we make several approximations. First, rather than updating the weights continuously, we update them at times separated by Δt. (In a slight abuse of notation, here Δt does not have the same numerical value as in the single neuron update rules.) Bayes’ theorem, Eq. (M.20), then becomes

$$P\left(w_{\text{tar},i}\mid {𝒟}_i\right)\propto P\left(d_i\mid w_{\text{tar},i},{𝒟}_i(t-\Delta t)\right)P\left(w_{\text{tar},i}\mid {𝒟}_i(t-\Delta t)\right)$$ (M.52)

where, as in the single neuron case, the data for synapse i is the presynaptic input, x_i, the actual weight, w_i, and the error signal, δ. To derive this expression we made two simplifications: we did not add noise to the error signal, so the synapses see δ rather than f_lin, and we did not enforce Dale’s law, so the weights can change sign. Because of the latter simplification, we let the weights, rather than the log weights, have a Gaussian distribution; that is why Eq. (M.52) is written in terms of w_tar,i rather than λ_tar,i.

In one respect the analysis is simpler than it was for single neurons: because the target weights do not evolve over time (see comments at the end of Sec. M1.1), we can avoid the integral in Eq. (M.21). However, in another respect it is more complicated: as just discussed, the likelihood (the first term on the right hand side of Eq. M.52) depends on past data. An exact treatment in this regime is beyond the scope of this work. What we do instead is choose the time step, Δt, so it is much larger than the correlation time of δ(t). This allows us to drop the dependence on 𝓓_i(t − Δt) in the likelihood, giving us

$$P\left(d_i\mid w_{\text{tar},i},{𝒟}_i(t-\Delta t)\right)\approx P\left(d_i\mid w_{\text{tar},i}\right)\propto P\left(\delta \mid x_i,w_i,w_{\text{tar},i}\right)$$ (M.53)

where, as in Eq. (M.30), we used the fact that without an error signal, x_i and w_i do not provide any information about w_tar,i.

While this gives us a very good approximation to the likelihood if Δt is large, large Δt means that updates would be made very rarely, and so learning would be slow. We thus make a second approximation, which is to optimize our learning rule (via numerical simulation, as discussed below) with respect to Δt. This gives us approximate Bayesian update rules, which presumably could be improved upon. However, as we will see, the approximate update rules already outperform the classical ones by an order of magnitude. Thus, any improvement would only make the case for Bayesian plasticity stronger.

To find an expression for P(δ|x_i, w_i, w_tar,i), we again write δ as in Eq. (M.32) (but without noise, so ξ_δ = 0, which reduces ${\sigma }_0^2$; see Eq. M.33). Now, however, we are interested in the log likelihood with respect to the target weights, w_tar,i, rather than the log of the target weights, λ_tar,i (as mentioned above). Thus, the distribution over δ simplifies relative to Eq. (M.37),

$$\delta \mid w_i,x_i,w_{\text{tar},i}\sim 𝒩\left(\left(w_{\text{tar},i}-w_i\right)x_i,{\sigma }_{\delta }^2\right).$$ (M.54)

As above, we made the approximation ${\sigma }_{\delta ,i}^2\approx {\sigma }_{\delta }^2$; see Eq. (M.36). It is now straightforward to write down the log likelihood,

$$L\left(w_{\text{tar},i}\right)=-\frac{{\left(\delta -\left(w_{\text{tar},i}-w_i\right)x_i\right)}^2}{2{\sigma }_{\delta }^2}+\text{const}.$$ (M.55)

The first and second derivatives evaluated at w_tar,i = μ_i are

$$L^{\prime }\left({\mu }_i\right)=\frac{\left(\delta -\left({\mu }_i-w_i\right)x_i\right)x_i}{{\sigma }_{\delta }^2}\approx \frac{\delta x_i}{{\sigma }_{\delta }^2}$$ (M.56a)

$$L^{″}\left({\mu }_i\right)=-\frac{x_i^2}{{\sigma }_{\delta }^2}.$$ (M.56b)

(We are justified in dropping the term (μ_i − w_i)x_i because it is a factor of $\sqrt{n}$ smaller than δ. That follows because ${\sigma }_{\delta }^2$, which is the variance of δ, is $𝒪(n)$; see Eq. M.36.) Inserting these expressions into Eq. (M.29) (with τ taken to ∞ because, as discussed in Sec. M1.1, we are assuming the target weights do not drift over time), we have

$$\Delta {\mu }_i=\frac{{\sigma }_i^2}{{\sigma }_{\delta }^2}\delta x_i$$ (M.57a)

$$\Delta {\sigma }_i^2=-\frac{{\sigma }_i^4}{{\sigma }_{\delta }^2}{x}_i^2,$$ (M.57b)

where Δμ_i = μ_i(t + Δt) − μ_i(t) and similarly for $\Delta {\sigma }_i^2$.

Primarily for convenience, we make a third approximation, which is to update the weights continuously rather than at discrete points separated by Δt. To do that we simply make the approximation Δμ_i ≈ Δtdμ_i/dt, and similarly for $\Delta {\sigma }_i^2$. This allows us to turn the update rules into ordinary differential equations,

$$\frac{d{\mu }_i}{dt}=\frac{1}{\Delta t}\frac{{\sigma }_i^2}{{\sigma }_{\delta }^2}\delta x_i$$ (M.58a)

$$\frac{d{\sigma }_i^2}{dt}=-\frac{1}{\Delta t}\frac{{\sigma }_i^4}{{\sigma }_{\delta }^2}{x}_i^2.$$ (M.58b)

Then, defining

$${\eta }_i\equiv \frac{{\sigma }_i^2}{{\sigma }_{\delta }^2\Delta t},$$ (M.59)

inserting this into into Eq. (M.58) and, in our fourth approximation, ignoring the time dependence in ${\sigma }_{\delta }^2$, those equations simplify to

$$\frac{d{\mu }_i}{dt}={\eta }_i\delta x_i$$ (M.60a)

$$\frac{d{\eta }_i}{dt}=-{\eta }_i^2{x}_i^2.$$ (M.60b)

Optimizing over Δt corresponds to optimizing over the initial value of η_i, which we assumed is the same for all i. This optimization is done via numerical simulations.

For the classical learning rules, we drop Eq. (M.60b) and fix η_i to the same value for all synapses.

Further methods details can be found in the attached Life Sciences Reporting Summary.

Code Availability

Code and data are available for download at: https://github.com/Jegmi/the-bayesian-synapse/releases/tag/v2

Data Availability

Code and data are available for download at: https://github.com/Jegmi/the-bayesian-synapse/releases/tag/v2