Skip to main content
PMC home page
eLife logoLink to eLife
. 2015 Oct 30;4:e09685. doi: 10.7554/eLife.09685

Retroactive modulation of spike timing-dependent plasticity by dopamine

Zuzanna Brzosko 1, Wolfram Schultz 1, Ole Paulsen 1,*
Editor: Marlene Bartos2
PMCID: PMC4626806  PMID: 26516682

Abstract

Most reinforcement learning models assume that the reward signal arrives after the activity that led to the reward, placing constraints on the possible underlying cellular mechanisms. Here we show that dopamine, a positive reinforcement signal, can retroactively convert hippocampal timing-dependent synaptic depression into potentiation. This effect requires functional NMDA receptors and is mediated in part through the activation of the cAMP/PKA cascade. Collectively, our results support the idea that reward-related signaling can act on a pre-established synaptic eligibility trace, thereby associating specific experiences with behaviorally distant, rewarding outcomes. This finding identifies a biologically plausible mechanism for solving the ‘distal reward problem’.

DOI: http://dx.doi.org/10.7554/eLife.09685.001

Research organism: mouse

eLife digest

To help someone learn a new task, we might give them a reward after they have performed well. However, these rewards tend to be given several seconds or minutes after the behavior they are supposed to promote. Therefore, it is unclear how the rewards affect the brain and help accelerate the learning process.

Information is processed and sent around the brain by networks of cells called neurons. These networks are constantly remodeled because learning changes the connections—called synapses—that neighboring neurons signal across. Synapses can be strengthened so that signals are sent across them more easily in the future. Synapses can also be weakened, making it harder for the neurons to subsequently communicate.

A chemical called dopamine is often produced in the brain when a reward is received. If dopamine is present in a synapse whilst a neuron is signaling to its neighbor, it can affect how effectively this communication occurs. Brzosko et al. have now investigated whether dopamine can also change the synapses if it is applied after signaling has already happened.

The strengthening or weakening of synapses can be triggered by electrically stimulating the neurons on either side of a synapse at particular times. Brzosko et al. did this to neurons in slices of mouse brain, and then applied dopamine to the neurons. The results suggest that dopamine can reverse synaptic weakening and can even cause the synapses to strengthen. However, the dopamine had to be applied immediately after stimulation to be able to strengthen the synapse. The next challenge is to establish whether this change in synaptic strength is responsible for the change in behavior.

DOI: http://dx.doi.org/10.7554/eLife.09685.002

Introduction

Spike timing-dependent plasticity (STDP) is a physiologically relevant form of Hebbian learning (Caporale and Dan, 2008). In its classic form, STDP depends on the order and precise timing of presynaptic and postsynaptic spikes: pre-before-post spike pairings induce timing-dependent long-term potentiation (t-LTP), whereas post-before-pre pairings induce timing-dependent long-term depression (t-LTD) (Markram et al., 1997; Bi and Poo, 1998). However, the quantitative rules of STDP are profoundly influenced by neuromodulators (Seol et al., 2007; Pawlak et al., 2010), including dopamine (DA) (Zhang et al., 2009; Edelmann and Lessmann, 2011; Yang and Dani, 2014). Although it is well established that reward-motivated behavior depends on the activity of DA neurons (Schultz et al., 1997; Suri and Schultz, 1999; Pan et al., 2005), the mechanisms that associate specific experiences with rewarding outcomes, which typically occur after a delay, are not well understood. This is referred to as the distal reward problem (Hull, 1943). To address this problem, here, we examined whether DA modulates STDP not only when applied during, but also—more importantly—when applied after the pairing event.

Results and discussion

We first sought to corroborate the shape of the STDP induction curve by varying the time interval between the presynaptic and postsynaptic activity (Δt; Figure 1A,B,G). To this end, we monitored excitatory postsynaptic potentials (EPSPs) that were evoked by extracellular stimulation of the Schaffer-collateral-CA1 pathway during whole-cell recordings of CA1 pyramidal cells in mouse horizontal slices (postnatal days 12–18; ‘Materials and methods’). Plasticity was induced in current clamp mode using an induction protocol that involved 100 pairings of a single EPSP followed by a single postsynaptic spike (t-LTP; Figure 1A) or a single postsynaptic spike followed by a single EPSP (t-LTD; Figure 1B) at 0.2 Hz. Consistent with previous studies (Bi and Poo, 1998; Zhang et al., 2009; Edelmann and Lessmann, 2011), the pre-before-post pairing protocol with Δt = +10 ms induced t-LTP (182 ± 14%; t(4) = 5.6, p = 0.0049 vs 100%, n = 5; Figure 1A) and the post-before-pre pairing protocol with Δt = −20 ms induced t-LTD (61 ± 9%; t(4) = 4.4, p = 0.0121 vs 100%, n = 5; Figure 1B). Surprisingly, however, we found that the post-before-pre pairing protocol with Δt = −10 ms instead of inducing t-LTD, elicited robust t-LTP (202 ± 21%; t(6) = 5.0, p = 0.0025 vs 100%, n = 7; Figure 1C,D,G). This conflicts with previous reports from hippocampal cultures (Bi and Poo, 1998; Zhang et al., 2009) and acute slices (Edelmann and Lessmann, 2011; Yang and Dani, 2014), where post-before-pre pairing protocols never elicited synaptic potentiation in baseline conditions. Given that DA has been found to widen the time window for the induction of t-LTP (Zhang et al., 2009; Yang and Dani, 2014), we wanted to assess whether endogenous DA could be responsible for the potentiation observed with the post-before-pre pairing under our experimental conditions. Therefore, we repeated this set of experiments using the post-before-pre pairing protocol with Δt = −10 ms in the presence of DA receptor (DAR) antagonists. Indeed, combined application of the D1/D5 receptor antagonist SCH23390 (10 μM) and D2-like receptor antagonist sulpiride (50 μM) from the start of the recordings prevented t-LTP and enabled t-LTD instead (72 ± 8%; t(5) = 3.6, p = 0.0160 vs 100%, n = 6; Figure 1C,D,G), rendering the STDP induction curve similar to that observed in hippocampal cultures (Bi and Poo, 1998; Zhang et al., 2009). These results suggest a modulatory action of endogenous DA, presumably released during the pairing protocol (Frey et al., 1990; Yang and Dani, 2014), which resulted in the changed polarity of plasticity at narrow negative spike-timing intervals.

Figure 1. Dopamine widens the time window for induction of t-LTP.

Figure 1.

Open in a new tab

Example plots of normalized EPSP slopes demonstrating that (A) pre-before-post pairing protocol with Δt = +10 ms induced t-LTP, whereas (B) post-before-pre pairing protocol with Δt = −20 ms induced t-LTD. Insets, pairing protocols. Traces show an excitatory postsynaptic potential (EPSP) before (1) and 40 min after (2) pairing. (C) Endogenous DA widens the spike time window for induction of t-LTP. In the absence of DA receptor antagonists, the post-before-pre pairing protocol with Δt = −10 ms induced t-LTP (red), whereas application of SCH23390 and sulpiride at the start of the recordings prevented this t-LTP and enabled t-LTD instead (black). Traces are presented as in A. (D) Summary of results. (E) Exogenous DA widens the spike time window for induction of t-LTP. In the presence of 20 μM DA, the post-before-pre pairing protocol with Δt = −20 ms induced t-LTP (red), whereas in control condition the same pairing protocol induced t-LTD (black). Traces are presented as in A. (F) Summary of results. (G) Summary of the spike timing-dependent plasticity (STDP) induction with various spike-timing intervals (Δt in ms) in control condition (green; data point at −20 ms represents data combined from B and E), in the presence of DA receptor antagonists (SCH23390 and sulpiride; black), or DA (red). Each data point is the group average EPSP slope percentage change from baseline. Error bars represent s.e.m. Significant difference (*p < 0.05, **p < 0.01, ***p < 0.001) compared with the baseline (one-sample two-tailed Student's t-test) or between the indicated two groups (paired two-tailed Student's t-test). The numbers of cells are shown in parentheses.

DOI: http://dx.doi.org/10.7554/eLife.09685.003

Figure 1—source data 1. Source data for Figure 1.
DOI: http://dx.doi.org/10.7554/eLife.09685.004
elife09685s001.xlsx (100.6KB, xlsx)
DOI: 10.7554/eLife.09685.004

Next, we wanted to examine whether the application of exogenous DA during pairing at negative spike-timing intervals facilitates synaptic potentiation. Indeed, the post-before-pre pairing protocol with Δt = −20 ms, which elicited robust t-LTD in control condition (74 ± 9%, t(8) = 3.0, p = 0.0165 vs 100%, n = 9; Figure 1E–G), induced significant t-LTP when exogenous DA (20 μM) was bath-applied for 10–12 min from 2 min before and during the post-before-pre pairings in interleaved experiments (144 ± 12%; t(5) = 3.7, p = 0.0148 vs 100%, n = 6; Figure 1E–G). Therefore, in accordance with previous findings (Zhang et al., 2009; Yang and Dani, 2014), the presence of DA during the coordinated spiking activity widens the spike time interval for induction of t-LTP.

A crucial aspect of reinforcement learning models is the ability of the reinforcing signal (DA) to strengthen active synapses, even when it arrives after the activity (Sutton and Barto, 1981; Izhikevich, 2007). To test this hypothesis experimentally, we applied DA after the t-LTD induction protocol. Exogenous DA (100 μM) added to the perfusion system for 10–12 min starting within 1 min after the post-before-pre pairing protocol with Δt = −20 ms converted t-LTD into t-LTP (169 ± 16%, t(5) = 4.3, p = 0.0078 vs 100%, n = 6; Figure 2A,B). This implies that DA can have a retroactive effect allowing negative spike pairings to induce t-LTP. The specificity of this DAergic conversion of STDP was assessed using DAR antagonists. Indeed, combined application of DAR antagonists, SCH23390 (10 μM) and sulpiride (50 μM), prevented DA-induced conversion of t-LTD into t-LTP, resulting in significant t-LTD instead (63 ± 12%, t(5) = 3.0, p = 0.0289 vs 100%, n = 6, Δt = −20 ms). Thus, the conversion of t-LTD into t-LTP was due to specific DAR activation. Importantly, when the test pathway was not stimulated following the pairing protocol until after DA washout (stimulation resumed 15 min after pairing), robust t-LTD was induced (61 ± 5%, t(5) = 8.1, p = 0.0005 vs 100%, n = 6; Figure 2A,B). The effect of DA was, therefore, activity dependent, demonstrating that the reinforcing signal is capable of acting specifically on the active inputs. To exclude the possibility that DA by itself could potentiate the test pathway, control experiments with ongoing synaptic stimulation over 60 min at 0.2 Hz, but without pairing with postsynaptic action potentials, were performed. Consistent with earlier reports (Otmakhova and Lisman, 1999), DA had no significant effect on the basal Schaffer-collateral transmission (110 ± 7%, t(7) = 1.4, p = 0.2169 vs 100%, n = 8; Figure 2A,B).

Figure 2. Dopamine retroactively converts t-LTD into t-LTP.

Figure 2.

Open in a new tab

(A) DA applied immediately after the pairing protocol with Δt = −20 ms converted t-LTD into t-LTP (+Pairing +Stim; red). If the pathway was not stimulated following the pairing protocol until after DA washout, t-LTD was induced (+Pairing −Stim; black). In the absence of the pairing protocol, DA had no effect on baseline EPSPs (−Pairing +Stim; blue). Traces show an EPSP before (1) and 40 min after (2) pairing. (B) Summary of results. Error bars represent s.e.m. Significant difference (**p < 0.01, ***p < 0.001) compared with the baseline (one-sample two-tailed Student's t-test). The number of cells is shown in parentheses.

DOI: http://dx.doi.org/10.7554/eLife.09685.005

Figure 2—source data 1. Source data for Figure 2.
DOI: http://dx.doi.org/10.7554/eLife.09685.006
elife09685s002.xlsx (58.4KB, xlsx)
DOI: 10.7554/eLife.09685.006

Subsequently, we wanted to determine whether the observed DA-induced conversion of t-LTD into t-LTP depends on the timing of DA application following the post-before-pre protocol (Figure 3A–D). We found that delayed application of DA (10 or 30 min after t-LTD pairing protocol) failed to convert t-LTD into t-LTP. Application of DA 10 min after the post-before-pre pairing caused a reversal of t-LTD back to baseline (94 ± 9%, t(11) = 0.7, p = 0.5230 vs 100%, n = 12; Figure 3B,D), whereas application of DA 30 min after the post-before-pre pairing failed to influence t-LTD altogether (59 ± 12%, t(5) = 3.4, p = 0.0200 vs 100%, n = 6; Figure 3C,D). Whilst it has previously been reported that DA applied after the induction of low frequency stimulation-induced LTD can reduce the magnitude of synaptic depression (Mockett et al., 2007), our data demonstrate, for the first time to our knowledge, that DA can change the polarity of STDP when acting within a short time window following the induction protocol.

Figure 3. Time dependence of the DA-induced conversion of t-LTD into t-LTP.

Figure 3.

Open in a new tab

(A) DA applied immediately after the pairing protocol with Δt = −20 ms converted t-LTD into t-LTP, whereas delayed application of DA failed to convert t-LTD into t-LTP and either (B) resulted in a reversal of t-LTD back to baseline (10 min after pairing) or (C) failed to affect t-LTD altogether (30 min after pairing). Traces show an EPSP before (1) and 40 min (A, B) or 60 min (C) after pairing (2). (D) Summary of results. Error bars represent s.e.m. Significant difference (*p < 0.05, **p < 0.01) compared with the baseline (one-sample two-tailed Student's t-test). The number of cells is shown in parentheses.

DOI: http://dx.doi.org/10.7554/eLife.09685.007

Figure 3—source data 1. Source data for Figure 3.
DOI: http://dx.doi.org/10.7554/eLife.09685.008
elife09685s003.xlsx (63.4KB, xlsx)
DOI: 10.7554/eLife.09685.008

Finally, we aimed to explore the possible mechanisms underlying the DA-induced conversion of t-LTD into t-LTP. Both hippocampal t-LTP and t-LTD (Edelmann and Lessmann, 2011; Yang and Dani, 2014), as well as the modulation of STDP by DA (Zhang et al., 2009), require functional NMDA receptors. We, therefore, asked whether the DA-induced conversion of t-LTD into t-LTP is also NMDA receptor dependent. Application of the NMDA receptor antagonist d-2-amino-5-phosphonopentanoic acid (d-AP5, 50 μM) after the post-before-pre pairing protocol did not by itself affect the development of t-LTD (57 ± 12%, t(5) = 3.6, p = 0.0156 vs 100%, n = 6, Δt = −20 ms; Figure 4A,D). Nevertheless, DA in the presence of d-AP5 reversed t-LTD back to baseline albeit failing to convert t-LTD into t-LTP (105 ± 12%, t(5) = 0.4, p = 0.6898 vs 100%, n = 6, Δt = −20 ms; Figure 4A,D). This suggests an important dissociation between two mechanisms involved in the DA-induced conversion of t-LTD into t-LTD, namely a reversal of synaptic depression (de-depression) and synaptic potentiation, one of which is NMDA receptor dependent.

Figure 4. Cellular mechanisms involved in the DA-induced conversion of t-LTD into t-LTP.

(A) DA applied immediately after the pairing protocol (Δt = −20 ms) converted t-LTD into t-LTP (red; data combined from Figures 2A, 3A). This effect requires NMDA receptors as application of d-AP5 1 min before the DA application partially blocked the conversion of t-LTD into t-LTP (blue), whereas application of d-AP5 alone failed to influence the development of t-LTD (black). Traces show an EPSP before (1) and 40 min after pairing (2). (B) The DA-induced conversion of t-LTD into t-LTP involves the activation of cAMP/PKA signaling cascade, which closely mimics the effect of DA. Forskolin, an AC activator, applied immediately after the pairing protocol (Δt = −20 ms) converted t-LTD into t-LTP (red). This effect requires NMDA receptors as application of d-AP5 1 min before forskolin application partially blocked the forskolin-induced conversion of t-LTD into t-LTP (blue), whereas application of d-AP5 alone failed to influence the development of t-LTD (black; data same as in A). (C) Downstream of cAMP, PKA is involved in the conversion of t-LTD into t-LTP as application of the PKA inhibitor, H-89, completely prevented the conversion of t-LTD into t-LTP. Traces are presented as in A. (D) Summary of results. Error bars represent s.e.m. Significant difference (*p < 0.05, **p < 0.01, ***p < 0.001) compared with the baseline (one-sample two-tailed Student's t-test) or between the indicated two groups (paired two-tailed Student's t-test). The number of cells is shown in parentheses.

DOI: http://dx.doi.org/10.7554/eLife.09685.009

Figure 4—source data 1. Source data for Figure 4.
DOI: http://dx.doi.org/10.7554/eLife.09685.010
elife09685s004.xlsx (71.4KB, xlsx)
DOI: 10.7554/eLife.09685.010
Figure 4—source data 2. Source data for Figure 4—figure supplement 1.
DOI: http://dx.doi.org/10.7554/eLife.09685.011
elife09685s005.xlsx (69.8KB, xlsx)
DOI: 10.7554/eLife.09685.011
Figure 4—source data 3. Source data for Figure 4—figure supplement 2.
DOI: http://dx.doi.org/10.7554/eLife.09685.012
elife09685s006.xlsx (55.2KB, xlsx)
DOI: 10.7554/eLife.09685.012

Figure 4.

Figure 4—figure supplement 1. The retroactive conversion of t-LTP into t-LTD is due to specific DA receptor activation.

Figure 4—figure supplement 1.

Plot of normalized EPSP slopes demonstrating that (A) application of 10 μM SCH23390 and 50 μM sulpiride prevented DA-induced conversion of t-LTD into t-LTP, which resulted in significant t-LTD, whereas (B) neither SCH23390 nor (C) sulpiride alone was able to completely block the DA effect. Traces show an EPSP before (1) and 40 min after (2) pairing. (D) Summary of results. Error bars represent s.e.m. Significant difference (*p < 0.05, **p < 0.01, ***p < 0.001) compared with the baseline (one-sample two-tailed Student's t-test) or between the indicated two groups (paired two-tailed Student's t-test). The number of cells is shown in parentheses.
DOI: http://dx.doi.org/10.7554/eLife.09685.013

Figure 4—figure supplement 2. Forskolin had no effect on baseline EPSPs.

Figure 4—figure supplement 2.

Plot of normalized EPSP slopes showing that in the absence of a pairing protocol, application of 50 μM forskolin does not result in significant synaptic modification (n = 7). Traces show an EPSP at 10 min (1) and 58 min (2) of a recording. Error bars represent s.e.m.
DOI: http://dx.doi.org/10.7554/eLife.09685.014
Open in a new tab

To investigate the intracellular signaling mechanisms involved, we initially set out to establish the DAR subtype associated with the DA-induced conversion of t-LTD into t-LTP. Even though combined application of D1-like and D2-like receptor antagonist completely blocked the DA effect (Figure 4—figure supplement 1A,D), application of either D1-like or D2-like receptor antagonist alone only partially prevented the conversion of t-LTD into t-LTP (SCH 23390: 131 ± 16%; t(6) = 1.4, p = 0.0113 vs DA (t-LTP); t(6) = 3.6, p = 0.2277 vs control (t-LTD); t(6) = 1.9, p = 0.0994 vs 100%; n = 7. Sulpiride: 94 ± 13%; t(6) = 3.2, p = 0.0228 vs DA (t-LTP); t(6) = 1.5, p = 0.1932 vs control (t-LTD); t(6) = 0.5, p = 0.6435 vs 100%; n = 7; Figure 4—figure supplement 1B–D). This suggests that both receptor subtypes might mediate the retroactive effect of DA on STDP. Given the astounding complexity of D2-like receptor pharmacology (Neve et al., 2004), we first wanted to evaluate the possible involvement of D1-like receptor-activated cAMP/PKA signaling cascade in the DA-induced conversion of t-LTD into t-LTP. D1/D5 receptor stimulation leads to the activation of adenylyl cyclase (AC) and subsequent increase in cyclic adenosine monophosphate (cAMP) and protein kinase A (PKA) activation (Greengard et al., 1999; Neve et al., 2004). We found that the general AC activator, forskolin (50 μM), applied for 10–12 min immediately after the pairing protocol with Δt = −20 ms resulted in robust conversion of t-LTD into t-LTP (167 ± 17%, t(6) = 3.9, p = 0.0078 vs 100%, n = 7; Figure 4B,D). Notably, forskolin (50 μM) applied in the absence of the pairing protocol had no significant effect on baseline EPSPs (83 ± 12%, t(6) = 1.4, p = 0.2046 vs 100%, n = 7; Figure 4—figure supplement 2). The forskolin-induced conversion of t-LTD into t-LTP was also NMDA receptor dependent since bath application of d-AP5 (50 μM) 1 min before forskolin treatment prevented synaptic potentiation from developing, although synaptic depression was reversed (102 ± 11%, t(5) = 0.2, p = 0.8616 vs 100%, n = 6; Figure 4B,D). This result suggests that the cAMP cascade works either upstream of or in parallel to NMDA receptor activation for the development of potentiation to occur. Downstream of cAMP, PKA is involved because the PKA inhibitor, H-89 (20 µM), blocked the DA-induced conversion of t-LTD into t-LTP, revealing significant t-LTD (76 ± 5%, t(6) = 5.0, p = 0.0025 vs 100%, n = 7; Figure 4C,D). Taken together, these results imply that the DA-induced conversion of t-LTD into t-LTP involves activation of the cAMP/PKA signaling cascade, which closely mimics the effects of DA (Figure 4A vs Figure 4B). Although, D2-like receptors are typically associated with the inhibition of AC (Neve et al., 2004), interestingly, there is also evidence that D2-like receptor stimulation can potentiate AC activity (Glass and Felder, 1997; Watts and Neve, 1997). Therefore, while the possibility that D2-like receptors contribute to the conversion of t-LTD into t-LTP via a different signaling cascade cannot be excluded, it is tempting to suggest that the DA-induced conversion of t-LTD into t-LTP is mediated primarily via the cAMP/PKA pathway. Hence, based on our results, we propose that the stimulation of postsynaptic (Figure 5ai) or presynaptic (Figure 5aii) DARs activates the cAMP/PKA pathway, which—via two cellular mechanisms (de-depression and potentiation)—leads to the retroactive conversion of t-LTD into t-LTP.

Figure 5. Proposed mechanisms underlying the DA-induced conversion of t-LTD into t-LTP.

Figure 5.

Open in a new tab

(A) Schematic diagram depicting core components of the proposed cellular mechanisms underlying the DA-induced conversion of t-LTD into t-LTP (de-depression and potentiation). (Ai) Model based on postsynaptic NMDAR-dependent potentiation (Bi and Poo, 1998; Caporale and Dan, 2008; Zhang et al., 2009; Edelmann and Lessmann, 2011; Yang and Dani, 2014) and metabotropic glutamate receptor-dependent (mGluRs) depression (Otani and Connor, 1998; Kemp and Bashir, 1999; Huber et al., 2000). De-depression (red, left): Activation of G protein-coupled D1/D5 receptors stimulates AC, increasing cAMP and activating PKA (Greengard et al., 1999; Neve et al., 2004), which, via phosphorylation of I-1 (Ingebritsen and Cohen, 1983), reverses the PP1-induced dephosphorylation of synaptic AMPARs (Lee et al., 2000; Mockett et al., 2007). Potentiation (red, right): PKA activation enhances NMDAR function (Westphal et al., 1999; Chen and Roche, 2007). (Aii) Model based on presynaptic depression (Bolshakov and Siegelbaum, 1994; Siegel et al., 1994; Oliet et al., 1997; Charton et al., 1999; Watabe et al., 2002; Jourdain et al., 2007). De-depression (red, left): Activation of presynaptic DA receptors stimulates AC, increasing cAMP and activating PKA (Greengard et al., 1999; Neve et al., 2004), which reverses the calcineurin-dependent presynaptic depression. Potentiation (red, right): as in Ai. Arrow indicates activation/phosphorylation, blunt-ended line indicates inhibition/dephosphorylation. Abbreviations: AMPAR, AMPA-type glutamate receptor; NMDAR, NMDA-type glutamate receptor; mGluR1/5, group I metabotropic glutamate receptor; DAR, dopamine receptor; AC, adenylate cyclase; cAMP, cyclic adenosine monophosphate; PKA, protein kinase A; I-1, inhibitor 1; PP1, protein phosphatase 1; PLC, phospholipase C; IP3, inositol 1,4,5-trisphosphate; ER, endoplasmic reticulum; DAG, diacylglycerol; eCB, endocannabinoid; CB1R, cannabinoid receptor type 1; CN, calcineurin; CaMKII, calcium-calmodulin-dependent protein kinase II. (B) Schematic diagram of synaptic and behavioral timescales in reward learning. During Exploration, the activity-dependent modification of synaptic strength due to spike timing-dependent plasticity (STDP) depends on the coordinated spiking between presynaptic and postsynaptic neurons on a millisecond time scale. Post-before-pre pairing leads to synaptic depression that develops gradually on a scale of minutes. When Reward, signaled via dopamine, follows Exploration with a Delay of seconds to minutes, synaptic depression is converted into potentiation.

DOI: http://dx.doi.org/10.7554/eLife.09685.015

The functional implications of our finding depend on the presynaptic source of DA in the hippocampus, which remains controversial because of the apparent discrepancy between DAergic terminals and receptors (Scatton et al., 1980; Gasbarri et al., 1997; Lisman and Grace, 2005). On one hand, it has been argued that noradrenergic terminals from the locus coeruleus, which mediates arousal and the optimization of behavioral performance (Aston-Jones and Cohen, 2005; Chamberlain and Robbins, 2013), may provide a major source of DA release (Smith and Greene, 2012). On the other hand, hippocampal pyramidal cell assemblies are directly affected by concurrent activity in midbrain DAergic neurons (McNamara et al., 2014), which have been linked to reward-seeking behavior (Schultz et al., 1997) and appetitive stimuli (Mirenowicz and Schultz, 1996; Fiorillo et al., 2013). This latter finding (McNamara et al., 2014) not only supports the hypothesis that hippocampal DA is relevant for reward processing, but is also consistent with our result that the activation of DAergic receptors during coordinated spiking activity changes the functional outcome of STDP (Figure 1C–G).

Previous studies examining reinforcement learning at the level of synaptic plasticity have showed that neuromodulators can affect timing-dependent plasticity in locust (Cassenaer and Laurent, 2012) and spine structural plasticity in striatal medium spiny neurons in mice (Yagishita et al., 2014) when acting within a delay time window of 1 s. While this narrow temporal detection window may be important in the striatum, it cannot account for the experimental evidence from behavioral studies of response acquisition with an extended reinforcement delay in rats and pigeons (Lattal and Gleeson, 1990; Sutphin et al., 1998), rhesus monkeys (Galuska and Woods, 2005), and humans (Okouchi, 2009). Meanwhile, our finding demonstrates that DA can modulate STDP in the CA1 with a reinforcement delay of at least 1 min (Figure 5B). Such extended reinforcement delay is likely to be particularly important in hippocampus-dependent learning during spatial exploration.

In conclusion, our work demonstrates a retroactive effect of DA on STDP—converting t-LTD into t-LTP. This effect is mediated at least in part through the activation of the cAMP/PKA cascade and requires the activation of synaptic NMDA receptors. This in turn suggests that the conversion can only occur at synapses that are re-activated following the initial pairing event. Interestingly, it has been reported that hippocampal reactivation events (sharp wave ripples) increase in frequency following reward (Singer and Frank, 2009; Atherton et al., 2015). Thus, in behaving animals, the conditions for the conversion of depression into potentiation might occur during reward-related sharp wave ripple activity. Together, these findings support the concept of a slowly decaying synaptic eligibility trace that is committed to memory by the occurrence of reward and provide a possible mechanism for associating specific experiences with behaviorally distant, rewarding outcomes in animals (Sutton and Barto, 1981; Suri and Schultz, 1999; Pan et al., 2005; Izhikevich, 2007; Harnett et al., 2009), including humans (Dunsmoor et al., 2015).

Materials and methods

Animals

Wild-type mice (C57BL/6; postnatal days 12–18; from Harlan, Bicester, UK or Central Animal Facility, Physiological Laboratory, Cambridge University) of both sexes were housed on a 12-hr light/dark cycle at 19–23 °C, with water and food ad libitum. Experimental procedures and animal use were in accordance with the animal care guidelines of the UK Animals (Scientific Procedures) Act 1986 under personal and project licenses held by the authors. Caution was taken to minimize stress and the number of animals used in the experiments.

Slice preparation

Mice were anesthetized with isoflurane and decapitated. The brain was rapidly removed, glued to the stage of a vibrating microtome (Leica VT 1200S, Leica Biosystems, Wetzlar, Germany) and immersed in ice-cold artificial cerebrospinal fluid (ACSF) containing the following (mM): 126 NaCl, 3 KCl, 26.4 NaH2CO3, 1.25 NaH2PO4, 2 MgSO4, 2 CaCl2, and 10 glucose. The ACSF solution, with pH adjusted to 7.2 and osmolarity to 270–290 mOsm l−1, was continuously bubbled with carbogen gas (95% O2/5% CO2). The brain was sectioned into 350-μm-thick horizontal slices. The slices were incubated in ACSF at room temperature in a submerged-style storage chamber for at least 1 hr. For recordings (1–7 hr after slicing), individual slices were transferred to an immersion-type recording chamber, perfused with ACSF (2 ml min−1) at 24–26 °C.

Electrophysiology

Whole-cell recordings

Whole-cell patch-clamp recordings were performed on CA1 pyramidal neurons (located adjacent to the stratum oriens). For stimulation of Schaffer collaterals, a monopolar stimulation electrode was placed in the stratum radiatum of the CA1 subfield. The hippocampal subfields were visually identified using infrared differential interference contrast (DIC) microscopy. Patch pipettes (resistance: 4–8 MΩ) were made from borosilicate glass capillaries (0.68 mm inner diameter, 1.2 mm outer diameter), pulled using a P-97 Flaming/Brown micropipette puller (Sutter Instruments Co., Novato, California, USA). The internal solution of patch pipettes was (mM) 110 potassium gluconate, 4 NaCl, 40 HEPES, 2 ATP-Mg, 0.3 GTP (pH adjusted with 1 M KOH to 7.2, and osmolarity with ddH2O to 270 mOsm l−1). The liquid junction potential was not corrected for. Cells were accepted for experiment only if the resting membrane potential at the start of the recording was between −55 and −70 mV. Membrane potential was held at −70 mV throughout further recording by direct current application via the recording electrode. At the beginning of each recording all cells were tested for regular spiking responses to positive current steps—characteristic of pyramidal neurons.

Stimulation protocol

EPSPs of amplitude between 3 and 8 mV were evoked at 0.2 Hz by adjusting the magnitude of direct current pulses (stimulus duration 50 μs, intensity 100 μA–1 mA). After a stable EPSP baseline period of at least 10 min, STDP was induced by repeated pairings of single presynaptic EPSP evoked by stimulation of Schaffer collaterals and single postsynaptic action potential elicited with the minimum somatic current pulse (1–1.8 nA, 3 ms) via the recording electrode. Pairings were repeated 100 times at 0.2 Hz. Spike-timing intervals (∆t in ms) were measured between the onset of the EPSP and the onset of the action potential. The EPSPs were monitored for at least 40 min after the end of the pairing protocol. Presynaptic stimulation frequency remained constant throughout the experiment.

Data acquisition and data analysis

Voltage signals were low-pass filtered at 2 kHz using an Axon Multiclamp 700B amplifier (Molecular Devices, Sunnyvale, California, USA). Data were acquired at 5 kHz via an ITC18 interface board (Instrutech, Port Washington, New York, USA), transmitting to a Dell computer running the Igor Pro software (WaveMetrics, Lake Oswego, Oregon, USA). All experiments were carried out in the current clamp (‘bridge’) mode. Series resistance was monitored (10–15 MΩ) and compensated for by adjusting the bridge balance. Data were discarded if series resistance changed by more than 30%. Data were analyzed using Igor Pro. EPSP slopes were measured on the rising phase of the EPSP as a linear fit between the time points corresponding to 25–30% and 70–75% of the peak amplitude. For statistical analysis, the mean EPSP slope per minute of the recording was calculated from 12 consecutive sweeps and normalized to the baseline. Normalized ESPS slopes from the last 5 min of the baseline (immediately before pairing) and from the last 5 min of the recording (35–40 min or 55–60 min after pairing) were averaged. The magnitude of plasticity, as an indicator of synaptic change, was defined as the average EPSP slope after pairing expressed as a percentage of the average EPSP slope during baseline.

Drugs

The following drugs were used: dopamine hydrochloride 20 μM, forskolin 50 μM, d-AP5 50 μM, SCH23390 hydrochloride 10 μM, sulpiride 50 μM, H-89 20 µM. All drugs (purchased from Sigma–Aldrich, Dorset, United Kingdom; Tocris Bioscience, Bristol, United Kindgom; or Abcam, Cambridge, United Kingdom) were bath-applied through the perfusion system by dilution of concentrated stock solutions (prepared in water or DMSO) in ACSF.

Statistical analysis

Statistical comparisons were made using one-sample two-tailed or paired two-tailed Student's t-test, with a significance level of α = 0.05. Data are presented as mean ± s.e.m. Significance levels are indicated by *p < 0.05, **p < 0.01, ***p < 0.001.

Acknowledgements

This research was supported by a studentship from the Medical Research Council (UK) to ZB, the School of Biological Sciences at the University of Cambridge, and the Wellcome Trust (WS).

Funding Statement

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Funding Information

This paper was supported by the following grants:

  • Medical Research Council (MRC) Graduate studentship to Zuzanna Brzosko.

  • University of Cambridge Student research support fund to Zuzanna Brzosko, Wolfram Schultz, Ole Paulsen.

Additional information

Competing interests

WS: Reviewing editor, eLife.

The other authors declare that no competing interests exist.

Author contributions

ZB, Conception and design, Acquisition of data, Analysis and interpretation of data, Drafting or revising the article.

WS, Conception and design, Drafting or revising the article.

OP, Conception and design, Drafting or revising the article.

Ethics

Animal experimentation: Experimental procedures and animal use were in accordance with the animal care guidelines of the UK Animals (Scientific Procedures) Act 1986 under Home Office personal license PIL- ICB486697 and project license PPL80/2440 held by the authors. Caution was taken to minimize stress and the number of animals used in experiments.

References

  1. Aston-Jones G, Cohen JD. An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance. Annual Review of Neuroscience. 2005;28:403–450. doi: 10.1146/annurev.neuro.28.061604.135709. [DOI] [PubMed] [Google Scholar]
  2. Atherton LA, Dupret D, Mellor JR. Memory trace replay: the shaping of memory consolidation by neuromodulation. Trends in Neurosciences. 2015;38:560–570. doi: 10.1016/j.tins.2015.07.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Bi GQ, Poo MM. Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. The Journal of Neuroscience. 1998;18:10464–10472. doi: 10.1523/JNEUROSCI.18-24-10464.1998. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Bolshakov VY, Siegelbaum SA. Postsynaptic induction and presynaptic expression of hippocampal long-term depression. Science. 1994;264:1148–1152. doi: 10.1126/science.7909958. [DOI] [PubMed] [Google Scholar]
  5. Caporale N, Dan Y. Spike timing-dependent plasticity: a Hebbian learning rule. Annual Review of Neuroscience. 2008;31:25–46. doi: 10.1146/annurev.neuro.31.060407.125639. [DOI] [PubMed] [Google Scholar]
  6. Cassenaer S, Laurent G. Conditional modulation of spike-timing-dependent plasticity for olfactory learning. Nature. 2012;482:47–52. doi: 10.1038/nature10776. [DOI] [PubMed] [Google Scholar]
  7. Chamberlain SR, Robbins TW. Noradrenergic modulation of cognition: therapeutic implications. Journal of Psychopharmacology / British Association for Psychopharmacology. 2013;27:694–718. doi: 10.1177/0269881113480988. [DOI] [PubMed] [Google Scholar]
  8. Charton JP, Herkert M, Becker CM, Schroder H. Cellular and subcellular localization of the 2B-subunit of the NMDA receptor in the adult rat telencephalon. Brain Research. 1999;816:609–617. doi: 10.1016/S0006-8993(98)01243-8. [DOI] [PubMed] [Google Scholar]
  9. Chen BS, Roche KW. Regulation of NMDA receptors by phosphorylation. Neuropharmacology. 2007;53:362–368. doi: 10.1016/j.neuropharm.2007.05.018. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Dunsmoor JE, Murty VP, Davachi L, Phelps EA. Emotional learning selectively and retroactively strengthens memories for related events. Nature. 2015;520:345–348. doi: 10.1038/nature14106. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Edelmann E, Lessmann V. Dopamine modulates spike timing-dependent plasticity and action potential properties in CA1 pyramidal neurons of acute rat hippocampal slices. Frontiers in Synaptic Neuroscience. 2011;3:6. doi: 10.3389/fnsyn.2011.00006. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Fiorillo CD, Song MR, Yun SR. Multiphasic temporal dynamics in responses of midbrain dopamine neurons to appetitive and aversive stimuli. The Journal of Neuroscience. 2013;33:4710–4725. doi: 10.1523/JNEUROSCI.3883-12.2013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Frey U, Schroeder H, Matthies H. Dopaminergic antagonists prevent long-term maintenance of posttetanic LTP in the CA1 region of rat hippocampal slices. Brain Research. 1990;522:69–75. doi: 10.1016/0006-8993(90)91578-5. [DOI] [PubMed] [Google Scholar]
  14. Galuska CM, Woods JH. Acquisition of cocaine self-administration with unsignaled delayed reinforcement in rhesus monkeys. Journal of the Experimental Analysis of Behavior. 2005;84:269–280. doi: 10.1901/jeab.2005.99-04. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Gasbarri A, Sulli A, Packard MG. The dopaminergic mesencephalic projections to the hippocampal formation in the rat. Progress in Neuro-psychopharmacology & Biological Psychiatry. 1997;21:1–22. doi: 10.1016/S0278-5846(96)00157-1. [DOI] [PubMed] [Google Scholar]
  16. Glass M, Felder CC. Concurrent stimulation of cannabinoid CB1 and dopamine D2 receptors augments cAMP accumulation in striatal neurons: evidence for a Gs linkage to the CB1 receptor. The Journal of Neuroscience. 1997;17:5327–5333. doi: 10.1124/jpet.108.145425. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Greengard P, Allen PB, Nairn AC. Beyond the dopamine receptor: the DARPP-32/protein phosphatase-1 cascade. Neuron. 1999;23:435–447. doi: 10.1016/S0896-6273(00)80798-9. [DOI] [PubMed] [Google Scholar]
  18. Harnett MT, Bernier BE, Ahn KC, Morikawa H. Burst-timing-dependent plasticity of NMDA receptor-mediated transmission in midbrain dopamine neurons. Neuron. 2009;62:826–838. doi: 10.1016/j.neuron.2009.05.011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Huber KM, Kayser MS, Bear MF. Role for rapid dendritic protein synthesis in hippocampal mGluR-dependent long-term depression. Science. 2000;288:1254–1256. doi: 10.1126/science.288.5469.1254. [DOI] [PubMed] [Google Scholar]
  20. Hull CL. Principles of behavior: an introduction to behavior theory. Oxford, England: Appleton Century; 1943. [Google Scholar]
  21. Ingebritsen TS, Cohen P. Protein phosphatases: properties and role in cellular regulation. Science. 1983;221:331–338. doi: 10.1126/science.6306765. [DOI] [PubMed] [Google Scholar]
  22. Izhikevich EM. Solving the distal reward problem through linkage of STDP and dopamine signalling. Cerebral Cortex. 2007;17:2443–2452. doi: 10.1093/cercor/bhl152. [DOI] [PubMed] [Google Scholar]
  23. Jourdain P, Bergersen LH, Bhaukaurally K, Bezzi P, Santello M, Domercq M, Matute C, Tonello F, Gundersen V, Volterra A. Glutamate exocytosis from astrocytes controls synaptic strength. Nature Neuroscience. 2007;10:331–339. doi: 10.1038/nn1849. [DOI] [PubMed] [Google Scholar]
  24. Kemp N, Bashir ZI. Induction of LTD in the adult hippocampus by the synaptic activation of AMPA/kainate and metabotropic glutamate receptors. Neuropharmacology. 1999;38:495–504. doi: 10.1016/S0028-3908(98)00222-6. [DOI] [PubMed] [Google Scholar]
  25. Lattal KA, Gleeson S. Response acquisition with delayed reinforcement. Journal of Experimental Psychology. 1990;16:27–39. doi: 10.1037/0097-7403.16.1.27. [DOI] [PubMed] [Google Scholar]
  26. Lee HK, Barbarosie M, Kameyama K, Bear MF, Huganir RL. Regulation of distinct AMPA receptor phosphorylation sites during bidirectional synaptic plasticity. Nature. 2000;405:955–959. doi: 10.1038/35016089. [DOI] [PubMed] [Google Scholar]
  27. Lisman JE, Grace AA. The hippocampal-VTA loop: controlling the entry of information into long- term memory. Neuron. 2005;46:703–713. doi: 10.1016/j.neuron.2005.05.002. [DOI] [PubMed] [Google Scholar]
  28. Markram H, Lubke J, Frotscher M, Sakmann B. Regulation of synaptic efficacy by coincidence of postsynaptic APs and EPSPs. Science. 1997;275:213–215. doi: 10.1126/science.275.5297.213. [DOI] [PubMed] [Google Scholar]
  29. McNamara CG, Tejero-Cantero Á, Trouche S, Campo-Urriza N, Dupret D. Dopaminergic neurons promote hippocampal reactivation and spatial memory persistence. Nature Neuroscience. 2014;17:1658–1660. doi: 10.1038/nn.3843. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Mirenowicz J, Schultz W. Preferential activation of midbrain dopamine neurons by appetitive rather than aversive stimuli. Nature. 1996;379:449–451. doi: 10.1038/379449a0. [DOI] [PubMed] [Google Scholar]
  31. Mockett BG, Guévremont D, Williams JM, Abraham WC. Dopamine D1/D5 receptor activation reverses NMDA receptor-dependent long-term depression in rat hippocampus. The Journal of Neuroscience. 2007;27:2918–2926. doi: 10.1523/JNEUROSCI.0838-06.2007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Neve KA, Seamans JK, Trantham-Davidson H. Dopamine receptor signalling. Journal of Receptor and Signal Transduction Research. 2004;24:165–205. doi: 10.1081/RRS-200029981. [DOI] [PubMed] [Google Scholar]
  33. Okouchi H. Response acquisition by humans with delayed reinforcement. Journal of the Experimental Analysis of Behavior. 2009;91:377–390. doi: 10.1901/jeab.2009.91-377. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Oliet SHR, Malenka RC, Nicoll RA. Two distinct forms of long-term depression coexist in CA1 hippocampal pyramidal cells. Neuron. 1997;18:1294–1297. doi: 10.1016/S0896-6273(00)80336-0. [DOI] [PubMed] [Google Scholar]
  35. Otani S, Connor JA. Requirement of rapid Ca2+ entry and synaptic activation of metabotropic glutamate receptors for the induction of long-term depression in adult rat hippocampus. The Journal of Physiology. 1998;511:761–770. doi: 10.1111/j.1469-7793.1998.761bg.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Otmakhova NA, Lisman JE. Dopamine selectively inhibits the direct cortical pathway to the CA1 hippocampal region. The Journal of Neuroscience. 1999;19:1437–1445. doi: 10.1523/JNEUROSCI.19-04-01437.1999. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Pan W-X, Schmidt R, Wickens JR, Hyland BI. Dopamine cells respond to predicted events during classical conditioning: evidence for eligibility traces in the reward-learning network. The Journal of Neuroscience. 2005;25:6235–6242. doi: 10.1523/JNEUROSCI.1478-05.2005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Pawlak V, Wickens JR, Kirkwood A, Kerr JND. Timing is not everything: neuromodulation opens the STDP gate. Frontiers in Synaptic Neuroscience. 2010;2:146. doi: 10.3389/fnsyn.2010.00146. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Scatton B, Simon H, Le Moal M, Bischoff S. Origin of dopaminergic innervation of the rat hippocampal formation. Neuroscience Letters. 1980;18:125–131. doi: 10.1016/0304-3940(80)90314-6. [DOI] [PubMed] [Google Scholar]
  40. Schultz W, Dayan P, Montague RR. A neural substrate of prediction and reward. Science. 1997;275:1593–1599. doi: 10.1126/science.275.5306.1593. [DOI] [PubMed] [Google Scholar]
  41. Seol GH, Ziburkus J, Huang S, Song L, Kim IT, Takamiya K, Huganir RL, Lee HK, Kirkwood A. Neuromodulators control the polarity of spike-timing-dependent synaptic plasticity. Neuron. 2007;55:919–929. doi: 10.1016/j.neuron.2007.08.013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Siegel SJ, Brose N, Janssen WG, Gasic GP, Jahn R, Heinemann SF, Morrison JH. Regional, cellular, and ultrastructural distribution of N-methyl-d-aspartate receptor subunit 1 in monkey hippocampus. Proceedings of the National Academy of Sciences of USA. 1994;91:564–568. doi: 10.1073/pnas.91.2.564. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Singer AS, Frank LM. Rewarded outcomes enhance reactivation of experience in the hippocampus. Neuron. 2009;64:910–921. doi: 10.1016/j.neuron.2009.11.016. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Smith CC, Greene RW. CNS dopamine transmission mediated by noradrenergic innervation. The Journal of Neuroscience. 2012;32:6072–6080. doi: 10.1523/JNEUROSCI.6486-11.2012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Suri R, Schultz W. A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task. Neuroscience. 1999;91:871–890. doi: 10.1016/S0306-4522(98)00697-6. [DOI] [PubMed] [Google Scholar]
  46. Sutphin G, Byrne T, Poling A. Response acquisition with delayed reinforcement: a comparison of two-lever procedures. Journal of the Experimental Analysis of Behavior. 1998;69:17–28. doi: 10.1901/jeab.1998.69-17. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Sutton RS, Barto AG. Toward a modern theory of adaptive networks: expectation and prediction. Psychological Review. 1981;88:135–170. doi: 10.1037/0033-295X.88.2.135. [DOI] [PubMed] [Google Scholar]
  48. Watabe AM, Carlisle HJ, O'Dell TJ. Postsynaptic induction and presynaptic expression of group 1 mGluR-dependent LTD in the hippocampal CA1 region. Journal of Neurophysiology. 2002;87:1395–1403. doi: 10.1152/jn.00723.2001. [DOI] [PubMed] [Google Scholar]
  49. Watts VJ, Neve KA. Activation of type II adenylate cyclase by D2 and D4 but not D3 dopamine receptors. Molecular Pharmacology. 1997;52:181–186. doi: 10.1124/mol.52.2.181. [DOI] [PubMed] [Google Scholar]
  50. Westphal RS, Tavalin SJ, Lin JW, Alto NM, Fraser IDC, Langeberg LK, Sheng M, Scott JD. Regulation of NMDA receptors by an associated phosphatase-kinase signaling complex. Science. 1999;285:93–96. doi: 10.1126/science.285.5424.93. [DOI] [PubMed] [Google Scholar]
  51. Yagishita S, Hayashi-Tkagi A, Ellis-Davies GCR, Urakubo H, Ishii S, Kasai H. A critical time window for dopamine actions on the structural plasticity of dendritic spines. Science. 2014;345:1616–1620. doi: 10.1126/science.1255514. [DOI] [PMC free article] [PubMed] [Google Scholar]
  52. Yang K, Dani JA. Dopamine D1 and D5 receptors modulate spike timing-dependent plasticity at medial perforant path to dentate granule cell synapses. The Journal of Neuroscience. 2014;34:15888–15897. doi: 10.1523/JNEUROSCI.2400-14.2014. [DOI] [PMC free article] [PubMed] [Google Scholar]
  53. Zhang JC, Lau PM, Bi GQ. Gain in sensitivity and loss in temporal contrast of STDP by dopaminergic modulation at hippocampal synapses. Proceedings of the National Academy of Sciences of USA. 2009;106:13028–13033. doi: 10.1073/pnas.0900546106. [DOI] [PMC free article] [PubMed] [Google Scholar]
eLife. 2015 Oct 30;4:e09685. doi: 10.7554/eLife.09685.016

Decision letter

Editor: Marlene Bartos1

eLife posts the editorial decision letter and author response on a selection of the published articles (subject to the approval of the authors). An edited version of the letter sent to the authors after peer review is shown, indicating the substantive concerns or comments; minor concerns are not usually shown. Reviewers have the opportunity to discuss the decision before the letter is sent (see review process). Similarly, the author response typically shows only responses to the major concerns raised by the reviewers.

Thank you for submitting your work entitled “Retroactive effect of dopamine on spike timing-dependent plasticity” for further consideration at eLife. Your article has been favorably evaluated by Eve Marder (Senior editor) and three reviewers, one of whom is a member of our Board of Reviewing Editors. One of the reviewers, Guoqiang Bi, has agreed to reveal his identity.

All three reviewers viewed the study as valuable for the readership of eLife. The main observation that spike-time dependent LTD can be converted into LTP upon DA release 10 min after the initial LTP-pairing protocol, was viewed as interesting. However, major criticism was formulated in relation to the missing explanation on how the timing of bath-applied DA may relate to reward-based conditioning. The reviewers asked for diagrams describing the timing issues, in particular the possible linkages between the cAMP pathway and its relation to Ca2+ signaling caused by action potentials and NMDAR activation. A stronger discussion of the data with existing literature is necessary. Moreover, one reviewer asked for additional experiments in which SCH23390 and sulpiride are separately applied. The reviewers were also curious for your thoughts about how the possible cAMP/PKA-mediated mechanism could be linked to Ca2+ signals, NMDA receptor activation and finally the switch from LTD to LTP. The specific comments of the reviewers are attached below.

Reviewer #1:

The study by the Paulsen's group shows nicely that spike-time dependent LTD can be converted into LTP upon DA release 10 min after the initial LTP-pairing protocol. They propose and show good experimental evidence for their hypothesis that this is caused by activation of the cAMP/PKA pathway plus activation of NMDARs. The data fit very well with a recent study from McNamara et al. (Nature Neuroscience, 2014) showing that DA improves reactivation of cell assemblies in CA1 and memory persistence. I have mild concerns regarding the precise timing of bath-applied drugs in the bath. How fast does the drug get to the DA receptors located at the recorded cell? This is important in defining the time window during which the DA effect can take place and induce the observed conversion of synaptic plasticity. The difference in the finding presented here to the one by the McNamara group should be emphasized.

Reviewer #2:

This study by Brzosko et al. addressed how dopamine (DA) retroactively regulates spike timing-dependent plasticity (STDP) in order to account for the distal reward problem, where rewards promote learning of experiences preceding rewards. The main result is that DA applied after the delivery of the post-before-pre pairing protocol converted LTD into LTP.

1) DA was bath applied for 10-12 min starting within 1 min after the delivery of the post-before-pre pairing protocol, which took ∼8 min. It is not clear how this relates to behaviorally relevant timing during reward-based conditioning, as described in Schultz, Dayan and Montague (1997), Suri and Schultz (1999), and Pan, Schmidt, Wickens and Hyland (2005). Furthermore, how does the millisecond order timing dependence in STDP relate to the second order timing dependence during reward-based learning (for example, see Drew and Abbott, PNAS, 2006, “Extending the effects of spike-timing-dependent plasticity to behavioral timescales”)? A schematic diagram on this timing issue (from millisecond to second to min in the current study) should be provided to account for the behavioral relevance.

2) In terms of the cellular mechanism, it is not clear how DA-induced activation of cAMP/PKA pathway relates to Ca signals caused by spikes and NMDA receptor activation. A schematic diagram describing the possible linkage should be provided with the timing diagram mentioned above. The authors may want to compare their data with a recent paper that addressed this issue on DA/spike timing dependent plasticity of dendritic spines (Yagishita et al., Science, 2015, A critical time window for dopamine actions on the structural plasticity of dendritic spines).

3) In the bar graphs, it is not clear what the significance mark on each bar means. I do understand the significance marks between bars.

4) It should be stated if any of the treatments (plasticity induction, DA/forskolin application, etc.) affected the amount of current necessary to hold the membrane potential at -70 mV, preferably with actual data.

5) According to the description in the Results section, LTP induced with -10 ms post-before-pre timing was “surprising”. This effect is ascribed to the release of endogenous DA by the stimulating electrode. It should be discussed in further detail how this observation compares to previous studies in brain slices and cultures.

Reviewer #3:

In this manuscript, the authors investigated the temporal specificity of dopaminergic modulation of STDP. They found that dopamine application within 10 minutes after STDP pairing could retroactively convert t-LTD into t-LTP. This conversion is apparently mediated by cAMP/PKA signaling, and requires synaptic activation as well as functional NMDA receptors. Together, the results provide an elegant cellular mechanism for the delayed effect of dopamine in reinforcement learning.

Overall I think the experiments were well designed and carried out. The results are important and intriguing, and should be of interests for both cellular and systems neuroscientists. I do have several questions:

1) Throughout the study, the authors used combined SCH23390+sulpiride to block both D1/D5-like receptors and D2-like receptors. Can the authors test differential roles of the two receptor types in STDP conversion by using the two antagonists separately, as has been done in previous studies?

2) Regarding the cellular signaling mechanisms, the authors argue that since AP-5 blocks the effect of DA and forskolin in t-LTD/LTP conversion, the cAMP/PKA signaling works either upstream of, or in parallel to NMDA receptor activation. Can the authors further allude to a potential mechanism(s) how such signaling interaction could happen?

eLife. 2015 Oct 30;4:e09685. doi: 10.7554/eLife.09685.017

Author response

Reviewer #1:

The study by the Paulsen's group shows nicely that spike-time dependent LTD can be converted into LTP upon DA release 10 min after the initial LTP-pairing protocol. They propose and show good experimental evidence for their hypothesis that this is caused by activation of the cAMP/PKA pathway plus activation of NMDARs. The data fit very well with a recent study from McNamara et al. (Nature Neuroscience, 2014) showing that DA improves reactivation of cell assemblies in CA1 and memory persistence. I have mild concerns regarding the precise timing of bath-applied drugs in the bath. How fast does the drug get to the DA receptors located at the recorded cell? This is important in defining the time window during which the DA effect can take place and induce the observed conversion of synaptic plasticity. The difference in the finding presented here to the one by the McNamara group should be emphasized.

The study by McNamara et al. (2014) shows that optogenetic activation of midbrain dopaminergic neurons during spatial exploration and learning enhances reactivation of newly encoded hippocampal representations and improves memory performance. This is consistent with our results demonstrating that the activation of DAergic receptors during coordinated spiking activity changes the functional outcome of STDP such that negative time pairings induce potentiation and not depression (Figure 1C-F). Please note, however, that the main finding of our study is significantly extending that of McNamara et al. in showing that dopamine can convert hippocampal timing-dependent synaptic depression into potentiation even when applied after the pairing event (as relevant for addressing the distal reward problem in the field of reinforcement learning). This effect was seen when dopamine was applied within one minute after the end of the pairing event. This has now been emphasized in the Results and Discussion section.

Reviewer #2:

1) DA was bath applied for 10-12 min starting within 1 min after the delivery of the post-before-pre pairing protocol, which took ∼8 min. It is not clear how this relates to behaviorally relevant timing during reward-based conditioning, as described in Schultz, Dayan and Montague (1997), Suri and Schultz (1999), and Pan, Schmidt, Wickens and Hyland (2005).

Our data show that DA can modulate STDP in the CA1 with a reinforcement delay of at least one minute (Figure 5B). This extended reinforcement delay is likely to be important in hippocampus-dependent memory tasks during spatial exploration.

Although behavioral evidence from hippocampus-based tasks is still lacking, our results are consistent with behavioral conditioning as demonstrated in multiple studies of response acquisition with delayed reinforcement in other brain areas: in rats and pigeons (Lattal and Gleeson, 1990; Sutphin, Byrne and Poling, 1998), rhesus monkeys (Galuska and Woods, 2005) and humans (Okouchi, 2009). This has now been emphasized in the manuscript (Results and Discussion section).

Please note that the duration of DA application (i.e. 10-12 minutes) mirrors the slow time profile of the gradually developing synaptic depression, where the synaptic weights reach their maximum depressed state after ∼10-15 minutes.

Furthermore, how does the millisecond order timing dependence in STDP relate to the second order timing dependence during reward-based learning (for example, see Drew and Abbott, PNAS, 2006, “Extending the effects of spike-timing-dependent plasticity to behavioral timescales”)? A schematic diagram on this timing issue (from millisecond to second to min in the current study) should be provided to account for the behavioral relevance.

We would like to thank for the suggestion of incorporating such a schematic, which is now presented in Figure 5B.

2) In terms of the cellular mechanism, it is not clear how DA-induced activation of cAMP/PKA pathway relates to Ca signals caused by spikes and NMDA receptor activation. A schematic diagram describing the possible linkage should be provided with the timing diagram mentioned above.

Again, we would like to thank you for this suggestion, and we have incorporated a figure to illustrate the links between the cAMP/PKA signaling cascade and the consensus signaling events underlying the induction of LTP and LTD at hippocampal synapses (Figure 5Ai and ii).

The authors may want to compare their data with a recent paper that addressed this issue on DA/spike timing dependent plasticity of dendritic spines (Yagishita et al., Science, 2015, A critical time window for dopamine actions on the structural plasticity of dendritic spines).

To address this request, we have incorporated the following paragraph to the Results and Discussion: “Previous studies examining reinforcement learning at the level of synaptic plasticity have showed that neuromodulators can affect timing-dependent plasticity in locust […]. Such extended reinforcement delay is likely to be particularly important in hippocampus-dependent learning during spatial exploration.”

We would like to emphasize that the timing rules for reinforcement are likely to be variable across different systems, ranging from 0.5-1.0 sec time delays for dopamine learning (Harnett, Bernier, Ahn and Morikawa, 2009) and fast striatal processing (Yagishita et al., 2014), via 1-40 sec for more complex behavioral tasks (Lattal and Gleeson, 1990; Sutphin, Byrne and Poling, 1998; Galuska and Woods, 2005; Okouchi, 2009) and aversive stimuli (Dickinson, Contemporary animal learning theory. Cambridge, UK: Cambridge University Press, 1980), to delays in the order of at least one minute for hippocampus-dependent learning as observed in our study.

3) In the bar graphs, it is not clear what the significance mark on each bar means. I do understand the significance marks between bars.

The significance mark on each bar indicates significant difference (*P < 0.05, **P < 0.01, ***P < 0.001) compared with the baseline (one-sample two-tailed Student’s t-test) or between the indicated two groups (paired two-tailed Student’s t-test). This has now been explicitly stated in each figure legend.

4) It should be stated if any of the treatments (plasticity induction, DA/forskolin application, etc.) affected the amount of current necessary to hold the membrane potential at -70 mV, preferably with actual data.

Author response image 1.

Author response image 1.

Open in a new tab

Example plot showing changes in the holding current over time for post-before-pre pairing protocol with Δt = −20 ms in control t-LTD (n = 9) and with DA added after pairing (n = 11). Error bars represent s.e.m.

DOI: http://dx.doi.org/10.7554/eLife.09685.018

While there was a small difference in holding current between Control t-LTD and DA condition, a similar difference was not seen with Forskolin, suggesting that the change in holding current is not likely to be related to the effect on synaptic efficacy. We believe that this figure does not contribute to explaining our finding and, if possible, we would prefer not to include it in the manuscript.

5) According to the description in the Results section, LTP induced with -10 ms post-before-pre timing was “surprising”. This effect is ascribed to the release of endogenous DA by the stimulating electrode. It should be discussed in further detail how this observation compares to previous studies in brain slices and cultures.

In view of the conventional presentation of STDP, we found this finding surprising. We now explicitly state that “This conflicts with previous reports from hippocampal cultures (Bi and Poo, 1998; Zhang, Lau and Bi, 2009) and acute slices (Edelmann and Lessmann, 2011; Yang and Dani, 2014), where post-before-pre pairing protocols never elicited synaptic potentiation in baseline condition.”

However, it has been shown earlier that DA widens the time window for the induction of t-LTP (Zhang, Lau and Bi, 2009; Yang and Dani, 2014), and this has now been discussed in the same paragraph.

Reviewer #3:

1) Throughout the study, the authors used combined SCH23390+sulpiride to block both D1/D5-like receptors and D2-like receptors. Can the authors test differential roles of the two receptor types in STDP conversion by using the two antagonists separately, as has been done in previous studies?

In response to this comment, we have carried out a new set of experiments, applying D1-like or D2-like receptor antagonist separately. Even though combined application of D1-like and D2-like receptor antagonist completely blocked the DA effect (Figure 4–figure supplement 1A, D), application of either D1-like or D2-like receptor antagonist alone only partially prevented the conversion of t-LTD into t-LTP (SCH 23390: 131 ± 16%; t(6) = 1.4, P = 0.0113 vs. DA (t-LTP); t(6) = 3.6, P = 0.2277 vs. control (t-LTD); t(6) = 1.9, P = 0.0994 vs. 100%; n = 7. Sulpiride: 94 ± 13%; t(6) = 3.2, P = 0.0228 vs. DA (t-LTP); t(6) = 1.5, P = 0.1932 vs. control (t-LTD); t(6) = 0.5, P = 0.6435 vs. 100%; n = 7; Figure 4–figure supplement 1B-D). This suggests that both receptor sub-types might mediate the retroactive effect of DA on STDP. This is now reported in the text (in the sixth paragraph of the Results and Discussion section), and presented in Figure 4–figure supplement 1.

We further discuss the implications of this in the same paragraph as follows: “D1/D5-receptor stimulation leads to the activation of adenylyl cyclase (AC) and subsequent increase in cyclic adenosine monophosphate (cAMP) […] to the retroactive conversion of t-LTD into t-LTP.”

2) Regarding the cellular signaling mechanisms, the authors argues that since AP-5 blocks the effect of DA and forskolin in t-LTD/LTP conversion, the cAMP/PKA signaling works either upstream of, or in parallel to NMDA receptor activation. Can the authors further allude to a potential mechanism(s) how such signaling interaction could happen?

Please see the response to Reviewer 2, point 1 above.

Additional changes to the manuscript:

To clarify the presentation in accordance to the reviewers’ comments, we have also updated Figure 1 and Figure 4, and expanded and moved Figure 2–figure supplement 1 to Figure 4–figure supplement 1.

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Supplementary Materials

    Figure 1—source data 1. Source data for Figure 1.

    DOI: http://dx.doi.org/10.7554/eLife.09685.004

    elife09685s001.xlsx (100.6KB, xlsx)
    DOI: 10.7554/eLife.09685.004
    Figure 2—source data 1. Source data for Figure 2.

    DOI: http://dx.doi.org/10.7554/eLife.09685.006

    elife09685s002.xlsx (58.4KB, xlsx)
    DOI: 10.7554/eLife.09685.006
    Figure 3—source data 1. Source data for Figure 3.

    DOI: http://dx.doi.org/10.7554/eLife.09685.008

    elife09685s003.xlsx (63.4KB, xlsx)
    DOI: 10.7554/eLife.09685.008
    Figure 4—source data 1. Source data for Figure 4.

    DOI: http://dx.doi.org/10.7554/eLife.09685.010

    elife09685s004.xlsx (71.4KB, xlsx)
    DOI: 10.7554/eLife.09685.010
    Figure 4—source data 2. Source data for Figure 4—figure supplement 1.

    DOI: http://dx.doi.org/10.7554/eLife.09685.011

    elife09685s005.xlsx (69.8KB, xlsx)
    DOI: 10.7554/eLife.09685.011
    Figure 4—source data 3. Source data for Figure 4—figure supplement 2.

    DOI: http://dx.doi.org/10.7554/eLife.09685.012

    elife09685s006.xlsx (55.2KB, xlsx)
    DOI: 10.7554/eLife.09685.012

    Articles from eLife are provided here courtesy of eLife Sciences Publications, Ltd

    RESOURCES