Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2017 Aug 28.
Published in final edited form as: Cell Rep. 2017 Aug 22;20(8):1765–1774. doi: 10.1016/j.celrep.2017.07.076

Dopamine encodes retrospective temporal information in a context-independent manner

Kaitlyn M Fonzi 1, Merridee J Lefner 1, Paul EM Phillips 2, Matthew J Wanat 1,2,3
PMCID: PMC5573123  NIHMSID: NIHMS898182  PMID: 28834741

Summary

The dopamine system responds to reward-predictive cues to reflect a prospective estimation of reward value, though its role in encoding retrospective reward-related information is unclear. We report cue-evoked dopamine release in the nucleus accumbens core encodes the time elapsed since the previous reward, or rather the wait time. Specifically, a cue that always follows the preceding reward with a short wait time elicits a greater dopamine response relative to a distinct cue that always follows the preceding reward with a long wait time. Differences in the dopamine response between short wait and long wait cues were evident even when these cues were never experienced together within the same context. Conditioned responding updated accordingly with a change in cue-evoked dopamine release, but was unrelated to a difference in the dopamine response between cues. Collectively, these findings illustrate that the cue-evoked dopamine response conveys a subjective estimation of the relative reward rate.

eTOC Blurb

Fonzi et al demonstrate that cue-evoked dopamine release encodes retrospective time-related information. They find that the dopamine system can discern differences between cues that have never been experienced together in the same context.

graphic file with name nihms898182u1.jpg

Introduction

Appetitive behavioral actions are influenced by the presence of reward-associated cues through neurobiological processes involving the mesolimbic dopamine system (Salamone and Correa, 2012). Dopamine neurons respond to cues to signal prospective reward-related information, such as reward size (Gan et al., 2010; Roesch et al., 2007; Tobler et al., 2005), the reward probability (Fiorillo et al., 2003; Hart et al., 2015), and the delay before a reward is delivered (Day et al., 2010; Fiorillo et al., 2008; Roesch et al., 2007).

In addition to conveying prospective reward-related information, cues can also signal retrospective reward-related information. Consider the scenario where there are two lemonade stands in the neighborhood: one is a short distance from your house on the corner and the other is a long distance from your house on a cul-de-sac. Both cues (the corner and the cul-de-sac) signal the identical prospective reward-related information (a cup of lemonade), but differ in the retrospective-reward related information that is conveyed (time elapsed and effort exerted before observing the cue). Given the role of dopamine encoding prospective reward-related information, could cue-evoked dopamine release also convey retrospective reward-related information, such as elapsed time? The dopamine response to both cues and rewards can be influenced by the passage of time (Fiorillo et al., 2008; Pasquereau and Turner, 2015; Soares et al., 2016; Starkweather et al., 2017), though it is unknown if the cue-evoked dopamine response encodes retrospective time-related information. Specifically, is there a difference in the dopamine response between a cue that always follows the previous reward with a short delay (Short Wait cue) and a cue that always follows the previous reward with a long delay (Long Wait cue)?

We performed voltammetry recordings of dopamine release in the nucleus accumbens (NAc) core during Pavlovian conditioning paradigms in which distinct cues solely conveyed differences in the time elapsed since the previous reward (wait time), as the reward size, reward probability, and delay to reward delivery were identical between the cues. As such, we could determine whether cue-evoked dopamine release encodes (i) prospective reward-related information exclusively, (ii) sunk costs, or (iii) the reward rate. For example, if cue-evoked dopamine release only conveys prospective information there would be no difference in the dopamine response between Short Wait and Long Wait cues, as both cues denote the delivery of an identical reward. If cue-evoked dopamine release signals sunk costs we would expect a larger dopamine response to the Long Wait cue. In support, reward-evoked dopamine neuron firing and release is enhanced following longer delays (Fiorillo et al., 2008; Wanat et al., 2010). Therefore, a cue paired with rewards delivered after long delays could evoke a greater dopamine response relative to a cue paired with rewards delivered after short delays. Finally, if cue-evoked dopamine release signals the reward rate we would expect a larger dopamine response to the Short Wait cue, as the short temporal interval from the previous trial results in a higher rate of reward.

Results

Cue-evoked dopamine release encodes wait time

Voltammetry recordings of dopamine levels in the NAc core were performed on rats trained on Pavlovian conditioning paradigms where distinct audio conditioned stimuli (CSs) solely signaled the time elapsed since the previous reward (Figure 1A). On Short Wait trials the CS was presented 15 – 25 s following the previous reward delivery. On Long Wait trials the CS was presented 65 – 75 s following the previous reward delivery. Therefore, the Short Wait CS and the Long Wait CS signal the identical prospective reward-related information (same reward size, probability, and delay to reward delivery), but differ in the retrospective temporal information that is conveyed (time elapsed since the previous reward).

Figure 1.

Figure 1

Open in a new tab

CS-evoked dopamine release encodes wait time. (A) Illustration of Intermixed training paradigm using distinct audio CSs for the different trial types. Short and Long Wait trials were presented in a pseudorandom pattern. (B) Representative color plots and dopamine current on a Short Wait (left) and Long Wait trial (right). Voltammetry recordings of dopamine release in the NAc core were performed in well-trained rats (> 24 sessions). (C) Wait time dopamine response prior to the CS presentation. (D) CS dopamine response normalized to pre-CS levels (* p < 0.05, paired t-test). (E) Hazard rate prediction of CS dopamine response as a function of wait time duration. (F) CS dopamine response normalized to pre-CS levels as a function of wait time duration (** p < 0.01, effect of trial type). (G) CS dopamine response normalized to trial start as a function of wait time duration (*** p < 0.001, effect of trial type). Data are represented as mean ± SEM.

Dopamine levels were examined in rats trained to experience both Short and Long Wait trials together in the same context (Intermixed training). Intermixed training sessions were comprised of 25 Short Wait and 25 Long Wait trials, presented in a pseudorandom pattern so that the identity of the upcoming trial could not be predicted (Figure 1A). Our results illustrate two components of the dopamine response: a decrease in dopamine levels throughout the wait time and an increase in dopamine release to the CS (Figure 1B–D). Wait time dopamine levels decayed according to an exponential function for the first 15 s (prior to the first possible CS presentation) and by a linear function thereafter in Long Wait trials (Figure 1C, Figure S1). This reduction in wait time dopamine levels is consistent with the time-dependent decrease in dopamine neuron firing prior to the presentation of a CS (Bromberg-Martin et al., 2010; Pasquereau and Turner, 2015), as well as with the cumulative decrease in NAc dopamine levels observed across trials and throughout behavioral sessions (Bassareo et al., 2015; Oleson et al., 2014). Inhibiting dopamine neuron activity accelerates the subjective estimate of elapsed time (Soares et al., 2016), which suggests the decrease in wait time dopamine levels could signal the passage of time.

To quantify the CS dopamine response while accounting for differences in wait time dopamine levels, we calculated the average dopamine response during the 5 s CS relative to the average dopamine response during the preceding 5 s. The Short Wait CS elicited a greater dopamine response relative to the Long Wait CS (paired t-test: t9 = 3.1, p = 0.013, n = 10 electrodes; Figure 1D), consistent with the interpretation that CS-evoked dopamine release signals retrospective temporal information to reflect the relative reward rate.

Although these results suggest dopamine encodes wait time, an alternative possibility is that the CS dopamine response reflects the hazard rate, or rather the likelihood a CS will be presented as a function of time. If the dopamine response were driven by the hazard rate, CS-evoked dopamine release should decrease with longer wait times as the presentation of the CS becomes more expected (Figure 1E). However, there was no systematic change in CS-evoked dopamine levels with increasing wait times during Short Wait trials (15 – 25s wait time; r2 = 0.00, p = 0.79) or Long Wait trials (65 – 75s wait time; r2 = 0.00, p = 0.92; two-way repeated measures ANOVA: effect of trial type, F(1,27) = 12.8, p = 0.0013; Figure 1F). Since the CS dopamine response is superimposed upon changes in wait time dopamine levels, we additionally examined if the CS dopamine response reflected the hazard rate when calculated as a difference relative to the trial start. However, there was no relationship between the wait time duration and the CS dopamine response calculated in this manner (Short Wait trials: r2 = 0.00, p = 0.70; Long Wait trials: r2 = 0.02, p = 0.42; two-way repeated measures ANOVA: effect of trial type, F(1,27) = 129.5, p < 0.0001; Figure 1G).

While CS-evoked dopamine release does not reflect changes in the hazard rate in 5 s increments, the difference in the dopamine response between Short and Long Wait CSs could be mediated by changes in the hazard rate over longer time periods (> 40 s). To address this possibility, a cohort of rats were trained in an identical manner except that the same CS was used for both Short and Long Wait trials (Intermixed training-Same CS; Figure 2A). Observing a difference between trial types with this training paradigm would indicate the CS dopamine response is driven by the hazard rate, as the CS does not provide explicit information about the wait time. Rats trained on the Intermixed-Same CS paradigm exhibited a decrease in wait time dopamine levels that decayed by an exponential function for the first 15 s and by a linear function thereafter (Figure 2B, Figure S1). However, there was no difference in the dopamine response between Short Wait and Long Wait trials when the same CS was used for both trial types (paired t-test, t6 = 1.2, p = 0.29, n = 7 electrodes; Figure 2C). Furthermore, the CS-evoked dopamine response for each trial type did not reflect changes in the hazard rate (Figure 2D), as increasing wait times were not associated with a reduction in the CS dopamine response calculated as a difference relative to pre-CS dopamine levels (Short Wait trials: r2 = 0.07, p = 0.23; Long Wait trials: r2 = 0.00, p = 0.81; Figure 2E), or to dopamine levels at the trial start (Short Wait trials: r2 = 0.03, p = 0.46; Long Wait trials: r2 = 0.03, p = 0.44; two-way repeated measures ANOVA: effect of trial type, F(1,18) = 45.6, p < 0.0001; Figure 2F). These results demonstrate the CS dopamine response is not driven by changes in the hazard rate, but rather encodes retrospective temporal information to relay the relative reward rate.

Figure 2.

Figure 2

Open in a new tab

CS-evoked dopamine release is not reflecting temporal expectancy. (A) Illustration of Intermixed-Same CS training paradigm using the same CS for both trial types. Short and Long Wait trials were presented in a pseudorandom pattern. (B) Wait time dopamine response prior to the CS presentation. (C) CS dopamine response normalized to pre-CS levels. (D) Hazard rate prediction of CS dopamine response as a function of wait time duration. (E) CS dopamine response normalized to pre- CS levels as a function of wait time duration. (F) CS dopamine response normalized to trial start as a function of wait time duration (*** p < 0.001, effect of trial type). Data are represented as mean ± SEM.

Dopamine encodes wait time in a context-independent manner

For dopamine to accurately assign reward-related information to a particular cue, the dopamine system must be able to differentiate between cues that have never been experienced together in the same context. To determine if CS-evoked dopamine release encodes wait time in a context-independent manner, a cohort of rats was trained to experience Short Wait trials and Long Wait trials in separate sessions, with voltammetry recordings for each session performed on different days (Separate training, Figure 3A). As with the Intermixed paradigms, dopamine levels decayed throughout the wait time in rats trained with the Separate paradigm (Figure 3B, Figure S1). Furthermore, the Short Wait CS elicited a greater dopamine response relative to the Long Wait CS, demonstrating that dopamine encodes wait time in a context-independent manner (paired t-test: Separate: t12 = 2.4, p = 0.031, n = 13 electrodes; Figure 3B,C). The magnitude of the CS dopamine response differed across the Pavlovian training paradigms (two-way repeated measures ANOVA: effect of training paradigm F(2,28) = 4.0, p = 0.030; effect of trial type F(1,28) = 11.3, p = 0.0024; Figure 3D), with greater dopamine release observed during sessions where two distinct CSs were presented (post-hoc Tukey’s test: Intermixed vs Separate training sessions t28 = 2.8, p = 0.028). There was no difference in the histological location or characteristics of the voltammetry electrodes between the groups (Figure S2), illustrating that the magnitude of CS dopamine response relates to the context in which the CS is experienced.

Figure 3.

Figure 3

Open in a new tab

Dopamine encodes differences between Long and Short Wait cues in a context-independent manner but does not mediate a corresponding difference in conditioned responding. (A) Illustration of Separate training paradigm where sessions consist of only a single trial type. (B) Wait time dopamine response prior to the CS presentation. (C) CS dopamine response normalized to pre-CS levels (* p < 0.05, paired t-test). (D) CS-evoked dopamine response across the different training paradigm (* p < 0.05, paired t-test or Tukey’s post-hoc test). (E) Conditioned responding across the different training paradigms. Data are represented as mean ± SEM.

Cue-evoked dopamine release and conditioned responding

To assess the relationship between behavior and CS-evoked dopamine release, we quantified conditioned responding in an identical manner to the analysis for the CS dopamine response. Specifically, conditioned responding was calculated as the rate of head entries into the food port during the 5 s CS relative to the preceding 5 s. Although conditioned responding increased across training sessions (Figure S2), there was no difference in the level of responding between Short and Long Wait trials in rats trained under the Intermixed, Separate, or Intermixed-Same CS paradigms (two-way repeated measures ANOVA: effect of training paradigm F(2,20) = 2.2, p = 0.14; effect of trial type F(1,20) = 0.04, p = 0.85; Intermixed: paired t-test, t6 = 1.0, p = 0.36, n = 7 rats; Separate: paired t-test, t9 = 0.7, p = 0.53, n = 10 rats; Intermixed-Same CS: Wilcoxon matched pairs, z = 0.1, p = 0.09, n = 6 rats; Figure 3E). Thus, a difference in dopamine release between cues does not translate into corresponding difference in conditioned responding.

Activating dopamine neurons during the presentation of a reward-predictive cue can subsequently alter cue-based actions (Hamid et al., 2016; Steinberg et al., 2013). Therefore, a change in dopamine release to a particular cue could mediate a corresponding change in behavior towards that cue. To address this possibility we capitalized on the ability of the dopamine system to alter the magnitude of the cue-evoked dopamine response depending upon the context in which the cue was experienced (Cone et al., 2016; Papageorgiou et al., 2016). Rats were initially exposed to Short Wait trials and Long Wait trials in separate contexts (Separate training) before experiencing both trial types within the same context (Intermixed training; Figure 4A). Switching rats from the Separate to Intermixed training paradigm selectively enhanced dopamine release to the Short Wait CS (paired t- test, t11 = 5.0, p = 0.0004), without affecting dopamine release to the Long Wait CS (paired t-test, t11 = 1.0, p = 0.32, n = 12 electrodes; Figure 4B–D). A Second Intermixed session resulted in no additional change in CS-evoked dopamine release (paired t-tests: Short Wait CS, t11 = 1.2, p = 0.24; Long Wait CS, t11 = 1.3, p = 0.23; Figure 4E). Furthermore, Pavlovian responding mirrored changes in the CS- evoked dopamine response, as conditioned responding to the Short Wait CS was enhanced during the First Intermixed session relative to the preceding Separate training sessions (paired t-tests: Short Wait CS, t8 = 2.4, p = 0.04, Long Wait CS, t8 = 1.6, p = 0.14), with no further change in responding during the Second Intermixed session (paired t-tests: Short Wait CS, t8 = 0.7, p = 0.53, Long Wait CS, t8 = 1.9, p = 0.10, n = 9 rats; Figure 4F–G). Collectively, these results illustrate that conditioned responding is not related to a difference in the dopamine response between cues, but rather is related to when the dopamine response towards a specific cue changes.

Figure 4.

Figure 4

Open in a new tab

Dopamine release and conditioned responding update in a CS-specific manner upon altering the context in which Short and Long Wait trials are experienced (A) Training paradigm. (B–C) Dopamine release to Short and Long Wait CSs under different training contexts. (D) Elevated dopamine release to the Short Wait CS during the First Intermixed session following Separate training sessions (*** p < 0.001, paired t-test). (E) No change in CS-evoked dopamine release between the First and Second Intermixed training sessions. (F) Elevated conditioned responding to the Short Wait CS during the First Intermixed session following Separate training sessions (* p < 0.05, paired t-test). (G) No change in conditioned responding between the First and Second Intermixed training sessions. Data are represented as mean ± SEM.

Trial history influences the dopamine response to the CS and wait time in opposing directions

The selective increase in dopamine release to the Short Wait CS during the First Intermixed training session could be mediated by the recent trial history coupled with the novel context of experiencing both trial types together within the same session (Cone et al., 2016; Nakahara et al., 2004; Papageorgiou et al., 2016). To assess the influence of trial history on CS dopamine release we performed a multiple linear regression analysis with the current trial and four previous trials as independent variables and dopamine release on the current trial as the dependent variable (n = 624 trials). This analysis identified the current trial and the preceding trial as the only significant factors contributing to the observed CS dopamine response (F(5,592) = 16.0, p < 0.001, r2 = 0.11; Best fit equation: dopamine (nM) = 6.35 – 0.12 * (current trial wait time) + 0.033 * (previous trial wait time)). In particular, dopamine release to the Short Wait CS was greater when preceded by a Long Wait trial (Long Wait → Short Wait) than when preceded by a Short Wait trial (Short Wait → Short Wait) (paired t-test t12 = 2.9, p = 0.0124, n = 13 electrodes; Figure 5A). While the preceding trial influences the CS dopamine response during the First Intermixed training session, this effect was absent on a subsequent training session and in rats exclusively trained using the Intermixed paradigm (Figure S3). Additionally, the dopamine response to the Short Wait CS was greater when experienced during Intermixed training sessions relative to the Separate training sessions (Figure 4, Figure S3). These results illustrate that the dopamine response to the Short Wait CS is influenced by the identity of preceding trial as well as by the context in which the CS is experienced. To evaluate the relationship between the change in CS dopamine release and conditioned responding trials were binned according to the wait time duration on the current trial as a function of the preceding trial (Short Wait or Long Wait) and the training context (Separate or First Intermixed). Conditioned responding correlated with dopamine release to the Short Wait CS (r2 = 0.79, p = 0.001; Figure 5B), and not to the Long Wait CS (r2 = 0.01, p = 0.86), demonstrating that an increase in CS-evoked dopamine release is related to a corresponding elevation in CS-evoked behavior.

Figure 5.

Figure 5

Open in a new tab

The preceding trial influences CS-evoked dopamine release and wait time dopamine levels in opposing directions. (A) The identity of the preceding trial influenced the dopamine response to the Short Wait CS during the First Intermixed training session (* p < 0.05, paired t-test). (B) Conditioned responding is related to a CS-specific change in dopamine release. (C) Average dopamine response during the current Short Wait trial (left) and the current Long Wait trial (right) as a function of the identity of the preceding trial. (D) Wait time dopamine response in 2.5 s bins. Grey overlay denotes temporal window where the identity of the current trial is unknown. Blue overlay denotes temporal window experienced only during Long Wait trials. The previous trial influences the magnitude of the wait dopamine response when the identity of the current trial is unknown (*, *** p < 0.05, p < 0.001; effect of previous trial). (E) Head entries during the wait time in 2.5 s bins. The previous trial influences the number of head entries during the period of time only experienced during Long Wait trials. (* p < 0.05, effect of previous trial). Data are represented as mean ± SEM.

An observed increase in dopamine release can be mediated by a reduction in baseline dopamine levels (Hamid et al., 2016). As such, the influence of the preceding trial on the CS dopamine response could be driven by a change in wait time dopamine levels and/or a change in the CS-evoked dopamine response. To address this we examined how the preceding trial influenced wait time dopamine levels on the current trial focusing on two phases of the wait time: the early phase when the identity of the current trial was unknown (grey overlay) and the late phase that was experienced only during Long Wait trials (blue overlay, Figure 5). During the First Intermixed session, wait time dopamine levels during the early phase were reduced if the preceding trial was a Short Wait trial (current Short Wait trial: two-way repeated measures ANOVA, effect of time F(5,60) = 30.2, p < 0.0001, effect of previous trial F(1,12) = 22.5, p = 0.0005; current Long Wait trial: two-way repeated measures ANOVA, effect of time F(5,60) = 23.1, p < 0.0001, effect of previous trial F(1,12) = 9.1, p = 0.01; n = 13 electrodes; grey overlay, Figure 5C–D, Figure S4). In contrast, the previous trial did not affect the dopamine response during the late phase of the wait time (two-way repeated measures ANOVA, effect of time F(14,168) = 16.1, p < 0.0001, effect of previous trial F(1,12) = 0.0, p = 0.90; blue overlay, Figure 5D).

These results demonstrate that the preceding trial influences wait time dopamine levels and CS dopamine release in opposing directions. For example, during the First Intermixed session there was a larger dopamine response to the Short Wait CS when preceded by a Long Wait trial (Long Wait → Short Wait; Figure 5A). If this effect were mediated by a selective change in wait time dopamine levels, we would anticipate wait time dopamine levels would be lower on Long Wait → Short Wait trials relative to Short Wait → Short Wait trials. However, we observed the opposite result as wait time dopamine levels were higher on Long Wait → Short Wait trials (Figure 5C–D), illustrating that wait time dopamine levels and CS-evoked dopamine release are independent processes.

Silencing midbrain dopamine neurons accelerates the estimation of elapsed time (Soares et al., 2016). As such, a greater reduction in wait time dopamine levels could similarly reflect an accelerated subjective perception of time, thereby altering behavioral responding. While the preceding trial influenced dopamine levels during the early phase of the wait time (Figure 5D), there was no effect on the number of head entries during this temporal window (current Short Wait trial: two-way repeated measures ANOVA, effect of time F(5,45) = 36.9, p < 0.0001, effect of previous trial F(1,9) = 1.3, p = 0.28; current Long Wait trial: two-way repeated measures ANOVA, effect of time F(5,45) = 21.2, p < 0.0001, effect of previous trial F(1,9) = 3.0, p = 0.11; n = 10 rats; grey overlay, Figure 5E). In contrast, rats performed more head entries during the late phase of the wait time on trials that were preceded by a Short Wait trial (two-way repeated measures ANOVA, effect of time F(14,126) = 3.7, p < 0.0001, effect of previous trial F(1,9) = 8.3, p = 0.0183; blue overlay, Figure 5C). These results suggest a greater reduction in dopamine levels during the early phase of the wait time could promote an increase in anticipatory responding during the late phase of the wait time. Notably, the influence of the preceding trial on wait time dopamine levels and behavioral responding was evident only during the First Intermixed session (Figure 5) and was absent in a subsequent training session or in rats exclusively trained on the Intermixed paradigm (Figure S5). Therefore, the recent trial history influences wait time dopamine levels and CS dopamine release only when the training context changes.

Discussion

To determine if cue-evoked dopamine release encodes (i) prospective reward-related information exclusively, (ii) sunk costs, or (iii) the reward rate, we recorded dopamine transmission in behavioral tasks where distinct cues solely conveyed differences in the time elapsed since the previous reward (wait time). Our results demonstrate that cue-evoked dopamine release in the NAc core encodes differences in the wait time to reflect the relative rate of reward in a context-independent manner. Theoretical and experimental data highlight that tonic dopamine levels scale with the reward rate (Hamid et al., 2016; Niv et al., 2007). Our data complements these findings and illustrates that the phasic dopamine response to cues also reflects a subjective assessment of the rewards earned per time. This interpretation that cue-evoked dopamine release encodes the reward rate is a unifying principle that could account for the differential dopamine responses observed in paradigms where cues signal different reward sizes, reward probabilities, or delays in the time before a reward is delivered (Day et al., 2010; Fiorillo et al., 2008; Fiorillo et al., 2003; Gan et al., 2010; Hart et al., 2015; Roesch et al., 2007; Tobler et al., 2005).

Increasing evidence illustrates the midbrain dopamine system is both sensitive to elapsed time and required for accurate temporal discrimination. For example, the dopamine response to cues can be influenced by the hazard rate, though this has been observed over short time scales (< 2s) and in response to cues functioning as a temporal discrimination signal (Pasquereau and Turner, 2015; Soares et al., 2016). In contrast, we found that when cues encode the wait time there was no influence of the hazard rate on cue-evoked dopamine release. Therefore the influence of the hazard rate on cue-evoked dopamine transmission likely depends upon on the duration of the temporal interval as well as the information that is signaled by the cue.

Midbrain dopamine neurons are required for accurate temporal discrimination (Soares et al., 2016), though elapsed time is not necessarily reflected in the cue-evoked dopamine response, which suggests dopamine signals the passage of time prior to when a cue is presented. Indeed, dopamine neuron firing decreases prior to the presentation of reward-predictive cues (Bromberg-Martin et al., 2010; Pasquereau and Turner, 2015) and dopamine levels decrease throughout the wait time in all variants of the Pavlovian conditioned paradigm we examined. Since inhibiting dopamine neuron activity accelerates the subjective estimate of elapsed time (Soares et al., 2016), a decrease in wait time dopamine levels could signify the passage of time. In support, a greater reduction in dopamine levels during the early phase of the wait time was associated with a subsequent increase in anticipatory head entries.

The dopamine response to a cue is thought to represent the incentive salience towards the cue, thereby influencing cue-driven behaviors (Berridge and Robinson, 1998). Activating dopamine neurons during the cue presentation can subsequently alter behavioral outcomes (Hamid et al., 2016; Steinberg et al., 2013). Similarly, we found that an increase in cue-evoked dopamine release was related to an increase in conditioned responding. This change in the cue-evoked dopamine response occurred without increasing the reward size and was driven by the recent trial history as well as by altering the context in which the cue was experienced. Furthermore, the noted differences in the magnitude of cue-evoked dopamine release across the different Pavlovian conditioning paradigms illustrate that the parameters of a particular task can heavily influence the level of engagement by the dopamine system in response to reward-predictive cues (Clark et al., 2013; Flagel et al., 2011).

Although the dopamine response between cues is often related to behavioral outcomes, dopamine is not always sufficient for mediating cue-elicited actions (Gan et al., 2010; Hollon et al., 2014; Roesch et al., 2007). Our results illustrate that the difference in dopamine release between Short and Long Wait cues was not related to a difference in Pavlovian conditioned responding. However, it is possible that the dopamine response between cues is critical for mediating a differential behavioral responding in operant assays. Ultimately, the manner by which increases and decreases in dopamine levels influence behavior ultimately depends upon the local circuit in which dopamine acts. Projections to the NAc from the prefrontal cortex and amygdala are also involved with behavioral responding to cues (Ambroggi et al., 2008; Otis et al., 2017). Therefore the capacity for NAc dopamine to influence behavior during the wait time or the cue likely depends upon (i) coincident input from specific projections to the NAc and (ii) the net effect of these converging inputs on medium spiny neurons.

Our data highlights that elapsed time is encoded by the dopamine response to cues. While cue-evoked dopamine release conveys a future estimate of reward value, our findings demonstrate that dopamine also signals retrospective temporal information. As such, the cue-evoked dopamine response can reflect an integration of both retrospective and prospective reward-related information to signal the reward rate.

Experimental Procedures

Subjects and surgery

All procedures were approved by the Institutional Animal Care and Use Committee at the University of Washington and the University of Texas at San Antonio. Male Sprague-Dawley rats were pair-housed upon arrival, maintained on a 12-hour light/dark cycle, and given ad libitum access to water and lab chow. Surgeries were performed under isoflurane on rats weighing 300–350g (~9–12 weeks old), after which rats were single-housed. Carbon-fiber electrodes targeting the NAc core (relative to bregma: 1.3 mm anterior; ±1.3 mm lateral; 7.0 mm ventral) and a Ag/AgCl reference electrode were implanted for voltammetry recordings.

Behavioral procedures

All procedures were performed during the light cycle. After > 1 wk recovery from surgery, rats were placed and maintained on mild food restriction (~15 g/day standard lab chow) to target 90% free-feeding weight, allowing for an increase in weight of 1.5% per week. Prior to Pavlovian conditioning, rats underwent a magazine training session where 20 food pellets (45 mg, Bio-Serve) were delivered with a 90 s inter-trial interval (ITI). Pavlovian sessions consisted of 50 trials where the termination of a 5 s audio CS (tone or whitenoise, counterbalanced across animals) resulted in the delivery of a single food pellet and illumination of the food port light for 4.5 s. The Short Wait CS was presented after a 20 ± 5 s ITI and the Long Wait CS was presented after a 70 ± 5 s ITI. Pavlovian sessions using the Intermixed or Intermixed-Same CS paradigm consisted of 25 Short Wait and 25 Long Wait trials, presented in a pseudorandom pattern. The Separate training paradigm involved 50 trial sessions where only a single trial type was presented, with the order of Short and Long Wait sessions occurring in a pseudorandom pattern. Rats either received 30 sessions of Intermixed training or 28 sessions of Separate training followed by 2 sessions of Intermixed training. Voltammetry recordings were performed during sessions 25–30 (one session per day) after rats established stable conditioned responding (Figure S2).

Voltammetry recordings

Chronically-implanted electrodes were connected to a head-mounted voltammetric amplifier for dopamine detection in behaving rats using fast-scan cyclic voltammetry as described previously (Clark et al., 2010; Wanat et al., 2013; Wanat et al., 2010). Chemical verification of dopamine was achieved by obtaining a high correlation of the cyclic voltammogram during a reward-related event to that of a dopamine standard (r2 ≥ 0.75). The voltammetry data and corresponding behavioral data for a session were not analyzed if the detected voltammetry signal did not satisfy the chemical verification criteria, identical to the exclusion criteria used in prior studies (Wanat et al., 2013; Wanat et al., 2010). The characteristics of the electrode background current were no different between experimental groups (Figure S2).

Data analysis

Dopamine was isolated from the voltammetric signal using chemometric analysis (Heien et al., 2005) with a standard training set of stimulated dopamine release detected by chronically implanted electrodes, as has been used in previous studies (Wanat et al., 2013; Wanat et al., 2010). The dopamine concentration was estimated based upon the average post-implantation sensitivity of electrodes (34 nA/μM) (Clark et al., 2010). The CS dopamine response was quantified by calculating the average dopamine response during the 5 s CS relative to the average dopamine response during the preceding 5 s or dopamine levels at the trial start. Pavlovian conditioned responding was calculated as the difference in the rate of head entries to the food port during the 5 s CS minus the head entry rate during the preceding 5 s. The multiple linear regression analysis used the entry method with CS-evoked dopamine release as the dependent variable and the wait time duration on the current and four preceding trials as independent variables. The number of animals/electrodes used per experiment was determined by a power analysis with an alpha level of 0.05 and power level of 0.8, using the effect size and variance estimated from preliminary data. Normality was assessed using the Kolmogorov-Smirnov test. If data failed this test, nonparametric statistical tests were performed. Significant effects were determined by Student’s t-tests, Wilcoxon ranked sum tests, ANOVAs, and linear regression analyses, with appropriate adjustments for differences in sphericity. Data were analyzed using MATLAB, Prism and SPSS. All relevant data are available from the authors.

Supplementary Material

s1

Highlights.

  • Cue-evoked dopamine release encodes retrospective temporal information

  • Dopamine signals differences between cues in a context-independent manner

  • Conditioned responding is not related to a difference in dopamine release between cues

  • Conditioned responding relates to a change in dopamine release to a given cue

Acknowledgments

This work was supported by National Institutes of Health grants DA033386 (M.J.W) and DA027858 (P.E.M.P). We thank S. Ng-Evans for technical support and C. Stelly and I. Oliva for critical input on the manuscript. The authors declare no competing financial interests.

Footnotes

Author contributions: M.J.W. and P.E.MP. designed experiments. M.J.W., K.M.F., and M.J.L. collected and analyzed data. M.J.W. wrote the manuscript.

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. Ambroggi F, Ishikawa A, Fields HL, Nicola SM. Basolateral amygdala neurons facilitate reward-seeking behavior by exciting nucleus accumbens neurons. Neuron. 2008;59:648–661. doi: 10.1016/j.neuron.2008.07.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Bassareo V, Cucca F, Frau R, Di Chiara G. Differential activation of accumbens shell and core dopamine by sucrose reinforcement with nose poking and with lever pressing. Behavioural brain research. 2015;294:215–223. doi: 10.1016/j.bbr.2015.08.006. [DOI] [PubMed] [Google Scholar]
  3. Berridge KC, Robinson TE. What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience? Brain research Brain research reviews. 1998;28:309–369. doi: 10.1016/s0165-0173(98)00019-8. [DOI] [PubMed] [Google Scholar]
  4. Bromberg-Martin ES, Matsumoto M, Hikosaka O. Distinct tonic and phasic anticipatory activity in lateral habenula and dopamine neurons. Neuron. 2010;67:144–155. doi: 10.1016/j.neuron.2010.06.016. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Clark JJ, Collins AL, Sanford CA, Phillips PE. Dopamine encoding of Pavlovian incentive stimuli diminishes with extended training. J Neurosci. 2013;33:3526–3532. doi: 10.1523/JNEUROSCI.5119-12.2013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Clark JJ, Sandberg SG, Wanat MJ, Gan JO, Horne EA, Hart AS, Akers CA, Parker JG, Willuhn I, Martinez V, et al. Chronic microsensors for longitudinal, subsecond dopamine detection in behaving animals. Nature methods. 2010;7:126–129. doi: 10.1038/nmeth.1412. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Cone JJ, Fortin SM, McHenry JA, Stuber GD, McCutcheon JE, Roitman MF. Physiological state gates acquisition and expression of mesolimbic reward prediction signals. Proc Natl Acad Sci. 2016;113:1943–1948. doi: 10.1073/pnas.1519643113. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Day JJ, Jones JL, Wightman RM, Carelli RM. Phasic nucleus accumbens dopamine release encodes effort- and delay-related costs. Biol Psychiatry. 2010;68:306–309. doi: 10.1016/j.biopsych.2010.03.026. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Fiorillo CD, Newsome WT, Schultz W. The temporal precision of reward prediction in dopamine neurons. Nature neuroscience. 2008;11:966–973. doi: 10.1038/nn.2159. [DOI] [PubMed] [Google Scholar]
  10. Fiorillo CD, Tobler PN, Schultz W. Discrete coding of reward probability and uncertainty by dopamine neurons. Science. 2003;299:1898–1902. doi: 10.1126/science.1077349. [DOI] [PubMed] [Google Scholar]
  11. Flagel SB, Clark JJ, Robinson TE, Mayo L, Czuj A, Willuhn I, Akers CA, Clinton SM, Phillips PE, Akil H. A selective role for dopamine in stimulus–reward learning. Nature. 2011;469:53–57. doi: 10.1038/nature09588. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Gan JO, Walton ME, Phillips PE. Dissociable cost and benefit encoding of future rewards by mesolimbic dopamine. Nature neuroscience. 2010;13:25–27. doi: 10.1038/nn.2460. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Hamid AA, Pettibone JR, Mabrouk OS, Hetrick VL, Schmidt R, Vander Weele CM, Kennedy RT, Aragona BJ, Berke JD. Mesolimbic dopamine signals the value of work. Nature neuroscience. 2016;19:117–126. doi: 10.1038/nn.4173. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Hart AS, Clark JJ, Phillips PE. Dynamic shaping of dopamine signals during probabilistic Pavlovian conditioning. Neurobiology of learning and memory. 2015;117:84–92. doi: 10.1016/j.nlm.2014.07.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Heien ML, Khan AS, Ariansen JL, Cheer JF, Phillips PE, Wassum KM, Wightman RM. Real-time measurement of dopamine fluctuations after cocaine in the brain of behaving rats. Proc Natl Acad Sci. 2005;102:10023–10028. doi: 10.1073/pnas.0504657102. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Hollon NG, Arnold MM, Gan JO, Walton ME, Phillips PE. Dopamine-associated cached values are not sufficient as the basis for action selection. Proc Natl Acad Sci. 2014;111:18357–18362. doi: 10.1073/pnas.1419770111. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Nakahara H, Itoh H, Kawagoe R, Takikawa Y, Hikosaka O. Dopamine neurons can represent context-dependent prediction error. Neuron. 2004;41:269–280. doi: 10.1016/s0896-6273(03)00869-9. [DOI] [PubMed] [Google Scholar]
  18. Niv Y, Daw ND, Joel D, Dayan P. Tonic dopamine: opportunity costs and the control of response vigor. Psychopharmacology. 2007;191:507–520. doi: 10.1007/s00213-006-0502-4. [DOI] [PubMed] [Google Scholar]
  19. Oleson EB, Cachope R, Fitoussi A, Tsutsui K, Wu S, Gallegos JA, Cheer JF. Cannabinoid receptor activation shifts temporally engendered patterns of dopamine release. Neuropsychopharmacology. 2014;39:1441–1452. doi: 10.1038/npp.2013.340. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Otis JM, Namboodiri VM, Matan AM, Voets ES, Mohorn EP, Kosyk O, McHenry JA, Robinson JE, Resendez SL, Rossi MA, et al. Prefrontal cortex output circuits guide reward seeking through divergent cue encoding. Nature. 2017;543:103–107. doi: 10.1038/nature21376. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Papageorgiou GK, Baudonnat M, Cucca F, Walton ME. Mesolimbic Dopamine Encodes Prediction Errors in a State-Dependent Manner. Cell reports. 2016;15:221–228. doi: 10.1016/j.celrep.2016.03.031. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Pasquereau B, Turner RS. Dopamine neurons encode errors in predicting movement trigger occurrence. Journal of neurophysiology. 2015;113:1110–1123. doi: 10.1152/jn.00401.2014. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Roesch MR, Calu DJ, Schoenbaum G. Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nature neuroscience. 2007;10:1615–1624. doi: 10.1038/nn2013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Salamone JD, Correa M. The mysterious motivational functions of mesolimbic dopamine. Neuron. 2012;76:470–485. doi: 10.1016/j.neuron.2012.10.021. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Soares S, Atallah BV, Paton JJ. Midbrain dopamine neurons control judgment of time. Science. 2016;354:1273–1277. doi: 10.1126/science.aah5234. [DOI] [PubMed] [Google Scholar]
  26. Starkweather CK, Babayan BM, Uchida N, Gershman SJ. Dopamine reward prediction errors reflect hidden-state inference across time. Nature neuroscience. 2017;20:581–589. doi: 10.1038/nn.4520. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Steinberg EE, Keiflin R, Boivin JR, Witten IB, Deisseroth K, Janak PH. A causal link between prediction errors, dopamine neurons and learning. Nature neuroscience. 2013;16:966–973. doi: 10.1038/nn.3413. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Tobler PN, Fiorillo CD, Schultz W. Adaptive coding of reward value by dopamine neurons. Science. 2005;307:1642–1645. doi: 10.1126/science.1105370. [DOI] [PubMed] [Google Scholar]
  29. Wanat MJ, Bonci A, Phillips PE. CRF acts in the midbrain to attenuate accumbens dopamine release to rewards but not their predictors. Nature neuroscience. 2013;16:383–385. doi: 10.1038/nn.3335. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Wanat MJ, Kuhnen CM, Phillips PE. Delays conferred by escalating costs modulate dopamine release to rewards but not their predictors. J Neurosci. 2010;30:12020–12027. doi: 10.1523/JNEUROSCI.2691-10.2010. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

s1
NIHMS898182-supplement-s1.pdf (6.1MB, pdf)

RESOURCES