Skip to main content
PMC home page
The Journal of Neuroscience logoLink to The Journal of Neuroscience
. 2010 Sep 8;30(36):12020–12027. doi: 10.1523/JNEUROSCI.2691-10.2010

Delays Conferred by Escalating Costs Modulate Dopamine Release to Rewards But Not Their Predictors

Matthew J Wanat 1, Camelia M Kuhnen 2, Paul E M Phillips 1,✉
PMCID: PMC2946195  NIHMSID: NIHMS234057  PMID: 20826665

Abstract

Efficient reward seeking is essential for survival and invariably requires overcoming costs, such as physical effort and delay, which are constantly changing in natural settings. Dopamine transmission has been implicated in decisions weighing the benefits and costs of obtaining a reward, but it is still unclear how dynamically changing effort and delay costs affect dopamine signaling to rewards and related stimuli. Using fast-scan cyclic voltammetry, we examined phasic dopamine release in the nucleus accumbens (NAcc) core and shell during reward-seeking behavior in rats. To manipulate the effort and time needed to earn a reward, we used instrumental tasks in which the response requirements (number of lever presses) were either fixed throughout a behavioral session [fixed ratio (FR)] or systematically increased from trial to trial [progressive ratio (PR)]. Dopamine release evoked by cues denoting reward availability was no different between these conditions, indicating insensitivity to escalating effort or delay costs. In contrast, dopamine release to reward delivery in both the NAcc core and shell increased in PR, but not in FR, sessions. This enhancement of reward-evoked dopamine signaling was also observed in sessions in which the response requirement was fixed but the delay to reward delivery increased, yoked to corresponding trials in PR sessions. These findings suggest that delay, and not effort, was principally responsible for the increased reward-evoked dopamine release in PR sessions. Together, these data demonstrate that NAcc dopamine release to rewards and their predictors are dissociable and differentially regulated by the delays conferred under escalating costs.

Introduction

Our actions are dynamically molded by reward-related decisions that weigh perceived benefits against perceived costs and involve multiple neurobiological processes (McClure et al., 2004), including the mesolimbic dopamine system (Wise, 2004; Schultz, 2006). For example, phasic dopamine neurotransmission encodes the size of anticipated rewards (benefit) when they are preceded by predictive stimuli (Tobler et al., 2005; Gan et al., 2010). However, less is known on how dopamine neurotransmission is modulated by the costs involved with obtaining a reward, such as the response requirement in instrumental behaviors or the delay until the reward is delivered. Importantly, the cost of obtaining rewards in natural settings is often dynamic; for instance, the time to explore and the distance to travel for a foraging animal constantly changes as a function of food availability, and these temporal and energetic costs are often intimately linked (Stephens and Krebs, 1986; Stevens et al., 2005; Berger-Tal et al., 2009). Although dopamine neurotransmission, elicited by both cue presentation and reward delivery, is sensitive to discrete switches in both the delay and the effort associated with attaining rewards (Roesch et al., 2007; Gan et al., 2010), it is not known how dopamine release to rewards and their predictors are affected when these costs are continuously changing.

Dynamic changes in the effort and delay costs to obtain rewards can be modeled in animals using progressive ratio (PR) reinforcement schedules, in which the number of lever presses required to earn a reward is systematically increased from trial to trial. To fulfill the escalating response requirement, the animal must exert more effort and endure longer delays on each successive trial. Thus, the PR instrumental paradigm incorporates compound effort and delay costs that escalate throughout a behavioral session. Under this instrumental schedule, animals behave in a manner that is consistent with their ability to track these compound costs to earn a reward (Wanchisen et al., 1988). Additionally, with PR reinforcement schedules, one can assess the breakpoint, or rather the highest response requirement an individual will overcome before ceasing responding. Previous studies have demonstrated a link between dopamine in the nucleus accumbens (NAcc) and performance during PR paradigms. Specifically, enhancing dopamine transmission elevates breakpoints (Zhang et al., 2003; Cagniard et al., 2006), and inhibiting dopamine signaling in the NAcc is associated with attenuated breakpoints (Aberman et al., 1998; Hamill et al., 1999). However, there are mixed findings over the relative contribution of the core and shell subregions of the NAcc. For instance, dopamine-specific lesions implicate the NAcc core but not the NAcc shell in overcoming high-effort constraints (Sokolowski and Salamone, 1998), whereas dopamine receptor antagonism implicates both the NAcc core and shell (Nowend et al., 2001). Indeed, phasic dopamine release can be differentially regulated between these subregions (Aragona et al., 2008, 2009).

Thus, in the current study, we examined phasic dopamine release in the NAcc core and shell to rewards and predictors of their availability during instrumental tasks under different reinforcement schedules. Specifically, in separate sessions, rats lever pressed for food rewards in discrete trials under either fixed costs [fixed ratio (FR sessions)], escalating effort and time costs (PR sessions), or escalating time with fixed-effort costs (low-effort yoked sessions). Within these tasks, rewards were available in discrete trials, which permitted the analysis of dopamine release to both reward-related cues and reward delivery. Using fast-scan cyclic voltammetry at chronically implanted microelectrodes (Clark et al., 2010), we found that phasic dopamine release to reward-availability cues in the NAcc core and shell was unaffected by escalating effort and delay costs. In contrast, increasing response requirements augmented reward-evoked dopamine release in the NAcc core and shell, which was a function of the delay to reward delivery, independent of effort-related costs.

Materials and Methods

Subjects and surgery.

All procedures were approved by the University of Washington Institutional Animal Care and Use Committee. Thirty-nine male Sprague Dawley rats (Charles River Laboratories) were used for this study. Animals were group housed, given ad libitum access to water and lab chow, and maintained on a 12 h light/dark cycle (lights on at 7:00 A.M.). For voltammetry electrode implantation surgeries, rats (∼300 g) were anesthetized with isoflurane, and holes were drilled in the skull for anchor screws, a Ag/AgCl reference electrode, and recording electrodes. The reference electrode was implanted, and carbon fiber electrodes were lowered into the NAcc core (coordinates relative to bregma: anteroposterior, 1.3 mm; mediolateral, 1.3 mm; dorsoventral, −7.0 mm) and/or the NAcc shell (anteroposterior, 1.5 mm; mediolateral, 0.6 mm; dorsoventral, −7.5 mm) and were secured with cranioplastic cement. Animals were allowed to recover for at least 3 weeks before beginning behavioral training. Of the 39 animals entering the study, 11 provided data for the NAcc core, seven for the NAcc shell, and two for both subregions (from bilateral electrodes). The remaining 19 animals were excluded as a result of postsurgical complications or not satisfying the criteria for reliable dopamine detection (see below). Electrode recording locations are presented in Figure 1.

Figure 1.

Figure 1.

Open in a new tab

Voltammetry recording sites. NAcc core recording locations are represented as filled black circles and NAcc shell locations as filled gray circles. Figures modified from Paxinos and Watson (2005).

Behavioral training.

Rats were placed on mild food restriction until they reached ∼90% free-feeding weight (∼2–3 d). Rats were maintained on food restriction throughout the experiment and were given standard lab chow (∼15 g/d) in the home cage to allow for an increase in weight of 1.5% per week. Experimental 45 mg food pellets (F0021; BioServ) were placed in their home cages on the day before the first training session to familiarize the rats with the food pellets. Operant chambers (Med Associates) had sloped floors and a house light and contained a food tray and two cue lights above two retractable levers on a single wall. The cue lights and their corresponding levers were located on either side of the food tray. All behavioral sessions commenced with illumination of the house light. On the first training session (single-lever training), one lever extended with its corresponding cue light illuminated. The lever would remain extended throughout the session, and a single lever press would deliver a food pellet (maximum of 100 pellets earned within 90 min). In some instances, inaccessible food pellets were placed behind the lever to promote lever pressing. After successful completion of the single-lever training session for one lever, rats were then trained to lever press on the opposite lever using the same training conditions. In the following session (pseudorandom single-lever training), the side of the active lever was alternated between trials within a session in which a lever press on the active lever would cause the delivery of a food pellet, retraction of the lever, and the cue light to turn off for a 15 s intertrial interval (ITI). A pseudorandom pattern dictated which side (left or right) was the active lever for a given trial. The pseudorandom single-lever training ended after the successful completion of 80 trials within 2 h. Finally, rats were trained to lever press multiple times for a single food pellet in FR sessions consisting of 60 trials in which the active lever side remained constant throughout the session. FR sessions began with both levers (active and inactive) extending and illumination of the house light and the cue light over the active lever. Completion of the correct number of lever presses led to a pellet delivery, retraction of the levers, and the cue and house lights turning off for a 30 s ITI. Rats were trained under an FR4 reinforcement schedule (four lever presses to complete a trial), first with one lever active and then with the opposite lever active in separate sessions. In a similar manner, rats were then trained using an FR8 reinforcement schedule. To habituate rats to the recording procedure, they performed a single FR4 session with voltammetry recordings, although data from this session were not included in the analysis.

Behavioral sessions.

After successful completion of the training described above, rats then performed a single session per day according to the schedule in Figure 2A [i.e., a single FR4 session (session 1) would be performed on the first experimental day, and a single PR session (session 2) would be performed on a subsequent experimental day]. FR4 sessions were the same to those during training sessions with 60 pellets earned typically over 30–35 min. PR sessions were identical to FR4 sessions except that the operant requirement on each trial (T) is the integer (rounded down) of 1.4(T − 1) lever presses, starting at one lever press (i.e., 1, 1, 1, 2, 3, 5, 7, 10, 14, 20, 28, 40, 56, 79, 111, 155, 217, 304, 426). PR sessions ended after 15 min elapsed without the subject completing the response requirement in a trial. In a preliminary set of animals, we found that PR sessions lasted ∼30 min under this schedule, to which the duration of FR4 sessions was matched. To minimize inflexible behaviors, we interleaved FR4 sessions between PR sessions and alternated the side of the active lever (Fig. 2A). Voltammetry data were collected from PR sessions and the FR4 sessions immediately preceding PR sessions (see below). A subset of animals (n = 5) were tested on a low-effort yoked session that was used to discern the effects of effort and time on dopamine release to reward delivery in PR sessions. The low-effort yoked sessions were similar in structure to PR sessions, except that (1) the required effort to complete a trial remained constant throughout the session (FR1), (2) the time until pellet delivery in a trial was modeled after the average trial duration to complete that same number trial in the PR sessions, and (3) the session ended after 18 trials. These low-effort yoked sessions were assessed on days in which a PR session would normally be performed (Fig. 2A).

Figure 2.

Figure 2.

Open in a new tab

Training schedule and behavior on PR sessions. A, Schedule for behavioral session (n = 20 rats). B, Average number of food pellets earned for each PR session. C, Average breakpoint for each PR session. D, Percentage of the operant requirement of the previous trial that was completed on the last, breakpoint trial.

Recording sessions.

During experimental recording sessions, the chronically implanted carbon-fiber microelectrodes were connected to a head-mounted voltammetric amplifier for dopamine detection by fast-scan cyclic voltammetry as described in detail previously (Phillips et al., 2003a; Heien et al., 2005; Clark et al., 2010). In brief, the potential applied to the carbon fiber was ramped from −0.4 V (vs Ag/AgCl) to +1.3 V and back at a rate of 400 V/s during a voltammetric scan and held at −0.4 V between scans. Scans were repeated at a frequency of 10 Hz throughout the session. The application of this triangular waveform causes redox reactions in electrochemically active species at the carbon fiber (including dopamine: approximately +0.7 and −0.3 V peak oxidation and reduction potentials, respectively), which can be measured as changes in current. To ensure that electrodes were capable of detecting dopamine, unexpected food pellets were delivered before and after a recording session to elicit dopamine release (Clark et al., 2010). Chemical verification of dopamine was achieved by obtaining high correlation of the cyclic voltammogram (electrochemical signature) to that of a dopamine standard (correlation coefficient, r2 ≥ 0.75 by linear regression). The voltammetry data for a session were not analyzed if food pellet delivery did not elicit dopamine release that satisfied the chemical verification criteria.

Data analysis.

To be included for data analysis, animals must have a minimum of two FR4 and two PR sessions with dopamine release satisfying the chemical verification criteria described above. Voltammetric data analysis were performed using software written in LabVIEW and low-pass filtered at 2000 Hz. Dopamine was isolated from the voltammetric signal using chemometric analysis (Heien et al., 2005) using a standard training set of stimulated dopamine release detected by chronically implanted electrodes. Dopamine concentration was estimated based on the average postimplantation sensitivity of electrodes (Stuber et al., 2008; Clark et al., 2010). Data were smoothed using a 0.5 s moving average. Analysis of extracellular dopamine concentration was restricted to a period of 3 s after cue onset or reward delivery. Because of the close temporal relationship between cue onset and reward delivery in FR sessions, dopamine concentration to reward delivery in FR sessions was calculated by determining the peak change in dopamine levels in the 3 s after pellet delivery relative to the 0.5 s before reward delivery. Statistical analyses were performed in Prism (GraphPad Software) using linear regression or one or two-way ANOVA with repeated measures as needed, followed by post hoc Bonferroni's t tests. Data are presented as mean ± SEM.

Histology.

After completion of the experimental sessions, animals were anesthetized with ketamine/xylazine (100 mg/kg) and the recording site was marked by making a small electrolytic lesion at the electrode tip by passing a current (∼70 μA) through the carbon-fiber microelectrode for 20 s. Animals were subsequently perfused transcardially with physiological saline and then with 4% paraformaldehyde in PBS, before the brains were removed and postfixed in a paraformaldehyde solution. The brains were then placed in 30% sucrose solution in PBS for 48 h, flash frozen, and sectioned coronally (60 μm). All sections were mounted and stained with cresyl violet.

Results

In this study, we explored how phasic dopamine release to rewards and related stimuli was affected by changing effort and delay costs during instrumental behavior. Specifically, we examined dopamine release to the cues presented at the start of each trial and to reward delivery in the NAcc core and shell either with fixed response requirements, under a FR reinforcement schedule, or when effort and delay costs escalated under a PR reinforcement schedule. Animals were subjected to a single behavioral session per day with FR and PR sessions interleaved and the side of the active lever alternated between sessions (Fig. 2A). This design was used to minimize concerns related to satiety and to maintain flexible goal-directed behaviors.

Behavior

Behavioral performance was consistent across multiple days in FR sessions and PR sessions. As expected, rats completed all 60 trials in FR sessions (FR4, total of 60 pellets earned, n = 20 rats in 234 sessions). In PR sessions, the number of food pellets earned (F(3,72) = 0.7, p = 0.6, one-way ANOVA) (Fig. 2B) and the lever presses on the last, uncompleted (breakpoint) trial (F(3,72) = 0.4, p = 0.8, one-way ANOVA) (Fig. 2C) were unchanged over multiple PR sessions. On average, rats earned 14.4 ± 0.2 food pellets per PR session, corresponding to a breakpoint of 107.7 ± 8.2 lever presses for a single food pellet (n = 20 rats in 76 sessions) (Fig. 2B,C). Interestingly, on the breakpoint trial, rats completed 100.7 ± 3.3% of the operant requirement of the previous trial (F(7,67) = 1.7, p = 0.1, one-way ANOVA) (Fig. 2D), demonstrating that the behavior is stable even within a single PR session.

Dopamine release during fixed-ratio sessions

To control for aspects of dopamine transmission in instrumental tasks that do not relate to escalating response requirements, we first used an FR reinforcement schedule. Specifically, we examined phasic dopamine release in individual trials during FR sessions to the cues presented at the start of each trial (cue-light illumination and lever insertion) and to reward delivery. Throughout the session, the presentation of trial-onset cues consistently elicited phasic dopamine release (Fig. 3A). Although there was no difference in cue-evoked dopamine release between the NAcc core and shell, there was a notable trial-dependent decrease in dopamine release to the cue over the session (region effect: F(1,1180) = 1.4, p = 0.3; trial effect: F(59,1180) = 11.8, p < 0.0001, two-way repeated measures ANOVA; n = 13 core, n = 9 shell rats) (Fig. 3B). Cue-evoked dopamine release reached asymptotic levels by the third trial in FR sessions in both the NAcc core and shell (region effect: F(1,1180) = 1.7, p = 0.2; trial effect: F(57,1180) = 1.2, p = 0.2; for trials 3–60, two-way repeated measures ANOVA) (Fig. 3B). We also examined the change in dopamine release to the food pellet delivered at the completion of a trial and found that reward-evoked dopamine release was negligible in both the NAcc core and shell throughout FR sessions (region effect: F(1,1140) = 0.2, p = 0.6; trial effect: F(59,1140) = 1.3, p = 0.08, two-way repeated measures ANOVA) (Fig. 3A,C). Therefore, under fixed-effort and delay costs, reward-evoked dopamine release in the NAcc was minimal, whereas cue-evoked dopamine release rapidly attenuated and then remained stable throughout the behavioral session.

Figure 3.

Figure 3.

Open in a new tab

Dopamine release to trial-onset cues and reward delivery with fixed costs. A, Examples of dopamine release in the NAcc core to reward-related stimuli in the first, intermediate, and last trials. Current at the peak oxidation potential of dopamine is plotted as a function of time, with the inset showing the cyclic voltammogram identifying the detected current as dopamine. Below are two-dimensional pseudocolor plots of cyclic voltammograms over time. Rectangular dashed lines denote the cue onset, and square dashed lines denote the pellet delivery. B, Peak dopamine response in the NAcc core (n = 13) and shell (n = 9) to cue presentation. C, Peak dopamine response in the NAcc core and shell to pellet delivery.

Dopamine release during progressive-ratio sessions

After determining how reward-related stimuli elicited dopamine release in FR sessions, we next assessed the effect of escalating effort and delay costs on dopamine release to trial-onset cues in PR sessions. Similar to FR sessions, there was a trial-dependent decrease in cue-evoked dopamine release present in both the NAcc core and shell during PR sessions (trial effect: F(13,260) = 21.0, p < 0.0001 for trials 1–14; region effect: F(1,260) = 0.03, p = 0.9; two-way repeated measures ANOVA; n = 13 core, n = 9 shell rats) (Fig. 4A,B). Additionally, dopamine release to trial-onset cues reached stable levels after the first few trials throughout the NAcc (trial effect: F(10,200) = 1.6, p = 0.1; region effect: F(1,200) = 0.3, p = 0.6; for trials 4–14, two-way repeated measures ANOVA) (Fig. 4B). To assess cue-evoked dopamine release in relation to the decision to cease responding, it is necessary to examine the pattern of striatal dopamine release when referenced to the breakpoint trial. Therefore, we analyzed cue-evoked dopamine release when aligned to the last trial of the PR session and again found no effect of trial or difference between the NAcc core and shell on phasic dopamine release (trial effect: F(9,180) = 1.3, p = 0.2; region effect: F(1,180) = 0.4, p = 0.5; last 10 trials, two-way repeated measures ANOVA) (Fig. 4C). Importantly, a subsequent analysis determined that the pattern of cue-evoked dopamine release across trials is identical between FR and PR sessions in both the NAcc core (trial effect: F(13,312) = 25.0, p < 0.0001; session type: F(1,312) = 0.9, p = 0.4; analyzed for trials 1–14 with two-way repeated measures ANOVA; n = 13) (Fig. 5A) and NAcc shell (trial effect: F(13,208) = 23.0, p < 0.0001; session type: F(1,208) = 0.2, p = 0.7; analyzed for trials 1–14 with two-way repeated measures ANOVA; n = 9) (Fig. 5B). Together, these results suggest that escalating effort and delay costs do not influence dopamine release to trial-onset cues.

Figure 4.

Figure 4.

Open in a new tab

Dopamine release to trial-onset cues with escalating costs. A, Examples of dopamine release in the NAcc core to the presentation of trial-onset cues are shown for the first, intermediate, and last trials. Current at the peak oxidation potential of dopamine is plotted as a function of time, with the inset showing the cyclic voltammogram identifying the detected current as dopamine. Below are two-dimensional pseudocolor plots of cyclic voltammograms over time. Rectangular dashed lines denote the cue onset. B, C, Peak dopamine response in the NAcc core (n = 13) and shell (n = 9) to cue presentation as a function of trials aligned to the first trial (B) and the last trial (C).

Figure 5.

Figure 5.

Open in a new tab

No difference in dopamine release to trial-onset cues between FR and PR sessions in either the NAcc core or shell. Voltammetry data from the NAcc core (n = 13; A) and NAcc shell (n = 9; B) found in Figures 3 and 4 are replotted to highlight the lack of difference in dopamine release to trial-onset cues between these behavioral sessions.

In contrast to the cue-evoked response, there was a notable difference between FR and PR sessions in dopamine release to reward delivery. Specifically, reward-evoked dopamine release was negligible in FR sessions (Fig. 3C) but increased in magnitude in later trials during PR sessions (Fig. 6A). There was a significant effect of trial on reward-evoked dopamine release with no difference between the NAcc core and shell (trial effect: F(7,140) = 18.7, p < 0.0001; region effect: F(1,140) = 0.01, p = 0.9; last eight trials, two-way repeated measures ANOVA; n = 13 core, n = 9 shell rats) (Fig. 6B,C), suggesting that overcoming escalating effort and/or delay costs in PR sessions elicits more dopamine release to reward delivery.

Figure 6.

Figure 6.

Open in a new tab

Dopamine release to pellet delivery increases with escalating costs. A, Examples of dopamine release in the NAcc core to food pellet delivery in progressively later trials in a PR session. Current at the peak oxidation potential of dopamine is plotted as a function of time, with the inset showing the cyclic voltammogram identifying the detected current as dopamine. Below are two-dimensional pseudocolor plots of cyclic voltammograms over time. Rectangular dashed lines denote the pellet delivery. B, C, Peak dopamine response to pellet delivery increased with higher costs in both the NAcc core (n = 13; B) and NAcc shell (n = 9; C).

Dissociating the contribution of effort and delay costs on reward-evoked dopamine release

The costs in PR sessions comprise two components, those relating to the exerted effort and those associated with the delay to reward delivery (i.e., time to complete a trial). The delay is highly correlated with the response requirement (p < 0.0001, r2 = 0.93) (Fig. 7A), making it impossible, from the PR-task data, to discern whether effort and/or delay costs are driving the increase in reward-evoked dopamine release. Therefore, data were compared between the PR task and a low-effort yoked control task in which a single lever press yielded a food pellet delivered with a progressive delay matched to the behavior in PR sessions. In agreement with our previous results (Fig. 5), there was a significant effect of trial on cue-evoked dopamine release, but, importantly, there was no difference between PR, FR, and low-effort yoked sessions (trial effect: F(13,598) = 21.8, p < 0.0001; session effect: F(2,598) = 0.7, p = 0.5; analyzed for trials 1–14 with two-way repeated measures ANOVA; n = 22 for FR and PR, n = 5 for low-effort yoked). When assessing reward-evoked dopamine release, there was a significant effect of trial duration, but not session type, in both PR and low-effort yoked sessions (trial duration: F(2,575) = 15.0, p < 0.0001; session type: F(1,575) = 1.8, p = 0.2, two-way ANOVA) (Fig. 7B). Importantly, there was not a significant interaction between session type and trial duration on reward-evoked dopamine release (F(2,575) = 0.1, p = 0.9, two-way ANOVA) (Fig. 7B), indicating that trial duration in PR sessions accounts for the increasing reward-evoked dopamine release. Subsequent analyses identified a main effect of trial duration in both PR and low-effort yoked sessions (PR: F(2,526) = 49.3, p < 0.0001, one-way ANOVA, n = 529 trials from 20 rats; Low-effort yoked: F(2,49) = 6.5, p < 0.01, one-way ANOVA, n = 52 trials from 5 rats) (Fig. 7B), with significant differences found between all trial duration comparisons in PR sessions (***p < 0.001, Bonferroni's multiple comparison test) and a significant difference between short- and long-duration trials in low-effort yoked sessions (**p < 0.01, Bonferroni's multiple comparison test). Together, these findings demonstrate that increasing reward-evoked dopamine release in PR sessions was a function of time costs (i.e., trial duration) independent of costs resulting from exerted effort.

Figure 7.

Figure 7.

Open in a new tab

Temporal costs mediated elevated dopamine release to pellet delivery in PR sessions. A, Time to complete a trial and the number of lever presses required to complete a trial were highly correlated in PR sessions (n = 20 rats). B, Reward-evoked dopamine release in the NAcc scaled with increasing trial duration in both PR (n = 22) and low-effort yoked (n = 5) sessions. **p < 0.01, ***p < 0.001.

Discussion

Reward availability is variable in natural settings, which confers constantly changing costs that must be overcome to obtain rewards (Stephens and Krebs, 1986; Stevens et al., 2005; Berger-Tal et al., 2009). Thus, decisions to pursue rewards need to accommodate these dynamic conditions. There is increasing experimental evidence that dopamine encodes reward-related information in the execution of economic cost–benefit decisions (Fiorillo et al., 2003, 2008; Phillips et al., 2003b, 2007; Roitman et al., 2004; Tobler et al., 2005; Roesch et al., 2007; Kobayashi and Schultz, 2008; Stuber et al., 2008; Salamone et al., 2009; Zhang et al., 2009; Gan et al., 2010). Previous work demonstrated that phasic dopamine transmission to reward-related stimuli was insensitive to the effort and delay associated with obtaining rewards, except under conditions when these costs changed (Roesch et al., 2007; Gan et al., 2010). Therefore, in the current work, we examined phasic dopamine release to reward-related stimuli under reinforcement schedules that incorporate cost structures with different dynamics. We found that progressively increasing the response requirement did not affect dopamine release in the NAcc core or shell to cues that predict reward availability. However, this manipulation elicited increases in dopamine release to the reward delivery itself in both of these NAcc subregions. This elevation in reward-evoked dopamine release was a function of the delay to reward delivery, independent of the effort exerted to obtain the reward.

Cue-evoked dopamine release

In all of the tasks used in this study, we noted that dopamine release to trial-onset cues was greatest for the first trial, decreased over the next few trials, and remained stable thereafter. Satiety is unlikely to be a significant contributor to this pattern because the attenuation of cue-evoked dopamine release occurred early in the session, whereas animals continued to work for food for many more trials (e.g., 60 total in FR sessions). Larger dopamine release in the first trial could be an effect of novelty associated with the unexpected start of the session, as has been suggested by other studies observing a similar profile of dopamine transmission (Takikawa et al., 2004; Roesch et al., 2007; Gan et al., 2010), or could be explained if this trial acts as an incentive cue for the entire session (Berridge, 2007). Regardless of the interpretation, attenuation of dopamine release to cue presentation over the first few trials is not attributable to changes in effort or delay costs because this pattern was identical between reinforcement schedules that had different cost structures.

Electrophysiological and neurochemical studies demonstrate that the predicted reward value is encoded by phasic dopamine transmission to reward-associated cues (Tobler et al., 2005; Gan et al., 2010). Pharmacological and genetic studies demonstrate that dopamine in the NAcc is used to promote high-effort behaviors (Aberman et al., 1998; Hamill et al., 1999; Salamone and Correa, 2002; Zhang et al., 2003; Cagniard et al., 2006). These seemingly divergent functions have made it difficult to conceptualize the role of dopamine signaling in the NAcc. However, the assessment of reward value and the decision to overcome costs to obtain rewards are necessarily linked. For example, anticipation of larger rewards promotes greater effort to be exerted to earn these rewards and elicits high levels of dopamine in the NAcc that facilitates this effort. Accordingly, it is suggested that phasic dopamine sets a threshold for the maximum cost that should be overcome to obtain rewards (Phillips et al., 2007). In the current work, cue-evoked dopamine release was stable for all but the first few trials of a session, which implies that the threshold to work remains constant from trial to trial. Consistent with this notion, the effort that rats exerted in the last (uncompleted) trial of PR sessions was very similar to that on the previous (completed) trial. Thus, the steady cue-evoked dopamine release with escalating effort and delay costs is indicative of a stable valuation and willingness to work for a single food pellet in the PR session.

Reward-evoked dopamine release

In contrast to the cue-evoked response, we found that reward-evoked dopamine release scaled with the increasing response requirement (Fig. 8). Although dopamine release to reward delivery was minimal in FR sessions, it increased throughout PR sessions. The cost of obtaining rewards in PR sessions comprised two highly correlated components: effort exerted and delay to reward delivery. As such, we could not determine which component(s) mediated the effect on reward-evoked phasic dopamine release from the data obtained in the PR task. Therefore, we experimentally separated effort and time costs in low-effort yoked sessions in which the effort costs were fixed and the delay costs progressively increased throughout the session. The identical pattern of reward-evoked dopamine release between PR and low-effort yoked sessions (i.e., no main effect of session type, no difference between session type for any trial duration, and no interaction) indicates that increasing delays, and not exerted effort, mediated the elevation of dopamine release to food pellet delivery in PR sessions. Analogous to our results in instrumental tasks, dopamine neuron activity to reward delivery is elevated with longer delays during pavlovian paradigms in primates (Fiorillo et al., 2008; Kobayashi and Schultz, 2008).

Figure 8.

Figure 8.

Open in a new tab

Dissociation of dopamine release to cues and rewards in PR sessions. A, Peak dopamine release in the NAcc core to trial-onset cues is plotted together with dopamine release to reward delivery aligned to the last completed trial (n = 13 rats). B, Average dopamine response in the NAcc core to cue presentation in short (10–25 s), medium (25–100 s), and long (>100 s) duration trials. C, Average dopamine response in the NAcc core to pellet delivery in short (10–25 s), medium (25–100 s), and long (>100 s) duration trials. D, Peak dopamine release in the NAcc shell to reward predictive cues is plotted together with dopamine release to reward delivery aligned to the last completed trial (n = 9 rats). E, Average dopamine response in the NAcc shell to cue presentation in short (10–25 s), medium (25–100 s), and long (>100 s) duration trials. F, Average dopamine response in the NAcc shell to pellet delivery in short (10–25 s), medium (25–100 s), and long (>100 s) duration trials.

Contemporary theories of dopamine function can provide a framework to interpret the increased dopamine release to delayed rewards in the context of cognition. The prediction error theory posits that dopamine neurons respond to rewards only when they are unexpected (Schultz et al., 1997; Fiorillo et al., 2003). Because timing becomes less accurate for longer intervals (Weber's law), it follows that dopamine release to rewards, which are delayed under escalating costs, could be a function of the temporal uncertainty of reward delivery. Indeed, dopamine neurons are phasically activated by rewards delivered at unpredictable times (Fiorillo et al., 2008). However, it is problematic to empirically test whether temporal uncertainty is the primary source of delayed-reward evoked dopamine release because there is no overt behavioral measure of uncertainty (Fiorillo et al., 2008). Nonetheless, others have concluded that enhanced phasic dopamine transmission associated with delayed reward delivery reflects uncertainty in the timing of the reward delivery (Fiorillo et al., 2008; Kobayashi and Schultz, 2008). Under an alternative framework, dopamine in the nucleus accumbens has been posited to represent the attribution of incentive value to stimuli (Berridge, 2007). Because dopamine release to reward delivery scaled with increasing delays, this raises the provocative question whether rewards are perceived as more valuable after longer responses have been endured to obtain them. Behavioral studies have examined this question by assessing the choice between distinct stimuli that were paired to the reward after completing the required response under different cost conditions. Interestingly, these studies support the prediction from the incentive hypothesis of dopamine function by demonstrating a preference for conditioned stimuli associated with reward delivery after high costs (Clement et al., 2000; Kacelnik and Marsh, 2002; Friedrich and Zentall, 2004; Navarro and Fantino, 2005; Alessandri et al., 2008). Therefore, both the reward prediction error and incentive salience theories of dopamine function provide competing but plausible rationales for the elevated reward-evoked dopamine release observed with the delay conferred by escalating costs.

Comparing dopamine release between the nucleus accumbens core and shell

In our assessment of anatomical specificity of reward- and cue-evoked dopamine release, we found no differences between the NAcc core and shell. Functionally, these subregions are thought to mediate different aspects of behaviors (Di Chiara, 2002). Recent studies have determined that phasic dopamine release can be heteregenous (Wightman et al., 2007) and differentially regulated throughout the NAcc core and shell in certain behaviors (Aragona et al., 2008, 2009). However, in the operant tasks used in the current study, reward-related stimuli elicited identical phasic dopamine responses throughout the NAcc core and shell, demonstrating that the degree of cooperability of dopamine signaling between these NAcc subregions is task specific.

Conclusions

Overall, the current findings demonstrate that, in both the NAcc core and shell, dopamine release to rewards and their predictors are separable and independently modulated during increasing instrumental-response requirements. Reward-evoked dopamine release in both regions is affected by escalating costs in a manner mediated by delays to reward delivery rather than increased work requirements, whereas cue-evoked dopamine release is unaffected by either temporal- or effort-related costs. Together, these results reveal the pattern of phasic dopamine release to reward-related stimuli in economic decisions under escalating costs.

Footnotes

This work was supported by National Institutes of Health Grants R01 MH079292 (P.E.M.P.), R21-AG030775 (P.E.M.P.), R01-DA016782 (P.E.M.P.), T32-AA009455 (M. E. Larimer), and F32-DA026273 (M.J.W.). We thank S. Ng-Evans for invaluable technical support, C. Akers for technical assistance, and J. Clark, S. Sandberg, J. Gan, and N. Daw for insightful comments.

References

  1. Aberman JE, Ward SJ, Salamone JD. Effects of dopamine antagonists and accumbens dopamine depletions on time-constrained progressive-ratio performance. Pharmacol Biochem Behav. 1998;61:341–348. doi: 10.1016/s0091-3057(98)00112-9. [DOI] [PubMed] [Google Scholar]
  2. Alessandri J, Darcheville JC, Delevoye-Turrell Y, Zentall TR. Preference for rewards that follow greater effort and greater delay. Learn Behav. 2008;36:352–358. doi: 10.3758/LB.36.4.352. [DOI] [PubMed] [Google Scholar]
  3. Aragona BJ, Cleaveland NA, Stuber GD, Day JJ, Carelli RM, Wightman RM. Preferential enhancement of dopamine transmission within the nucleus accumbens shell by cocaine is attributable to a direct increase in phasic dopamine release events. J Neurosci. 2008;28:8821–8831. doi: 10.1523/JNEUROSCI.2225-08.2008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Aragona BJ, Day JJ, Roitman MF, Cleaveland NA, Wightman RM, Carelli RM. Regional specificity in the real-time development of phasic dopamine transmission patterns during acquisition of a cue-cocaine association in rats. Eur J Neurosci. 2009;30:1889–1899. doi: 10.1111/j.1460-9568.2009.07027.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Berger-Tal O, Mukherjee S, Kotler BP, Brown JS. Look before you leap: is risk of injury a foraging cost? Behav Ecol Sociobiol. 2009;63:1821–1827. doi: 10.1007/s00265-009-0809-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Berridge KC. The debate over dopamine's role in reward: the case for incentive salience. Psychopharmacology (Berl) 2007;191:391–431. doi: 10.1007/s00213-006-0578-x. [DOI] [PubMed] [Google Scholar]
  7. Cagniard B, Balsam PD, Brunner D, Zhuang X. Mice with chronically elevated dopamine exhibit enhanced motivation, but not learning, for a food reward. Neuropsychopharmacology. 2006;31:1362–1370. doi: 10.1038/sj.npp.1300966. [DOI] [PubMed] [Google Scholar]
  8. Clark JJ, Sandberg SG, Wanat MJ, Gan JO, Horne EA, Hart AS, Akers CA, Parker JG, Willuhn I, Martinez V, Evans SB, Stella N, Phillips PE. Chronic microsensors for longitudinal, subsecond dopamine detection in behaving animals. Nat Methods. 2010;7:126–129. doi: 10.1038/nmeth.1412. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Clement TS, Feltus JR, Kaiser DH, Zentall TR. “Work ethic” in pigeons: reward value is directly related to the effort or time required to obtain the reward. Psychon Bull Rev. 2000;7:100–106. doi: 10.3758/bf03210727. [DOI] [PubMed] [Google Scholar]
  10. Di Chiara G. Nucleus accumbens shell and core dopamine: differential role in behavior and addiction. Behav Brain Res. 2002;137:75–114. doi: 10.1016/s0166-4328(02)00286-3. [DOI] [PubMed] [Google Scholar]
  11. Fiorillo CD, Tobler PN, Schultz W. Discrete coding of reward probability and uncertainty by dopamine neurons. Science. 2003;299:1898–1902. doi: 10.1126/science.1077349. [DOI] [PubMed] [Google Scholar]
  12. Fiorillo CD, Newsome WT, Schultz W. The temporal precision of reward prediction in dopamine neurons. Nat Neurosci. 2008;11:966–973. doi: 10.1038/nn.2159. [DOI] [PubMed] [Google Scholar]
  13. Friedrich AM, Zentall TR. Pigeons shift their preference toward locations of food that take more effort to obtain. Behav Processes. 2004;67:405–415. doi: 10.1016/j.beproc.2004.07.001. [DOI] [PubMed] [Google Scholar]
  14. Gan JO, Walton ME, Phillips PE. Dissociable cost and benefit encoding of future rewards by mesolimbic dopamine. Nat Neurosci. 2010;13:25–27. doi: 10.1038/nn.2460. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Hamill S, Trevitt JT, Nowend KL, Carlson BB, Salamone JD. Nucleus accumbens dopamine depletions and time-constrained progressive ratio performance: effects of different ratio requirements. Pharmacol Biochem Behav. 1999;64:21–27. doi: 10.1016/s0091-3057(99)00092-1. [DOI] [PubMed] [Google Scholar]
  16. Heien ML, Khan AS, Ariansen JL, Cheer JF, Phillips PE, Wassum KM, Wightman RM. Real-time measurement of dopamine fluctuations after cocaine in the brain of behaving rats. Proc Natl Acad Sci U S A. 2005;102:10023–10028. doi: 10.1073/pnas.0504657102. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Kacelnik A, Marsh B. Cost can increase preference in starlings. Anim Behav. 2002;63:245–250. [Google Scholar]
  18. Kobayashi S, Schultz W. Influence of reward delays on responses of dopamine neurons. J Neurosci. 2008;28:7837–7846. doi: 10.1523/JNEUROSCI.1600-08.2008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. McClure SM, Laibson DI, Loewenstein G, Cohen JD. Separate neural systems value immediate and delayed monetary rewards. Science. 2004;306:503–507. doi: 10.1126/science.1100907. [DOI] [PubMed] [Google Scholar]
  20. Navarro AD, Fantino E. The sunk cost effect in pigeons and humans. J Exp Anal Behav. 2005;83:1–13. doi: 10.1901/jeab.2005.21-04. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Nowend KL, Arizzi M, Carlson BB, Salamone JD. D1 or D2 antagonism in nucleus accumbens core or dorsomedial shell suppresses lever pressing for food but leads to compensatory increases in chow consumption. Pharmacol Biochem Behav. 2001;69:373–382. doi: 10.1016/s0091-3057(01)00524-x. [DOI] [PubMed] [Google Scholar]
  22. Paxinos G, Watson C. San Diego: Academic; 2005. The rat brain in stereotaxic coordinates. [DOI] [PubMed] [Google Scholar]
  23. Phillips PE, Robinson DL, Stuber GD, Carelli RM, Wightman RM. Real-time measurements of phasic changes in extracellular dopamine concentration in freely moving rats by fast-scan cyclic voltammetry. Methods Mol Med. 2003a;79:443–464. doi: 10.1385/1-59259-358-5:443. [DOI] [PubMed] [Google Scholar]
  24. Phillips PE, Stuber GD, Heien ML, Wightman RM, Carelli RM. Subsecond dopamine release promotes cocaine seeking. Nature. 2003b;422:614–618. doi: 10.1038/nature01476. [DOI] [PubMed] [Google Scholar]
  25. Phillips PE, Walton ME, Jhou TC. Calculating utility: preclinical evidence for cost-benefit analysis by mesolimbic dopamine. Psychopharmacology (Berl) 2007;191:483–495. doi: 10.1007/s00213-006-0626-6. [DOI] [PubMed] [Google Scholar]
  26. Roesch MR, Calu DJ, Schoenbaum G. Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nat Neurosci. 2007;10:1615–1624. doi: 10.1038/nn2013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Roitman MF, Stuber GD, Phillips PE, Wightman RM, Carelli RM. Dopamine operates as a subsecond modulator of food seeking. J Neurosci. 2004;24:1265–1271. doi: 10.1523/JNEUROSCI.3823-03.2004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Salamone JD, Correa M. Motivational views of reinforcement: implications for understanding the behavioral functions of nucleus accumbens dopamine. Behav Brain Res. 2002;137:3–25. doi: 10.1016/s0166-4328(02)00282-6. [DOI] [PubMed] [Google Scholar]
  29. Salamone JD, Correa M, Farrar AM, Nunes EJ, Pardo M. Dopamine, behavioral economics, and effort. Front Behav Neurosci. 2009;3:13. doi: 10.3389/neuro.08.013.2009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Schultz W. Behavioral theories and the neurophysiology of reward. Annu Rev Psychol. 2006;57:87–115. doi: 10.1146/annurev.psych.56.091103.070229. [DOI] [PubMed] [Google Scholar]
  31. Schultz W, Dayan P, Montague PR. A neural substrate of prediction and reward. Science. 1997;275:1593–1599. doi: 10.1126/science.275.5306.1593. [DOI] [PubMed] [Google Scholar]
  32. Sokolowski JD, Salamone JD. The role of accumbens dopamine in lever pressing and response allocation: effects of 6-OHDA injected into core and dorsomedial shell. Pharmacol Biochem Behav. 1998;59:557–566. doi: 10.1016/s0091-3057(97)00544-3. [DOI] [PubMed] [Google Scholar]
  33. Stephens DW, Krebs JR. Princeton, N.J.: Princeton University Press; 1986. Foraging theory. [Google Scholar]
  34. Stevens JR, Hallinan EV, Hauser MD. The ecology and evolution of patience in two New World monkeys. Biol Lett. 2005;1:223–226. doi: 10.1098/rsbl.2004.0285. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Stuber GD, Klanker M, de Ridder B, Bowers MS, Joosten RN, Feenstra MG, Bonci A. Reward-predictive cues enhance excitatory synaptic strength onto midbrain dopamine neurons. Science. 2008;321:1690–1692. doi: 10.1126/science.1160873. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Takikawa Y, Kawagoe R, Hikosaka O. A possible role of midbrain dopamine neurons in short- and long-term adaptation of saccades to position-reward mapping. J Neurophysiol. 2004;92:2520–2529. doi: 10.1152/jn.00238.2004. [DOI] [PubMed] [Google Scholar]
  37. Tobler PN, Fiorillo CD, Schultz W. Adaptive coding of reward value by dopamine neurons. Science. 2005;307:1642–1645. doi: 10.1126/science.1105370. [DOI] [PubMed] [Google Scholar]
  38. Wanchisen BA, Tatham TA, Hineline PN. Pigeons' choices in situations of diminishing returns: fixed- versus progressive-ratio schedules. J Exp Anal Behav. 1988;50:375–394. doi: 10.1901/jeab.1988.50-375. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Wightman RM, Heien ML, Wassum KM, Sombers LA, Aragona BJ, Khan AS, Ariansen JL, Cheer JF, Phillips PE, Carelli RM. Dopamine release is heterogeneous within microenvironments of the rat nucleus accumbens. Eur J Neurosci. 2007;26:2046–2054. doi: 10.1111/j.1460-9568.2007.05772.x. [DOI] [PubMed] [Google Scholar]
  40. Wise RA. Dopamine, learning and motivation. Nat Rev Neurosci. 2004;5:483–494. doi: 10.1038/nrn1406. [DOI] [PubMed] [Google Scholar]
  41. Zhang L, Doyon WM, Clark JJ, Phillips PE, Dani JA. Controls of tonic and phasic dopamine transmission in the dorsal and ventral striatum. Mol Pharmacol. 2009;76:396–404. doi: 10.1124/mol.109.056317. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Zhang M, Balmadrid C, Kelley AE. Nucleus accumbens opioid, GABaergic, and dopaminergic modulation of palatable food motivation: contrasting effects revealed by a progressive ratio study in the rat. Behav Neurosci. 2003;117:202–211. doi: 10.1037/0735-7044.117.2.202. [DOI] [PubMed] [Google Scholar]

Articles from The Journal of Neuroscience are provided here courtesy of Society for Neuroscience

RESOURCES