The power of price compels you: Behavioral economic insights into dopamine-based valuation of rewarding and aversively motivated behavior

Abstract

The mesocorticolimbic dopamine pathway is generally considered to be a reward pathway. While shortsighted, there is a strong basis for this view of dopamine function. Here, we first describe the role of phasic dopamine release events in reward seeking. We then explain why these release events are being reconsidered as value signals and how we applied behavioral economics to confirm they play a causal role in the valuation of reward. Just because dopamine release can function as a dopamine reward value signal however, does not imply that dopamine is solely a reward molecule. Rather, mesocorticolimbic dopamine appears to mediate many adaptive behaviors, including: reward seeking, avoidance, escape and fear-associated conditioned freezing. While more studies are needed before a consensus is reached on when, where and how dopamine mediates aversively-motivated behavior, its involvement is almost irrefutable. Thus, we next describe the role dopamine plays in these ethologically-relevant defensive behaviors. We conclude by describing our recent behavioral economics results that reveal a causal role for dopamine in the valuation of avoidance.

1. Brief introduction to the mesocorticolimbic dopamine system

It is becoming increasingly evident that dopamine systems interact within complex neural networks to mediate not only appetitively, but also aversively motivated behaviors. A growing body of evidence supports a role for nigrostriatal dopamine in defensive behaviors, such as fear-induced conditioned freezing (Bouchet et al., 2018; Mika et al., 2015). And, recent evidence suggests dopamine release events arising from non-canonical dopamine populations in the periaqueductal gray and in the dorsal raphe interact with the amygdala to influence behavioral responses to fear (Groessl et al., 2018).

In the current review we will restrict our discussion to the mesocorticolimbic dopamine system, a neural pathway that is highly conserved across vertebrate species (Smeets et al., 2000). This pathway originates from the ventral tegmental area (VTA) and projects to various neural substrates throughout the brain, including the frontal cortex, amygdala, olfactory tubercle and—most prominently, the nucleus accumbens (NAc) (Swanson, 1982). While dopamine released into various terminal fields influences motivated behavior, we are focusing on the NAc because of its predominance not only in mesolimbic neuroanatomy, but also in the historic motivational literature. The NAc is often described as a limbic-motor (Mogenson et al., 1980) or a Pavlovian-instrumental (Cardinal et al., 2002) interface because it theoretically integrates emotional information with learned associations to guide advantageous behavior. While dopamine neurons predominate in the VTA (~60%), it is important to note that GABA and glutamate neurons are also prevalent and capably modulate dopamine neural activity (Margolis et al., 2012; Morales and Margolis, 2017; Nair-Roberts et al., 2008; Swanson, 1982; Tagliaferro and Morales, 2008; Yamaguchi et al., 2007). Secondary messengers within the VTA (e.g., NO, H₂O₂ and endocannabiniods) and neuropeptides/hormones (e.g., ghrelin, estrogen) arising from outside the VTA influence dopamine function as well (Becker, 1999; Chen et al., 2001; Cone et al., 2015; Munro et al., 2006; Oleson et al., 2012a; Oleson and Cheer, 2012; Oleson et al., 2014; Prast and Philippu, 2001; West et al., 2002). Investigating the role dopamine plays in behavior is further complicated by the growing recognition that dopamine neurons co-release multiple primary neurotransmitters, including GABA and glutamate (Stuber et al., 2010; Sulzer and Rayport, 2000; Tritsch et al., 2012). While co-release was thought to play exclusive roles in early development, including circuit formation and synaptic fortification, we are beginning to recognize that this form of neurotransmission continues to modify existing circuits and synapses into adulthood (Vaaga et al., 2014). And, even if we come to understand the precise impacts of co-release and neuromodulation, we must accept that dopamine function is shaped by its interactions within a wide array of neural networks. Indeed, neural input into the VTA greatly influences dopamine release (Brown et al., 2017; Tian and Uchida, 2015), and the effects of dopamine release in the NAc are dictated by converging cortical, amygdalar, and hippocampal input (Brady and O’Donnell, 2004; Floresco et al., 2001).

When considering the role dopamine plays in behavior it is also important to recognize that dopamine neurons fire in one of two distinct patterns. When at rest, dopamine neurons exhibit regular pacemaker activity (1–5 Hz) that produces a tone on high-affinity dopamine receptors (Goto and Grace, 2005; Venton et al., 2003). When presented with motivationally salient stimuli, dopamine neurons fire in phasic bursts (≥20 Hz) (Goto and Grace, 2005). These bursts of neural activity contribute to transient release events that are sufficient in concentration to occupy low-affinity dopamine receptors (Dreyer et al., 2010; Sombers et al., 2009; Venton et al., 2003). This review will exclusively focus on the phasic form of dopamine release. While it is likely that the magnitude of release events in the NAc reflects the summation of phasic activity in the VTA (Sombers et al., 2009), it is noteworthy that dopamine neurons within the midbrain heterogeneously represent aversive stimuli at the level of the single unit (Lammel et al., 2012; Pignatelli and Bonci, 2015). Although the majority of electrophysiological studies report that dopamine neurons are inhibited by aversive stimuli (Mileykovskiy and Morales, 2011; Tan et al., 2012), there are numerous reports of excitations as well (Anstrom et al., 2009; Brischoux et al., 2009; Matsumoto and Hikosaka, 2009). Despite the fact that some of these results might be confounded by the inclusion of non-dopamine neurons (Ungless and Grace, 2012), they support the growing belief that many factors influence whether dopamine neurons are excited or inhibited by an aversive event—including the subpopulation of neurons surrounding the recording electrode (Lammel et al., 2012) and the environmental context in which aversion is introduced (Matsumoto et al., 2016).

In fact, variables such as environmental context and behavioral history greatly influence the emotional valence of stimuli at the behavioral level. Indeed, whether a stimulus is perceived to be rewarding or aversive by an individual subject is malleable and influenced by variables such as history and environment context (Nasser and McNally, 2012). Although electrical shock is generally assumed to be an aversive stimulus, animals can be trained to respond for its delivery (McKearney, 1968). Similarly, universally recognized rewarding stimuli, including abused drugs, can produce aversive responses (Grigson, 1997). These observations do not however, mean we should revert to ignoring emotional valence as was done in the historical behavioral analysis literature (Baron and Galizio, 2005). Rather, it should be acknowledged that the emotional valence of ‘rewarding’ and ‘aversive’ stimuli can change, and caution should be exercised during the design and analysis of studies on the neuroscience of motivation and emotion. While we are generations away from completely understanding how subpopulations of dopamine neurons interact at the pathway and network level to control rewarding vs. aversively motivated behavior, recent neuroscientific advances provide considerable insight into the role that transient dopamine release events play in guiding advantageous outcomes.

2. Accumbal dopamine release in reward directed behavior

By performing in vivo electrophysiological recordings in the midbrain of awake and behaving monkeys, Schultz et al. (1997) first demonstrated that putative dopamine neurons fire in phasic bursts when animals are presented with an unexpected reward or a reward predictive stimulus but are suppressed when an expected reward is withheld. This observation led to the prominent reward prediction error (RPE) theory, which holds that phasic bursts of dopamine neural activity encode a reward prediction signaled by the earliest predictor of its availability. While we now recognize that the traditional electrophysiological criteria used to identify dopamine neurons produced false positives (Ungless and Grace, 2012), many different labs have since used a variety of techniques to confirm the general precepts of RPE (Abler et al., 2006; Daw and Doya, 2006; Hart et al., 2014; Pagnoni et al., 2002).

By performing fast-scan cyclic voltammetry (FSCV), an electrochemical technique that allows for the detection of subsecond changes in dopamine release, we (Oleson et al., 2012a) confirmed that RPE can occur within a single brain stimulation reward session (Owesson-White et al., 2008) (Fig. 1A–C). Here, we measured subsecond changes in dopamine release as rats pressed a lever that delivered electrical current into the VTA (i.e., brain stimulation reward). These electrical currents lead to the depolarization of dopamine neurons, albeit indirectly. Rather than directly activating dopamine neurons or their unmyelinated axons, dopamine neurons are thought to integrate input from non-dopaminergic pathways that can be directly activated by electrical stimulation (Hernandez et al., 2010; Moisan and Rompre, 1998). As a result of this integration, dopamine neurons burst fire to produce high-concentration, transient release events in terminal fields of the mesocorticolimbic pathway (Sombers et al., 2009).

Fig. 1.

Republished from Oleson et al., 2012a with permission. (A) Pavlovian associations sculpt patterns of transient dopamine release in response to a predictive cue during reward seeking. A representative surface-plot shows changes in dopamine concentration (z-axis) occurring across trials (y-axis) while responding is maintained by brain stimulation reward in an ICSS task. Cue presentation, which is indicated by the gray rectangle, occurred for 1-s prior to lever extension. (B) Representative traces show the mean cue-evoked dopamine concentration increasing across trials. Each dopamine concentration trace represents the mean of 30 consecutive trials. (C, D) The conditioned cue begins to strengthen reward seeking as the concentration of cue-evoked dopamine increases across trials. Linear regression analyses show increases in mean dopamine concentration and decreases in response latency across binned responses. Dashed lines represent the 95% confidence region.

Within each trial of the task, reward availability was signaled by a cue-light placed above the operant lever. When given the opportunity to respond on a lever that delivers electrical currents to the VTA, animals rapidly acquire responding within a single session (Fig. 1D). As would be predicted by RPE, the concentration of dopamine time-locked to a reward predictive cue increased as the latency to respond decreased (Fig. 1C, D). It is also noteworthy that dopamine concentration can increase prior to cue presentation when animals are able to anticipate the timing of reward availability (Cheer et al., 2007). Fig. 1A, B depicts these anticipatory dopamine responses preceding a reward predictive cue that was presented according to a fixed schedule. In such cases, dopamine may represent interoceptive timing cues, which are known to exert considerable power over ongoing behavior (Lake et al., 2016). Because dopamine release begins to occur at the earliest predictor of reward, it became widely accepted that these transient release events function as teaching signals to guide reward learning (Schultz et al., 1997). This causal relationship was first demonstrated by Adamantidis et al. (2011), who implemented optogenetics technology to show that increasing dopamine release at reward delivery accelerates reward learning.

3. Transition from RPE to value signals

RPE is currently being considered within the context of reward value. Reconsidering RPE-associated phasic dopamine release events as value signals provides the most parsimonious description of how DA functions within this behavioral context. The initial RPE studies were inspired by the fundamental classical conditioning studies of Pavlov and Rescorla-Wagner (Pavlov, 2010; Rescorla and Wagner, 1972), and, therefore, focused on the formation of conditioned associations through associative learning. From this perspective, it might be tempting to assume that with repeated associations between a reward and its predictive cue, that DA release will begin to exclusively occur at the earliest predictor of reward. While the data presented herein will show that DA release events tend to be higher in concentration at the predictive cue, they continue to occur at the outcome as well. In addition, while stimulus-response associations are important, they fail to completely account for the patterns of DA release observed during reinforced behavior. As will be discussed throughout this review, various factors influence DA release as animals pursue desired outcomes, including the timing of reinforcement delivery (Fonzi et al., 2017; Lake et al., 2016; Meck, 1986; Oleson et al., 2014), the motivational/emotional state of the animal (Cone et al., 2016), and the environment in which reinforcement is provided (Nakahara et al., 2004). Thus, a more apt description might be that DA release events guide animals to highly-valued outcomes in a state- and contextually-dependent manner. In this review, we will present data supporting that the DA signal evoked by a routinely presented reward-predictive stimulus reflects predicted value; whereas, the DA signal evoked by the outcome reflects experienced value. It is also worth noting that the field shifting toward a value-based framework was greatly influenced by computational assessments on the role of DA in reinforcement learning. An interdisciplinary combination of economic theory, computer science and mathematics led to value-based computational models that better align with the DA and behavioral observations we observe during action-outcome learning (Glimcher, 2011).

A growing body of electrophysiological and electrochemical studies suggest that the concentration of dopamine evoked by reward predictive cues corresponds to the magnitude of reward predicted by the cue (Bayer and Glimcher, 2005; Day et al., 2010; Day et al., 2011; Enomoto et al., 2011; Lak et al., 2014; Schelp et al., 2017; Stauffer et al., 2014; Tobler et al., 2005). Imaging studies reveal similar observations in human subjects and further refine the output regions involved. The BOLD signal in the VTA corresponds to RPE (Klein-Flügge et al., 2011) but disparate effects are observed across striatal output regions. Klein-Flügge et al., 2011 report that BOLD activity in the NAc encodes variables related to the receipt of reward, such as time of reward delivery, rather than RPE itself. This is an important distinction to consider in its own right. Using FSCV, we also find that changes in dopamine release events in the NAc are primarily related to time under conditions of periodic reinforcement, such as reward seeking maintained on a fixed-interval schedule (Oleson et al., 2014). Together, we interpret these results to suggest that dopamine release events play multiple roles in ongoing behavior rather than an exclusive role in RPE. Despite these potentially confounding time-related signals, multiple human imaging studies report that the BOLD signal in the ventral striatum (aka NAc in rats) and caudate (aka dorsal medial striatum in rats), but not the putamen (aka lateral medial striatum in rats) corresponds to RPE in humans (Haruno and Kawato, 2006a; Haruno and Kawato, 2006b; McClure et al., 2003; O’Doherty et al., 2003). Thus, a large body of evidence obtained using a variety of approaches supports the conception that cue-evoked dopamine release events actually represent a value prediction signaled by the cue (Schultz et al., 2015; Schultz et al., 2017). This shift in thought is leading investigators to incorporate elements of economic theory to investigate the relationship between dopamine and valuation (Arvanitogiannis and Shizgal, 2008; Burke et al., 2016; Hernandez et al., 2012; Pasquereau and Turner, 2013; Schultz et al., 2015). For example, a recent series of electrophysiological studies reported that dopamine neurons respond to gambles and outcomes to guide economic decisions (Stauffer et al., 2014), and are able to integrate various factors that underlie value (Lak et al., 2014).

4. Using demand curves to investigate natural reward value

We measure value by assessing the rate at which demand curves decay (Fig. 2). Demand curves are a common tool used by economists to measure price sensitivity. They depict the relationship between consumption (mg of sugar received or mA footshock avoided, in this review) and price. In operant studies, price can be defined as the response requirement per unit reward (Bickel et al., 1990). Demand curves usually show a negative gradient (i.e., the law of demand), where consumption decreases with increasing price. The rate at which the negative slope decays can be used to make inferences regarding the value individuals place on the commodity being consumed. We fit demand curves using freely available R-studio programs (https://gitlab.com/oleson/schelp-pnas) and measure the price elasticity of demand by computing the variable α, an established dependent measure of value (Hursh and Silberberg, 2008). This variable represents the cost at which the elasticity of demand is exactly −1, meaning consumption drops by one percent in response to a one percent increase in price. When demand curves decay at a faster rate they are said to be more elastic. A higher a value indicates the demand for the desired outcome is more elastic, suggesting the value of the commodity is diminished. In contrast, a lower α value indicates the demand for the commodity is relatively inelastic, suggesting the value of the commodity is enhanced (Fig. 2).

Fig. 2.

Measuring value: Illustrative demand curve showing how we determine value. α is our dependent measure of value and represents the rate at which the demand curve decays. Demand decays at a faster rate when the animal becomes more sensitive to price. As the animal is willing to pay less for the commodity, we would interpret the resulting increase in α as a decrease in value.

While the fields of psychopharmacology (Bickel et al., 1990) and behavioral analysis (Herrnstein, 1974) have a long history of applying demand curve analysis, the predominance of ‘between-sessions’ approaches has made it difficult to reliably perform neuroscientific assessments within daily sessions. We overcame this obstacle by developing a within-session approach that allows demand curves to be generated in daily sessions. Working within a behavioral economics framework, we first designed a ‘between-sessions’ procedure where rats were presented with cocaine at different prices across daily sessions (Oleson and Roberts, 2009). One unique feature of this approach is that we manipulated price by reducing the amount of drug delivered per lever press (Zittel-Lazarini et al., 2007). As the unit price ratio (lever presses required per unit of desired outcome) might suggest, unit price can be manipulated not only by increasing response requirement as in the progressive ratio schedule, but also by fixing the response requirement at 1 and decreasing the available unit of reward (Bickel et al., 1990). This latter approach reduces the temporal confound of opportunity cost that is inherent to tasks that manipulate price by increasing response requirement (Schelp et al., 2017), and allows the investigator to seamlessly change unit-price within daily sessions. The ‘within-session’ behavioral economics approach has several advantages over comparable tasks. Unlike the commonly used progressive ratio schedule, within-session behavioral economic tasks involve multiple pairings between each unit-price, reward and its predictive cue. These cue-associations are critically important for the cue to represent reward value and evoke lawful dopamine value signals. An additional benefit is that session duration is malleable, allowing the investigator to tailor their experimental design around commonly encountered temporal constraints (e.g., drug half-life, post-reinforcement pauses, fatigue, etc.). Finally, behavioral economics uses common semantics that may facilitate interdisciplinary collaboration.

5. Dopamine encodes reward value and causally modifies the price rats will pay for reward

To test the revised value-based theory of RPE we performed demand analyses while rats responded on a lever for sugar. We reasoned that if transient dopamine signals do indeed represent value, then the transient dopamine response should be sensitive to price and modulating dopamine release should alter the rate at which demand curves decay. Access to sugar was provided across ten unit prices (defined as lever presses required per mg of sucrose) within a single session, with each price presented for a fixed epoch of time. A predictive cue light placed above the lever signaled sugar availability. We manipulated price by either increasing response requirement and reducing reward magnitude in independent assessments. Regardless of how price was manipulated, the concentration of accumbal core dopamine time-locked to cue presentation and sucrose-delivery significantly decreased with increasing price (Fig. 3A). These data show that the concentration of transient dopamine release events represent price and support the theory that dopamine represents reward value. To assess the causal influence of dopamine on price sensitivity, we then optically augmented release during the task. Augmenting release at the reward predictive cue increased dopamine concentration at the cue (Fig. 3B), but reduced release at reward delivery (Fig. 3C). As might be predicted from the latter dopamine observation, increasing release at cue presentation also made demand for sugar more sensitive to price (Fig. 3D–F). From these observations, we infer that value is decreased because of a negative RPE (i.e., the animal receives less reward than expected). Conversely, enhancing dopamine at reward made demand less sensitive to price (Fig. 3D–F). We attribute this finding to a positive RPE, whereby the animal perceives they received a better value than anticipated. These data confirm and extend the notion that dopamine release events encode reward value, and further demonstrate that increasing dopamine release causally modifies price sensitivity.

Fig. 3.

Republished from Schelp et al., 2017 with permission. (A) Dopamine concentration (mean ± sem) decreases across the first five prices in the sugar-based behavioral economics task. (B, C) Optogenetic stimulation at cue increases dopamine release at cue but decreases dopamine release at reward delivery. When compared to baseline animals (A), stimulating dopamine release at cue presentation increased dopamine at cue (B) but decreased dopamine at reward delivery (C). (D-F) Optogenetic stimulation alters price sensitivity in a representative rat. (D) Cumulative records from one animal responding in sugar-based behavioral economics task. Colors denoting stimulation condition are shown in the legend. (E) The same data are re-plotted to show responses as a function of unit price. (F) Demand curves show consumption as a function of unit price. Demand curves are fitted to the data to estimate α. α measures price sensitivity; a high α values indicate consumption decays rapidly with price.

6. Accumbal dopamine in avoidance

Given the field’s emphasis on the role of dopamine in reward directed behavior, relatively less is known regarding dopamine’s role in avoiding harm. The few studies on the subject suggest that avoidance requires dopamine signaling, particularly dopamine release into the NAc. Animals fail to acquire avoidance following 6-hydroxydopamine lesions of midbrain dopamine neurons, a deficit that is reversed by L-dopa treatment (Cooper et al., 1973; Zis et al., 1974). While multiple terminal fields of the mesocorticolimbic pathway are likely involved in avoidance, an abundance of evidence clearly demonstrates a role for striatal dopamine within the NAc. Dopamine terminal lesions in the NAc (McCullough et al., 1993) and dopamine receptor antagonists locally administered in the NAc (Arnt, 1982; Wadenberg et al., 1990) impair the avoidance of electrical footshock while genetic restoration of dopamine within the striatum of otherwise dopamine-deficient mice is necessary to maintain avoidance (Darvas et al., 2011). While dopamine in the striatum was necessary to maintain avoidance, dopamine in the amygdala was necessary for avoidance learning (Darvas et al., 2011). Within the context of the existing literature, these findings suggest that while the amygdala is critically important for aversively motivated learning (Ledoux and Muller, 1997; LeDoux, 2003), the striatum may be recruited later on during the maintenance and habituation of avoidance (Poremba and Gabriel, 1999).

Early attempts to monitor dopamine release during avoidance using microdialysis revealed that striatal dopamine concentration is increased during the avoidance of electrical footshock (McCullough et al., 1993). However, it remained unknown precisely when dopamine is released during avoidance. In a typical active avoidance experiment multiple footshocks, avoidance outcomes, escape outcomes, and safety periods occur over the course of commonly used microdialysis sample collection times (10–20 min).

7. Transient dopamine release in avoidance, escape and conditioned fear

To characterize dopamine release in the NAc core during conditioned avoidance we measured subsecond accumbal dopamine release events during behavior maintained in an active, signaled operant avoidance task (Oleson et al., 2012b). Within each experimental trial, a cue light was presented to the animal as a warning signal for 2 s prior to the delivery of recurring footshock. A response lever was simultaneously extended into the operant chamber when the warning signal was illuminated. Two outcomes were possible: avoidance and escape. Animals could produce an avoidance outcome by pressing the lever once during the initial 2 s warning period. If an avoidance response occurred, footshock delivery was prevented and the animal entered a 20 s safety period signaled by a tone. Animals could produce an escape outcome by failing to press the lever during the initial 2 s warning period. Thus, the animal must respond on the lever to terminate ongoing footshock. As in avoidance, after each escape response animals enter a 20 s safety period signaled by a tone.

Under these conditions, the concentration of dopamine evoked by the warning signal increased during avoidance, but decreased during escape (cf. Fig. 4A, B). In other words, the occurrence of accumbal dopamine release during warning signal presentation predicted whether or not the animals would successfully avoid footshock. These data suggest that accumbal dopamine release encodes cues predicting avoidance and may motivate incipient actions devoted to this advantageous outcome. Further microinjection studies revealed that dopamine likely facilitates avoidance by acting on low-affinity D1, rather than high-affinity D2 receptors in the NAc (Wenzel et al., 2018). In both avoidance and escape outcomes, dopamine release increased during the safety period (Fig. 4A, B). It is possible that the distinct responses observed at the warning signal and safety signal during escape/avoidance may contribute to avoidance learning (Oleson and Cheer, 2013). Recent evidence from Luo et al. (2018) supports this supposition. These investigators demonstrated that dopamine signals originating in the VTA during safety are necessary to reduce fear responses by engaging molecular mechanisms in the NAc shell (Luo et al., 2018).

Fig. 4.

Republished from Oleson et al., 2012b with permission. (A) Dopamine encodes conditioned stimuli during negative reinforcement. Representative color plots (left) and dopamine concentration traces (right) show avoidance (top), one-footshock escape (middle), and two-footshock escape (bottom) responses. Arrows indicate lever responses, lightning bolts indicate footshocks, trumpets indicate safety periods, levers + lights indicate warning signals. Left, Voltammetric current (z-axis) plotted against applied scan potential (Eapp; y-axis) and time (x-axis). Right, Dopamine concentration traces plotted as a function of time. Inset shows cyclic voltammogram for dopamine. (B) Warning signal presentation increases dopamine release when rats successfully avoid footshock. Mean (n = 9) ± SEM traces depict the time course of changes in subsecond dopamine release as animals minimize punishment by avoiding footshock. Dashed lines represent warning stimulus onset, around which mean data are grouped. Color representations: light gray, maximum warning stimulus duration; dark gray, safety period. (C) Warning signal presentation inhibits dopamine release when rats fail to avoid or escape footshock. Only one-footshock escape trials are included in the mean (n = 9). (D) Maximal dopamine concentration evoked by warning signal presentation predicts successful punishment avoidance.

By contrast, an attenuation in accumbal dopamine concentration may suppress ongoing goal-directed behavior (Kravitz et al., 2012). It is likely that the suppression in dopamine release observed in escape training may underlie equally adaptive species-specific freezing responses that are observed during operant avoidance (Bolles, 1970). If so, operant avoidance provides a highly-objective approach to study the neural mechanisms underlying distinct ethologically-relevant defensive behaviors, including: fear, escape, and avoidance. In agreement with Badrinarayan et al. (2012), we also observed a transient suppression in dopamine concentration in the NAc core during the extinction of fear memories using a more traditional fear conditioning approach. Here, inescapable footshock was first conditioned to an otherwise neutral tone. 24hr later the conditioned tone was replayed while we monitored changes in accumbal dopamine release. A tone-induced suppression in dopamine release (Fig. 4C) was only apparent in trials in which the animal froze upon its presentation (Oleson et al., 2012b). We recently expanded upon these findings by applying optogenetics technology to show that augmenting dopamine release during the fear-associated tone facilitates the extinction of conditioned freezing (Wenzel et al., 2018; Badrinarayan et al., 2012).

8. Using demand curves to investigate avoidance value: feasibility and additional observations on the role of dopamine in conditioned suppression

While accumbal dopamine release appears to contribute to different defensive behaviors, it remains controversial whether transient dopamine signals represent the value of avoiding harm. A notable electro-physiology study recently reported that dopamine neurons predominately encode the value of reward rather than avoidance (Fiorillo, 2013). To assess whether accumbal dopamine release events represent the value of active avoidance, we performed demand analysis as rats avoided electrical footshock across a range of prices in daily operant sessions. Our approach builds upon emerging economic approaches designed to study the value of avoidance (Spiegler et al., 2018; Sweis et al., 2018). We defined unit-price as lever responses per mA current (resp./mA) and manipulated price by increasing the number of lever presses required to avoid in fixed epochs within each session. While this task produced lawful demand curves, they were uniquely hallmarked by an attenuation of consumption at session onset (cf. Fig. 3E vs Fig. 5A). Because this attenuation developed over training, we infer that it represents a learned suppression resulting from repeated training sessions that only terminate when the animal receives 15–20 consecutive footshocks. This suppression at session onset is not observed early in economic avoidance training (Fig. 5A) and was absent during behavior maintained under a simple fixed ratio 1 schedule FR1) (Oleson et al., 2012b; Wenzel et al., 2018). In these earlier simple reinforcement studies, daily sessions terminated after a fixed period of time rather than after a fixed number of consecutive footshock. Thus, session onset did not necessarily predict an aversive session terminus.

Fig. 5.

Republished from Pultorak et al., 2018 with permission. (A) As animals acquired responding in the economic avoidance task, a suppression of consumptions at the lowest unit-price (2 resp/mA) developed over the course of training. This may represent a conditioned suppression induced by a session onset that predicts an aversive session terminus (15–20 consecutive foot-shocks). (B). Dopamine concentration was concurrently suppressed with behavior at session onset. (C). Optically stimulating dopamine neurons at successful avoidance rectified the suppression of behavior at session onset.

As in our investigations into the role of dopamine during escape and conditioned fear, the concentration of NAc core dopamine evoked by the warning signal was concurrently suppressed with behavior at session onset (Fig. 5B). Optogenetic stimulation of VTA dopamine neurons at successful avoidance rectified the conditioned suppression of consummatory behavior at session onset—further supporting the notion that mesolimbic dopamine release causally modifies this defensive response (Fig. 5C).

9. Dopamine encodes avoidance value and causally modifies the price rats will pay to avoid

Despite the initial suppression, consumption of avoidance generally conformed to the law of demand, allowing us to assess whether dopamine represents the value to avoid aversive outcomes (Pultorak et al., 2018). To evaluate the role of NAc core dopamine in the valuation of avoidance, we first employed FSCV to monitor dopamine evoked by the warning signal across a range of prices. As in reward motivated behavior, dopamine concentration at the warning signal decreased with price (Fig. 6A). Also, similarly to reward-directed behavior (Fig. 1A, B) (Cheer et al., 2007; Oleson et al., 2012a), we observed anticipatory dopamine signals emerging prior to the warning signal (Fig. 6A, inset). One distinction from our previous work is that dopamine at the warning decreased with price in both avoidance and escape outcomes. While Oleson et al. (2012b) reported a suppression in dopamine concentration and behavior when the warning signal resulted in escape rather than avoidance (Fig. 4A, B), animals were simply trained to avoid footshock under an FR1 schedule in up to 50% of trials (~6d). The economic avoidance task required approximately 30d of training. Thus, it is possible that the warning signal no longer produces a conditioned suppression during escape outcomes after extended training.

Fig. 6.

(A) The concentration of dopamine evoked by a warning signal that predicted the opportunity to avoid decreased with the price to avoid. (inset). Representative avoidance trial shows that dopamine concentration began increasing in anticipation of warning signal presentation, as the safety period occurred in fixed 30 s intervals. (B) The concentration of dopamine release events during the safety period decrease with price. (C, D) Optogenetic activation of VTA dopamine neurons at the warning signal made animals more sensitive to price, consistent with a negative reward prediction error. In contrast, optically stimulating dopamine neurons at successful avoidance made animals less sensitive to price, consistent with a positive reward prediction error.

We also analyzed how dopamine responses change with price during the safety period. The pattern of dopamine release during the safety period is distinct, as this period is not transient; rather, each safety period lasted 30 s in duration. Thus, to more accurately assess dopamine transient activity during the safety period, we analyzed the amplitude of individual transient events throughout this period. We found that concentration of dopamine released during safety decreased with price, but only in avoidance outcomes (Fig. 6B). We interpret these distinct outcome-specific responses to suggest that dopamine concentration prior to behavioral action was predictive of value; whereas, dopamine concentration following action was reflective of the outcomes value.

We next sought to assess the causal role of dopamine in the valuation of avoidance by employing optogenetics. As in reward motivated behavior (Fig. 3C–E), optically stimulating dopamine neurons in the VTA at the warning signal made rats more sensitive to price (Fig. 6C), consistent with a negative RPE. We infer that heightened release at cue presentation signaled an advantageous outcome, a prediction that was then violated by the occurrence of footshock. By contrast, optically increasing release at successful avoidance made animals less sensitive to price (Fig. 6D), consistent with a positive RPE. We infer that heightened release at successful avoidance signaled the outcome was better than predicted, indicating a good value worth seeking. These results support the theory that transient dopamine signals causally influence valuation and further clarify that these value signals not only represent the value of pursuing reward, but also the value of avoiding harm.

10. How does DA interact with other circuits that mediate avoidance?

While our data suggest that DA release events are involved in both the valuation or reward and avoidance, it is likely that the mesolimbic DA pathway is merely a shared component of larger diverging circuits that ultimately control these distinct behaviors. Thus, we next consider several other brain regions implemented in avoidance and how these circuit nodes may interact with DA value signals. Within the striatum DA primarily acts on the predominate cell type present, GABAergic medium spiny neurons. These neurons then project to the globus pallidus—the primary output nucleus of the basal ganglia. For a thorough review on the role of the globus pallidus in aversion we refer the reader to (Wulff et al., 2018). The globus pallidum was traditionally divided into an internal segment that is associated with action initiation, and an external segment that is often associated with action inhibition (Stephenson-Jones et al., 2013). While these nuclei are clearly involved in defensive behaviors including avoidance, we are beginning to recognize the behavioral importance of a unique subset of habenula-projecting globus pallidal neurons (Stephenson-Jones et al., 2016). Habenula-projecting globus pallidal neurons effectively close a valuation loop by conveying value signals from the striatum to the habenula, a structure well-known to indirectly inhibit dopamine neurons in the VTA (Stephenson-Jones, 2019; Tian and Uchida, 2015). DA release events may directly influence these pallidal value signals during avoidance as local administration of DA D1R and D2R agonists into this region facilitates inhibitory avoidance learning (Lénárd et al., 2017; Péczely et al., 2014). Outside of the canonical basal ganglia extra-pyramidal motor pathway, amygdalar interactions between the amygdala and NAc are well known to influence avoidance. Of its three major subdivisions, the basal and medial, but not central amygdala are required for active avoidance (Lázaro-Muñoz et al., 2010). In well trained animals, the warning signal is thought to activate neurons in the lateral amygdala. These lateral amygdala neurons project to the basolateral amygdala which in turn projects to the NAc to influence active avoidance (LeDoux and Daw, 2018). By contrast, a distinct circuit involving the central amygdala and periaqueductal gray is thought to mediate a competing fear-induced freezing response (Johansen et al., 2011). It is possible that this latter ‘fear’ circuit contributes to the cue-evoked suppression in DA release observed early in avoidance learning and that the initial attenuation in avoidance observed in the within-session economic avoidance task. Indeed, optogenetic activation of the periaqueductal gray produces defensive escape-like responses that we believe contribute to these suppressed responses (Assareh et al., 2016) in addition to the aforementioned habenula. The prefrontal cortex is thought to mediate the switch from the central amygdala-periaqueductal gray mediated freezing response and the basolateral amygdala-NAc mediated active avoidance response (LeDoux and Daw, 2018; Moscarello and LeDoux, 2013). Of the various regions that make up the prefrontal cortex, the orbitofrontal cortex seems to be particularly important for aversive learning (Jean-Richard-dit-Bressel and McNally, 2016). While dopamine concentration within the prefrontal cortex may play a particularly important role in active avoidance, the data presented herein suggest that it is not an exclusive one.

11. Concluding statement

In this review we describe how our recent behavioral economic investigations fit into the existing literature on the role of dopamine in appetitively and aversively motivated behavior. By performing demand analyses on operant data, we confirm that dopamine causally modifies the valuation of reward. However, we also demonstrate that dopamine plays a role in aversively-motivated defensive behaviors, including: avoidance, escape, and conditioned freezing. These observations support a more nuanced role for dopamine in adaptive behavior. Few neurochemicals serve a single behavioral function like reward processing. Rather, the evolutionarily-conserved mesocorticolimbic dopamine pathway was likely selected because it promotes an array of advantageous behaviors by integrating experience with emotion. What remains unknown is when, where, and how dopamine release mediates these distinctly advantageous behaviors. As this special issue will attest, various groups report that dopamine is released into different neural substrates at different times by a variety of appetitively- and aversively-associated stimuli. While many studies are required before a consensus is reached, recent advances in behavioral design, neural monitoring, and computational analysis will ensure we know exponentially more on the role dopamine plays in aversion ten years into the future than we knew ten years ago.

HIGHLIGHTS.

• Within an economic context DA causally modifies the price rats pay for reward.

• We also describe DA’s roles in in avoidance, escape and conditioned fear.

• A new economics-based approach to assess the value of avoidance is described.

• In this context, DA causally modifies the price rats pay to avoid footshock.

Abbreviations:

VTA

ventral tegmental area

NAc

nucleus accumbens

RPE

reward prediction error

DA

dopamine

GABA

gamma-aminobutyric acid

FSCV

fast-scan cyclic voltammetry

H2O2

hydrogen peroxide

NO

nitric oxide

Hz

hertz