Aversion Hot Spots in the Dopamine System

Abstract

Through the development of optogenetics and other viral vector-based technologies, our view of the dopamine system has substantially advanced over the last decade. In particular, progress has been made in the reclassification of dopamine neurons based on subtypes displaying specific projections, which are associated with different features at the molecular, anatomical, molecular and behavioral level. Together, these discoveries have raised the possibility that individual groups of dopamine cells make a unique contribution to the processing of reward and aversion. Here, we review recent studies that have identified non-canonical dopamine pathways that are activated in response to aversive stimuli, including dopamine projections to the ventromedial shell of the nucleus accumbens, prefrontal cortex, tail of the striatum and amygdala.

Dopamine: From Reward to Aversion

For the survival of any organism, it is crucial to avoid stimuli that are potentially harmful to the body, including toxins, predators and mechanical stressors. Accordingly, a large part of the brain is dedicated to avoiding environments and stimuli that may expose one to such harmful agents, providing an aversion-based learning system that promotes avoidance and defensive behaviors [1,2]. Historically, the dopamine (DA) system has mostly been associated with reward and reward-based learning, but a substantial body of evidence suggests a similarly important role for this system in the processing of aversive experiences [3–8].

A quantitative way to describe the role of DA in value-based learning is based on the reward prediction error (RPE) hypothesis, which represents the difference between the predicted reward and the reward actually received [9,10]. As such, the activity of DA neurons, dependent on the experience of an organism at any moment in time, is thought to correlate directly with the magnitude of this RPE. Thus, when a received reward is higher than the predicted reward, there is a positive RPE, resulting in increased firing of DA neurons; and when a received reward is lower than the predicted reward, the RPE is negative, and DA neuronal firing is decreased. Finally, when a received reward could be fully predicted by an organism, the RPE is zero and the rate of DA neuron firing does not change. Similarly, an unexpected aversive event, such as a punishment, evokes a negative RPE-evoked reduction in DA neuron firing, whereas a fully predictable aversive event does not change firing [9,11,12]. In this way, DA is thought to directly contribute to computations that are necessary to guide learning processes in the brain, consistent with influential theoretical models of value-based learning [13,14].

Although encoding of RPE by DA neurons has been well established, data from many laboratories over the last decades have shown that that not all DA neurons strictly follow the RPE model. Indeed, our view of the DA system has changed substantially over the last decade. For example, reclassification of DA neurons based on subtypes displaying specific projections, which are correlated with different features at the molecular, anatomical, and electrophysiological level [15–19], has given strong emphasis to the possibility that individual groups of DA cells show very different responses to rewarding and aversive stimuli. In direct contrast with the RPE hypothesis, increased DA activity or DA release in response to aversive events has been reported [20–25]. The idea that DA may not act as a unitary ‘reward signal’ has led to a long-standing controversy in the field [5,8,26]. However, through the advancement of viral vector-based approaches (e.g., cell type- and projection-specific electrophysiology, in vivo imaging and optogenetics), convincing evidence has been provided that DA neurons can be divided into a much larger number of functionally distinct populations than previously assumed [27–33].

In this short review, we highlight recent studies that have identified non-canonical (i.e., aversion-activated) DA neurons based on their specific downstream projection targets (Figure 1). While we mainly focus on ventral tegmental area (VTA) DA neurons projecting to subregions of the nucleus accumbens (NAc; mesolimbic pathway) or medial prefrontal cortex (mPFC; mesocortical pathway), we will also briefly discuss amygdala-projecting DA neurons (mesoamygdaloid pathway) and DA neurons in the lateral aspect of the substantia nigra (SN) projecting to tail of the striatum (TS), as well as DA neurons located in the dorsal raphe nucleus (DR) and periaqueductal gray (PAG).

Figure 1. Aversion pathways within the dopamine system.

In the schematic overview, dopamine subpopulations that have been shown to be activated by aversive stimuli are highlighted based on their anatomical connectivity between ventral tegmental area (VTA) and substantia nigra (SN) and projection targets in the medial prefrontal cortex (mPFC), ventromedial nucleus accumbens shell (vNAcMed), basolateral amygdala (BLA) and tail of the striatum (TS). Dopamine neurons in the dorsal raphe nucleus and periaqueductal gray (PAG) project to the bed nucleus of the stria terminalis (BNST) and central nucleus of the amygdala (CeA) (BNST and CeA projections are not shown).

The Striatum

With regard to aversive learning, a formal distinction can be made between (1) instrumental aversive learning, which is comprised of (1a) punishment learning and (1b) active avoidance and escape learning, and (2) aversive Pavlovian conditioning [34]. A punishment refers to a direct adverse outcome of an action, and it suppresses the future expression of that behavior. In contrast, during active avoidance and escape learning, an animal learns to take an action in order to avoid an aversive stimulus, such as a foot shock (in a laboratory) or predator attack (in the wild). Finally, during aversive Pavlovian conditioning, a neutral stimulus (e.g., sound) is paired with an aversive stimulus (e.g., foot shock), leading to a stimulus-outcome association that is independent of an animal’s behavior.

Although such a formal distinction is useful for studying aversion in a laboratory setting, in real life situations, these processes often occur in concert. For example, when a foraging animal endures a predator attack, it may suppress future exploratory behaviors in order to avoid attacks (punishment). In addition, it may escape from a location that resembles the place in which the attack has occurred (active avoidance), which may especially be the case when it hears, sees or smells the same predator (Pavlovian conditioning). That being said, there is considerable evidence that different brain structures are involved in these distinct types of aversive learning [34].

The striatum is a major target of midbrain DA neurons, and it is ideally positioned to be a key structure for processing aversive stimuli. The striatum not only receives many cortical and subcortical inputs related to processing of sensory cues, location, memory and emotion, and as such may guide which cues need to be avoided, but it also represents an important output to basal ganglia structures, thereby allowing an animal to quickly respond to overt threats [35–38]. While the striatum is often divided into a ventral and dorsal subdivision, based upon the function of and connection from VTA and SN DA neurons, respectively, more recent studies have suggested more complex DA innervation patterns along a mediolateral gradient that applies to both the ventral and dorsal striatum [16,28,30,39–41]. However, whether DA neurons projecting to the striatum, and in particular the NAc, are excited or inhibited during aversive experiences has been controversial for many decades. There is a large body of literature based on microdialysis and fast-scan cyclic voltammetry showing that a number of different noxious stimuli (e.g., tail pinch, foot shock, stress) can both increase and decrease DA transmission in the NAc [6,20,21,24,42,43]. In contrast, most electrophysiological studies reported that the majority of VTA DA neurons are inhibited in response to aversive stimuli [9,11,12] (cf. [22,23,44]), consistent with RPE theory.

Several attempts have been made to explain the increase in DA release in response to aversive stimuli, and to resolve the apparent discrepancy between microdialysis and voltammetry recordings (showing differential responses to aversive stimuli) and electrophysiological recordings (showing unitary encoding of RPE). For example, it has been proposed that (1) DA neurons form a heterogeneous population tuned to either rewarding, aversive and/or both (in terms of general motivational salience) stimuli [5,8], (2) aversive stimuli may reflect physical impact rather than aversiveness [9,26,45], (3) reward or relief is experienced when an aversive stimulus is terminated [9,46], (4) high-reward contexts may contribute to the excitatory responses to aversive stimuli [47], and (5) DA neurons transmit a safety signal when an animal avoids an aversive event successfully [48,49].

With the advancement of novel cell type- and projection-specific targeting strategies, the argument that DA neurons show high levels of anatomical and functional heterogeneity has received considerable attention over recent years. These studies have given strong emphasis to the possibility that DA neurons, defined by their downstream projection targets, make a unique contribution to the processing of rewarding and aversive stimuli [27,28,30–33]. For example, de Jong et al. simultaneously recorded DA release in different ventral striatal subregions during an aversive conditioning task using dLight fiber photometry [28]. They found that DA was released in the ventromedial striatum (i.e., ventral NAc medial shell (vNAcMed)) in response to unexpected aversive outcomes and to cues that predict them, whereas DA release in other ventral striatal subregions (e.g., NAc lateral shell) was persistently decreased. Such a spatial separation is consistent with a previous fast-scan cyclic voltammetry study demonstrating that a fear-evoking cue increased DA release in the NAc medial shell [21]. The finding that vNAcMed-projecting neurons release DA in response to aversive stimuli was corroborated by another research group that systematically compared calcium dynamics in DA terminals across different subregions of the NAc [33]. Interestingly, ablation of glutamatergic neurons in the lateral hypothalamus led to an inability of vNAcMed-projecting DA neurons to respond to aversion-predictive cues, without affecting the neurons’ propensity to release DA in response to the aversive event itself [28], which points to a pivotal role of this pathway in aversive learning.

In addition to the vNAcMed, other striatal aversive hot spots have been identified in the posterior dorsolateral striatum and tail of the striatum. For example, Lerner et al. described two functional classes of DA neurons located medially and laterally in the SN which project to dorsomedial and dorsolateral striatum, respectively; the dorsomedial striatum-projecting DA neurons showed a marked decrease in activity at the time of an aversive stimulus, whereas the dorsolateral striatum-projecting DA neurons showed an increase [30]. These findings are consistent with those put forward by Matsumoto and Hikosaka (2009), who showed that DA neurons located more laterally within the SN increase firing in response to aversive stimuli [23], as well as Menegas et al., who reported that aversion-activated DA neurons in the lateral SN project to the tail of the striatum (TS) [50]. This latter study also demonstrated that optogenetic stimulation of DA terminals in the TS caused avoidance in a real-time place preference assay in mice [32], as opposed to place preference that would be expected when optogenetically stimulating VTA DA cell bodies in a non-projection defined manner [51]. Interestingly, DA terminals in the TS were activated by novel stimuli (either neutral, aversive or rewarding), and these novelty responses attenuated over time when these cues were paired with certain unconditioned stimuli [52]. Similarly, de Jong et al. showed that vNAcMed-projecting DA neurons responded to unpredicted rewarding stimuli [28] suggesting that both vNAcMed- and TS-projecting DA neurons, though anatomically separated in terms of cell body location (medial VTA versus lateral SN), may signal motivational salience. Additionally, medial VTA and lateral SN DA neurons express vesicular glutamate transporter 2 genes (VGLUT2) suggesting that they are capable of co-releasing glutamate and DA. Conversely, canonical (i.e., RPE-encoding) DA neurons in the lateral VTA and medial SN DA neurons lack expression of VGLUT2 [18,41,53]. Thus, an unresolved question is what the functional role of glutamate and DA co-release in these neurons is, and whether it is necessary for aversion-related behaviors [54].

The Medial Prefrontal Cortex

While DA release in the mPFC has shown to be involved in complex forms of decision-making, planning and working memory [55–57], there is also evidence that it is crucial for certain aversion-based learning processes. This could be due to the mPFC being a “top-down” controller of reward-based decision making [56,58]. In other words, when consequences associated with a behavior become aversive, the mPFC may be involved in decreasing the frequency of that behavior through behavioral inhibition. Abercrombie et al. (1989) was one of the first studies that observed increased DA transmission in the mPFC in response to aversive tail-shock stress [59]. Since then, several studies have suggested, though often based on indirect evidence, that the mesocortical DA pathway may play a role in processing aversion-related information.

For example, Lammel et al. (2011) showed that excitatory inputs onto mesocortical DA neurons are potentiated in response to an aversive experience [60]. The same authors showed in a subsequent study that lateral habenula neurons selectively target both mesocortical DA neurons and GABAergic neurons in the rostromedial tegmentum and that activation of these lateral habenula efferents promotes place aversion [61]. More direct evidence for a role of the mesocortical DA pathway in aversion is based on experimental data showing that optogenetic stimulation of VTA DA terminals in the mPFC promotes conditioned place aversion [62], although it should be noted that in another study optogenetic stimulation of that same pathway had no effect on place preference behavior [63]. Indeed, a key challenge for studying the mesocortical pathway is the relatively sparse DA innervation of the mPFC in mice as compared to the dense DA innervation of the striatum. In addition, there seems to be no or very little expression of the dopamine transporter in the mesocortical DA system [16], which together presents a caveat for performing genetically-targeted optogenetic manipulations or fiber photometry recordings when using transgenic mouse lines [64,65].

Despite these limitations, Kim et al. (2016) successfully used fiber photometry to study calcium transients in DA terminals in the mPFC in response to different salient stimuli. They found increased calcium transients in response to an aversive tail shock, but not to a water reward [29]. More recently, Vander Weele et al. (2018) confirmed these findings and further suggested that DA release increases the signal-to-noise ratio of neuronal responses to aversive stimuli of mPFC neurons projecting to the dorsal periaqueductal gray [66]. Although these studies point to a role for DA release in the mPFC in aversive behaviors, it will be important to further examine how these findings integrate with other well-established DA-dependent cortical functions such as working memory.

The Amygdala

VTA DA neurons also project to the basolateral amygdala, a brain region that is well known to be involved in fear responses through aversive Pavlovian conditioning [67–69]. As such, it has been demonstrated that restoration of DA transmission in the basolateral amygdala of DA-deficient mice can restore some short-term aspects of Pavlovian fear memory [70]. However, these effects may extend beyond aversive Pavlovian conditioning, since DA can also potentiate appetitive Pavlovian conditioning through activation of DA D1 receptors in the lateral amygdala [71]. More recently, Lutas et al. (2019) demonstrated that VTA DA terminals in the basal amygdala are excited by stimuli that carry motivational salience and cues that predict them. The authors further showed that hunger reduced excitatory responses of DA terminals to reward while simultaneously strengthening excitatory responses to aversive stimuli, indicating that activity of the mesoamygdaloid DA pathway is dependent on the motivational state of the animal [31]. Together, these studies suggest an important role of amygdala-projecting DA neurons in mediating cue-outcome associations, irrespective of the valence of the outcome.

Dopamine Neurons in Dorsal Raphe and Periaqueductal Gray

Thus far, we have focused on brain regions that represent prominent downstream projection targets of VTA and SN DA neurons. However, DA neurons are also located in the DR and PAG, although they are more sparsely distributed when compared to the ventral midbrain [72]. It remains debatable whether DA neurons in the DR and PAG represent separate populations, since both structures are adjacent to each other and DA cells in both regions project to the bed nucleus of the stria terminalis and the central nucleus of the amygdala [27,72–74]. That being said, recent studies have suggested an important role of these DA neurons in aversion-related behaviors. For example, Matthews et al. demonstrated a role for DR DA neurons in the aversive properties of social isolation in mice. As such, social isolation evoked synaptic changes in DR DA cells, and in vivo calcium imaging revealed that these neurons showed higher responses during social interaction when the subject had been socially isolated. In addition, optogenetic activation of DR DA neurons caused place avoidance, while it also increased social preference (similar to prolonged social isolation), suggesting a potential role of these neurons in mediating ‘loneliness’ [74]. More recently, Groessl et al. (2018) showed that DA is released by DR/PAG neurons in response to foot shock, and contributes to aversive Pavlovian conditioning by facilitating plasticity in the central amygdala. Interestingly, during the aversive Pavlovian learning process, the responses of these DR/PAG DA neurons shifted from the unconditioned (shock) to the conditioned (shock-predicting tone) stimulus, suggesting that these neurons encode a ‘sign inversed’ RPE signal, rather than general motivational salience, thereby providing a neurocomputational mechanism through which DR/PAG DA may facilitate learning [27]. Together, these studies point to a unique role of DR/PAG DA neurons in specific types of aversion-related motivated behaviors.

Conclusions

Historically, DA is thought to contribute to aversive learning processes through homogeneous coding of RPEs, thereby reinforcing and punishing actions that triggered rewarding or aversive stimuli, respectively. However, a growing body of evidence suggests a more complex picture: several subpopulations of DA neurons have been discovered that show robust activation in response to aversive stimuli and cues that predict them. These DA neurons may be identified based on their specific projection targets to vNAcMed, mPFC, amygdala and TS. They seem to represent non-canonical DA neurons that have electrophysiological and molecular properties that are largely different from classical DA neurons in the lateral VTA, which are known to be inhibited in response to aversive events. Interestingly, DA subpopulations that are activated in response to aversive stimuli are clustered in the medial VTA and to some degree in the lateral SN as well as DR/PAG.

The discovery and delineation of these aversive hot spots now paves the way for a more thorough investigation into the anatomy, genetics and function of the DA subtypes that innervate these aforementioned structures. While an extensive amount of research has focused on the role of lateral VTA DA neurons underlying RPE encoding, much remains to be learned from medial VTA DA neurons and how these cells may contribute to aversive learning. A key challenge will be to identify pathway-specific molecular markers that can be used to selectively target DA cell subtypes [18,75,76]. Although it will certainly take some time to develop a detailed understanding of medial VTA DA neurons in a way we have achieved it for the lateral VTA, the initial discoveries of aversive DA hot spots provide crucial new insights into the functional architecture of the DA system. Together with recent advances in human neuroimaging of the DA system [4,77,78], it is possible that continued research on the heterogeneity of DA neurons will shed light on how anatomically and functionally distinct DA subsystems contribute to the variety of symptoms in neuropsychiatric disorders.

Highlights.

• DA neurons show high levels of anatomical and functional heterogeneity

• Role for projection-defined DA neurons in processing aversive stimuli

• Identification of aversive hot spots in vNAcMed, TS, mPFC, BLA