Dopamine, Updated: Reward Prediction Error and Beyond

Abstract

Dopamine neurons have been intensely studied for their roles in reinforcement learning. A dominant theory of how these neurons contribute to learning is through the encoding of a reward prediction error (RPE) signal. Recent advances in dopamine research have added nuance to RPE theory by incorporating the ideas of sensory prediction error, distributional encoding, and belief states. Further nuance is likely to be added shortly by convergent lines of research on dopamine neuron diversity. Finally, a major challenge is to reconcile RPE theory with other current theories of dopamine function to account for dopamine’s role in movement, motivation, and goal-directed planning.

Introduction

Dopamine neurons of the ventral tegmental area (VTA) have been intensely studied for their role in reinforcement learning. These dopamine neurons encode reward prediction error (RPE) – the difference between predicted and received rewards. RPE encoding in dopamine neurons was first suggested by Schultz et al. in 1997 [1]. The hypothesis that dopamine neurons function to encode RPE gripped the attention of many neuroscientists due to its excellent coherence with the predictions of temporal difference reinforcement models of learning [2,3]. Now, more than 20 years after the initial suggestion, there is little doubt that RPE is an essential part of what dopamine does in the brain. But is it everything? Recent progress in dopamine research has led to stronger formulations of the RPE hypothesis, which have refined the theory while largely confirming the initial observation. However, it remains a challenge to incorporate other hypothesized functions of dopamine, for example in motivation and motor behavior, into the RPE hypothesis. Whether dopamine’s functions can indeed be united under one theory is not clear: many recent studies have also brought attention to previously underappreciated dopamine neuron diversity. This diversity – in gene expression, intrinsic currents, synaptic connectivity, and neurotransmitter co-release – could give rise to a diversity of functions not amenable to description under one theory. In this review, we provide a brief synopsis of exciting advancements in understanding how dopamine neurons encode RPE, as well as an overview of emerging evidence for dopamine’s other functions in reward-related, motor, and cognitive behaviors and their relationship to the theme of cell type diversity.

Advancements in Understanding Dopamine’s Encoding of Reward Prediction Error

The advent of optogenetics in the early 2000s brought a revolution to our investigations of dopamine function. In particular, an important early study by Steinberg et al. [4] formally demonstrated causality in RPE by testing a strong, formal version of the RPE hypothesis using a behavioral procedure known as “blocking.” In the blocking procedure, an animal first learns that a single cue (A) fully predicts a reward (R). Once the association is learned, it “blocks” new learning about redundant cues. For example, if animals are trained that A →R and then that AX →R, they will not learn that X→R. Since A is fully predictive of R, the additional presentation of X does not add any predictive value. Therefore, there is no RPE generated by dopamine neurons at the reward delivery time and no new learning about X occurs. Steinberg et al. found that optogenetically imposing an RPE by stimulating VTA dopamine neurons at the time of reward during compound (AX→R) training could unblock learning. This experiment showed that VTA dopamine neurons were not only computing a signed, quantitative RPE, but that this RPE signal was sufficient to update the animal’s value learning system and change behavior.

Recently, Maes et al. [5] have built on these findings using a similar blocking paradigm in conjunction with optogenetic inhibition of VTA dopamine neurons, in this case during the time of cue presentation. By manipulating cue-evoked dopamine signals, they tested whether cue-evoked dopamine transmission is an RPE signal. They trained animals that A→R, and then that both AX →R and AY→ R. Normally, learning about X and Y would be blocked. The experiment was then to optogenetically inhibit dopamine neurons during presentations of AX (but not AY) and ask whether this inhibition unblocks learning about X compared to Y. This clever design can distinguish between two possibilities for what cue-evoked dopamine might be encoding: if cue-evoked dopamine encodes a value prediction, suppressing it should unblock learning about X (X did not predict reward; therefore, reward was unexpected; therefore, one should learn about X). On the other hand, if cue-evoked dopamine encodes reward prediction error, then the lack of an error should not generate any new learning. The authors found that learning about X and Y was equally blocked. This result provides evidence that dopamine encodes a strictly-defined RPE that is not to be conflated with the prediction itself.

Other recent optogenetic stimulation experiments also support the same conclusion: optogenetic stimulation of VTA dopamine neurons can promote associations between cues and rewards or between cues and other valueless cues, without causing any cues to acquire value [6,7]. There is evidence that predicted value is encoded by inputs to dopamine neurons – e.g., the prefrontal cortex – but NOT by dopamine neurons themselves [8,9]. Together, these experiments help to rule out the direct encoding of value or value prediction by dopamine neurons, solidifying RPE theory.

Additionally, Morrens et al. [10] showed that novel, but not familiar, cues evoke dopamine release and that if dopamine release is inhibited during a novel cue, learning about that cue is impaired. These findings help explain why animals learn more quickly about novel cues. They also fit well with another important, recently developed idea that dopamine acts not only as a prediction error for reward values but also as a prediction error for sensory experiences (“sensory prediction errors”; [11–14]). The idea that cue-evoked dopamine occurs before associations are learned, and helps guide that learning, also fits with ideas about how novelty responses can help mark events with an unknown potential for reward association [15].

How do dopamine neurons encode prediction errors? An exciting new idea, that of a distributional RPE code, is inspired by artificial intelligence research. The idea is that instead of all dopamine neurons encoding a similar mean prediction error, each neuron might encode either a more “optimistic” or more “pessimistic” prediction, distributed around the mean. Using many cells to represent a range of predictions allows the brain to capture a full probability distribution for future rewards and in theory improves reinforcement learning. Indeed, Dabney et al. [16] found strong evidence for distributional RPE encoding in single-unit recordings from VTA dopamine neurons in mice performing probabilistic learning tasks. These findings fit with previous recordings by the same group, which showed that individual VTA dopamine neurons compute complete RPEs (via subtraction), albeit with heterogeneity in scaling [17–19]. Distributional RPE encoding is likely achieved through variations in intrinsic excitability as well as in the complement of inputs each individual dopamine neuron receives from a variety of brain areas [20].

A final, important recent addition to our understanding of how dopamine neurons encode RPE is the proposal that RPEs depend on hidden belief states. A “state” is a neural representation of environmental conditions at a given time. However, deterministic external signals about the animal’s current state are not always available. Therefore, belief states are created in cases where an animal must rely on some probabilistic assessment of what state it is likely to be in. To test whether belief states contribute to RPE encoding by dopamine neurons, several different tests have been designed in which the external cues indicating state are ambiguous. One test used perceptual ambiguity in the cues predicting reward [21], another created ambiguity by using the same cue to predict both large and small rewards in randomly alternating (unsignaled) blocks [22], another used variable cue-reward intervals combined with probabilistically-delivered rewards to cause ambiguity about whether particular trials were long-interval trials vs. omission trials [23]. In all of these manipulations, the responses of VTA dopamine neurons were well accounted for by RPE models incorporating belief states. Therefore, RPE signals are not necessarily accurate representations of error, but reflect an animal’s internal understanding of its environment based on previous experiences in that environment.

Going Beyond Reward Prediction Error

In the studies discussed thus far, researchers focused on VTA dopamine neurons. Even more specifically, these studies focused primarily on dopamine neurons in the lateral VTA, which have a specific set of cellular, molecular, and synaptic properties compared to dopamine neurons as a whole, and which primarily project to the nucleus accumbens lateral shell [24,25]. Not all dopamine neurons are alike. For example, using the blocking paradigm described above, Keiflin et al. [14] showed that while optogenetic stimulation of VTA dopamine neurons at the time of reward can unblock learning, optogenetic stimulation of nearby substantia nigra pars compacta (SNc) dopamine neurons cannot. These results, and others, suggest that SNc dopamine neurons do not encode RPE, despite being capable of acting as a primary reinforcer [14,26–28]. Another set of dopamine neurons in the far lateral portion of the substantia nigra (SNL), which project to the caudal tail of the striatum, receive a distinct array of inputs compared to VTA and SNc, and respond to salient, novel stimuli [29–32]. Other recent studies additionally emphasize heterogeneity in the encoding properties of VTA dopamine neurons [33,34], adding to a story of heterogeneity from earlier studies of projection-defined VTA dopamine neurons [25,35]. Thus, there is a growing need in the field to add specificity in our methods of identifying dopamine neurons and to carefully report which subsets of dopamine neurons are under scrutiny in any given experiment.

Diverse Molecular Phenotypes of Midbrain Dopamine Neurons

Studies identifying heterogeneity in dopamine neurons based on their locations and projection targets have emphasized the need to answer the question: just how heterogenous is the midbrain dopamine system? How do we define dopamine cell types? Molecular approaches are offering some insight. Using single-cell gene expression analyses, researchers have proposed the existence of ~4–7 distinct groups of midbrain dopamine neurons defined by the expression of genes such as Aldh1a1, Sox6, and Vglut2 [36–41]. These exciting studies offer promise in explaining the heterogeneity of dopamine neuron responses in vivo. Still, work remains to be done to map molecularly-defined groups of neurons onto anatomically-defined groups. Poulin et al. [42] took a step in this direction when they used intersectional genetic strategies to create mouse lines for selecting out molecularly-defined dopamine neuron subtypes for study. They mapped the projection patterns of various subtypes and noted different (but overlapping) patterns of forebrain innervation. In terms of input connectivity, previous studies have indicated that projection-defined dopamine neurons receive distinct patterns of input [29,43–45], but it is as yet unclear how this aspect of connectivity interfaces with the molecular phenotypes of dopamine neurons.

How do molecularly-defined populations of dopamine neurons contribute to behavior? One recent study addressed this question for aldehyde dehydrogenase-positive (Aldh1a1+) dopamine neurons in the SNc and found that this population is crucial for the acquisition of motor skills in the rotarod task [46]. However, the question of what Aldh1a1+ dopamine neurons are encoding during motor skill acquisition remains. Another recent study defined VTA dopamine neuron subtypes by their expression of the neuropeptidergic markers Crhr1 (corticotropin-releasing hormone receptor 1) and Cck (cholecystokinin) [47]. They found that Crhr1- and Cck-expressing VTA dopamine neurons project to the core and medial shell of the nucleus accumbens, respectively. These two groups of dopamine neurons were critical for distinct parts of behavioral reinforcement: Crhr1-expressing VTA dopamine neurons were critical for establishing instrumental action-outcome and Pavlovian cue-reward associations, while Cck-expressing VTA dopamine neurons helped motivate responding for a reward after an action-outcome relationship was established. These findings suggest that while most dopamine neurons increase their activity in response to rewards, different subtypes are critical for individual features of reward processing. Future studies examining the interface between neuropeptidergic phenotype and the gene expression patterns defined by single-cell sequencing studies, however, are still necessary to fully define dopaminergic heterogeneity at the molecular level and to align these definitions with behavioral observations.

Dopamine Neurons Use More than Dopamine to Communicate

One important consequence of variations in gene expression in midbrain dopamine neurons is variations in neurotransmitter co-release. It has become increasingly evident that dopamine neurons can release more than their namesake: subsets of dopamine neurons co-release classical fast neurotransmitters such as glutamate and GABA [48–54]. The relevance of co-release for behavior is an area of active investigation and debate. For example, concerning glutamate co-transmission, Wang et al. demonstrated that co-release of glutamate from dopamine terminals in the ventral striatum does not contribute to acquisition of intracranial self-stimulation (ICSS) or real-time place preference (RTPP) for optogenetic stimulation [55]. Meanwhile, Zell et al. found that conditionally ablating tyrosine hydroxylase from neurons that co-release glutamate and dopamine also did not affect ICSS acquisition and RTPP, despite a loss of dopamine release in the ventral striatum [56]. Together, these studies suggest redundant roles of glutamate and dopamine release in “dopamine” neurons, at least in the specific behavioral paradigms examined. Future studies are necessary to ascertain the exact timing, signaling dynamics, and downstream effects of co-transmitter release by dopamine neurons, and elucidate their functions across all of striatum. Notably, the possibility of co-release means that studies of dopamine cell body firing in the midbrain cannot necessarily claim that the downstream circuit effects of this firing are due to dopamine release alone. To determine the contributions of dopamine release per se to downstream circuit function in future experiments, new tools such as fluorescent dopamine sensors [57–60] and sensors allowing researchers to track the biochemical responses of dopamine receptor-expressing neurons during behavior [61] will be instrumental.

Dopamine and Movement

Parkinson’s disease – in which SNc dopamine neurons slowly degenerate – has long made evident the importance of dopamine in spontaneous movement. Yet, it remains unclear how dopamine neurons participate in motor control. Recent studies have deepened our understanding, demonstrating that dopamine neurons – primarily in the SNc but also in the VTA – show phasic activity related to different components of movement such as action choice, initiation, vigor, and velocity [62–70]. These signals have not been explained by RPE theory and, using excitatory optogenetics, have been shown to be sufficient to trigger movement. Still, the physiological role of movement-related dopamine neurons is under debate. Coddington and Dudman have suggested that the portion of dopamine neurons that are naturally active during movement initiation is small and that optogenetic stimulation of dopamine neurons must be extremely supraphysiological to evoke movement directly [71]. Notably, when Wu et al. ablated Aldh1a1+ SNc dopamine neurons (a sizeable portion of the SNc dopamine population, and the dopamine neurons most vulnerable in Parkinson’s Disease), they did not observe immediate parkinsonism, but merely a slight decrease in high-speed movements combined with a profound motor learning deficit [46]. This result suggests that SNc dopamine neurons are not primarily responsible for movement initiation and that parkinsonism arises mostly from slower basal ganglia-wide adaptations to progressive dopamine loss.

Movement-related dopamine neurons are largely located in SNc, where they encode contralateral actions [62,63,72]. Yet, there is also evidence for the representation of movement in VTA dopamine neurons. Using a ‘Go-NoGo’ task, Syed et al. [68] demonstrated that if an animal did not need to initiate movement to obtain a reward, dopamine release in the NAc core in response to a cue predicting the reward was attenuated. Engelhard et al. [33] have also argued for the representation of kinematic variables by at least a subset of VTA dopamine neurons, alongside RPE representation. Furthermore, Hughes et al. used sensitive measurements of force generated by mice in a head-fixed fixed-time interval task to observe three populations of VTA dopamine neurons representing different aspects of forward and backward movement relative to a reward lick spout [70]. Although the firing rates of dopamine neurons they recorded were tightly linked to movement, these movement relationships were only observed in highly trained, not naïve, mice. Based on these results, it seems likely that dopamine neurons track both movement and RPE, often simultaneously in the same cells. There may be good reason for dopamine neurons to track both signals together: to assign credit to our actions accurately, we need to detect when rewards are missed not because of misunderstandings in cue-outcome or action-outcome relationships, but because of errors in motor execution [73; see also 74]. This is an enticing idea, but more work is still required to understand how movement signals interact with RPE to drive motivated behavior and motor learning.

Dopamine, Motivation, and Planning: What Are Dopamine Ramps For?

Dopamine “ramps,” in which dopamine activity and release increases as an animal gets closer to reward, have been proposed as a mechanism of maintaining motivation to work for distal rewards and properly linking credit back to the initial cue or action that resulted in a positive outcome [69,75,76]. Notably, ramping occurs more readily when mice are participating in instrumental rather than Pavlovian tasks [77–79]. The phenomenon has been observed robustly in the VTA and ventral striatum [69,75,76,78–82], as well as in the dorsomedial striatum [77,83]. There is less evidence for strong ramping in the dorsolateral striatum [75,77,83].

The origin of ramping activity remains controversial. Ramping is reliably observed when measuring the postsynaptic release of dopamine (using microdialysis, voltammetry, and dopamine sensor imaging) [69,75–78,81]. It has also been observed when measuring the activity of dopamine axons in the striatum using calcium sensors [77,78,83]. However, controversy exists as to the observation of ramps in cell body activity. Mohebi et al. [81] report an absence of ramping in VTA cell bodies (measured using electrophysiology) even when ramps are observed in the NAc (measured using voltammetry and dopamine sensor imaging), suggesting ramps could be locally generated in axons. However, others have observed ramps when measuring cell body activity, which are more obvious when population measures such as fiber photometry are used instead of electrophysiological recordings of single units [33,78–80,82]. Thus, there is an argument about whether ramps are generated in cell bodies, and, if they are, whether axonal mechanisms could still accentuate them.

What is the computational role of ramps? Again, controversy exists, and the answer may vary depending on the subpopulations of dopamine neurons being examined. To date, investigations of dopamine heterogeneity have largely not been brought to bear on the question of dopamine ramping. One school of thought is that dopamine ramps can be successfully incorporated into RPE theory [2,69,77,84,85]. Ramping could be explained by the back propagation of RPEs during learning and thus used to reinforce action choices [2,69,85]. Mikhael et al. [84] argue that sensory feedback plays an important role in generating ramps from RPE by reducing the uncertainty about time to reward.

Another school of thought is that dopamine ramps are better explained as motivational signals, or as a means of coordinated goal-directed action planning [75,76,79,81]. Song and Lee [82] suggest a model in which a ramping-to-phasic transition in dopamine signaling is related to a reduction in task dimensionality as subjects narrow their focus to relevant stimuli. They suggest that ramps should fade as instrumental learning asymptotes and habits begin to form [76,82]. In contrast, Guru et al [79] observe that ramps generally fade with repetitive training, and provide evidence that ramps are only maintained under conditions where internal goal representations are necessary to support behavior. These explanations of the roles of dopamine ramping in behavior, as well as a fuller characterization of when dopamine ramps occur and how they are generated, await further investigation. At present, the wide range of tasks and recording methods used to detect ramps, as well as the lack of an agreed-upon definition for ramping, are hindering progress in this burgeoning area of interest.

Conclusion

Much exciting progress has been made in dopamine research in recent years, but challenges lie ahead. The field must begin to map studies of computational function onto molecularly-defined groups of dopamine neurons. Such studies would help to clarify whether or not molecularly-defined groups of dopamine neurons all serve to encode RPE, and whether each group of neurons serves single or multiple computational functions. A comprehensive mapping of computational function onto molecularly-defined dopamine cell types will add richness to both RPE theory and other theories on dopamine’s role in supporting reward- and motor-related behaviors.

Going further, we must also better understand how dopamine signals in distinct subsets of neurons are being generated by upstream circuits and interpreted by downstream circuits. On the former problem, input mapping studies have led the way [20,29,43,44,86] but a good deal of work is left to describe the detailed mechanisms by which these inputs contribute to the encoding properties of dopamine neurons, as well as to understand how such connectivity might be altered by sex, across development and aging, and in conditions (e.g., stress) that are related to the emergence of psychiatric conditions. On the latter problem, we are only beginning to understand the transformations that take place between dopamine cell body activity, axon activity, and neurotransmitter release [81]. The astoundingly complex axonal arbors of midbrain dopamine neurons [87] provoke many questions about how action potential propagation is regulated. Complex neuromodulatory mechanisms are known to regulate dopamine release at terminals [88,89]. Different release machinery can regulate dopamine release on different time scales [90,91], and yet further mechanisms may regulate the differential co-release of other neurotransmitters [50,52]. Thus, even when we observe RPE signals while recording dopamine neuron action potentials at cell bodies, we may question whether and in what form downstream circuits are receiving these signals. As researchers tackle these difficult questions in the coming years, theories of dopamine function will, no doubt, again need to be updated.

Highlights.

• Reward Prediction Error (RPE) remains a compelling theory accounting for VTA dopamine neuron activity

• RPE theory now incorporates sensory prediction errors, distributional encoding, and belief states

• Molecular subtyping of dopamine neurons provides a scaffold for probing additional computational functions

• Whether dopamine neuron activity related to movement, motivation, and goal-directed planning can be explained by RPE is under debate