How do dopamine neurons represent positive and negative motivational events?

Abstract

Midbrain dopamine neurons are activated by reward or sensory stimuli predicting reward^1–4. These excitatory responses increase as the reward value increases⁵. This response property has led to a hypothesis that dopamine neurons encode value-related signals and are inhibited by aversive events. Here we show that this is true only for a subset of dopamine neurons. We recorded the activity of dopamine neurons while monkeys were conditioned using a Pavlovian procedure with appetitive and aversive outcomes (liquid reward and airpuff directed at the face, respectively). We found that some dopamine neurons were excited by reward-predicting stimuli and inhibited by airpuff-predicting stimuli, as the value hypothesis predicts. However, more numerous dopamine neurons were excited by both of these stimuli, inconsistent with the hypothesis. Some dopamine neurons were also excited by both reward and airpuff themselves, especially when they were unpredictable. Neurons excited by the airpuff-predicting stimuli were located more dorsolaterally in the substantia nigra pars compacta, whereas neurons inhibited by the stimuli were located more ventromedially, some in the ventral tegmental area. A similar anatomical difference was observed for their responses to airpuff itself. These findings suggest that different groups of dopamine neurons convey motivational signals in distinct manners.

If midbrain dopamine neurons actually encode value-related signals, their activity should be inhibited by aversive stimuli because aversive stimuli have negative motivational values. However, the results are inconsistent, some studies showing inhibitions⁶ while others showing both inhibitions and excitations^7–11 by aversive stimuli. Few of these studies examined the effects of rewards on the same dopamine neurons^{12, 13}, partly because the animals were anesthetized.

To test if dopamine neurons encode motivational values, we conditioned two monkeys using a Pavlovian procedure with two distinct contexts (Fig. 1): one in which a liquid reward was expected (appetitive block, Fig. 1a), and the other in which an aversive airpuff was anticipated (aversive block, Fig. 1b). In each block, three conditioned stimuli (CSs) were associated with the unconditioned stimulus (US, reward or airpuff) with 100%, 50% and 0% probability, respectively. These three CSs were considered to convey three different levels of motivational value. Indeed, in the appetitive block, anticipatory licking increased as the probability of reward increased (Fig. 1c). In the aversive block, anticipatory blinking increased as the probability of airpuff increased (Fig. 1d).

Figure 1. Pavlovian procedure.

a, Appetitive block. Three conditioned stimuli (CSs) were associated with apple juice with 100%, 50% and 0% probability, respectively. b, Aversive block. Three CSs were associated with aversive airpuff with 100%, 50% and 0% probability, respectively. In both blocks, each trial started after the presentation of a timing cue (central small spot) on the screen. After 1 s, the timing cue disappeared and one of the three CSs was presented. After 1.5 s, the CS disappeared and the US (reward or airpuff) was delivered. In addition to the cued trials, uncued trials were included in which a reward alone (free reward) was delivered during the appetitive block and an airpuff alone (free airpuff) was delivered during the aversive block. c, Average of normalized magnitude of anticipatory licking during the presentation of reward CSs for monkey D (solid line) and monkey N (dashed line). d, Average of number of anticipatory blinks during the presentation of the airpuff CSs for monkey D (solid line) and monkey N (dashed line). Double asterisks indicate a significant difference between two data points (P < 0.01, Wilcoxon rank-sum test). Error bars indicate s.d.

While the monkeys were conditioned using the Pavlovian procedure, we recorded single unit activity from 103 putative dopamine neurons (68 in monkey N and 35 in monkey D) in and around the substantia nigra pars compacta (SNc) and ventral tegmental area (VTA). Their electrophysiological properties were distinctly different from other neurons in the SNc and VTA (Supplementary Fig. 1), and hereafter we call them dopamine neurons.

Most previous studies on midbrain dopamine neurons have characterized dopamine neurons as a functionally homogeneous population¹. We found that this is not true.Figure 2a and e show the activity of two dopamine neurons, separately for different CSs. Their activity was similar in the appetitive block (top row). Both of them were excited by 100% reward CS (the CS associated with reward with 100% probability). This excitation decreased in response to 50% reward CS, and even changed to an inhibition in response to 0% reward CS. However, the dopamine neurons showed completely different responses in the aversive block (bottom row). In response to 100% airpuff CS, the neuron shown in Fig. 2a was inhibited whereas the neuron shown in Fig. 2e was excited. Furthermore, as the probability of airpuff decreased, their response magnitudes were graded in opposite directions.

Figure 2. Responses of dopamine neurons to CSs.

a, e, Activity of two example neurons in the appetitive block (top row) and aversive block (bottom row) which were classified as ACS-inhibited type (a) and ACS-excited type (e). Histograms (20ms bins) and rasters are aligned at the onset of the CS and are shown for 100% reward CS, 50% reward CS, 0% reward CS, 100% airpuff CS, 50% airpuff CS, and 0% airpuff CS. b, c, Averaged activity of 24 ACS-inhibited type neurons. f, g, Averaged activity of 38 ACS-excited type neurons. Spike density functions (SDFs) are shown for 100% reward CS (dark red), 50% reward CS (light red) and 0% reward CS (gray) in the appetitive block (b, f), and for 100% airpuff CS (dark blue), 50% airpuff CS (light blue) and 0% airpuff CS (gray) in the aversive block (c, g). Gray area indicates the period that was used to analyze CS-evoked response. d, h, The magnitudes of the responses of the ACS-inhibited type neurons (d) and ACS-excited type neurons (h) to the reward CSs (red) and airpuff CSs (blue). Filled symbols indicate a significant deviation from zero (P < 0.05, Wilcoxon signed-rank test). Red and blue asterisks indicate a significant difference between two responses for reward and airpuff CSs, respectively (double asterisks, P < 0.01; single asterisks, P < 0.05, Wilcoxon signed-rank test). Error bars indicate s.d.

In order to characterize the CS-evoked responses, we classified the 103 neurons into three groups based on the response to 100% airpuff CS (Supplementary table 1). Neurons showing a significant inhibition and excitation were classified as airpuff CS inhibited type (ACS-inhibited type, n=24) and airpuff CS excited type (ACS-excited type, n=38), respectively (P < 0.05, Wilcoxon signed-rank test). Neurons showing no significant response were classified as airpuff CS non-responsive type (ACS-nonresponsive type, n=41) (P > 0.05, Wilcoxon signed-rank test). The CS-evoked responses of individual neurons are shown in Supplementary Fig. 2 and Supplementary table 2. In the following, we will focus on the ACS-inhibited and ACS-excited type neurons (see Supplementary Fig. 3 for ACS-nonresponsive type neurons, see also Supplementary note A and Supplementary table 3 for the electrophysiological properties of each type).

The averaged activity of the ACS-inhibited type neurons was modulated by the reward probability (Fig. 2b) and airpuff probability (Fig. 2c) in the opposite directions. The excitatory response to the reward CS decreased and reversed to an inhibition as the reward probability decreased (Fig. 2b, red line in Fig. 2d). In contrast, the inhibitory response to the airpuff CS decreased as the airpuff probability decreased (Fig. 2c, blue line in Fig. 2d). The same trend was found in individual ACS-inhibited type neurons (Supplementary note B and Supplementary Fig. 4a). These results suggest that the ACS-inhibited type neurons encode motivational value on a single scale, most strongly excited in response to the most positive stimulus (100% reward CS) and most strongly inhibited in response to the most negative stimulus (100% airpuff CS).

The averaged activity of the ACS-excited type neurons was also modulated by the reward probability (Fig. 2f) and airpuff probability (Fig. 2g), but in the same direction. The excitatory response decreased as the outcome probability decreased for both of reward- and airpuff-predicting CSs (Fig. 2h, see also Supplementary note B and Supplementary Fig. 4b for individual neurons). These results suggest that the ACS-excited type neurons do not encode motivational value.

Previous studies have repeatedly shown that dopamine neurons were excited by reward itself when it was unexpected¹. On the other hand, it is still debatable whether they are excited or inhibited by aversive stimuli and, if so, in what context. Figure 3a shows the responses of the same neuron shown in Fig. 2a to reward and airpuff. This neuron was strongly excited when reward was presented without preceding CS (free reward) and inhibited when airpuff was presented without preceding CS (free airpuff), consistent with value coding. In contrast, the neuron shown in Fig. 3e was excited by both free reward and free airpuff.

Figure 3. Responses of dopamine neurons to USs.

a, e, Activity of two example neurons in the appetitive block (top row) and aversive block (bottom row) which were classified as AUS-inhibited type (a) and AUS-excited type (e). Histograms and rasters are aligned at the onset of the US and are shown for 100% reward, 50% reward, free reward, 100% airpuff, 50% airpuff, and free airpuff. b, c, Averaged activity of 47 AUS-inhibited type neurons. f, g, Averaged activity of 11 AUS-excited type neurons. SDFs are shown for 100% reward (dark red), 50% reward (light red) and free reward (gray) in the appetitive block (b, f), and for 100% airpuff (dark blue), 50% airpuff (light blue) and free airpuff (gray) in the aversive block (c, g). Gray area indicates the period that was used to analyze US-evoked response. d, h, The magnitudes of the responses of the AUS-inhibited type neurons (d) and AUS-excited type neurons (h) to reward (red) and airpuff (blue). Conventions are the same as Fig. 2d and h.

We then re-classified the 103 neurons into three groups based on the response to free airpuff (Supplementary table 1). Neurons showing a significant inhibition and excitation were classified as airpuff US inhibited type (AUS-inhibited type, n=47) and airpuff US excited type (AUS-excited type, n=11), respectively (P < 0.05, Wilcoxon signed-rank test). Neurons showing no significant response were classified as airpuff US non-responsive type (AUS-nonresponsive type, n=45) (P > 0.05, Wilcoxon signed-rank test). The US-evoked responses of individual neurons are shown in Supplementary Fig. 5 and Supplementary table 2. Note that this classification was different from that based on the response to 100% airpuff CS. In the following, we will focus on the AUS-inhibited and AUS-excited type neurons (see Supplementary Fig. 6 and 7 for AUS-nonresponsive type, see also Supplementary note C and Supplementary table 4 for the electrophysiological properties of each type).

The averaged responses to the reward and airpuff are shown for the AUS-inhibited type neurons (Fig. 3b and c) and AUS-excited type neurons (Fig. 3f and g). In both types, the excitatory response to reward disappeared when the reward was completely predictable by following 100% reward CS, and decreased when the reward was partially predictable by following 50% reward CS (Fig. 3b and f). This is consistent with the reward prediction error hypothesis that the activity of dopamine neurons represents a difference between the expected and actual values of reward^{14, 15}.

The AUS-inhibited type neurons appeared to encode prediction error even for aversive outcomes, albeit partially, because it was inhibited by an unexpected aversive airpuff (free airpuff) (Fig. 3c) and this inhibitory response decreased monotonically as the airpuff was more predictable (Fig. 3d, see Supplementary note D and Supplementary Fig. 8a for individual neurons). Interestingly, the excitatory response of the AUS-excited type neurons to the airpuff also decreased as the airpuff was more predictable (Fig. 3g and h, see Supplementary note D and Supplementary Fig. 8b for individual neurons).

The prediction error hypothesis predicts that neurons should respond in opposite directions when an outcome is unexpectedly omitted, compared to when the same outcome is unexpectedly delivered^{14, 15}. We found that AUS-inhibited type neurons, but not AUS-excited type neurons, tended to show the responses to both reward omission and airpuff omission (Supplementary note E and Supplementary Fig. 9).

The current consensus that dopamine neurons carry reward-related information is thought to be applied to all dopamine neurons located in the midbrain, including both the SNc and the VTA¹. Since we have found different types of dopamine neurons with regard to their responses to aversive events, we now ask whether they were located in different regions in the midbrain. Figure 4a shows the recording sites of the 68 dopamine neurons in monkey N in relation to the response to 100% airpuff CS. Neurons showing a significant excitation (i.e., ACS-excited type neurons, red circles) tended to be located in the more dorsolateral part, and neurons showing a significant inhibition (i.e., ACS-inhibited type neurons, blue circles) tended to be located in the more ventromedial part. To statistically test this trend, we examined the relation between the recording depth and the response to 100% airpuff CS for monkey N (Fig. 4b) and monkey D (Fig. 4c). As shown by the scatter plots, a significant negative correlation was found for both monkeys (monkey N, r = −0.50, P < 0.01; monkey D, r = −0.57, P < 0.01). This negative correlation confirmed the dorsolateral-ventromedial differentiation of the airpuff-predicting CS evoked excitatory and inhibitory responses among dopamine neurons. Similar location differences were found in relation to response to airpuff itself (Supplementary note F and Supplementary Fig. 10).

Figure 4. Locations of dopamine neurons in relation to their responses to airpuff-predicting CS.

a, Recording sites of 68 dopamine neurons in monkey N are plotted on five coronal sections shown rostrocaudally from left to right (interval: 1 mm). Red circles indicate neurons showing significant excitations to 100% airpuff CS (i.e., ACS-excited type neurons). Blue circles indicate neurons showing significant inhibitions to 100% airpuff CS (i.e., ACS-inhibited type neurons). White circles, no significance (i.e., ACS-nonresponsive type neurons). Black lines indicate electrode penetration tracks which were tilted laterally by 35 degrees. b, c, Relation between recording depth and the response to 100% airpuff CS for monkey N (b) and monkey D (c). Red, blue, and white circles indicate ACS-excited, ACS-inhibited, and ACS-nonresponsive type neurons. The recording depth was measured from a reference depth set by a manipulator to advance the recording electrode.

It has generally been assumed that midbrain dopamine neurons form a unified functional group, all representing reward-related signals in a similar manner¹. Our results are roughly consistent with this idea as far as the reward-related signals are concerned. However, clear heterogeneity was revealed when we examined their responses to aversive events. We found two types of dopamine neurons, one inhibited and the other excited by airpuff or its predictor. Clearly, the unified concept of dopamine neurons needs to be changed (see Supplementary note G for the relationship between our findings and previous studies).

We propose that there are at least two functional groups of dopamine neurons. Dopamine neurons in the first group (airpuff-inhibited type, i.e., ACS- and AUS-inhibited types) would represent motivational value. Their responses covaried with prediction errors associated with both reward and airpuff, and therefore would be useful in learning to approach rewards and avoid aversive stimuli. The function of the second group (airpuff-excited type, i.e., ACS- and AUS-excited types) is not immediately clear, but we found that their response to the CS was correlated with the latency of the monkey’s orienting response (gaze shift) to the CS and that this correlation appeared only after the CS was paired with reward or airpuff (Supplementary note H and Supplementary Fig. 11). These results raise the possibility that the CS-evoked responses of the airpuff-excited dopamine neurons reflect the motivational salience of the CS. However, this interpretation may not be valid for the US or US omission responses of these neurons.

Importantly, the two types of dopamine neurons were distributed differently, the airpuff-excited type in the dorsolateral region in the SNc, and the airpuff-inhibited type in the ventromedial region in the SNc as well as the VTA (see Supplementary note I for details). In monkeys¹⁶ as well as rats¹⁷, dopamine neurons in the dorsolateral SNc project mainly to the dorsal striatum, while those in the ventromedial SNc and VTA project mainly to the ventral striatum. The airpuff-inhibited dopamine neurons in the ventromedial region in the SNc and VTA may thus transmit value-related information to the ventral striatum which is known to represent reward values^18–20. On the other hand, the airpuff-excited dopamine neurons in the dorsolateral region in the SNc would respond to motivationally salient stimuli, whether they are appetitive or aversive, and send the signal to the dorsal striatum which is related to orienting behavior^21–23. This may be part of the mechanism with which orienting behavior such as saccadic eye movement is induced by motivationally salient stimuli²⁴.

The two types of dopamine neurons may receive inputs from different sources. The airpuff-excited dopamine neurons may receive inputs from areas such as the basal forebrain in which neurons also show excitatory responses to both appetitive and aversive events^{25, 26} (see Supplementary note J for further discussion). On the other hand, the airpuff-inhibited dopamine neurons may receive inputs, at least partially, from the lateral habenula. Using the same Pavlovian procedure, we have shown that lateral habenula neurons are excited by the airpuff-predicting CS and inhibited by the reward-predicting CS, indicating that they encode motivational value similarly to the airpuff-inhibited dopamine neurons, but in the opposite manner²⁷. The value signals in the lateral habenula would then be transmitted to the dopamine neurons by inhibiting them²⁸ and this effect was stronger on dopamine neurons located in the ventromedial SNc or the VTA where the airpuff-inhibited type dominates (Supplementary note K and Supplementary Fig. 12).

We so far have classified dopamine neurons into two types. However, the real picture is more complex. First, the difference between the two types was gradual, not distinct, so that there was another group of dopamine neurons that did not belong to any of these types (i.e., airpuff-nonresponsive type). Second, the classification was different for CSs and USs (Supplementary note L, Supplementary table 1 and Supplementary Fig. 13b). More neurons were excited by the airpuff-predicting CS, whereas more neurons were inhibited by the airpuff itself. This might indicate flexible operations of the dopamine system. If a salient stimulus (i.e., CS) is presented, it would be beneficial to orient attention to the stimulus and judge if it predicts a rewarding event or an aversive event. This is the time when a majority of dopamine neurons are excited, thus promoting the orienting behavior. If an aversive event (i.e., US) occurs, it would be crucial to learn to avoid the action that led to the aversive event. This is the time when a majority of dopamine neurons are inhibited, thus promoting avoidance learning.

METHODS SUMMARY

Two adult rhesus monkeys (Macaca mulatta) were used for the experiments. All procedures for animal care and experimentation were approved by the Institute Animal Care and Use Committee and complied with the Public Health Service Policy on the humane care and use of laboratory animals. A plastic head holder and plastic recording chamber were fixed to the skull under general anesthesia and sterile surgical conditions. The recording chamber was placed over the fronto-parietal cortex, tilted laterally by 35 degrees, and was aimed at the SNc and VTA. Two search coils were surgically placed under the conjunctiva of the eyes. The head holder, the recording chamber and the eye coil connectors were all embedded in dental acrylic that covered the top of the skull and were connected to the skull by acrylic screws. We conditioned two monkeys using a Pavlovian procedure with an appetitive US (liquid reward) and an aversive US (airpuff). During the Pavlovian procedure, we recorded the activity of dopamine neurons in and around the SNc and VTA. We estimated the position of the SNc and VTA by MRI and identified dopamine neurons by their electrophysiological properties. After the end of recording sessions in one monkey, we confirmed the recording sites histologically. We analyzed anticipatory licking, anticipatory blinking and neuronal responses during the Pavlovian procedure. We focused on three kinds of neuronal responses: 1) responses elicited by CS presentation, 2) responses elicited by US delivery, and 3) responses elicited by US omission. Details of the Pavlovian procedure, identification of dopamine neurons, analysis methods, and histological procedure can be found in Full Methods.

METHODS

Pavlovian procedure

Our Pavlovian procedure consisted of two blocks of trials, an appetitive block (Fig. 1a) and an aversive block (Fig. 1b). In the appetitive block, three conditioned stimuli (CSs) (red circle, green cross and blue square for monkey N; yellow ring, cyan triangle and blue square for monkey D) were associated with a liquid reward (apple juice) as an unconditioned stimulus (US) with 100%, 50% and 0% probability, respectively. In the aversive block, three CSs (yellow ring, cyan triangle and blue square for monkey N; red circle, green cross and blue square for monkey D) were associated with an airpuff directed at the monkey’s face as an US with 100%, 50% and 0% probability, respectively. The liquid reward was delivered through a spout which was positioned in front of the monkey’s mouth. The airpuff (20 – 30 psi) was delivered through a narrow tube placed 6 – 7 cm from the face. Each trial started after the presentation of a timing cue for both blocks. The monkeys were not required to fixate the timing cue. After 1 s, the timing cue disappeared and one of the three CSs was presented pseudo-randomly. After 1.5 s, the CS disappeared and the US was delivered. In addition to the cued trials, uncued trials were included in which a reward alone (free reward) was delivered during the appetitive block and an airpuff alone (free airpuff) was delivered during the aversive block. All trials were presented with a random inter-trial-interval (ITI) that averaged 5 s (3 – 7 s) for monkey N and 4.5 s (3 – 6 s) for monkey D. One block consisted of 42 trials with fixed proportions of trial types (100%: 12 trials, 50 %: 12 trials, 0 %: 12 trials, uncued: 6 trials). For 50 % trials, the CS was followed by the US on 6 trials and was not followed by the US on the other 6 trials. The block changed without any external cue. For each neuron we collected data by repeating the appetitive and aversive blocks twice or more.

We monitored licking and blinking of the monkeys. To monitor licking, we attached a strain gage to the reward spout, and measured strains of the spout by licking. To monitor blinking, a magnetic search coil technique was used. A small Teflon-coated stainless steel wire (< 5 mm diameter, 5 or 6 turns) was taped to an eyelid. Eye closure was identified by the vertical component of the eyelid coil signal.

Identification of dopamine neurons

We searched for dopamine neurons in and around the SNc and VTA. Dopamine neurons were identified by their irregular firing, tonic baseline activity around 5 spikes/s, broad spike potential and phasic excitation to free reward.

Data analysis

We analyzed anticipatory licking, anticipatory blinking and neuronal activity during the Pavlovian procedure.

To evaluate the frequency and strength of anticipatory licking, the strain gage signal was used. We first calculated the velocity of the strain of the spout. Then we integrated the absolute velocity during CS presentation for each trial. This integrated velocity becomes larger if the monkeys more frequently and strongly lick the spout. We defined this value as the magnitude of anticipatory licking in the trial. The magnitude was normalized by the following formula, normalized magnitude = (X − Min) / (Max − Min), where X is the magnitude of anticipatory licking in the trial, Max is the maximum magnitude in the recording session, and Min is the minimum magnitude in the recording session.

To count the number of anticipatory blinks during CS presentation, the vertical component of the eyelid signal was used. We first calculated the downward velocity of eyelid movement. We set a threshold and counted how many times the velocity crossed the threshold during CS presentation for each trial. This count was defined as the number of anticipatory blinks in the trial.

In analyses of neuronal activity, responses to each CS were defined as the discharge rate during 150–325 ms after CS onset minus the background discharge rate during the 250 ms before CS onset. Response to reward was defined as the discharge rate during 200–400 ms after reward onset minus the background discharge rate during the 250 ms before reward onset. Response to airpuff was defined as the discharge rate during 50–200 ms after airpuff onset minus the background discharge rate during the 250 ms before airpuff onset. Response to reward omission was defined as the discharge rate during 200–500 ms after CS offset minus the background discharge rate during the 250 ms before CS offset. Response to airpuff omission was defined as the discharge rate during 150–350 ms after CS offset minus the background discharge rate during the 250 ms before CS offset. These time windows were determined based on the averaged activity of dopamine neurons. Specifically, we set the time windows such that they include major parts of the excitatory and inhibitory responses.

Because the 0% reward and 0% airpuff CSs were physically identical, they could only be distinguished by the block context (appetitive block or aversive block). Therefore, to analyze responses to 0% reward and 0% airpuff CSs, we excluded all 0% reward and 0% airpuff CSs that were presented before the block context could be known, that is, before the block's first presentation of 100% CS, 50% CS or free outcome.

We characterized the electrophysiological properties of recorded neurons by (1) baseline firing rate, (2) irregularity of firing pattern, (3) spike wave form. Baseline firing rate is the mean firing rate during the 250ms before the onset of the timing cue. To quantify irregularity of firing pattern, we used an irregularity metric introduced by Davies et al.²⁹ which they called “IR”. First, interspike interval (ISI) was computed for each “between-spikes.” If spike_i−1, spike_i, and spike_i+1 occurred in this order, the duration between spike_i−1 and spike_i correspond to ISI_i; the duration between spike_i and spike_i+1 correspond to ISI_i+1. Second, the difference between adjacent ISIs was computed as |log(ISI_i / ISI_i+1)|. The value was then assigned to the timing when the spikei occurred. Thus, small IR values indicate regular firing and large IR values indicate irregular firing. We then computed a median of all IR values during ITI (during the 1000ms before timing cue onset). To quantify spike wave form, we measured the spike duration of 67 dopamine neurons (whose spike wave forms were successfully recorded). The typical spike consisted of the following waves: first, sharp negative; second, sharp positive; third, slow negative; fourth, slow positive. We measured the spike duration from the peak of the first wave (sharp negative) to the peak of the third wave (slow negative).

Histology

After the end of the recording session in monkey N, we selected representative locations for electrode penetrations. When typical dopamine activity was recorded, we made electrolytic microlesions at the recording sites (12 µA and 30 s). Then, monkey N was deeply anesthetized with an overdose of pentobarbital sodium, and perfused with 10 % formaldehyde. The brain was blocked and equilibrated with 10 % sucrose. Frozen sections were cut every 50 µm in coronal plane. The sections were stained with cresyl violet.