Abstract
The transient response of dopamine neurons has been described as reward prediction error (RPE), with activation or suppression by events that are better or worse than expected, respectively. However, at least a minority of neurons are activated by aversive or high-intensity stimuli, casting doubt on the generality of RPE in describing the dopamine signal. To overcome limitations of previous studies, we studied neuronal responses to a wider variety of high-intensity and aversive stimuli, and we quantified and controlled aversiveness through a choice task in which macaques sacrificed juice to avoid aversive stimuli. Whereas most previous work has portrayed the RPE as a single impulse or “phase,” here we demonstrate its multiphasic temporal dynamics. Aversive or high-intensity stimuli evoked a triphasic sequence of activation-suppression-activation extending over a period of 40–700 ms. The initial activation at short latencies (40–120 ms) reflected sensory intensity. The influence of motivational value became dominant between 150 and 250 ms, with activation in the case of appetitive stimuli, and suppression in the case of aversive and neutral stimuli. The previously unreported late activation appeared to be a modest “rebound” after strong suppression. Similarly, strong activation by reward was often followed by suppression. We suggest that these “rebounds” may result from overcompensation by homeostatic mechanisms in some cells. Our results are consistent with a realistic RPE, which evolves over time through a dynamic balance of excitation and inhibition.
Introduction
The firing of dopamine neurons is typically modulated for ∼0.5 s after sudden stimulus onset. This entire period has often been viewed as a single “phasic” response signaling reward prediction error (RPE). According to a simple interpretation of RPE, firing rate should be suppressed by aversive stimuli and unchanged by neutral stimuli. Contrary to this notion, it has been found that dopamine neurons can be activated by sensory stimuli that lack obvious motivational value (Strecker and Jacobs, 1985; Horvitz et al., 1997; Schultz, 1998; Horvitz, 2000; Dommett et al., 2005) and that a minority of dopamine neurons can be activated by aversive stimuli (Schultz and Romo, 1987; Mirenowicz and Schultz, 1996; Guarraci and Kapp, 1999; Coizet et al., 2006; Joshua et al., 2008; Brischoux et al., 2009; Matsumoto and Hikosaka, 2009; Mileykovskiy and Morales, 2011; Wang and Tsien, 2011; Budygin et al., 2012; Cohen et al., 2012). These observations have caused doubt that RPE is an adequate description of the output of all dopamine neurons at all times (reviewed by Redgrave, 1999; Horvitz, 2000; Bromberg-Martin et al., 2010).
Most previous studies tested only a single type of aversive stimulus, and little was done to demonstrate that neuronal responses were related to aversiveness as opposed to other aspects of the stimulus, such as sensory intensity (e.g., sound intensity, visual contrast). Of those studies that provided evidence that stimuli were aversive, most relied on conditioned avoidance responses, such as eye blink (Guarraci and Kapp, 1999; Joshua et al., 2008; Matsumoto and Hikosaka, 2009). Avoidance does not ensure net aversiveness, especially if stimuli can be easily avoided at low cost (for example, blinking in response to a cool breeze on a hot day). If stimuli were aversive, it is not known how aversive they were relative to the reward values that have been carefully studied with respect to dopamine neurons. The latter issue is of particular importance if the aversive stimuli are presented in a context that includes reward stimuli because the responses of dopamine neurons are strongly dependent on reward context (Tobler et al., 2005). A context that causes the expectation of relatively large rewards may render dopamine neurons virtually unable to distinguish neutral from weakly aversive or appetitive stimuli.
To address these issues, we have quantified aversiveness in a choice task by measuring how much juice monkeys would sacrifice to avoid aversive stimuli. We then studied neural responses to stimuli having aversiveness approximately equal and opposite to juice reward. Stimuli included aversive air, saline, and bitter solution, neutral loud sound, appetitive juice, juice omission, and conditioned stimuli that predicted these. Here we focus on the temporal dynamics of neuronal responses, an issue that has received relatively little attention (but see Nomoto et al., 2010). In an accompanying article based on the same set of data, we address the issue of response diversity across dopamine neurons (Fiorillo et al., 2013). Our results are consistent with a realistic RPE in which stimulus identity is not known with certainty at short latencies.
Materials and Methods
Animals.
Two rhesus macaques (Macaca mulatta) were studied, one female (Monkey O; 10.5 kg) and one male (Monkey F; 5.5 kg). These same two monkeys contributed data on dopamine neurons to two other studies (Fiorillo, 2011; Fiorillo et al., 2013), and Monkey O contributed data to a third study (Fiorillo et al., 2008). Procedures complied with guidelines established by the National Institutes of Health and were overseen locally by the Stanford University Animal Care and Use Committee. Liquid intake was restricted to ensure the motivation of monkeys to participate in experiments.
Eye tracking and data acquisition.
A monkey's head was fixed in place in front of a computer monitor in a sound-insulated room. Eye position was monitored with an infrared eye tracking system (Eyelink II; SR Research), except for initial experiments performed in Monkey O, in which a scleral search coil was surgically implanted and eye position was monitored within a magnetic field. Expo software (written by Peter Lennie and modified by Julian Brown) was used to deliver stimuli and to collect all data.
Stimuli.
Juice, saline, and bitter solutions were delivered directly into the mouth by a computer-controlled solenoid valve (part 002-0010-900; Parker Hanafin) under the force of gravity. The solenoids were located ∼0.7 m from the monkey's head, and their opening and closing emitted a click of 72 dB as measured at the location of the monkey's head. One delivery system (reservoir, solenoid valve, polyethylene tube, and metal spout) was used to deliver juice, and a second system was used to deliver saline or bitter solutions. Thus, appetitive and aversive solutions could be delivered together or separately on each trial.
Juice was two-thirds apple juice and one-third water. In choice experiments of the sort illustrated in Figure 1, choosing the “standard” option resulted in delivery of 180 μl of juice delivered over a period of 200 ms, except in choice experiments with loud sound (see Fig. 1E), in which 130 μl was delivered >150 ms for the standard option. In all experiments performed during neuronal recordings, 130 μl was delivered over a period of 150 ms.
Figure 1.
Measuring aversiveness with a choice task. A, The choice task consisted of 4 or 5 trial types, with 20 trials of each type per session. On each trial, two visual images appeared, and fixation of gaze on an image resulted in its associated reward outcome. Choosing the “standard” option, which was available on each trial, always resulted in delivery of a large volume of juice plus an aversive stimulus. The “alternative” option resulted in juice alone, with the volume of juice depending on the image. B–E, For all stimulus types, the average percentage of choices for the alternative option increased with its associated juice volume. The solid vertical line indicates the amount of juice associated with the standard option (180 μl, except in the case of loud sound). The hyphenated vertical lines show the estimated points of indifference at which, on average, choices were evenly divided between the two options. Curves represent logistic functions of the form as follows: y = a/(b + exp(− cx)). Two concentrations of bitter solution were tested: 1 and 10 mm denatonium. Numbers of sessions are indicated in parentheses. F, Aversiveness for each type of stimulus (mean ± SD across sessions, in microliters of juice). From left to right, stimuli are air, saline, bitter, and loud sound in Monkey O and air and saline in Monkey F. Bitter corresponds to the lower concentration of denatonium (1 mm). Aversiveness was defined as the difference between the standard juice volume and the juice volume corresponding to the point of indifference (the distance between the hyphenated and solid lines in B–E). Aversiveness was calculated for each session before averaging it across sessions (as opposed to averaging the choice data across sessions and then fitting curves, as was done in B–E).
The intensities of aversive stimuli that were delivered during neuronal recordings were selected based on the choice task (see Fig. 1), as described in detail below. “Saline” was an aqueous solution of 8% NaCl delivered >60 ms (30 μl) in Monkey O, and 4% NaCl for 30 ms (10 μl) in Monkey F (seawater typically has a salinity of 3.5%). In initial choice experiments, concentrations of 0.5, 1, 4, and 8% were tested in Monkey O, but only 4% was tested in Monkey F. Bitter solution was 1 or 10 mm denatonium for 80 ms (40 μl), and was only tested in Monkey O. Before choosing these concentrations, concentrations of 0.001, 0.003, 0.1, and 0.3 mm denatonium were tested in choice experiments. Whereas 1 mm was delivered during initial recordings of neurons, 10 mm was delivered during the latter recordings (see Figs. 1 and 2).
Figure 2.
Increased aversiveness of saline and bitter solutions diminishes neuronal activation. A, Saline and bitter solutions became more aversive over the course of the day in Monkey O. Aversiveness was measured in 1–10 sessions each day in the choice task (Fig. 1). Aversiveness (mean ± SEM in microliters of juice) as a function of daily session number in chronological order, for all days on which at least three sessions were performed (8 d with saline and another 8 d with bitter solution). At the start of some days, Monkey O worked for saline in a distinct choice task, demonstrating that saline was sometimes appetitive (see Results). B, Neuronal activation (mean ± SEM) to saline in Monkey O during the early period (100–200 ms) declined as a function of the number of neurons recorded each day in chronological order (black) but did not show a clear relation to relative DV position (green), a conclusion further supported by regression analysis (see Results). DV position was measured relative to the midpoint between the most dorsal and most ventral neuron recorded during a single electrode penetration. The numbers of cells contributing to each data point are shown: in two cases, two distinct waveforms were recorded simultaneously, causing there to be more neurons recorded second and third than first. C, Responses (mean ± SEM) to saline in Monkey O near the beginning and end of the day. Among 46 neurons recorded over 10 d (3–7 neurons per day), average responses are shown for the first 2 (n = 20) and last 1 or 2 (n = 19) neurons recorded each day. Bin size, 20 ms. D, Responses to 1 and 10 mm denatonium, a bitter substance. After recording responses to 1 mm denatonium in 19 neurons over a total period of ∼2 weeks, the concentration was increased to 10 mm, and responses of 14 more neurons were recorded over several additional weeks. The higher concentration was more aversive (Fig. 1D).
Some experiments in Monkey O used only sound as a stimulus. “Morse,” a sound available in the Mac OSX 10.4 operating system, was delivered for 150 ms. It was amplified to 90 dB in experiments with “loud sound,” and to 72 dB in experiments with moderate sound (as measured at the location of the monkey's head). The walls of the experimental room were insulated to minimize sound originating outside the room.
The Pavlovian CS used in some experiments (see Fig. 6) had both visual and auditory components. The visual stimuli consisted of circles or squares of 4 degrees of visual angle in diameter or width, which differed in color and grating frequency depending on the experiment. They were always presented in the center of the monitor. In all experiments with audiovisual Pavlovian stimuli, the same sound of 72 dB was presented (“ping,” available in the Mac OSX 10.4 operating system).
Figure 6.
Neuronal responses to onset of CS predicting aversive stimuli. A, Population PSTHs (mean ± SEM) to conditioned stimuli predicting air, saline, or bitter (with baseline rates subtracted). Bin size = 50 ms. B, Scatter plot of responses of each neuron to a CS predicting air (n = 77). C, Responses to a CS predicting saline or bitter solutions (n = 75). A total of 29 neurons were excluded, as described for Figure 4B. B, C, Firing rates are from the two time periods indicated by shading in A (50–150 ms and 220–320 ms). Data points are labeled according to the results of significance tests (p < 0.05, unpaired t tests across trials) comparing firing rates in the specified periods relative to baseline rates: blue triangles, activation or suppression during the early period only; green squares, the late period only; red diamonds, both; black circles, neither. The large number of data points in the upper left quadrants indicates that activation followed by suppression was a common response pattern.
Air was delivered by a computer-controlled solenoid valve (part 009-0339-900; Parker Hanafin) connected to a pressurized gas tank. The gas tank and solenoid valve were located in an adjacent room. Air was delivered from a tube having an inner diameter and length of ∼3 mm and 3 m, respectively. Air was directed, parallel to the ground, at the left nostril from the left side, exiting the tube at a distance of ∼1 cm from the nostril. During all neuronal recordings and in Figure 1, the air pressure (measured at the gas tank) was 35 and 20 psi in Monkeys O and F, and the solenoid valve was open for 200 ms. The sound of the air exiting the tube had an intensity of 82 dB at the location of the monkey's head.
Choice tasks.
Choice tasks were used to quantify the aversiveness of stimuli, although other behaviors were observed as well (see below). On each trial, monkeys made eye movements to choose between two visual images presented on a computer monitor (see Fig. 1A). Each image spanned 4 degrees of visual angle and was centered 6 degrees to the left or right of the center of the monitor. The positions of the two images varied randomly between left and right across trials. Continuous fixation of gaze on either image for 500 ms was immediately followed by a sound (72 dB) that signaled correct performance. The sound was identical for each stimulus and reward outcome. One second after the sound, the reward outcome was delivered, consisting of juice, with or without the simultaneous delivery of an aversive stimulus. Regardless of the animal's choice, both images were extinguished at the same time (synchronous with offset of juice delivery on trials in which juice was delivered). The intertrial interval from reward outcome to presentation of images on the next trial was random between 0.5 and 1.5 s. If fixation on one of the images was not acquired within 1.0 s, both images disappeared and another intertrial interval began.
The goal was to decide upon a single intensity for each type of aversive stimulus, having a single level of aversiveness, and then to later deliver stimuli of that same intensity during recordings of dopamine neurons. Intensity corresponded to concentration, air pressure, or duration (volume), as described in detail above (see Stimuli). We identified an intensity for which the motivational value was equal and opposite to a small drop of juice (50–100 μl) by asking how much juice the monkey would sacrifice to avoid a given intensity of an aversive stimulus. In initial experiments, it was necessary to estimate the motivational values of many stimulus intensities. This was done using a choice task that consisted of just two visual conditioned stimuli. In a typical case, one conditioned stimulus (CS) was associated with 180 μl of juice plus the aversive stimulus (the intensity of which was fixed across all trials), and the other was associated with 120 μl of juice in the absence of any aversive stimulus. Approximately 100 trials were performed. The goal was to find an intensity for which choices would be split approximately evenly between the two options, which in the above example would correspond to an aversive value of −60 μl (120–180 μl) of juice. If the animal displayed a strong preference for one option or the other, a new intensity for the aversive stimulus was chosen, novel visual stimuli were introduced, and another session of choice trials was performed. At the start of experiments, intensities were chosen to be very low and were then gradually increased (typically by a factor of 10–1000 over 1–3 weeks).
Once an intensity was found that had a relatively stable value (across multiple sessions andd) between −50 and −100 μl, the choice task depicted in Figure 1 was performed to more precisely quantify the aversiveness. As in the choice task described above, there were two options (corresponding to two images) on each trial. The “standard” image was presented on every trial, and choosing it resulted in a large volume of juice (180 μl) plus an aversive stimulus (which had an intensity that was fixed within and across sessions). Fixation on the “alternative” image resulted in delivery of juice only. Four or five alternative images were presented in each session (but only one per trial), each associated with a particular volume of juice (see Fig. 1A). Each session consisted of 80 or 100 total trials, with 20 trials of each trial type (a particular alternative vs the standard). Each trial type was presented once within every sequence of 4 or 5 consecutive trials, with the precise order being randomized.
Once a particular intensity was chosen, it was important to ensure that its aversiveness was sufficiently stable over time. Therefore, the same choice task, with precisely the same visual and aversive stimuli, was performed one or more times per day over a period of 1–2 weeks before the start of neuronal recording (and for one or more days after the completion of all neuronal recordings).
It is worth noting that, whereas the aversiveness measured in the choice task was for stimuli that were well predicted by the monkeys, the neuronal responses to these stimuli that are reported here are for the less predictable delivery of stimuli. Unpredicted air puff could be more aversive than predicted air puff, despite the likelihood that there was little the monkeys could do to mitigate the aversive effects of air directed toward the nostril. Similarly, although the loud sound was found to be neutral in the choice task, it could have been aversive when it was more surprising.
In addition to the choice task, other behaviors also suggested the aversiveness of the stimuli. Anticipatory blinking was often observed after stimuli that predicted air. Whereas gaze remained fixed on visual conditioned stimuli that predicted juice, it was often averted from aversive conditioned stimuli after an orienting response. During a block of trials in which saline or bitter solutions were delivered (in the absence of juice), the solutions were seen to run out of the mouth, suggesting that the monkeys tried not to swallow them. A gaping response was often observed, and the bitter solution sometimes caused a gag response, especially at high concentration.
Judging aversiveness based on choice versus avoidance behaviors.
Avoidance behaviors, such as eye blink, have been used to measure aversiveness of air puff in previous studies (Joshua et al., 2008; Matsumoto and Hikosaka, 2008, 2009). One limitation of relying on avoidance behavior is that it does not demonstrate net aversiveness, as in the example of eye blink in response to a cool breeze on a hot day. It also does not indicate the motivational value of air puff relative to juice or other rewards, a factor of particular importance if rewards are present in the same context as air puff (resulting from adaptation of dopamine neurons to reward context) (Tobler et al., 2005). We tracked eye position and observed conditioned eye blink to conditioned stimuli predicting air puff (data not shown). As described below, we found that our air puff was much more aversive than necessary to elicit eye blink, suggesting that our air puff may have been much more aversive than that used in previous studies (Joshua et al., 2008; Matsumoto and Hikosaka, 2008, 2009).
In initial behavioral experiments, we adjusted air pressure, duration, and body location (Fiorillo et al., 2013). Our intention was to start with minimally aversive air puff and to increase its intensity until the monkey would sacrifice at least 50 μl of juice to avoid it. In the first choice experiments, air (10 psi for 10 ms) was directed at the left cheek (∼2 cm below the eye) from a distance of ∼2 cm (in comparison, Matsumoto and Hikosaka, 2009 delivered air 6–7 cm from the face; see their Online Methods). This was sufficient to cause eye blink, as well as conditioned anticipatory eye blink in a choice task, but the monkey was unwilling to sacrifice any juice to avoid the air. Air pressure and duration were gradually increased, and the air was directed closer to the eye: >50 psi (measured at the valve adjacent to the gas tank, ∼2.5 m from the monkey's face), the solenoid valve leaked air. A total of 50 psi for 200 ms was sufficient to push loose skin from the cheek over the eye, but still the monkey was unwilling to sacrifice any juice to avoid the air. Air was then directed at the ear, again without aversive value in the choice task. Finally, it was found that aversiveness could be measured in the choice task when air (35 psi for 200 ms) was directed at the nostril from a distance of <1 cm (parallel to the ground rather than directly up the nostril). Thus, in all subsequent experiments, 200 ms of air was directed at the nostril from a distance of 1 cm in both Monkeys O and F. These observations suggest that the minimal intensity of air that evokes conditioned eye blink may have a motivational value that is far below that of a very small drop of juice (∼50 μl), probably by a factor of at least 10. This may be in part because the aversiveness of air puff near the eye could be avoided substantially by eye blink, a very small movement that requires minimal effort. By contrast, our air was directed at the nose and could not be avoided. It had a value of ∼−75 μl of juice, and it seems likely that this was much more aversive than the air puff administered in previous studies.
Experimental design for neuronal recordings.
Most of the data analyzed here were from recordings made while unconditioned stimuli were delivered in the absence of any CS. A block of 30–80 trials consisted of just one stimulus (juice, air, saline, bitter solution, sound) delivered with a randomly distributed intertrial interval of 2–16 s. The delivery of just one stimulus in a block of trials would be expected to minimize “generalization” and therefore improve the ability of the neurons to discriminate stimulus identity at short latencies. However, the long intertrial intervals make the stimuli less predictable, which is likely to reduce discrimination at short latencies.
Other experiments were performed with Pavlovian conditioned stimuli. Conditioning with visual or audiovisual conditioned stimuli for several hundred trials was performed over 2–4 d before the start of neuronal recordings. The unconditioned stimulus (US) onset occurred 1.0 s after CS onset, and CS offset either coincided with the offset of the US or occurred 1.0 s after CS onset in the case that the US was omitted. Offset of the CS therefore signaled omission of the US. Intertrial intervals were 2–6 s. The conditioned stimuli were followed by unconditioned stimuli on pseudo-randomly selected trials with a probability of 0.5, although we expect that this was inconsequential with respect to the time course of the transient neuronal responses to CS onset.
Because most of the present stimuli had motivational value to the monkeys and because they were not explicitly conditioned to predict future stimuli, they meet the common definition of “unconditioned stimuli.” However, they are better understood as “conditioned” stimuli. The concept of an “unconditioned” stimulus, including its timing, is poorly defined (Wise, 2002). It is not clear which stimuli are truly unconditioned (and thus have motivational values that are insensitive to conditioning). The consequence of this issue to the present work arises from the fact that, if a truly US exists (specific activation of nociceptors being a plausible example), it could function through “hard-wired” circuitry. Although this would obviously limit the ability of the nervous system to learn, it would make it possible to always and unambiguously recognize a stimulus as appetitive or aversive, even at short latencies. In contrast, the discrimination of CS of a general nature is inherently more ambiguous and must necessarily require more time (because almost identical stimuli could have opposite values, or the same stimulus could have opposite values depending on context). It is known that dopamine neurons respond to the first predictive stimulus in a series, and not to later stimuli that are well predicted. The first stimulus in all cases studied here was presumably auditory, followed within milliseconds by tactile stimulation (except in the case of pure sound stimuli). The auditory and tactile stimuli associated with appetitive juice were nearly identical to those associated with aversive saline and bitter solutions. The motivational value of these auditory and tactile stimuli was presumably a result of conditioning, in which animals learned to associate them with the gustatory and nutritive consequences that followed (gustation is known to be a relatively slow sense). Thus, it would be misleading to characterize these stimuli as “unconditioned.” We use the term “US” to follow convention and to distinguish these stimuli from our conventional “conditioned stimuli,” which could instead be labeled as “higher-order conditioned stimuli.”
Recording and localization of dopamine neurons.
Glass-insulated tungsten electrodes (Alpha-Omega) were plated with gold and platinum. The region of dopamine cell bodies in the ventral midbrain was localized with the aid of physiologically identified landmarks. Dopamine neurons were distinguished from other neurons in the region by the characteristics of their extracellularly recorded impulses, including long, multiphasic waveforms and slow, fairly regular basal firing rates (Schultz and Romo, 1987). For the same two monkeys used in this study, details of the electrodes, physiological landmarks, and method of localizing neurons were given by Fiorillo (2011), whereas details of waveforms and estimated locations of neurons are provided by Fiorillo et al. (2013).
Chronology of experiments.
Neuronal data were collected in four sets of experiments in Monkey O, followed by one set in Monkey F. For each set, choice behavior was studied over a period of 3–5 weeks before the start of neuronal recordings, which extended over another 3–6 weeks. A total of 35–66 neurons were recorded in each set, with 1–6 neurons recorded per day and an average of 3 or 4. Multiple experiments were performed while recording each neuron, with each experiment consisting of a discrete block of trials lasting ∼5–10 min and consisting of 40–80 trials per trial type (most experiments reported here consisted of just one trial type). Most neurons were recorded for ∼1.0–1.5 h. Set 1 in Monkey O included experiments with high-intensity sound, as well as experiments with air puff that are not presented here. Set 2 included experiments with both high- and moderate-intensity sound, as well as previously reported experiments on timing (Fiorillo et al., 2008). Set 3 included experiments with saline as well as conditioned stimuli that predicted saline. Set 4 included experiments with air puff and bitter solution, and conditioned stimuli that predicted air puff or bitter solution. Only one set of experiments was performed in Monkey F, and it included the same types of experiments as set 4 in Monkey O, except that saline solution was studied instead of bitter solution. In addition, in set 4 in Monkey O, and the one set in Monkey F, an experiment was performed in which a CS predicted juice with a probability of 50%. Analysis of the neuronal responses to the CS provided the basis of a previous report (Fiorillo, 2011). Here we report on the neuronal responses after omission of expected juice in this same experiment.
Data analyses.
Data were analyzed using Matlab. Firing rates were measured during the time periods indicated in Table 1. For all types of stimuli, the baseline firing rate was measured in the 1.0 s period before stimulus onset. To quantify the statistical significance of these firing rates relative to baseline firing rate in a single neuron, firing rates (spike counts) were calculated for all individual trials and compared with baseline rates with an unpaired t test; p < 0.05 was taken to be significant, without any correction for the fact that the same test was performed separately on all neurons. For comparisons across the whole population of recorded neurons, the mean firing rate across trials was calculated for each condition in each neuron, and these single neuron mean firing rates were then compared between conditions across the population of neurons using paired or unpaired t tests.
Table 1.
The time periods in which firing rates were measured in each cella
| Stimulus type | Early | Middle | Late |
|---|---|---|---|
| Juice, saline, and bitter solutions | 100–200 | 220–320 | 400–600 |
| Air puff and sound | 40–100 | 160–260 | 380–480 |
| Visual CS offset (juice omission) | 80–160 | 160–280 | 360–560 |
| Visual CS onset | 50–150 | 220–320 | None |
a Data are given as milliseconds after stimulus onset. These periods were chosen to capture the majority of each phase of response based on visual inspection of population PSTHs (illustrated in Fig. 4). The main reason for using different time periods was the earlier responses to air and loud sound relative to other types of stimuli.
Onset latencies of the early activation were identified according to criteria that made no use of either the predefined windows used to measure firing rates or significance tests. Neurons with control firing rates of <1.0 Hz were excluded from this analysis. An activation was defined as the earliest event in which firing rate in at least 3 consecutive bins of 10 ms was greater than twice baseline firing rate. The onset was then defined as the onset of the first bin in the series. To restrict our analysis to the “early” period, activations were required to have onsets within 200 ms after stimulus onset.
Regression analyses.
We performed two multiple linear regression analyses, in relation to Figures 2B and 7, using the function “regstats” in Matlab. With respect to Figure 2, we wished to determine whether chronological order or dorsal-ventral (DV) position had the greater influence in the shift from activation to suppression in the responses to saline in Monkey O. Chronological order was quantified by the rank order of neuronal recordings from 1 to 7. DV position of each neuron was determined relative to the midpoint between the most dorsal and ventral dopamine neurons recorded during a single electrode penetration (Fiorillo et al., 2013). Each factor was normalized to cover an equivalent range. All neurons (n = 46) were analyzed in a single regression analysis.
Figure 7.
The temporal influence of sensory intensity and motivational value based on regression analysis. A, Coefficients for sensory intensity (blue) and motivational value (red) as a function of time relative to stimulus onset. Black represents an offset (β0), and its deviation from 0 indicates failure of the model to fully account for mean firing rates. Linear regression was performed with responses to juice, juice omission, saline-bitter, air, loud sound (90 dB), and moderate sound (72 dB) (bin size = 20 ms; baseline firing rates subtracted). Asterisks indicate significant effects of sensory or motivational values on firing rates, and horizontal bars indicate time periods during which the regression model accounted for a significant portion of the stimulus-dependent variance in firing rates (p < 0.05, ANOVA). B, Model prediction (based on coefficients in A) of responses to stimuli. The predicted responses are similar to measured population average PSTHs (compare with Fig. 4E), with the most obvious exception being the early suppression of firing rate to juice omission in the model, which was not observed in recorded responses.
A second linear regression analysis was used to examine the time course of the influence of sensory intensity and motivational value on firing rate. This included responses to juice, saline, bitter, air, loud sound, moderate sound, and juice omission. We used the mean firing rate for each stimulus in each neuron to perform one regression analysis across the entire population, with the number of data points for each stimulus condition depending on the number of cells recorded (which varied from 14 in the case of 72 dB sound to 194 in the case of juice). Because different combinations of stimuli were tested in different cells, we did not perform individual regression analyses on responses of individual neurons.
For cell firing rates in each 20 ms bin, we fit a line according to the equation:
 |
where FR(t) is firing rate in bin t, M is motivational intensity, S is sensory intensity, and βM(t) and βS(t) are fit coefficients representing their estimated contributions to firing rate. β0 effectively represents the residual firing rate not accounted for by sensory intensity or motivational value. Because the analysis was performed after subtracting baseline firing rates, all coefficients are near 0 before stimulus onset (see Fig. 7).
To perform this analysis, it was necessary to estimate sensory and motivational intensities on a common scale. In the absence of a clear rationale for treating the two dimensions differently, we normalized values in each dimension to cover a range of 0–1. All of the intensities tested here are likely to have been within a range that can be well discriminated at behavioral and neuronal levels, and within such a range, firing rates have generally been found to increase as an approximately linear function of the logarithm of intensity in a variety of sensory systems.
In trying to quantify sensory intensity, we relied on measured sound intensities, which signaled the onset of all stimuli, except juice omission. These were 72 dB for moderate sound and for delivery of juice, saline, and bitter solutions, 82 dB for air puff, and 90 dB for loud sound (see Stimuli above for details). Whereas onsets of these stimuli were relatively unpredictable, the omission of juice was signaled by the well-predicted offset of a 1.0 s duration visual stimulus, in the absence of any sound. We therefore judged juice omission to have a sensory intensity that was substantially less than the relatively unpredicted onset of a 72 dB sound [our unpublished data, and observations of others (Bromberg-Martin et al., 2010), suggest that the effects of sensory intensity on firing rate are suppressed by predictability]. In the absence of a precise method to quantify the sensory intensity of such diverse stimuli on a common scale, we assigned equidistant values of 0, 0.33, 0.66, and 1 for juice omission, all liquid solutions and moderate sound, air puff, and loud sound, respectively. These values are almost the same as if we had considered visual CS offset (juice omission) to have an intensity equivalent to 64 dB, which would have normalized to give values of 0, 0.31, 0.69, and 1.0.
Whereas the values described above were used to generate Figure 7, we obtained qualitatively identical results for two other sets of estimated sensory intensities. Specifically, we obtained the same temporal order of dependence on sensory and motivational intensities as in Figure 7A, and the same rank order of predicted peristimulus time histograms (PSTHs) across stimuli as in Figure 7B. In one of these cases, we entirely omitted the data with visual CS offset (juice omission; because it had no sound intensity and was thus difficult to compare with other stimuli) and used equidistant values of 0, 0.5, and 1.0 for the stimuli that included sound (data not shown). This produced results extremely similar to Figure 7. In another case, we considered that the sensory intensity of visual CS offset was equivalent to just a 45 dB sound, much less than our 72 dB sound; therefore, we used normalized values of 0, 0.6, 0.8, and 1.0. Still, the results were similar to Figure 7, but the effect of sensory intensity was less, and the early activation in the predicted PSTHs was less than for the actual PSTHs. When we went even further in this direction, considering that visual CS offset had a sensory intensity of 0 on a dB scale, and then normalized all sound intensities by a range of 90 dB (to arrive at 0, 0.8, 0.9, and 1.0), the effect of sensory intensity nearly vanished, and the predicted PSTHs for all stimuli, other than juice and juice omission, were nearly identical. Thus, we obtained results similar to Figure 7 for all reasonable estimates of subjective sensory intensity that were based on raw sound intensities, and less reasonable estimates performed less well in predicting actual PSTHs.
Juice had a volume of 130 μl during neuronal recordings. We used data from the choice task to quantify motivational value relative to juice. Although there was some variation across stimuli in average aversiveness, neuronal responses displayed little variation with aversiveness (see Fig. 10). For regression analysis, we treated all aversive stimuli as having values of −100 μl of juice (see Fig. 1). We further modified these values in an attempt to account for the “rectification” of firing rates, which limits the decrease in firing rate that can be caused by aversiveness. For this, we estimated the average baseline firing rate to be 3 Hz and the peak firing rate after juice delivery to be 12 Hz (see Fig. 4A), and we therefore multiplied the appetitiveness of juice (130 μl) by a factor of 4 (12/3). We then normalized to cover a range of 1, arriving at values of −0.16, 0, and 0.84 for aversive, neutral, and appetitive stimuli, respectively.
Figure 10.
Firing rates across the entire population of dopamine neurons as a function of motivational value. Each data point represents the mean ± SEM of between 14 and 194 neurons (filled symbols, Monkey O; unfilled symbols, Monkey F). A, Firing rates were measured between 150 and 250 ms after stimulus onset, when discrimination of motivational value was most robust (Fig. 7). B, Firing rates were measured between 0 and 500 ms after stimulus onset. It is possible that the rates in this longer period might more closely reflect the functional dopamine concentrations acting on recipient neurons (see Results).
Figure 4.
Population average responses (PSTHs) of dopamine neurons to relatively unpredicted stimuli. All recorded neurons were included in the average, without selection for any response. The shaded regions illustrate the “early,” “middle,” and “late” periods (Table 1) in which firing rates were measured for further analyses (Fig. 5). Black, Monkey O; red, Monkey F. A, Juice (mean ± SEM). B, Saline and bitter solutions. Responses of 29 neurons from Monkey O were excluded here; these were among the first 2 neurons recorded at the start of each day with saline or a low concentration of denatonium (1 mm, bitter) that may have been appetitive (Fig. 2). C, Air. D, Sound. The same sound was delivered at 90 or 72 dB (within partially overlapping populations of neurons). E, Population PSTHs for each type of stimulus are shown together for comparison, averaging across all cells from each monkey. Baseline firing rates were subtracted, and responses were smoothed by calculating a moving average with 50 ms bins and increments of 1 ms. F, To compare activation kinetics, the same PSTHs shown in E were normalized to have a peak amplitude of 1.0. Air and loud sound had early onsets, whereas juice and saline-bitter had later onsets.
Results
Choice task to measure aversiveness
The first step in the present work was to select an appropriate intensity for aversive stimuli. One criterion was that the aversive stimuli must have low enough intensity so that the overall experience of the monkeys in the laboratory was positive, which was necessary to ensure their continued cooperation. Thus, the positive value of juice reward, summed over time, must exceed the value of aversive stimuli. A second criterion was that we wished to have the aversiveness of stimuli be comparable to the appetitiveness of small juice rewards that have been carefully studied with respect to primate dopamine neurons, to minimize the possibility that aversiveness might be too weak to modulate dopamine neurons. In studies of primate neurophysiology, typical juice volumes on rewarded trials are between 100 and 200 μl. Tobler et al. (2005) found that the firing rate of dopamine neurons is highly effective in discriminating juice volumes as small 50 μl, especially when the juice volumes are well predicted. We therefore wished to deliver aversive stimuli with values that were equal but opposite to ∼50–100 μl of juice. Delivery of larger juice volumes (130–180 μl), in other blocks of trials, ensured cooperation of the monkeys.
To quantify the value of aversive stimuli, a choice task was used to measure how much juice a monkey would sacrifice to avoid the aversive stimuli (see Materials and Methods). On each trial, monkeys made eye movements to choose between two visual images (Fig. 1A). The “standard” image was presented on every trial, and choosing it resulted in a large volume of juice (180 μl) plus an aversive stimulus (which had an intensity that was fixed within and across sessions). Fixation on the alternative image resulted in delivery of juice only. The greater the volume of juice associated with the alternative, the greater the fraction of trials on which it was chosen. The percentage of choices for the alternative options was plotted as a function of the alternative juice volume, and curves were fit (Fig. 1B–F). The difference between the standard juice volume (180 μl) and the point of indifference (at which 50% of choices were for the alternative) provided our measure of the value of the aversive stimulus.
After a period of several weeks of adjusting the intensity of an aversive stimulus and performing choice experiments, a single intensity was chosen that was measured to have an average value between −70 and −110 μl of juice (Fig. 1F). This intensity was then used in all neuronal recordings, except in the case of bitter solution, for which the concentration was increased 10-fold approximately midway through several weeks of neuronal recordings. The increase in concentration led to an increase in average aversiveness from ∼−110 μl to −200 μl (Fig. 1D).
It is noteworthy that the intensity of air that was required for the monkeys to sacrifice juice was much greater than that required to elicit conditioned blinking. Air puff near the eye, despite having at least 10-fold greater pressure than that required to elicit eye blink, was motivationally neutral in our choice task. We were not able to get Monkey O to sacrifice any juice to avoid air directed near the eye or ear, and we therefore directed air at the nose (see Materials and Methods). Thus, it seems likely that our air puff was much more aversive than that delivered in previous studies that judged aversiveness based on conditioned eye blink (Joshua et al., 2008; Matsumoto and Hikosaka, 2008; Matsumoto and Hikosaka, 2009).
Juice, saline, and bitter solutions were delivered directly into the mouth and appeared to be very similar to one another in terms of sensory modalities and intensities. In all three cases, the opening and closing of the solenoid valves caused a sound of ∼72 dB. By contrast, the air puff emitted a sound of 82 dB and could be seen to cause a strong tactile stimulation of the nose (see Materials and Methods). Past studies have shown that dopamine neurons can be activated by high-intensity stimuli (see Introduction), although it was not clear in previous studies whether such stimuli were appetitive, aversive, or neutral. We therefore also studied a loud sound of 90 dB in Monkey O. Although the loud sound was experienced as aversive by the experimenter, it was found to be neutral in the choice task (despite the fact that we used a narrow range of juice alternatives centered on the standard volume to be sensitive to weak aversiveness or appetitiveness) (Fig. 1E,F).
We were surprised to find that the concentrated saline solution (8% NaCl) could sometimes have a net appetitive value in Monkey O and that it became more aversive over the course of the day. The definitive evidence of appetitiveness came from choice experiments performed in the weeks after the end of the 3 week period during which neuronal responses to saline were recorded. The choice task illustrated in Figure 1 demonstrated that saline had the same average aversiveness after neural recordings as it had before recordings (data not shown). In a distinct choice task performed after neural recordings, Monkey O chose between two visual stimuli (each of which was novel at the start of each session): one delivering saline alone and the other delivering nothing. In 4 such sessions over 3 d, Monkey O chose saline between 31 and 65 times in each session while choosing “no outcome” only 7–13 times across these 4 sessions. These sessions were performed near the start of each of the 3 d in the laboratory. In later sessions on the same 3 d, the monkey would not work for saline while still working for juice. On 2 other days, the monkey would not work for saline even at the start of the day. Thus, saline was sometimes appetitive in Monkey O, and this was probably for its water content because water but not salt intake was restricted, and because a low concentration of bitter solution also became more aversive over the course of the day (Fig. 2A).
The choice experiments of Figure 1 were designed to carefully measure aversiveness but not appetitiveness (because there was little or no option to choose a larger volume of juice rather than a smaller volume plus the test stimulus). We observed that, in some sessions at the start of the day, Monkey O appeared nearly indifferent in choosing between juice versus juice of equal volume plus saline (or bitter). For saline and the low concentration of bitter in Monkey O, curves of the type shown in Figure 1 shifted up and to the left over the course of most days (data not shown), and the aversiveness therefore increased over the course of the day for both saline and bitter in Monkey O (Fig. 2A). This explains the flatness of the average curves in these cases (Fig. 1C,D), which was not observed for air puff in either animal, or saline in Monkey F (Fig. 1B,C).
Neuronal recordings
We recorded extracellularly from 195 neurons (134 neurons in Monkey O, 61 in Monkey F) in the ventral midbrain that were judged to be dopamine neurons based on long duration waveforms as well as low and moderately regular rates (see Materials and Methods). Other than location, these were the only criteria used in the present study to identify putative dopamine neurons. Consistent with past studies, the firing rate of 78% of neurons was significantly increased after delivery of relatively unpredicted juice (p < 0.05). However, neither responsiveness to juice nor any other stimulus was used as a criterion for selecting which cells to record. We estimated that 81% of all 195 neurons were in substantia nigra, and the rest were in ventral tegmental area and the retrorubral field. The companion paper provides details of waveforms, locations of recorded neurons, and correlations between neuronal locations and responses (Fiorillo et al., 2013). Neuronal locations had only a modest influence on responses; therefore, the analyses presented here do not distinguish responses by neuronal location.
During neuronal recordings, aversive stimuli were delivered having the same intensity, and presumably the same aversiveness (70–110 μl) as in the choice task (Fig. 1). The volume of juice delivered during neuronal recordings (130 μl) was less than in choice experiments (180 μl). Experiments with juice omission were performed with a volume of 240 μl of juice delivered with a probability of 0.5; ignoring the critical issue of subjective scale (Tobler et al., 2005), this would correspond to a negative prediction error of 120 μl. Thus, our stimuli were all of comparable absolute values, except for neutral sound. Each stimulus was presented in a relatively unpredictable manner in a block of trials in which no other stimuli were presented (with the exception of “juice omission”; see Materials and Methods).
Increasing aversiveness of saline and bitter diminishes neuronal activation
An obvious and important experiment is to systematically vary the aversiveness of a stimulus, by changing its intensity, and examine the effect this has on neuronal responses. Despite its fundamental importance, this experiment has not been done, at least in part because of the difficulty of recording responses to multiple stimuli in dopamine neurons. It would also be of particular interest to examine responses to the same stimulus under differing motivational states, as in the case that salt can be appetitive or aversive depending on the nutritional needs of the animal. We did not set out to perform either of these experiments, but we did ultimately collect data that relates to each of them.
As described above, we found that saline (and perhaps bitter solution as well) was nearly neutral or even appetitive at the start of the day in Monkey O (probably for its water content), and became strongly aversive by the end of the day (Fig. 2A). In neurons recorded near the start of the day, the average response was an activation followed by a suppression slightly below baseline firing rate (Fig. 2C, black). As the day progressed and the monkey received more saline as well as juice, the activation gradually disappeared (Fig. 2B) (linear regression, r = −0.39, p = 0.008, n = 46). In neurons recorded later in the day, there was virtually no activation but a clear suppression (Fig. 2C, red). A similar observation was made for responses to a CS predicting saline in Monkey O (data not shown). No clear effect was observed with saline in Monkey F, commensurate with the lack of a shift in aversiveness over the day (data not shown).
The apparent effect of daily chronological order of recording on early activation to saline (Fig. 2B,C) could conceivably be related instead to the position of neurons in the DV dimension. Electrodes were advanced vertically from dorsal to ventral, and usually (but not always) the first neurons recorded each day were more dorsal. We therefore performed a multiple linear regression analysis with two factors: chronological order of recording and relative DV position. Each of these was determined independently for each electrode penetration (one penetration per day, recording 3–6 neurons) and then pooled across all 46 neurons. Order of recording had a strong and significant effect (β = −3.8, p = 0.009), whereas DV position had little or no effect (β = 1.6, p = 0.51). This result is supported by Figure 2B, in which responses are plotted both as a function of neuron order (black) as well as relative DV position (green).
Satiation of thirst over the course of the day seems to be the most likely explanation for the increased aversiveness of saline and bitter. At the start of the day, the appetitive water content of saline and bitter solutions may have offset and sometimes overcome the aversive properties of the salt or bitter substance. If so, then there could also be a decrease in activation to juice over the course of the day. This was indeed observed in Monkey O (data not shown). However, we could not be confident this was explained by chronological order of recording as opposed to DV position. The latter appeared to be an important factor with respect to suppression and late activation (Fiorillo et al., 2013).
Further evidence that an increase in aversiveness abolished the initial activation came from experiments with two concentrations of bitter solution (1 and 10 mm denatonium) in Monkey O. The lower concentration was aversive on average (Fig. 1D,F), but like saline in Monkey O, it was nearly neutral or perhaps appetitive near the start of some days, and it become more aversive over the course of each day (Fig. 2A). The higher concentration had an average aversiveness approximately twice that of the lower concentration, and it appeared to be aversive under all conditions (Fig. 1D; error bars are too small to be visible for choices between juice of equal volume with or without bitter, indicating that the monkey virtually never chose the bitter solution). After recording responses to the low concentration of bitter solution in 19 neurons, the high concentration was delivered while recording an additional 14 neurons. The neuronal activation observed with the lower concentration was virtually absent with the higher concentration (p = 0.04, unpaired t test of rates in the early period, 100–200 ms), whereas the suppression of firing rate appeared more pronounced in response to the higher concentration (Fig. 2D).
These data support the view that appetitive stimuli induce a net excitation of dopamine neurons, and aversive stimuli induce a net inhibition. Here we focus on the population average response, which illustrates the typical response of most dopamine neurons. In additional analyses, we search, but do not find, evidence to support the hypothesis of Matsumoto and Hikosaka (2009) that a minority of dopamine neurons are activated by aversiveness (Fiorillo et al., 2013).
Multiphasic responses
We observed triphasic responses to aversive stimuli and neutral loud sound, consisting of a sequence of early activation, suppression, and late activation, as illustrated by a neuron responding to air (Fig. 3A). Many neurons exhibited just one or two of the three phases. Figure 3B, C shows a neuron responding to saline with suppression followed by late activation and another neuron responding to bitter with a long-lasting suppression in the absence of any activation.
Figure 3.
Three individually recorded dopamine neurons responding to aversive stimuli with three phases of response. Rasters and PSTHs are aligned to the onset of each stimulus at time 0. Trials are arranged in chronological order from bottom to top. Each PSTH has a different vertical scale. A, The response to air in this cell displayed all three response phases that we observed across the population of neurons, consisting of an early and late activation separated by suppression. B, A cell that responded to saline solution with suppression followed by late activation. C, A cell that responded to bitter solution with a long-lasting suppression in the absence of any activation.
In population average PSTHs, the response to juice consisted of what appeared to be just one “phase” of activation (Fig. 4A), as previously reported. Saline and bitter solutions caused a very small activation, followed by suppression below baseline, followed by late activation (Fig. 4B); because responses to saline and bitter were so similar, we pool them together in most subsequent analyses. Aversive air and neutral loud sound caused a suppression and late excitation that were quite similar to saline-bitter, but they also caused an initial activation that was substantially earlier and of shorter duration than activations observed with other stimuli (Fig. 4C,D). The activation by the neutral loud sound (90 dB) was virtually absent when the same sound was delivered at moderate intensity (72 dB) (Fig. 4D). All modulations in firing rate were small compared with the activation observed in response to juice (Fig. 4E), a fact that is unlikely to result from the small differences in absolute motivational value across stimuli (Tobler et al., 2005). Normalization of average PSTHs to their peaks illustrates the relative time courses (Fig. 4E). The early activation to air and loud sound was virtually back to baseline by the time of peak activation to juice and saline-bitter. All three phases of the response appeared to occur earlier for air and loud sound than for saline-bitter.
To characterize the three phases of response across all neurons, we measured firing rates in periods (“early,” “middle,” and “late”), which were chosen by visual inspection of the population PSTHs to be centered on the average peak of each phase (Fig. 4). Firing rates were measured in the same periods in each neuron, but the periods differed across stimuli, all three being earlier for air and sound than for juice and saline-bitter (Table 1; Fig. 4). The scatter plots of Figure 5 display all firing rates from all neurons.
Figure 5.
Scatter plots of firing rates during prestimulus baseline and in early, middle, and late periods in all individual neurons. “Baseline” was the 1.0 s period before stimulus onset. Early, middle, and late periods are specified previously (Table 1; Fig. 4). Each point represents a single neuron. Red corresponds to neurons in which firing rate was significantly greater than or less than baseline firing rate (p < 0.5; unpaired t tests across trials). Diagonal lines represent equality. “Zero” has been shifted slightly off the axes to aid visualization of some data points. A, Juice. B, Juice omission. C, Air. D, Saline or bitter solutions (29 neurons were excluded, as described for Fig. 4B). E, Sound. Circles and triangles represent neurons tested with 90 and 72 dB of the same sound, respectively. Red and green indicate significant deviations from baseline firing rates.
The first point to be made about the scatter plots is that there is no clear evidence of more than one cluster, suggesting an absence of statistically discrete populations of dopamine neurons (Fiorillo et al., 2013). Second, the response distribution is skewed, with most neurons showing small changes from baseline rates and a few neurons showing large changes, especially increases. Because the skew of the distribution is not extreme, population average PSTHs provide reasonably good descriptions of the population. Nonetheless, because dopamine neurons have a low basal firing rate, activations tend to be larger than suppressions, and in taking an average, a few cells (or trials) with large activations can overwhelm a majority with weak suppression.
We also recorded responses to audiovisual conditioned stimuli that predicted the unconditioned stimuli. According to theory (Montague et al., 1996), and consistent with empirical evidence, conditioned stimuli and unconditioned stimuli both modulate dopamine neurons by inducing prediction errors. We can safely assume that the responses we observed to our unconditioned stimuli were actually to auditory and perhaps tactile aspects of the stimuli that had been conditioned to predict gustatory, nutritional, and other consequences. Thus, we would not expect for responses to audiovisual conditioned stimuli to differ qualitatively, and that is what we found. The average population response to conditioned stimuli predicting air, saline, or bitter was a small activation followed by suppression (Fig. 6A). Likewise, the most typical response was activation followed by suppression (Fig. 6B,C; the large number of responses shown in the upper left quadrants). Late activations were observed in some cells but were not readily apparent in averages across the entire population (Fig. 6A).
Initial activation is related to sensory intensity, not motivational value
The initial activation in response to air was unlike responses to juice, saline, or bitter. It had a median onset latency of 55 ms (n = 28), compared with 95 ms (n = 138) for juice (p = 0.006), and the activation was of shorter duration (Fig. 4F). Very similar activations to aversive air puff have been seen previously in primate dopamine neurons (Joshua et al., 2008; Matsumoto and Hikosaka, 2009), as well as neurons of lateral habenula (Matsumoto and Hikosaka, 2008). Our air puff caused very strong tactile stimulation of the nose, as well as a sound of 82 dB (see Materials and Methods). We suspected that the early activation of dopamine neurons, starting before 100 ms, was related to sensory intensity rather than motivational value (Bromberg-Martin et al., 2010). It is well known that the initial activation of “high-level” sensory neurons to their preferred stimulus (such as a face) is strongly dependent on sensory intensity (such as contrast), and that the influence of specific, high-level features emerges later. Likewise, it has been widely observed that conditioned stimuli can evoke an initial activation in dopamine neurons that is similar regardless of the reward value associated with the CS, and that firing rates discriminate reward value at longer latencies (e.g., Fiorillo et al., 2008; Nomoto et al., 2010).
To test the hypothesis that short-latency activation is related to sensory intensity, we administered a loud sound (90 dB) in Monkey O. High-intensity stimuli have been found to activate dopamine neurons (see Introduction), but previous studies have not verified their motivational value. Our loud sound was found to be neutral in the choice task (Fig. 1E). It evoked a neuronal response that appeared virtually indistinguishable in shape and time course from the response to air, consisting of a triphasic sequence of activation-suppression-activation (Fig. 4E,F). Like the response to air, the initial activation was brief and had a short latency (median of 55 ms; n = 15). The same sound at moderate intensity (72 dB) caused virtually no early activation (Fig. 4D). Saline and bitter solutions, which had moderate sensory intensity (72 dB) but were as aversive as air puff, did not cause large or short-latency activation (Fig. 4E,F). Juice had moderate sensory intensity (72 dB) and was very similar to saline-bitter in its auditory and tactile aspects. Likewise, juice and saline-bitter caused early activations that differed in amplitude and duration but had very similar onset latencies (Fig. 4E,F). Juice omission caused virtually no early activation (Figs. 4E and 5B). It was signaled by the predictable offset of a visual CS, an event that would be expected to have lower sensory intensity than any other sensory event studied here.
To better quantify the influence of sensory and motivational intensities over time, we performed a linear regression analysis (see Materials and Methods). We distinguished four sensory intensities based on sound intensity, ranging from visual CS offset on the low end (which signaled juice omission and had no sound) to loud sound on the high end. Based on the choice task, we distinguished three motivational intensities, which we modified in an attempt to account for the expected nonlinear relationship with firing rate (which is naturally rectified; see Materials and Methods). Regression analysis suggested that both sensory and motivational intensities each had a strong influence on firing rate but that sensory intensity exerted its influence earlier, between ∼40 and 150 ms, whereas motivational value had its effect between 90 and 300 ms (Fig. 7A). The regression model predicted firing rates that were similar in amplitude and time course to the recorded rates, suggesting that our linear model with two factors accounted reasonably well for average neuronal responses (Fig. 7B; compare withFig. 4E). Because our quantitative assignments for sensory intensity were somewhat arbitrary, we repeated the regression analysis with other parameter values, but the results remained similar (see Materials and Methods). This suggests that, with our stimuli and test conditions, the firing rate of dopamine neurons did not reliably reflect motivational value until ∼150 ms after stimulus onset. However, the time period during which firing rates best reflect motivational value will depend on the type of stimuli and their predictability. The period of sensitivity to reward value is substantially later and longer-lasting for perceptually difficult discriminations (Nomoto et al., 2010). However, it could also be earlier than that observed here if discrimination is made easier by having stimuli that differ in spatial location or sensory modality, or if conditioning were performed for a longer period of time.
Suppression by aversive and neutral stimuli
Between one-third and one-half of the population of neurons displayed statistically significant suppression to each of the aversive or neutral stimuli, as well as juice omission, and a substantial majority displayed some suppression, even if it did not reach significance (Fig. 5): saline-bitter, 43% of neurons (57 of 134), median firing rate of 65% of baseline firing rate across the entire population of cells; air, 34% of neurons (30 of 87), median rate of 73%; 90 dB sound, 48% of neurons (15 of 31), median rate of 60%; 72 dB sound, 36% of neurons (5 of 14), median of 41%; juice omission, 51% of neurons (45 of 88), median rate of 61%. In those cells with statistically significant suppression, the average timing and peak amplitudes were approximately similar for the various stimuli, with peak suppressions occurring over a range of ∼200–280 ms (Fig. 8).
Figure 8.
The time course of suppression. For each type of stimulus, neurons with significant suppression during the middle period were selected (numbers of neurons in parentheses). PSTHs were made using a moving average over 50 ms, with 1 ms increments. Before averaging across cells, firing rates were divided by baseline rates. Firing rates >50% above baseline are not shown.
The most interesting aspect of these data is the suppression by neutral stimuli, which appeared to be identical to aversive stimuli. The suppression by moderate intensity sound was statistically significant across the population despite the smaller number of cells tested (p = 0.01, paired t test, n = 14). This suggests that suppression is not specific to aversive stimuli. It should be noted that, although many previous studies of dopamine neurons in primates have examined responses to neutral stimuli, most previous studies have presented the neutral stimuli in the context of other stimuli that were associated with reward; thus, the neutral stimuli caused mild initial activation through “generalization,” often followed by suppression (e.g., Mirenowicz and Schultz, 1996). In the present study, generalization was minimized (but not necessarily eliminated) by delivering only one stimulus in a block of 30–80 trials.
The suppression by loud sound, like that to air, was often seen to follow a large, short-latency activation, and the suppression could therefore represent a “rebound.” However, the initial activation was not present in all cells with suppression. Furthermore, the moderate intensity sound caused an apparently identical suppression in the absence of any activation (Figs. 4D, 5E, and 8), suggesting that suppression was not dependent on a preceding activation.
Late activation
Late activation was observed between ∼400 and 700 ms after onset of aversive stimuli and loud sound. Although this late activation was robust in some cells (Figs. 3A,B and 9B,C), its average magnitude was small (Fig. 4). Nonetheless, it was statistically significant across the population of neurons for each type of aversive stimulus and neutral loud sound in each monkey, especially in the case of air puff (p = 0.05–0.08 for saline in each monkey, and for bitter and loud sound in Monkey O, p < 0.002 for air puff in each monkey). The late activation reached statistical significance in 29% (25 of 86), 16% (21 of 134), 16% (5 of 31), and 15% (13 of 88) of neurons tested with air, saline-bitter, loud sound, and juice omission, but in only 8% (16 of 194) and 7% (1 of 14) of cells tested with juice and moderate sound, respectively. It was earlier and of greater peak amplitude in the case of air than saline-bitter or juice omission (Figs. 4E and 9B,C), which may be related to the fact that air had higher sensory intensity and therefore evoked shorter latency triphasic responses that were better time-locked and synchronized across trials and across neurons. The late activation may correspond to a “rebound” because it was usually preceded by suppression (Figs. 3A,B and 9B,C).
Figure 9.
“Rebound” responses follow large deviations in firing rates. A, Neurons were selected for significant activation to juice in the early period and then sorted into those with and without significant suppression during the middle period. The shaded region of each PSTH represents the period that was used for sorting cells; thus, the observed differences in firing rate during this period were intentionally created by the sorting process. The interesting difference in firing rates is seen in the response phase preceding the shaded region. Neurons with significant suppression to juice also have greater preceding activation. B, Neurons were selected for significant suppression to air in the middle period and then sorted into those with and without late activation. Thus, neurons with late activation have greater preceding suppression. C, Analogous to B, but showing responses to saline-bitter.
The late activation appears not to have been observed previously. There have been previous reports of “rebound” activation, but these were linked to the sudden offset of relatively long duration aversive stimuli (Brischoux et al., 2009; Mileykovskiy and Morales, 2011; Wang and Tsien, 2011). In the present study, neither the rebound activation nor any other responses appeared to be linked to stimulus offset (which was very predictable given the short and invariant duration of each stimulus). The late activation occurred earlier after air than saline or bitter, despite the fact that offset of air occurred at 200 ms after onset, whereas offset of saline was at 30 and 60 ms in Monkeys O and F, respectively, and offset of bitter in Monkey O was at 80 ms. The late activation was evidently absent or less prominent in previous studies with air puff in primates (Joshua et al., 2008; Matsumoto and Hikosaka, 2009). The apparent discrepancy might be explained if the sensory intensity or aversiveness of our air puff was much greater than that used in previous studies, which appears highly likely (see Judging aversiveness based on choice versus avoidance behaviors).
Evidence for homeostatic inhibition and excitation
Prediction error in dopamine neurons is thought to arise from a dynamic balance of excitation and inhibition in single neurons (e.g., Brown et al., 1999; Fiorillo, 2008), and there is strong evidence for this in neurons of other brain regions (e.g., Wehr and Zador, 2003; Hosoya et al., 2005; Berg et al., 2007). According to theory, predicted excitation or inhibition should be counteracted to maintain a homeostatic level of excitability. Because predictions tend to be accurate at most times, excitation and inhibition tend to rise and fall in approximate synchrony and to substantially cancel one another, with spikes occurring when excitation briefly exceeds inhibition. Whereas the onset of our sensory stimuli was relatively unpredictable, their continued presence and its associated cellular excitation or inhibition could be predicted and counteracted in principle, thereby keeping the increase or decrease in firing rate brief. One obvious means to achieve this would be negative feedback (which can be understood as a primitive form of prediction). The theoretical ideal would be for homeostatic feedback to perfectly predict and counteract excitation or inhibition so that deviations in firing rate are kept brief without any “rebound.” But in the absence of such perfection, we might expect to see small rebound responses caused by compensatory mechanisms that slightly overshoot their mark in some neurons.
Consistent with this hypothesis, past studies have shown rasters and PSTHs of responses of single dopamine neurons to reward stimuli in which a suppression below baseline firing rate can be seen after strong activation (e.g., Schultz et al., 1993). However, this suppression has generally not been quantified or discussed, and it is not observed in population average PSTHs (e.g., Fig. 4A). We observed that firing rate decreased in most neurons during the middle and late periods after juice. Of 194 neurons, 89, 46, and 37% had even the slightest increase above baseline firing rate in the early, middle, and late periods, respectively. During the middle and late periods, 23% and 20% of neurons had significant decreases in firing rate, whereas only 12 and 8% had increases (Fig. 5A, red points). However, as would be expected given the low basal firing rates of dopamine neurons, the increases in firing rate in a minority of neurons tended to be larger in amplitude than the decreases observed in the majority (Fig. 5A). Thus, the mean firing rate was slightly elevated during the middle period (111% of baseline), whereas the median was marginally suppressed (96% of baseline). In those cells that displayed significant suppression during the middle period, the average suppression after juice was similar in amplitude to the suppressions observed with neutral and aversive stimuli (Fig. 8).
Because the theory of predictive homeostasis is based on a dynamic balance of excitation and inhibition, it suggests that neurons with greater activation may show greater “rebound” suppression. This is a striking hypothesis, given the obvious alternative that neurons with strong early activation will also show stronger activation in later periods. To test this, we selected neurons with significant activation to juice during the early period, and we then sorted them into two groups: those with and without significant suppression during the middle period. As expected, if suppression reflects a homeostatic mechanism, suppression in the middle period was associated with greater activation in the early period (Fig. 9A). Therefore, greater peak activation was not associated with prolonged activation.
Just as homeostatic inhibition may follow pronounced excitation, homeostatic excitation may follow pronounced inhibition. Thus, we may be able to extend the rationale given above for the activation–suppression pattern to understand the “rebound” activation that followed suppression. We selected neurons with significant suppression to saline-bitter or air in the middle period, and we then sorted them into those with and without significant late activation. As predicted, those with significant late activation also had larger suppression (Fig. 9B,C); therefore, larger suppression was not associated with longer-lasting suppression. This supports our proposal that rebound activation may be a hallmark of a homeostatic process. As noted above, the activation itself would not be homeostatic or beneficial but could occur in some cells in which feedback was excessive and slightly “overshot the mark.” Consistent with this interpretation, significant late activation was observed in a minority of cells, whereas some other neurons displayed prolonged suppression with no late activation (Figs. 3C and 5).
How effectively does dopamine signal reward value?
As reported above, we found the firing rate of dopamine neurons robustly discriminated the reward value of our stimuli at a latency of 150–250 ms (Fig. 7). Figure 10A shows the mean ± SEM firing rates during this period as a function of motivational value, clearly illustrating the larger response of dopamine neurons to juice reward compared with aversive and neutral stimuli. However, it is important to consider how the dopamine signal might be interpreted by recipient neurons in regions, such as the striatum. Dopamine concentration should be interpreted continuously in “real time,” and it is not clear to what extent recipient neurons possess information indicating that sensory stimulation has occurred or how much time has elapsed since its onset. Furthermore, the spatial and temporal dynamics of dopamine concentration (in striatum and other target regions), together with the relatively slow nature of G-protein coupled dopamine receptors, suggest that the effective dopamine signal is substantially averaged over space (“volume transmission”) and possibly over time as well (Arbuthnott and Wickens, 2007). The multiple phases that we observed in firing rates (as well as the variability across individual neurons) could conceivably be of limited consequence for dopamine recipient neurons, which might effectively observe only the total spike count over several hundred milliseconds. As a simple and rough means to address this, we averaged firing rates of all neurons over the 500 ms period after stimulus onset. Although firing rate was greatest after juice, it was also substantially above baseline in the case of stimuli with high sensory intensity (air and loud sound) in Monkey O (Fig. 10B). In the case of air in Monkey O, the population response was only marginally less than the response to juice (2.2 ± 0.3 and 1.4 ± 0.3 Hz above baseline for juice and air, respectively; p = 0.06, paired t test across 31 neurons; 11 of 31 neurons were significantly more activated by juice, and 4 by air).
The above result (Fig. 10B) suggests a potential problem in using the dopamine signal to distinguish reward from high sensory intensity. One solution would be to have an opposing signal cancel the sensory-related effect of dopamine in target regions. This is not unlikely because some neurons are activated by stimuli associated with absence of reward or aversiveness, and they too are likely to be activated at short latencies by sensory intensity. For example, neurons in lateral habenula are activated by air puff at short latencies (Matsumoto and Hikosaka, 2008); and given the present results, this is probably related to its sensory intensity rather than aversiveness. A partial solution is also suggested by careful consideration of the activity of dopamine neurons. Previous work suggests that the system is optimized so that the activity of dopamine neurons robustly discriminates among the most likely reward values at each moment, consistent with principles of efficient (optimal) coding (Tobler et al., 2005). It can be misleading to compare responses to distinct stimuli presented in distinct contexts that naturally evoke distinct predictions (as done in Fig. 10) because both the amplitude and polarity of responses depend strongly on prediction (Tobler et al., 2005). In the present case, each stimulus was delivered repeatedly with long and variable intervals in a block of trials in which no other stimuli were presented; thus, the likely alternative at any given moment was “nothing.” Under such conditions, the system should (and apparently does) adjust its prediction and response properties to optimally discriminate the likely reward values, and the important discrimination given the present task design is whether the stimulus is better or worse than “nothing” at a given moment. Dopamine neurons would be expected to display much more robust discrimination between appetitive juice and a high-intensity sensory stimulus (neutral or aversive) were these presented in the same context, so that the animal expected one or the other to occur at a given moment.
Discussion
Response diversity has been observed both across dopamine neurons and across time. Whereas some have suggested how diversity might be explained within a unified framework of RPE (e.g., Kakade and Dayan, 2002; Fiorillo et al., 2008; Nomoto et al., 2010), others have proposed a multiplicity of dopamine signals that could serve distinct functions (e.g., Fiorillo et al., 2003; Schultz, 2007; Brischoux et al., 2009; Bromberg-Martin et al., 2010). According to these latter proposals, RPE is an appropriate description of the activity of only a subset of dopamine neurons, or only during particular periods of time. Here we suggest that RPE can account for the multiphasic temporal dynamics that we observed, and it may also account for response diversity across neurons (Fiorillo et al., 2013).
Short latency activation by sensory intensity
After sudden onset of moderately unexpected stimuli, the initial activation of dopamine neurons was primarily determined by sensory intensity, whereas motivational value exerted its influence later (150–250 ms). It has been proposed that short latency activation may serve an “alerting function,” whereas the firing rate that follows within ≤100 ms in the same dopamine neurons signals RPE and drives positive reinforcement (Bromberg-Martin et al., 2010). For the same dopamine to mean different things at different times to the same postsynaptic neurons would require additional information, not provided by dopamine, to interpret the meaning of the dopamine signal. In this case, it would be critical for postsynaptic neurons to know that a stimulus had occurred and how much time had elapsed since its occurrence. If they do have such information, the problem would remain of resolving two signals separated in time by <100 ms and carried by relatively slow G-protein-coupled dopamine receptors.
Because short latency activation is more closely related to sensory intensity than reward value, it might seem that it could not signal RPE. However, the system necessarily has little information about stimulus identity and reward value at short latencies (Mirenowicz and Schultz, 1996; Redgrave et al., 1999; Kakade and Dayan, 2002; Nomoto et al., 2010). The key to understanding neuronal responses as RPE is to try to take the perspective of neurons in “real time” (Kakade and Dayan, 2002; Fiorillo, 2008). Ideally, RPE would have almost no dependence on sensory intensity, just as it is desirable for an object-selective neuron in visual cortex to be insensitive to contrast. However, subjective evidence for reward is necessarily dependent on a sensory stimulus having some intensity, and complete independence of the two is impossible. It is well known that, even though neurons can respond in a highly selective manner to certain stimuli (e.g., faces), the initial activation after sudden stimulus onset is strongly dependent on “lower” sensory features (e.g., contrast). In inferotemporal cortex, visual object discrimination occurs at longer latency than the initial increase in firing rate, and discrimination based on reward value occurs later still (Mogami and Tanaka, 2006).
Prediction and prediction error actually help to solve the problem of discriminating a neuron's stimulus at short latencies. Prediction errors are generated in many types of neurons by a wide array of homeostatic (adaptive) mechanisms that predict and counteract synaptic drive, and this predictive homeostasis increases the information content of a neuron's output by maintaining its sensitivity to input intensity (e.g., Barlow, 1961; Hosoya et al., 2005; Tobler et al., 2005; Fiorillo, 2008). At most times, inhibition accurately predicts excitation and the two substantially cancel one another, as has been observed with intracellular recordings in vivo in a variety of brain regions (e.g., Wehr and Zador, 2003; Berg et al., 2007). Prediction errors correspond to brief mismatches in the strength of excitation and inhibition.
The ability of a neuron's output to carry information about its stimulus (e.g., reward value) improves with the accuracy of its prediction, which naturally improves over time after the sudden onset of a static stimulus. A good example is the center-surround structure of receptive fields in retinal ganglion neurons (RGNs), the mechanisms of which are well understood. RGNs use information about light intensities in the spatial and temporal surround to predict and counteract the effect of light intensity in the center (e.g., Hosoya et al., 2005). Immediately after an instantaneous change in light intensity, RGNs appear insensitive to spatial contrast, just as dopamine neurons appear insensitive to reward value. This is because the prediction provided by surround inhibition, which endows these neurons with sensitivity to spatial contrast, must cross one more synapse (en route from the photoreceptors) than excitation coming from the center of the receptive field. Prediction is necessarily inaccurate at short latencies, but at all times, the output of RGNs is aptly described as an error in their prediction of light intensity (which at most times corresponds to spatial contrast).
It is concluded that, from the perspective of dopamine neurons, their activation signifies only their subjective estimate of reward value, which is naturally crude at short latencies. Because many previous studies did not distinguish short from long latency modulations in firing rate, they may have mistakenly concluded that short-latency modulation by aversive stimuli was caused by aversiveness (e.g., Matsumoto and Hikosaka, 2008; Jhou et al., 2009; Matsumoto and Hikosaka, 2009). Further analysis of the present data failed to find evidence of any dopamine neurons activated by aversiveness (Fiorillo et al., 2013).
Two types of inhibition
We found suppression of firing rate by aversive, neutral, and appetitive stimuli. There is reason to suspect that dopamine neurons receive two fundamentally different types of inhibition, which we denote as “homeostatic” and “opponent.” All biological models of prediction error include inhibitory mechanisms that mediate the “prediction,” and these are homeostatic because they function to stabilize membrane excitability by predicting and counteracting synaptic excitation (e.g., Brown et al., 1999; Hosoya et al., 2005; Fiorillo, 2008). In models of RPE, synaptic excitation and homeostatic inhibition both contribute evidence of reward value. By contrast, opponent inhibition contributes evidence against reward value and would naturally be evoked by aversive stimuli.
The visual system may provide insight into how neurons could represent motivational value. ON and OFF visual neurons represent evidence favoring higher and lower light intensities, respectively. We can think of dopamine neurons as analogous to ON cells, being activated by evidence of higher than expected reward value, and suppressed by evidence against reward. Presumably there are also “OFF” cells that are activated by evidence against reward (e.g., Matsumoto and Hikosaka, 2007). In lateral geniculate nucleus, an ON thalamocortical neuron receives opponent inhibition from an OFF-type GABAergic interneuron having nearly the same spatial receptive field (reviewed by Wang et al., 2011). Thus, the ON cell adds up opposing evidence for (excitation) and against (inhibition) higher light intensities. However, the ON cell also receives homeostatic inhibition from ON-type interneurons. Whereas homeostatic inhibition mediates a prediction of high light (reward) based on the positive correlation between activity of ON-type neurons, opponent inhibition mediates a prediction of low light (reward) based on the anticorrelation between OFF and ON neurons. In the case of reward, the anticorrelation would occur because at most times a single stimulus provides either evidence for or against reward, but less commonly, two or more stimuli provide contradictory evidence.
We observed suppression to aversive and neutral stimuli, each of which had been associated with an absence of juice reward and could function as a “conditioned inhibitor” (Tobler et al., 2003) to drive opponent inhibition, possibly through activation of neurons in rostromedial tegmental nucleus (Jhou et al., 2009; Hong et al., 2011). Homeostatic inhibition likely comes from multiple sources. Ventral tegmental area interneurons are activated by reward stimuli and directly inhibit dopamine neurons (Cohen et al., 2012). This relatively tonic inhibition could contribute to suppression by juice omission if CS offset causes a reduction in excitation. We also observed evidence of phasic homeostatic inhibition after juice delivery. Among multiple likely mechanisms, an interesting possibility is synaptic inhibition via metabotropic glutamate receptors, which follows excitation by ionotropic glutamate receptors (Fiorillo and Williams, 1998). This inhibition would be expected to occur only after the strong synaptic excitation that is thought to accompany reward stimuli. In an analogous vein, strong synaptic inhibition could elicit homeostatic excitation and rebound activation, possibly mediated by “H-type” cation channels (see Fiorillo et al., 2013).
The original concept of RPE was inspired by the literature on reinforcement learning and corresponded to a single “phasic” response, with the prediction derived from a CS or action typically occurring at least a second earlier (Montague et al., 1996). A more general view of prediction error developed independently from the literature on “efficient coding” (e.g., Barlow, 1961; Hosoya et al., 2005; Tobler et al., 2005; Fiorillo, 2008). An important generalization beyond the standard view of RPE is that multiple predictions occur simultaneously over multiple timescales, even as short as 1 ms (Fiorillo, 2008). Here we demonstrated modulations in firing rate unfolding over hundreds of milliseconds that are consistent with a more general, realistic, and physiological view of prediction error.
Footnotes
This work was supported by the World Class University program through the National Research Foundation of Korea, funded by the Ministry of Education, Science and Technology Grant R32-2008-000-10218-0 to C.D.F. and a grant to William T. Newsome from the Howard Hughes Medical Institute. We thank Bill Newsome for advice and helpful discussions, Julian Brown for excellent technical assistance, and Mehwish Bhatti for assistance in constructing some figures.
The authors declare no competing financial interests.
References
- Arbuthnott GW, Wickens J. Space, time and dopamine. Trends Neurosci. 2007;30:62–69. doi: 10.1016/j.tins.2006.12.003. [DOI] [PubMed] [Google Scholar]
- Barlow HB. Possible principles underlying the transformation of sensory messages. In: Rosenblith WA, editor. Sensory communication. Cambridge, MA: MIT; 1961. pp. 217–234. [Google Scholar]
- Berg RW, Alaburda A, Hounsgaard J. Balanced inhibition and excitation drive spike activity in spinal half-centers. Science. 2007;315:390–393. doi: 10.1126/science.1134960. [DOI] [PubMed] [Google Scholar]
- Brischoux F, Chakraborty S, Brierley DI, Ungless MA. Phasic excitation of dopamine neurons in the ventral VTA by noxious stimuli. Proc Natl Acad Sci U S A. 2009;106:4894–4899. doi: 10.1073/pnas.0811507106. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Bromberg-Martin ES, Matsumoto M, Hikosaka O. Dopamine in motivational control: rewarding, aversive, and alerting. Neuron. 2010;68:815–834. doi: 10.1016/j.neuron.2010.11.022. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Brown J, Bullock D, Grossberg S. How the basal ganglia use parallel excitatory and inhibitory learning pathways to selectively respond to unexpected rewarding cues. J Neurosci. 1999;19:10502–10511. doi: 10.1523/JNEUROSCI.19-23-10502.1999. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Budygin EA, Park J, Bass CE, Grinevich VP, Bonin KD, Wightman RM. Aversive stimulus differentially triggers subsecond dopamine release in reward regions. Neuroscience. 2012;201:331–337. doi: 10.1016/j.neuroscience.2011.10.056. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Cohen JY, Haesler S, Vong L, Lowell BB, Uchida N. Neuron-type-specific signals for reward and punishment in the ventral tegmental area. Nature. 2012;482:85–88. doi: 10.1038/nature10754. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Coizet V, Dommett EJ, Redgrave P, Overton PG. Nociceptive responses of midbrain dopaminergic neurones are modulated by the superior colliculus in the rat. Neuroscience. 2006;139:1479–1493. doi: 10.1016/j.neuroscience.2006.01.030. [DOI] [PubMed] [Google Scholar]
- Dommett E, Coizet V, Blaha CD, Martindale J, Lefebvre V, Walton N, Mayhew JE, Overton PG, Redgrave P. How visual stimuli activate dopaminergic neurons at short latency. Science. 2005;307:1476–1479. doi: 10.1126/science.1107026. [DOI] [PubMed] [Google Scholar]
- Fiorillo CD. Towards a general theory of neural computation based on prediction by single neurons. PLoS ONE. 2008;3:e3298. doi: 10.1371/journal.pone.0003298. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Fiorillo CD. Transient activation of midbrain dopamine neurons by reward risk. Neuroscience. 2011;197:162–171. doi: 10.1016/j.neuroscience.2011.09.037. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Fiorillo CD, Williams JT. Glutamate mediates an inhibitory postsynaptic potential in dopamine neurons. Nature. 1998;394:78–82. doi: 10.1038/27919. [DOI] [PubMed] [Google Scholar]
- Fiorillo CD, Tobler PN, Schultz W. Discrete coding of reward probability and uncertainty by dopamine neurons. Science. 2003;299:1898–1902. doi: 10.1126/science.1077349. [DOI] [PubMed] [Google Scholar]
- Fiorillo CD, Newsome WT, Schultz W. The temporal precision of reward prediction in dopamine neurons. Nat Neurosci. 2008;11:966–973. doi: 10.1038/nn.2159. [DOI] [PubMed] [Google Scholar]
- Fiorillo CD, Yun SR, Song MR. Diversity and homogeneity in responses of midbrain dopamine neurons. J Neurosci. 2013;33:4693–4709. doi: 10.1523/JNEUROSCI.3886-12.2013. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Guarraci FA, Kapp BS. An electrophysiological characterization of ventral tegmental area dopaminegic neurons during differential pavalovian fear conditioning in the awake rabbit. Behav Brain Res. 1999;99:169–179. doi: 10.1016/s0166-4328(98)00102-8. [DOI] [PubMed] [Google Scholar]
- Hong S, Jhou TC, Smith M, Saleem KS, Hikosaka O. Negative reward signals from the lateral habenula to dopamine neurons are mediated by rostromedial tegmental nucleus in primates. J Neurosci. 2011;31:11457–11471. doi: 10.1523/JNEUROSCI.1384-11.2011. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Horvitz JC. Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events. Neuroscience. 2000;96:651–656. doi: 10.1016/s0306-4522(00)00019-1. [DOI] [PubMed] [Google Scholar]
- Horvitz JC, Stewart T, Jacobs BL. Burst activity of ventral tegmental dopamine neurons is elicited by sensory stimuli in the awake cat. Brain Res. 1997;759:251–258. doi: 10.1016/s0006-8993(97)00265-5. [DOI] [PubMed] [Google Scholar]
- Hosoya T, Baccus SA, Meister M. Dynamic predictive coding by the retina. Nature. 2005;436:71–77. doi: 10.1038/nature03689. [DOI] [PubMed] [Google Scholar]
- Jhou TC, Fields HL, Baxter MG, Saper CB, Holland PC. The rostromedial tegmental nucleus (RMTg), a GABAergic afferent to midbrain dopamine neurons, encodes aversive stimuli and inhibits motor responses. Neuron. 2009;61:786–800. doi: 10.1016/j.neuron.2009.02.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Joshua M, Adler A, Mitelman R, Vaadia E, Bergman H. Midbrain dopaminergic neurons and striatal cholinergic interneurons encode the difference between reward and aversive events at different epochs of probabilistic classical conditioning trials. J Neurosci. 2008;28:11673–11684. doi: 10.1523/JNEUROSCI.3839-08.2008. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kakade S, Dayan P. Dopamine: generalization and bonuses. Neural Netw. 2002;15:549–559. doi: 10.1016/s0893-6080(02)00048-5. [DOI] [PubMed] [Google Scholar]
- Matsumoto M, Hikosaka O. Lateral habenula as a source of negative reward signals in dopamine neurons. Nature. 2007;447:1111–1115. doi: 10.1038/nature05860. [DOI] [PubMed] [Google Scholar]
- Matsumoto M, Hikosaka O. Representation of negative motivational value in lateral habenula. Nat Neurosci. 2009;12:77–84. doi: 10.1038/nn.2233. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Matsumoto M, Hikosaka O. Two types of dopamine neuron distinctly convey positive and negative motivational signals. Nature. 2009;459:837–841. doi: 10.1038/nature08028. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Mileykovskiy B, Morales M. Duration of suppression of ventral tegmental area dopamine neurons encodes a level of conditioned fear. J Neurosci. 2011;31:7471–7476. doi: 10.1523/JNEUROSCI.5731-10.2011. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Mirenowicz J, Schultz W. Preferential activation of midbrain dopamine neurons by appetitive rather than aversive stimuli. Nature. 1996;379:449–451. doi: 10.1038/379449a0. [DOI] [PubMed] [Google Scholar]
- Mogami T, Tanaka K. Reward association affects neuronal responses to visual stimuli in macaque TE and perirhinal cortices. J Neurosci. 2006;26:6761–6770. doi: 10.1523/JNEUROSCI.4924-05.2006. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Montague PR, Dayan P, Sejnowski TJ. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J Neurosci. 1996;16:1936–1947. doi: 10.1523/JNEUROSCI.16-05-01936.1996. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Nomoto K, Schultz W, Watanabe T, Sakagami M. Temporally extended dopamine responses to perceptually demanding reward-predictive stimuli. J Neurosci. 2010;30:10692–10702. doi: 10.1523/JNEUROSCI.4828-09.2010. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Redgrave P, Prescott TJ, Gurney K. Is the short-latency dopamine response too short to signal reward error? Trends Neurosci. 1999;22:146–151. doi: 10.1016/s0166-2236(98)01373-3. [DOI] [PubMed] [Google Scholar]
- Schultz W. Predictive reward signal of dopamine neurons. J Neurophysiol. 1998;80:1–27. doi: 10.1152/jn.1998.80.1.1. [DOI] [PubMed] [Google Scholar]
- Schultz W. Multiple dopamine functions at different time courses. Annu Rev Neurosci. 2007;30:259–288. doi: 10.1146/annurev.neuro.28.061604.135722. [DOI] [PubMed] [Google Scholar]
- Schultz W, Romo R. Responses of nigrostriatal dopamine neurons to high-intensity somatosensory stimulation in the anesthetized monkey. J Neurophysiol. 1987;57:201–217. doi: 10.1152/jn.1987.57.1.201. [DOI] [PubMed] [Google Scholar]
- Schultz W, Apicella P, Ljungberg T. Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J Neurosci. 1993;13:900–913. doi: 10.1523/JNEUROSCI.13-03-00900.1993. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Strecker RE, Jacobs BL. Substantia nigra dopaminergic unit activity in behaving cats: effect of arousal on spontaneous discharge and sensory evoked activity. Brain Res. 1985;361:339–350. doi: 10.1016/0006-8993(85)91304-6. [DOI] [PubMed] [Google Scholar]
- Tobler PN, Dickinson A, Schultz W. Coding of predicted reward omission by dopamine neurons in a conditioned inhibition paradigm. J Neurosci. 2003;23:10402–10410. doi: 10.1523/JNEUROSCI.23-32-10402.2003. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Tobler PN, Fiorillo CD, Schultz W. Adaptive coding of reward value by dopamine neurons. Science. 2005;307:1642–1645. doi: 10.1126/science.1105370. [DOI] [PubMed] [Google Scholar]
- Wang DV, Tsien JZ. Convergent processing of both positive and negative motivational signals by the VTA dopamine neuronal populations. PLoS ONE. 2011;6:e17047. doi: 10.1371/journal.pone.0017047. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Wang X, Sommer FT, Hirsch JA. Inhibitory circuits for visual processing in thalamus. Curr Opin Neurobiol. 2011;21:726–733. doi: 10.1016/j.conb.2011.06.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Wehr M, Zador AM. Balanced inhibition underlies tuning and sharpens spike timing in auditory cortex. Nature. 2003;476:442–446. doi: 10.1038/nature02116. [DOI] [PubMed] [Google Scholar]
- Wise RA. Brain reward circuitry: insights from unsensed incentives. Neuron. 2002;36:229–240. doi: 10.1016/s0896-6273(02)00965-0. [DOI] [PubMed] [Google Scholar]