Phasic dopamine signals: from subjective reward value to formal economic utility

Abstract

Although rewards are physical stimuli and objects, their value for survival and reproduction is subjective. The phasic, neurophysiological and voltammetric dopamine reward prediction error response signals subjective reward value. The signal incorporates crucial reward aspects such as amount, probability, type, risk, delay and effort. Differences of dopamine release dynamics with temporal delay and effort in rodents may derive from methodological issues and require further study. Recent designs using concepts and behavioral tools from experimental economics allow to formally characterize the subjective value signal as economic utility and thus to establish a neuronal value function. With these properties, the dopamine response constitutes a utility prediction error signal.

Introduction

The function of reward is derived from the biological needs for nutritional and other substances and reproduction. Thus, rewards have specific value for individual survival and gene propagation. Although rewards have physical aspects that are detected by sensory receptors, there are no specific receptors for the typically polysensory rewards, and their value needs to be inferred from eliciting preferences in behavioral choices. Furthermore, reward value depends on the organism’s momentary requirements. Satiation induced by a meal reduces the value of foods but may render liquids such as digestive drinks more attractive. Thus, value is subjective and constructed by the brain; it cannot be estimated entirely from the physical parameters and sensory properties of the rewards. The usual way to estimate subjective value in animals involves behavioral measures, including break points in fixed ratio schedules, preferences in binary choices and psychophysical indifference points against a common reference reward (subjective equivalents). Subjective value estimated in these ways is expressed in physical measures of break points, choice frequency or reference reward amount (e.g. ml of juice or numbers of pellets). By contrast, a more general, and theoretically well defined, measure for subjective value is formal economic utility, which constitutes a mathematical characterization of reward preferences and provides an internal metric of subjective value (sometimes called util) [1]. Individuals have the best chance to survive by preferring rewards with the highest subjective value. Economic theory formalizes this idea with axioms defining the conditions for utility maximization [2].

Maximization of subjective value and utility requires decision mechanisms in which inputs from neuronal value signals compete with each other, and only the option with the highest value gets selected. Neuronal reward signals that serve as appropriate inputs to competitive decision mechanisms should process subjective value or, in their best defined form, economic utility, in a monotonic but usually nonlinear relationship to objective value.

This review describes the neuronal coding of subjective value and formal economic utility in one of the brain’s prominent reward systems, the dopamine neurons. We review both the electrophysiological responses of mid-brain dopamine neurons and the voltametrically assessed dopamine concentration changes in axonal terminal areas in nucleus accumbens. We also address recent issues concerning voltammetric changes reflecting subjective value in rats.

Different rewards

Concepts and behaviour

How can we maximize subjective value when choosing between apples and oranges? These objects contain important substances for bodily functions, like glucose and water, but their precise contents are difficult to quantify. As different rewards often have no common physical unit, one can assign a ‘common currency’ value to one particular object, called ‘numeraire’ in economic theory. Behavioral preferences serve to estimate the subjective values of all other objects relative to this common reference, which allows comparison of subjective values between rewards. The value of the numeraire is usually set to 1, and the values of all other rewards are expressed as real number multiples of this value. Choice preferences provide numeric measures of subjective value along objective physical scales but not along subjective value scales; the subjective measure they allow is a rank order of rewards relative to the numeraire (and each other). Money is the key numeraire for modern humans, and any sufficiently familiar reward can serve as numeraire for animals. Monkeys show stable, but individually different, rankings of subjective value, as estimated from direct choices and from psychophysical variations against a numeraire [3,4••].]. Thus, monkeys estimate subjective reward value from different rewards in a common currency.

Neurobiology

The phasic, neurophysiological dopamine reward prediction error response is a brief value signal that increases monotonically with increasing reward amount and probability [5–7]. It is associated with corresponding dopamine concentrations changes in rat nucleus accumbens [8,9••,10]. Aversive stimuli induce occasionally dopamine activations and often depressions [11,12]; the phasic activations consist of briefly increased impulse activity that reflects physical stimulus impact but does not vary positively with aversiveness [13••] and thus does not represent an indiscriminate response to reward and punishment; the depressions consist of briefly reduced impulse activity that codes either negative aversive value [13••] or absence of reward (negative reward prediction error) [14•]. Dopamine reward responses follow closely the rank-ordered preferences among different liquid and food rewards (Figure 1a) [4••]. The responses reflect also the sum of positive and negative reward values, as shown when rewards are delivered together with punishers (Figure 1b) [14•]. Thus, dopamine neurons integrate different positive and negative outcomes into a subjective value signal in a common currency.

Figure 1.

Subjective value coding. (a) Graded impulse responses of dopamine neurons to different reward-predicting stimuli (blackcurrant juice and mashed banana mix). Arrows indicate subjective behavioral preferences; ~ indifferent. imp/s: firing rate. From Lak et al. [4••]. (b) Incorporation of negative aversive value into the common currency dopamine signal. The impulse response to reward juice alone (black) is reduced when an aversive salt or bitter solution is delivered together with the juice. +imp/s: firing rate subtracted from baseline firing. From Fiorillo [14•]. (c) Influence of risk on dopamine impulse responses to stimuli predicting differentially two liquid rewards (blue, blackcurrant juice; green, orange juice). Top: stimuli predicting binary, equiprobable, risky gambles (double horizontal bars) and safe rewards (single bars) with identical mean amounts. Vertical bar height indicates reward amount. Bottom: corresponding neuronal responses, closely following subjective values inferred from behavioral preferences shown above. Arrows indicate response increases with risky over safe rewards in risk seeking animal. From Lak et al. [4••]. (d) Differential, risk-attitude dependent influences of risk on voltammetric dopamine responses in rat nucleus accumbens to cue lights. Equiprobable risk reduces response in risk avoiders (top) but increases response in risk seekers (bottom). Mean reward is identical for gamble and safe reward (1 pellet). From Sugam et al. [9••].

Risk

Concepts

Rewards are inherently risky. The terms of risk avoidance and risk seeking refer to individual psychological tendencies of hating or loving risk and characterize the influence of risk on subjective reward value. Risk avoiders value risky outcomes lower than safe outcomes with same objective value, whereas risk seekers do the opposite. Risk is distinct from probability, as it increases up to probabilities of p = 0.5 and then declines again. The most simple and best controlled behavioral risk test employs binary, equiprobable gambles (p = 0.5 each outcome) in which risk is defined by statistical variance (Figure 1c top). A typical gamble involves two rewards whose amounts are above and below a mean by equal amounts, and higher risk consists of larger spread around the same mean (‘mean-preserving spread’) [15]. This review considers only risk defined by variance, without skewness.

Behavior

The subjective value of risky options can be assessed psychophysically in choices between certain and risky outcomes. The amount of the certain outcome at choice indifference (‘certainty equivalent’) indicates the subjective value of the gamble. A certainty equivalent below the mean value of the gamble outcomes indicates risk avoidance, whereas the opposite indicates risk seeking. In these tests, rhesus monkeys are often risk seeking with liquid amounts below 0.4–0.6 ml and risk avoiding with larger amounts [16,17,18••]. This reward amount-dependent pattern of risk attitude resembles the peanuts effect in humans [19]. In choices between a certain reward (1 sugar pellet) and a risky outcome (0 or 2 sucrose pellets at p = 0.5 each), rats show specific, risk-avoiding or risk-seeking attitudes [9••]. Thus, risk solely involving variance affects subjective value in monkeys and rats.

Neurobiology

Phasic dopamine responses to variance risk-predicting cues in monkeys are enhanced in the low value, risk-seeking range and reduced in the high value, risk-avoiding range (Figure 1c) [4••,18••,20]. In formal economic tests for the influence of risk on subjective value [15,21], dopamine responses follow second order stochastic dominance for risk seeking [18••]. Striatal voltammetric dopamine concentration changes following risky cues are reduced in risk-avoiding rats and enhanced in risk seekers (Figure 1d) [9••]. These data suggest meaningful incorporation of risk into the dopamine signal of subjective value.

Temporal discounting

Concepts and behaviour

The subjective value of rewards decays with the delay between a stimulus or action and the reward delivery. This temporal discounting may have its biological origin in the physical decay of many nutrient rewards. Temporal discounting applies to rewards in general, even when they remain physically unchanged and do not decay. Inter-temporal choices between a variable early and a set late reward serve to psychophysically assess temporal discounting. The subjective value of the late reward is inferred from the amount of early reward at choice indifference. The decay is captured quantitatively by hyperbolic, exponential or combined discounting functions [22–24]. Monkeys show slightly better hyperbolic than exponential discounting across several seconds [25]. In direct choices between early and late rewards, rats prefer earlier over later rewards of equal amounts [26], and exhibit delay discounting behavior [10]. Thus, temporal discounting dissociates subjective from objective reward value.

Neurobiology

Monkey dopamine neurons show hyperbolically decreasing responses to reward-predicting stimuli across reward delays of several seconds, despite constant physical reward amount, thus matching closely the behavioral discounting (Figure 2a) [25,27]. Lower reward amounts are associated with steeper neuronal discounting [25]. The close similarity between behavioral and neuronal discounting suggests that dopamine responses are sensitive to the reduced subjective reward value of delayed rewards. Voltammetric dopamine responses in rat nucleus accumbens show comparable temporal discounting, despite constant amount and effort (Figure 2b) [10,26], although another study found no appreciable temporal discounting (combined with effort cost) using the same method [28,29•]. Further, optogenetic dopamine axon stimulation in nucleus accumbens enhances the choice of delayed reward [10], presumably by enhancing the subjective value of the late reward in line with the general, causal influence of dopamine stimulation on behavioral learning and approach [30,31,32•]. Thus, the dopamine signal appears to reflect and influence the subjective reward value derived from reward delay.

Figure 2.

Temporal discounting. (a) Decreasing impulse responses of dopamine neurons to stimuli differentially predicting increasing reward delays (red), corresponding to subjective value decrements assessed in intertemporal choices (blue). Y-axis shows behavioral value and neuronal responses in % of reward amount at 2 s delay (0.56 ml). From Kobayashi and Schultz [25]. (b) Lower voltammetric dopamine responses in rat nucleus accumbens to visual cues specifying longer reward delay with identical effort. From Day et al. [26].

Effort cost

Concepts and behaviour

Caloric rewards provide energy for body functions. However, reward acquisition often involves effort, which amounts to energy expenditure. The gain from reward is reduced by the loss. This notion can be extended to all rewards; effort is considered an economic cost that should be subtracted from income value (however, subtracting cost from income does not define formal economic utility as viewed by economists [2,33,34]). Monkeys show longer reaction times, more task errors and lower task engagement when performing more effortful hand or arm movements against mechanical resistance [35,36]. Rats prefer low over high ratios of lever pressing [26,29•,37•]. Thus, effort cost affects subjective reward value.

Neurobiology

Movement effort reduces phasic neurophysiological value responses of some substantia nigra dopamine neurons in monkeys [35,36]. Increasing ratios of lever pressing reduce substantially the voltammetric dopamine value responses in rat nucleus accumbens beyond the effects of associated temporal discounting (Figure 3a) [26], whereas another study finds no influence of effort on voltammetric dopamine responses (Figure 3b) [28,29•,37•]. These differences may be addressed by considering that higher effort is positively correlated with temporal discounting in these studies [26,28,29•,37•]. Temporal discounting on its own is known to reduce electrophysiological and voltammetric dopamine responses when effort stays constant [25–27] (Figure 2a,b). Thus, effort should have decreased the voltammetric dopamine responses due to its correlation with temporal discounting. Explanations for these differences might lie in the slower (lasting >8 s) and smaller (0.5–15 nM) effort-insensitive voltammetric responses in one study [28,29•,37•], as compared to the typical dopamine changes of 30–70 nM associated with neurophysiological dopamine responses in the other studies [10,26]. The effort-insensitive voltammetric dopamine changes might not involve dopamine impulse responses or derive from methodological differences.

Figure 3.

Voltammetric dopamine responses under different effort loads. (a) High effort (lever press FR16) compared to low effort (FR1) decreases dopamine response in rat nucleus accumbens (from 70 nM to 50 nM). From Day et al. [26]. (b) Behavioral preference for low effort–low reward (LL) is not associated with higher voltammetric dopamine response in 3 nM range. From Hollon et al. [29•].

Methodology of dopamine voltammetry

The reports by Gan et al. [28], Hollon et al. [29•] and Wanat et al. [37•] used background-subtracted fast-scan cyclic voltammetry (FSCV) with chronically implanted carbon-fiber microelectrodes to monitor dopamine [38]. While this approach has been characterized [38], it is controversial in that it differs from the classically used acute electrodes, which are positioned with micromanipulators at specific locations in brain [39]. Given the well-established regional heterogeneity of dopamine release in the brain [40], acute electrodes allow for the location of dopamine ‘hotspots’ that cannot be done with chronically implanted electrodes, possibly explaining why dopamine responses measured with acute electrodes are often an order of magnitude greater than those with chronic electrodes [41]. Further, after each experiment, acute electrodes can be removed from the brain for calibration and temporal characterization. By contrast, chronic electrodes, constructed with polyimide-coated fused silica [42,43], are glued into the brain, preventing optimization of recording positions for best responses, post hoc calibration and temporal response assessment following each experiment.

Chronically implanted microelectrodes are also controversial because they have been only partially successful in other applications. They have been investigated for electrical–brain interfaces and chemical monitoring. Their insertion into tissue activates the immune response [44,45], which, over a month, can lead to encapsulation in glial scars that yield distorted but still useful electrical signals [46]. However, chronically implanted chemical sensors, typified by glucose electrodes needed for diabetic patients (for review see [47]), are recommended only for relatively short durations (3–7 days). While carbon-fiber electrodes implanted for a month seem to avoid glial encapsulation [38], other problems including biofouling, passivation, and degeneration can occur that necessitate frequent calibration. Methods for in situ calibration have yet to be developed.

The time course of dopamine fluctuations in the brain is a function of release and uptake [48–50], and temporal distortion could obfuscate these processes. For example, irrespective of whether an animal prefers low effort/low reward (LL) trials or high effort/high reward (HH) trials, cue-evoked dopamine during the HH trials has a pro-longed elevation at chronic electrodes (see lower panel of Fig. 3b) [29•]. The authors claim that this reflects sustained release. Alternatively, the delayed return to baseline could indicate diminished uptake rates in HH trials because electrochemical gradients across neuronal membranes, necessary for uptake, are diminished by prolonged firing [51]. Temporal distortion makes it impossible to resolve these possibilities. Note, however, that while dopamine shows prolonged elevations when measured with acute electrodes [10], the responses are regionally specific and temporally calibrated.

Recordings from chronic sensors differ in other ways from results obtained with acute sensors. For example, dopamine responses measured with the chronic probe often dip far below baseline (Figure 3b). In some instances the dip below baseline obtained with chronic electrodes is larger than the cue-evoked increase (Figure 1e and g in Ref. [29•]). Does this mean uptake is enhanced around chronically implanted electrodes? In most cases with acutely implanted electrodes, dopamine returns near to baseline, within the standard error of measurements, after its behavioral activation [9••,10,26], although it may remain elevated when reward-related behavior is ongoing [9••,10]. Because all electrode responses exhibit some drift, interpretations of failure to return to baseline have not yet been attempted in the literature.

A failure to return to baseline could also arise from chemical interferences. Baseline dips occur when dopamine and basic pH changes overlap — a common finding during behavior [52]. Display of color plots of the voltammetric data allows the possibility of chemical interferences to be evaluated. Unfortunately, we cannot evaluate this possibility because Hollon et al. [29•] do not show the fast-scan cyclic voltammograms from which the chemical changes are extracted. Further, chronic electrodes are frequently used with principal component analysis to extract dopamine contributions, employing a ‘standard training set’. A major problem with standard training sets is that they eliminate an important method of validating the principle component analysis [53]. This approach is also problematic because cyclic voltammetric responses of each carbon fiber differs, requiring building of training sets at each measurement site [53,54].

Taken together, the reliability of recordings with chronic voltammetric electrodes needs further confirmation. Although multiple interpretations have been suggested here, in our view the correct interpretation requires additional data and validation.

Utility

Concepts

The quantification of subjective reward value relative to a numeraire employs an objective scale. The £20,000 price of car is an objective money amount, even though my preference for that car over another car reveals my subjective value. By contrast, formal economic utility advances by one crucial step in providing a mathematical function of objective value u (x). Knowing such a function allows to determine the subjective value for goods solely based on their objective amounts, without every time requiring behavioral choices. Thus, by contrast to simple subjective value, utility employs an internal, subjective scale. This property makes utility a fundamental variable for economic decision theory.

Being subjective, utility lacks a physical measure that could serve as an anchor for quantification. Therefore, in principle, utility can only be estimated by choice preferences among rewards, which involves the ranking of rewards relative to each other and results in ordinal utility. However, choices under risk allow to estimate certainty equivalents that can serve to construct numeric, cardinal utility functions with unique shapes that are meaningful up to positive affine transformation (y = a + bx) [1,2,33,34]. The construction of formal, mathematically defined utility does not require other factors such as reward type, delay or effort, although these factors affect net benefit utility according to various economic models. Thus, formal economic utility is a well-defined, highly constrained form of subjective value that is expressed in internal units (utils).

Behavior

The fractile procedure is a useful behavioral tool for estimating the utility of reward income while keeping reward type, delay and effort constant. It estimates certainty equivalents in choices between specifically set, binary, equiprobable gambles and psychophysically adjustable safe outcomes [55,56]. Monkeys show convex utility functions with small rewards (risk-seeking, below 0.4–0.6 ml of blackcurrant juice) that turn concave with larger amounts (risk-avoiding) [18••]. The risk attitudes apparent in the curvatures of the utility function correspond to out-of-sample risky choices specifically tested at low and high outcomes [18••]. Thus, formal economic utility provides an excellent means for establishing meaningful neuronal value functions, as long as stable environments prevent adaptive value rescaling.

Neurophysiology

The identification of neuronal utility coding would involve the establishment of a neuronal utility function n (x) based on a behavioral utility function u (x). Such a signal would go well beyond the neuronal coding of subjective value shown above, as it would document that neuronal processing can derive utility u (x) from objective reward measures x that are detected by neuronal sensory signals. As neuronal activity has numeric properties, it could be necessary to relate the neuronal reward responses to numeric, rather than ordinal, utility functions. Thus, as a minimal condition, numeric neuronal value functions n (x) should have similar, usually nonlinear, forms as numeric behavioral utility functions u (x), which can be estimated in choices under risk.

The positive reward prediction error response of dopamine neurons increases non-linearly with linear increases of unpredicted juice reward (Figure 4a) [18••]. The responses (black bars) follow well the curvature of the utility function (red), increasing mildly in the low and high ranges and supralinearly in between. Similar nonlinear changes are also seen with three identical, well-defined, binary gambles that allow estimation of numeric utility; the dopamine response to the larger of the two gamble outcomes (positive prediction error) is small in lower and higher ranges, where the slope of the utility function is relatively flat, and large in the center where the utility slope is steeper (Figure 4b). In following the curvature of the utility function, the dopamine responses reflect the changes in marginal utility (first derivative of utility function) that underlie nonlinear utility. However, dopamine neurons do not code marginal utility explicitly, as the responses to unpredicted reward (Figure 4a) follow the monotonically increasing utility prediction error (blue lines) but not the nonmonotonic marginal utility. Thus, the dopamine reward prediction error response represents a numeric neuronal value function n (x) that follows closely the behavioral utility function u (x) and appears to constitute a utility prediction error signal. This neuronal value function is valid in the stable environment tested in the specific experiment [18••]; it might be subject to adaptive rescaling in more variable contexts, which remains to be tested. In sum, the neuronal signal coding a well-constrained, mathematical utility function goes well beyond the coding of subjective value derived from simple behavioral preferences. With these characteristics, the dopamine utility prediction error signal constitutes a measurable, physical signal that implements the elusive utility in the brain.

Figure 4.

Utility prediction error coding by dopamine neurons. (a) Nonlinear impulse response to unpredicted juice reward generating positive reward prediction errors. Red: nonlinear utility function. Black: dopamine responses in same monkey. Blue: linear increases in unpredicted reward, generating linearly increasing positive reward prediction errors. The linearly increasing reward prediction errors are coded by dopamine neurons as nonlinearly increasing utility prediction errors, thus showing a neuronal utility signal. (b) Nonmonotonic utility prediction error signal with constant-risk gambles. The higher gamble outcomes constitute positive utility prediction errors (Δu) whose magnitudes depend on the local slope of the utility function (top). Dopamine responses follow the nonmonotonic changes of the positive utility prediction error (Δu). a and b from Stauffer et al. [18••].