Fuel not fun: reinterpreting attenuated brain responses to reward in obesity

Abstract

There is a well-established literature linking obesity to altered dopamine signaling and brain response to food-related stimuli. Neuroimaging studies frequently report enhanced response in dopaminergic regions during food anticipation and decreased responses during reward receipt. This has been interpreted as reflecting anticipatory “reward surfeit”, and consummatory “reward deficiency”. In particular, attenuated response in the dorsal striatum to primary food rewards is proposed to reflect anhedonia, which leads to overeating in an attempt to compensate for the reward deficit. In this paper, we propose an alternative view. We consider brain response to food-related stimuli in a reinforcement-learning framework, which can be employed to separate the contributions of reward sensitivity and reward-related learning that are typically entangled in the brain response to reward. Consequently, we posit that decreased striatal responses to milkshake receipt reflect reduced reward-related learning rather than reward deficiency or anhedonia because reduced reward sensitivity would translate uniformly into reduced anticipatory and consummatory responses to reward. By re-conceptualizing reward deficiency as a shift in learning about subjective value of rewards, we attempt to reconcile neuroimaging findings with the putative role of dopamine in effort, energy expenditure and exploration and suggest that attenuated brain responses to energy dense foods reflect the “fuel”, not the fun entailed by the reward.

Graphical abstract

Obesity results when energy intake chronically exceeds energy expenditure [1]. The neurotransmitter dopamine plays a role in both sides of this energy balance equation by the regulation of food intake [e.g., 2], the invigoration of behavior [3, 4], and the integration of metabolic signals into brain reward circuits [5-9]. There is evidence in humans [10-21] and in animals [22-29] that obesity is associated with alterations in dopaminergic neurotransmission. For example, attenuated blood oxygen level dependent (BOLD) response to palatable food receipt is consistently observed in the dorsal striatum in overweight and obese individuals [19, 20, 30-32]. Although BOLD is only an indirect marker of neural activation, this response is linked to dopamine signaling because it is associated with polymorphisms that affect dopamine D2 receptors [18, 19] and positron emission tomography (PET) derived measures of dopamine signaling [33]. Specifically, carriers of the A1 allele of the TaqIA A1 polymorphism show reduced D2 receptor density compared to non-carriers [34, 35] and largely drive the inverse association between response in the dorsal striatum to caloric liquids such as milkshake and body mass index (BMI) or weight gain [18-20], while D2 receptor binding potential in the striatum is also associated with BMI [10-17].

Anhedonia commonly refers to a reduced ability to experience pleasure or a diminished response to rewarding stimuli [36-38]. According to the Diagnostic and Statistical Manual of Mental Disorders (DSM), anhedonia is one of the core symptoms of major depression [39]. Although anhedonia has received considerable interest in basic neurobiological research, the understanding of the concept has been hampered by ambiguities in the definition and operationalization of anhedonia [38]. In past decades, converging evidence has established that mesolimbic dopamine is related mainly to “wanting”, but not “liking” of rewards [40-42]. However, clinical diagnosis and most anhedonia self-report questionnaires do not distinguish between reduced motivation to obtain a reward (wanting) and reduced experience of pleasure when the reward is being obtained (liking; anhedonia [38]).

Similarly, the interpretation of the blunted dorsal striatal response as evidence of anhedonia fails to consider the other functions of dopamine signaling, as well as the considerable evidence that dopamine contributes to regulating motivated behavior rather than “pleasure” per se [43]. We therefore offer a reinterpretation of dorsal striatal attenuations associated with obesity and weight gain susceptibility. Specifically, we apply a reinforcement learning framework [44] and argue that the data are better represented as differences in reward-related learning, which has been consistently linked to dopamine D2 receptor function in animals [e.g., 45] and humans [46-48] but not to anhedonia.

A reinterpretation of the attenuated dorsal striatal response

The reduction of D2 receptors in the dorsal striatum has been considered a hallmark finding in drug addiction [49]. Whereas initial evidence suggested similar effects in obese individuals [17], flanked by animal studies indicating that this deficit could be diet-induced [28], this finding has not been consistently replicated to date [10]. Consequently, Horstmann et al. [10] have proposed an alternative interpretation of the conflicting data by arguing in favor of a nonlinear relationship between BMI and dopaminergic tone. This notion echoes the non-linear relationship that has been observed between BMI and reward sensitivity [50]. It posits that dopaminergic tone is lowest for overweight and mildly obese individuals, which amplifies phasic dopamine signals (relative to the overall dopaminergic tone), thereby increasing sensitivity to reward [10].

Neuroadaptive changes have been commonly observed with the progression of addictive behavior. Whereas drug-seeking is initially guided by the prospect of positive reinforcement, the “dark side” of reinforcement, which is represented by negative reinforcement due to the reduction of withdrawal, becomes increasingly important with continued substance use [51, 52]. This is also reflected in a shift from impulsive to compulsive behavior. The addiction cycle is characterized by three stages: 1) binge/intoxication, 2) withdrawal/negative affect, and 3) preoccupation/anticipation [51]. These stages have been associated with distinct brain networks. Binge/intoxication is supported by the mesolimbic network of ventral tegmental area and ventral striatum, withdrawal/negative affect by the extended amygdala, and the preoccupation/anticipation stage by a more distributed “craving” network involving the orbitofrontal cortex, dorsal striatum, prefrontal cortex, basolateral amygdala, hippocampus, and insula as well as “cognitive control/inhibition” network involving the cingulum, the dorsolateral prefrontal, and the inferior frontal cortex [51]. Notably, food addiction appears to resemble many aspects of other substance-related addictions, which points to a perhaps more generalized mechanism [53]. Critically, during protracted withdrawal (i.e., after acute withdrawal symptoms have declined), hypofunction in dopamine pathways has been observed as indicated by decreases in D2 receptor expression and decreases in dopamine release [51]. As a result, it has been hypothesized that these neuroadaptations may contribute to anhedonia and amotivation, which are commonly reported by individuals suffering from addiction, while sensitivity to conditioned drug cues might be enhanced at the same time [51]. To summarize, escalation of substance intake driven by positive reinforcement may contribute to neuroadaptations within the striatum that, in turn, lead to changes in reward-seeking behavior, effectively reinstating addictive behavior via negative reinforcement.

In order to improve our understanding of the neurobiological substrates of anhedonia, “reward deficits” can be grouped into different facets. At least four broad categories can be distinguished that have been linked to anhedonia in the past, depending on the experimental operationalization, namely consummatory (e.g., sucrose intake), anticipatory (e.g., food cue), motivational (e.g., effort), and learning (e.g., speed of reward-related learning) deficits [54]. While consummatory deficits map most intuitively onto the definition of anhedonia as reduced experience of pleasure, it is important to note that patients with major depression do not show an attenuated preference for sweet solutions over water, which is a common animal model for studying anhedonia [54]. In the following perspective, we argue that brain response to palatable milkshake may reflect all four facets of the “reward” response depending upon the exact task design.

Although it seems intuitive that the brain response to a palatable food mainly reflects the pleasure derived from it, there is conflicting evidence that blunted dorsal striatal response to caloric beverages such as milkshake reflect experienced pleasure. First, in healthy weight individuals responses in the dorsal striatum during chocolate consumption correlate with changes in wanting and liking that occur as the chocolate is consumed to beyond satiety [55] and meal-induced dopamine release in the dorsal striatum correlates with ratings of meal pleasantness [56]. However, studies that observe an association between BMI and dorsal striatal response to milkshake fail to find associations between BOLD response and individual differences in the rated liking of the milkshake [30]. Second, in a case-control study, individuals with the A1 allele and obesity or binge eating disorder had higher reward sensitivity (as assessed by two personality questionnaires) than normal-weight controls [57]. Collectively, these studies suggest that dorsal striatum response during palatable nutrient intake reflects the effect of internal state on the reinforcing properties of food but that the blunted response in obesity is unlikely to reflect anhedonia.

Positive and Negative Outcome Learning

Recently, theorists have applied a Bayesian approach to understanding brain function. The central feature of this approach is that the mind assigns probabilities to hypotheses and updates them according to Bayesian rules of inference. Here “agents” seek to optimize the value of future behavior and minimize surprise by trying to minimize the “free energy” [58]. Analogous to the minimization of free energy, reinforcement learning algorithms (e.g., Q-learning) seek to minimize surprise by sequentially improving value predictions. The predictions can be improved by the use of reward prediction errors, which serve as “teaching signals” to optimize action. Whereas surprise is present whenever expectations are violated (i.e., surprise is an “unsigned” prediction error), the concept of positive and negative outcome learning is key to understanding reward-related learning in particular. Within a reinforcement learning framework, positive updates occur whenever a reward is obtained that exceeds the current expectation (or an expected punishment is omitted), whereas negative updates occur whenever a reward is omitted that was being expected (or an unexpected punishment occurs). The Bayesian brain perspective, which can be seen as a more general framework than reinforcement learning, has also been applied to interoceptive processing. Here the perception of a food reward is thought to be dependent on an initial expectation of the stimulus, which upon consumption may produce a prediction error to update future inferences according to Bayesian principles [59]. This concept is supported by differences in the brain response to taste based on expectations of taste identity [60].

The dorsal striatal response to food-related stimuli appears to reflect aspects of prediction and, consequently, prediction error signals. Specifically, the direction of the association between BMI and dorsal striatal response to milkshake in A1 carriers depends upon whether the milkshake is cued and predictable or uncued and unpredictable. More specifically, the initial studies reporting a decreased response to milkshake in obese A1 employed a paradigm where cues predict the arrival of a milkshake [19, 20]. More recently, the same quality and quantity of milkshake has been delivered without cueing and at random intervals. In this case increased dorsal striatal response in A1 carriers is associated with weight gain over a one-year period [61]. In contrast to deterministic designs, probabilistic cue designs promote the generation of value expectations based on the likelihood of receiving a reward within a certain time following a cue. BOLD responses in the dorsal striatum may therefore reflect distinct parameters in a reinforcement learning framework, depending upon design. These parameters distinguish between anticipatory and consummatory aspects of anhedonia on the one hand and learning on the other hand [54].

Striatal Circuits and Positive and Negative Outcome Learning

Striatal dopamine receptors can be mapped onto two distinct neural pathways, namely the D1 “go” circuit and the D2 “no-go” circuit [e.g., 47, 48]. D1 receptor function mainly predicts learning from positive outcomes, whereas D2 receptor function mainly predicts learning from negative outcomes. The hypothesized neurophysiological mechanism is that dopamine D1 receptors have a lower affinity, which makes them primarily responsive to large phasic bursts of dopamine, whereas the high-affinity D2 receptors are sensitive to the smaller dips in dopamine release that occur after unexpected reward omission [47]. Dopaminergic tone is thought to modulate the balance between reward-related vs. punishment-related learning with high dopaminergic tone favoring learning from positive outcomes [62]. These interpretations have been corroborated by a recent PET study demonstrating that a) D1 receptor availability in the dorsal striatum is positively related to positive outcome learning, b) D2 receptor availability in the dorsal striatum is related to negative outcome learning in an inverted u-shape manner, and c) that dopamine depletion improves negative outcome learning, but not positive outcome learning [46]. In particular, D2 receptor function seems to be involved in learning from trials where reward is at stake (compared to punishment involving the loss of money). Specifically, whereas enhancement of tonic dopamine by administration of L-DOPA increases the dynamic range of prediction errors (i.e., by increasing the magnitude of positive and negative updates) and improves reward-related learning, the inverse D2 receptor agonist haloperidol decreases the dynamic range of prediction error signals and attenuates reward-related learning [63]. BOLD responses to food-related cues may therefore reflect a combination of tonic and phasic responses driven by interaction between D1 and D2 receptor signaling.

There is emerging evidence that reinforcement learning is impaired in obese individuals. Preliminary evidence is pointing to a perhaps stronger impairment in negative outcome learning in obesity [cf. 10]. However, it is also possible that both forms of learning are impaired. Whereas obese participants in Coppin, Nolan-Poupart [64] showed impaired negative, but not positive outcome learning in a probabilistic learning task, the same participants showed impaired conditioning for both the “positive” and the “negative” pattern in a conditioning cue preference test. Thus, an alternative interpretation would be that obese participants have a general impairment in value-based learning and that the balance between learning from positive and negative outcomes could be shifted in addition because of differences in dopaminergic tone [62]. This would explain why obese individuals do not show improved learning for negative outcomes as normal weight individuals do, but lower acquisition of the contingencies in general. Likewise, food-reward specific impairments in reversal learning have been reported in obese women that could be explained by impairment in both positive and negative outcome learning [65].

A general insensitivity to learning cues may also account for the observation that impulsivity is inversely associated with BOLD response to milkshake in the dorsal striatum, especially in overweight and obese individuals [30]. Accordingly, impulsivity has been associated with D2 receptor polymorphisms [66] and linked with reduced D2/D3 receptor availability in the striatum [67] and the midbrain [68]. The association in the midbrain is in turn correlated with enhanced amphetamine-induced dopamine release in the striatum, suggesting impulsive responding results from diminished inhibitory autoreceptor control over striatal dopamine release [68]. Taken together, these studies suggest that the negative association between BMI and striatal response to milkshake is associated, at least in part, with D2-dependent dopamine signaling, which in turn influences impulsive responding. However, impulsivity can be defined as a tendency for premature choices without foresight and despite negative consequences [69]. According to the influential reinforcement learning algorithm called Q-learning [70], agents seek to maximize the total discounted expected reward and the degree of temporal discounting is captured by the discount rate γ. A higher discount rate reduces the subjective value of rewards that are more distant in time making them less attractive to pursue. The association between dorsal striatal response and impulsivity may therefore reflect to some degree increased temporal discounting resulting in diminished sensitivity to learn about future reward outcomes. Relatedly, high trait impulsivity has recently been linked to diminished generation of prediction error signals in the prefrontal cortex that reflect deliberative (“model-based”) control and foresight [71].

In summary, there are viable interpretations of the blunted striatal responses besides anhedonia. Below we consider the possibility that the robust inverse association between BMI and dorsal striatal responses to nutrient intake is an indication of attenuated reward-related learning signals. We then outline a model mapping this interpretation onto the observed published findings.

A reinforcement learning perspective on anhedonia

In contrast to most self-report questionnaires of anhedonia, reinforcement learning models provide a clear mathematical distinction between the primary sensitivity to reward and the ability to learn from reward feedback [37]. This distinction is of critical importance because it is mainly learning from reward feedback that is consistently linked to D2 receptor function. Although many different designs have been used to study the brain response to food rewards using fMRI (Figure 1), the reinforcement learning framework can be applied to any design. Furthermore, separate learning rates can be introduced to capture differences in learning from positive or negative outcomes or rewards versus punishment. While these additional parameters can substantially improve our understanding of the learning mechanisms involved in value-based decisions, they also come at the cost of increased complexity. Thus, the use of additional free parameters always has to be justified by the additional variance that they may capture in a given dataset. Here, we provide one example to illustrate the overall approach.

Figure 1.

Many different designs can be used to study the effects of rewards on the brain response. However, subtle differences in the design may have an impact on the interplay between brain responses at the cue and outcome event. Reinforcement learning models can be used to generate predictions given the choice of design and may help to disambiguate the acquired brain response to reward. Brown = milkshake; blue = rinse; white = tasteless.

In our hypothetical study, we define the outcomes as receipt of a bolus of caloric milkshake or a noncaloric tasteless and odorless solution. Each outcome has a specific cue associated with it which never predicts the other outcome and only predicts the associated outcome in a probabilistic fashion. Such a design was used by Burger and Stice [72] to examine cue–reward learning. They found differences in cue-reward learning slopes and reward-receipt habituation slopes across milkshake runs predict weight change [72]. This approach is complementary to what we propose, which is applied at the trial level, and provides an excellent example of how different aspects of learning can be quantified given careful design of a study. In our approach we denote the probabilistic cue design as p(milkshake|cue_milkshake) = .5 and p(milkshake|cue_tasteless) = 0. There is no uncued receipt of milkshake, p(milkshake|no cue) = 0. Following the notation of Huys, Pizzagalli [37], we denote the delivery of a milkshake reward on trial t as r_t = 1 and the omission of the reward as r_t = 0. The participant assigns this reward the value ρ and we assume that participants build an expectation of the reward value, Q_t. This expectation can be derived by learning via trial-based reward-prediction errors δ_t.

$${\delta }_t=\rho \times r_t-Q_t$$ (1)

which reflect the difference between the expected reward value (Q_t) and the received reward (r_t), adjusted by the reward sensitivity parameter ρ. Expectations can be formed by adding the error term δ_t to the previous expectation.

$$Q_{t+1}=Q_t+\epsilon \times {\delta }_t$$ (2)

The parameter ε (alternatively denoted as α; [73]) acts as learning rate and adjusts how fast expectations are formed by the use of prediction errors (also called delta rule). Within this simple reinforcement learning framework, anhedonia might be associated with two parameters, namely the reward sensitivity parameter ρ, and/or the learning rate ε. High values of ρ indicate higher sensitivity to, or greater subjective value of, the reward because the reward is multiplied with ρ [37]. Thus, the reward sensitivity parameter maps onto the anticipatory and the consummatory facets of anhedonia that are commonly employed to measure anhedonia [54]. In contrast, high values of ε increase the speed at which reward affects behavior because the prediction errors are multiplied with ε and thereby maps onto the aspect of learning [54]. Critically, the prediction-error signals δ_t have been commonly associated with phasic activity of dopamine neurons [74-76]. Hence, differences in dopaminergic neuromodulation that act via prediction-error signaling would be expected to modulate the learning rate ε [37]. It is also important to note that it is possible to include information on the cue association provided by practice or instructions into the equation 2. For our simulated example, we set the expected value of the cue to half of ρ * r_t because only 50% of the rewards are predicted by the cue. In other words, the participants in our hypothetical study know the probabilistic association of the cues when they start the task. We selected this instructed design to show that differences in the brain signal can also arise due to differences in learning when participants start with the same value expectations (Q_t).

To investigate the selective effect of anhedonia or dopamine on the two reinforcement learning parameters, we need to establish that these two aspects can actually be separated [cf. 37]. Although reward sensitivity and learning rates affect distinct elements in the equations 1 and 2, respectively, the obtained parameters will tend to be inversely coupled. This is because the reward r_t is multiplied by both ρ (to calculate the prediction error in equation 1) and ε (to update the expectation for the next trial Q_t+1 in equation 2; [73]). Nevertheless, since ρ only affects one term in equation 1 to calculate δ_t, whereas ε controls the effect of δ_t on the expectation for the next trial in equation 2, these quantities are mathematically distinguishable. Whereas reward sensitivity governs the value that the reward would approach over time, the learning rate would affect how quickly this asymptote is reached [37]. This independence has allowed researchers to disentangle subtle aspects of appetitive behavior. For example, drawing on a behavioral meta-analysis that involved data from six experiments and 313 individual participants, Huys, Pizzagalli [37] were able to address the question of the selectivity of anhedonia on motivated behavior at this quantitative level. Employing a probabilistic reward learning task, they found that anhedonic depression, a subscale of the Mood and Anxiety Symptom Questionnaire [77], was significantly and specifically correlated with the reward sensitivity parameter ρ, but not the learning rate ε [37]. Similarly, major depression was most likely associated with a change in reward sensitivity, not the learning rate, in categorical model comparisons using the so-called Bayes factor (i.e., reflecting the evidence in favor of one model/explanation over the others given the observed data). Intriguingly, administration of a low dose of the D2 receptor agonist pramipexole mainly affects performance by reducing the learning rate ε [37]. Taken together, the elegant study by Huys et al. outlines the enormous potential of model-based analyses in disentangling distinct psychological aspects of reinforcement by the use of reinforcement learning techniques and provides a quantitative framework for the conclusions derived from our literature review.

Differences in brain response to reward illustrated by simulated reinforcement learning

To create quantitative predictions based on the equations of the reinforcement learning framework outlined above, we created random task sequences of 50 trials that start with the milkshake cue. All of the sequences were fully balanced (i.e., 25 trials involved predicted administration of milkshake and 25 trials only involved the cue and the “predicted” reward was omitted; outcome null event). Trials including the administration of the tasteless solution did not have to be incorporated in the model because we can assume that a) the reward value of tasteless solution is close to zero and b) it does not interact with the reward value of milkshake. Both assumptions are standard in the analysis of fMRI data. Second, we created a parameter grid to simulate participants with all combinations of reward sensitivity and learning rates. Since both parameters are orthogonal, we can depict the effect of altering each parameter separately (see Figure 2 and 3). For each of the 16 × 19 pairs of parameters, we simulated 10 participants leading to a total of 3,040 simulated participants and 152,000 simulated trials.

Figure 2.

Simulated brain responses based on a parametric grid (see Figure 3) of the two parameters learning rate and reward sensitivity. Whereas reward sensitivity has an effect on both predicted cue-value and prediction-error signals, the learning rate mainly influences prediction errors. Since increases in reward sensitivity will lead to corresponding increases in prediction error signals, the two parameters can only be separated by looking at both cue and outcome signals in a conjoint analysis.

Figure 3.

Sampling of simulated participants with high reward sensitivity and low learning rates can reproduce the observed pattern of “reward surfeit” during anticipation (cue response) and “reward deficiency” during reward receipt.

The simulated data illustrate the main differences between the learning rate parameter and the reward sensitivity parameter in a probabilistic cue task as it would be assessed with fMRI group statistics. Whereas changes in learning rates do not affect the average cue responses (i.e., value expectations), changes in reward sensitivity lead to proportional changes in cue responses (i.e., anticipatory aspect of anhedonia). Critically, when the reward is being received, both reward sensitivity (i.e., consummatory aspect of anhedonia) and learning rates make highly similar predictions because both parameters scale positively with the modeled brain signal at the outcome stage (Figure 2). In other words, the brain response to palatable food does not only dependent on reward sensitivity (reflecting reward surfeit vs. deficit), but also on the associated update signals. Consequently, the attenuated brain response to food in the dorsal striatum in obesity could also be explained by alterations in learning or a combination of both reward sensitivity and reward-related learning.

This pattern of results demonstrates an important aspect about study design within the reinforcement learning framework. In order to differentiate between reward deficiency and “learning deficiency”, it is necessary to look at the cue and the outcome stage in combination, because neither alone might be sufficient to explain the results at each event. We illustrate this by selecting a subset of participants with high reward sensitivity, but low learning rates (Figure 3). Due to the high sensitivity of this subgroup to reward, we see high reward expectations at the cue stage (Figure 3, lower left panel). While this would also lead to stronger prediction error signals at the reward outcome stage, this effect is overshadowed by the low learning rates of this subset of participants (Figure 3, lower right panel). On the one hand, we see a reward surfeit at the cue (anticipatory) stage and on the other hand, we see a reward deficit at the outcome (consummatory) stage. This pattern strongly resembles the results obtained in the dorsal striatum in overweight/obese individuals.

Although simulations cannot provide an answer to the question of whether this pattern of differences is truly caused by higher reward sensitivity and lower learning rates, the simulations can demonstrate that such a reinforcement learning policy is able to generate exactly this diverging pattern. Thus, blunted reward outcome signals can even occur when there is heightened reward sensitivity in a probabilistic cue task such as the one employed in our representative example. This leads to the perhaps non-intuitive conclusion that reward sensitivity can be observed best in the brain response to the cue, not to the reward itself.

Endocrine modulation of dopamine signaling

Feedback signals generated by metabolic utilization of nutrients have been linked to dopaminergic signaling and thus entail the potential to modulate food reinforcement [78]. For example, food restriction increases tonic dopamine levels, which accounts for the common observation that animals are more motivated to work for food reward when hungry [79], and increases D2 receptor availability [25]. Whereas homeostatic signals such as ghrelin and leptin consistently modulate food intake, mesolimbic dopamine release is not precisely mirroring these effects [80, 81]. Nevertheless, ghrelin-treated rats work harder to obtain food [5] and leptin regulates effort-based responding for food rewards and mesolimbic dopamine via the midbrain [6]. Recent studies also show that food-evoked and food-cue evoked dopamine spikes are regulated by metabolic state with endogenous fluctuations in ghrelin as a key contributor [7, 8]. Likewise, leptin was found to attenuate the reward value of a nutrient via dopamine neurons, effectively opposing the result of food restriction [9]. Notably, leptin and D2 receptors appear to be coupled in a reciprocal manner and low availability of D2 receptors might be an indication of leptin resistance [82]. In humans, ghrelin and leptin have been found to modulate food-cue reactivity [83-86] and consummatory responses [87] in the mesocorticolimbic system.

In addition to ghrelin and leptin, glucose metabolism is an important modulator of dopamine signaling. The striatum (i.e, the NAcc) contains neurons that sense glucose [88], expresses insulin receptors [89] and insulin action in the brain provides a negative feedback signal [90]. Moreover, insulin modulates dopamine signaling effectively reducing reinforcement potency [91, 92]. In humans, reduced insulin sensitivity is associated with less endogenous dopamine at D2/3 receptors in the NAcc and acute dopamine depletion, in turn, reduces insulin sensitivity [93]. Importantly, a recent PET study has shown that insulin sensitivity is not correlated with D2 receptor binding per se using a non-replaceable tracer [94]. Insulin has been shown to reduce food-cue reactivity in the brain including the striatum [95, 96] and, accordingly, cue-induced appetite [96]. Moreover, NAcc response to a flavor predicting calories is directly proportional to increases in blood glucose when the flavor was previously consumed with calories [97] and the signal-to-noise ratio of NAcc response to milkshake receipt is related to peripheral insulin sensitivity [98].

Taken together, these findings suggest a close, but complex, interaction of metabolic state, endocrine signals, and the modulation of motivated behavior via dopaminergic projections. However, it is largely unknown to date how leptin, ghrelin, or insulin modulate reward-related learning processes in healthy and obese humans. Nevertheless, endocrine signals might be analogous to an interface, supporting a close coupling between metabolic state, availability of energy in the environment, and the action policy necessary to pursue those reinforcing options.

Where is the “fun”? Opioid receptor signaling and hedonics

Since we have argued that dopamine signaling is not directly related to anhedonia and pleasure, this raises the question how food hedonics is represented. Hedonic hotspots have been identified in the ventral pallidum and the NAcc shell [42, 99, 100]. Within these spatially restricted hotspots, stimulation of, for example, μ-opioid receptors leads to an increase in sucrose liking of 200-400% [99]. Likewise, hedonic “coldspots” have been identified adjacent to the hotspot in the NAcc shell where opioid receptor stimulation leads to a decrease in liking of sweetness. Furthermore, anatomical differences between NAcc opioid-based modulations of “liking” versus “wanting” to eat suggest that these aspects are even separable within the opioid system [99]. Intriguingly, two recent studies have shown that μ-opioid receptor binding is reduced in obese individuals [11, 101] and that weight loss induced by caloric restriction [101] or bariatric surgery [102] leads to an increase in receptor binding such that they approach the binding levels obtained in healthy weight controls.

Collectively, these results strongly suggest that μ-opioid signaling is a promising candidate for obesity-related differences in reward sensitivity and hedonics. This also calls for a better integration of future research on dopamine- or opioid-dependent mechanisms that may differentially modulate wanting or liking of rewards in order to improve our understanding of the neurobiological alterations that contribute to obesity. As reward-seeking behavior is guided by the integration of wanting and liking, computational approaches such as reinforcement learning could provide insights into their interplay in action control and reward-related learning.

Future directions

To determine if responses in the dorsal striatum to milkshake that are observed in overweight and obesity are partly explained by changes in reward sensitivity and/or learning rates, the ideal food administration task would incorporate multiple elements including cue only events, uncued food events, and cued food events. In addition, to optimize the estimation of learning rates, it is desirable to change the cue probabilities [73], as is typically done in reversal learning paradigms by swapping the associated probabilities or alternatively, by using random Gaussian walks of the associated reward probabilities. The question could also be answered retrospectively by employing meta-analytic techniques on acquired data across different designs. Ultimately, the proposed reinforcement learning framework could help to synthesize the current body of evidence such as in the study by Huys, Pizzagalli [37]. Moreover, although reward-related BOLD responses in the striatum have been shown to be associated with markers of dopamine function [33], the observed differences in the signal could also be modulated by other mechanisms unrelated to dopamine. Thus, while BOLD fMRI provides a reasonable proxy of reward-related learning signals, caution is warranted in interpreting such differences in BOLD signal with regard to differences in dopamine function. Future studies should thereby target the dopamine system more directly, for example using combined PET/fMRI measurements.

Finally, recent studies and reviews indicate that the dopamine D2 receptor is critically involved in energy allocation [3, 4, 103]. We suggest that reward-related learning might be the mechanism linking food reward expectations and consumption on the one hand and energy expenditure on the other hand. Specifically, these elements could be integrated in a complex, yet comprehensive, framework where dopamine signals the value of work [104]. More specifically, dopaminergic signals might guide learning and motivation to regulate the selection of options (e.g. foods) in our environment that are worth the effort needed to obtain them [104]. Within this theoretical framework, the D2 no-go circuit would put the D1 go circuit on an energy budget [4] and represent the opportunity cost of action [105, 106]. The average reward rate, which tracks the recent reward history and is learned via prediction error signals, has been linked to the invigoration of behavior and tonic dopamine signaling [106, 107]. While the average reward rate may, literally, set the tone, the error signal in the striatum could nevertheless be attenuated because it is less effective in driving behavioral approach by the facilitation of energy expenditure [3]. In line with the idea of an approach signal, fMRI reward-cue signals in the ventral striatum can predict the invigoration of behavior on a trial-by-trial basis indicating that the strength of the signal “fuels” approach behavior [108]. A recent study by Stopper et al. [109] has demonstrated that phasic dopamine signals provide feedback on whether recent actions were rewarded in order to direct future actions towards options that pay off. Moreover, in rodents, the dorsal striatum has been recently shown to track the nutritional value of food [110] and human studies indicate that the dorsal striatal response is sensitive to the availability of energy-dense food [111]. Future studies are needed to test the hypothesis that “fuel” and learning signals are indeed entangled.

Conclusions

Converging evidence suggests that overweight and obesity is characterized by increased anticipatory, but decreased consummatory reward responses in the striatum. The attenuated striatal response to food reward receipt has been interpreted as potential indication of reward deficit or anhedonia. In this perspective, we have argued that the blunted dorsal striatal response to caloric and palatable liquids that is consistently observed in obesity does not provide a conclusive indication of anhedonia. Instead, we demonstrate that differences in reward sensitivity translate into symmetric effects on cue-value and outcome signals using a reinforcement learning framework. Specifically, the observed pattern of brain responses can also be explained by heightened reward sensitivity coupled with decreased reward-related learning signals. Within this framework, decreased striatal responses reflect weaker learning signals rather than anhedonia. Notably, dopamine has been conclusively shown to be critically involved in encoding reward-related learning signals, whereas dopamine function is not necessary for “pleasure”. Instead, the hedonic aspect is thought to be encoded in the opioid system, which might be altered in obesity as well. How the attenuated learning signal maps onto the recently elucidated function of D2 receptors in energy allocation is unknown but highly relevant because it may be that the dorsal striatum integrates available energy and energy costs to dynamically adjust the balance between the costs and benefits of action. If so, then striatal response to energy sources may reflect the “fuel” rather than “fun”.

Highlights.

• Attenuated response to food reward in dorsal striatum in obesity

• Response to reward does reflect more than “pleasure”

• Reinforcement learning indicates importance of error signals

• Obesity may be characterized by impairment in reward learning