Reward and decision processes in the brains of humans and nonhuman primates

Abstract

Choice behavior requires weighing multiple decision variables, such as utility, uncertainty, delay, or effort, that combine to define a subjective value for each considered option or course of action. This capacity is based on prior learning about potential rewards (and punishments) that result from prior actions. When made in a social context, decisions can involve strategic thinking about the intentions of others and about the impact of others' behavior on one's own outcome. Valuation is also influenced by different emotions that serve to adaptively regulate our choices in order to, for example, stay away from excessively risky gambles, prevent future regrets, or avoid personal rejection or conflicts. Drawing on economic theory and on advances in the study of neuronal mechanisms, we review relevant recent experiments in nonhuman primates and clinical observations made in neurologically impaired patients suffering from impaired decision-making capacities.

Introduction

Neuroeconomics is a general framework for studying the neural basis of economic decision making. At first glance, one may wonder how this discipline can be relevant to the sort of issues that neurologists and psychiatrists are concerned with, beyond identifying the causes of pathological gambling or compulsive spending disorders. However, neuroeconomics is neither based on nor primarily concerned with monetary exchanges. Although money is often used as a convenient proxy of value in behavioral and neuroimaging studies, economic analysis of choice behavior can be applied in various situations where individuals need to estimate the desirability of goods and services in the broadest possible sense, for example, when making choices about food items on a restaurant menu, physical exercise, career prospects, commitment in a relationship, a child's education. The goal of economic theory is to account for choice behavior, using a restricted set of assumptions, the most fundamental one being that agents are essentially self-interested, rational maximizers, computing utility functions, weighing the expected value of an outcome by its probability or the delay in obtaining it. Behavioral economics further attempts to account for observations that seem to deviate from the predictions of standard economic theory, like the fact that individuals sometimes systematically over- or underestimate probabilities or act altruistically, by taking into account how cognitive, emotional, or social factors interact to bias our choices. Bringing concepts from economic theory into the realm of everyday behavior thus provides a formalization of what makes a prospect attractive or repulsive in terms of specific, tractable computations. These computations can then be traced to the activity of single neurons or populations of neurons in different brain regions, or revealed through specific selective deficits in pathological conditions.

Our objectives in this brief review are to introduce some of the core concepts used in behavioral economics, to make arguments based on data from animal research for the evolutionary origin of the basic neuronal mechanisms for value-based decision making, and to illustrate with clinical studies the relevance of a neuroeconomic approach in appraising different types of behavioral and cognitive disturbances.

Behavioral economics

In most natural situations, our choices are rarely totally black or white, and predicting how people will choose requires an accurate model of how value is assigned to outcomes. The simplest possible computation of subjective value is a product of “utility,” u, and its probability of occurrence, P (let us assume that the cost associated with 1-P is null). This seems like a reasonable starting point since most decisions are probabilistic: for example, expectations of a successful day at the beach hinge on rainfall statistics, and deciding to wait for a hotline operator is influenced by the probability of simultaneous calls from other customers at that particular time of the day. However, options with seemingly identical expected values may lead to different choice preferences. The prospects of winning 1€ with a fixed probability of P=1 or winning 10€ with a fixed probability of P=1/10 lead, in the long run, to similar average gains, but can generate different preferences depending on an individual's attitude toward risk. Furthermore, in real-world settings, probabilities are generally not fixed and fluctuate over time. Experimental studies have shown that expected values are computed by taking into account the mean and variance of outcome probability distributions.¹ Another key issue is time. Decisions generally produce outcomes in the future, and outcomes that occur later are devalued relative to identical, but immediate, ones. Again, experimental data show that optimal decision making in humans is based on time discounted expected values of outcomes.

More complex situations arise when multiple individuals interact. In behavioral economics, game-theory approaches are used to analyze scenarios in which an outcome is contingent upon mutual decisions made by two or more individuals. A typical example is the “prisoner's dilemma” Two players must simultaneously choose one of two possible options. Payoffs are organized such that it is most advantageous for both players to choose option A. However, if only the first player picks option A and the second player opts for B, the first player will end up with a far worse outcome than had he/she chosen the less attractive option B.This type of game is a model of actual social interactions where individuals need to anticipate each other's intentions. If the prisoner's dilemma is played only once, game theory predicts that rational self-interested players will choose option B. If repeated multiple times, players can respond to each other's actions and have opportunities to “retaliate” for or “forgive” noncooperative behavior, and this may engender different dominant strategies (eg, tit for tat, win-stay/lose-shift).²

Rationality assumptions posited by economic theory generate predictions that are often verified, but the literature is replete with paradoxes and violations to what is sometimes referred to as the “standard model.” In fact, much current work in behavioral economics is devoted to refining or proposing alternative theories in order to accommodate experimental observations. Calculating subjective values is not always a straightforward process. Probability weighting and delay discounting functions are notably nonlinear. For instance, biases—such as the tendency to overestimate small probabilities—can lead to an exaggerated aversion to certain types of risky choices. Utilities also show nonlinearities with losses looming larger than same-sized gain, a phenomenon referred to as loss aversion.³ Unexpected outcomes in social games occur because of individual differences in the “theory of mind” abilities required for strategic reasoning. However, some of the most important challenges to the standard model are those that question the rationality assumption itself and highlight the impact of emotions, such as regret, envy, altruism, and inequity aversion. For example, in another widely used test, the “ultimatum game,” a player is given a sum of money, part of which has to be transferred to a second player; that player can then accept the offer, and both players keep their respective share, or reject the offer, and both players leave empty-handed. Strict utilitarianism predicts that the first player should transfer the smallest nonzero amount possible and the second player should accept it, because something is always better than nothing. But the common finding is that the first player transfers a sizeable share of the money, possibly because of an inherent prosocial motivation for cooperation and reciprocity or perhaps more pragmatically because, anticipating the other's inequity aversion, he/she wishes to preempt rejection of an offer that would be considered unfair.⁴

Neuronal mechanisms

It can thus be seen that empirical descriptions provided by models of real-world decisions can be considered as normative behavior when evaluating pathological deviations in patient populations. These models are also useful in the quest to elucidate the underlying neurobiological mechanisms. Neuroimaging studies have contributed to the identification of the cerebral networks activated by value estimations and choice behavior in humans. Many of the behavioral observations made in humans have been verified in animals, including probability distortions, long-term reward rate maximization,⁵ and inequity aversion, thus underlining the relevance of animal research.⁶ In fact, much of the human neuroimaging work has been directly inspired by findings from neuronal recording experiments conducted in rodents and nonhuman primates (Figure 1). We briefly summarize some of the key results regarding the neuronal basis of reward processing and choice behavior, drawing on work conducted in nonhuman primates, the species whose cerebral organization is closest to ours. An extensive presentation of the flourishing literature based on human functional magnetic resonance imaging (fMRI) is beyond the scope of the present paper and interested readers can find several detailed reviews in refs 7-9.

Figure 1. Simplified schematic representation of the primate reward and decision-making network. Midbrain nuclei containing dopaminergic neurons: AMYG, amygdala; GPi, internal globus palIidus; SNc/VTA, substantia nigra pars compacta/ventral tegmental area. VL/VA, ventral lateral and anterior thalamic nuclei. Subdivisions of the prefrontal cortex: ACC, anterior cingulate cortex; dIPFC, dorsolateral prefrontal cortex; OFC, orbitofrontal cortex; vmPFC, ventromedial prefrontal cortex.

The biological cornerstone of valuation processes is the brain's reward system and one of its major components, the dopamine neuronal pathway. Midbrain dopamine neurons have been implicated in different functions but first and foremost in signaling the hedonic value of stimuli in the environment and are thus often considered as forming the brain's “reward retina.” The response properties of macaque dopamine neurons have been found to compute variables essential for reinforcement learning, such as the difference between expected and actual reward outcomes. In simple classical conditioning experiments, dopamine neurons respond to unexpected rewards, with a sharp increase in discharge rate, and to expected but absent rewards, with a sharp decrease in discharge rate.¹⁰ These signals are believed to contribute to learning by signaling both positive and negative changes in the environment, corresponding to reward or utility prediction error postulated by models of economic decision. Dopamine neurons also encode reward expectation through tonic discharges that, interestingly, scale with reward probability and thus convey information about risk and uncertainty.¹¹

Explicit reward signals are found in most midbrain dopamine neurons, as well as in subsets of neurons receiving dopaminergic projections in the orbitofrontal cortex (OFC), in the ventral striatum, and in the amygdala. These neurons further respond to conditioned stimuli that predict future reward. Another class of cells link information about reward with sensory or action information. The OFC contains a representation of primary reinforcers, such as food or sexual stimuli. These are neurons that discriminate sensory quality within and across category, but do not merely encode the physical characteristics of stimuli. Activity is elicited by a specific food item when the animal is motivated to acquire it, but stops if the animal is sated.¹² In other words, reward-coding neurons respond if the “marginal utility” of that particular food is sufficiently high, ie, if it satisfies a current need or desire. Rewards and reward expectation also affect the activity of more dorsal and medial regions of the prefrontal cortex involved in action selection and planning. In fact, many cortical (prefrontal, cingulate, parietal, infero temporal cortex) and subcortical (striatum, amygdala, superior colliculus) structures involved in high-level sensory and/or motor integration receive reward information.¹³ Unrewarding and aversive events are also represented most commonly in distinct neuronal subpopulations in the midbrain, cortex, and amygdala. However, cells in the posterior parietal cortex have been shown to encode both rewarding and aversive stimuli, in which case activity is interpreted as reflecting motivational salience rather than value.¹⁴ Together, these results suggest that reward information serving to select relevant stimuli and to guide goal-directed approach and avoidance behavior is broadcast widely throughout the primate brain.

If reward neurons are to encode decision variables, their activity should reflect subjective value in a generic way; that is, preferences for different goods should be encoded in a common “currency.” Monkeys do not use money to purchase goods. However, their preferences can be precisely estimated by offering fruits or juices in different amounts and measuring subjective equivalences; for example, how much a grape is worth in bits of cucumber. When recorded during the performance of such a task, action potentials of dopamine neurons and reward-sensitive neurons in the OFC have been shown to be discharged in equal numbers for subjectively equivalent food rewards, irrespective of nature or amount, and to scale their firing rate in direct function of behavioral preference, thus effectively representing value in a common neural currency.^15,16 Further studies have focused on how the brain translates subjective value into actual choices and actions. Real-world economic decisions can sometimes be straightforward but mostly involve multiple, ambiguous outcomes and require the integration of several factors, for example, delay, uncertainty, and action cost. Formal models of decision generally postulate a competition between signals that accumulate in favor of each option, racing toward a threshold and culminating into a sort of winner-take-all decision. Evidence for such a process has been observed in several regions, including the dorsal and ventral striatum, and the dorsolateral prefrontal, posterior parietal, and anterior cingulate cortex (ACC), although the models best fitting the data remain a subject of debate.^17,18

Patient studies

Reward and dopamine

Identifying the elementary neural mechanisms used to compute expected values and guide rational decision making is a central objective of research in neuroeconomics, but also in the domains of reinforcement learning (concerned with animals and humans) and machine learning (concerned with computer models and robots). All three disciplines are interested in how rewards and punishments shape responses to the environment bymeans of signals arising in midbrain dopamine neurons. Studies conducted in patients in whom dopamine function is altered, like Parkinson disease, Tourette syndrome, or attention-deficit hyperactivity disorder, are thus directly relevant to this issue. For example, patients with Tourette syndrome show early onset physical and vocal tics, believed to result from dopamine-induced basal ganglia dysfunction and often treated with neuroleptics and dopamine antagonists. In a series of elegant studies, Worbe et al¹⁹ and Palminteri et al^20,21 studied reinforcement learning in Tourette patients, both on and off medication. They used a simple cued go/no-go task to expose participants to cue-reward associations (monetary gain) as well as cue-punishment associations (monetary loss). Patients showed learning profiles according to their treatment status that varied in a manner consistent with the hypothesis of dopamine hyperfunctioning in Tourette syndrome. When off medication, patients had impaired punishment learning, whereas when on medication they had impaired reward learning. The authors interpret this pattern of findings according to the theory of dopamine's dual action in reward and punishment learning through, respectively, positive and negative prediction error signals.¹⁰ In untreated patients, because baseline levels of dopamine are too high, punishments cannot pull the dopamine response low enough to generate a negative prediction error signal. By contrast, in patients treated with dopamine antagonists, baseline levels of dopamine are kept too low, and rewards cannot push the dopamine response high enough to generate a positive prediction error. In support of this hypothesis, a similar, but mirror-reversed dissociation in reward and punishment learning pattern has been found in patients with Parkinson disease on (high baseline dopamine) and off (low baseline dopamine) L-Dopa treatment.^22,23

Although a faulty reinforcement learning mechanism cannot account for the entire spectrum of deficits observed in basal ganglia disorders, it could partly explain a number of motor and cognitive symptoms. Reduced negative reinforcement due to excessive dopamine activity could be involved in the inability to suppress tics in Tourette syndrome, but also in impulse control disorders, such as pathological gambling and hypersexuality, all of which have also been observed as side effects of L-Dopa treatment in Parkinson disease. The major drugs of addiction have a variety of pharmacological actions resulting in overstimulation of the dopaminergic reward system.²⁴ Reduced reward sensitivity due to low dopamine levels might be involved in early nonmotor symptoms of Parkinson disease and in psychiatric conditions associated with depression and apathy. Considerations about the role of dopamine imbalance must also take into account the heterogeneity of lesions and of functional anomalies in the midbrain, striatum, and associated cortical and limbic networks. Understanding the specificity of behavioral disturbances in light of the known anatomy and pharmacology of the basal ganglia is a key objective for much human and nonhuman primate research in this field.²⁵

Nonsocial decision making

The ventromedial and orbitofrontal regions are important cortical targets of dopaminergic projections, but also receive inputs from limbic structures, hippocampus, and other cortical areas. Probably the first and best-known study of economic decision making in brain-lesioned patients is that of Bechara and collaborators,²⁶ who introduced the Iowa gambling task, in which participants are presented with four different decks from which they pick cards in order to earn game money. Two decks contain some cards with small gains and fewer cards with small losses, yielding, on average, positive payoffs. Two other decks contain some cards with high gains and a larger number of cards with high losses, yielding, on average, negative payoffs. Normal subjects, after sampling cards from each deck, develop a preference for the good decks and avoidance of the bad ones. Also, as learning takes place, subjects show an increase in skin conductance before picking cards from the bad decks, even before they become aware of any difference in expected value among the four decks. This implicit stress response is considered evidence for the activation of somatic and emotional signals in guiding decisions, the so-called somatic marker hypothesis. When tested with the Iowa gambling task, patients with ventromedial prefrontal cortex (vmPFC) lesions and patients with bilateral amygdala lesions failed to develop a preference for the safer advantageous deck and either chose randomly or had a preference for the risky gamble, leading to overall monetary losses. Interestingly, while control subjects and prefrontal cortex-lesioned patients showed skin conductance responses to winning and losing, amygdala-lesioned patients failed to show such responses. Furthermore, contrary to controls, neither patient group showed anticipatory skin conductance responses to the risky deck, suggesting that defective emotional signaling in the amygdala and defective routing of this information through prefrontal cortex resulted in a similar negative impact on loss (or risk) aversion learning. Bechara et al²⁷ point out that some of the vmPFC patients were aware that the decks were bad yet persevered in their risky choices; the authors draw a parallel with other pathological conditions, such as drug addiction or psychopathy where maladaptive or pathological behavior persists despite intact awareness of the consequence of one's actions.

Our group used a different type of gambling game in order to study the interactions between emotions and value-based decisions in patients with OFC lesions.²⁸ Subjects chose between two gambles of known probabilities and rated subjectively felt emotions after the outcome was revealed. Different types of emotions could be measured depending on how much information was supplied. In the presence of feedback on the chosen gamble only, subjects experienced either satisfaction or disappointment, depending on the outcome value. When feedback was given on both gambles, they could compare “what is” with “what could have been” and hence could experience more complex forms of emotions, such as rejoicing or regret. This is an important issue, because in behavioral economics, decisions made under uncertainty generate regret aversion, leading subjects to estimate (and sometimes overestimate) future regret when making choices in order to minimize the risk of experiencing it later.²⁹ Patients with OFC lesions reported feeling elated and saddened when their chosen gamble produced wins and losses, respectively. They also showed commensurate emotional arousal, as measured by their skin conductance response, in agreement with previous findings.²⁶ However, contrary to control subjects, neither their emotional responses nor their decision strategies were affected by information on the foregone option. For instance, when the chosen gamble yielded a small positive earning, their satisfaction level was not reduced by discovering that the outcome of the other gamble was even more advantageous, nor was it enhanced when discovering it was much worse.

Although lack of regret could also reflect blunted somatic markers, it should be emphasized that this is a cognitive-based emotion requiring a capacity to perform counterfactual comparison, a process that might be impaired in these patients. Work on monkeys shows that the ACC, an area closely connected to limbic and prefrontal regions, contains neurons that encode the size of rewards, obtained as well as unobtained, in a simple decision task.³⁰ The ACC is thus suited to provide “Active” outcome value signals needed to compare factual and counterfactual information. Neuroimaging studies in humans have shown a cerebral regret network that includes the amygdala, OFC, ACC, and hippocampus. Whereas simple monetary wins and losses in the two-choice gambling task modulate reward-related areas, such as the ventral striatum, experienced and anticipated regret are correlated with enhanced activity in the OFC, amygdala, and ACC.³¹ Other imaging studies have confirmed the key role of the OFC in the regret mechanism, with a possible functional distinction between lateral and medial OFC (see ref 32 for a review).

Social decision making

Interest in the mechanisms of social decision making is more recent. Access to vital resources requires the establishment of trust and cooperation with social partners. One function of social information is its intrinsic rewarding quality. The tremendous success of smartphone photo exchange apps and online social networks is a patent testimony to the high value we assign to observing and learning about our conspecifics. Another function is how “others” are used as comparison points in own-reward evaluation. Social comparison is a basic human activity. As Karl Marx noted in Wage Labor and Capital,³³ it is a major driving force in the burning pursuit of wealth, and it is a source of no less burning frustration:

A house may be large or small; as long as the neighboring houses are likewise small, it satisfies all social requirement tor a residence. But let there arise next to the little house a palace, and the little house shrinks to a hut. The little house now makes it clear that its inmate has no social position at all to maintain, or but a very insignificant one; and however high it may shoot up in the course of civilization, if the neighboring palace rises in equal or even in greater measure, the occupant ot the relatively little house will always find himself more uncomfortable, more dissatisfied, more cramped within his four walls. The role of inequity aversion in decision making is well documented in humans and in several nonhuman primate species.³⁴ In contrast, if the other's reward serves as reference point, ie, when an individual has to make decisions about somebody else's money rather than about t heir's own money, emotional engagement is reduced and risk evaluation less constrained by loss aversion.³⁵ Impairments in socially based cognition and communication have been reported in brain-lesioned patients. Patients with vmPFC lesions engage in socially inappropriate behavior, lacking in awareness of their social norm violations,³⁶ and fail to adjust their communicative behavior according to the social partner,³⁷ thus raising the question of the role of prefrontal areas in social decision making.

Experiments in nonhuman primates have explored how neural activity is affected when decisions are made in a social context. When given the opportunity to provide food or to prevent infliction of unpleasant stimuli to conspecifics, monkeys express variable, partner-dependent social tendencies but mainly make prosocial decisions.³⁸ In the OFC, we have shown that own-reward sensitive neurons respond to a fixed reward to self with different discharge rates, depending on whether monkeys are working to obtain a reward for self only or for both self and a partner monkey. Activity modulations are consistent with simultaneously recorded behavior, as equity-averse animals show reduced neuronal activity to “shared” reward (same amount granted to both) and equity-seeking animals show enhanced activity.³⁹ OFC thus appears suited to perform the neural computations involved in social comparisons. Whereas neurons in the OFC and in the striatum only encode own rewards in a social context,^39,40 ACC neurons distinguish between rewards to self and to others,⁴¹ and medial prefrontal neurons track rewards and action of partner monkeys.⁴² Finally, in a recent experiment using as experimental paradigm an iterated prisoner's dilemma game, neuronal activity in the dorsal ACC was shown to predict another player's intention to cooperate or to “defect,” and disruption of its activity by electrical stimulation reduced the tendency to produce mutually beneficial choices.⁴³ Although food trading may seem quite rudimentary in comparison with complex human social exchanges, these results highlight the key role played by prefrontal cortical areas in regulating the basic mechanisms of cooperation and competition.

Few studies have adopted a neuroeconomic perspective on social decisions made after brain damage. Patients with vmPFC lesions were found to reject unfair offers more often than control subjects in a single-round ultimatum game, but only when outcomes were obtained after a delay and not when outcomes were displayed and gains immediately available.⁴⁴ Thus, rather than reflecting anomalous inequity aversion, the explanation advanced by the authors is that after damage to the vmPFC, future gains are more rapidly discounted (a so-called myopia for the future), thereby making the subjective value of an unfair offer appear even lower—and more unacceptable—than it would to a normal individual. Interestingly, patients also exhibited a more global social impairment. Whereas control subjects display more inequity aversion when facing a human opponent than a computer opponent, choices made by patients with lesions in the vmPFC did not distinguish between the two situations and only depended on the distribution of monetary outcomes Differentiation of human versus computer partners is believed to reflect the role of social norms in economic decision making and, in the present case, the need felt to administer costly altruistic punishment to human partners who behave unfairly.⁴⁵ By contrast, since inanimate objects cannot be held responsible for their choices, the dominant strategy when playing against a computer is rational maximization (ie, accepting a very small offer is better than nothing). In a later study, decisions made by vmPFC-lesioned patients were analyzed in the context of the “trust game,” a social exchange test evaluating trust and reciprocity.⁴⁶ In this game, participants play the role of investor and trustee engaged in a financial exchange. The investor has to transfer all or part of an initial endowment, knowing that the amount invested is tripled as it is transferred to the trustee, and the trustee must then decide what fraction of that amount to return to the investor. As in the previous study, in contrast to control subjects who were more risk adverse (less trusting) with a human than a computer partner, vmPFC-lesioned patients did not act differently in the two settings and they displayed less generosity as trustee. The human vs computer dissociation is taken as evidence of fear of betrayal, another social emotion influencing risky decisions. The complexities of social-economic games call for caution in data interpretation. Nevertheless, observations made in patients with OFC and vmPFC lesions suggest that this region plays a key role in regulating decision making through nonsocial emotions, such as anticipated regret and social/moral emotions, such as envy, fairness, betrayal, and guilt.

Conclusions

The use of experimental approaches and concepts from behavioral economics is a relatively new direction in neuroscience and neuropsychology. In interaction with other domains such as reinforcement learning, computational neuroscience, and social psychology, economic analysis provides a mechanistic framework and explicit hypotheses to investigate covert reward evaluation processes and emotional regulation of individual and social decision making. Further work drawing on comparative electrophysiological and neuroimaging approaches in animals and humans is needed to develop an integrated view of motivated behavior and pave the way toward novel approaches to the treatment of neuropsychiatric disorders.