The good, the bad and the brain: Neural correlates of appetitive and aversive values underlying decision making

Abstract

Approaching rewards and avoiding punishments could be considered as core principles governing behavior. Experiments from behavioral economics have shown that choices involving gains and losses follow different policy rules, suggesting that appetitive and aversive processes might rely on different brain systems. Here we contrast this hypothesis with recent neuroscience studies exploring the human brain from brainstem nuclei to cortical areas. Although some circuits show rigid specialization, many others appear to process both appetitive and aversive stimuli, in a flexible manner that depends on a context-wise subjective reference point. Moreover, appetitive and aversive aspects are often integrated into net values that are signaled with enhanced activity in ‘positive regions’, and suppressed activity in ‘negative regions’. This dichotomy might explain why drugs or lesions can produce valence-specific effects, biasing decisions towards approaching a reward or avoiding a punishment.

Introduction

« Good and evil, reward and punishment, are the only motives to a rational creature: these are the spur and reins whereby all mankind are set on work, and guided. » Since these famous words of John Locke, various experiments have been conducted in order to understand how reward and punishment can guide behavior. In standard decision theory [1], choices are based on option values, i.e. estimates of how good or bad the outcome would be. After outcomes are experienced, option values can then be updated following reinforcement learning theory [2]. In principle, the same decision and learning rules could be applied to appetitive and aversive values. In other words, reward and punishment would be two sides in the same dimension.

Yet several findings in behavioral economics have suggested that human agents have different attitudes toward gains and losses, as if these were split in two domains separated by a reference point. For instance, the endowment effect shows that people are willing to pay more to retain some thing they own than to obtain the same thing from someone else [3]. According to the framing effect, people are risk-averse when considering potential monetary gains, but risk-seeking when dealing with losses [4]. Also, losses are not discounted with time in the same manner as gains, as seen in the so-called “sign effect” [5]. These observations suggest that the human brain has evolved two different systems dedicated to appetitive and aversive processes, which may exert distinct influences on behavioral biases and clinical symptoms. Meta-analyses of neuroimaging studies lend support to this distinction by highlighting neural circuits associated with processing appetitive and aversive reinforcers (see Figure 1).

Figure 1.

A large-scale meta-analysis of neuroimaging studies of reward (560 total reports) and aversive (169 total reports) processes (Neurosynth; Yarkoni et al. 2011). Aversive includes dorsal medial prefrontal cortex (dMPFC), anterior insula (aINS) and amygdala (Amyg). Reward includes striatum (Str, encompassing nucleus accumbens, caudate and putamen) and ventromedial prefrontal cortex (vMPFC).

In this review, we explore this dissociation by examining recent evidence from experiments using functional magnetic resonance imaging (fMRI), pharmacological manipulations and clinical studies. Specifically, we discuss a) the dissociation of neural systems signaling appetitive and aversive value in the human brain; b) the integration of positive and negative features (benefits and costs) into a net decision value; and c) the effect of contextual shifts in appetitive versus aversive value coding. When discussing appetitive values, we consider data using primary, secondary and more abstracts rewards (e.g., food, money and art respectively), while aversive values include action costs like effort or punishing outcomes such as pain or financial loss. We take the approach of first examining neuromodulatory signals sent by deep brainstem nuclei and then extend our discussion to higher brain areas that are more accessible to neuroimaging techniques.

Appetitive and aversive signaling by neuromodulators

Appetitive and aversive processes appear to be disturbed in many pathological conditions. For instance, depression might be characterized by a negative bias - an exacerbation of the aversive aspects of life events and/or dampening of positive experiences, which is typically alleviated by serotonin reuptake inhibitors [6]. In decision-making, such biases may affect how potential benefits are weighed against potential costs. If the potential reward seems less attractive, or if the required effort seems too demanding, patients would prefer to do nothing. This reduction of behavior would be clinically classified as apathy and commonly seen in conditions such as Parkinson’s disease, which is typically treated with dopamine enhancers [7]. These observations motivated the investigation of neuromodulators such as dopamine and serotonin in the study of appetitive and aversive processing, since they were implicated in the etiology or treatment of neuropsychiatric diseases associated with a shift in how benefits are weighed against costs.

Weighing costs and benefits

The trade-off between effort and reward has been explored for a long time in animals, using tasks involving a choice between going for a small food reward that requires little effort or larger reward that requires more effort (e.g., climbing a barrier). The willingness to accept higher efforts for higher rewards is critically dependent on dopamine levels in the nucleus accumbens [8]. This has been replicated in humans, using monetary rewards in exchange of physical efforts exerted on a handgrip. The propensity to choose high effort – big reward options was shown to depend on dopamine level in the striatum and ventromedial prefrontal cortex measured with PET imaging, and to be enhanced by administration of d-amphetamine [9, 10]. Thus, the arbitrage between costs and benefits may depend on neuromodulators such as dopamine. This might explain cases of apathy, such as those observed in depression, negative schizophrenia or Parkinson’s disease, where dopaminergic targets such as the striatum have a blunted response to reward [11, 12].

The shift in cost/benefit trade-off does not imply that different brain systems are involved in appetitive and aversive processes. It could be that dopamine level increases with positive events, favoring approach behavior, and decreases with negative events, favoring avoidance behavior. An a priori argument against this possibility is that the possible range for coding is too narrow on the negative side, since the baseline firing rate of dopamine neurons is quite low (3–5Hz, [13]). This argument suggests the existence of an opponent system that would code negative events positively (with increased firing rate). Dissociating these opponent systems involves comparing choices made between two appetitive on the one hand and two aversive options on the other hand. This is not trivial, since subjects might reframe their expectations if they know the condition (appetitive or aversive) in which they are, such that not being punished can become rewarding, or vice-versa [14].

Learning by carrots or by sticks

Clear separation of appetitive and aversive processing has been operationalized in instrumental learning paradigms. These tasks involve subjects learning by trial and error to select the option that is more rewarding or to avoid the option that is more punishing. Simple models of instrumental learning [2] propose that chosen option value is updated based on the outcome, in proportion to reward prediction error (actual minus predicted reward). Single-unit recording studies in monkeys have suggested that dopamine neurons might precisely encode reward prediction errors [15]. Encoding of reward prediction errors has also been reported in humans using both single-unit electrophysiology [16] and brainstem fMRI [17]. Whole-brain fMRI results mostly show reward prediction errors correlating with dopaminergic targets in the prefrontal cortex [18] and the striatum [19]. Interestingly, the magnitude of the prediction error signal in the striatum has been found to correlate with behavioral performance [20, 21].

The claim that dopamine neurons could also encode aversive events such as effort and punishment is more debated, as many electrophysiology and voltammetry studies have observed that the effect of aversive stimuli is small compared to that of unexpected reward [22]. However, reliable activations related to noxious stimuli have been recorded in dopaminergic nuclei, sometimes in distinct populations of dopaminergic cells [23, 24], or possibly in non-dopaminergic cells [25, 26]. Direct manipulations using optogenetics and microstimulation have been more conclusive on showing a specific involvement of dopamine neurons in reward processing and appetitive behavior [27, 28]. These studies are consistent with the idea that dopamine might influence future choices by increasing the values of chosen options following reward obtainment.

This idea has recently been tested in humans. Dopamine enhancers given to healthy subjects or to patients with Parkinson’s disease improve reward learning but leave unaffected, or sometimes even impair, punishment learning [20, 29]. Conversely, dopamine blockers given to healthy subjects or patients with Tourette syndrome impair reward learning but not punishment learning [30, 31]. Furthermore, reward-related learning has been shown to depend on subject-wise striatal dopamine level measured with PET, and on polymorphism of dopamine receptor or transporter genes [32, 33]. The increase in sensitivity to positive reinforcement might explain the compulsive behaviors, such as hypersexuality or pathological gambling, developed by patients with Parkinson’s disease under dopamine receptor agonists [34].

In face of the evidence for dopamine being implicated in the reward domain, many authors have looked for opponent systems that would be dedicated to the punishment domain. Among neuromodulators, an influential theory has suggested that serotonin could encode punishment prediction errors and therefore teach the brain to avoid repeating unfortunate choices [35]. However, evidence for this proposal is scant, since electrophysiological recording and pharmacological manipulation studies have implicated serotonin in both reward and punishment processing [36, 37]. The theory has been revisited to incorporate the role assigned to serotonin in inhibition, as opposed to invigoration, of behavior [38]. Some studies have indeed reported that decreasing serotonin level through tryptophan depletion impaired punishment-induced inhibition [39, 40] but the evidence remains mixed. Serotonin has been implicated in other types of costs such as the cost of information sampling [41] or the opportunity cost induced by waiting for reward delivery [42, 43]. Thus, it is tempting to conclude that serotonin has a differential implication in the cost rather than the benefit domain. However, whether serotonin helps avoiding costs or overcoming costs (in order to get more reward) is still a matter of debate [44, 45].

Beyond neuromodulation: whole-brain mapping of appetitive and aversive values

Common neural currencies

Outside the brainstem, the search for opponent systems underlying appetitive and aversive processes has been extended to the entire brain in humans using fMRI. Seminal studies in animals have allowed for the delineation of a neural circuit involved in processing rewards (for review see [46]). Central to this circuit is the role of the striatum and ventromedial prefrontal cortex (vmPFC) in representing and adjusting the value of a stimulus to inform decision-making (e.g., [47, 48]), particularly in humans, as highlighted by recent neuroimaging meta-analyses (e.g., [49, 50]). Both regions are engaged by typical rewards such as food, liquids and money (for review see [51, 52]), but also more abstract rewards including art (e.g., [53]), attractive faces (e.g., [54]), social feedback (e.g., [55]) and even intrinsic experiences related to well-being and positive emotions (e.g., [56]). The ability of the striatum and vmPFC to represent distinct types of rewards allows for the flexible comparison of such stimuli when making decisions (e.g., valuation of a monetary reward in comparison with social praise). Indeed, the vmPFC in particular has been associated with representing the difference in subjective value of distinct goods (e.g., mugs; [57]; snacks; [58]) as a function of how much people are willing to pay for it (e.g., [59]), which provides a “common currency” where value signals from distinct rewards are coded on a common scale to inform decision-making [52].

While the neural circuits underlying reward processing are well characterized, there are more questions surrounding the neural basis of aversive processing. Aversive outcomes that elicit intense feelings of arousal such as disgusting pictures are associated with activity in the insula (e.g., [60]). Aversive conditioning with primary reinforcers such as shock is thought to depend on the integrity of the amygdala [61], which has been posited to potentially code for the associability of a conditioned stimulus [62] and assisting in the computations of how aversive an outcome is prior to decision-making [63]. In contrast to primary reinforcers such as shock, aversive outcomes associated with secondary reinforcers (e.g. financial loss) yield inconsistent results with respect to amygdala activation, with some observations [64] particularly in the context of framing [65], but other non-observations perhaps due to the intensity of the stimuli [66]. However, monetary loss is often observed in the striatum [67] and medial prefrontal cortex [68], characterized by a decrease in activity (compared to a neutral baseline) that can correlate with behavioral decisions [69]. The insula has also been shown to respond to monetary losses, especially the anticipation of a loss [68], which has been related to risk predictions [70] and decisions to be risk-averse [71]. Finally, the dorsal anterior cingulate, part of the dorsomedial prefrontal cortex has been linked with aversive values such as calculating the effort (i.e., cost) necessary to obtain a reward [72].

FMRI studies also provide several examples of overlapping appetitive and aversive processes in the human brain. The striatum, for example, while often linked to reward processes, is also recruited during aversive learning [66] perhaps reflecting a role in actively coping with negative reinforcers [73] as in avoidance learning [74, 75]. Consistent with a role of the striatum in reinforcement learning, aversive prediction error signals are also observed in this region, although compared to reward prediction errors, aversive signals are argued to be spatially distinct within the striatum [76] and selectively potentiated under a stressful state [77]. Finally, activity in the putamen to both appetitive [78] and aversive [79] Pavlovian cues exerts a behavioral influence on instrumentally learned behavior. Taken together, neuroimaging evidence suggests that appetitive and aversive processing overlaps in the human striatum, however with some noted spatial distinctions within the striatum. Notably, appetitive and aversive processing have been associated with anterior and posterior parts of the striatum respectively [76] and even ventral and dorsal divisions respectively, with the dorsal striatum reflecting more anxious feelings associated with a decision [80].

Contextual dependencies

The activity in value related regions to appetitive and aversive information has been suggested to reflect the relative magnitude of the potential outcome with respect to a reference point, as observed by increases in activation for gains and decreases for losses during receipt in striatum [67] and mPFC [68]. This could be viewed as evidence for separate systems, or could be thought of as an overlapping system that represents value in a continuous scale. Indeed, when decisions involve both gains and losses and the integration of cost-benefit must occur to inform choices then vmPFC in particular has been observed to reflect appetitive and aversive values in an economically coherent way. For example, it has been argued that loss aversion during decisions – i.e., greater sensitivity to losses than gains –is reliant on this relative coding with respect to a reference point in neural regions associated with value [81]. This is also observed with primary appetitive and aversive reinforcers (e.g., food) which have been shown to be similarly represented in the vmPFC to inform decision-making [82].

Thus, the calculation of costs and benefits to inform decisions involves putative reward circuitry in a manner that suggests a calculation of positive values (to attain a reward or avoid a punishment). Importantly, this value calculation has to take into account not only the appetitive and aversive values but the context in which they are experienced. Context can refer to a person’s state (e.g., stressed individuals show blunted striatal responses to monetary gains - [83] - akin to blunted response by individuals with depression; [11]), but can also simply refer to the subjective value of a reinforcer (e.g., benefit) in the presence of a different reinforcer (e.g., cost). For instance, an individual’s preference for control during decision-making has been associated with ventral striatum activity [84] a bias posited to be driven by greater prediction errors [85]. Interestingly, this preference for control is apparent when the choice is solely tied to a potential monetary gain or to avoiding a monetary loss, but it is abolished when losses and gains are presented concurrently [86]. It suggests that context in which an appetitive reinforcer is viewed (i.e., in the presence or absence of a potential aversive stimulus) can influence value related neural signals. In accordance with this, context-dependent shifts in risk preferences (greater preference for risk in the context of a prior loss) are observed in vmPFC [87]. An interesting effect of context is the case of the “near miss” during gambling where an aversive event (a miss) is treated as an appetitive event due to the proximity to what would have been a “win” response. In such cases, near miss trials evoke activity in reward-related regions such as the striatum and have increased connectivity with the insula [88].

The calculation of subjective value when a decision involves potential gains and losses often involves a consideration of cost-benefit comparisons that factors an individual’s susceptibility to loss aversion and sensitivity towards risk. However, at times decisions that can lead to a reward may also be associated with costs of different modalities (e.g., effort or avoiding pain). When costs (pain by electric shock) and benefits (monetary rewards) are presented, decision-making is influenced by the interaction of the two as the appeal of benefits is attenuated with increases in costs, a process that leads to attenuation of ventral striatum and vmPFC prediction error signals [89] and stronger coupling of vmPFC and amygdala pathways [90]. A potential mechanistic explanation of how cost-benefit computations may occur supports the idea of two potential systems that interact prior to decision-making [63]. Specifically, Basten and colleagues suggest that the vmPFC and dorsolateral PFC calculate the difference between costs and benefits that arise from anticipatory signals within the amygdala and ventral striatum respectively.

Conclusions and moving forward

In this review, we highlight neural systems involved in appetitive and aversive processes that can exert distinct behavioral biases. While some neural regions have been primarily linked with appetitive or aversive processes (e.g., ventral striatum and anterior insula respectively), alluding to the existence of two separate systems, we also note examples in which this dichotomy is not fully supported, suggesting that there may not be regions responsible for processing solely one domain or the other. Rather, the classification of values as appetitive or aversive may flexibly depend on the context the information is perceived and reflect changes in value in accordance with a subjective reference point. Even when correlation with appetitive and aversive processes are observed at the scale of brain networks using fMRI, electrophysiological studies often show much more diversity within the single-cell level, as exemplified by findings of dopaminergic and non-dopaminergic cells in the ventral tegmentum area. One question for future investigations is to understand how aggregate signals recorded with human neuroimaging techniques can emerge from the computations operated by individual neurons.

Yet some functional specialization has been consistently observed at the population level and helps explain why certain neural perturbations have valence-dependent effects, such as those induced by dopaminergic drugs or by focal brain lesions (Box 1). This has been helped by the use of parametric designs, which avoid confounding value coding with differences or commonalities in sensory features. A key finding is that some brain regions such as the vmPFC integrate information about appetitive and aversive values across different reinforcer modalities (from basic rewards such as food to more abstract ones such as social reinforcers), so as to compute a net value that might guide decision making in a cost-benefit type of calculation. How such net value signal connects to the motor representations that underpin the behavior is another matter for future investigations. It remains possible that some low-level representations of aversive and appetitive values (as perhaps in neuromodulatory systems) might directly map onto approach and avoidance motor patterns, and compete for behavioral control in a more implicit fashion than the cost-benefit arbitrage that is operated in higher-level cortical areas. New and more direct techniques, such as optogenetics, or a combination of methods, such as the use of pharmacological manipulations with neuroimaging techniques, may allow for better insight and characterization of neural systems involved in appetitive and aversive processes.

BOX 1. Effects of focal lesions on appetitive versus aversive processes.

Activation studies using either electrophysiology or fMRI have failed to identify brain regions that would be purely dedicated to reward or punishment. However, some consensus has emerged about some neural representation of net value (in which appetitive aspects are discounted by aversive aspects). For instance, common findings in fMRI studies are the positive correlation with net value in ventral fronto-striatal circuits, and the negative correlation in the anterior insula and sometimes amygdala [50]. These correlations do not mean that these regions are necessary for integrating appetitive and aversive values in decision making. However, these findings can be corroborated by studies with brain-damaged patients which provide a better argument for causality. For instance, patients with Huntington’s disease were impaired in reward-based learning as soon as neural degeneration reached their ventral striatum [75]. On the contrary, patients with glioma affecting the anterior insula were impaired in punishment-based learning but not reward-based learning [75]. Damage to the amygdala is known to impair avoidance of aversive objects, as described in the Kluver-Bucy syndrome [91]. A recent study investing choices between lotteries reported that patients with amygdala calcification did not exhibit the change in risk attitude between gain and loss domains (the so-called framing effect) and were therefore considered as more rational than healthy subjects [92]. Thus, damage to regions that positively encode net value is likely to reduce the contribution of appetitive processes and damage to regions encoding net value negatively might impair the contribution of aversive processes to decision making. The situation is more balanced with lesions to the ventromedial prefrontal cortex, in keeping with the putative role of this region in integrating appetitive and aversive dimensions. Such a lesion was shown to affect information integration in multi-attribute decision-making [93], to amplify inconsistencies in choices between appetitive food items [94], but also to impair learning from negative feedback [95].

Highlights.

• -The same brain regions process rewards or punishments across reinforcer modalities
• -No strict separation of brain systems processing appetitive and aversive events
• -Appetitive or aversive depends on a context-dependent subjective reference point
• -Some brain regions integrate appetitive and aversive aspects into net values
• -Net values are positively encoded in some brain regions, negatively in others