Reward

Abstract

Neurons throughout frontal cortex show robust responses to rewards, but a challenge is determining the specific function served by these different reward signals. Most neuropsychiatric disorders involve dysfunction of circuits between frontal cortex and subcortical structures, such as the striatum. There are multiple frontostriatal loops and different neuropsychiatric disorders involve different loops to greater or lesser extents. Understanding the role of reward in each of these different circuits is a necessary step in developing novel treatments for these disorders. In this chapter, we summarize the recent literature that has identified the role of reward in different subregions of the frontal cortex. Orbitofrontal cortex integrates information about multiple aspects of expected rewards in order to derive their value, which can then be used to decide between alternative potential rewards. Neurons in anterior cingulate cortex encode the difference between the expected reward and the actual outcome. This information is useful for learning, since it can ensure that behavior changes when the outcome was not anticipated. Reward also affects signals in lateral prefrontal cortex related to attention and response selection, ensuring that behaviors are optimally prioritized. Finally, we discuss how reward signals contribute to social processing and autonomic control.

Introduction

A central function of executive control is to manage capacity bottlenecks that arise in cognitive processing and behavior. We are only able to interact with, attend to or hold in mind a limited number of things at any given instant in time. In order to manage this optimally, mechanisms of prioritization are necessary. This in turn requires the organism to compute how valuable a given course of behavior will be over the long run. Reward signals play a crucial role in this process and so it is not surprising that such signals are evident throughout the frontal lobe.

A problem in addressing the ubiquity of reward signals is that reward can serve many different functions. Rewards have sensory (e.g. flavor, intensity) and emotional components (they make us happy), they satisfy motivational drives, they have a value (some rewards are more preferable than others), they are reinforcing (we repeat behavior that produced a reward), and they can guide the allocation of cognitive and attentional resources. In this chapter, we examine the differences in reward signals across the prefrontal cortex and discuss how the signals contribute to the function of different prefrontal areas. We begin with orbitofrontal cortex (OFC) whose functions are most directly related to reward processing.

Orbitofrontal cortex

OFC occupies the ventral surface of the frontal lobe. It consists of five cytoarchitectonic subregions: frontal polar area 10, area 11 anteriorly, area 13 posteriorly, area 14 medially, and area 47/12 laterally (Carmichael and Price, 1994; Petrides and Pandya, 1994). OFC extensively connects with areas responsible for processing reward information, such as the amygdala, the hypothalamus, cingulate cortex and brainstem structures such as the periaqueductal gray matter (Carmichael and Price, 1995a). It also connects with areas responsible for processing sensory properties of rewards, such as primary olfactory and gustatory cortex (Carmichael and Price, 1995b). In summary, the connections of OFC are compatible with a structure that integrates sensory and reward information.

The position of OFC in the skull, resting on top of our eye orbits and the ridges created by the sphenoid bone, makes it particularly susceptible to damage from head trauma. Aneurysm of the anterior cerebral artery also affects OFC. Yet damage to OFC often appears to have remarkably little effect. One case that has been extensively described is that of Elliott who was diagnosed with a brain tumor at the age of 35 (Damasio, 1994; Eslinger and Damasio, 1985). The operation to remove the tumor was successful, but the surgery left Elliot with bilateral damage to his OFC. However, neuropsychological tests could find no evidence of brain damage. Tests of his intelligence, memory, reading and writing comprehension, verbal fluency, visuospatial abilities, and facial recognition revealed average to superior performance. Even tests designed specifically to tax frontal lobe processes, such as working memory, rule switching, and cognitive estimation, failed to reveal any deficits. However, within months of the operation, Elliott made a series of catastrophic life decisions, including losing his job, his life savings, divorcing his wife and marrying a prostitute. This is typical of OFC damage: intact cognitive abilities with a devastating loss in the ability to make good, everyday decisions.

Dysfunction of OFC is also implicated in many neuropsychiatric disorders that are characterized by poor decision-making. OFC is the cortical area most implicated in addiction. Researchers studying addicts have seen reductions in the overall volume of OFC (Tanabe et al., 2008; Franklin et al., 2002) as well as changes in neuronal activity (Bolla et al., 1998; Volkow et al., 1991). Addicts also show the same pattern of impaired decision-making as OFC patients (Bechara and Damasio, 2002; Bechara et al., 2002; Bolla et al., 2003; Rogers et al., 1999). OFC dysfunction is also implicated in obsessive-compulsive disorder, pathological gambling, self-injurious behaviors and eating disorders (Pelchat, 2002; Everitt et al., 2007; Cavedini et al., 2002; Fernando and Robbins, 2011). Impaired choice behavior is a common feature of these disorders: the patient feels compelled to make a particular choice, despite the negative consequences associated with that choice (Voon et al., 2015).

Since OFC patients were first described, several tests have been developed that are able to detect these decision-making deficits in the laboratory. A very simple test was devised by Fellows and colleagues. The patient is presented with two colored swatches and asked which they prefer. There is no right or wrong answer, but normal subjects show consistency in their choices. First, they are temporally consistent: if they prefer red over blue, then a week later they will still prefer red over blue. Second, they are internally consistent: if they prefer red over blue and green over red, then they will prefer green over blue. Patients with OFC damage do not show any consistency; they appear to be choosing almost at random (Fellows and Farah, 2007). The same pattern of impaired decision-making is observed using more formal tests of subjective decision-making that have been developed by behavioral economists (Camille et al., 2011).

Studies in monkeys have begun to reveal what computations are being performed by OFC that makes it so critical to decision-making. In a now seminal study, Padoa-Schioppa and colleagues trained monkeys to make choices between different volumes of different types of juice reward (Padoa-Schioppa and Assad, 2006). To make their choice effectively, the monkey needed to consider both variables. For example, a thirsty monkey might prefer the taste of fruit juice to water. If the choice is between equal volumes of both, he will obviously choose the juice. However, increasing the volume of water available can compensate for its less desirable taste. If the volume of water is sufficiently large, relative to the juice volume, then the monkey will pick the water. At some point, the volume of water will compensate for its less desirable taste exactly, and the monkey will be indifferent between the two choices. This measures the monkey’s value of one reward’s taste relative to the other. For example, if the monkey is equally likely to choose four drops of water or one drop of fruit juice, we know that the monkey considers the taste of juice four times more valuable than water. Padoa-Schioppa found that during the decision, the firing rate of OFC neurons correlated with the value of the juices on offer. For example, a neuron might show a higher firing rate when the monkey was choosing one drop of juice compared with when he was choosing one drop of water. However, the neuron’s firing rate would be the same when the monkey was choosing one drop of juice compared with when he was choosing four drops of water. We cannot explain this pattern of neuronal activity on the basis of the drinks’ volume because equal volumes of the drinks produce different neuronal firing rates. Nor can we explain it solely by the drinks’ taste because certain volumes of the drinks produce equal levels of neuronal firing. However, we can explain it in terms of the monkey’s valuation: when his valuation of the two drinks is the same (such as when there are four drops of water or one drop of juice), the neuronal firing rate is also equivalent.

Over the past decade, OFC neurons have been shown to encode a variety of signals that would be important for determining the value of an expected outcome. One of the clearest examples of this is a study we performed in which we trained monkeys to make binary choices among a large set of pictures that was associated with outcomes that varied along three dimensions (Kennerley et al., 2009). We trained monkeys to fixate a central dot, before two pictures appeared on the left or right of the screen (Fig 1A). The animal had to choose between these pictures, and indicate its choice through a left or right lever movement. Each picture was associated with either a specific probability of reward, a specific amount of reward, or a specific number of lever presses that the animal needed to make before the reward would be delivered (Fig 1B). Choices always consisted of pictures that were adjacent to one another in value, where value was defined as more juice or less work. Individual OFC neurons often changed their firing rate according to the value of the choice under consideration, and they did so irrespective of the physical way in which we had manipulated the value of the choice (Fig 1C). OFC neurons have subsequently been found to encode many other factors relevant to decision-making including the physical effort necessary to earn a reward (Hosokawa et al., 2013), the time until a reward is delivered (Roesch and Olson, 2005) and the confidence one has in the decision (Kepecs et al., 2008). All of these factors are integrated into an abstract signal that indicates the value of choosing a given option. These findings in monkeys help to explain why patients with OFC damage are so poor at making decisions. When we are faced with a decision our OFC neurons fire in a way that indicates the value of the likely outcome associated with either choice option. OFC patients are forced to make decisions more or less at random because they lack the signals as to the value of the expected outcomes that might result from different possible choices.

Figure 1.

A) Timeline of the behavioral task. B) Set of pictures and outcomes which were used to test animals’ choice behavior. C) Orbitofrontal neuron whose firing rate increased as the value of the choice increased, irrespective of whether we manipulated value via the probability or size of the reward, or the amount of work necessary to earn the reward.

Functional organization of orbitofrontal cortex

OFC is a large cortical area. In humans, it occupies about 10% of our total cortical area (Semendeferi et al., 2002). A number of studies have focused on understanding how OFC is organized. A meta-analysis of neuroimaging studies suggested that OFC is organized along two axes: a mediolateral axis whereby positive outcomes are encoded medially and negative outcomes are encoded laterally, and an anterior-posterior axis whereby concrete outcomes are encoded more posteriorly than abstract outcomes (Kringelbach and Rolls, 2004).

Evidence supporting the anterior-posterior axis has continued to accumulate over the subsequent years. For example, decisions involving concrete rewards, such as food or erotic stimuli activate more posterior regions of OFC compared to decisions involving abstract rewards, such as money (Sescousse et al., 2010). In contrast, the idea that negative outcomes are encoded more laterally than positive outcomes has received little support. Studies using Pavlovian conditioning found that OFC neurons that encode upcoming rewards (fruit juice) were intermingled with those encoding upcoming punishments (air puffs to the face) (Morrison and Salzman, 2009). More recently, we trained two monkeys to perform a visuomotor association task for secondary reinforcement (Rich and Wallis, 2014). Monkeys learned that the length of a reward bar shown on their task screen corresponded to the amount of juice they would receive after completing a block of six trials. The advantage of this experimental design is that the same physical stimulus, a reward bar, can be used to either reward the animal, by increasing its length, or punish the animal, by decreasing its length. Although many OFC neurons encoded the valence of the expected outcome, neurons encoding rewarding or punishing outcomes were randomly intermingled and there was no evidence of a mediolateral organization.

An alternative organization has been proposed by Rushworth and colleagues (Rushworth et al., 2011). They contrasted the effects of selective lesions of either the lateral or medial OFC in monkeys. The results suggested the lateral OFC is responsible for credit assignment. This is the computation by which we correctly assign a reward to the behavior that produced that reward so that we can repeat that behavior in future and obtain more reward. This can be a complex process and the computations that underlie it remain obscure. For example, consider eating a delicious meal in a restaurant. Which behavior should we reinforce? Should we reinforce the act of opening the door that led us into the restaurant? Or should we reinforce the behavior that got us a pay raise that enabled us to eat in the upscale restaurant? The latter makes sense; we need to earn money if we want the finer things in life. Increasing our tendency to open doors, on the other hand, is unlikely to improve our ability to eat at fancy restaurants. However, it is unclear how the brain bridges the temporal and causal discontiguities that would allow us to associate the meal with the pay raise in order to ensure that the correct behavior is reinforced. Lateral OFC appears to play some role in this process. Monkeys with damage to lateral OFC showed impairments in reward-guided learning because they had difficulty assigning the rewarded outcome to the immediately preceding choice (Noonan et al., 2010). They were more likely to assign the reward to earlier or even subsequent choices.

In contrast, medial OFC seems to play a more central role in reward-based decision-making. Monkeys with damage to medial OFC had difficulty with value comparison, particularly when options were close in value, and had a greater propensity to allow irrelevant alternatives to interfere with their decisions (Noonan et al., 2010). Subsequent work in humans has supported this distinction between the medial and lateral OFC. For example, in humans performing value-based decisions, frontal lobe magnetoencephalography signals reflected the difference in value of the two alternatives on offer and this value difference was localized to medial OFC (Hunt et al., 2012). Similar signals have been recorded from single neurons in medial OFC. In monkeys trained to make choices between two sequentially presented offers, value coding in medial OFC neurons is anti-correlated between the two offers, again consistent with the two offers being compared and the difference in value being computed (Strait et al., 2014). In sum, these studies suggest a key role for OFC in comparing options in order to make decisions.

Neuroimaging and the ventromedial prefrontal cortex

One of the most robust signals in human neuroimaging studies is the tendency for ventromedial prefrontal cortex (vmPFC) to be activated during a wide variety of value-based decision-making tasks (Montague et al., 2006). This region consists of area 14, which occupies medial OFC and the ventral part of the medial PFC. In contrast, neurophysiological studies in monkeys usually record from central OFC, which consists of areas 11 and 13 (Padoa-Schioppa and Assad, 2006; Kennerley et al., 2009). Studies that have recorded from vmPFC have found neurons that encode value, but they are typically less numerous and more weakly tuned than neurons in areas 11 and 13 (Bouret and Richmond, 2010; Rich and Wallis, 2014; Strait et al., 2014). The distinction between these areas has important anatomical implications, because medial PFC and OFC belong to distinct anatomical networks (Carmichael and Price, 1996). Areas within the medial wall, including area 14, connect strongly with one another, but only weakly with OFC. In contrast, areas within OFC, including areas 11 and 13, connect strongly with one another, but only weakly with the medial wall.

There are a number of possibilities that might explain this discrepancy between vmPFC in human and monkey studies (Wallis, 2012). First, it may be that the area is not homologous between the two species. This is unlikely. Cytoarchitectonic studies using quantifiable image-processing methods show that vmPFC shares similar cytoarchitectonics in both species (Mackey and Petrides, 2010). Studies that have compared patterns of connectivity across species using diffusion tensor imaging have also supported the close similarity in monkey and human OFC organization (Croxson et al., 2005).

Second, the cognitive demands of tasks that activate vmPFC in humans may not match those used to probe OFC in monkeys. For example, it is possible that there could be an additional social component to human decision-making tasks that is not present in monkey tasks, leading to greater activation of vmPFC and its adjacent regions in humans. In many of the tasks used in humans, subjects are trying to maximize the amount of money that they win, but the amounts of money are not usually large, and subjects’ motivations might have more to do with impressing the experimenter than winning money per se. Another difference is that tasks in monkeys usually employ a limited set of decisions with which the animals are already familiar, whereas studies in humans employ novel decisions. Indeed, vmPFC in humans is activated when you have to use your imagination to determine whether you would like a novel experience, such as the flavor of tea-flavored jelly (Barron et al., 2013). There is also remarkable habituation of value signals in vmPFC if a subject is repeatedly faced with the same decision (Hunt et al., 2012). These properties of vmPFC may make it difficult to see value signals in monkey neurophysiology studies, where the inherent stochasticity of single neuron recordings requires repeated presentation of decisions in order to average neural activity across trials.

The different methods used to study decision-making in monkeys and humans could also contribute to the differences observed between the species. Neuroimaging data may be more sensitive to value signals in vmPFC than in OFC. Susceptibility artifacts arise in functional magnetic resonance imaging (fMRI) scans near air-tissue boundaries, and the nasal sinuses lie directly underneath OFC, making it particularly prone to these kinds of artifacts (Glover and Law, 2001). Supporting this, other imaging methods that are not prone to susceptibility artifacts, such as positron emission tomography, do show activation of central OFC in human decision-making studies (Arana et al., 2003; Chaudhry et al., 2009). Different results could also arise because fMRI and single-unit neurophysiology are sensitive to different physiological parameters. The blood oxygen level–dependent (BOLD) response, measured by fMRI, correlates with the local field potential (LFP) rather than the action potentials of individual neurons (Logothetis et al., 2001). This has sometimes been interpreted to mean that the BOLD response reflects the inputs of an area, whereas single-unit neurophysiology reflects the outputs, but the reality is more complex. For example, an increase in activity in inhibitory interneurons can increase energy consumption and the BOLD response (Buzsaki et al., 2007), even though the functional consequence may be deactivation of the area. In addition, neuromodulatory systems can affect large numbers of cells and potentially induce greater changes in the fMRI signal than changes in the spiking rate of a small set of function-specific neurons (Logothetis, 2008). Similarly, top-down feedback signals can induce a larger BOLD response in sensory cortex than bottom-up signals related to the processing of the stimulus (Sirotin and Das, 2009). The interaction of these factors could considerably complicate the interpretation of the fMRI signal in vmPFC.

Finally, the functional organization in an area may affect how difficult it is to detect signals with fMRI. Sensorimotor areas frequently show a topographic mapping of the sensorimotor parameter space. In such cases, averaging across large populations of neighboring neurons, as the BOLD response does, could still extract the parameter. However, there is little evidence of such topography in OFC, with neurons recorded on the same electrode showing selectivity to very different decision parameters (Morrison and Salzman, 2009; Kennerley et al., 2009). Furthermore, OFC neurons show a diametrically opposed encoding scheme: approximately half of the value-encoding neurons increase their firing rate as value increases, whereas half increase their firing rate as value decreases (Kennerley et al., 2009; Padoa-Schioppa and Assad, 2006; Morrison and Salzman, 2009). These two populations could potentially have opposing effects on the BOLD signal, canceling one another out. Given this, multivariate decoding methods might still be able to extract the value signal from the fMRI data but, to date, studies using this method have broadly reached the same conclusion as univariate methods: significant reward information could be decoded from vmPFC rather than areas 11 and 13 (Kahnt et al., 2010).

One possible way to reconcile the findings in humans and monkeys would be for neurophysiologists to analyze LFPs, particularly in vmPFC, as LFPs may better correlate with the fMRI response. LFPs in rat OFC do contain decision-related information, such as the magnitude (van Duuren et al., 2007) and probability (van Duuren et al., 2009) of expected rewards. In addition, there is evidence that the LFP may be one mechanism by which functional ensembles of neurons across the frontal lobe can be coordinated and communicate with one another (Canolty et al., 2010). For example, in an odor-discrimination task, spikes from movement-related OFC neurons phase-locked to the gamma band of to the theta band (van Wingerden et al., 2010). The LFP may be crucial for coordinating functional ensembles of OFC neurons that are responsible for implementing distinct cognitive processes that may underlie decision-making.

Anterior cingulate cortex

Patients with bilateral damage to ACC are considerably rarer than those with bilateral damage to OFC. Of the handful of studies that have been performed, a common finding is that ACC patients appear to have difficulty in modifying their behavior in response to feedback (Modirrousta and Fellows, 2008; Williams et al., 2004). For example, Williams and colleagues showed that patients with ACC damage were less likely to change their behavior when the monetary reward was decreased (Williams et al., 2004). With regard to the clinical picture, ACC has been particularly implicated in depression and anxiety (Drevets, 2000) and it is one of the principal targets for deep brain stimulation to alleviate treatment-resistant depression (Mayberg et al., 2005).

Neurophysiological studies in animals have revealed remarkable similarity in the reward -related neuronal responses observed in ACC and OFC (Kennerley et al., 2009; Cai and Padoa-Schioppa, 2012), raising the question as to what the two areas are doing differently from one another. A comparison of ACC and OFC anatomy suggests potential answers to this question. Although both ACC and OFC connect with areas responsible for processing rewards, ACC has strong connections with motor areas but few direct connections with sensory cortex, while OFC shows the opposite pattern (Carmichael and Price, 1995a; Carmichael and Price, 1995b; Dum and Strick, 1993). Thus, there may be a division of labor between OFC and ACC with regard to choice evaluation, with OFC calculating the value of possible outcomes, while ACC calculates the value of the action producing the outcome and monitors the success of behavioral outcomes over time to guide adaptive behavior. Consistent with this, lesions to ACC in rats disrupts choices between options differing in their payoff and the effort necessary to obtain that payoff (Walton et al., 2002), but such lesions do not affect other types of cost-benefit decisions (e.g. delay-based decisions), that do not require the evaluation of an action (Rudebeck et al., 2006b). In contrast, rats with OFC lesions show impaired delay-based decision-making, but intact effort-based decision-making (Rudebeck et al., 2006b). These findings could help explain the role of ACC in depression which is characterized by both anhedonia (the patient does not derive pleasure from previously rewarding activities) and anergia (the patient has little energy with which to engage in previously rewarding activities). These symptoms could conceivably reflect a disruption in the ability to integrate reward and effort information to determine the value of a given behavior.

To obtain neurophysiological data to support these ideas, we trained monkeys to perform effort-based or delay-based cost-benefit analyses (Hosokawa et al., 2013). Effort trials required monkeys to lift and hold a lever in place in order to earn a reward, where the amount of necessary force could be pneumatically adjusted (Fig 2A). Delay trials required the monkey to sit a wait a specific length of time until the reward would be delivered. There were two sets of 16 pictures, each of which was associated with a specific reward and a specific cost (Fig 2B). We found that some neurons in both OFC and ACC integrated information about costs and benefits in order to calculate an abstract value. However, neurons responding to delay or effort costs occurred in both areas. In a follow-up study, we examined the dynamics of this process (Hunt et al., 2015). First, we identified a signal that was unique to lateral prefrontal cortex (LPFC). Neurons in this region initially encoded the value expected from a given choice, but then switched to encoding the motor response necessary to obtain that outcome. We then found that dynamics in the LFP in ACC and OFC predicted when this switch would occur, suggesting that these two areas are controlling the flow of information into LPFC to translate a decision into action. Critically, this control was most evident in OFC for delay-based decisions and in ACC for effort-based decisions (Fig 2C). These results highlight the fact that the relationship between lesion effects and the underlying neurophysiology can be complex. The important differences may sometimes lie, not in what information the neurons are encoding, but rather in how they are using that information.

Figure 2.

A) Timeline of the behavioral task. B) Set of pictures and outcomes which were used to test animals’ choice behavior. On effort trials, each picture was associated with receiving a specific reward amount and having to exert a specific force in order to lift the lever. On delay trials, the outcome was a specific reward amount that was delivered after a specific delay. C) We calculated the percentage of variance that could be explained in LPFC firing rates by the LFP in ACC or OFC. The plot shows the difference in this measure on effort trials versus delay trials. Positive values indicate a larger effect on effort versus delay trials, while negative values indicate a larger effect on delay versus effort trials. Both OFC and ACC LFPs affect LPFC firing shortly after choice onset, but in ACC the effect is larger on effort trials, while in OFC the effect is larger on delay trials.

Another difference between ACC and OFC is that ACC neurons robustly encode the outcome of a choice, whereas OFC neurons show stronger encoding at the time of the choice rather than the outcome (Cai and Padoa-Schioppa, 2012). More formally, OFC neurons appear to be encoding a reward prediction, the value expected to arise from a given choice, whereas ACC neurons appear to be encoding a reward prediction error (Figure 3), whether what just occurred was better or worse than anticipated (Kennerley et al., 2011). This information can be used as a teaching signal (Wallis and Rich, 2011), increasing the likelihood of behaviors that lead to reward, similar to the signals encoded by dopamine. Indeed, ACC receives the heaviest dopamine projection in prefrontal cortex (Williams and Goldman-Rakic, 1993). ACC has also been associated with error-related activity (Holroyd and Coles, 2002; Debener et al., 2005). However, for most behavioral tasks, errors occur less frequently than rewards, suggesting ACC outcome activity might instead reflect violations in expectancy (Oliveira et al., 2007; Jessup et al., 2010) or how informative an outcome is for guiding adaptive behavior (Walton et al., 2004; Jocham et al., 2009; Behrens et al., 2007), consistent with a role in encoding prediction errors.

Figure 3.

A) ACC neuron encoding a positive prediction error. At the time of the choice, the neuron responded more to pictures that predicted a higher probability of receiving a reward. At the time of the outcome, the neuron responded to the delivery of reward, particularly when the animal was least expecting to receive a reward, i.e. those trials in which the pictures predicted a low probability of reward delivery. B) ACC neuron encoding a negative prediction error. The neuron responds more to pictures that predicted a lower probability of receiving a reward. It showed little response to the delivery of reward, and instead responded when reward was omitted, particularly on those trials where the animal was most expecting to receive a reward.

A recent theory has attempted to bring together the role of ACC in effort-based decision-making and encoding of prediction errors. The expected value of control theory argues that the critical function of ACC is to determine how much effort to invest in cognitive processes as well as behavior (Shenhav et al., 2013). Implementing certain cognitive processes, such as cognitive control, is postulated to require effort and ACC is responsible for determining whether this effort is worthwhile given the potential payoff from the behavior, how much effort should be invested in a task and, when several potential tasks are competing with one another, which is the most worthwhile. In support of this, neuroimaging studies have shown that ACC is activated when either cognitive or physical effort must be exerted (Botvinick et al., 2009a). Thus, the prediction error signals in ACC serve dual roles: indicating that cognitive control may need to be exerted since outcomes were not as expected, as well as indicating the value of the unexpected outcome to determine how much control is worthwhile. ACC could then initiate the implementation of cognitive control via its interactions with LPFC (Kouneiher et al., 2009).

Lateral prefrontal cortex

LPFC is responsible for the implementation of cognitive control (Miller and Cohen, 2001), which requires multiple cognitive processes, such as working memory, attentional selection and planning. Many of these processes are affected by reward. For example, one of the most long-standing findings regarding LPFC neurophysiology is that the neurons are spatially tuned and that this tuning carries across intervening delays (Fuster and Alexander, 1971; Kubota and Niki, 1971). However, working memory is also affected by reward: information is stored more precisely when more reward is at stake (Morey et al., 2011). This process likely occurs in LPFC. We trained monkeys to perform a task where they had to saccade to the remembered location of a space cue after a short one second delay (Fig 4A). A cue at the beginning of the trial indicated how much juice the animal would receive for correctly performing the task. We recorded throughout prefrontal cortex and found that only neurons in ventral LPFC encoded both the expected reward as well as information in spatial working memory (Kennerley and Wallis, 2009b; Kennerley and Wallis, 2009a). Furthermore, the precision of the spatial encoding increased as the expected reward increased (Fig 4B). Reward-dependent modulation is not limited to spatial information; reward can also modulate LPFC encoding of high-level information, such as categories (Pan et al., 2008). Furthermore, the modulation may be bidirectional: the contents of working memory may also be able to modulate the reward signal. In dieters exercising self-control regarding choices involving healthy and unhealthy snacks, there is increased activation of LPFC and a concomitant decrease in areas representing value information, as though LPFC is dampening the value signal (Hare et al., 2009).

Figure 4.

A) Monkeys had to saccade to the remembered location of a space cue that could appear at one of 24 locations. One of ten cues appeared at the beginning of the trial, which indicated that one of five juice amounts would be delivered for correctly performing the task. B) Spike density histogram from an LPFC neuron. The different colors indicate the position of the spatial cue, grouping locations together as indicated by the spatial key. The inset plot is a heat map of the neuron’s firing rate when the spatial cue appeared at different locations on the screen. The neuron fires most strongly when the cue appeared in the top left of the screen, but this selectivity was much stronger when then animal expected a large reward for correct performance (bottom plot) compared to a small reward (top plot).

Reward can also serve simply as feedback. For example, consider a child learning an association between a stimulus (e.g. the configuration of a door handle and hinges) and a response (e.g. push or pull). The precise value of the reward is not necessarily relevant and can be very variable (e.g. food in a refrigerator, a toy in a cupboard, an adult saying, “Good job”). Instead, the reward serves as feedback indicating to the child that they selected the correct response. This differs from the actions themselves having different values (e.g. a child choosing between opening two cupboards, one full of toys, the other containing clothes). Neurophysiological evidence shows that LPFC neurons encode the success or failure of a selected response (Watanabe, 1989), consistent with a feedback signal. LPFC neurons integrate such signals across multiple trials (Seo et al., 2007), thereby potentially providing a mechanism by which sequences of actions can be learned (Averbeck and Lee, 2007). In addition, LPFC feedback signals are sustained and influence both future behavior as well as the neuronal encoding of that behavior (Histed et al., 2009).

The above results suggest that LPFC neurons may play an important role in linking behavior with temporally separated feedback, in order to determine correct responses. Although neurons in LPFC encode signals relating to the history of previously performed actions, signals relating to the values of different actions are more common in ACC. For example, in a task where subjects must learn what response to make to a stimulus in order to obtain a specific outcome, LPFC neurons encode the association between the stimulus and response rather than information about the outcome (Matsumoto et al., 2003). In contrast, neurons in ACC encoded which response led to which outcome (Matsumoto et al., 2003). Similarly, in a task where a monkey chose between two different responses each associated with one of three different juices, LPFC neurons only encoded the responses whereas ACC neurons encoded the responses and the outcomes (Luk and Wallis, 2009). Furthermore, the firing rate of ACC neurons to the outcomes correlated with the subject’s preferences between the different juices, consistent with a value signal (Luk and Wallis, 2009). In sum, the activity of LPFC neurons is consistent with using value information as feedback to determine the correct response to make given a particular sensory environment. In contrast, ACC neurons are similar to OFC neurons in that they appear to encode value information directly.

Social

Patients with frontal damage are not just impaired at processing simple rewards, such as food or money. They are also impaired at processing more complex social and emotional stimuli. The vmPFC seems to be particularly important, as well as the immediately adjacent brain areas in ACC and OFC. For example, vmPFC patients show impairments in theory of mind (Stone et al., 1998), empathy (Shamay-Tsoory et al., 2003) and detecting whether faces are expressing emotion (Tsuchida and Fellows, 2012). In addition, vmPFC patients do not show autonomic responses to emotionally charged social stimuli, such as mutilation or pornography, despite normal autonomic responses to simple unconditioned stimuli, such as loud noises (Damasio et al., 1990). Neuroimaging studies consistently show that vmPFC is activated by a wide variety of social rewards, including cooperation (Rilling et al., 2002), love (Bartels and Zeki, 2004) and trust (King-Casas et al., 2005), activate vmPFC. Furthermore, in male monkeys, lesions of area 32 (the area directly dorsal to area 14) disrupt behavioral responses to socially relevant stimuli, such as other aggressive males or female genitalia (Rudebeck et al., 2006a).

The importance of vmPFC in social processing may simply reflect its role in valuation more generally. Primates are inherently social animals and social interactions are rewarding. Indeed, monkeys will forgo juice rewards simply to see pictures of other monkeys (Deaner et al., 2005). In support of this idea, neuroimaging studies have shown that the same region of vmPFC is activated when the correct performance of a task is rewarded with either a juice reward or a social reward, such as an experimenter saying, “Good job” (van den Bos et al., 2007). However, studies using more formal computational models of learning have painted a more complex picture. Behrens and colleagues used reinforcement learning models to examine how subjects integrated information about monetary rewards with information regarding the trustworthiness of advice from a confederate in order to learn probabilistic reward contingencies (Behrens et al., 2008). Dorsal ACC was responsible for tracking the reliability of the monetary rewards, whereas a more ventral region in the cingulate gyrus was responsible for tracking the reliability of the confederate’s advice. Both signals were then combined by vmPFC to derive the overall value of a given choice. Thus, the response of vmPFC to social rewards is not simply a confound of its response to reward in general, but rather reflects a more abstract integration in which both social and non-social rewards are integrated in order to determine the optimal choice.

The response properties of single neurons in these areas also suggest that they play an important role in social processing above and beyond reward evaluation. In monkeys, OFC neurons respond to socially relevant stimuli, such as faces and genitalia (Watson and Platt, 2012). When two monkeys sit facing one another, OFC neurons show different responses to visual cues depending on whether those cues predict that reward will be delivered to one or both animals (Azzi et al., 2012). When monkeys are making decisions about whether reward will be delivered to themselves, to their partner or to neither animal OFC neurons predominantly encode the amount of juice the animal expects to receive, whereas ACC neurons encode the amount of juice the animal expects the partner to receive (Chang et al., 2013). Importantly, these neuronal responses are abolished if the partner is absent and the juice is delivered to an empty primate chair. In other words, the neurons are not simply responding to the sight of juice being delivered, but rather specifically reflect another animal receiving juice.

In sum, there is a good deal of evidence to suggest that the areas of the frontal lobe most critical for value-guided decision-making are also strongly implicated in the control of social behavior. However, the exact contribution remains unclear. A recent study by Williams and colleagues has begun to elucidate what this contribution may be (Haroush and Williams, 2015). Monkeys were trained to perform a prisoner’s dilemma game, in which reward delivery depended on the extent to which they cooperated. They found that there were two populations of neurons in ACC, those that were encoding the animal’s own decision, and those that were trying to predict the action that the other monkey would take. Interestingly, these two factors could be used to predict the expected reward payoff, but this information was not encoded by ACC. This suggests that monitoring one’s own actions in relation to those of others may be one of the primary functions of ACC, rather than predicting the rewards associated with those actions.

Physiological

Prefrontal cortex is anatomically well-positioned to control the physiological and autonomic responses to reward. OFC and ACC strongly connect with the amygdala (Carmichael and Price, 1995a) and hypothalamus (Ongur et al., 1998), respectively. During the 1990s, a series of experiments by Damasio and colleagues led to the development of the ‘somatic marker hypothesis’, which stated that the decision-making impairments experienced by patients with damage to OFC and ACC resulted from a failure to activate the autonomic state that would help signal the optimal choice (Damasio, 1996). In other words, the patients were lacking the ‘gut feel’ that often accompanies decisions (Kahneman, 2011). This theory was largely based on a study where patients with OFC damage failed to show skin conductance responses prior to making risky choices on a gambling task (Bechara et al., 1996). The theory has since been discredited (Dunn et al., 2006). The design of the gambling task confounded the ability to track reward contingencies with the ability to flexibly switch behavior in response to changing reward contingencies, the latter of which is known to depend on the integrity of OFC (Dias et al., 1996; Walton et al., 2010). When this confound was removed from the gambling task, patients with OFC damage were no longer impaired (Fellows and Farah, 2005).

Despite the failure to directly link autonomic changes to decision-making, there is nevertheless convincing evidence that prefrontal cortex is involved in autonomic processing. For example, increased ACC activity is observed when people are asked to monitor their own heartbeats (Critchley et al., 2004) or when their esophagus or large bowel are directly stimulated (Hobday et al., 2001). Electrical stimulation of ACC in animals causes changes in heart rate and blood pressure (Burns and Wyss, 1985; Kaada et al., 1949). In addition, ACC activity was observed in a neuroimaging study where heart rate was increased by having the subject engage in a physically or cognitively effortful task (Critchley et al., 2003). Although this might be explained by the role of ACC in effort-based behavior, as described above, the BOLD response appeared to correlate better with heart rate than the amount of effort exerted. Furthermore, the same study tested three ACC patients who were found to be unimpaired at performing the tasks, but showed blunted cardiovascular responses.

Animal studies have also implicated OFC in autonomic control. In an appetitive Pavlovian task, animals with OFC lesions showed a normal pattern of elevated autonomic arousal (as measured by heart rate and blood pressure) when presented with a reward-predictive cue (Reekie et al., 2008). However, during either extinction or reversal, when the cue no longer predicted reward, animals with OFC lesions continued to show autonomic responses for far longer than control animals. These results are consistent with some studies in humans. Electroencephalography measures of arousal indicate that patients with OFC damage show enhanced responses to emotional cues (Rule et al., 2002). Thus, rather than showing reduced autonomic responses, as predicted by the somatic marker hypothesis, OFC damage in both animals and humans produces enhanced responses to emotional stimuli consistent with the role for OFC in the top-down control of autonomic processes.

Future research

Responses to reward can be observed in every frontal lobe area and this reflects the ubiquity of reward in cognitive and behavioral processes. Although reward coding is omnipresent, it appears to be involved in different computational processes in different prefrontal regions. The role of reward in OFC is linked to the processes of decision-making and evaluation, in ACC it guides learning and action selection and in LPFC it helps to control attentional and executive processes. While this serves as a useful roadmap for the role of reward in the frontal lobe, it is important to avoid the oversimplification of a box and arrow model. Current research aims to more precisely define the contribution of reward to these processes at a computational and algorithmic level. Drawing from the field of artificial intelligence, advances have been made in linking reward-related responses in prefrontal cortex to the computational algorithms that underlie reinforcement learning (Kennerley et al., 2011; Alexander and Brown, 2011). Extensions of these models, such as hierarchical reinforcement learning (Botvinick et al., 2009b), are being used to develop computational accounts of how prefrontal cortex establishes a model of the world that can be used to guide learning and behavior (Badre and Frank, 2012; Frank and Badre, 2012; Wilson et al., 2014). These same models are increasingly providing a more sophisticated understanding of neuropsychiatric disorders (Huys et al., 2016), which will guide the development of interventions over the coming decade.