Heterogeneous reward signals in prefrontal cortex

Summary

Neurons encode upcoming rewards throughout frontal cortex. Recent papers have helped to determine that these signals play different roles in different frontal regions. Neurons in orbitofrontal cortex (PFo) appear to be responsible for calculating the specific value of an expected reward, information that can help efficiently guide decision-making. Similar signals are also present in the cingulate sulcus (PFcs). In contrast, reward signals in lateral prefrontal cortex (PFl) are consistent with a role in using reward to guide other cognitive processes, such as the allocation of attentional resources and using value information to guide learning other relationships in the environment such as arbitrary stimulus-response mappings. A remaining issue for future work is to specify the precise roles of PFo and PFcs. These two areas show very different patterns of connectivity with other brain areas, and it is currently unclear how this effects their contribution to decision-making.

Introduction

During the 1980s, two distinct lines of research highlighted the importance of PFo for processing reward information and using such information to guide behavior. First, PFo neurons encoded whether a sensory stimulus was predictive of a rewarding outcome [1]. Second, patients with PFo damage had specific impairments in their ability to make decisions in everyday life [2]. These findings elegantly complemented one another. PFo patients would be forced to make decisions more or less at random if they lacked the signals as to the expected rewards that might result from different possible choices. Despite this initial consilience, neuropsychological and neurophysiological findings subsequently diverged. Laboratory tests confirmed that frontal patients had very specific deficits in value-based decision-making (e.g. which color do you prefer?), and not other forms of decision-making such as perceptual decision-making (e.g. which color is more blue?) [3]. Furthermore, such deficits only occur following damage to PFo, not PFl [3], consistent with the notion that PFo has a specialized role in this process. However, reward-predictive neurons, far from being specific to PFo, were subsequently found in every frontal area that neurophysiologists cared to record from, including PFl [4], PFcs [5], and motor areas such as the premotor and primary motor cortex [6].

These findings raised two important questions. First, if the presence of reward-predictive neurons does not indicate a necessary role in decision-making, what is it about the neuronal encoding in PFo that makes it so important to this process? Second, what function does reward-predictive encoding serve in other frontal areas? Rewards have many aspects. They have sensory (e.g. flavor, intensity) and emotional components (they make us happy), they satisfy motivational drives, they have a value (some rewards are more preferable than others are), they are reinforcing (we repeat behavior that produced a reward), and they can guide the allocation of cognitive and attentional resources. How do reward-predictive neurons in different frontal areas relate to these different functional properties of reward?

Reward signals underlying the calculation of value

How does one determine the value of one good relative to another? One approach is through barter: if I want three chickens for my pig then we can say that I find pigs three times more valuable than chickens. By bartering with a monkey using different types of juice, researchers showed that many PFo neurons encode the value of one juice reward relative to another [7,8]. One problem facing such a signal is that there is a huge range in the value of potential goods. Consider browsing the real estate section of the newspaper in a café, when the server interrupts to take our order. We are effortlessly able to switch from valuations in the hundreds of thousands of dollars to those for just a few dollars. Yet the PFo neurons representing this value range have a limited firing rate range (typically <50 Hz). A recent study shows that PFo neurons solve this problem by dynamically adjusting their firing rate range to the range of values on offer [9] (Fig. 1). An interesting question is whether and how top-down influences, such as the context one is in, might be able to accomplish such an adjustment. PFo neurons also encode a wide range of other variables necessary for decision-making [10], including positive and negative expected outcomes [11,12], the costs involved in acquiring an outcome [13,14], the probability that one's choice will be successful [14] and one's confidence in the choice [15].

Figure 1.

a) The figure illustrates the hypothetical relationship between firing rate and value if the neuron adapts its firing rate to match the value of the range under consideration (ΔV). Specifically, as the value range increases, the slope of the relationship between firing rate and value decreases. b) The raw data taken from the entire population of recorded neurons from PFo. Each line is the average firing rate of a specific neuron for a given value. Different colors indicate the different value ranges under consideration. c) Average of the neuronal responses in b). The data clearly match the predictions of the hypothetical model shown in a). As the value range under consideration increases, the slope of the relationship between value and firing rate decreases, indicative of neuronal adaptation. Reprinted with permission from [9].

A recent neuropsychological study in rats provided greater insight into the nature of the PFo value signal by using the phenomenon of transreinforcer blocking [16]. A subject only learns that a cue predicts a reward if the reward's occurrence is surprising. Thus, learning about a cue can be blocked if a second cue is present that fully predicts the delivery of a reward. However, supposing the two cues predict different rewards (e.g. apple versus orange). Now the second cue only partially blocks learning to the first cue: it blocks learning to those factors that the two rewards share (general reward information) but leaves intact learning to the factors that differentiate the two rewards (specific reward information). It is only this latter information that is sensitive to PFo lesions. But what exactly is this specific reward information? One explanation is that PFo encodes the sensory features (e.g. appearance) of the specific reward that the cue predicts [17]. A problem with this interpretation is that it seems to draw a somewhat arbitrary distinction between reward-predictive cues and the sensory properties of a reward. For example, is the sight of an apple a predictive cue of the rewarding properties of the apple (e.g. its sweet taste) or a sensory feature of the apple? An alternative explanation is that there are multiple value scales in the brain, a value system that signals the general affective properties of apples (i.e. they are positive things that will make you happy) and a value system that calculates the specific value of the apples (i.e. 2 apples are worth 5 oranges) [16]. This specific value system is the one instantiated by PFo. The notion of distinct value systems is not novel. For example, previous investigators have suggested distinct value systems related to liking something (hedonic value) and wanting something (incentive value) [18].

In addition to PFo, dysfunction of PFcs has also been associated with impairments in using reward information to guide decision-making [19,20]. Neurons in PFcs also encode many of the same factors as PFo neurons including positive and negative outcomes [21-23], probability of success [14], and reward preference [24]. We trained monkeys to make choices between different behavioral outcomes that varied in terms of either payoff (volume of juice), cost (number of lever presses necessary to earn the juice) or probability (probability that the juice would be delivered) [14]. We recorded from PFo, PFcs and PFl and found that neurons encoded value across the different decision variables in diverse ways. For example, some neurons encoded the value of just a single decision variable, others encoded the value of choices for two of the decisions but not the third, while still others encoded value across all three decision variables (Fig. 2). The prevalence of neurons encoding a single decision variable was similar in all areas, but neurons encoding two or three decision variables were most prevalent in PFcs and PFo. Thus, PFcs and PFo neurons encode a multiplexed representation, which may allow the integration of the individual components of a decision and underlie their critical contribution to decision-making. Consistent with this notion, patients with PFo damage are impaired in integrating multiple attributes of a choice outcome in order to make a decision (e.g. choosing between a small apartment in a low crime neighborhood versus a larger apartment in a higher crime neighborhood) [25].

Figure 2.

Spike density histograms illustrating the activity recorded from single neurons under three different types of value manipulation (probability, payoff or effort). The vertical lines indicate the onset of the pictures indicating the value of the choice (left) and the time at which the animal was able to make his choice (right). The different colored lines indicate the value of the choice under consideration. Four neurons are shown: a) encodes the value of probability choices alone, b) encodes payoff and cost but not probability, c) encodes probability and payoff but not cost and d) encodes the value of all three decision variables. Across the population, there was an approximately equal number of neurons showed a positive relationship between firing rate and value (d) as showed a negative relationship (a, b, and c).

Using value signals for other cognitive processes

Value information can potentially serve other functions beyond allowing efficient action selection. More broadly, value information allows prioritization, which is necessary whenever one has limited resources. Action selection is just one example of system with capacity constraints - we can only pursue one course of action at a time. However, cognitive processes also have capacity constraints, most notably the process of working memory. Neurons in PFl encode information in working memory [26,27]. A common finding is that performance of a working memory task improves when a subject expects a larger reward, which might arise if value information can modulate the fidelity of information stored in working memory. To examine this, we trained monkeys to maintain spatial locations in working memory [28,29]. At the beginning of each trial, a picture informed the subject how much juice they would receive for correct performance of the task. Figure 3 illustrates an example of reward modulating the spatial selectivity of a PFl neuron. We recorded from PFl, PFo and PFcs, but we only saw such modulation in PFl. Reward-dependent modulation is not limited to spatial information; reward can also modulate PFl encoding of high-level information, such as categories [30]. Furthermore, the modulation may be bidirectional: the contents of working memory may also be able to modulate the reward signal. In dieters exercising self-control regarding choices involving healthy and unhealthy snacks, there is increased activation of PFl and a concomitant decrease in areas representing value information, as though PFl is dampening the value signal [31].

Figure 3.

Spike density histograms illustrating how the size of an expected reward can modulate spatial information held in working memory. The graphs illustrate neuronal activity as animals remember different locations on a computer screen under the expectancy of receiving either a small or a large reward for correct performance. The grey bar indicates the presentation of the mnemonic spatial cue. To enable clear visualization, the spatial data is collapsed into four groups consisting of 6 of the 24 possible spatial locations tested. The inset indicates the mean standardized firing rate of the neuron across the 24 spatial locations. a) When the subject expected a small reward, the neuron showed moderate spatial selectivity, which consisted of an increase in firing rate when the subject was remembering locations in the top right of the screen. When the subject expected a large reward for correct performance, spatial selectivity dramatically increased with a high firing rate for locations in the top right of the screen and a low firing rate for locations in the bottom left. b) A neuron that showed little spatial selectivity when the subject expected a small reward, but a dramatic increase in spatial selectivity when the subject expected a large reward.

Value information can also serve simply as feedback. For example, consider a child learning an association between a stimulus (e.g. the configuration of a door handle and hinges) and a response (e.g. push or pull). The precise value of the reward is not necessarily relevant and can be very variable (e.g. food in a refrigerator, a toy in a cupboard, an adult saying, “Good job”). Instead, the reward serves as feedback indicating to the child that they selected the correct response. This differs from the actions themselves having different values (e.g. a child choosing between opening two cupboards, one full of toys, the other containing clothes).

Neurophysiological evidence shows that PFl neurons encode the success or failure of a selected response [32], consistent with a feedback signal. PFl neurons integrate such signals across multiple trials [33], thereby potentially providing a mechanism by which sequences of actions can be learned [34]. In addition, PFl feedback signals are sustained and influence both future behavior as well as the neuronal encoding of that behavior [35]. Taken together, these results suggest that PFl neurons may play an important role in linking behavior with temporally separated feedback, in order to determine correct responses. Although neurons in PFl encode signals relating to the history of previously performed actions, signals relating to the values of different actions are more common in PFcs. For example, in a task where subjects must learn what response to make to a stimulus in order to obtain a specific outcome, PFl neurons encode the association between the stimulus and response rather than information about the outcome [5]. In contrast, neurons in PFcs encoded which response led to which outcome [5]. Similarly, in a task where a monkey chose between two different responses each associated with one of three different juices, PFl neurons only encoded the responses whereas PFcs neurons encoded the responses and the outcomes [24]. Furthermore, the firing rate of PFcs neurons to the outcomes correlated with the subject's preferences between the different juices, consistent with a value signal [24]. In sum, the activity of PFl neurons is consistent with using value information as feedback to determine the correct response to make given a particular sensory environment. In contrast, PFcs neurons are similar to PFo neurons in that they appear to encode value information directly.

Functional differences between PFo and PFcs

If PFo and PFcs are both capable of encoding value signals and damage to either area impairs decision-making, a clear question is what the two areas are doing differently from one another. A comparison of PFcs and PFo anatomy suggests potential answers to this question. PFcs and PFo both connect with areas responsible for processing rewards, such as the amygdala and hypothalamus [36,37], and receive dopaminergic inputs [38] that also carry reward signals [39,40]. PFcs strongly connects with cingulate motor areas and limbic structures but has few direct connections with sensory cortex [41-43]. In contrast, PFo only weakly connects with motor areas, but heavily connects with all sensory areas, including visual, olfactory and gustatory cortices [41,42,44]. Consistent with this anatomy, PFcs neurons tend to encode rewards and the action that led to the reward in monkeys [5,45] and humans [46], whereas PFo neurons encode the value of sensory stimuli with little encoding of motor responses [7,8,14,29,47]. PFcs may have an additional role in monitoring the outcome of the selected action to guide behavioral adjustment, as PFcs activity is sensitive to errors or deviations in reward expectancy [48], reflects the history of reward associated with different actions [19,49], tracks volatility in reward rate of different actions [50], and encodes “fictive” rewards, counterfactual rewards that might have been obtained had the subject selected an alternate course of action [51]. Thus, there may be a division of labor between PFo and PFcs with regard to choice evaluation, with PFo calculating the value of possible outcomes, while PFcs calculates the value of the action producing the outcome and monitors the success of behavioral outcomes over time to guide adaptive behavior.

Recent neuropsychological studies support this dissociation. Lesions of PFo impaired probabilistic matching tasks when the discriminanda were visual pictures, but not when the discriminanda were motor responses, while the opposite pattern occurred following lesions of PFcs [19,52]. In rats, lesions to the corresponding PFcs region disrupts choices when rats are faced with options differing in their payoff and the effort necessary to obtain that payoff [53], but such lesions do not effect other types of cost-benefit decisions (e.g. delay-based decisions), that do not require the evaluation of an action [54]. In contrast, rats with PFo lesions show impaired delay-based decision-making, but intact effort-based decision-making [54].

Interpretational issues

Adaptive decision-making depends on a number of different computations, including representing the value of one's current internal (motivational) state, the value of possible (chosen and unchosen) alternatives, and the generation of a prediction error signal when outcomes deviate from expectations [55]. Importantly, although these value representations reflect different computations, they can be highly correlated with one another [50,56]. Moreover, given choices can be made on the order of a few hundreds milliseconds, these processes are likely to occur in close temporal proximity and thus be difficult to differentiate. Finally, it is difficult to infer functional specialization by comparing results across studies that use different tasks, subjects and statistical analyses. To address these issues several studies have recorded from different populations of neurons in the same subjects, demonstrating the degree to which different frontal areas are recruited for different value computations [6,12,14,21,57], as well as demonstrating a hierarchy in the order in which reward is encoded, first in PFo and then later in PFl and PFcs [29,47].

A final issue is that it is not clear how these ideas relate to the functional organization of the human frontal cortex. The human frontal area that is most consistently activated across a broad range of value-based decision-making studies is the ventral portion of the medial surface of the frontal lobe (PFvm) [31,58-61]. This area most likely corresponds to either area 32 or area 10 (Fig 4). Primate neurophysiologists have done little recording from these areas due to their relative inaccessibility (although see [62]).

Figure 4.

Cytoarchitectonic maps of the ventral and medial surface of the human and macaque frontal cortex. Granular cortex (containing a layer 4) appear in blue, agranular areas in green and allocortical areas in yellow. Abbreviations: a, agranular; AON, anterior olfactory ‘nucleus’; I, insula; Pir, piriform cortex; l, lateral; m, medial; o, orbital; r, rostral; c, caudal; i, inferior; p, posterior; s, sulcal; v, ventral. (Note that the letter ‘a’ has two meanings: in Ia it means agranular; in 13a it distinguishes that area from area 13b.) Architectonics are taken from [63,64]. Reprinted with permission from [65].

Conclusion

Over the last decade, it has become clear that reward signals are ubiquitous in the frontal lobe. This should not be surprising given that a central goal of behavior is to obtain reward. It is also clear that reward information is useful for many different functions, and so reward signals in different frontal regions may be serving very different roles. A thesis compatible with the most recent research is that neurons in PFcs and PFo are directly involved in calculating value, with PFo calculating the value of possible outcomes, while PFcs calculates the value of the action to guide future behavior. In contrast, PFl may use value information to modulate the contents of working memory and as a feedback signal to learn arbitrary relationships such as those that underlie stimulus-response mappings.