Orbitofrontal Cortex and the Computation of Subjective Value: Consolidated Concepts and New Perspectives

Abstract

Remarkable progress has been made in recent years towards understanding the functions of the orbitofrontal cortex (OFC). The finding that neurons in this area encode the subjective value monkeys assign to different goods while choosing has been confirmed and extended by numerous studies using both primate neurophysiology and human imaging. Moreover, new lesion studies demonstrated that subjective values computed in the OFC are causally and specifically related to choice behavior. Importantly, values in the OFC are attached to goods, not to actions or to spatial locations. Furthermore, subjective values appear to be computed in this area even if the situation does not require a choice. In the light of this growing body of work, we propose that the computation of good identities and subjective values in an abstract representation is the primary function of the OFC. In this view, OFC neurons compute the subjective value of a good whenever that good is behaviorally relevant.

In the paper published in the proceedings of the first conference on orbitofrontal cortex (OFC) held at the New York Academy of Sciences, one of us reviewed evidence suggesting that one function of the OFC is to compute subjective values¹. The main arguments presented then can be roughly summarized as follows. First, an economic choice study by Padoa-Schioppa and Assad, in which monkeys chose between different juices offered in variable amounts, had shown that neurons in the central OFC (in and around area 13m) encode the subjective values animals assigned to the different goods². In that study, subjective value was defined at the behavioral level from the trade-off between the two dimensions (or determinants) on which offers varied, namely juice type and juice amount. At the neuronal level, three types of neurons were found: offer value cells encoded the value on only one of the two juices; chosen value cells encoded the value of the chosen juice; taste cells encoded the identity of chosen juice in a binary way. These three variables were found to explain the activity of the neuronal population in OFC significantly better than numerous other variables tested in that study, including other value, value difference, total value, etc. Second, confirming previous observations^{3, 4}, that study had also shown that the neuronal representation of subjective value in OFC did not depend on the visuo-motor contingencies of choice. For example, the activity of neurons encoding the offer value of apple juice did not depend on whether apple juice was presented on the left or on the right of the fixation point. Similarly, the activity of neurons encoding the chosen value did not depend on whether obtaining the chosen juice required a saccade to the left or to the right. Values were thus attached to particular goods (juices), not to particular spatial locations or to particular actions². Third, research work in multiple species had shown that OFC lesions induce specific deficits in choice behavior^5–7. Based on results from single cell recordings and lesion studies, our review paper thus proposed that economic choices might be based on values computed in the OFC. The further hypothesis that choices actually take place within this neuronal representation has been recently formalized in a comprehensive model⁸.

Our current purpose is to describe how the understanding vis-à-vis the role of OFC in the computation of subjective value has evolved in the past four years. Indeed, remarkable progress has been made in this relatively short time, thanks to work conducted in many laboratories using various experimental approaches. In essence, a wealth of new data supports the proposal that OFC encodes subjective values and that these neuronal signals underlie economic choices. In particular, several studies generalized the original findings² to choices between goods that vary on dimensions such as probability, time delay and physical effort. Other studies characterized how the encoding of value in this brain region depends (or does not depend) on the behavioral context of choice. Finally, new lesion studies provided more stringent evidence that OFC is specifically and causally linked to valuation and economic choice. The evidence accumulated in recent years thus motivates the proposal that the computation of subjective values in an abstract representation is not just one function of OFC, but rather the primary function of OFC.

Choices between goods that vary on multiple dimensions

The value an individual assigns to a good – for example a scoop of chocolate ice cream – is intrinsically subjective. Consequently, any operational measure of value requires a choice and thus a trade-off between different goods⁸. For example, if an individual in an ice cream shop is indifferent between one scoop of chocolate and two scoops of vanilla, one can say that these two goods have equal subjective value. Thus assuming that value functions are linear, one can measure the subjective value of given quantities of either ice cream on the same scale (e.g., in units of vanilla scoops). Importantly, the concept of “good” is rather general and goods can vary on many dimensions. In the ice cream example discussed here and in our original study², offered goods varied for the commodity (i.e., the flavor) and the quantity. However, goods can also vary on dimensions such as the probability of delivery, the time delay, the ambiguity associated with the offer, the physical effort necessary to obtain the good, other kinds of cost, etc. By definition, subjective values integrate all these dimensions and thus provide the basis to make a decision⁸.

The proposal that neurons in the OFC encode subjective value was originally based on data from trade-offs between juice type and juice quantity². In the past few years, important work has brought support to this proposal by examining choices that involve several other determinants. Roesch and Olson had previously found that neurons in the primate OFC are modulated both by the quantity of juice delivered to the animal and by the delay intervening before juice delivery. Consistent with a unitary representation of value, the two modulations had opposite sign⁴. Along similar lines, Wallis and colleagues found that the activity of neurons in the OFC (and other frontal areas) depends not only on the quantity of juice delivered to a monkey, but also on the probability and on the physical effort exerted by the animal to obtain the juice^{9, 10}. More recently, these same authors recorded from the central OFC using a choice task. They found that neurons in this area encode the subjective value defined by the trade-off between juice quantity and delay, and between juice quantity and physical effort¹¹. Similarly, ongoing work in our laboratory examines choices between goods that vary on three dimensions – juice type, quantity and probability. Preliminary results indicate that neurons in OFC encode subjective values as defined by the integration of all these determinants¹². Another striking example of integration came from a study by Watson and Platt. In their experiment, monkeys chose between drinking a given amount of juice and drinking a different amount of juice while watching the image of a conspecific. Male macaques were willing to forgo some amount of juice for the opportunity to observe female perinea. Importantly, the activity of neurons in OFC encoded subjective value as defined by this behavioral trade-off¹³. Another interesting study showed that neurons in the OFC encode both positive and negative values¹⁴.

Choices between goods that vary on multiple dimensions have also been examined in numerous imaging studies in humans. In particular different experiments examined trade-offs between money amount and food type^{15, 16}, money amount and quantity of non-food consumables¹⁷, money amount and probability^{18, 19}, money amount and ambiguity^{18, 20}, money amount and time delay^{21, 22}, probability and time delay²³, probability and number of electric shocks²⁴, and (in hypothetical moral choices) number of lives at risk and likelihood of successful rescue²⁵. In essence, all these studies found neural activity associated with subjective values in the medial or in the central OFC – an impressive consistency across laboratories, experimental procedures and value determinants.

Although results from primate neurophysiology and human imaging are in many ways remarkably consistent, it is worth noting that some degree of discrepancy actually exists. In particular, imaging studies often found value-related activity more medially compared to single cell studies. This discrepancy is particularly puzzling because from an anatomical standpoint orbital regions and medial regions are clearly distinct, they are interconnected with different sets of cortical and subcortical areas, and they are scarcely interconnected with each other²⁶. The origins of this discrepancy remain unclear²⁷. Possibly relevant elements include the fact that anatomical regions in the two species might not correspond perfectly; the fact that single cell recordings in primates generally take place after a long training process; the fact that BOLD signals from central OFC often suffer from attenuation and/or distortion²⁸ (hence, negative results from central OFC should sometimes be taken with a grain of salt); and the fact that in the medial OFC neural activity related to value is potentially confounded with neural activity related to autonomic responses^29–33. This said, several human imaging studies did actually find the most prominent choice- and value-related activity in central OFC^34–37. Thus although more work is necessary to understand possibly important differences across species and procedures, current evidence from human imaging overall supports the notion that subjective values are computed in the OFC.

With respect to rodents, new results confirmed that if subjective values are indeed encoded in the rodent OFC, their representation does not integrate multiple determinants^38–42. For example, food quantity and time delay are encoded by different groups of cells³⁸. Moreover, neurons in the rodent OFC are spatially selective^{38, 43}. Physiological differences between rodents and primates might simply reflect a poor anatomical homology⁴⁴. However, the diversity of results might at least in part reflect differences in behavioral paradigm – a gap that should stimulate further research in the coming years.

Effects of lesions

The disruptive effects of OFC lesions on choice behavior had long been documented^{45, 46}. In particular, using the reinforcer devaluation paradigm, it had been found that the effects of selective satiation on choices are greatly diminished following OFC ablation^5–7. Nonetheless, critical new information has been gathered in the past few years thanks to a series of studies conducted in several laboratories. Specifically, deficits in value-based decision-making (often tested using the reinforcer devaluation paradigm) were consistently found after lesions placed in central OFC (areas 11/13) but not after lesions placed in other brain regions including the medial OFC (area 14)⁴⁷, the dorso-lateral PFC^{48, 49}, the ventro-lateral PFC^{49, 50}, the anterior cingulate cortex⁴⁹ and the hippocampus^{51, 52}. (Amygdala lesions did affect value-based decisions^52–54.) Conversely, lesions to the central OFC failed to induce any deficit in a variety of reward-related and cognitive tasks including basic food preference⁵², satiation^{54, 55}, rule- or strategy-based decision-making^{49, 50}, serial object reversal learning^{47, 56}, response inhibition⁴⁷, discrimination learning⁵², and working memory maintenance⁴⁹. Taken collectively, these studies reveal a high degree of specificity. On the one hand, central OFC (with the amygdala) appears to be the only brain region indispensible for the computation of subjective values. On the other hand, computing and/or comparing subjective values appears to be the only mental function clearly disrupted by lesions of the central OFC.

Further support for a causal role of OFC in economic choice came from a recent study in which Camille et al examined choices between different foods in humans. In economic theory, it is known that choices can be described by a value function if and only if preferences are transitive. In other words, if given any 3 goods A, B and C an individual chooses A over B, B over C and C over A, her choices cannot be based on a functional representation of value. Extending earlier work⁵⁷, Camille et al found that patients with OFC lesions choosing between different combinations of foods violated preference transitivity significantly more often than matched controls⁵⁸. (Since lesions were typically large, the study did not disambiguate between subregions of OFC.) Compared with data from the reinforcer devaluation experiments, the importance of this result is twofold. First, it confirms a causal link between values computed in OFC and choice behavior using an independent behavioral paradigm. Second, it makes a more categorical point: OFC lesions do not just alter preferences in favor or against one particular determinant of value, or make it more difficult to integrate internal and external determinants. Rather, OFC lesions disrupt the very ability to compute and/or to compare any subjective value.

In summary, lesion studies in recent years provide new and more stringent evidence suggesting that the relationship between (central) OFC and the computation of subjective value is specific – in both senses – and causally related to economic choice behavior.

Menu invariance and gain adaptation

The results discussed so far buttress the hypothesis that subjective values are computed in the OFC. Another recent set of studies examined in more detail the properties of this neuronal representation and, specifically, how the encoding of subjective value depends on the behavioral context of choice. Two main results have been reported.

First, a study by Padoa-Schioppa and Assad found that OFC neurons encoding the value of one particular good (e.g., apple juice) do so independently of the good offered as an alternative (e.g., grape juice or water) – a property called menu invariance⁵⁹. The significance of this result follows from the close link between menu invariance and preference transitivity. To appreciate this link, consider three goods A, B and C and one individual who chooses good A over B, B over C, and C over A – and thus violates transitivity. If one describes them in terms of subjective value, these choices can be explained only if values depend on the menu (e.g., if the value assigned to good A is high when the alternative is good B and low when the alternative is good C). In other words, choices that violate transitivity can be explained only if values depend on the menu. Equivalently, choices based on a menu invariant representation of value are necessarily transitive. Thus menu invariance observed in the OFC may explain the fact that human and animal choices most typically satisfy preference transitivity^60–62.

Second, values computed in different behavioral contexts can vary by orders of magnitude. For example, the same individual might choose some times between different ice cream flavors and other times between different cars for sale. In a computational sense, this large variability poses a serious challenge – an issue highlighted by the fact that OFC neurons encode values in a linear way². In principle, the valuation system could meet this computational challenge through mechanisms of gain adaptation. Two recent studies, one in our laboratory and one by Kobayashi and colleagues, indeed demonstrated the existence of such mechanisms^{63, 64}. In essence, both studies found that the encoding of value undergoes range adaptation such that a given range of firing rates represents different ranges of values in different behavioral contexts. Range adaptation was observed both in offer value and chosen value cells, and both in positive and negative encoding cells (i.e., neurons for which the activity increased or decreased as a function of value)⁶³. Partial adaptation occurred on a trial-by-trial basis. However, full adaptation seemed to require at least 10–20 trials⁶⁴. Finally, for offer value cells, the activity range did not depend on the preference status of the encoded juice⁶³. Interestingly, these findings shed new light on earlier work by Tremblay and Schultz³ and others^{65, 66}. Their observations appear best interpreted in terms of a range-adapting representation of subjective values (a cardinal measure) as opposed to a representation of relative preferences (an ordinal relationship).

These recent studies significantly deepen our understanding of how subjective values are represented in the OFC and provide key elements in the effort to explain choice behavior in terms of neurobiological mechanisms. In essence, the encoding of subjective value in the OFC is at the same time menu invariant (and thus instantiative of preference transitivity) and range adapting (and thus capable of supporting choices on different value scales). Interestingly, work in behavioral economics shows that under carefully designed conditions it is possible to elicit choices that systematically violate transitivity^{67, 68}. Thus it will be interesting to investigate in future work the limits under which menu invariance holds true in the OFC.

Generalizations and new perspectives

To summarize the results reviewed in the previous sections, a wealth of data indicates that neurons in the OFC encode the identity and the subjective value of different goods, that this neuronal representation is abstract (independent of the spatial contingencies of the task), and that during economic choice subjective values computed in the OFC are causally related to the decision. As we often emphasized⁸, proving that a neuron or a functional imaging signal actually encodes subjective value ultimately requires a choice task in which subjects trade-off between different determinants. However, this fact does not mean that individuals – and neurons in the OFC – compute subjective values only while engaging in economic choices. In fact, there is ample evidence to the contrary. Behaviorally, subjective values can guide a variety of cognitive processes in addition to choice, including perceptual attention, affective behavior, associative learning, etc.⁸ Neuronally, numerous studies that did not include a choice in the sense defined here found in the primate OFC neuronal responses that are naturally interpreted in terms of subjective value^{3, 4, 9, 14, 69, 70}. Hence, it is reasonable to wonder whether any experimental result from the OFC eludes such an interpretation. As a generalization of our previous argument, we thus propose that the computation of good identities and subjective values in an abstract representation is not just a function of the OFC, but rather the primary function of the OFC. In this view, neurons in the OFC compute the subjective value of a good whenever that good is behaviorally relevant, whether or not the situation requires a choice.

The ramifications of this proposal remain to be examined and counter-examples might be found in future work. As an initial test, we discuss in the remaining of this section the results of two recent studies that revisited previously addressed issues – spatial selectivity and hypothetical outcomes.

In the study by Tsujimoto and colleagues⁷¹, monkeys were presented in each trial with a visual instruction, after which they executed a saccade towards one of two targets. The two targets were always located on the left and on the right of the fixation point; in any given trial, a juice reward was associated with only one of the two targets (hence, the task did not involve a choice in the sense discussed above). In each trial, the correct target was identified by applying a rule – either “stay” or “shift”, depending on the instruction – to the target selected in the previous trial. Thus to receive a juice reward, the animal had to remember the previous target and to apply the instructed rule. Among other results, the authors reported that the activity of neurons in the OFC recorded at the end of the trial varied depending on the target selected in that trial⁷¹. In principle, such differential activation could be due to spatial selectivity and, more specifically, to an underlying representation of motor responses or spatial locations in Euclidean space – an interpretation that would be at odds with other reports^{2, 4, 72, 73}. However, as emphasized by the authors, this signal is unlikely to represent a motor response per se. Likewise, this signal seems unlikely to represent a spatial location per se. Instead, this signal may have represented the identity of the chosen target in an abstract or symbolic representation. Indeed, it can be noted that in this study targets were defined uniquely by their spatial location. Thus a neuron encoding the identity of one of the two targets symbolically (as opposed to the spatial location of the target per se) would have presented the same differential activation found in the study. In this light, the (categorical) chosen target signal recorded by Tsujimoto can be viewed as akin to the taste (chosen juice) signal recorded in economic choice tasks^{2, 59}. Both signals encoded the identity of the chosen option (and both signals were most prominent at the end of the trial).^a Notably, the identity of the chosen target was behaviorally relevant in the Tsujimoto study because it guided saccade selection in the following trials.

In the study by Abe and Lee⁷⁴, monkeys executed saccades towards one of three possible targets. The task was designed as a rock-scissor-paper game in which the three targets represented the three options (R, S and P) and a computer program played as the animal’s opponent. On win trials, the three options delivered different amounts of juice. Thus animals selected R, S and P in different proportions. At the end of each trial, the animal was delivered the amount of juice won in that trial. In addition, the animal was visually shown the juice amounts it would have received for the two non-selected targets (hypothetical outcomes). The spatial locations of options R, S and P remained fixed within a trial block and changed unpredictably at the end of a trial block. The task design induced two learning processes. First, at the beginning of each new trial block, the animal learned the new spatial configuration of options R, S, and P. Second, at steady state within a trial block, choices on any given trial were to some extent biased by the outcome of the previous trial. For example, following a loss trial, animals were more likely to select the target that would have lead to a win. In other words, both actual and hypothetical outcomes guided learning. The authors recorded from OFC (and lateral PFC). Their analyses highlighted three variables: the actual outcome, the chosen location and the hypothetical outcome (in tie and loss trials). Prima facie, OFC neurons encoding the chosen location seems at odds with numerous studies that did not find any spatial selectivity in this region^{2, 4, 72, 73}. Furthermore, OFC neurons encoding the hypothetical outcome seems at odds with the fact that during simple economic choices vanishingly few neurons encoded the variable other value (analogous to the hypothetical outcome)^{2, 59}. However, considerations similar to those discussed for the Tsujimoto study apply here. Indeed within each trial block, options R, S and P were only defined by their spatial location. Thus a neuron encoding not the spatial location of options R, S and P per se, but rather their identity in any abstract or symbolic representation would have appeared as spatially selective. Moreover, both the spatial location and the value of the hypothetical outcome guided a learning process and thus were behaviorally relevant. Thus the results of this study are in fact consistent with an abstract representation of option identities and their subjective values.

In conclusion, the results of both studies can be accounted for by an abstract representation of goods or options and subjective values in the OFC.

Conclusions

Remarkable progress in understanding the functions of the OFC has been made in the past few years thanks to an impressive convergence of results from animal neurophysiology, human imaging and lesion studies. This large body of work suggests that the primary function of this brain region is the computation of good identities and subjective values in an abstract representation. Numerous important questions remain open, including addressing possible differences between species, understanding the neuronal mechanisms that result in choice fallacies, and many other. Yet, the successes achieved in recent times promise new breakthroughs in the years to come.