Is the core function of orbitofrontal cortex to signal values or make predictions?

Abstract

One dominant hypothesis about the function of the orbitofrontal cortex (OFC) is that the OFC signals the subjective values of possible outcomes to other brain areas for learning and decision making. This popular view generally neglects the fact that OFC is not necessary for simple value-based behavior (i.e., when values have been directly experienced). An alternative, emerging view suggests that OFC plays a more general role in representing structural information about the task or environment, derived from prior experience, and relevant to predicting behavioral outcomes, such as value. From this perspective, value signaling is simply one derivative of the core underlying function of OFC. New data in favor of both views have been accumulating rapidly. Here we review these new data in discussing the relative merits of these two ideas.

Introduction

In everyday life, both humans and animals choose between different alternative outcomes or goods (e.g., what to eat for dinner); this has been described as value-based decision making, since it is presumed to reflect subjective valuation of the different options [1]. The rubric underlying such value-based decision making assumes two consecutive steps: valuation of available options on a common scale and action selection based on these valuations. With the finding of neural correlates of this scalar value in the brain, especially in the OFC, this view has gained popularity [1,2]. OFC hence has become widely seen as a dedicated neural substrate for calculating subjective value, which other brain areas then use to guide choices among available outcomes or goods [3]. This is despite the relative lack of studies showing that OFC is fundamentally necessary for deciding between goods based on value, the widespread finding of value correlates in many other brain regions, and the lack of any definition of value that is independent of behavior [4,5]. With an emphasis on new data, below we will consider each of these problems, and then explore another emerging view suggesting that OFC plays a critical role in representing task structure and is only necessary for value-based behavior when such a representation – a cognitive model or map – is required for calculating the value underlying normal choice.

Neural correlates of value in the OFC

Perhaps the most influential evidence for the idea that the OFC’s core function is to signal value comes from primate recordings on an economic choice task [3,6], in which subjective value is measured though choice behavior. Although this idea certainly has a long history prior to this work, earlier studies generally emphasized the representation of associative information or associative conjunctions [7–10]. Work using the economic choice task was arguably the first to clearly dissociate neural correlates of “economic” value, or what is called revealed preference, from sensory and motor aspects of the underlying associations. That is, some OFC neurons appeared to respond only based upon the scalar value of the goods on offer and not about any of the unique information comprising the choice (cues, outcome features or quantities, direction or type of response). Of course, such correlates can only be interpreted within the narrow parametric space of the task (i.e., the specific cues used and particular outcomes on offer), but nevertheless the activity of this category of neurons, along with subsequent results in humans using fMRI [11,12], have been taken as strong evidence that OFC is fundamental to determining the value underlying our choices.

And correlative evidence supporting this role continues to accumulate; in various choice tasks, neuronal activity in the OFC correlates with value in fMRI imaging, electrophysiological, and calcium imaging studies [13–20]. These value representations are adaptive to outcome and context changes [15,21,22], modulated by gaze fixation [23] and covert shift of attention [24], and supported by a mixture of temporally dynamic and stable coding schemes [25]. OFC neurons projecting to striatum predominantly encode integrated value among several other decision variables [26], and even theta oscillations in the OFC are reported to track the values of new reward-predictive cues during learning [27].

When interpreting these data, however, it is important to bear in mind that what can be found in a correlative study is constrained by particular task designs. While each of these studies found value encoding in the OFC, each was designed for this purpose. And many do not employ the multidimensional approach that characterizes the economic choice studies. Instead, for practical reasons, these designs assume that neural correlates tracking just a single dimension of reward, such as likelihood, amount, or caloric content, necessarily represent value. Neural correlates tracking value across only one reward dimension may not be abstractly tuned ‘value’ neurons. Indeed even two dimensions provide a relatively weak test of the null hypothesis, given the variability of single-unit activity and the amount of training the subjects typically receive, which may drive changes in how specific outcomes within a narrow range (i.e., juices) are represented. As a result, tasks using few reward dimensions to determine value likely overestimate the number of neurons that represent a multi-dimensional, truly abstract value. Consistent with this idea, this permissive definition of value has led to observations of value coding being prevalent across the entire brain [4], even in the cerebellum [28] and hypothalamus [29]. Against this interpretation, while some OFC neurons do appear to strongly encode information that can be interpreted as value, nearly all neural recording studies have found that OFC neurons also respond to a variety of other task variables.

For instance, individual OFC neurons can be found with activity tuned to cue and outcome identity [16,30], spatial locations [31,32], action selection [14,33,34], integration of prior and current information [14,18], decision confidence [26,35], and task rules [36]. Even in now-classic studies of neuroeconomic value, in which value is determined from integration of multiple reward features, many OFC neurons represent the value of only some options, and still more respond based on sensory features of the goods [6]. Although not necessarily contradictory to the idea that the OFC signals value, this heterogeneity of findings does raise the possibility that value representation is not the core function of OFC, since it fails to explain so many neural correlates found there, even when the area is faced with what is a very narrow range of outcomes compared to what we experience in our daily lives.

Possibility of value as part of task structure representation

How can we account for such diverse neural correlates? An alternative hypothesis, which would predict such diversity, proposes that OFC represents states or “locations” and their transitions in the task space (i.e., the task structure or cognitive map) [37,38]. If this is the case, value can be seen as one important aspect or derivative of such a representation, particularly in the experimental settings typically employed in behavioral neuroscience studies. This hypothesis has considerable support from both older work as well as recent correlative studies in humans [39–44], non-human primates [45], and rodents [31,46–51].

For example, in perhaps the first human imaging study to focus on this specific question [40], subjects were trained to judge the age (young or old) of a face or a house that were spatially superimposed in the same image (Figure 1a). Whether to switch to judge a face or a house on a given trial was determined by a change of age on the previous trial. This created 16 unique trial types or states, the identification of which had to be inferred through trial transitions (Figure 1b). These “hidden” states, when necessary for behavioral learning and decision making, were found to be uniquely decodable from BOLD signals in the OFC (Figure 1c). The data provide direct evidence that the OFC represents task states that are crucial for cognitive mapping.

Figure 1:

Task state representations in human OFC. (a) Participants were asked to keep judging the age (young versus old) of one of the two categories (face or house) until an age change occurred. When an age change occurred, participants had to switch to judging the other category starting from the first trial after the change. The task rules created an alternating mini-block structure of judging either the age of faces or houses. (b) Possible transitions between 16 task states. Each circle denotes a particular state. Arrows indicate possible transitions. (c) Average classification of 16 task states identified from fMRI patterns within the OFC (blue bar) and following a permutation test (black bar). Dashed lines, chance level; error bars, s.e.m. *p < 0.05. Plots recreated from Ref. [40].

Critically, such hidden states are not the same as value, however their appropriate identification can clearly be useful for predicting value. For instance, performing properly in reversal tasks – something that is classically dependent on the OFC – can be facilitated by recognizing the hidden “reversal” state as distinct from initial learning [37]. This allows a new behavior to be acquired more quickly because it does not require first unlearning or modifying the prior learning, since it is not relevant in the new reversal state.

In an effort to show the interaction between the representation of value and states, we recently recorded single-unit activity from OFC while rats were performing a simple odor discrimination task in which performance reflected a knowledge of the odor sequence across trials [48]. Like the work above, proper responding in this task required tracking of prior events in order to discriminate hidden states predictive of reward (Figure 2a). In this context, we found that current value was a prominent feature of both the single-unit and neural-ensemble activity, but that the task states – positions in the sequence – were also decodable independent of this value information (Figure 2b,c). This included the hidden states, which were only discriminable based on prior events. The dissociation of value from structural information about the task suggests that the two are multiplexed in the OFC but separable at the neural-ensemble level, which argues against the idea that value is the ultimate output of the OFC and in favor of the proposal that encoding of structure is one fundamental function of the OFC.

Figure 2:

Value-dissociable task state representations in rat OFC. (a) Odor sequence task. Each number (0 – 15) indicates a unique odor that was used as a cue in a single trial of the “go, no-go” odor discrimination task. The 16 different odor cues were presented in four sequences (S1a, S1b, S2a, and S2b). There are six trials or positions (P1 – P6) within each sequence. Arrows indicate the transitions between sequences. The task design created 24 task states. (b) Hypothesized confusion matrices resulted from decoding of 24 task states. (c) A linear discriminant analysis (LDA) was performed on the original neural data recorded from the rat OFC with the labels of reward value, to dissociate neural representations of the current value and current state. The three panels show actual confusion matrices as results of classification of the 24 task states using original neural data (left) and the first linear discriminant component (middle; similar to the pattern of the ‘current value’ model), and other linear discriminant components (right; similar to the pattern of the ‘current state’ model). (d) Another odor sequence task consisting of two alternating odor sequences (S1 and S2). This task was intended to resemble a continuous T-maze. Note that S1 and S2 have identical reward availability at each position (P1 – P4). (e) Mean classification accuracy of S1 versus S2 using recorded OFC neurons at all positions. Error bars, s.d. * denotes that mean decoding accuracy exceeds 95% confidence interval from label-shuffled decoding. Plots in (a), (b) and (c) recreated from Ref. [48]; (d) and (e) from Ref. [46].

The representation of hidden states independent of reward or other confounding factors is also evident in another odor sequence task, in which two alternating sequences of odor-guided trials were used to mimic a continuous T-maze [46]. Like the continuous T-maze, the two sequences have a common central path (the same odors at each position) with unique entry and exit paths (different odors at the same positions) (Figure 2d). Critically, there is no difference in value between the two sequences at any position – from a value-based perspective, there is simply a single loop in the task structure.

Recording single unit activity in OFC in this task, we found that neurons nevertheless differentiated the sequences at all positions; successful decoding was even observed at positions where both odor identities and values were exactly the same (i.e., hidden states) (Figure 2e). Thus, these data demonstrate that OFC neurons are able to disambiguate hidden states, even in the absence of differences in value or observable sensory inputs.

Encoding of structural information about the environment is also evident in OFC during sensory preconditioning. In this setting, when the reward is completely absent, OFC has been found to acquire representations of sensory-sensory associations in both rats [49] and humans [52]. Specifically, during the initial phase of the task – when two value-neutral sensory cues are paired – activities of both single units in rats and BOLD signals in humans change to reflect the associations between the two cues by exhibiting similar response patterns.

OFC is required for model-based value computation

Correlative studies discussed above collectively suggest that the value signaling is not something peculiar in the OFC but can be seen as part of the task structure representation. One could argue, however, that the task structure is represented for the sole purpose of computing value within the OFC, and thus the value signaling is still the major functional output of OFC. If this is the case, disrupting OFC function should always disrupt behavioral performance in value-based decision-making tasks, without affecting behavior when value is not at issue. And yet this is not what causal data shows. Instead, it is easy to find value-based behavior – even in explicit-choice tasks – in which inactivating OFC has no effect, and there are examples of behavior that do not seem to involve value at all that still require the OFC.

Value-based behavior that is independent of the OFC is easy to find and ranges from Pavlovian conditioning to operant responding to discrimination learning [4]. Indeed most behaviors in experimental settings, the vast majority of which are value-based, do not require OFC. However the most striking such evidence comes directly from a rodent version of the economic choice task first developed in monkeys to search for value representations [6]. In this task, rats were trained to make choices between pairs of food pellets [53] (Figure 3a). On each trial, the shape and quantity of each pellet on offer was indicated by the shape and number of segments of two symbols presented on touch screens in front of the rat. By touching the appropriate screen, the rat gained access to the type and quantity of the pellet associated with the symbol. Optogenetically inactivating either lateral [53] or medial [54] OFC had no effect on the choice behavior of well-trained rats in this setting, as indexed by the subjective preference, steepness of the choice curve, or transitivity between different pairs of goods (Figure 3b). Similar negative results were also reported in a choice task requiring information about probability and amount to be integrated; value-based choices on individual trials were not affected by lateral OFC inactivation [55]. While there are examples to counter these reports, in subjects that have less experience in the task or in which the OFC is artificially over-activated instead of being inhibited [13,56], the above work demonstrates that in very well-controlled settings, the OFC is not strictly necessary for value-based behavior, even when current value must be integrated across multiple dimensions of information. This should not be possible if the OFC’s core function is to calculate economic value.

Figure 3:

Dependence of behavior on OFC in the rat economic choice task. (a) Illustration of the economic choice task. Rats choose by pressing one of the two touch screens displaying two-dimensional offers of food pellets that differ in identity and quantity. (b) Optogenetic inactivation of OFC during the cue period did not change the economic choice behavior of well-trained rats. (c) One of the food pellets was pre-fed before well-trained rats were tested in the economic choice task. (d) In control sessions, the effect of pre-feeding was related to whether and how much the pre-fed pellet was preferred. Specifically, if the preferred pellet was pre-fed, choices shift towards the pellet, otherwise choices shifted away. With OFC inactivation, this relationship was abolished, as if the current pellet value was no longer consistently related to choice. (d) Introducing novel offer pairs in the economic choice task. Solid lines show food pellet pairs that rats were previously trained on. Dotted lines indicate food pellet pairs that rats had not experienced together previously. The modified task tested how rats would incorporate new information to modify an established “goods space” describing the preference between food pellet pairs. (e) It took longer for OFC-inactivated rats to reach a stable preference for novel food pellet pairs. Plots in (a) and (b) recreated from Ref. [53]; (c) and (d) from Ref. [71]. (c) and (d) from Ref. [74].

Notably, at the same time that OFC is not necessary for value-based choices in the above studies, it is critical for value updating that relies on model-based inference. This was first evident in Pavlovian reinforcer devaluation, an iconic behavioral task to assess model-based inference. In this task, subjects are initially trained to associate a cue and a reward. After the outcome is paired with an unpleasant experience such as LiCl injection or satiety, normal subjects update value of the cue through its model-based association with the outcome, which is reflected in reduced responding to the cue during the probe test. Work using this procedure has shown repeatedly that OFC manipulations disrupt normal changes in behavior after devaluation [57–64]. This conclusion is consistent with many other studies using different behavioral tasks, such as overexpectation [65] and sensory preconditioning [66,67], in which model-based inference is needed for value updating. It is worth noting that testing the necessity of OFC in classic instrumental devaluation tasks has led to negative results, suggesting a dissociation in the support of strictly instrumental versus Pavlovian representations [58,68,69], although effects of OFC manipulations on instrumental devaluation have been reported when prior learning of the operanda has occurred, perhaps due to the increased complexity of the setting that may introduce Pavlovian information regarding hidden states [70].

If representing the task structure underlying model-based behavior is the core function of the OFC, then it predicts that OFC should be necessary for economic choice when the underlying value depends on a model of the task for its construction. In well trained subjects, it is unclear when this is true; however one way to force this would be to devalue one of the goods on offer prior to the critical testing. As in the Pavlovian devaluation task described above, this manipulation would force the use of a model to determine the value underlying normal behavior. We have recently done exactly this experiment, inactivating the lateral OFC in rats during economic choice when one of the outcomes was pre-fed prior to test (Figure 3c). In this setting, we found that inactivation again had no effect on the established choice, while at the same time it disrupted the normal effects of pre-feeding on the choice [53,71] (Figure 3d).

Of course, deficits in devaluation tasks after OFC manipulations, while consistent with a wider role, do not necessarily require one; instead they may simply reflect a more limited role in value-based decision-making [3]. To argue that behavioral deficits caused by OFC dysfunction actually reveal a failure in the general use of model-based information, outside the realm of value signaling per se, requires evidence of impairments in behavior that is orthogonal or even independent of value. Though this at first seems far-fetched, it turns out that such evidence is actually relatively abundant. For instance, OFC is required for outcome-specific Pavlovian-to-instrumental transfer [58,68], outcome-identity unblocking [72], facilitated learning caused by differential outcomes [73], and for proper sensory learning in the first phase of sensory preconditioning [59]. In each of these tasks, the critical role of the OFC is difficult to explain as signaling of scalar value, while being easy to explain as related to representing task structure. This is particularly true in a recent study showing that optogenetically inactivating OFC during the initial cue-cue learning during sensory preconditioning abolishes value inference in the later the probe test [59]. The initial cue-cue learning occurs in the absence of any overt value or reward; there is no behavioral response. However learning of the valueless sensory associations is critical to the later inference; thus these results demonstrate that the OFC is required for building associative models, even in the absence of any reward. Together with the electrophysiological recording data on the sensory preconditioning task [49] showing that OFC keeps track of these sensory-sensory associations even before rewards are introduced, these results provide clear evidence that OFC is critical for learning task structure independent of value.

Why would this be? One key aspect of the sensory preconditioning task is that it involves the acquisition of new information about the world – the cue pairs have never been encountered before. This is also true in economic choice, when subjects first encounter new pairs of goods; even if they have previously encountered the two options separately, relating them and accurately determining subjective preferences when options are compared for the first time requires new mapping. Interestingly, we have recently found that the OFC is necessary for economic choice under these conditions. That is, while inactivation of OFC has no effect on economic choice in our hands once the rats have experienced a pair of goods together, if we inactivate the lateral OFC in the very first session with a new pair, then rats show impairments at establishing their initial subjective preference [74] (Figure 3e, f). This is true even if the rats have experienced the pellets many times before as part of other pairs. This finding suggests that the OFC is necessary for establishing the relationships between pellets in the “goods space”, rather than for using the goods space to read out the economic value.

Conclusions

There is still a strong belief in the field that the core function of the OFC is to represent subjective or economic value, and that this value signal is the main output from OFC used by other brain areas to form behavioral plans and instruct action selection. Although this idea seems to be supported by an abundance of correlative studies, we argue against the proposal that value signaling is the core function of the OFC based on the poor definition of value, the widespread finding of similar value correlates in most of the brain, as well as causal data showing that the OFC is often not necessary for value-based choice. Conversely, OFC is often required for behaviors for which value is either not critical or is entirely absent during periods when OFC is required. Overall, these data are more consistent with the proposal that the primary function of OFC is to represent task structure, from which value is partly derived.