Prioritising the relevant information for learning and decision making within orbital and ventromedial prefrontal cortex

Abstract

Our environment and internal states are frequently complex, ambiguous and dynamic, meaning we need to have selection mechanisms to ensure we are basing our decisions on currently relevant information. Here, we review evidence that orbitofrontal (OFC) and ventromedial prefrontal cortex (VMPFC) play conserved, critical but distinct roles in this process. While OFC may use specific sensory associations to enhance task-relevant information, particularly in the context of learning, VMPFC plays a role in ensuring irrelevant information does not impinge on the decision in hand.

Introduction

It is an obvious, but sometimes overlooked, fact that it frequently takes many weeks to get an experimental animal to perform a task that could be explained to a human participant in a matter of minutes. From one perspective, this neatly encapsulates how useful language is to communicate information. However, it also highlights just how important, and often difficult, it can be without such input to determine which specific elements of a complex environment should be used to guide and update behaviour. This is particularly evident in situations where stimuli and rewards are separated in space and time, can have different meanings depending on the external context or internal state, and can also provide several different types of information (for instance, a food or fluid reward might both satisfy an internal need and provide information that the correct response has been made) [1].

One pressing question is therefore what neural structures help select relevant information and inhibit irrelevant information for the task in hand and how these relate to neural mechanisms implicated in value-guided decision making [2-6••,7]. A related issue concerns the mechanisms that allow us to determine, and potentially seek out, information relevant to satisfy a current need, and also how these systems interrelate with circuits implicated in reward seeking [8].

While these are complex topics, in this brief review we will focus on converging evidence that the lateral parts of orbitofrontal cortex (OFC) and ventromedial prefrontal cortex (VMPFC) play key roles in these faculties.

Anatomical considerations

OFC and VMPFC are large structures consisting of multiple distinct areas. Nonetheless, there are anatomical similarities between certain regions, which has allowed Price to define two distinct, though interconnected, networks in rodents, monkeys and humans [9]. First, an “orbital sensory” network, including Walker’s areas 11, 12 and 13 and parts of anterior insula in primates, receives rich sensory information from all sensory modalities and also projects back to sensory structures. The equivalent network in the rat would include LO, VLO and AIv. By contrast, a “medial visceromotor” network, including medial OFC area 14 as well as areas 25 and 32 and medial area 10, is characterised by strong connections with the medial temporal lobe as well as projections to limbic regions such as ventral striatum and lateral hypothalamus. In the rat, this network is likely made up of MO (medial orbital), prelimbic and infralimbic cortex. This network has access to information about motivational states and the current context, and can influence arousal states through connections to regions such as the hypothalamus.

Therefore, for the purposes of this review, we will refer to the former set of areas collectively as “OFC” and the latter, including medial OFC as “VMPFC”. However, we acknowledge that the information encoded and specific function of particular structures within these general areas may have important differences [cf.7].

OFC and specific reward representations

There is general agreement from both single unit [10,11] and fMRI [12•] studies that parts of OFC encode the precise identity of rewards, and can represent specific associations between stimuli and economic parameters such as reward size, probability and delay [12•, 13-14]. Arguably the most reliable effect of disruption to this region is to reduce the influence of reinforcer devaluation on subsequent choices [15,16].

What remains a matter of much debate is the function these signals play during learning and decision making. One possibility is this information is used to construct an integrated value signal that could underpin “goods-based” decision making [4]. OFC represents the value of options (large negative < neutral < large positive) rather than their salience as defined by their divergence from indifference (large negative > neutral < large positive) [17,18].

However, the interpretation of such value coding has been challenged. Schoenbaum and colleagues have demonstrated in a series of elegant studies that cells in rat OFC are sensitive to parameters such as identity or associative salience even when reward value is carefully controlled for [19,20]. Perhaps most compellingly, McDannald and colleagues [21••] recently showed that a population of OFC cells would increase their firing when a new stimulus combination was followed by either an increase in reward magnitude or a different, but equally-preferred, flavour of reward. In fact, these cells would generally signal the degree of sensory and outcome divergence from the original learned state, a finding that chimes with several other studies showing rich, rapid sensory encoding in OFC [22,23].

Indeed, outside of the domain of reward quality and quantity, few OFC neurons encode combinations of economic parameters; instead, individual value parameters are encoded in overlapping small populations of neurons [13,14,24•,25]. Given that OFC can encode information about the specific association between a stimulus and the sensory properties of a reward separate to any information about value, this implies that OFC’s role in the decision process is better described as the formation of stimulus-based predictions based on the attributes of rewards and the information to be gained from their outcomes, key inputs for a decision process.

Crediting outcomes to the correct choices

Another way of considering the functional significance of specific representations of expected outcomes is that they can facilitate appropriate updating of value estimates [6••,26,27]. Walton, Noonan and colleagues [28,29•] showed that lesions to lateral OFC, but not to medial OFC, in macaque monkeys caused a specific and selective deficit in crediting a particular stimulus choice with its consequent outcome in a 3-option probabilistic decision making task.

While the precise locus of such an effect is a matter of current debate [30], under this perspective, it seems plausible that specific types of outcome are not represented in OFC to control choices directly, but instead to facilitate rapid updating of stimulus-based associations by allowing animals to accurately assign credit to a particular stimulus or choice that produced them. This in turn will enable accurate stimulus-based value estimates to be passed on to structures involved in choosing what option to select.

OFC, task structure and selecting how to learn

If correct, the next pressing question is to determine what exact computations OFC performs and how the OFC resolves which elements of the world are relevant for learning. Some potential clues can be found in the study by Walton and colleagues discussed above [28]. One consequence of the loss in appropriate credit assignment observed in the OFC-lesioned animals was that it unmasked a separate, intact learning mechanism that could approximate stimulus-outcome associations by using recent choice and reward histories. It is important to note that this faculty was not a novel learning strategy acquired after the lesion; logistic regression analyses showed that these recency-weighted choice and reward histories affected choices to an almost equal extent pre- and post-operatively in the control and lesion groups. However, in the absence of an OFC lesion, their impact on behaviour was dwarfed by the much stronger influence of specific stimulus-outcome pairings. This implies that the way the OFC promotes appropriate credit assignment might therefore be to enhance current task-relevant associations rather than to suppress irrelevant ones.

A number of studies have provided evidence for a role of OFC in such a faculty. For instance, excitotoxic OFC lesions in rats cause them to have abnormally persistent latent inhibition [31]. The lesion rendered them slower to respond to a stimulus relative to unlesioned control animals when it switched from being neutral to becoming reinforced; in other words, the OFC group were impaired at upregulating attention to a familiar but previously behaviourally irrelevant stimulus once it became a useful predictor of future events. By contrast, there is little evidence that OFC lesions that spare medial OFC directly disrupt extinction learning, implying no role for this region in disengaging with a stimulus when it no longer predicts reward [15,32].

There is also evidence that OFC might play a role in identifying the type of decision environment the agent currently faces, a sort of ‘relevance filter’ over the vast stimulus (decision) space available to an agent at any given time [6••]. For instance, in a task where monkeys were trained to make decisions on separate trials between either delay- or effort-discounted rewards (Fig. 1A), the single biggest factor explaining OFC neuronal variance was the type of decision presented on that trial rather than the value of the options [24•; See Fig. 1B, C].

Figure 1. OFC neurons signal task context.

A) Task design in [24•]. Subjects made choices between pairs of presented pictures. On effort trials, they had to lift a lever and hold it for 1.5-s to earn the reward. On delay trials, they simply had to wait for a specific delay before they received the reward. The amount of reward they received, the force to lift the lever and the length of delay varied depending on the picture they chose. Effort and delay trials were intermingled in the same session. B) Percentage of neurons that were identified in OFC, the anterior cingulate cortex (ACC) and the cingulate motor area (CMA), using stepwise regression, as encoding specific variables during the first half of the choice epoch. For OFC, the 3 variables that were most frequently encoded are shown. The horizontal dotted line represents chance levels of neuronal selectivity. Red bars indicate that the proportion of selective neurons is significantly higher than the average proportion of selective neurons in the fixation period, assessed using a binomial test at p < 0.05, with a Bonferroni correction for multiple comparisons. C) Time course of activity encoding the type of decision for effort-preferring and delay-preferring neuronal populations. Each plot shows the mean firing rate relative to baseline ± S.E.M. The grey shading illustrates the 500-ms epoch used for analysis which is centered on the mean latency at which the information is encoded in each area. The blue numbers indicate the number of neurons included in each plot. OFC neurons were modulated primarily by delay- but not effort-based decisions. This is consistent with studies showing effects of OFC lesions on delay- but not effort-related decision making in situations where there is no bridging cue between the choice point and the delayed reward [63,64]. Adapted from [24•].

Whether OFC is able to select the appropriate task structure or just applies this information computed by other frontal cortical regions is not yet known; as is shown in Fig. 1B, encoding of decision type predominated across multiple regions of frontal cortex and was not unique to OFC. What is evident is that OFC can utilise information about task structure to promote rapid contingent learning.

VMPFC, valuation and value comparison

Unlike research into OFC function, evidence for the role of VMPFC in value-guided decision making has to date been largely driven by human studies. The BOLD signal in this region has often been shown to correlate with the current subjective value of various different types of options [33-35]. This holds true even in the case where the particular item has never previously been directly experienced [36].

However, as with the OFC, the functional role of VMPFC value signals remains disputed. Representations of decision value are evident in many brain regions [37], thus an important question is to identify a neural signature of a decision. A version of a biophysically plausible attractor network model of a binary probabilistic choice process [38] suggests decision inputs (values) are initially summed, and then compete via mutual inhibition, producing a later, second signal reflecting the difference in value between the chosen and unchosen options [39••]. Critically, VMPFC activity contained both such signatures in the correct timeframe [39••].

In fact, in many situations when two choice options are presented, the BOLD signal in this region not only correlates positively with the subjective value of a chosen, attended option, but also negatively with the value of the next best, but rejected option [40-42]. Recently, Strait and colleagues have reported comparable antagonistic effects between the values of two sequentially presented options in area 14 in macaques [43•]. Together, this evidence points towards an important role for VMPFC in a competitive value comparison necessary for decision making [3,39••].

When and why does value comparison take place in VMPFC?

Nonetheless, while VMPFC activation is common to a range of studies (outside the domain of decision making as well as within), it is not a signature of all decisions and is instead critically dependent on the local context. For instance, VMPFC value comparison signals not observed when selecting whether to take an available option or to forego this to search for something better in the environment; only when a decision is made to engage with the current option does the VMPFC BOLD signal represent the value of this chosen item [44].

VMPFC also only appears involved in the context of “difficult” choices, such as when choosing between two options that are close in value or when different decision variables advocate opposing choices, and even this may depend on how unfamiliar subjects are with such decisions [29•,39••,45]. Monkeys with medial, but not lateral, OFC lesions also exhibit irrational context-dependence of their choices in a 3-option probabilistic decision making task; after surgery, logistic functions describing the pattern of choices between pairs of options became affected by the value of the 3^rd available option in these animals, violating normative models of rational choice [29•]. Such effects were particularly prominent during difficult choices.

VMPFC, attention and selection

What is common to situations that recruit or require VMPFC during value-guided decision making is that, first, the goal is clearly selectable from currently presented stimuli and, second, the task environment requires relevant information to be sampled and selected for an optimal choice to be made. Indeed, an alternative account of the chosen minus unchosen comparison signal in VMPFC is that it instead reflects the difference between an attended and an unattended option, especially as chosen items generally are attended longer than unchosen ones [46,47•]. Neurons in dorsal parts of VMPFC encode value information particularly around attentional shifts, suggesting integration between the allocation of attention and valuation processes [48].

A change in the way information is attended to and acquired following VMPFC damage [49] might explain why the predominant deficit observed experimentally in monkeys and humans with VMPFC damage is an increased tendency for inconsistent choices [15,50,51]. Unlike the maladaptive increase in exploratory choices seen following OFC lesions [28], this cannot be explained by impaired value learning [29•].

One way of integrating these ideas is to suggest that VMPFC does not just mediate value comparison, but is also required to maintain selective focus on information that is most relevant to the current goal. Chau and colleagues [52••] investigated how the presence of a third, but unavailable and therefore irrelevant, alternative would influence speeded choices between two other relevant options (Figure 2A). They found that people would on average make more suboptimal choices during difficult decisions when the value of the unavailable distractor was comparatively low and the presence of such a low value distractor reduced the VMPFC value comparison signal (Figure 2B-C). Moreover, subjects who showed the greatest influence of the distractor on the VMPFC value comparison signal also made fewer choices of the best option (Figure 2D). There was also evidence that this process was influenced by interactions with OFC. The value comparison signal in VMPFC was positively coupled with activity in lateral OFC whereas the influence of the distractor on the VMPFC signal was negatively coupled with a similar part of lateral OFC.

Figure 2. The behavioural and neural influence of an irrelevant distractor on decision making.

A) Task design in [52••]. Subjects were presented with three options each consisting of a rectangular bar of a particular colour presented at a particular orientation. Each colour and orientation combination was associated with specific reward magnitude and probability. 100ms after presentation of the 3 options, one of the options became an unavailable distractor, D, that could not be selected (indicated by the purple box). Participants had to make speeded decisions between the 2 relevant, available options (higher value option, HV, and lower value option, LV). B) During difficult trials when the two available options were close in value (green box), more suboptimal decisions were paradoxically made when the difference between the value of the best available option and the distractor (HV – D) was larger. C - D) The VMPFC HV-LV value comparison signal was also attenuated with increasing size of HV – D (panel C) and individuals who exhibited a stronger influence of the distractor on the VMPFC signals (i.e., a more negative HV-D effect) resulted in more suboptimal decisions (panel D). Adapted from [52••].

This could be interpreted to suggest that VMPFC might help facilitate context-appropriate value comparison in part through mechanisms that suppress information that is not currently relevant for their current motivational needs [44,53-55]. Such an account is congruent with recent evidence in rodents that stimulus-selective cells in medial OFC, unlike in lateral OFC, show a small but significant increase in firing to odours associated with the least valuable option in a delay/reward decision task [56]. Lesions to an adjacent structure – prelimbic cortex – also cause rats to fail to downregulate attention to a novel cue in a blocking paradigm even though it provides no new information to guide predictions and choice [57]. It will be interesting to determine whether a similar process of competition by mutual inhibition, which can successfully account for VMPFC value comparison signals and even the paradoxical effects of a distracting alternative [39••,52••], might be extended to generally predict such a function.

Conclusions

In this brief review, we have outlined ideas that suggest that OFC and VMPFC have key complimentary roles in selecting the appropriate information to allow appropriate value learning and value comparison to occur. OFC, through interactions with sensory cortex, can use stimulus-reward associations to enhance attention towards specific, task-relevant environmental information, which in turn can allow rapid contingent learning when new information is acquired; VMPFC, with access to information about the current motivational goals, can help suppress irrelevant value information impinging on an ongoing decision. These regions clearly do not perform these functions in isolation [cf. 52••] and it will be critical in the coming years to investigate how these two networks cooperate to promote selection. This will also require a comparison between OFC and VMPFC signals with interconnected brain areas [14,24•,48,52••], examining interactions between structures [52••,58,59], and particularly looking at how interference in one part of the network affects coding elsewhere [27,60]. Moreover, understanding the way in which these or other regions determine current task relevance and gather information in a dynamic setting is of primary importance [61,62].