A new perspective on the role of the orbitofrontal cortex in adaptive behaviour

Abstract

The orbitofrontal cortex (OFC) is crucial for changing established behaviour in the face of unexpected outcomes. This function has been attributed to the role of the OFC in response inhibition or to the idea that the OFC is a rapidly flexible associative-learning area. However, recent data contradict these accounts, and instead suggest that the OFC is crucial for signalling outcome expectancies. We suggest that this function — signalling of expected outcomes — can also explain the crucial role of the OFC in changing behaviour in the face of unexpected outcomes.

The orbitofrontal cortex (OFC) has long been associated with adaptive, flexible behaviour in the face of changing contingencies and unexpected outcomes. In 1868, John Harlow¹ described the erratic, inflexible, stimulus-bound behaviour of Phineas Gage, who reportedly suffered extensive damage to the orbital and midline prefrontal regions². Since then, owing to increasingly refined experimental work, the regulation of flexible behaviour has been attributed to the OFC, a set of loosely defined areas in the prefrontal regions that overlie the orbits.

In this Perspective, we first review data showing that the OFC is crucial for changing behaviour in the face of unexpected outcomes. Then we describe the two dominant hypotheses that have been advanced to explain this function, and follow this with data that directly contradict both of these. We focus on a region of the OFC for which there are strong anatomical and functional parallels across rodents and primates; this area encompasses the dorsal bank of the rhinal sulcus in rats, including lateral orbital regions and anterior parts of the agranular insular cortex. These cortical fields have a pattern of connectivity with the rostral basolateral amygdala, the ventral striatum, the mediodorsal thalamus and sensory regions that is qualitatively similar to the pattern of connectivity of areas 11, 12 and 13 in the primate orbital prefrontal cortex^3,4. Furthermore, as we outline below, neurophysiological and lesion studies show remarkable similarities in the functions of these areas across species. These studies indicate that this part of the OFC is crucial for signalling information about expected outcomes and for using that information to guide behaviour. We suggest that signals from this part of the OFC that relate to expected outcomes also contribute to the detection of errors in reward prediction when contingencies are changing, thereby facilitating changes in associative representations in other brain areas and, ultimately, behaviour.

The OFC is crucial for flexible behaviour

The role of the OFC in flexible behaviour is typically assessed using tasks in which subjects must change an established behavioural response in order to adapt to new contingencies. FIGURE 1 shows an example in rats performing a simple odour discrimination task. Here, rats are trained to sample odour cues at a centrally located port and then visit a nearby fluid well. They learn that one odour predicts that the well contains sucrose and that a different odour predicts that it contains quinine. Because rats like sucrose but want to avoid drinking quinine, they learn to discriminate between the two odour cues, drinking from the well after sampling the sucrose-predicting cue and withholding that response after sampling the quinine-predicting cue. After learning, their ability to rapidly reverse their responses can be assessed by switching the odour–outcome associations. OFC lesions cause an impairment in animals’ ability to rapidly reverse responding in this setting, a deficit that has been shown in different laboratories over the past 50 years, across species, experimental designs and learning tasks^5–16. The reliability of these findings provides convincing evidence that the OFC performs a crucial function in flexible behaviour. Two hypotheses have dominated the discussion of what this function might be; they are reviewed below.

Figure 1. Showing orbitofrontal involvement in flexible, adaptive behaviour in rats using an odour discrimination reversal task.

The events in a simple odour discrimination reversal task that demonstrates the involvement of the orbitofrontal cortex in flexible, adaptive behaviour. Rats sample an odour from a port. After sampling, they decide whether to respond at a nearby fluid well. One odour predicts a tasty sucrose solution, whereas a second odour predicts an unpleasant-tasting quinine solution. As illustrated in the top panel, rats rapidly learn to discriminate between the odours, drinking from the fluid well after sampling the sucrose-predicting odour but withholding that response after sampling the quinine-predicting odour. After this has been learned, the odour–outcome associations can be reversed, such that the sucrose-predicting odour comes to predict quinine and vice versa. This is illustrated in the bottom panel. Initially rats respond on the basis of what they have learned previously. However, normal rats rapidly recode these associations (a) and inhibit their old response patterns in favour of new ones (b); rats with damage to the orbitofrontal cortex fail to change their behaviour as rapidly.

The OFC as response inhibitor

The first and perhaps most prevalent hypothesis is that the OFC is crucial for flexible behaviour generally — and reversal learning in particular — because it inhibits inappropriate responses (FIG. 1b). This idea was introduced by David Ferrier, who proposed, on the basis of experiments in dogs and monkeys, that the frontal lobes are crucial to suppressing motor acts in favour of attention and planning¹⁷. During the past few decades, response inhibition has become specifically associated with orbitofrontal function because it provides an attractive description of the general and specific behavioural effects of orbitofrontal damage¹⁸. For example, this explanation fits well with the constellation of symptoms caused by orbitofrontal damage in humans, which include impulsivity, perseveration and compulsive behaviours¹⁹, and it also provides an excellent description of reversal-learning deficits^5–16, as well as deficits in detour reaching tasks and stop signal tasks^20,21.

However, this hypothesis does not have good predictive power outside of these settings. In fact, it is easy to find behaviours that require response inhibition that are not affected by OFC damage. For example, in many of the reversal studies described earlier, subjects with lesions of the OFC could successfully inhibit responses during initial learning even though they exhibited difficulties in inhibiting them after reversal. Similarly, the OFC is not required for inhibition of ‘pre-potent’ or innate response tendencies. For example, in reinforcer devaluation tasks, animals with orbitofrontal lesions have trouble changing cue-evoked responding, but they are readily able to withhold the selection or consumption of food that has been paired with illness or that has been fed to satiety^12,22–26 (but see REF. 27). Another example comes from a study²⁸ in which monkeys were allowed to choose between differently sized peanut rewards but had to select the smaller reward in order to receive the larger one. Monkeys with OFC damage inhibited their innate tendency to select the larger reward just as well as controls. So overall the OFC — or at least the region that seems the most analogous across species — does not seem to have a general role in response inhibition.

Of course, this does not imply that whatever role the OFC has is not important for inhibiting responses in some settings — it only implies that this is not the root function of the OFC. Furthermore, it is of course possible that other prefrontal areas — perhaps even some that have been termed OFC but that lie outside of the area that we focus on here — mediate a more general inhibitory function. Indeed, this has recently been suggested²⁷. However, at least for the region of the OFC that is targeted by the studies described above, response inhibition does not seem to be a general function.

The OFC as rapidly flexible encoder of associative information

More recently it was suggested that the OFC is crucial for flexible behaviour because it is better than other brain areas at rapidly encoding new associations, particularly those between cues and outcomes²⁹. According to this hypothesis, the OFC supports flexible behaviour by signalling this associative information to other areas, thereby driving correct responses or, perhaps, inhibiting incorrect ones (FIG. 1a).

This idea was based on single-unit recording studies showing that cue-evoked neural activity in the OFC rapidly reflects new associative information during reversal learning. This was first reported by rolls and colleagues³⁰, who noted that, in monkeys, single units in the OFC often reversed their firing response between two stimuli (that is, switched from firing in response to one stimulus to firing in response to another stimulus) when their associations with reward were reversed. This reversal of encoding was initially shown for different syringes used to deliver appetitive and aversive fluids³⁰ and subsequently shown for visual and olfactory discrimination^31,32.

These findings, together with OFC-dependent reversal learning impairments, are consistent with the idea that the OFC supports flexible behaviour because it stores the new associative information more rapidly or in a more functional way than other brain areas. However, like the response inhibition hypothesis, this idea is not consistent with more recent evidence from studies that directly tested its predictions.

For example, only 25% of cue-selective neurons in the OFC reverse encoding³³. During reversal learning, cue-selective neurons in many other brain regions reverse firing much more rapidly and in greater proportions than do OFC neurons. In the rat basolateral amygdala, for instance, 55–60% of cue-selective neurons reverse firing³², and similar findings have been reported in the monkey amygdala³⁴. Moreover, there is no relationship between cue preference or the degree of cue selectivity before and after reversal in OFC neurons. Thus, reversal of encoding in the OFC is neither uniquely rapid nor particularly pervasive.

These results are contrary to the basic version of this proposal in which the OFC is simply quicker or more efficient than other areas at storing new associations. An alternative version of this proposal is that storage in the OFC might be more functional — owing to connectivity or for some other reason — than similar information storage in the amygdala or elsewhere. Importantly, this version would not require that associations be stored more rapidly in the OFC, but it would predict a close relationship between reversal of encoding in the OFC and reversal of behaviour. In other words, reversal of encoding in OFC neurons should be associated with rapid reversal learning. But we have observed precisely the opposite relationship; that is, rats reverse their behaviour more slowly when there is rapid reversal of encoding in the OFC³³. This, together with other evidence presented below³⁵, is inconsistent with the rapidly flexible encoding hypothesis.

The OFC signals outcome expectancies

If the OFC does not have a special role in response inhibition and is not better than other regions at encoding new associative information, then what is its crucial function in flexible behaviour? We suggest that this function involves signalling outcome expectancies. Specifically, we suggest that the OFC signals the predicted characteristics — such as sensory properties (size, shape, texture and flavour) and perhaps even contextual cues such as particular timing or likelihood — and unique value of specific outcomes that an animal expects given particular circumstances and cues in the environment³.

Evidence from neuronal recording studies

Evidence in support of this proposal comes from neuronal recording studies that demonstrate anticipatory firing in OFC neurons. Although these anticipatory signals are also apparent before other events, they are particularly strong in the OFC before the animal experiences primary rewarding or punishing events. For example, during discrimination learning, distinct populations of neurons in the rat OFC develop selective firing in anticipation of delivery of sucrose or quinine³⁶. Often, these neurons initially fire in response to one or the other outcome, then come to fire in anticipation of that outcome and finally also fire in response to cues that predict the outcome (FIG. 2a). Unlike reward-responsive dopamine neurons, which also transfer activity from outcomes to predictive events^37–41, these OFC neurons do not stop firing in response to the reward after learning⁴², nor do they show stronger (or weaker) activity when reward is delivered (or omitted) unexpectedly⁴³ (FIG. 2b). Therefore, their activity is not well described as a prediction error signal (see also below); rather, activation of these neurons during progressively earlier periods in the trial could be considered as a representation of the expected outcome.

Figure 2. Stylized plots of reward-responsive neurons in the orbitofrontal cortex.

a | In discrimination learning tasks, neurons in the rat orbitofrontal cortex (OFC) initially fire in response to either the rewarding reinforcer (for example, sucrose) or the aversive reinforcer (for example, quinine) (dark blue line). After a number of trials the neurons also start to fire in anticipation of the reinforcer (red line). Finally, they also come to fire in response to cues that predict the reinforcer (purple line). b | These OFC neurons do not stop firing in response to the reward, even after many trials. Moreover, their firing is not stronger in response to an unexpected reward or weaker in response to the omission of a reward left panel). In this respect, OFC neurons are unlike dopamine (DA) neurons in the ventral tegmental area (VTA), which are also reward responsive but which fire more strongly in response to unexpected rewards and decrease firing when an expected reward is not delivered (right panel). Based on data from REFS ³⁶–⁴³.

Similar findings have been reported in monkeys and humans^44–50. For example, Schultz and colleagues showed that OFC neurons exhibit outcome-expectant activity during a visual delayed response task⁵⁰. The strength of anticipatory firing depended on which particular food reward the monkey expected to receive: it was stronger when preferred foods were expected. Thus, anticipatory activity reflects both the physical and/or sensory attributes of the expected food reward and the subjective value that the monkey places on the item.

Anticipatory firing has also been observed in other areas^{34,36,44,47,51–53}, but it occurs first in the OFC^36,44. Furthermore, in other areas — the basolateral amygdala in particular — it is reduced to near chance levels by (even unilateral) lesions of the OFC⁵⁴. These results suggest that, to some extent, OFC neurons construct predictions on the basis of afferent input rather than simply passing on information received from other brain areas. Indeed, neural activity in the OFC of an animal performing a task is striking in that it typically anticipates trial events rather than being triggered by them⁵⁵. Each predictable event holds value because it provides information about the probability or proximity of reward. Thus, signalling expected outcomes could be considered a more general property of the OFC if ‘outcomes’ are broadly construed to include not only primary rewarding or punishing events but also events that have acquired value or meaning because they provide information about the likelihood of receiving such primary reinforcers in the future.

In this regard, such signalling would be conceptually similar to the ‘state value’ defined by temporal difference reinforcement learning models⁵⁶, which reflects the value of all current and future rewards predicted by cues and circumstances that are present in the environment. Indeed, even the neural response to an actual reward might reflect both the reward’s current hedonic properties and its future nutritive (or other) consequences (for example, a reward can also serve as a sensory cue that predicts a subsequent reward).

Evidence from behavioural studies

Of course, the neural firing patterns described above are only correlates of behaviour. If the OFC is crucial for signalling outcome expectancies, then manipulations that disrupt or alter the output from this brain area should preferentially disrupt behaviours that depend on information about outcomes. This turns out to be an excellent description of most OFC-dependent behaviours. For example, as rats learn in the odour discrimination task described above, they begin to respond faster (that is, approach the well faster) after sampling a positive predictive odour (that is, when they expect sucrose) than after sampling a negative predictive odour (that is, when they expect quinine)¹⁶. This behavioural differentiation emerges at the time that neurons in the OFC begin to show differential firing in anticipation of sucrose or quinine³⁶ and, unlike changes in choice performance (the decision to go or not to go to the well), changes in the latency of the behavioural response are abolished by OFC lesions¹⁶.

A second and perhaps more conclusive example of the crucial role of the OFC in using information about expected outcomes comes from studies using Pavlovian reinforcer devaluation. In these studies, an animal is trained to associate a cue with a particular reward. Subsequently, the value of the reward is reduced by pairing it with illness or by feeding it to satiety. The animals’ ability to access and use that new value to guide the learned responding is then assessed by presenting the cue alone. Animals normally show a reduced response to the predictive cue, reflecting their ability to access and use cue-evoked representations of the reward and its current value. However, monkeys and rats with OFC lesions fail to show this effect of devaluation^12,22,57. The deficit caused by OFC damage is evident even if the OFC is intact during the initial training and devaluation^24,25; therefore, the OFC is necessary for mobilizing information about the value of the expected outcome to guide or influence responding. This observation is consistent with data from rats, monkeys and humans showing that neural activity in the OFC changes as a result of satiation^45,58–60. The role of the OFC in encoding and using information about expected outcomes in this setting differs from that of the amygdala or the mediodorsal thalamus, which are necessary during the earlier phases of the reinforcer devaluation task^25,61–64.

A third example comes from Pavlovian-to-instrumental transfer experiments. These involve a task in which animals are trained to independently associate a cue and a behaviour (for instance, pressing a lever) with a reward. Subsequently, they will increase this behaviour in the presence of the cue. This increased responding is thought to reflect the effect of Pavlovian information, triggered by cue presentation, on the behaviour (that is, lever pressing), which is controlled by instrumental representations. Consistent with the proposal that the OFC is crucial for signalling outcome information triggered by cues, Balleine and colleagues reported that outcome-specific (but not general) Pavlovian-to-instrumental transfer is abolished by OFC lesions⁶⁵.

These examples indicate that one fundamental role of neural activity in the OFC is to drive behaviour on the basis of the specific features of expected outcomes. Other examples of OFC-dependent behaviours corroborate this account. These include delayed discounting^66–68, conditioned reinforcement and other second-order behaviours^26,69–71, Pavlovian approach responses¹³, the enhancement of discriminative responding by different outcomes⁷², and even cognitive and affective processes, such as regret and counterfactual reasoning⁷³. In each of these examples, normal performance can be plausibly argued to require the ability to signal, in real-time, information about the characteristics and values of outcomes predicted by cues and circumstances in the environment.

Of note, the brain areas that mediate these types of value-guided behaviour are also typically important for reversal learning^12,16,22,65. One exception to this general rule is described in a recent study by Bachevalier and colleagues that showed that reversal learning was unimpaired in monkeys with lesions of areas 11 and 13 (REF. ²⁷); the same monkeys were previously reported to exhibit deficits in behavioural changes after reinforcer devaluation. Although this report indicates that reversal learning can be mediated by OFC-independent mechanisms, it does not necessarily contradict earlier findings more consistent with colocalization of reversal learning and devaluation in the OFC. OFC-dependent reversal deficits are transient and disappear quickly with practice^74,75. Indeed, we can completely eliminate OFC-dependent reversal deficits by damaging other brain areas, whereas deficits in behaviours (latencies to respond) that are probably crucially dependent on value representations persist^35,76. This is relevant because these monkeys were tested ~18 months after surgery and had undergone extensive training in various tasks before reversal testing²⁷. In any event, although this report²⁷ raises important questions about reversal learning and about whether this process depends on different brain regions, it does not necessarily contradict the basic idea presented in the next section, that adaptive behaviour in general — epitomized by reversal deficits — can be supported by the same information as that used in devaluation or in other value-guided tasks; it merely confirms that the behaviour does not have to be, which is a topic that we address later.

Expectancies and flexible behaviour

The OFC-dependent behaviours described above differ from flexible behaviour that can be assessed by reversal learning or related tasks because they generally do not involve changes in established associations or response contingencies. For example, OFC inactivation impairs changes in conditioned responding after devaluation, even though nothing about the underlying associations has changed and only the value of the outcome has been altered. Similarly, OFC damage disrupts Pavlovian-to-instrumental transfer even when lesions are made after rats have learned the underlying cue–reward and response–reward associations. These data suggest that signals from the OFC that relate to expected outcomes influence judgment and decision making when contingencies are stable.

But how does signalling of expected outcomes account for the role of the OFC in modifying behaviour when contingencies have changed? One possibility is that these signals are also crucial for updating associative representations in other brain regions in the face of unexpected outcomes. The OFC might play this part by contributing information that is necessary for calculating the teaching signals that are thought to drive learning in such situations⁵⁶. These teaching signals (also known as prediction errors) result from differences between the value of the expected outcome and that of the actual outcome. There is strong evidence that these prediction errors are signalled by phasic activity in dopamine neurons in the substantia nigra and ventral tegmental area^37–41, and activity consistent with error signalling has also been observed in other brain areas^77–80. Although OFC neurons do not signal prediction errors themselves⁴³, the OFC might still contribute to the signalling of these prediction errors by providing a source of information (although not the only source; see below) about the expected outcome value; this information could then be used to compute prediction errors (FIG. 3). The loss of this function would explain why OFC damage disrupts behavioural flexibility under conditions of changing contingencies.

Figure 3. The proposed role of the orbitofrontal cortex in flexible, adaptive behaviour.

A summary of several proposals reviewed in the text for explaining the role of the orbitofrontal cortex (OFC) in facilitating reversal learning. At the top is the historical proposal that output from the OFC directly inhibits ‘pre-potent’ responses and is therefore crucial for reversal learning. Below is the more recent idea that the OFC functions as a highly flexible associative reference table to directly guide correct responding and so is vital for reversal learning. At the bottom is our alternative proposal that the OFC is crucial for reversal learning because it drives associative learning in other structures. According to this proposal, the OFC does this by supporting the generation of teaching signals (for example, by the ventral tegmental area (VTA)) when actual outcomes do not match signals from the OFC regarding expected outcomes. BLA, basolateral amygdala; mPFC, medial prefrontal cortex.

Predictions and implications

Our hypothesis makes a couple of testable predictions. The first is that changes in associative representations in downstream regions, such as the amygdala, should be at least partly dependent on the OFC. Consistent with this, associative encoding in the basolateral amygdala is markedly less flexible in rats with lesions of the OFC⁵⁴. moreover, lesions or inactivation of the basolateral amygdala abolish the reversal deficit caused by OFC lesions³⁵. This result suggests that this downstream (amygdala) inflexibility underlies the deficit in reversal behaviour that results from OFC lesions. As recoding of associations in the OFC occurs more slowly than does recoding in the basolateral amygdala³², these results cannot be easily explained by the ‘rapidly flexible encoding’ hypothesis of OFC function. However, they are fully consistent with the proposal that signalling of old associations by OFC neurons drives reversal learning by supporting error signalling in some other region (for example, the ventral tegmental area; see below). These error signals would then facilitate the encoding of new associations by downstream areas such as the amygdala. The better the OFC signals the old associations, the stronger the teaching signals and the faster the recoding process. This proposal also fits with the finding, discussed earlier, that better reversal behaviour occurs when cue selectivity in the OFC fails to reverse³³.

The abolition of the OFC-dependent reversal deficit by lesions or inactivation of the basolateral amygdala also implies that the OFC–amygdalar system is not actually required for normal reversal learning. This somewhat unconventional statement makes sense when one considers that reversal learning probably involves changes in encoding of various types of associative information maintained in different brain regions; the speed of reversal learning will therefore depend in part on the task and the type of associative strategy it emphasizes, as well as on the rate at which information in different brain regions is updated (with the ‘slowest’ regions determining the speed of reversal learning). normally, the basolateral amygdala changes its representations very rapidly^32,34, and so is probably not the rate-limiting step. Consistent with this, damage to the basolateral amygdala does not affect reversal learning, at least not in the sorts of task often used to assess OFC-dependent reversal learning^16,27,81. However, when the OFC is lesioned, representations in the basolateral amygdala are slow to change, and this area becomes an impediment to the speed of reversal learning. In other words, damage to the OFC introduces an artificial bottleneck that slows learning. When the basolateral amygdala is damaged or inactivated in an OFC-lesioned rat, reversal might go back to being controlled by whatever area normally governs its rate.

A second prediction of our hypothesis is that the OFC should be important for learning in other situations, when error signals are crucially dependent on information about expected outcomes. Results from a Pavlovian over-expectation task are consistent with this prediction. In this task, rats initially learn that several Pavlovian cues are each independent predictors of a reward. Then, two of these cues are presented together (compound training), followed by the same reward. When either of these cues is subsequently presented alone, rats exhibit a spontaneous reduction in their conditioned response. This reduced response is thought to result from negative prediction errors induced by violation of the animal’s ‘summed’ expectation for the reward. Reversible inactivation of the OFC during compound training prevents summation and the later reduction in response to the individual cues⁴³. This result cannot be explained as a simple deficit in using associative information encoded by the OFC, because the OFC is fully functional when the use of new information is assessed. Instead, it suggests that the OFC is crucial for the prediction-error-induced learning. Notably, disconnecting the OFC from the ventral tegmental area during compound training in the over-expectation task also abolishes the response reduction⁴³. This indicates that the role of the OFC in supporting learning in this setting requires interaction with dopamine neurons in the ventral tegmental area.

Of course, the OFC is not required for all learning; clearly, other brain areas can acquire new representations without input from the OFC. Indeed, much learning — including initial discrimination learning and simple Pavlovian learning — is unaffected by OFC damage²². One possible explanation for this preserved learning is that these situations, which involve new learning, do not require expectancies for the production of error signals. If anything, expectancies would actually impede learning by reducing the size of the teaching signal as learning progressed. So it is not surprising that the OFC is not necessary for all learning.

Moreover, it is likely that the OFC is not the only source of information about expected outcomes. Below, we suggest that this role of the OFC in driving learning may be constrained by the circuits in which it is embedded and by the particular role that these circuits have in particular categories of associative learning. As we describe, outside these domains learning may not depend on signalling from the OFC.

Conclusions and future directions

Recent evidence contradicts long-standing ideas that the role of the OFC in adapting behaviour in the face of changing contingencies and unexpected outcomes reflects response inhibition or rapid flexibility of associative encoding. Instead, we suggest that it reflects contributions of the OFC to changing associative representations in other brain regions, mediated indirectly through the support of prediction-error signalling by systems such as the midbrain dopamine system. This is consistent with evidence that the OFC is crucial for signalling outcome expectancies.

This proposal is not unlike ideas about the role of the ventral striatum in actor–critic models of reinforcement learning⁸². According to these theories, the ventral striatum serves as the critic, providing information necessary to compute a ‘state value’, which is, in turn, required by downstream areas such as the midbrain to calculate reward prediction errors. These errors serve as teaching signals to facilitate the acquisition of rules or policies in other regions — such as the dorsolateral striatum⁸³ — that serve as the actor to drive behaviour. This idea of state value is comparable to the outcome expectancies that we suggest are signalled by the OFC. Thus, another way to view our proposal is that the OFC functions as a critic.

Importantly, we do not argue that the OFC supplants the ventral striatum in this regard. Rather, we suggest that the OFC might act in parallel or in series with the ventral striatum and perhaps other regions, such as the amygdala or the medial prefrontal cortex, in signalling this sort of information. Similarly to the fact that there are different systems for different kinds of memory, there are likely to be multiple critics, each providing a particular type of information relevant to computing state value.

For example, it is now clear that even simple Pavlovian conditioning results in the formation of multiple associations in the brain, each capturing different aspects of the cue–outcome pairing⁸⁴. As a result, cues become able to evoke a representation of the outcome, including its particular sensory features and its unique, changing value. In addition, cues become linked to the general emotion associated with the outcome, which that outcome shares with other, similar outcomes. Analogous distinctions have been drawn for goal-directed versus habitual behaviour in instrumental learning⁸⁵.

These different representations support different types of behaviour and depend on somewhat non-overlapping neural circuits. As described above, signalling of information about specific outcomes depends on a circuit that includes the OFC and the basolateral amygdala. By contrast, signalling of general affective information depends on other areas, such as the nucleus accumbens in the ventral striatum⁸⁶.

We suggest that each of these regions might function as a relatively independent critic, depending on what type of expectancy is being violated. Input from the OFC may be particularly important when the predicted outcome changes. This would be consistent with recent data showing that the OFC is necessary for learning when one outcome is substituted for another²⁶. On the other hand, input from regions that signal general affective information might be crucial for learning when the general affective value of the outcome, rather than its particular features, are altered — for example, when an animal’s internal motivational state changes after learning.

Similarly, the OFC might be particularly important in signalling Pavlovian information, providing the crucial predictions and playing a lesser or perhaps inconsequential part in signalling information about expected outcomes derived from instrumental or action–outcome associations. This proposal has been advanced by Balleine and colleagues on the basis of observations that OFC damage affects changes in Pavlovian responding but not changes in instrumental responding after reinforcer devaluation⁶⁵. Although neurophysiological and imaging results show action–outcome correlates in neural activity in the OFC, these could reflect sensory information that differs between different responses^87–89. If the OFC has a crucial role in signalling only cue-evoked or Pavlovian information, then some other area would have to provide instrumental information. Leading candidates would be parts of the medial prefrontal cortex and the dorsomedial striatum, which, Balleine and colleagues have shown, are crucial for goal-directed behaviour^90,91.

Inputs from multiple critics might converge on midbrain areas through direct or indirect connections. For example, the ventral striatum receives input from the OFC and the medial prefrontal cortex; thus, the ventral striatum could function as a super-critic, integrating information and sending it off to be used in error signalling. Alternatively, this information could reach the midbrain through direct projections. Direct and indirect pathways might both be used, with each influencing learning in subtly different ways. Of course, it is also possible — indeed likely — that different types of information interact with different error-signalling regions. There are several regions (other than the midbrain dopamine neurons) that seem to signal prediction errors.

Importantly, proposals regarding the differing contributions of the OFC and other brain areas to error encoding are testable. Functional MRI and single-unit studies combined with lesions or inactivation and with behavioural tasks that manipulate expectancies based on the specific and general values of actions and Pavlovian cues can directly confirm or invalidate our hypothesis. Although substantial experimental evidence supports the ideas laid out in this Perspective, further work holds the potential to sketch out this circuit and its functions in more detail, hopefully leading to the creation of a truly useful framework for how brain circuits implement the simple associative-learning processes that help us to navigate our ever-changing world.

Glossary

Conditioned reinforcement

The process by which a Pavlovian-acquired value can reinforce instrumental action

Delayed discounting task

A task that assesses the effect of delaying a reward on a subject’s choice behaviour

Detour reaching task

A task that assesses the ability of a subject to learn to reach around a clear barrier in order to retrieve a reward

Instrumental

Related to a training situation in which associations are arranged between actions and outcomes

Pavlovian

Related to a training situation in which associations are arranged between cues and outcomes

Pavlovian approach responses

Approaches to a conditioned stimulus

Reinforcer devaluation task

A task that assesses the ability of a subject to modify conditioned responding after the expected reward is devalued, typically by pairing it with illness or by feeding it to satiety

Stop signal task

A task that assesses the ability of a subject to stop or inhibit an ongoing reward-seeking behaviour in response to a ‘stop’ signal

Temporal difference reinforcement learning models

A family of non-trial-based reinforcement learning models in which the difference between the expected and actual values of a particular state in a sequence of behaviours is used as a teaching signal to facilitate the acquisition of associative rules or policies to direct future behaviour

Visual delayed response task

A task that assesses the ability of a subject to respond to visual cues to obtain a reward that is presented after a short delay