Reconciling the Roles of Orbitofrontal Cortex in Reversal Learning and the Encoding of Outcome Expectancies

Abstract

Damage to orbitofrontal cortex (OFC) has long been associated with decision-making deficits. Such deficits are epitomized by impairments in reversal learning. Historically, reversal learning deficits have been linked to a response inhibition function or to the rapid reversal of associative encoding in OFC neurons. However here we will suggest that OFC supports reversal learning not because its encoding is particularly flexible—indeed it actually is not—but rather because output from OFC is critical for flexible associative encoding downstream in basolateral amygdala (ABL). Consistent with this argument, we will show that reversal performance is actually inversely related to the flexibility of associative encoding in OFC (i.e., the better the reversal performance, the less flexible the encoding). Further, we will demonstrate that associative correlates in ABL are more flexible during reversal learning than in OFC, become less flexible after damage to OFC, and are required for the expression of the reversal deficit caused by OFC lesions. We will propose that OFC facilitates associative flexibility in downstream regions, such as ABL, for the same reason that it is critical for outcome-guided behavior in a variety of setting—namely that processing in OFC signals the value of expected outcomes. In addition to their role in guiding behavior, these outcome expectancies permit the rapid recognition of unexpected outcomes, thereby driving new learning.

Orbitofrontal cortex (OFC) has long been implicated in cognitive flexibility.^1,2 The loss of this ability is evident in impairments in rapid reversal learning that are observed after OFC damage in a variety of species.^1,3-13 In these settings, animals are first taught to respond to one cue to receive reward and to withhold or inhibit a similar response to avoid punishment or non-reward. After animals are responding correctly based on these contingencies, the meaning of the cues is reversed, such that animals must respond to the cue to which they had previously withheld responding and withhold responding to the cue to which they had previously responded. Animals with OFC damage typically acquire the initial discrimination normally but require many more trials than controls to relearn the discrimination after the cue-outcome associations are reversed. This deficit has been taken as evidence that OFC promotes flexible responding. Yet why is OFC important for this function?

THE ORBITOFRONTAL CORTEX AND RESPONSE INHIBITION

The role of OFC in promoting reversal learning has been explained historically as a result of the involvement of OFC in inhibiting responses.^1,2 While this has been proposed to be a general prefrontal function, reports often identify disinhibited, perseverative, or impulsive responding as hallmarks of the “orbitofrontal syndrome.” Thus, much of the behavior of orbitofrontal damaged humans has been conceptualized as an inability to inhibit so-called “prepotent” responses. To the extent that this is more than just a restatement of the deficit, this proposal suggests that output from OFC somehow directly inhibits expression of the response, as illustrated in Figure 1A. Unfortunately this account is contradicted by the observation that, in most of the discrimination reversal studies in which OFC lesions impair reversal, lesioned animals are fully capable, before reversal, of withholding a response that is identical to the one that they are unable to withhold after reversal. This is particularly true in go, no-go tasks, in which this initial response is often highly ingrained. For example, in our studies the rats are shaped for many hundreds of trials to respond for reward on every trial. Yet rats with OFC lesions learn to inhibit this response at the same rate as controls when presented with a series of discrimination problems, both on the initial and subsequent problems. Only after reversal are these rats unable to withhold responding.

FIGURE 1.

Schematic summarizing proposed roles for the OFC in reversal learning. (A) Historical proposal that output from OFC directly inhibits “prepotent” responses and thus is critical for reversal learning. (B) More recent idea that OFC acts as a highly flexible associative look-up table to guide correct responding and thus is critical for reversal learning. (C) Our proposal that OFC is critical for reversal learning because it facilitates associative learning in other structures, by facilitating the generation of teaching signals when actual outcomes do not match OFC signals of expected outcomes. DS, dorsal striatum; VS, ventral striatum; DA, dopamine; 5-HT, serotonin.

Moreover, animals with OFC lesions are also able to inhibit responses normally in many other settings. For example, although OFC-lesioned monkeys and rats fail to suppress cue-evoked conditioned responses after reinforcer devaluation, they do inhibit consummatory responses like controls.^8,14,15 In these tasks, the animals are trained to associate cues with rewards. If the rewards are devalued by overfeeding or pairing with illness, normal animals exhibit reduced responding to the cues that predict those rewards. OFC-lesioned animals fail to show this decline in conditioned responding. However, they are able to suppress actual consumption of the devalued food item after this critical probe test. Further, during the probe test, which is conducted under extinction conditions, lesioned rats show normal extinction of conditioned responding. Thus while OFC-lesioned rats fail to suppress conditioned responding as a result of prior devaluation of the outcome, they are able to suppress conditioned responding as a result of non-reward. In fact, it has recently been reported that rats with OFC lesions can even suppress instrumental responses normally after reinforcer devaluation.¹⁶

Furthermore, it has recently been shown that OFC-lesioned monkeys are able to inhibit prepotent responses normally in the so-called reversed reward contingency task.¹⁷ In this task, the monkeys were asked to select the smaller of two available food quantities in order to obtain presentation of the larger quantity. Thus monkeys were presented with a choice between one half peanut or four half peanuts. In order to obtain the four half peanuts, the monkeys had to select the one half peanut. This requires the monkeys to suppress their natural tendency to select the thing that they want to receive, which is the larger of the two rewards. OFC-lesioned monkeys learned to do this at the same rate as controls.

These studies show that there are circumstances in which OFC is not required for response inhibition. Thus while there may be situations in which OFC promotes response inhibition, response inhibition per se is not the underlying function of this area. In other words, disinhibited and impulsive responding after OFC lesions is a symptom, rather than an explanation.

THE ORBITOFRONTAL CORTEX AS AN ASSOCIATIVE LOOK-UP TABLE

More recently, the role of OFC in promoting reversal learning has been linked to the finding that cue-selective neurons in OFC reverse firing selectivity for cues during reversal learning.^18-20 Reversal of cue-selective activity in OFC neurons presumably reflects acquisition of the reversed associations. An example of such flexible associative encoding is given in Figure 2A, which shows a single neuron recorded in OFC in a rat learning and reversing a novel odor discrimination problem. During initial learning, the neuron becomes selective to odor two, which predicts quinine, then after reversal, the same neuron switches to fire to odor one now that this odor predicts quinine. Similar correlates are also observed for the positive odor cue. When these neurons were first reported by Rolls and colleagues in great numbers in OFC,¹⁹ flexible associative encoding had not been prominently demonstrated in other brain areas. The apparent uniqueness of these correlates, in combination with the close correspondence between their apparent function and the effects of OFC damage, led to the readily accepted proposal that the critical contribution of OFC to behavior was to be a rapidly modifiable associative look-up table of sorts. By this account, illustrated in Figure 1B, the reversed associations would be learned most rapidly in OFC and then employed to drive the behavioral reversal.

FIGURE 2.

Flexibility of associative encoding in OFC is inversely related to speed of reversal learning. (A) Example of associative encoding in a rat OFC neuron recorded during acquisition and reversal of a two-odor discrimination problem. Neural activity is shown synchronized to odor onset on each trial, in both raster and peri-event time histograms, for trials before learning (pre-criterion), after learning (post-criterion), and after reversal (reversal). Before reversal, odor 1 predicts sucrose, and odor 2 predicts quinine. The neuron exhibits a phasic response during odor sampling that develops with learning and tracks the predicted outcome (quinine) across reversal. (B) Average trials required to attain criterion on the reversal in sessions in which reversing cue-selective neurons were recorded versus sessions in which cue-selective neurons were recorded that did not reverse. Rats performed significantly worse in sessions with reversing neurons. (Data adapted from Stalnaker et al.²²) (In color in Annals online.)

However more recent neurophysiological findings are inconsistent with this hypothesis. Specifically, though single-unit recording studies in rats and primates have shown that about a quarter of cue-selective OFC neurons reverse their stimulus selectivity after reversal of the stimulus-outcome associations,^18,20,21 the population of cue-selective neurons actually becomes non-selective after reversal and is, in effect, replaced by the emergence of a new cue-selective population. This is illustrated in Figure 3A, which shows the effect of reversal of a two-odor discrimination problem on the firing of OFC neurons selective for each odor cue during learning. Both populations fire selectively during sampling of their respective cue before reversal but not after reversal. Loss of selectivity is also shown graphically for each neuron in the two populations by the inset scatter plot in Figure 3A. This figure plots a cue-selectivity index for each neuron before and after reversal. This calculation consisted of the difference in firing rate to the odor cues divided by the sum of those rates. If a population of neurons maintained the same cue-selectivity before and after reversal, one would expect a regression with a positive correlation. On the other hand, if a population of neurons reversed cue-selectivity, one would expect a regression with a negative correlation. However, this plot shows that cue-selective neurons in OFC are actually equally likely to fire to either odor cue after reversal. As a result, there is no correlation between the cue-selectivity indices of these OFC neurons before and after reversal. This encoding pattern is at odds with the proposal that OFC rapidly encodes reversals of cue-outcome associations.

FIGURE 3.

Flexibility of associative encoding is greater in ABL than in OFC. Population response of neurons in OFC (A) and ABL (B) identified as cue-selective during learning. Average activity per neuron is shown, synchronized to odor onset, during and after reversal. The population response reverses cue-selectivity in ABL but not in OFC. Inset scatterplot compares the cue-selectivity indices before (X-axis) and after reversal (Y-axis) for all the cue-selective neurons used to construct the population histogram. Blue and red symbols show data for “Odor One Selective” neurons and “Odor Two Selective” neurons, respectively. The cue-selectivity indices are inversely correlated in ABL, consistent the reversal of cue-selectivity, whereas they show no correlation in OFC. (Data adapted from Stalnaker et al..²²,³⁶) (In color in Annals online.)

Furthermore, we have recently reported that the probability of observing reversal of cue-selectivity in OFC neurons is actually inversely related to the rate of reversal learning.²² This relationship, illustrated in Figure 2B, is inconsistent with the notion that flexible encoding in OFC drives behavioral reversal, since if it did one would expect the opposite relationship (i.e., better performance when reversing neurons are plentiful). Instead, this relationship suggests that reversal of cue-selectivity in OFC reflects feedback regarding discrepancies between expected and actual outcomes. Such feedback would be greatest when reversal performance is poor. Indeed, OFC BOLD response is sensitive to violations of expected outcomes, and a small number of neurons in OFC respond strongly when errors are detected.^23-25

Finally, recent work shows that associative encoding in other brain regions can be more flexible than that in OFC. The best example of this is activity in basolateral amygdala (ABL), a region that receives strong projections from OFC.^26-28 ABL is critical for associative learning,^29-32 and neurons in ABL rapidly become selective to cues that predict biologically relevant appetitive or aversive outcomes.^18,33-35 An analysis of the effect of reversal on cue-selectivity in ABL^18,34,36 is presented in Figure 3B. This figure, which parallels the analysis of OFC neurons in Figure 3A, shows the effect of reversal of a two-odor discrimination problem on the firing of ABL neurons selective for each odor cue during learning. Both populations switch their cue-selectivity across reversal, thereby rapidly encoding the reversed cue-outcome associations. This pattern is also evident in the inset scatterplot, which plots the cue-selectivity index for each neuron before and after reversal. These values were inversely correlated across the population, consistent with a reversal of cue-selectivity. Comparison of data in Figure 3A and B is particularly revealing, showing that though OFC has received the greatest attention for the flexibility of its neural correlates, encoding in ABL neurons actually appears to provide a more rapidly flexible associative look-up table. Importantly this impression has been confirmed in primates in the past year in an elegant study by Salzman and colleagues showing that primate amygdala neurons reverse cue-selectivity in large proportions during reversal learning.³⁷

THE ORBITOFRONTAL CORTEX AND ENCODING OF OUTCOME-EXPECTANCIES

So what then is the contribution of OFC to flexible responding in situations such as reversal learning? An alternative explanation for the role of OFC in promoting flexible responding is found in the neural activity in OFC that anticipates expected outcomes.^25,38-45 This is evident across different species and tasks; neural activity or BOLD changes in OFC are often triggered by cues and events that convey information about the value of impending outcomes.^{25,38,40,43,44,46-51} (Note that this is different from prediction error signaling thought to characterize dopaminergic activity.⁵²) The importance of OFC for signaling such information is evident in lesion studies in which damage to OFC causes selective deficits in the ability of animals to use information about the value of expected outcomes to guide existing or established behavior. For example, as described earlier OFC-lesioned animals fail to modify responding to cues after reinforcer devaluation.^8,14,15 Deficits in these settings, which do not involve new learning, likely reflect a critical function for OFC in modulating the expression of previously acquired associative information, perhaps in downstream areas such as the basal ganglia or amygdala.⁵³

We have proposed that neural encoding of expected outcomes by OFC might also provide a signal that could be compared to actual outcomes to drive new learning.⁵⁴ When an actual outcome fails to match expectations, this comparison could promote changes in old associative representations and the acquisition of new ones. Presumably this function would be helpful even in initial learning, but it might be particularly critical during reversals, in which the discrepancy between actual and expected outcomes is maximal. By this account, illustrated in Figure 1C, OFC would support reversal learning not directly due to its role in rapid encoding new associations but rather indirectly, by facilitating changes in associative encoding in other brain regions. Such old or persistent memories might otherwise slow or impede behavioral reversal. Perhaps the best candidate region for showing this effect is ABL; as outlined above, ABL is critically involved in associative learning, and associative encoding in ABL normally changes rapidly after reversal. This proposal makes at least two testable predictions:

1. The flexibility of associative correlates in ABL should depend on input from OFC.

2. Expression of the reversal deficit caused by OFC lesions should be mediated through ABL.

We will describe these predictions—and the data that addresses them—in the remaining sections of this chapter.

Flexibility of Associative Encoding in Basolateral Amygdala Depends on Input from Orbitofrontal Cortex

The first prediction of our hypothesis is that associative encoding in amygdala should depend on OFC. Consistent with this proposal, we have recently reported that cue-selective firing in ABL neurons fails to reverse in rats with unilateral OFC lesions.³⁴ These data are illustrated in Figure 4, which shows the effects of reversal of a two-odor discrimination on the firing of cue-selective ABL neurons recorded in rats with ipsilateral neurotoxic lesions of OFC. Because the connections between OFC and ABL are largely (though not completely) ipsilateral, this lesion would have substantially diminished the influence of OFC on ABL in the lesioned hemisphere while leaving processing in the intact hemisphere largely unaffected. This manipulation caused associative correlates in ABL to become both less prevalent and, when followed across reversal, significantly less flexible. This is evident in the population histograms, in which neither population of cue-selective neurons shows any selectivity to the odor cues after reversal, and in the scatterplot, where there was no correlation between cue-selectivity before and after reversal in the cue-selective neurons. Thus removal of OFC dramatically reduced the flexibility of encoding in ABL.

FIGURE 4.

Flexibility of associative encoding in ABL depends on input from OFC. Population response of cue-selective neurons in ABL in rats with ipsilateral lesions of OFC. Average activity per neuron is shown, synchronized to odor onset, during and after reversal. Unlink the populations recorded in intact rats, illustrated in Figure 3B, the population response recorded in OFC-lesioned rats does not reverse cue-selectivity. Inset scatterplot compares the cue-selectivity indices before (X-axis) and after reversal (Y-axis) for all the cue-selective neurons used to construct the population histograms. Blue and red symbols show data for “Odor One Selective” neurons and “Odor Two Selective” neurons, respectively. Again in contrast to the inverse correlation in intact rats, illustrated in Figure 3B, the cue-selectivity indices in OFC-lesioned rats showed no correlation, indication that ABL neurons in OFC-lesioned rats had lost their tendency to reverse. (Data adapted from Saddoris et al.³⁴) (In color in Annals online.)

Of course associative encoding (and flexibility of that encoding) was not completely eliminated by OFC lesions. Obviously such preserved encoding could reflect input from the intact contralateral hemisphere or compensatory mechanisms in the lesioned brain. However the partially preserved encoding is also consistent with recent studies showing that some outcome-guided behaviors depend on ABL but not OFC. For example, ABL lesions cause deficits in fear conditioning,^30,55-57 but OFC lesions typically do not.⁵⁸ ABL lesions also abolish conditioned stimulus-potentiated feeding,^59,60 while OFC lesions do not.⁶¹ Thus ABL neurons may be able to encode associative information, albeit at a slower rate, even in the absence of signaling from OFC. This would be consistent with the idea that other brain areas may signal aspects of impending outcomes independent of contributions from OFC.

Reversal Deficits Caused by Orbitofrontal Lesions Are Mediated through Basolateral Amygdala

The second prediction of our hypothesis is that the reversal deficits caused by OFC lesions should be mediated by the inflexible correlates in ABL. In other words, if OFC normally promotes reversal by facilitating changes in associative encoding in ABL, then damage to ABL in OFC-lesioned animals should mitigate or even abolish the reversal impairment. Note that this prediction contrasts with the prediction that other ideas regarding OFC function would make. For example, if OFC normally promotes reversal directly either by inhibiting responding or by rapidly acquiring the reversed associations, then damage to these downstream areas should either worsen or have no effect on the OFC-dependent reversal deficit. To test this, we assessed reversal learning in rats with bilateral neurotoxic lesions of either OFC or ABL and rats with bilateral OFC lesions combined with bilateral ABL lesions or unilateral ABL lesions plus inactivation of contralateral ABL during reversal.⁶² The results are shown in Figure 5. Consistent with our hypothesis, we found that removal or inactivation of ABL completely abolished the reversal deficit caused by OFC lesions.

FIGURE 5.

Reversal impairments caused by OFC lesions are abolished by lesions or inactivation of ABL. Bars show the average number of trials (±SEM) required to retain and reverse a two-odor discrimination problem for controls, rats with OFC lesions, and rats with OFC lesions combined with bilateral ABL lesions or unilateral ABL lesions and inactivation of contralateral ABL via infusions of muscimol + baclofen immediately prior to reversal (red arrows). As expected, OFC lesions impaired reversal learning. This impairment was abolished by pre-training lesions of ABL or by inactivation at the time of reversal. ABL lesions alone had no effect. *Significant difference with P < 0.05. (Data adapted from Stalnaker et al.,⁶²) (In color in Annals online.)

Importantly, lesions of ABL in otherwise intact rats did not facilitate reversal learning, and neither did ABL lesions have any effect on the acquisition of the initial discriminations. This argues against an account by which ABL lesions corrected the reversal impairment by an effect on initial acquisition of the discriminations or by independently improving cognitive flexibility. The absence of effects of ABL lesions on reversal learning is consistent with our own prior work and other reports that have used fiber-sparing lesion techniques to examine the role of amygdala in reversal learning.^6,8,61,63 In these studies, damage to amygdala did not facilitate or impair reversal learning performance, although ABL damage did affect differential changes in response latencies, consistent with a role in encoding the associative information acquired with learning. (Note that OFC + ABL lesions do not fix the latency effects observed with either lesion in isolation—see supplemental material in Ref. 62.) The simplest explanation of these results is that changes in such associative encoding in ABL during reversal learning normally occur more rapidly than those in other regions (at least in paradigms similar to ours). As a result, removing ABL has little effect in otherwise intact rats. However, OFC lesions impair reversal of associative encoding in ABL, rendering it slow enough to retard the rate of behavioral reversal.

CONCLUSIONS

Damage to OFC has long been associated with deficits in adaptive decision making. Such deficits are epitomized by impairments in reversal learning. Here we have shown that reversal performance is actually inversely related to the flexibility of associative encoding in OFC. Further, we have demonstrated that associative correlates in ABL are more flexible than those in OFC during reversal learning, become less flexible after damage to OFC, and are required for the expression of the reversal deficit caused by OFC lesions. This is consistent with the proposal that OFC supports rapid reversal learning by facilitating associative flexibility in downstream regions, such as ABL, rather than due to any role in directly driving reversals either through a role in response inhibition or in rapidly encoding the reversed associations. This proposal is consistent with the idea that OFC is critical for signaling the value of expected outcomes. In addition to guiding behavior, these outcome expectancies permit the rapid recognition of unexpected outcomes, thereby driving new learning.