Orbitofrontal inactivation impairs reversal of Pavlovian learning by interfering with disinhibition of responding for previously unrewarded cues

Abstract

Orbitofrontal cortex (OFC) is critical for reversal learning. Reversal deficits are typically demonstrated in complex settings that combine Pavlovian and instrumental learning. Yet recent work has implicated the OFC specifically in behaviors guided by cues and the features of the specific outcomes they predict. To test whether the OFC is important for reversing such Pavlovian associations in the absence of confounding instrumental requirements, we trained rats on a simple Pavlovian task in which two auditory cues were presented, one paired with a food pellet reward and the other presented without reward. After learning, we reversed the cue-outcome associations. For half the rats, OFC was inactivated prior to each reversal session. Inactivation of OFC impaired the ability of the rats to reverse conditioned responding. This deficit reflected the inability of inactivated rats to develop normal responding for the previously unrewarded cue; inactivation of OFC had no impact on the ability of the rats to inhibit responding to the previously rewarded cue. These data show that OFC is critical to reversal of Pavlovian responding, and that the role of OFC in this behavior cannot be explained as a simple deficit in response inhibition. Furthermore, the contrast between the normal inhibition of responding, reported here, and impaired inhibition of responding during Pavlovian over-expectation, reported previously, suggest the novel hypothesis that OFC may be particularly critical for learning (or behavior) when it requires the subject to generate predictions about outcomes by bringing together or integrating disparate pieces of associative information.

Introduction

Orbitofrontal cortex (OFC) is critical for reversal learning. This has been demonstrated in rats, cats, mice, monkeys, and humans (Teitelbaum, 1964; Butter, 1969; Dias et al., 1996; Bechara et al., 1997; Ferry et al., 2000; Chudasama and Robbins, 2003; Fellows and Farah, 2003; Schoenbaum et al., 2003; Hornak et al., 2004; Izquierdo et al., 2004; Bissonette et al., 2008). Indeed, reversal deficits have come to epitomize the effects of damage to OFC.

More recently, work using very different behavioral paradigms has highlighted a role for OFC in the performance of behaviors that depend on Pavlovian associations, specifically associations between cues and outcomes. For example, OFC is critical for spontaneous changes in conditioned responding after reinforcer devaluation (Gallagher et al., 1999; Izquierdo et al., 2004; Pickens et al., 2005) and also for changes in the expression of outcome-specific Pavlovian-to-instrumental transfer (Ostlund and Balleine, 2007a) and in outcome-guided conditioned reinforcement (Burke et al., 2008). All of these settings illustrate a specific role for OFC in mediating Pavlovian representations, linking cues to the outcomes they predict. Consistent with this, recording studies in both rats and monkeys and imaging work in humans have shown that cue-evoked neural signals in OFC are particularly attuned to these associations (Thorpe et al., 1983; Rolls et al., 1996; Schoenbaum et al., 1999; Ramus and Eichenbaum, 2000; Gottfried et al., 2003; O’Doherty et al., 2003; Wallis and Miller, 2003; Roesch and Olson, 2004; Hampton et al., 2006; Morrison and Salzman, 2006; Padoa-Schioppa and Assad, 2006; Roesch et al., 2006). Indeed at least one recent report has suggested that when behaviors do not depend on cue-evoked information about outcomes, OFC may not even be involved (Ostlund and Balleine, 2007a, b).

Yet nearly all of the evidence linking OFC to reversal learning is derived from complex settings that combine Pavlovian and instrumental learning. This is because reversal deficits are typically demonstrated in discrimination settings in which instrumental responses are rewarded after presentation of one cue but not the other. Thus the role of OFC in reversal learning might be related to its role in guiding other Pavlovian behaviors or it might reflect another function, such as rule learning, required in these complex settings (Murray and Izquierdo, 2007).

Here we tested this simple question by inactivating OFC during reversal of a Pavlovian discrimination. This setting differs from the reversal tasks cited above in that there was no contingency between the animal’s response and reward delivery. As expected, OFC inactivation impaired the ability of the rats to reverse conditioned responding. However, somewhat surprisingly, the deficit reflected an inability of these rats to develop normal conditioned responding for the previously unrewarded cue. Inactivation of orbitofrontal cortex had no impact on the ability of the rats to inhibit responding to the previously rewarded cue. These data show that although orbitofrontal cortex is critical to reversal of Pavlovian responding, the role of orbitofrontal cortex in this function cannot be explained as a simple deficit in response inhibition. Notably OFC inactivation in these same rats impaired the normal inhibition of responding in a Pavlovian over-expectation task (Takahashi et al., 2009). As we will discuss, the contrast between the impairment reported in that study and the lack of effect of OFC inactivation on response inhibition here suggests that orbitofrontal signaling may be particularly critical when normal learning (or behavior) requires the subject to generate predictions about outcomes by bringing together or integrating disparate pieces of associative information.

Methods

Subjects

A total of 16 Long-Evans rats (Charles River Laboratories, Wilmington, MA), weighing between 320–450 grams, were housed individually and were placed on a 12 house light/dark schedule. During non-testing periods, all rats had ad libitum access to food and water but during behavioral testing, all rats were food restricted to 85% of their baseline weights. Testing was conducted during the light period of the rats’ cycles. All testing was conducted in accordance with NIH guidelines and was approved by the Animal Care and Use Committee at the University of Maryland, Baltimore.

Surgical procedures

Twenty-three gauge cannulae (Plastics One Inc., Roanoke, VA) were implanted in all rats bilaterally into orbitofrontal cortex (coordinates: 3.0 mm anterior to bregma, 3.2 mm lateral and 5.0 mm ventral; Paxinos and Watson atlas, 1998) during stereotaxic surgery to allow for infusions of a GABA-A/B agonist cocktail (Musicmol/Baclofen) (n=8) or saline for controls (n=8). Surgical and infusion protocols were identical to those used in previously (Schoenbaum et al., 2007; Takahashi et al., 2009). Rats were anesthetized with isoflurane and then placed into a stereotax. Here the head placement was adjusted to ensure that bregma and lambda were placed at the same level. A midline incision was made to expose the skull, burr holes were drilled over the OFC, and guide+dummy cannulae (Plastics One Inc.) were lowered and cemented to the skull. During non-infusion periods, dummy cannulae were left in place. Two rats in the experimental group lost one of their cannulae during the course of the experiment and were excluded from the study.

Apparatus

Testing was conducted in eight standard sized rat behavioral boxes from Coulbourn instruments (12″W × 10″D × 12″H) (Allentown, PA), enclosed in a sound resistant shell. One recessed food cup was placed in the center panel, 2 cm above the floor. Infared photocells placed inside the food troughs were used to measure conditioning behavior. Each food trough was connected to a feeder placed outside the behavior chamber. Each feeder was set up to deliver 45 mg sucrose pellets (Banana or Grape flavored pellets, Bio-Serv, Frenchtown, NJ). Infared illuminating lights placed above the food trough were used for the experimenter to detect cue periods recorded on DVDs. Auditory cues were used in the conditioning training. Speakers were mounted inside the behavior chamber were used to deliver noise cues, such as a white noise and a tone cue (4 kHz, ~76 dB). Additionally, a clicker auditory cue was used (2 Hz).

Pavlovian conditioning

Prior to training, all rats in the current study received two weeks of training in a Pavlovian over-expectation task (Takahashi et al., 2009). In this task four cues were used: three auditory cues (white noise, tone and clicker) and one visual cue (cue light). Rats received all four cues (30 seconds each) in a blocked design. In this design, each cue was presented eight times in their own respective block (four blocks, counterbalanced in terms of block) over 10 days. The average ITI between cues in a particular block was 2.5 min and between blocks was 5–10 min. Two cues (one visual and one auditory) terminated with the delivery of three pellets, noted as O1 (grape or banana flavored, Bio-Serv, Frenchtown, NJ). Additionally, a second auditory cue terminated with three pellets of the alternative flavored sucrose pellet, O2. The fourth cue served as a control cue and was paired with no food. Next, in compound conditioning, the visual cue and the auditory cue that led to the same outcome were presented together as a compound and led to the same three O1 pellets. The other two auditory cues were presented separately during this time and remained unchanged. These two cues subsequently served as the CS+ and CS− in this current study and were trained in an identical fashion as used previously. Note, during compound conditioning, some rats in the experimental groups received infusions of bacolofen/muscimol (see methods below). Additionally, rats received a probe test in the earlier study at the end of training where all cues were presented eight times each under extinction settings. After approximately three weeks between the completion of this testing, training for this current experiment began. During this time, the rats remained in their home cages and received periodic handling and ad lib food and water.

Reminder training was conducted over three days. During each daily session, the rats were presented with two of the auditory cues used in the previous training: a 30 second CS+, designated A1, which terminated with the delivery of three flavored sucrose pellets (same pellets that were paired with this cue previously) and a 30 second CS−, designated A2, which terminated with no reward. Each cue was presented 8 times in a blocked design (counterbalanced in terms of order, using the same ITI’s as listed above both within and between blocks). The specific cues (tone, white noise, and clicker) and food pellet rewards (banana and grape flavored sucrose pellets) were counterbalanced. By the third day, responding in both groups had returned to a ceiling similar to that observed in the initial training. There were no effects of prior experience on responding during reminder training in the current study. After three days of reminder training, we began reversal sessions. These sessions were identical to the reminder sessions except that the cue-outcome associations were switched such that A2 predicted reward and A1 predicted no reward. Training on this one reversal continued for five sessions.

OFC inactivation

Prior to each of the five Pavlovian reversal sessions, rats in each group were bilaterally infused with either a GABA agonist cocktail (experimental group, n=6) or saline (control group, n=8). Both musicmol, a GABA-A agonist (Sigma, St. Louis, MO) and bacolofen, a GABA-B agonist (Sigma, St. Louis, MO), were used as the inactivating agents. For each infusion, a 30 G injector cannula was inserted into each guide cannula. The injector extended 1 mm below the end of the guide. Injectors were connected to a Hamilton Syringe (Hamilton, Reno, NV) with PE20 tubing (Thermo Fisher Scientific, Inc., Waltham, MA). The syringe was placed in infusion pump (Thermo Fisher Scientific, Inc., Waltham, MA). Infusions into each hemisphere consisted of 103 ng of musicmol and 32 ng of bacolfen in 150 nl of saline, infused at a rate of 250nl/min. We have previously found that this concentration and procedure is sufficient to induce discrimination reversal deficits equivalent to those caused by neurotoxic lesions of OFC (Takahashi et al., 2009). Furthermore, larger amounts of fluorescently-conjugated muscimol injections into other prefrontal regions have produced significant spreads (approx. 0.5–0.7mm from cannula tip in the medial-lateral and dorsal-ventral axis, respectively)(Allen et al., 2008). After the infusion, the cannulae were left in place for two to three minutes to allow for proper diffusion of the drugs. Approximately five to ten minutes following the bilateral injections, rats were placed in the behavioral chambers for reversal sessions.

Histology

After the completion of the study, all rats were anesthetized and then infused with 150 nl of thionin (0.25%) using the same procedure used to infuse the inactivating GABA agents into the OFC. Rats were then perfused with saline followed by 4% paraformalydehyde. Brains were extracted and later cut on a microtome at 40 um per section. Every other section was kept to determine cannulae placement. Both cannulae tracks and thionin deposits helped to identify the exact location of the injection site in the OFC. Sections were later Nissl stained.

Behavioral Measurements and Statistical Analyses

Percent of time spent in the food cup was analyzed during both cue and non-cue periods, pre-CS periods (30 s before cue onset) before and after reversal learning. Data is shown as percent of time spent in the food cup during the 30 s CS cue period. These data, measured by photobeam breaks, were collected from Graphic State software (Coulbourn Inc., PA) and analyzed in Matlab (Mathworks) and Statistica (Statsoft).

Results

Cannulae placement is illustrated in Figure 1A. Cannulae were located within OFC for all rats, and there were no obvious differences in placement between groups. Figure 1B shows a photomicrograph of a guide cannulae track in one of our experimental rats. No additional damage was found outside cannula track damage in our experimental group.

Figure 1.

A. Location of cannulae in OFC. B. Photomicrograph of one experimental rat that received infusions of baclofen/muscimol. Note the damage made by the guide cannula ends above OFC; there was little or no apparent permanent damage from the repeated infusions.

Average responding during reminder training is shown in Figures 2A (to be saline) and 2B (to be musc/bac). No infusions were given during this time. Rats in both groups responded significantly more for the rewarded A1 cue than for the non-rewarded A2 cue, and there were no differences between groups. A three-way ANOVA (cue × treatment × day) revealed a main effect of cue (F (1,12)=21.429, p<0.001) but no main effects nor interactions with treatment (F’s< 1.663, p’s>0.2107). A comparison of responding to A1 and A2 during the three reminder days versus the last three days of initial conditioning showed a main effect of cue (F(1,12)=27.080, p<0.001), but no main effects nor any interactions with any other factors (F’s< 3.273, p’s>0.05).

Figure 2.

Effects of OFC inactivation on reversal of Pavlovian responding. A. Conditioned responding (as measured by percent behavior spent in the food cup only during the CS) during reminder conditioning days (as an average over three days) and reversal training. Rats only received infusions of saline in OFC during reversal sessions (five days). B. Conditioned responding during reminder conditioning days (as an average) and reversal training in rats that received infusions of muscimol and baclofen in OFC (only during the five reversal sessions). (*p<0.05)

Average responding during subsequent reversal sessions is shown in Figures 2A (saline) and 2B (muscimol/baclofen). Saline controls learned to inhibit responding to A1 while at the same time increasing responding for A2 over the five days of reversal learning. By contrast, OFC inactivated rats showed a normal decline in their conditioned responding to A1 but failed to increase their responding to A2 normally. This failure was particularly evident in the final several sessions. In accord with this interpretation, two-way ANOVA (cue × day) revealed a main effect of cue (F(1,7)=7.348, p<0.0302) and a cue × day interaction (F(4,28)=9.165, p<0.0001) in saline controls, and further planned comparisons on each day indicated that there were significant differences in responding for the two cues on each of the last three days of reversal training. A similar analysis of responding in OFC-inactivated rats revealed a main effect of cue (F(1,5)=9.334, p<0.028) but no cue × day interaction (F(4,20)=0.953, p<0.4545), and planned comparisons demonstrated no significant differences in responding for the two cues at any point during reversal learning. Additionally ANOVA (group × day) comparing responding for the rewarded A2 cue and the non-rewarded A1 cue in the two groups during reversal training demonstrated no main effect nor any interactions for cue A1 (F(4,48)<0.310, p>0.870) but a significant group × day interaction in responding for cue A2 (F(4,48)=3.921, p<0.008). Thus, OFC inactivation did not disrupt the rats’ ability to inhibit responding for the previously rewarded cue (A1) but did prevent them from learning to respond normally for the previously non-rewarded cue (A2).

Discussion

Here we tested the role of OFC in reversal learning using a Pavlovian discrimination task. This setting differs from the reversal tasks typically used to assess OFC function in that there was no contingency between the animal’s response and reward delivery. As expected, OFC inactivation impaired the ability of the rats to reverse conditioned responding. However, somewhat surprisingly, the deficit reflected an inability of these rats to develop normal conditioned responding for the previously unrewarded cue; inactivation of OFC had no impact on the ability of the rats to inhibit responding to the previously rewarded cue. This result is consistent with a recent report from our lab, in which OFC-lesioned rats exhibited normal extinction of responding to previously rewarded Pavlovian cues (Burke et al., 2008). Further it replicates findings from Roberts and colleagues in marmosets, showing that OFC damage also prevented normal changes in conditioned motor and autonomic responses for the previously rewarded cue (Reekie et al., 2008). In this study, marmosets were trained on a similar task with two cues: one paired with reward (CS+) and one paired with nothing (CS−). After initial training, half of the animals were lesioned, retrained to reach similar levels of responding and then reversed on cue-outcome contingencies. As in the current study, they found that lesioned animals had significant behavioral impairments in reversing their behavior, primarily due to an inability to learn to respond appropriately to the new CS+. The lesioned animals had no significant difficulty inhibiting their responding to the previously rewarded CS.

There are three aspects of these results that are worth comment. The first concerns the general role of the OFC in reversal learning. The prior report in marmosets and the results presented here show that OFC is critical to reversal of Pavlovian responding. As noted in the introduction, this is important because the discrimination tasks typically used to study OFC’s contribution to reversal learning include both Pavlovian and instrumental elements. That OFC is specifically necessary for Pavlovian reversals is consistent with the view that OFC is critical to reversal learning (and in a variety of other tasks) due to its role in signaling the Pavlovian associations between the cues and outcomes rather than due to any specific role in instrumental learning. OFC neurons encode associations between cues and outcomes in reversal settings (Thorpe et al., 1983; Rolls et al., 1996; Schoenbaum et al., 1999), and such encoding seems to be similar even without instrumental contingencies (Morrison and Salzman, 2006). Indeed it has recently been suggested that OFC plays no role in learning about actions and is involved exclusively in Pavlovian learning (Ostlund and Balleine, 2007a, b). Our data is consistent with this idea.

However, while OFC may not be critical for action-outcome learning, clearly signaling from OFC can influence the selection of actions. This is true in discrimination reversal, and we have also recently shown a critical role for the OFC in instrumental learning for conditioned reinforcement (Burke et al., 2008). Additionally neural activity appears to signal the value of different actions, both in rat OFC (Feierstein et al., 2006; Roesch et al., 2006) and also in ventromedial OFC in humans (Valentin et al., 2007). While these correlates may reflect the unique sensory aspects of the different responses, it is also possible that they reflect a complex role for OFC in influencing action selection, which is not revealed by lesion studies.

The second aspect of these results worth commenting on is their significance for the notion that OFC plays a general role in response inhibition. While is it true that damage to OFC often affects behavior in situations that require animals to inhibit a response, these data, as well as evidence of normal inhibition of prepotent responding in the reversed-reward-contingency task (Chudasama et al., 2007) and even in go, no-go tasks (Schoenbaum et al., 2002b), indicate that “response inhibition” is unlikely to be a root function of the OFC. Instead it may be a symptom of an underlying failure in signaling of outcome expectancies (Schoenbaum and Roesch, 2005). This failure is clearly evident in reinforcer devaluation settings, in which there is no requirement for learning (Gallagher et al., 1999; Izquierdo et al., 2004). We have argued that this same function - signaling of expected outcomes by OFC - may also facilitate learning in the face of unexpected outcomes via a contribution to the generation of prediction errors. Consistent with this idea, OFC damage is associated with miscoding of old associations in basolateral amygdala during reversal learning (Saddoris et al., 2005; Stalnaker et al., 2007).

Of course OFC is a large area, encompassing both lateral and medial regions and extending back into caudal agranular insular regions in some reports; our conclusions may not be relevant to orbital areas outside our target region. In the current study, we targeted a region of lateral OFC in rats that has reciprocal connections with subregions of amygdala, ventral striatum, and mediodorsal thalamus that parallel the connectivity of areas 11, 12, and 13 in primates (Schoenbaum et al., 2002a). This region, located in the dorsal bank of the rhinal sulcus, would include the lateral and ventral orbital areas and agranular insular cortex back to the genu of the corpus callosum. We specifically avoided posterior agranular insular regions, presumed to be primary gustatory regions (Saper, 1982; Kosar et al., 1986; Krushel and Van Der Kooy, 1988). Additionally we did not target medial orbital areas, and it is unlikely that our infusions reached these regions. It is possible that these medial regions mediate inhibitory functions reported in some studies of OFC function that utilize very large or more medial lesions.

However the area targeted in the current report does mediate reversal learning; indeed inactivation of the lateral part of OFC targeted here, using the same agents, dose and cannulae location caused reversal deficits in an olfactory discrimination task identical to those caused by neurotoxic lesions (Takahashi et al., 2009). Furthermore lesions centered on this same area (and not extending into medial regions) also cause devaluation deficits in rats (Pickens et al., 2003; Pickens et al., 2005). Thus, at least in rats, these functions appear to be mediated by this lateral area (but see also Kazama and Bachevalier, 2009).

Finally the third aspect and perhaps most interesting aspect of these results that deserves comment is the contrast between the negative effect of OFC inactivation on the inhibition of responding to a previously rewarded cue, reported here, and the positive effect of OFC inactivation on inhibition of responding to a previously compounded cue in a Pavlovian over-expectation task, reported previously (Takahashi et al., 2009). Indeed, as noted in the methods section, the rats used here were a subset of those animals. The striking contrast between the ability of these rats to inhibit responding in extinction here but not as a result of over-expectation may point to a specific role for OFC when it is necessary to integrate elemental Pavlovian associations in order to generate accurate predictions about expected outcomes. OFC is critical in the final stage of performance in a number of tasks, such as devaluation and discounting tasks and outcome-specific transfer (Mobini et al., 2002; Pickens et al., 2003; Izquierdo et al., 2004; Winstanley et al., 2004; Pickens et al., 2005; Rudebeck et al., 2006; Ostlund and Balleine, 2007a). Each of these settings requires the subject to generate predictions about outcomes and to do so by bringing together or integrating disparate pieces of associative information. Summation – which is required for normal learning during over-expectation – is just a special case of such integration. OFC inactivation prevents learning from over-expectation and, critically, also disrupts the increased responding observed in controls to the compound cue (Takahashi et al., 2009). By contrast, extinction doesn’t necessarily require summation, though it can be an incidental feature of some procedures (eg if the context has also been disproportionately rewarded or the animal has a prior training history of some sort). This could explain why OFC is sometimes necessary for extinction learning (Butter, 1969; Izquierdo and Murray, 2005).

Of course if OFC is only critical when normal learning (or behavior) requires summation, then it is perhaps surprising that the acquisition of a new response was impaired by OFC inactivation in the current experiment, since this would seem to only require simple elemental expectancies. While this seems at odds with the proposal that OFC is important for integrating information about expected outcomes, it is worth considering that reversal learning is quite complex, thus new learning in this setting may not be as straightforward as it appears. Processes involving contextual conditioning, occasion setting, temporal features of the environment, and even uncertainty may all play a larger role in reversal learning than they do in initial conditioning. Indeed, initial appetitive Pavlovian conditioning in naïve rats is not normally affected by OFC lesions (Gallagher et al., 1999; Burke et al., 2008; Takahashi et al., 2009). We would speculate some feature or requirement of the task or training history of the rats used here may be allowing summation to facilitate new learning after reversal but not extinction. Further work using behavioral procedures designed to specifically to manipulate the need for summation is necessary to directly test this hypothesis.