The role of the orbitofrontal cortex in the pursuit of happiness and more specific rewards

Abstract

Cues that reliably predict rewards trigger the thoughts and emotions normally evoked by those rewards. Humans and other animals will work, often quite hard, for these cues. This is termed conditioned reinforcement. The ability to use conditioned reinforcers to guide our behaviour is normally beneficial; however, it can go awry. For example, corporate icons, such as McDonald’s Golden Arches, influence consumer behaviour in powerful and sometimes surprising ways¹, and drug-associated cues trigger relapse to drug seeking in addicts and animals exposed to addictive drugs, even after abstinence or extinction^2,3. Yet, despite their prevalence, it is not known how conditioned reinforcers control human or other animal behaviour. One possibility is that they act through the use of the specific rewards they predict; alternatively, they could control behaviour directly by activating emotions that are independent of any specific reward. In other words, the Golden Arches may drive business because they evoke thoughts of hamburgers and fries, or instead, may be effective because they also evoke feelings of hunger or happiness. Moreover, different brain circuits could support conditioned reinforcement mediated by thoughts of specific outcomes versus more general affective information. Here we have attempted to address these questions in rats. Rats were trained to learn that different cues predicted different rewards using specialized conditioning procedures that controlled whether the cues evoked thoughts of specific outcomes or general affective representations common to different outcomes. Subsequently, these rats were given the opportunity to press levers to obtain short and otherwise unrewarded presentations of these cues. We found that rats were willing to work for cues that evoked either outcome-specific or general affective representations. Furthermore the orbitofrontal cortex, a prefrontal region important for adaptive decision-making⁴, was critical for the former but not for the latter form of conditioned reinforcement.

Rats (n = 13) received neurotoxic lesions of the orbitofrontal cortex (OFC; see Supplementary Information for lesion drawings); controls (n = 19) received either sham surgeries with saline infusions into the OFC or no surgery. Sham and non-surgical controls showed no differences in subsequent measures of conditioning, thus they were combined into a single group. After surgery, all rats were trained in a specialized pavlovian conditioning task, termed transreinforcer blocking^5,6. Blocking refers to the observation that a cue that predicts reward will prevent the formation of associations between that reward and any other cues that are present (see Supplementary Discussion for details on alternative interpretations of blocking). Thus, if a rat is trained that the illumination of a light predicts food and is later presented with that same light and a tone followed by the same food, the rat will not learn to associate the tone with any of the information evoked by the presentation of that food (for example, its sensory qualities and general affective properties). The light prevents, or blocks, rats from forming associations between the tone and the food.

The form of transreinforcer blocking that we have used varies this procedure by using two different but equally preferred outcomes. The light is initially presented alone followed by one outcome. Subsequently, the light is presented in compound with the tone, followed by the second outcome. Because both outcomes are equally preferred, they trigger comparable emotional responses⁵. These general affective properties are already predicted by the light, thus the tone is blocked from forming associations with them. However, the light does not predict information that is specific to the second outcome, such as its particular reinforcing sensory properties. Thus, the tone is able to form associations with these outcome-specific properties. As a result, the tone evokes thoughts of a specific outcome without triggering general affective representations. Evidence to support this comes from studies showing that cues trained this way facilitate responses leading to the same outcome, thought to be mediated through the activation of a neural representation of that outcome and its specific features. These cues, however, fail to facilitate responses that lead to different but similarly valenced outcomes, thought to be mediated by neural representations of their shared affective properties⁵ (see Supplementary Discussion for additional details). Therefore, transreinforcer blocking creates a cue that can be used to test whether conditioned reinforcement is mediated by outcome-specific information.

Our implementation of transreinforcer blocking is illustrated in Fig. 1 (see Methods for further details). Rats received 12 days of conditioning in which two unique visual cues, A and B (cue light or house light, counterbalanced), were paired with two unique outcomes, O1 and O2 (grape- or banana-flavoured sucrose pellets, counterbalanced), respectively. These pellets were equally preferred but could be discriminated and their value independently manipulated; there was no effect of lesion on these preferences (see Supplementary Results). Acquisition of pavlovian conditioned responding is shown in Fig. 1a. Both control and lesioned rats learned to respond to both cues, and there were no differences in their rates of responding. A three-way analysis of variance (ANOVA) test (lesion ×cue ×day) showed a significant main effect of day (F_11,330 = 9.54, P < 0.001) but no other main effects nor any interactions with either cue or lesion (F < 1.41, P > 0.16 in all cases).

Figure 1. Effect of orbitofrontal lesions on pavlovian conditioning and reinforcer devaluation.

Shown is the experimental timeline, with boxes linking conditioning, compound conditioning, and reinforcer devaluation phases to data from each phase. In the timeline and figures, A, B, X and Y are training cues; R1 and R2 are instrumental responses; and O1 and O2 are different flavoured sucrose pellet reinforcers. a, Responses per minute in the food cup during the first period of the conditioned stimulus (CS) (first 8 s) for A and B during pavlovian conditioning sessions. There were no effects of lesions. b, Responses per minute in the food cup during the first cue period, for AX and BY during compound pavlovian conditioning sessions. The timing of the two probe tests and the pre-devaluation conditioned reinforcement (CRf) testing are indicated by arrows. There were no effects of lesions. c, Food pellets consumed during the final two days of devaluation (days 5 and 6) for control (open bar) and lesion (filled bar) groups. There were no effects of lesions. D, devalued pellet; ND, non-devalued pellet. Error bars denote s.d.

After the initial conditioning, the rats underwent 9 days of compound conditioning in which they received presentations of the two ‘fully conditioned’ cues, A and B, in compound with two new cues, termed X and Y (tone and white noise, counterbalanced). AX led to delivery of O1, which was the same outcome predicted by A alone. Because A already predicted this outcome, X was ‘blocked’ from becoming associated with any of the information evoked by O1. BY also led to presentation of O1; however, O1 was a different outcome than that predicted by B alone. As a result, Y was ‘partially conditioned’, becoming associated with properties unique to O1, including its specific sensory properties, while remaining blocked from becoming associated with the general affective properties shared by O1 and O2. Conditioned responding during compound training is shown in Fig. 1b. Both control and lesioned rats continued to respond to the cues in this phase, and there were no differences in their rates of responding. A three-way ANOVA (lesion ×cue ×day) showed neither a main effect of lesion nor any interactions with lesion (F < 1.32, P > 0.23 in all cases).

As part of this compound conditioning, we tested the ability of the pavlovian cues to serve as conditioned reinforcers. For this, control and lesioned rats were each divided into two groups such that there were no differences in conditioned responding between them. Rats in each group received two 30-min training sessions in which two levers were made available in the training chambers. For group A, pressing one lever resulted in a 1-s presentation of A, and pressing the other resulted in a 1-s presentation of X. For group Y, pressing one lever resulted in a 1-s presentation of Y, and the other resulted in a 1-s presentation of X.

The average number of responses (presses) on each lever for rats in groups A and Y is shown in Fig. 2a and b, respectively. Control rats responded similarly for A and Y; lesioned rats responded for A, but not for Y (see Supplementary Results for additional responding for B in sham and OFC-lesioned rats). A three-way ANOVA (lesion × A/Y group × lever) showed main effects of lever (F_1,28 = 47.3, P < 0.0001), group (F_1,28 = 20.1, P < 0.001) and lesion (F_1,28 = 4.437, P < 0.044), and significant interactions between lesion and group (F_1,28 = 16.427, P < 0.001) and between lesion, group and lever (F_1,28 = 8.582, P < 0.007). Planned post-hoc tests showed significant differences in responding for A versus X in control and lesioned rats and for Y versus X in controls (P < 0.05), but there was no significant difference in responding for Y versus X in lesioned rats (P = 0.7).

Figure 2. Effect of orbitofrontal lesions on conditioned reinforcement for a fully conditioned A cue, for a blocked X cue, and for the partially blocked Y cue before and after reinforcer devaluation.

Shown is the experimental timeline linked to data from each conditioned reinforcement (CRf) test. In the timeline and figures, A, B, X and Y are training cues; R1 and R2 are instrumental responses; and O1 and O2 are different flavoured sucrose pellet reinforcers. a–d, Lever pressing for A versus X, or Y versus X, in control (open bars) and lesioned (filled bars) rats before (a, b) and after (c, d) devaluation. Lesions diminished responding for Y before devaluation (a, b), controls diminished lever pressing for Y after devaluation (c, d). Lever pressing is averaged across two 30-min sessions in each figure. Asterisks indicate significance at P < 0.05 on post-hoc contrast testing; the grey numbers indicate the ratio of responding on the two levers for each significant comparison. NS, not significant. Error bars denote s.d.

These results demonstrate that conditioned reinforcement in rats can be mediated by outcome-specific representations. Indeed, a comparison of response ratios (A/X versus Y/X in controls, given in Fig. 2) demonstrated no significant difference in the strength of the conditioned responding between A and Y in controls (P = 0.18). Thus, outcome-specific information was sufficient for normal responding in our task. Furthermore, because OFC-lesioned rats were impaired only when responding for Y (compare the lesion groups in Fig. 2a and b), these results show that the OFC is critical for conditioned reinforcement only when responding is mediated by outcome-specific information.

Although lesioned rats failed to respond to Y, when tested for A they responded more than controls. This increased responding was not specific to A. Unplanned post-hoc tests showed significant differences in responding in lesioned versus control rats for A (P = 0.0002) and for X (P = 0.034). Moreover, the increased responding for each was proportional, thus a comparison of response ratios (A/X in controls and lesioned rats, given in Fig. 2) resulted in no significant difference between controls and lesioned rats (P = 0.40). Although a decline in responding to A in lesioned rats owing to the loss of a contribution from outcome-specific information might have been expected, general increases have been seen in other studies on the effects of prefrontal, including orbitofrontal, damage on conditioned reinforcement^7,8. It is possible that affective information, which is still mobilized by A in lesioned rats, is able to compensate for or take over from the loss of the outcome information and even generalizes to activate behaviour on the other lever. This idea is supported by the fact that reinforcer devaluation, which should also preferentially affect behaviour mediated by outcome representations, fails to affect conditioned reinforcement⁹. Indeed an inability to signal the expected outcome may actually contribute to this over-responding, because it might impair extinction of the conditioned reinforcer’s affective value when primary reward is not presented after the cue.

To confirm that responding for Y (but not A) was dependent on the value of the specific outcome predicted by these cues, we reassessed the ability of the cues to support conditioned reinforcement after devaluation of O1. Devaluation was conducted over 6 days. On days 1, 3 and 5, the rats received 100 pellets of the O1 outcome followed by illness, induced by lithium chloride injection. On days 2, 4 and 6, rats received the same amount of O2 without illness. As illustrated in Fig. 1c, devaluation caused a significant and selective reduction in consumption of O1 in both groups. ANOVA (lesion × outcome) showed that both control and lesioned rats consumed significantly less of O1 than O2 on the final day of testing (F_1,30 = 311.07, P < 0.0001), and there were no effects of lesion on this devaluation effect (F < 3.59, P > 0.068, in all cases). There were also no effects of flavour (see Supplementary Results).

After devaluation, rats were returned to the training chambers with the levers available. Locations and programmed consequences remained unchanged. Average lever pressing over two 30-min sessions is shown in Fig. 2c, d. Devaluation diminished conditioned reinforcement for Y in controls but had little apparent effect on conditioned reinforcement for A in either controls or lesioned rats. A three-way ANOVA (lesion × A/Y group × lever) resulted in main effects of lever (F_1,28 = 10.02, P < 0.004) and group (F_1,28 = 6.675, P < 0.015), as well as interaction between group and lever (F_1,28 = 14.237, P < 0.001). However, there were no longer any main effects or any interactions with lesion (F < 1.73, P > 0.19 in all cases). Planned post-hoc tests showed significant differences in responding in both control and lesioned rats for A versus X (P < 0.05 in both cases) but not for Y versus X (P > 0.50 in both cases). Comparison of response ratios across devaluation demonstrated no significant change in the strength of conditioned responding for A in either controls (P = 0.37) or lesioned rats (P = 0.7).

These results demonstrate that conditioned responding in controls for Y is mediated by information about the value of the specific outcome that Y predicts, because control rats stopped responding for Y when that outcome was devalued (Fig. 2d, control group). The fact that the OFC would be critical for responding for Y brings the role of the OFC in conditioned reinforcement into accord with a growing consensus that the OFC is critical for cue-evoked behaviours when those behaviours depend critically on the value of the specific outcome that the cue predicts⁴. Second, these results confirm previous reports that conditioned reinforcement can be mediated by the ability of cues to trigger devaluation-insensitive general affective representations, because rats continued to respond for A even after devaluation (control group in Fig. 2c)⁹. The OFC does not seem to be necessary for this form of conditioned reinforcement (lesion group in Fig. 2c).

Notably, the effects of OFC lesions on conditioned reinforcement parallelled the effects of these lesions on pavlovian responding to these cues, assessed by presenting the rats with each cue (A, B, X and Y) individually in two extinction tests. These probe tests were conducted before and after the first conditioned reinforcement test. Each cue was presented six times without reinforcement. Average conditioned responding across trials in these extinction sessions is shown in Fig. 3. Both control and lesioned rats showed conditioned responding to cues A and B, whereas control but not lesioned rats acquired conditioned responding to Y. Neither group acquired a conditioned response to X. Consistent with this description, a three-way ANOVA (lesion × cue × trial) showed main effects of cue, trial, and cue × trial interactions (F_3,90 = 19.2, P < 0.0001; F_5,150 = 28.8, P < 0.0001; F_15,450 = 3.49, P = 0.01, respectively). These effects reflected greater responding for A and B in responding across trials in the extinction probe tests, which was observed in both controls and lesioned rats. ANOVA examining the effect of lesions on responding to the fully conditioned cues (A and B) yielded no significant main effects or any interactions (F < 1.2, P > 0.29 in all cases). In contrast, ANOVA examining the effect of lesions on responding to the blocked cues, X and Y, yielded a significant interaction between cue and lesion (F_1,30 = 10.156, P < 0.003). Post-hoc tests showed that control rats responded significantly more for Y than for X (P < 0.001) whereas lesioned rats did not (P > 0.34). Thus, the OFC seems to be critical for both pavlovian and instrumental responses guided by information about the value of specific outcomes.

Figure 3. Effect of orbitofrontal lesions on pavlovian conditioned responding after transreinforcer blocking in extinction probe tests.

Shown is the experimental timeline linked to data from the blocking probe tests. In the timeline and figures, A, B, X and Y are training cues; R1 and R2 are instrumental responses; and O1 and O2 are different flavoured sucrose pellet reinforcers. Responses per minute at the food cup are shown individually for each of six unrewarded presentations of A, B, X and Y (small black (lesioned) and white (control) boxes) and also averaged over all six presentations (large grey (lesioned) and white (control) bars). Lesions had no effect on responding for A, B or X, but significantly diminished responding for Y. Error bars represent s.d.

Here we have shown that conditioned reinforcers can drive behaviour either by activating representations of specific outcomes or by activating more general emotional or affective representations. These two types of information differ in critical ways: principally, the former reflects the animal’s current desire for the outcome that the cue predicts, whereas the latter operates independently of that information. In normal settings, both affect and outcome information probably influence responding in concert; however, in theory, their influence on behaviour can be divergent. This was true in the current study for rats responding for the fully conditioned cue after devaluation of the outcome that this cue predicted. Notably, despite devaluation of the outcome, these rats continued to work for this cue. This imbalance is similar to the effects of conditioned reinforcers in humans, which are often independent of our desire for the outcomes they predict, as for example when one is lured into McDonald’s despite a desire to avoid unhealthy fare. A similar but more severe imbalance is also prominent in neuropsychiatric disorders, including drug addiction and eating and anxiety disorders, in which cues come to control behaviour out of all proportion to the desirability of the predicted outcomes^2,3.

The balance in how different types of associative information are mobilized by conditioned reinforcers may depend on the relative efficiency of processing in different neural circuits specialized for handling different types of associative information. Evidence presented here indicates that outcome-specific conditioned reinforcement is mediated by the OFC, whereas conditioned reinforcement mediated by general affective information is not. Although at least one previous report has implicated the OFC in conditioned reinforcement⁸, our results indicate that its role may be entirely subordinate to this region’s well-documented involvement in signalling expected outcomes (see Supplementary Discussion for consideration of alternative interpretation involving attention). The OFC is activated by cues and during responding in a way that seems to reflect features of the specific outcomes that are to follow—particularly their value^10–16—and damage to the OFC has been shown to impair changes in cue-evoked responding after devaluation^17–19. Here we show that the OFC is also critical for instrumental responding when that responding is driven by cue-evoked representations of expected outcomes.

Of course the OFC is ultimately only one part of the brain circuit used by conditioned reinforcers to mobilize information. Areas such as the amygdala and the ventral striatum have also been implicated in responding for conditioned reinforcers^20–27. In the same way that the OFC supports outcome-specific conditioned reinforcement, it seems likely that a subset of these other brain areas support conditioned reinforcement driven by cue-evoked representations of the general affect or emotion evoked by the cue^28–30. An anatomical segregation of the brain circuits supporting conditioned reinforcement mediated by thoughts of specific outcomes versus more general affective information could help explain why disease states so often cause an imbalance in how we respond to conditioned reinforcers.

METHODS SUMMARY

Thirty-two food-deprived male Long-Evans rats (Charles River Laboratories) served as subjects. Testing began after lesion/sham surgeries and was conducted using equipment from Coulbourn Instruments. Transreinforcer blocking consisted of pavlovian conditioning, compound conditioning, and probe tests. Rats received 12 days of conditioning in which two visual cues (a house light and a cue light, designated A and B, counterbalanced) were paired 16 times with sucrose pellets (45 mg grape- and banana-flavoured sucrose pellets, designated O1 and O2, respectively, counterbalanced) (Research Diets).

After conditioning, all rats received 1 day of pre-exposure to two auditory cues (white noise and tone, designated X and Y, counterbalanced). The next day, the rats began 9 days of compound conditioning. Compound cue AX was paired with O1, which was the same flavour sucrose pellet associated with A. Compound BY was also paired with O1, which was a different flavour sucrose pellet than that associated with B. Each session consisted of eight presentations of each compound. The rats also received eight presentations of A–O1 and B–O2 reminder training. During compound conditioning, all rats received two probe test days consisting of six unrewarded presentations of A, B, X and Y.

Conditioned reinforcement was conducted during compound conditioning in the original training chambers with response levers on the right and left walls. One lever led to a 1-s presentation of a conditioned stimulus (either A or Y depending on group) on a fixed ratio two (FR2) schedule, and the second lever led to a 1-s presentation of X on an FR2 schedule. The lever–cue associations were counterbalanced for side in each group. Conditioned reinforcement testing was conducted once before, and again after, devaluation of O1. O1 was devalued by pairing it with illness induced by 0.3 M lithium chloride (5 mg kg⁻¹, intraperitoneal injection).