Interaction Between Orbital Prefrontal and Rhinal Cortex Is Required for Normal Estimates of Expected Value

Abstract

Predicting and valuing potential rewards requires integrating sensory, associative, and contextual information with subjective reward preferences. Previous work has identified regions in the prefrontal cortex and medial temporal lobe believed to be important for each of these functions. For example, activity in the orbital prefrontal cortex (PFo) encodes the specific sensory properties of and preferences for rewards, while activity in the rhinal cortex (Rh) encodes stimulus-stimulus and stimulus–reward associations. Lesions of either structure impair the ability to use visual cues or the history of previous reinforcement to value expected rewards. These areas are linked via reciprocal connections, suggesting it might be their interaction that is critical for estimating expected value. To test this hypothesis, we interrupted direct, intra-hemispheric PFo-Rh interaction in monkeys by performing crossed unilateral ablations of these regions (functional disconnection). We asked whether this circuit is crucial primarily for cue–reward association or for estimating expected value per se, by testing these monkeys, as well as intact controls, on tasks in which expected value was either visually cued or had to be inferred from block-wise changes in reward size in uncued trials. Functional disconnection significantly affected performance in both tasks. Specifically, monkeys with functional disconnection showed less of a difference in error rates and reaction times across reward sizes, in some cases behaving as if they expected rewards to be of equal magnitude. These results support a model whereby information about rewards signaled in PFo is combined with associative and contextual information signaled within Rh to estimate expected value.

Introduction

The subjective desirability of an expected reward, i.e., its value, is a key factor in determining the latency, accuracy, and vigor of goal-directed behavior (Dickinson and Balleine, 1994). Behavioral studies have identified several factors—such as reward type, size, or cost—that typically determine expected value (Toates, 1986) Physiological studies suggest that these factors are encoded across a broad network including: neuromodulatory systems, limbic structures, and association areas in the medial temporal lobe and prefrontal cortex (Haber and Knutson, 2010). Given this widespread neural representation, determining to what extent, and under what conditions, direct interaction between circuit elements is required is a crucial step both in unraveling information flow within this network and in revealing the neural basis of reward-guided behavior.

There are several reasons to suspect that direct interaction between two key areas in the aforementioned reward circuitry, the orbital prefrontal (PFo) and rhinal (Rh) cortex, might be critical for estimating the value of an expected reward. First, bilateral lesions of either PFo or Rh disrupt the ability to use visual-reward cues (Liu et al., 2000; Simmons et al., 2010) or the history of previous reinforcement (Walton et al., 2010; Clark et al., 2012) to estimate expected value. Second, these structures seem to play complementary roles in behavior. Activity in Rh is significantly modulated by multiple mnemonic factors (Brown et al., 1987; Sakai and Miyashita, 1991; Liu and Richmond, 2000), and lesions of Rh impair many types of learning and memory (Murray et al., 1993, 1998; Mumby and Pinel, 1994; Yonelinas et al., 1998; Sauvage et al., 2010). Conversely, activity in PFo is significantly modulated by the sensory and hedonic properties of rewards (Rolls, 1989; de Araujo et al., 2003; De Araujo and Rolls, 2004) as well as subjective reward preferences (Padoa-Schioppa and Assad, 2006; Chaudhry et al., 2009), and damage to PFo alters the influence of previous outcomes on current decisions (Rudebeck and Murray, 2008; Camille et al., 2011) and impairs learning driven by changes in the identity of an expected reward (Burke et al., 2008; McDannald et al., 2011). Finally, anatomical studies have reported direct ipsilateral connections reciprocally linking PFo and Rh (Kondo et al., 2005; Saleem et al., 2008). Given the above evidence, we hypothesized that information about rewards signaled in PFo is combined with associative and contextual information signaled in Rh to estimate expected value.

To determine whether PFo-Rh interaction is critical for the normal assessment of expected value we interrupted their direct communication using a crossed disconnection design. Following disconnection, monkeys were still able to learn cue–reward associations, but their assessments of expected reward value were significantly altered. To test whether this deficit reflected a weakening of associative relationships or a more general impairment in estimating expected value, we tested these monkeys, as well as a group of unoperated controls, on tasks in which reward value was predictable, but was not signaled by visual cues. These manipulations also resulted in significant impairments, suggesting this circuit plays a widespread role in evaluating expected reward.

Materials and Methods

Subjects.

We tested nine rhesus monkeys (Macaca mulatta)—eight males and one female—weighing between 5 and 14 kg as subjects in this study. Two monkeys served as unoperated controls, four monkeys were given crossed PFo and Rh lesions (PFo X Rh), and three monkeys were given bilateral PFo lesions. The four monkeys that served as the PFo X Rh group were experimentally naive. The bilateral PFo group was naive to our tasks before testing; however, they had previously been trained and tested in several standard neuropsychological assessments of learning and memory. The two monkeys that served as unoperated controls had extensive experience with each of our behavioral paradigms. All experimental procedures conformed to the National Institute of Health Guide for the Care and Use of Laboratory Animals and were approved by the National Institute of Mental Health Animal Care and Use Committee.

Surgery.

Aspiration lesions of Rh and PFo were carried out as described previously (Meunier et al., 1993; Izquierdo et al., 2004). For the PFo X Rh group, lesions were made in two stages; a unilateral ablation of PFo or Rh was made following the completion of preoperative training (two monkeys were given Rh lesions and two monkeys were given PFo lesions at this stage), complementary (i.e., PFo for animals that first received an Rh lesion) lesions were then performed in the contralateral hemisphere after an intervening recovery and testing period lasting 5–6 weeks. All bilateral PFo lesions were made in a single surgery.

Surgeries were carried out under aseptic conditions in a fully equipped operating suite with veterinary supervision. Before surgery, animals were sedated with ketamine hydrochloride (10 mg/kg, i.m.); a surgical level of anesthesia was then induced and maintained with isoflurane gas (2–4% to effect). Body temperature, heart rate, blood pressure, and expired CO₂ were monitored throughout all surgical procedures. After removal of a bone flap overlying the region of interest and reflection of the dura mater, intended lesion boundaries were marked via electrocautery and tissue was then removed through a combination of suction and electrocautery using a fine gauge metal pipette under the guidance of an operating microscope (Zeiss).

Tasks and training.

During all testing sessions, monkeys sat in a primate chair inside a darkened, sound-attenuated testing chamber. They were positioned 57 cm from a computer monitor subtending 40 × 30 degrees of visual angle. Task timing and visual stimulus presentation were under the control of networked computers running, respectively, custom written (Real-time Experimentation and Control—REX) and commercially available software (Presentation, Neurobehavioral Systems) for the design and control of behavioral experiments.

Red–green color discrimination.

Monkeys were initially trained to grasp and release a touch sensitive bar to earn fluid rewards. After this initial shaping, we introduced a red–green color discrimination task (Bowman et al., 1996). Red–green trials began with a bar press, 100 ms later a small red target square (0.5°) was presented at the center of the display (overlaying a white noise background). Animals were required to continue grasping the touch bar until the color of the target square changed from red to green. Color changes occurred randomly between 500 and 1500 ms after bar touch. Rewards were delivered if the bar was released between 200 and 1000 ms after the color change, releases occurring either before or after this epoch were counted as errors. All correct responses were followed by visual feedback (target square color changed to blue) after bar release and reward delivery 200–400 ms after visual feedback.

Visually cued task.

After an animal reached criterion in the red–green task (3 consecutive sessions with >85% correct performance) we introduced a visually cued reward size task (Fig. 1, visually cued; Fig. 2). Each trial began when animals grasped the touch bar, bar press was now initially followed (by 100 ms) by the presentation of a cue image (grayscale natural images, 10° × 10°, superimposed on a white noise background). Each cue signaled which of four different reward sizes (1, 2, 4, or 8 drops, random draw) the animal would earn upon successful completion of the trial. Four-hundred milliseconds after cue presentation a red target square appeared, now centered on the cue. Animals were once more required to hold the touch bar until the target square color changed from red to green (500–1500 ms). Releases that occurred outside of the 200–1000 ms interval following the color change were counted as errors; error trials were repeated until completed correctly. Successful bar releases were signaled via visual feedback (blue square). Reward delivery followed feedback by 200–400 ms and lasted for between 150 and 2500 ms (200 ms interdrop interval). Periodically, the reward system was calibrated to ensure an average drop size of 0.1 ml.

Figure 1.

Schematic illustration of the three instrumental tasks. In the visually cued task (top row) each reward size was paired with a unique cue image presented at the start of each trial. To earn the reward, monkeys had to release a lever a short time after the color of a target stimulus changed from red to green. In the visually instructed task (middle row), monkeys were required to react to the color change in the target stimulus but were not given a visual cue to reward size. In both the visually cued and visually instructed tasks, correct performance was followed by visual feedback and reward delivery, error trials were repeated until performed correctly. In the self-initiated task (bottom row), monkeys simply had to press and release a lever at their own pace to earn reward. They were given visual feedback after a rewarded bar release (bar releases occurring during the reward period were not reinforced). In both the visually instructed and self-initiated tasks, reward size varied across blocks of trials (25 trials per block).

Figure 2.

Lesion reconstructions. Intended PFo and Rh lesions are shown on a ventral view of the macaque brain as well as on coronal sections at the indicated levels. Estimates of the extent of aspiration lesions for two of the four monkeys in the PFo X Rh group are plotted on coronal sections at the indicated levels and reconstructed onto a ventral view of the macaque brain; reconstructions for each case are shown at the top of each column. Lesions were reconstructed using MR images; representative MR images for monkey P are shown next to the corresponding coronal section for both the unilateral PFo and Rh lesion. Yellow arrows mark the lesion boundaries. The two monkeys not shown received the opposite pattern of PFo and Rh lesions (PFo left hemisphere, Rh right hemisphere). Across the group, lesions largely covered the areas of interest, and damage to adjacent structures was minimal and distributed idiosyncratically across monkeys.

Monkeys were tested for 10–15 sessions using the same set of four cue images (cue set); we used a total of eight unique cue sets. Monkeys were tested in the visually cued task for two hours each session, 1 session per day, 5–6 d per week.

Self-initiated task.

Self-initiated trials are detailed schematically in Figure 1 (self-initiated). In contrast to the red–green color discrimination and visually cued tasks, the self-initiated task contained neither visual cues to reward size nor visual targets that the monkey was required to attend to, to successfully complete a trial. To earn a reward, monkeys simply had to touch and release a bar. All bar releases were followed by visual feedback and a reward (200–400 ms after feedback, 1, 2, 4, or 8 drops); to provide a basis for formulating an expected value, reward sizes were varied in blocks (random draw), with 25 responses per block. To distinguish the self-initiated from the visually cued task we displayed a different background image on the computer monitor (gray-scale fractal image). The duration of a self-initiated session was adjusted for each monkey to approximately equate the total volume of reward earned across the visually cued and self-initiated tasks. Monkeys were run in the self-initiated task for a total of 15 sessions, 1 session per day, 5–6 d per week.

In a control version of the self-initiated task, at the beginning of each block we presented a visual cue that signaled the reward size available in that block. Cues were grayscale natural images (10° × 10°) presented at the center of the computer monitor (presented over the same background image used in the standard self-initiated task). Cues were presented in the first trial of every block and remained visible for the duration of the block. This constant presentation ensured that the animals were not able to use the appearance/disappearance of the cue image as a trigger for their motor response.

Visually instructed task.

To determine the effect of providing monkeys a visual cue to response but not to reward size, we adapted the red–green color discrimination task (Fig. 1, visually instructed). Instead of a constant reward size, correct responses were now followed by 1, 2, 4, or 8 drops of reward; to provide a basis for formulating an expected value, reward sizes varied in blocks (random draw) with 25 correct trials per block. Monkeys were run on the visually instructed task for a total of 15 sessions, 1 session per day, 5–6 d per week.

Data analysis.

Standard modeling and statistical methods (see below) were applied to our data using software written in Matlab (MathWorks) and R (Team, 2004).

The visually cued and visually instructed tasks both contained a target signal (red–green color change) that monkeys had to respond to, to earn fluid reward. Thus, for these tasks, we were able to calculate both error rates and reaction times. Because error rates were more consistent across animals, and because error rate versus reward size data could be fit with a simple model for estimating learning rates, we focus here on error rates as a dependent measure of the monkeys' performance. However, results were largely qualitatively similar whether error rates or reaction times were used as a dependent measure. In the one instance for which results obtained using error rates or reaction times differed we present both measures. For each session, to assess the effect of both the reward size offered on a trial and the amount of reward consumed to that point in the session, performance was calculated: (1) separately according to reward size using all trials in a session, and (2) after binning trials according to reward size and normalized accumulated reward. To estimate changes in monkeys' reward expectations with continued exposure to a set of cue–reward pairings, for each session, we fit the error rates from each monkey to the hyperbolic model: E = 1/(aR), where (E) is the error rate, (a) is a scaling factor, and (R) the reward size. We plotted the goodness-of-fit (R²) of this model versus session number, smoothing curves with a three session moving average, to estimate learning rates.

We took the interval between the time monkeys touched and released the bar (release interval) as our dependent measure of performance in the self-initiated task. To facilitate comparisons between the visually cued and/or instructed and self-initiated tasks, we excluded release intervals that exceeded the maximum duration of a visually cued trial (taking the logarithm of the release interval or excluding release intervals that were greater than the standard deviation by a factor of three yielded similar results).

To examine the trial-by-trial dynamics of monkeys' self-initiated performance, we first normalized release intervals to the mean and SD in each session, we then averaged across within-block trial numbers within each session, across sessions for each monkey, and across monkeys for each experimental group.

We used generalized linear mixed models (GLMM) to evaluate whether the covariates, reward size and normalized accumulated reward, or the factors session number, surgical treatment, cue set, number of cues, or within-block trial number, had a significant effect on performance, quantified as error rates, reaction times, or release intervals. This approach assumes that variation in repeated measures data is due to both fixed (e.g., reward size) and random (e.g., monkey) effects, allows independent variables to be treated as continuous (e.g., reward size) or categorical (e.g., surgical treatment), and allows for non-normal dependent measures (i.e., error rates) (Longford, 1993). We compared random intercept models (which constrain the covariance between any pair of repeated measurements to be equal) with random slope and intercept models (which include the covariance between repeated measurements as an additional parameter) using a log likelihood ratio test (Zar, 2010). We selected the random slope and intercept model only if it provided a significantly better fit to the data. Depending on the distribution of the dependent variable, we used either a binomial (error rates) or Gaussian (reaction times and release intervals) link function.

We also used conventional ANOVA models to analyze these data, applying the variance stabilizing arcsine transformation (Zar, 2010) to all proportional data before hypothesis testing. Several diagnostic measures (Q-Q plots, data, residual and transformed distributions, standardized residual versus fitted values, etc.) were evaluated to ensure that in all cases the transformed data conformed to ANOVA assumptions. The variability accounted for by individual monkeys was included as an error term within each ANOVA model; mixed-model (“split-plot”) ANOVAs (Zar, 2010) were used for comparing experimental groups. Where appropriate, Tukey's HSD test was used for post hoc multiple comparisons. Results obtained using ANOVA were qualitatively similar to those obtained using GLMM, only the results of the GLMM analysis are reported here.

To test for group differences in the correlation in performance across two conditions we first transformed correlation coefficients using Fisher's r-to-z transformation; critical values were calculated as Z = (z₁ − z₂)/√(1/n₁−3) + (1 /n₂-3) where Z ∼ t_α(df) (Zar, 2010).

Lesion reconstruction.

Intended PFo X Rh lesions are shown in the left column of Figure 2. The intended borders of the PFo lesion were as follows: the medial–lateral extent ran from the fundus of the lateral orbital sulcus to the fundus of the rostral sulcus; rostral–caudally, the lesion ran from a line joining the rostral tips of the lateral and medial orbital sulci to ∼5 mm from the junction of the frontal and temporal lobes (in sum, Walker's areas 11, 13, 14, and a caudal portion of area 10). The intended Rh lesion subsumed both the entorhinal (Brodmann's areas 28) and perirhinal cortical fields (Brodmann's areas 35 and 36), including cortex on both sides of the rhinal sulcus and extending ∼2 mm lateral to the sulcus.

The full extent and location of all lesions was assessed using T1-weighted magnetic resonance image (MR) scans (1 mm slices, 0.4 mm in-plane resolution). To plot lesions, coronal MR images were first matched to coronal plates in a stereotaxic rhesus monkey brain atlas; following alignment, lesion boundaries were marked on each plate. After determining the boundaries of the lesion in each section, the full extent of the lesion was reconstructed onto a ventral view of the macaque brain. Representative reconstructions for two cases from the PFo X Rh group, along with MR images for one of the two cases, are shown in Figure 2, a representative reconstruction from the bilateral PFo group appears in the study by Simmons et al. (2010).

Results

Preoperative training

Monkeys (n = 4) were initially trained to perform a sequential red–green color discrimination task (Bowman et al., 1996) (see Materials and Methods, Tasks and training). When a monkey achieved a criterion of 85% correct responses in three consecutive sessions (mean = 13.5, SEM = 0.866), a visually cued reward size task (Fig. 1) was introduced.

On each trial of the visually cued task a visual cue signaled the amount of reward to be delivered following a correct red–green color discrimination (1, 2, 4, or 8 drops). Although information about reward size is not required for successful completion of the color discrimination trials, we found that error rates decreased with increasing reward size (Fig. 3A; GLMM; reward size: z = 2.8, p < 10⁻³). Additionally, error rates across all reward sizes increased as monkeys consumed more rewards (Fig. 3B; GLMM; accumulated reward: z = 11.2, p < 10⁻¹⁶). As previously reported (Minamimoto et al., 2009), there was a significant interaction between the effects of reward size and accumulated reward (see Fig. 3C; GLMM; reward size X accumulated reward: z = 9.2, p < 10⁻¹⁶). Because sensory and motor demands were constant across trials, these differences in performance across reward size and accumulated reward should reflect the monkeys' subjective valuation of the expected rewards.

Figure 3.

Preoperative performance in the visually cued task. A, Group average performance in the visually cued task is plotted as error rates (ordinate) versus reward size (abscissa), data are from the final 15 sessions of testing before the first unilateral lesion. There was a significant effect of reward size on performance, with error rates decreasing with reward size. B, Overall performance (collapsing across reward sizes) for the same 15 sessions shown in A is plotted as error rates (ordinate) versus normalized accumulated reward (abscissa, see Methods and Materials, Data analysis). There was a significant effect of accumulated reward on performance, with error rates increasing as monkeys became more sated. C, Performance for the same 15 sessions shown in A and B is plotted as error rates (ordinate) versus normalized accumulated reward (abscissa) separately for each reward size. There was a significant interaction between the effects of reward size and accumulated reward. D, Heat maps depicting group average error rates (color scale) as a function of reward size (ordinate) and session number (abscissa). Data in each panel are from a different cue set. There was a significant interaction between the effects of reward size, session number and cue set. On average, performance stabilized earlier in testing for later cue sets. E, Heat maps depicting group average error rates (color scale) as a function of accumulated reward (each session divided into sextiles, ordinate) and session number (abscissa). Data in each panel are from a different cue set (4 preoperative cue sets). The effect of accumulated reward was not significantly different across sessions or cue sets. Norm. Accum. Reward, Normalized accumulated reward.

Sensitivity to both expected reward size and accumulated reward developed rapidly. After no more than two sessions following their initial exposure to the task the performance of all four monkeys showed a significant effect of both expected reward size and accumulated reward (Fig. 3D,E, first panels; χ² test, all p < 0.01). Although they showed early sensitivity to both reward size and accumulated reward, performance continued to change until reaching a plateau later in testing (Fig. 3D,E first panels). It is possible that the number of sessions monkeys require to reach this asymptotic performance varies with their experience with different sets of cue–reward pairings (learning set). To control for the possibility of a learning set, each monkey was exposed to four cue sets (four cue–reward pairings per set, 15 sessions per cue set) during preoperative training.

Several aspects of performance continued to change with increasing experience. First, there was a significant negative correlation between overall error rate and session number for the first (r = −0.35, p < 10⁻⁶) and a trend toward significance for the second (r = −0.14, p = 0.07) cue set, but no such relationship for the third or fourth cue sets (both p > 0.3) (data not shown). This suggests that overall accuracy in the task continued to increase throughout testing on the first cue set, but had stabilized by the time of testing on the second-fourth cue sets. Second, this effect appeared to be largely driven by a sharp decrease in error rate for large but not small rewards. This conditional change in performance also stabilized more rapidly during testing on later cue sets (Fig. 3D; GLMM; reward size × session number × cue set: z = 4.3, p < 10⁻⁵). In contrast to these experience dependent changes in performance across reward sizes, the effect of accumulated reward remained fairly stable across cue sets (Fig. 3E; GLMM; accumulated reward × session × cue set: z = 1.3, p = 0.2).

A previous study reported that performance in this task is well described by a simple hyperbolic discounting model (Minamimoto et al., 2009). Accordingly, we used a hyperbolic decrease in error rates with increasing reward size as our standard of asymptotic performance (Fig. 4A). Learning curves were estimated by examining the evolution of the R² statistic obtained from fits of this model to data for each session (Fig. 4B). Consistent with the observations noted above, the learning curves in Figure 4B demonstrate the emergence of a learning set in this task. Post hoc multiple comparisons revealed that this effect was driven by a difference in performance across the first and all other cue sets (Tukey's HSD, p < 0.01).

Figure 4.

Estimating learning curves in the visually cued task. A, In each panel, group average error rates (ordinate) are plotted against reward size (abscissa), solid curves are the best fitting hyperbolic model. Within a row, data is from separate sessions with the same cue set, within a column, data is from the same session number but different cue set (cue set and session number are indicated in each panel). During training, the rate at which performance changed with additional testing on a given cue set varied across cue sets. B, Learning curves are plotted as the goodness-of-fit (R²) of the best-fit hyperbolic model (ordinate) versus session number separately for each cue set (smoothed with a 3 session moving average filter). Animals took significantly longer to achieve asymptotic performance on the first cue set. hyp, Hyperbolic.

Unilateral ablation of PFo or Rh

After the completion of preoperative testing we performed a unilateral aspiration lesion of either PFo (monkeys B and H) or Rh (monkeys P and S). After a two-week recovery period, monkeys were tested on the basic sequential color discrimination task until they achieved criterion performance (mean = 4, SEM = 0.7); we then resumed testing in the visually cued task. In the following, all analyses include data from all 15 sessions for each condition the monkeys were tested under.

To determine whether either unilateral lesion had a specific effect on retention or new learning, monkeys were tested for retention of the last preoperatively learned set of cues as well as for their ability to learn a new set of four cue–reward pairings (referred to as old/new respectively). We evaluated two measures of performance. First, we found no significant effect of unilateral ablations on performance, quantified as error rates, with either an old or new cue set (GLMM; all p > 0.12) (Fig. 5A). There was a trend for unilateral PFo but not Rh ablations to alter the effect of accumulated reward on performance for a new cue set [GLMM; accumulated reward X lesion (PFo): z = 1.8, p = 0.07]. Second, to test whether either or both types of unilateral ablation had an effect on the rate at which animals learn the cue–reward associations we repeated the model-fitting procedure used to estimate learning curves during preoperative training. We found no significant difference in the rate at which animals learned the cue–reward associations after unilateral PFo or Rh ablations (Fig. 5C, ANOVA; p > 0.5 for both old and new cue sets).

Figure 5.

PFo-Rh functional disconnection disrupts performance in the visually cued task. A, In each panel, group average error rates (ordinate) are plotted against normalized accumulated reward (abscissa), data are from the final preoperative and unilateral ablation conditions (middle panel, old cue set; right panel, new cue set). There was no significant effect of unilateral PFo or Rh ablations. B, Conventions as in A but for data collected following PFo-Rh functional disconnection. Unlike unilateral ablation of PFo or Rh, PFo-Rh functional disconnection had a significant effect on performance. C, Preoperative and postoperative data from testing with the old cue set are plotted as error rates (ordinate) versus normalized accumulated reward (abscissa) separately for each reward size. Following PFo-Rh functional disconnection, monkeys made fewer errors in 1 drop trial early in a session, and more errors in 2, 4, and 8 drop trials late in a session. D, Goodness-of-fit for the best fit hyperbolic model (ordinate) is plotted versus session number (abscissa) for data from the final preoperative training set and the unilateral (middle) and PFo X Rh (right) conditions. Norm. Accum. Reward, Normalized accumulated reward.

Functional disconnection of PFo and Rh: visually cued task

To interrupt direct interaction between PFo and Rh, in a second surgery, we removed the remaining structure (Rh in monkeys B and H, PFo in monkeys P and S) in the contralateral hemisphere. If contralateral projections between PFo and Rh are either weak or nonexistent, this treatment should selectively abolish direct PFo-Rh interaction.

After the second-stage surgery to complete the functional disconnection, monkeys were tested under the same schedule as after the first-stage surgery. We found that PFo-Rh functional disconnection significantly altered monkeys' performance of the visually cued task. Specifically, the PFo X Rh group exhibited a significant change in the distribution of errors across expected reward sizes. This was true for both old (Fig. 5A–C; GLMM, lesion × reward size × accumulated reward: z = 5.4, p < 10⁻⁸) and new cue sets (Fig. 5A,B; GLMM; lesion × reward size z = 4.7, p < 10⁻⁶), where old refers to the last cue set tested before PFo-Rh functional disconnection and new designates a novel set of cues. This effect can be seen separately for each reward size for the old cue set in Figure 5C. For both old and new cue sets, there was no significant difference in overall performance the number of trials completed (ANOVA; both p > 0.28) following functional disconnection. Furthermore, the lesions did not produce a reduction in general movement outside of the testing chamber, nor was the monkeys' consummatory behavior affected, as evidenced by the fact that all monkeys either maintained or gained weight over the duration of the experiments. This indicates that neither the monkeys' global motivational level nor ability to perform the operant task was affected by the treatment. Additionally, functional disconnection did not affect the rate at which monkeys learned the cue–reward associations, as assessed by sessions taken to attain asymptotic performance (Fig. 5D; ANOVA, lesion × session, p > 0.1 for both old and new cue sets).

Although functional disconnection significantly altered performance on the visually cued task, this effect was relatively mild and was even smaller during a second round of old–new testing (GLMM; testing order: z = 2.2, p = 0.02; testing order × accumulated reward: z = 3.7, p < 10⁻⁴; testing order × accumulated reward × reward size: z = 1.7, p = 0.09; data not shown). To determine whether the severity of the impairment varied with task difficulty, we increased the number of cue–reward pairings in one set from four to eight. Given that the same four reward sizes were used in this condition, this resulted in an eight cues onto four reward sizes mapping. In addition to the PFo X Rh group we tested a group of control animals (n = 2) on this and the standard visually cued task (one task per session, 15 sessions per task).

Doubling the cue–reward pairings in a set from four to eight dramatically affected PFo X Rh but not control group performance (Fig. 6A). This effect was consistent whether tested via a within-group comparison across difficulty conditions for the PFo X Rh (preop four cue vs postop eight cue: GLMM; reward size × accumulated reward × cue number z = 3.9, p = 10⁻⁵) and control groups (four cue vs eight cue conditions: GLMM; main effect of cue number and all interactions, p > 0.19), or via a between group comparison for the eight cue condition (GLMM; group × reward size × accumulated reward z = 5.5, p < 10⁻⁸). This effect was not driven by a deficit in learning the cue–reward mappings (Fig. 6B; ANOVA; group X session F_(13,39) = 0.76, p = 0.7). Nor could it be attributed to confusion over the fact that two unique cues now signaled each reward size (Fig. 6C; cue1–cue2 correlation—PFo X Rh: r = 0.47, p < 10⁻¹²; control: r = 0.4, p < 10⁻⁷; t₍₂₎ = 0.47, p = 0.65).

Figure 6.

Doubling the number of cue–reward associations results in a significant impairment in the visually cued task. A, Performance in the visually cued task is plotted as error rate (ordinate) against reward size (abscissa), data are from conditions with either 4 unique cues and 4 reward sizes (top) or 8 unique cues and 4 unique reward sizes (bottom). Data in the left (right) panel are from the control (PFo X Rh) group. The PFo X Rh group made significantly fewer errors—and exhibited less of a difference in performance across reward sizes—in the 8 cue condition. B, Goodness-of-fit for the best fit hyperbolic model (ordinate) is plotted versus session number (abscissa) for both the control (left) and PFo X Rh (right) groups. C, Error rates for the two cues that signaled a given reward size are plotted versus one another, separate panels are data from the control (left) and PFo X Rh (right) groups. Norm. Accum. Reward, Normalized accumulated reward.

In conclusion, the primary effect of the PFo-Rh functional disconnection on performance in the cued reward size task was a change in the slope of the function relating performance to accumulated reward, the direction of this effect varied across reward sizes, and the greatest differences in performance were observed at large reward sizes with larger cue sets, when monkeys with PFo-Rh functional disconnection performed nearly optimally. There was no increase in the number of sessions it took for the monkeys to reach asymptotic performance following PFo-Rh functional disconnection and PFo-Rh functional disconnection affected performance for both old and new cue sets.

Functional disconnection of PFo and Rh: self-initiated task

Our results thus far suggest that PFo-Rh functional disconnection disrupts stimulus–reward associations. However, it is possible that the effect of this treatment is an alteration in the assessment of expected reward value per se, as suggested by the effect of PFo-Rh functional disconnection on the interaction between reward size and accumulated reward. To determine whether the deficit following PFo-Rh functional disconnection reflects impaired cue–reward associations or altered expected value estimates, we tested the PFo X Rh group, as well as a control group (n = 2), on a task in which neither reward nor response information was provided by visual cues (Fig. 1B, self-initiated task). In the self-initiated task, monkeys obtained reward by simply touching and releasing a bar; they were free to press and release the lever at their own pace. Reward size was varied in a block design, with blocks of different reward sizes (1, 2, 4, or 8 drops, 25 responses per block) randomly interleaved. Monkeys in both the PFo X Rh and control groups quickly learned that releases led to reinforcement; there was no significant difference between groups in the number of responses in a session (ANOVA, F_(1,4) = 0.005, p = 0.95).

In line with previous accounts (Bouret and Richmond, 2010; Clark et al., 2012), control group performance was significantly affected by reward size. Specifically, release intervals decreased with increasing reward size (Fig. 7A; GLMM; reward size: z = 4.7, p < 10⁻⁶). In contrast, there was no significant effect of reward size on release intervals for the PFo X Rh group (Fig. 7A; GLMM; reward size: z = 0.5, p = 0.57). Direct comparison of control and PFo X Rh group performance revealed a significant effect of PFo-Rh functional disconnection on self-initiated task performance (GLMM; group X reward size: z = 2.9, p < 10⁻³). Examination of the within-block response-by-response dynamics of normalized release intervals revealed that control group sensitivity to reward size in the self-initiated task developed rapidly, on average one response after block transitions (Fig. 7B control; GLMM; reward size × trial number: z = 2.7, p < 10⁻³). In contrast, the normalized release intervals for the PFo X Rh group only tend toward separation according to reward size in the last five responses of a block (Fig. 7B PFo X Rh; GLMM; reward size × trial number: z = 1.9, p = 0.06). Finally, both groups showed a trend toward an effect of accumulated reward in this task (Fig. 7C; GLMM; PFo X Rh, accumulated reward: z = 1.9, p = 0.05; Control, accumulated reward: z = 1.6, p = 0.11).

Figure 7.

PFo-Rh functional disconnection disrupts performance in the self-initiated task. A, Self-initiated performance is plotted as release interval (see Materials and Methods, Data analysis) (ordinate) versus reward size (abscissa) for the control and PFo X Rh group. The control group, unlike the PFo X Rh group, exhibited a significant decrease in release interval with increasing reward size. B, Within-block performance is plotted as standardized release interval (see Materials and Methods; ordinate) versus within-block trial number (abscissa) for the PFo X Rh (left) and control (right) groups. By definition, responses faster than average are positive, responses slower than average are negative. For the control, but not PFo X Rh, group responses separate according to reward size immediately (second trial of a block) after block transitions. C, Average release intervals (ordinate) are plotted versus normalized accumulated reward (abscissa) separately according to reward size. Left, PFo X Rh group; right, control group. Norm. Accum. Reward, Normalized accumulated reward; RI, release interval; msec, milliseconds.

In summary, control animals were able to use the block-wise structure in the self-initiated task to predict upcoming rewards and comparisons between the relative values experienced during different blocks to value different reward sizes. In contrast, the PFo X Rh group showed no such behavioral modulation according to reward size. Interestingly, neither group showed an effect of accumulated reward in this task. Two possible explanations for our failure to observe a robust increase in release intervals with increasing accumulated reward, relative to the effect of accumulated reward we observed in the visually cued task, are as follows: (1) visually cued sessions were approximately twice as long as self-initiated sessions, this could have resulted in increased fatigue or distraction, two factors that might have enhanced the effect of accumulated reward, and (2) in a related vein, self-initiated sessions did not require the monkey to attend to stimuli, nor inhibit responses during wait periods; this reduced cost on self-initiated trials could have diminished the effect of accumulated reward.

Functional disconnection of PFo and Rh: controlling for the visual environment

Although unlikely, it is possible that the impairment seen in the self-initiated task actually arises from an effect of the PFo-Rh functional disconnection on stimulus—reward association. During the self-initiated task, there were a number of stimuli present in the testing chamber that had a constant relationship to reward (e.g., response lever, IR camera, video monitor, etc.). It could have been that the monkeys began to associate a given reward size with any element of this unvarying context. Changes in reward size across blocks would have required flexible updating of this association. Failure for release intervals to begin to diverge according to reward size over a 25-trial block could reflect an impaired ability to perform such rapid cue–reward updating. Although such a broad hypothesis is not falsifiable, a reduced version, that monkeys had experience with inspecting the video monitor for information about reward size (from their extensive testing in the visually cued task) and could have been associating the constant background image with variable reward sizes, can be tested.

Accordingly, we altered the self-initiated task, providing information about reward size in each block via unique visual cues. Unlike in the visually cued task, visual cues in the self-initiated task were presented on the first trial of each block and then remained visible throughout (see Materials and Methods, Tasks and Training). Comparing the control and PFo X Rh groups once more reveals a significant difference in performance (Fig. 8A, GLMM; group × reward size: z = 3.7, p < 10⁻⁴). Thus, the impairment that we originally observed in the self-initiated task persisted in a condition that included unique visual cues to reward size.

Figure 8.

Introducing a visual reward cue does not rescue performance in the self-initiated task. A, Cued self-initiated performance is plotted as release interval (ordinate) versus reward size (abscissa) for the control and PFo X Rh group. As for the standard self-initiated task, the control but not PFo X Rh, group exhibited a significant decrease in release interval with increasing reward size. B, The trial-by-trial dynamics of the responses of the PFo X Rh (left panel) and control (right panel) groups in the cued self-initiated task are plotted as standardized release intervals (ordinate) versus within-block trial number (abscissa). C, Average release intervals (ordinate) for the trials highlighted within the rectangular regions in (B) are plotted versus reward size (abscissa). For the PFo X Rh group, release intervals on first trials in a block are significantly faster than release intervals on subsequent trials. Additionally, for the PFo X Rh but not control group, release intervals tend to decrease with reward size only on first trials. Unlike the PFo X Rh group, control group responses were not faster on first trials in a block and actually showed a more straightforward relationship with reward on later trials (compare responses on fourth with first trials). RI, Release interval; msec, milliseconds.

Inspection of the trial-by-trial dynamics of the PFO X Rh group's responses in the cued self-initiated task reveals that: (1) there is indeed a trend toward a decrease in release intervals with increasing reward size for this group, but this occurs only on the first trial of a block (Fig. 8B,C PFo X Rh), and, (2) there is a large increase in release intervals in all reward conditions on trials 2–25 (Fig. 8B,C PFo X Rh). This suggests that monkeys could be using the changing of the visual cue in the first trial of each block (the only trial in which cues disappear/appear) as a trigger to action. This could explain the overall hastening of their responses in these trials. Furthermore, it appears this imperative cue might allow them to better estimate (to some degree) a trial's expected value. In contrast to the PFo X Rh group, the control group did not exhibit either of these effects (Fig. 8B,C, Control).

To determine whether an explicit imperative stimulus provided some savings following PFo-Rh functional disconnection, we tested a condition that included an imperative stimulus (the color discrimination target) but no visual cue to reward size on each trial (visually instructed task—Fig. 1). As in the self-initiated task, reward sizes in the visually instructed task varied across blocks of 25 trials.

In the visually instructed task, both the control and PFo groups performed both more accurately and efficiently for larger reward sizes, with changes in performance occurring immediately after block transitions (Fig. 9A,B, control; GLMM; error rates, reward size X group: z = 2.8, p < 10⁻³; reaction times; main effect of group and all interactions, p > 0.21). Additionally, both error rates and reaction times increased with increasing accumulated reward in both groups (Fig. 9C; GLMM; error rates and reaction times, both p < 0.01). Thus, it appears that providing a visual imperative cue provides at least a partial remediation of the deficit observed in the self-initiated task.

Figure 9.

An imperative cue partially rescues performance. A, Performance in the visually instructed task is plotted as both error rates and reaction times (left, right; ordinate) versus reward size (abscissa) for the PFo X Rh and control groups. Reaction times were significantly affected by reward size for both groups. B, Trial-by-trial response dynamics are plotted as standardized reaction times (ordinate) versus within-block trial number (abscissa) for the PFo X Rh (left) and control (right) groups. C, Group average error rates (top row) and reaction times (bottom rows; ordinate) are plotted versus normalized accumulated reward (abscissa) separately according to reward size. Left, PFo X Rh group; right, control group. RT, reaction time; msec, milliseconds; Norm. Accum. Reward, Normalized accumulated reward.

Effect of bilateral PFo or Rh lesions in the self-initiated task

It has previously been shown that bilateral lesions of PFo and Rh (Fig. 10A) (Simmons et al., 2010; Clark et al., 2012) alter the influence of visual-associative information on monkeys' behavior in the visually cued task. The previous study of bilateral PFo lesions did not address the possibility that in addition to this impairment these monkeys could also have a deficit in estimating expected reward value when value is not signaled by visual cues, as has recently been shown for bilateral Rh lesions (Clark et al., 2012) (Fig. 10B,C) and as we have shown above for functional disconnection of PFo and Rh.

Figure 10.

Effect of bilateral PFo and Rh lesions. A, Performance in the visually cued task is plotted separately—as error rates (ordinate) versus normalized accumulated reward (abscissa)—for the control (left), bilateral PFo (middle), and bilateral Rh (right) groups. Both the bilateral PFo and bilateral Rh groups were significantly different from controls in this task. B, Performance in the self-initiated task is plotted as release interval (ordinate) versus reward size (abscissa), separate traces correspond to data from the control and bilateral lesion groups. Bilateral PFo and bilateral Rh lesions significantly altered monkeys' performance in the self-initiated task. C, Trial-by-trial response dynamics are plotted as standardized release interval (ordinate) versus within-block trial number (abscissa) for the bilateral PFo (middle) and bilateral Rh (right) groups (see Fig. 7B for comparison with control group response dynamics). D, Group average release intervals (ordinate) are plotted versus normalized accumulated reward (abscissa) separately according to reward size. Left, PFo group; right, Rh group. RI, release interval; msec, milliseconds; Norm. Accum. Reward, normalized accumulated reward.

To determine whether bilateral PFo lesions alter performance when reward information is not signaled via visual cues, we tested this group in our self-initiated task. Comparison of control performance with the PFo lesion group revealed a significant effect of PFo lesions (Fig. 10B,C; GLMM, group z = 3.3, p < 10⁻⁴). As described above for the control and PFo X Rh groups, the bilateral PFo group's self-initiated performance was unaffected by accumulated reward (Fig. 10D; GLMM main effect of accumulated reward and all interactions, p > 0.32).

Discussion

We have demonstrated that PFo-Rh interaction is required for normal estimates of expected reward value. In simple instrumental tasks, we varied reward magnitude, as well as visual cues to reward and action. Across all tasks, intact monkeys responded with greater accuracy or faster responses in trials ending in larger rewards. Following PFo-Rh functional disconnection, monkeys were less sensitive to differences in expected value whether value was signaled via visual cues or varied predictably across blocks of self-initiated trials.

Previous work has shown that bilateral removal of either PFo (Simmons et al., 2010) or Rh (Liu et al., 2000; Clark et al., 2012) impairs performance in tasks similar to those used here. Here, we provide an important extension to these earlier reports. The impairments observed following bilateral removal of either PFo or Rh are consistent with explanations based upon either the independent or interactive function of these structures. By interrupting direct PFo-Rh communication within each hemisphere we have demonstrated that it is their interaction that is critical.

Following PFo-Rh functional disconnection monkeys displayed a significant impairment in a condition in which reward varied across blocks of trials but was not explicitly signaled via visual cues (self-initiated task). This suggests the deficit produced by PFo-Rh functional disconnection extended beyond stimulus—reward association. Prior accounts have focused on the effect of PFo or Rh lesions on object recognition (Meunier et al., 1993; Meunier et al., 1997), stimulus–stimulus association (Murray et al., 1993; Buckley and Gaffan, 1998b), and the use of stimulus–reward history (Jones and Mishkin, 1972; Buckley and Gaffan, 1998a; Murray et al., 1998), stimulus–reward contingency (Walton et al., 2010), or cues to specific reward value (Thornton et al., 1998; Izquierdo et al., 2004; Burke et al., 2008; McDannald et al., 2011) to guide decision making. All of these studies suggest PFo and Rh are primarily involved in imparting behavioral significance to external stimuli; consequently, our demonstration of significant impairments in a task that lacked visual cues to reward is unexpected, although we have recently described a similar impairment in monkeys with bilateral Rh ablations (Clark et al., 2012). Below, we discuss several plausible alternative explanations of our results.

Because it lacked a visual imperative cue, the self-initiated task also lacked a visual selective attention requirement. Additionally, because monkeys were free to respond at their own pace, the time between rewards was on average shortest in the self-initiated task. Thus, it is possible that in the visually cued and visually instructed tasks the additional requirements of attending to the target stimulus and waiting for the target color change acted as a cost that varied with reward size; i.e., attending and waiting is less aversive in eight drop versus two drop trials. These variable costs might have served to increase the contrast between relative expected reward values to a level that the PFo X Rh group was able to detect. Accordingly, the greater impairment in self-initiated trials could reflect the more similar, i.e., harder to discriminate, expected values in this task. While we cannot rule out this explanation, the results of a recent study that used a Pavlovian measure of expected value (anticipatory lipping) suggest that in normal animals the expected value of identical reward magnitudes does not differ across our visually cued and self-initiated tasks (Bouret and Richmond, 2010).

Recent work describing the heterogeneity of the cortical areas subsumed within our large Rh and PFo ablations, the pattern of connections between these areas, and impairments that follow from more selective ablations of these regions offer another possible explanation for the difference in the severity of the impairment across our visually cued and self-initiated tasks. Our Rh lesions encompassed both the perirhinal (area 35/36) and entorhinal (area 28) cortex. Previous studies have reported a greater incidence of stimulus–reward encoding in perirhinal versus entorhinal cortex (Liu and Richmond, 2000; Sugase-Miyamoto and Richmond, 2007). Additionally, selective lesions of either perirhinal or entorhinal cortex produce different impairments in classic tests of visual recognition memory. Perirhinal cortex lesions yield severe and long lasting impairments in delayed-nonmatch-to-sample tasks whereas entorhinal cortex lesions yield weaker and more transient deficits that are more dependent upon contextual rather than purely visual associations (Meunier et al., 1993; Buckmaster et al., 2004). Along the same vein, our PFo lesions included both lateral (areas 11/13) and medial (area 14) subdivisions. A recent study reported greater encoding of expected value in a self-initiated versus visually cued task in medial PFo, and the converse pattern in lateral PFo (Bouret and Richmond, 2010). Similarly, in a recent experiment examining the contribution of PFo subregions to assessments of value comparisons, selective medial, but not lateral, ablations resulted in a significant impairment (Noonan et al., 2010). Finally, several studies have described the specificity of connections between the perirhinal cortex, lateral PFo, and sensory and association areas (orbital network) and the entorhinal cortex, medial PFo, and motor and visceromotor areas (medial network) (Carmichael and Price, 1996; Rempel-Clower and Barbas, 2000; Kondo et al., 2005; Price, 2007; Saleem et al., 2008). Given the preceding, it is possible our monkeys' impairments in the visually cued and self-initiated tasks actually arose from separate disconnection syndromes. Under this proposal, interruption of the orbital network results in the deficit in the visually cued task and interruption of the medial network results in the impairment in the self-initiated task. The difference in the severity of the impairment in the visually cued and self-initiated conditions could reflect the differing degree to which perirhinal and entorhinal interactions with lateral and medial PFo, respectively, are required for the respective tasks.

An explanation for our findings that is based upon the larger networks in which PFo and Rh are embedded is bolstered by the results of similar experiments involving groups of animals with bilateral PFo or Rh lesions (Simmons et al., 2010; Clark et al., 2012) (and compare Fig. 10). Importantly, the qualitative effect of all lesions was the same, namely, reduced discrimination of reward size in the visually cued task and increased reward rate in the self-initiated task. However, the difference in the specifics of the effect of bilateral PFo and Rh lesions on behavior in the self-initiated task suggests that these structures likely play different roles in estimating expected value (compare Fig. 10). In general, impairments observed following focal lesions could be due to the disruption of processing at some site distant from, but connected to, the ablated structure. For example, PFo lesions reduce activity in the bed nucleus of the stria terminalis (BNST), and the magnitude of reduced BNST activity correlates with threat anxiety (Fox et al., 2010). Here, the impairment observed following PFo-Rh disconnection could arise either from removing processing within PFo (Rh) that depends upon input from Rh (PFo) or from removing balanced PFo-Rh input to a third structure that is also required for normal reward-guided behavior. As areas with strong connectivity with both PFo and Rh and evidence of involvement in reward-guided behavior, the ventral striatum, amygdala, hippocampus, and nucleus mediodorsalis of the thalamus represent interesting candidates for this latter possibility. While the exact roles of PFo and Rh in reward-guided behavior remain open questions (Eichenbaum et al., 2007; Murray et al., 2007; Padoa-Schioppa and Cai, 2011; Schoenbaum et al., 2011), our results demonstrate that their interaction is critical for modulating behavior in response to expected reward value.

Finally, recent studies have claimed a role for Rh and PFo, and prefrontal-medial temporal lobe interactions generally, in generating retrospective and prospective value estimates for items that are not currently visible (Eichenbaum and Fortin, 2009; Schacter and Addis, 2009; Peters and Büchel, 2010; Sellitto et al., 2010; Clark et al., 2012). In the self-initiated task, monkeys are required to compare current outcomes with outcomes that were experienced in the more distant past. Thus, the decreased sensitivity to reward size that monkeys with bilateral Rh lesions or PFo-Rh functional disconnection display in this condition could reflect the importance of the PFo-Rh circuit in value representations with strong mnemonic or contextual requirements.

Finally, there is evidence that a prefrontal–hippocampal circuit is important for both short-term memory (Axmacher et al., 2008) and reward-guided behavior (Ballard et al., 2011). Thus, it is possible that at least some of our observed effects of removing Rh, either bilaterally or via functional disconnection, result from removing specific inputs to the hippocampus.

In conclusion, we observed a significant impairment in estimating expected value following PFo-Rh functional disconnection. Given the evidence for the involvement of these structures in visual memory and visually guided behavior, the persistence of this impairment in conditions that do not require the formation of stimulus–outcome associations seems counterintuitive. Previous authors have noted that when using visual short-term memory or stimulus–outcome associations to guide their behavior, subjects must flexibly assign behavioral significance to a particular stimulus. We suggest that our results can be viewed as an extension of these findings. Namely, it appears that the interaction between PFo and Rh is critical for generating estimates of expected value whether the cue to reward expectation is an external stimulus or internal knowledge.