Learned Value Shapes Responses to Objects in Frontal and Ventral Stream Networks in Macaque Monkeys

Abstract

We have an incomplete picture of how the brain links object representations to reward value, and how this information is stored and later retrieved. The orbitofrontal cortex (OFC), medial frontal cortex (MFC), and ventrolateral prefrontal cortex (VLPFC), together with the amygdala, are thought to play key roles in these processes. There is an apparent discrepancy, however, regarding frontal areas thought to encode value in macaque monkeys versus humans. To address this issue, we used fMRI in macaque monkeys to localize brain areas encoding recently learned image values. Each week, monkeys learned to associate images of novel objects with a high or low probability of water reward. Areas responding to the value of recently learned reward-predictive images included MFC area 10 m/32, VLPFC area 12, and inferior temporal visual cortex (IT). The amygdala and OFC, each thought to be involved in value encoding, showed little such effect. Instead, these 2 areas primarily responded to visual stimulation and reward receipt, respectively. Strong image value encoding in monkey MFC compared with OFC is surprising, but agrees with results from human imaging studies. Our findings demonstrate the importance of VLPFC, MFC, and IT in representing the values of recently learned visual images.

Introduction

Humans and other animals readily learn to associate an arbitrary stimulus with a meaningful outcome, such as a reward, and this capacity for association is central to many forms of motivated behavior. Although the neural basis of reward-guided decision making is an active area of research, we have an incomplete picture of how the brain links object representations to reward value, and how this value information is stored and later retrieved.

Several regions within the frontal cortex may encode value signals that contribute to reward-based decision making, including the orbitofrontal cortex (OFC), the ventrolateral prefrontal cortex (VLPFC), and the medial frontal cortex (MFC). All of these cortical areas are reciprocally connected with the amygdala, a subcortical structure that contributes to learning stimulus–reward value associations in both macaque monkeys (Paton et al. 2006; Belova et al. 2008; Bermudez and Schultz 2010) and humans (O'Doherty et al. 2001, 2002; Gottfried et al. 2002; Li et al. 2011). Here, we assess the role of these 3 frontal areas and the amygdala in representing recently learned reward-predictive images.

In monkeys, lesions of OFC lead to impairments in value-based choices of both objects and actions (Rhodes and Murray 2013; Rudebeck, Saunders, et al. 2013). In addition, neurons in OFC signal the values of reward outcomes predicted by different stimuli (Tremblay and Schultz 1999; Padoa-Schioppa and Assad 2006; Kennerley et al. 2009; Kennerley and Wallis 2009) and change their activity patterns with changing stimulus-reward contingencies (Thorpe et al. 1983; Morrison and Salzman 2009). Notably, value encoding in OFC is significantly reduced after removal of amygdala inputs (Rudebeck, Mitz, et al. 2013).

A second region that contributes to reward-guided decision making is VLPFC. VLPFC lesions in monkeys produce impairments in object reversal learning (Butter 1969; Iversen and Mishkin 1970), the flexible application of rules (Baxter et al. 2009; Buckley et al. 2009), and attentional selection (Rushworth et al. 2005). Consistent with the effects of lesions to this area, neurons in VLPFC encode image value (Rich and Wallis 2014) and fMRI BOLD signals in VLPFC similarly track monkeys' choices in a reward-guided decision making task (Chau et al. 2015). In humans, VLPFC has been implicated in reversal learning (Cools et al. 2002).

A third part of frontal cortex, MFC, has also been implicated in reward-guided decision making. MFC lesions in monkeys yield deficits in object reversal learning (Chudasama et al. 2013) and action reversal learning (Kennerley et al. 2006), and neurophysiological recordings have shown that neurons in MFC area 32 encode information about stimulus value (Kaping et al. 2011), as do more dorsal MFC sites (Kennerley et al. 2009; Rudebeck, Mitz, et al. 2013). In addition, the majority of functional imaging studies in humans have implicated MFC in encoding value (Knutson et al. 2005; Kable and Glimcher 2007; Noonan et al. 2011; Clithero and Rangel 2014), valence (Costa et al. 2010; Chikazoe et al. 2014), and salience (Metereau and Dreher 2015).

Human fMRI findings describing value effects in these 3 areas are difficult to reconcile with neurophysiological recordings from monkeys, partly because of the use of varying nomenclatures and experimental designs, but largely because the value encoding regions identified in the majority of human fMRI studies and monkey single-unit recordings seem to occupy different territories in humans (MFC) and monkeys (OFC). For example, as indicated earlier, several neurophysiological studies in OFC areas 11 and 13 of monkeys have reported neurons that signal stimulus-evoked value responses (Thorpe et al. 1983; Tremblay and Schultz 1999; Hikosaka and Watanabe 2000; Padoa-Schioppa and Assad 2006; Kennerley et al. 2009; Kennerley and Wallis 2009; Morrison and Salzman 2009; Rudebeck, Mitz, et al. 2013). However, many human fMRI studies have typically emphasized findings in a different frontal area; MFC (Levy and Glimcher 2011; Sescousse et al. 2013; Clithero and Rangel 2014; Metereau and Dreher 2015). We can make some headway towards reconciling these differences by using appropriately crafted fMRI studies in monkeys to characterize species similarities and differences.

In the present study, we used fMRI in macaque monkeys to document areas involved in representing learned value in an attempt to bridge single-unit recordings in monkeys with fMRI studies in humans that assess full-brain activity. We took the same approach in monkeys (e.g., full-brain fMRI) to help identify areas with potentially similar functions, which could prove invaluable to future studies targeting such areas with experimental manipulations or neurophysiological recording techniques. Given the difficulty in relating human and monkey findings regarding stimulus–reward value coding in MFC and OFC, together with the recent finding in monkeys that areas outside OFC, rather than OFC itself, are likely to be essential for tracking changes in stimulus-reward contingencies (Rudebeck, Saunders, et al. 2013), we set out to examine brain regions signaling object–reward value associations. We used the full-brain coverage afforded by fMRI to assess and localize value representations resulting from learning novel image–reward associations, identifying a number of candidate brain regions for future study as to their causal roles in reward-based decision making.

Based on prior findings in both species, we expected to observe stimulus–reward value signals not only in frontal cortex and the amygdala, of primary interest in this study, but also in the visual cortex. We were particularly interested in IT, which contains cells with responses tuned to complex visual objects (Desimone et al. 1984; Baylis et al. 1987; Tamura and Tanaka 2001). For example, inferior temporal visual cortex in monkeys (Mogami and Tanaka 2006; Eradath et al. 2015) and object or scene selective cortex in humans (Krawczyk et al. 2007; Hickey and Peelen 2015) respond to reward-predictive images. In addition, neurophysiological recordings in monkeys (Apicella et al. 1991; Hassani et al. 2001) and fMRI studies in humans (Delgado et al. 2000; Knutson, Adams, et al. 2001; Knutson, Fong, et al. 2001; Yamamoto et al. 2012; Miller et al. 2014; Sescousse et al. 2014) have demonstrated that the striatum is involved in processing reward-predictive images.

Here, we took the approach of manipulating reward probability associated with an arbitrary set of learned visual images. Macaque monkeys were trained to associate images of objects with a high (75%) or low (25%) probability of water reward in sessions composed of choice trials (2 images) and view trials (1 image). An event-related fMRI design allowed for the independent measurement of BOLD responses evoked by viewing reward-predictive images, reward receipt, or both. We modeled choice behavior during training sessions and subsequent scanning sessions with a reinforcement learning algorithm to estimate the values of images on each trial. Our specific aims were to document that BOLD signals in response to rewarded images were a result of experience (i.e., learning) and to identify cortical and subcortical regions involved in storage and/or retrieval of learned image value, reward receipt, or both.

Materials and Methods

Monkeys

Three adult male rhesus monkeys (Macaca mulatta; weighing 4.7–7.1 kg at the beginning of training) were used in this study. Monkeys were pair housed when possible and kept on a 12-h light–dark cycle with periodic access to food during scheduled feeding times. All procedures were carried out in accordance with the Guide for the Care and Use of Laboratory Animals and were approved by the National Institute of Mental Health Animal Care and Use Committee.

General Training Procedures

Each monkey was surgically implanted under sterile conditions with a plastic head post while under isoflurane anesthesia. After recovery, monkeys were trained to sit in an fMRI-compatible chair and accommodated to the training booth. Monkeys were then trained to gaze at a fixation spot (0.25°) on a monitor to earn water reward. Finally, over the course of a few weeks, they were accommodated to the MRI environment while wearing ear protection.

Behavioral Control Software

MonkeyLogic (Asaad et al. 2008) was used to record eye position in both the test chamber and scanner and to synchronize behavioral data acquired in the scanner to fMRI data.

Visual Stimuli

Visual stimuli consisted of images of colored objects arbitrarily selected from the Internet. Images were 72 dpi resolution, equated for luminance using the SHINE toolbox (Willenbockel et al. 2010), fit within an 11° diameter fixation window, and placed on a 50% gray background. In the training booth, images were viewed on a 19-in color LCD monitor placed 57 cm from the monkeys' eyes. In the scanner, an Avotec video projector (1024 × 768 pixels) was used to present images on a screen monkeys could view through a mirror. Image sets consisted of 2 groups of distinct object categories (fire hydrants, telephones, airplanes, guitars, boats, ski boots, etc.) composed of 10 exemplars each (e.g., 10 chairs and 10 fire hydrants). Only exemplars of a given category were offered on choice trials (i.e., chair1 vs. chair2). Monkeys saw a given image, on average, 38 times per day, or 152 times per week. The same set of visual images was used within a week, but novel sets of images were used across weeks.

Choice and View Task

Monkeys learned about the probability of reward associated with images through a combination of instrumental (choice) and Pavlovian (view) trials in a Choice and View task. Monkeys were trained in 4-day schedules each week. The Choice and View task consisted of interleaved 2 alternative forced-choice trials and view only trials (Fig. 1), hereafter referred to as choice and view trials, respectively. On choice trials, following 250 ms of initial fixation on a spot within a 3° window, 2 images were presented 7.5° from the center of the screen. The monkey was required to maintain central fixation for an additional 250 ms before making a saccade to one of the images. Once the monkey had made a saccade, as determined by the eye position entering an 11° window surrounding the chosen image, the image not chosen disappeared immediately from the screen. The monkey was then required to maintain fixation within the chosen image fixation window for 750 ms. Upon successful completion of this fixation requirement, the chosen image disappeared, and after a pseudorandom delay chosen from a standard uniform distribution of 2–4 s, the monkey either received a reward or not, according to the assigned reward probability of the chosen image. Choice trials involved presentation of pairs of images and were of 3 types: 1) 2 images predicting a high probability of reward, 2) 2 images predicting a low probability of reward, and 3) 1 image each predicting a high and low probability of reward. For ease in presentation, we will refer to images predicting a high and low probability of reward as “high probability” and “low probability” images, respectively.

Figure 1.

Choice and View Task. The Choice and View Task consisted of 2 interleaved types of trials, choice trials and view trials, presented in random sequence. Monkeys initiated all trials by fixating on a spot in the center of the screen for 250 ms within a 3° window. After this fixation period elapsed, a single, centrally placed image appeared (view trial) or a pair of images appeared (choice trial). Choice trials always involved a choice between 2 images. The offered image pairs could take 3 forms: 2 high probability images, a high and low probability image, or 2 low probability images. Monkeys indicated their choice by making a saccade to one of the 2 available images, presented 7° from a central fixation spot, after a continued fixation requirement for 250 ms was met, and were rewarded based on the reward probability of the chosen image and continued viewing of that image within an 11° diameter window. On view trials, monkeys were rewarded based upon reward probability of the viewed image and continued viewing of that image within the same sized window used for choice trials. Trials were separated by a 2–8 s.

On view trials, following 250 ms of initial fixation upon a spot, a single image was presented in the center of the screen. The monkey was required to maintain gaze on the centrally placed image for 1 s within an 11° fixation window. Upon successful completion of this fixation requirement, the image disappeared, and after a randomized delay chosen from a standard uniform distribution of 2–4 s, the monkey either received a reward or not, according to the assigned reward probability of the image. A variable intertrial interval chosen from a standard uniform distribution of 2–8 s followed the reward delivery event. The same images appeared in choice and view trials.

If eye position fell outside of the initial fixation window, if monkeys made a premature choice or gazed away from the chosen image (choice trials) or from the centrally placed image (view trials), those trials were considered errors and were terminated. After any such errors, monkeys were required to begin a new trial with the same image or image pair.

Experimental Design—Weekly Schedule

The Choice and View task was used for each day of the 4-day schedule (Fig. 2). On the first day of each week, termed Reference day, all images predicted the same probability of reward (66%). We were therefore able to assess whether monkeys exhibited an overall preference for groups of images to be assigned a high or low probability of reward before those reward contingencies were introduced. After the Reference day, new reward contingencies were introduced. Specifically, on Training Days 1 and 2, one-half of the images in the set were assigned a low probability of reward (25%), and the remaining images were assigned a high probability of reward (75%). Test scans followed the 2 days of training; images continued to be rewarded at 25 and 75% probability. Image–reward probability assignments were fixed during Training Days 1 and 2 and Test scans.

Figure 2.

Training and scanning schedule. Monkeys performed the Choice and View Task 4 days per week according to a fixed schedule for training and fMRI scans. Each weekly schedule consisted of 4 consecutive days with a given stimulus set: Reference scan, 2 consecutive days of training outside the scanner (Training Days 1 and 2), and a Test scan. This was repeated with novel images every week. During Reference day scans, all images predicted a 66% chance of reward. During Training Days 1 and 2 and Test scans, half of the images predicted a 75% chance of reward, and half predicted a 25% chance of reward.

On Reference days and Test days, scanner runs lasted 15 min each; monkeys typically performed 6–8 runs, each consisting of 85–110 trials. Training conducted outside the scanner consisted of blocks of 100 trials; monkeys typically completed six- to eight 100-trial blocks.

Each monkey performed the Choice and View task consecutively for 8–10 weeks, which varied slightly because of scanner scheduling or occasional experimental complications. Some incomplete sessions were not used; the remaining 8 complete sessions were used from each monkey in all analyses. Regular training and scanning occurred over the course of 2–3 months for each monkey. To summarize, the weekly schedule involving Reference scan, Training Day 1 and 2, and Test scan was run 8 times per monkey, each time with a new set of images.

Eye-Tracking Systems and Fluid Delivery

Eye position was monitored in the MRI scanner (iView eye-tracking system, SensoMotoric Instruments, SMI, sampling frequency 60 Hz) and in the training booth (Arrington ViewPoint EyeTracker, Arrington Research, Scottsdale, AZ, USA, sampling frequency 220 Hz). Eye position data were filtered (4th order 350 Hz Bessel low pass filter) and recorded at 1 kHz using Monkey Logic. Water was delivered as a single 0.3 mL bolus using an air-pressurized system controlled with an electronic solenoid activated by a TTL pulse from MonkeyLogic. Volume delivery was calibrated by solenoid open times and delivered though a Crist Instruments Pressurized Reward System: 5-RPD-M1A02 (Mitz 2005).

Analysis of Choice Performance

Choice performance was analyzed to assess monkeys' learning of reward-predictive images. We counted the number of instances monkeys chose the high probability image out of the total number of completed choice trials in which high versus low probability images were presented for choice. We then calculated the mean percentage, and standard error of the mean (±1 SEM), of high versus low probability choices across 8 sessions for each day (Reference scan, Training Day 1 and 2, Test scan) for each monkey.

fMRI Data Acquisition

MRI scanning was carried out in the Neurophysiology Imaging Facility Core (NIMH, NINDS, NEI). All functional data were collected with a 4.7 T, 60-cm vertical scanner (Bruker Biospec 47/60) equipped with Bruker gradient coils (6 Ga/cm). A transmit coil, 20 × 12.5 cm dimensions, was situated over the posterior portion of the monkeys' heads. A pair of receive coils designed for optimal coverage of IT, amygdala, and OFC were positioned over the temporal and frontal lobes. Receive coils were 8.5 × 7 cm. Behavioral data were aligned to fMRI data using custom written MATLAB code operating from MonkeyLogic by recording and counting volume pulses, or TRs, from the scanner.

During scanner setup in each session, shimming was carried out using a field map measurement. Single-shot echo planar images (EPI) were carried out with the following parameters optimized for BOLD imaging: echo-time (TE), 22 ms; flip angle, 76°; repetition time (TR), 2.2 s; matrix size 96 × 48 mm coronal; voxel size, 1.5 mm isotropic. A low-resolution anatomical scan was acquired in the same session to serve as an anatomical reference (modified driven equilibrium Fourier transform sequence (MDEFT), TE, 4.13 ms; flip angle, 12°; TR, 11.7 ms; matrix size: 92 × 112; voxel size: 0.5 × 0.5 × 1.5 mm).

We also collected high-resolution anatomical scans at 0.5 mm isotropic resolution from a horizontal 4.7 T magnet (Bruker Biospec 47/60), while monkeys were under isoflourane anesthesia in an MRI-compatible stereotaxic frame. We used CARET (Van Essen, et al. 2001) to create models of the cortical surface of 1 monkey, to which all others were aligned. Surface volumes and specification files from CARET were used to display statistical fMRI data upon the cortical surface.

Anatomical scans from each session (MDEFTs) were registered to the high-resolution scan for each monkey and then aligned to that from a single monkey. Spatial transformation matrices for aligning anatomical scans were applied to statistical data analyzed in each session's native space. Data from each complete session (8) from each monkey (3) for each fMRI scan (Reference, Test) were then transformed into the same space, allowing for group analysis.

We measured the temporal signal-to-noise ratio (tSNR) in each of the 8 Test scan sessions in the 3 monkeys and created an average tSNR map. tSNR values were calculated from residuals after regression. Given that our typical imaging sessions contained on average 3200 time points, we determined that an appropriate tSNR cutoff value would be 40, based on empirical data and models (Murphy et al. 2007); this estimate is consistent with previous work (Simmons et al. 2010).

fMRI Data Analysis

Functional data were analyzed with Analysis of Functional NeuroImages (AFNI) (Cox 1996) and plotted on the cortical surface with Surface Mapping with AFNI (SUMA). Data were converted from the raw Bruker format to AFNI BRIK and HEAD files using custom written UNIX scripts, which maintained all header information. This information was required for AFNI to adequately carry out subsequent preprocessing, alignment, and analysis.

Preprocessing

Each run was corrected to compensate for geometric field distortions using the PLACE method (Xiang and Ye 2007), slice-time corrected, and resampled in pitch from 20° to plumb (de-obliqued). For each monkey, anatomical scans (MDEFTs) from each session were manually skull stripped and registered to a high-resolution skull-stripped scan from the same monkey using a 12-dimension affine transformation. The first 5 volumes of each run were removed to allow the scanner to reach a steady state, and the data were then concatenated across runs within a single session, motion corrected using volume registration, and blurred at 2 mm.

Analysis Methodology

After preprocessing, 2 separate analyses were conducted. In the first analysis, a deconvolution analysis allowed for the inspection of each voxels temporal response profile, plotted in percent signal change from image or reward onset to trial end, and for the construction of relevant t-maps from the peak of the estimated BOLD response. Time courses for several clusters of interest are plotted in the corresponding figures, below.

Having established that monkeys learn high and low probability images, we quantified the values of images in a second analysis by fitting monkeys' choice behavior on Training Days 1 and 2, and the Test scan sessions with a reinforcement learning model. By fitting monkeys' choice behavior with a reinforcement learning model, we were able to estimate trial-by-trial variations in the values of images and to quantify results in terms of value. We then used image value estimates as regressors in a general linear model (GLM) of the fMRI data.

Deconvolution Analysis

Deconvolution analysis was carried out on all Reference and Test scan sessions to generate BOLD response time courses for chosen images at the time of choice, singly viewed images, and reward delivery. Choice times, image onset times from view trials, or reward delivery times were deconvolved as a linear combination of 8 uniform B-spline basis functions that generated a time series of β coefficients at each voxel approximating the maximum length of a single trial (15.4 s). Nuisance regressors included head motion (measured as residuals from the registration procedure) and scanner drift (linear detrending). This generated a time course of β coefficients at each voxel, which was transformed to percent signal change. Contrast maps (t-maps) and bar plots of percent signal change were created from an average of the peak BOLD response from the 8 sessions in the 3 monkeys, which was estimated to occur at 4.4 s from image onset, for Reference and Test scans. Average BOLD response time courses were generated from view trials in each session. (SEM is calculated across sessions; there were 24 sessions in total, 8 for each of the 3 monkeys).

Modeling Choices and Estimates of Value

In a separate analysis, we used the monkeys' choice behavior on Training Days 1 and 2, and the Test scan to derive trial-by-trial estimates of value for each of the images the monkey chose or viewed during that week. To fit the temporal difference (TD) learning model, we extracted the choice trials from the larger sequence of trials maintaining their temporal structure. We used choice trials in which the animals chose between high and low value images. We used a standard TD model to estimate learning from feedback, as well as to estimate the inverse temperature. Specifically, value updates were given by:

$$v_i(k+1)=v_i(k)+\alpha (R-v_i(k)),$$ (1)

where $v_i$ is the value estimate for option i, R is the reward feedback for the current trial, and α is the learning rate parameter. These value estimates were then passed through a logistic function to generate choice probability estimates:

$$d_1(k)=(1+e^{\beta (v_2(k)-v_1(k))}{)}^{-1},d_2(k)=1-d_1(k).$$ (2)

The likelihood is then given by:

$$f(D|{\alpha }_f)=\prod _k[d_1(k)c_1(k)+d_2(k)c_2(k)],$$ (3)

where $c_1(k)$ had a value of 1 if a low value image was chosen on trial k, and $c_2(k)$had a value of 1 if a high value image was chosen on trial k; otherwise they had a value of 0. The initial value of each stimulus was set to 0.66, consistent with the reward rate for all images during Reference scans prior to training. Standard function optimization techniques were used to maximize the log of the likelihood of the data given the parameters. As the estimation can settle on local minima, we used 20 initial values for the parameters. The maximum of the log-likelihood across fits was then used. The model was fit to each weekly set of training and test sessions.

Using the fitted learning rate parameter, we then computed trial-by-trial values (using Eq. 1) for the full sequence of choice and view trials the monkeys experienced across Training days and Test scans. Specifically, for view trials, the value of the image category (high or low probability of reward) was updated based on the fitted learning rate parameter of the model and whether the viewed image was rewarded or not. Likewise, on choice trials, the value of the chosen image category (high or low probability of reward) was updated based on the parameters of the model and whether the chosen image was rewarded or not.

These value estimates (plotted in Supplementary Fig. 1) were used to parametrically modulate regressors that modeled BOLD responses to chosen or singly viewed images during the Test scans. Value estimates were first mean centered and used to construct 2 regressors based on choice times from choice trials, or image onset times from view trials. The first regressor modeled the mean response relative to baseline. Of greater interest was the second regressor that modeled residual variance around the mean response in terms of fluctuations in image value for choice and view trials. Reward events were also modeled. Nuisance regressors included head motion and scanner drift. Regressors of interest were convolved with a standard hemodynamic response function (HRF) and entered into a GLM used to obtain regression coefficients for each regressor. Regression coefficients for each Test scan were then entered into a mixed-effects ANOVA with monkey nested in session, and session treated as a random factor, to generate contrast maps of value regressors from view trials during Test scan sessions. All contrast maps presented in the results are from view trials.

Thresholds and Multiple Comparison Corrections

All maps were initially thresholded at a P < 0.005 uncorrected. This corresponded to FDR-corrected P values ranging between 0.028 and 0.004 (documented in further detail in figure legends). In addition, a cluster size threshold of 7 voxels was used to control the familywise error rate at P < 0.001 in all maps shown. The cluster size threshold (7 voxels) corresponded to the 99.9% quantile of the distribution of the maximum cluster size simulated via 10 000 Monte Carlo stimulations (Forman et al. 1995), using estimates of spatial smoothness based on the residuals of the GLM based on the reinforcement learning model.

Results

Monkeys learned a novel set of visual–reward associations each week. Half the images predicted a high probability of reward (75%) and half predicted a low probability of reward (25%). Using fMRI, we identified brain regions that encoded the values predicted by the newly learned images. Given previous findings in both macaque monkeys and humans, we expected to identify image value effects in several frontal cortical areas, visual processing areas, including IT, and subcortical areas including the amygdala and striatum. In the following sections, we describe the behavioral results, followed by a systematic survey of cortical and subcortical areas whose activity is determined by recently learned image value, summarized in Table 1 (see also Supplementary Figs 2 and 3).

Table 1.

fMRI effects of image value

Region	Area	Peak t-value, view (P value)	Peak t-value, choice (P value)	tSNR at peak t
Visual cortex	V1–V4	7.3 (3.46 × 10−7)	6.75 (1.11 × 10−6)	132
IT	STS lower bank	6.6 (1.55 × 10−6)	6.2 (3.76 × 10−6)	100
	TEO, TE	7.2 (4.27 × 10−7)	6.8 (1.00 × 10−6)	87
Temporal polar	TG	4.8 (9.62 × 10−5)	6.2 (3.76 × 10−6)	66
VLPFC	12	4.6 (1.54 × 10−4)	4.8 (9.62 × 10−5)	115
Insula	Anterior	5.5 (1.85 × 10−5)	5 (5.99 × 10−5)	153
Striatum	Caudate head	6.2 (3.76 × 10−6)	5.6 (1.47 × 10−5)	136
	Putamen	11.9 (8.49 × 10−11)	8.9 (1.43 × 10−8)	151
Somatosensory cortex	3a, 3b, 1, & 2	6.9 (8.09 × 10−7)	6.7 (1.24 × 10−6)	140
	SII, area 7op	7.0 (6.53 × 10−7), 7.9 (1.00 × 10−7)	8.65 (2.3 × 10−8), 8.1 (6.75 × 10−8)	177, 149
Motor cortex	M1, premotor	6.9 (8.09 × 10−7)	6.9 (8.09 × 10−7)	191
Cingulate	Anterior	7.8 (1.232 × 10−7)	6.9 (8.09 × 10−7)	90
	Posterior	7.5 (2.28 × 10−7)	7.0 (6.53 × 10−7)	74
Thalamus	Pulvinar	5.8 (9.31 × 10−6)	6.3 (3.01 × 10−6)	125
	VP	10.1 (1.62 × 10−9)	9.1 (9.83 × 10−9)	121
MFC	10 m/32	5.1 (4.73 × 10−5)	4.98 (6.28 × 10−5)	125

Note: Peak t-statistics from either view or choice trials are shown in Table 1, with tSNR values of peak t-values. Data are from a regression analysis of image values taken from a reinforcement learning model of monkeys' choice behavior.

Behavioral Results

Each of the 3 monkeys completed 8 four-day schedules, with novel images each week. Averaging across the 8 weeks, Figure 3 shows the proportion of high probability images that were chosen on high versus low probability choice trials for each day of the 4-day training schedule, ± 1 SEM, calculated across the 8 sessions for each monkey. Monkeys showed no choice preferences during Reference scans, indicating that they valued the images to be assigned to the high and low probability reward conditions equally before the differential reward probabilities were introduced on Training Days. An ANOVA with factors of Monkey (subject 1, 2 and 3) and Day (Reference scan, Training Day 1, Training Day 2 and Test scan) revealed a significant effect of training (F_{3, 95} = 150.863, P < 1 × 10⁻¹⁵), but no effect of Monkey (F_{2, 95} = 1.963, P = 0.147), nor a significant interaction effect (F_{6, 95} = 0.252, P = 0.957) (Fig. 3). Monkeys expressed no systematic side bias (effect of side, ANOVA, F_{1, 95}, P = 0.281). Following 2 days of training outside the scanner, monkeys chose high probability images on 92.5% of choice trials involving high versus low probability images during the Test scans. Thus, as a result of training, each monkey developed a preference for images associated with a high probability of reward.

Figure 3.

Choice preferences. Plot shows image preferences for choice trials involving high versus low probability images for each day of the 4-day schedules. Preference for the high probability images increased over days, indicating that each of the monkeys learned the associations between images and the probability of reward. There was a significant effect of day (P < 1 × 10⁻¹⁵), but no effect of monkey (P = 0.147), nor an interaction effect (P = 0.957). Error bars are ±1 standard error of the mean (SEM) across sessions (8 sessions for each of the 3 monkeys).

Learning-Related Changes in BOLD Responses

We were interested in areas that changed their responses as a result of learning. To document the effects of learning, we created whole-brain activity maps and plotted corresponding BOLD response time courses from view trials that showed regions that responded more to high probability images than to low probability images. Based on this map, we then compared responses in several regions of interest to responses to the same stimuli from view trials from Reference scans, with the results shown in Figures 4 and 5.

Figure 4.

Training effects and BOLD responses in IT. (A,B) A cluster of voxels indicated by white outline (A) and corresponding BOLD time course from 3 monkeys (B) from view trials in Reference scan sessions; this significant cluster of voxels was first identified in Test scans (C). Before training, BOLD responses for high and low probability images do not differ from one another (A,B). Note, however, that IT does respond to reward-predictive visual images (B). (C) Voxels in IT (black outline) showing significant image value effects on view trials in the Test scan. (D) BOLD response time courses for images that predicted either high or low probability of reward for the voxels outlined in C, shown in the same manner as in B. After training, the same region of IT responds significantly differently to high and low probability images. The t-maps shown in A (no significant voxels) and C are a contrast of the peak BOLD response (4.4 s following image onset) of high probability images versus low probability images. Error bars are ±1 SEM of 24 sessions. Inset color bar shows a scale of P values from P = 0.001 to P < 6.5 × 10⁻⁷; data are FDR corrected to 0.001. amts, anterior middle temporal sulcus; rs, rhinal sulcus; sts, superior temporal sulcus.

Figure 5.

Image-related training effects. Percent BOLD signal change on view trials before training (Reference scans; black bars, images to be paired with high probability of reward; gray bars, images to be paired with a low probability of reward) and after training (Test scans; red bars, high probability images; yellow bars, low probability images) is shown for 6 areas in which significant voxel clusters were identified following training (P = 0.001, FDR = 0.01). The data indicate the peak BOLD response, 4.4 s following image onset. Error bars are ±1 SEM of 24 sessions.

To demonstrate how BOLD responses changed as a result of learning, we show reward-predictive effects and BOLD time courses from view trials in a cluster of voxels in IT in the right hemisphere (Fig. 4). For example, after training, a cluster in right IT that includes areas TEa, TEm, and TEad showed greater responses to images that predicted a high probability of reward relative to images that predicted a low probability of reward, as reflected in both t-maps (Fig. 4C, black outline) and BOLD time courses derived from the same cluster of voxels (Fig. 4D). We then assessed BOLD responses in the same voxels in the Reference scan (Fig. 4A, white outline). BOLD response time courses for images to be paired with either a high or low probability of reward, as determined from Reference scans (Fig. 4B), were estimated to peak at 4.4 s following image onset and return to baseline after 8.8 s. During the Reference scan, unlike the Test scan, although IT showed reward-predictive BOLD responses to images (see BOLD time course in Fig. 4B), responses to images to be paired with either a high versus low probability of reward were not significantly different from one another (note the lack of significant voxels in Fig. 4A). Therefore, any differential BOLD responses to high versus low probability images observed in the Test scan can be attributed to learning and were unlikely to be due to behavioral or neural biases present before training.

To demonstrate learning-related changes in other areas of interest, we plotted BOLD percent signal change at the peak BOLD response time during view trials, before and after training. To do this, we identified a number of significant clusters from Test scans from the contrast of high probability versus low probability images that survived a thresholded t-statistic (P < 0.0001, false discovery rate [FDR] = 0.001). We used these regions of interest to plot the peak BOLD percent signal change during Reference scans, prior to training. Figure 5 shows peak percent BOLD signal change for images before and after training in 2 distinct sites in IT, one in the fundus of the right STS (A; see also Fig. 4) and the other below the STS on the ventral lateral surface (B), as well as in MFC area 10 m/32, VLPFC area 12, the anterior putamen, and the anterior insula. All of these regions responded positively to visual images prior to training (during Reference scans), but none showed significantly greater BOLD responses to images to be paired with a high probability of reward (black bars) than images to be paired with a low probability of reward (gray bars). Following training, many regions exhibited a larger BOLD response to images that predicted a high probability of reward (red bars), together with a somewhat smaller BOLD response to images that predicted a low probability of reward (yellow bars). Whereas many areas showed an increase in BOLD response to the high probability images, together with a decrease in BOLD response to the low probability images, the MFC was an exception; the MFC only showed the significant decrease in BOLD response for images predicting a low probability of reward, and this decrease was more marked relative to other regions (Fig. 5C). Thus, regions initially showing no selectivity for 1 rewarded image category (25% or 75% reward) over another came to respond more to high probability of reward images than low probability of reward images following training (with MFC exhibiting slightly different training effects, as previously mentioned). We therefore conclude that any changes in BOLD responses to images were a result of experience gained in the 2 training sessions outside the scanner.

Given the clear effects of training on the responses to rewarded images, we asked whether training also changed the brain's responses to the receipt of reward. To determine whether the training procedure impacted BOLD responses to reward receipt in any area of the brain, we carried out a mixed-effects ANOVA, with monkey nested in session, and session treated as a random factor, on reward receipt with pretraining and posttraining as a factor. We found no significant effects, indicating that BOLD responses to reward receipt prior to training were not significantly different from BOLD responses to reward receipt after training. In sum, BOLD responses to reward receipt were not altered by the training procedure in any area of the brain.

Frontal Cortex: MFC, VLPFC, and OFC

Using the reinforcement learning analysis, we generated functional maps of image value from view trials and identified clusters of voxels that encoded image value in several frontal areas, including MFC, VLPFC, and OFC. We begin first by focusing on findings from MFC, as these may help to address a discrepancy between sites thought to encode value in monkeys (OFC) and humans (MFC). Figure 6A,B shows the locations of value encoding voxels in MFC area 10 m/32 (circle in Fig. 6B), immediately dorsal to the rostral sulcus. This region was only responsive to image value, showing no significant BOLD responses to reward receipt. Voxels in and dorsal to the cingulate sulcus (Fig. 6A,B) also encoded image value, likely in areas 9 and the supplementary motor area (SMA). Conversely, a region situated ventral to the rostral sulcus including OFC area 14 (indicated by an arrow in Fig. 6B, D) exhibited positive BOLD responses to reward receipt, but did not respond to image value. Thus, there was good signal from these voxels, but, despite our expectations, image value did little to modulate responses in this area. The cluster of voxels responsive to reward receipt extended laterally to include OFC areas 11 and 13 in both hemispheres. The distinction between cortex responding to image value and reward receipt is indicated by the circle and arrow, respectively, in Figure 6B,D. Figure 6E shows the time course of BOLD activation determined from a deconvolution analysis in a cluster of voxels in area 10 m/32 (see cluster outlined in black in Fig. 6A,B), demonstrating a greater BOLD response to high value images than low value images. The time course shown in Figure 6F shows the BOLD response to reward receipt in a cluster of voxels in areas 11/13 (cluster outlined in black in Fig. 6D). These effects were unlikely driven by a single monkey; the mean regression coefficients for image value (Fig. 6G, top) and reward receipt (Fig. 6G, bottom) are shown for each monkey, thresholded at the group-level t-statistic.

Figure 6.

Image value and reward receipt in MFC and OFC. A reinforcement learning model was used to estimate the values of images on each trial, which were used as a regressor in a general linear model of the data. (A,B) Significant effects of image value from view trials are shown in the mid-sagittal plane (A) and coronal plane (B), notably in a cluster of voxels on the medial wall (circle) in MFC area 10 m/32. t-Maps range from 3.125 (P = 0.005) to 7.0 (P < 6.5 × 10⁻⁷) and above; corrected FDR = 0.008. (C,D) Significant effects of reward receipt in OFC areas 11/13 and 14, shown in the mid-sagittal (C) and coronal (D) planes. t-Maps range from 3.125 (P = 0.005) to 7.0 (P < 6.5 × 10⁻⁷) and above; corrected FDR = 0.003. Note the distinction in the regions representing image value (circle) and reward receipt (arrow). (E) Percent BOLD signal change in MFC area 10 m/32 on view trials for high and low value images; cluster indicated in black outline in A and B. (F) Percent BOLD signal change following reward receipt in OFC areas 11/13; cluster indicated in black outline in D. Time courses were determined from a separate deconvolution analysis. Error bars are ± 1 SEM of 24 sessions. (G) The mean regression coefficients for image value (top) and reward receipt (bottom) are shown for each monkey, thresholded at the group-level t-statistic shown in panels A–D (P = 0.005 to P < 6.5 × 10⁻⁷; corrected FDR = 0.008). Scale bars are 5 mm. ac, anterior commissure; cs, cingulate sulcus; cc, corpus callosum; los, lateral orbital sulcus; mos, medial orbital sulcus; ps, principal sulcus; SMA, supplementary motor area.

VLPFC area 12 and OFC area 13b also encoded image value, as did a region dorsal to the cingulate sulcus (SMA). As shown in Figure 7, voxels in VLPFC encoding image value were found just lateral to the lateral orbital sulcus; these voxels are denoted with a black arrow in Figure 7A,B,C. Image value effects were present in both hemispheres. Significant voxels encoding image value were present in OFC area 13b, but only in the right hemisphere (Fig. 7C). The BOLD time course of the VLPFC cluster indicated in Figure 7A (cluster outlined in black) is shown in Figure 7D for high and low value images. Findings reported in MFC, VLPFC, and OFC were deemed to be reliable, in part, because tSNR values in these areas were >40 and were reasonable given the number of TRs collected each scan. Whole brain tSNR values are shown in Supplementary Figure 4.

Figure 7.

Image value effects in VLPFC area 12. (A–C) Significant effects of image value in VLPFC area 12 in the left and right hemispheres from view trials shown in the coronal plane (A), a sagittal plane through the right hemisphere (B), and on a reconstruction of the frontal and anterior temporal lobe cortical surface in the right hemisphere (C). The white line in panel C indicates the level of the coronal section shown in A. The black arrows in panels A, B, and C show the same voxel cluster in each view of the right hemisphere. The green outline in panel B indicates the location of the cortical “surface” shown in panel C. (D) Bold time courses for high and low value images from VLPFC cluster surrounded by black outline in A. Other sites of activation include area 13b, insula, and TG. t-Maps range from 2.8 to 5 (P = 0.01, FDR-corrected 0.01). Error bars are ±1 SEM of 24 sessions. Scale bar is 5 mm. cd, caudate; cs, cingulate sulcus; los, lateral orbital sulcus; mos, medial orbital sulcus; ps, principal sulcus.

Visual and Temporal Polar Cortex

As expected, based on previous studies, we also found a number of areas within visual cortex that encoded image value, including V1–V4, TEO, and TE. We also identified image value effects in temporal polar cortex (TG). Because we trained monkeys to learn the value of complex visual objects, we were particularly interested in IT, which contains neurons preferentially responsive to complex visual objects (Desimone et al. 1984; Baylis et al. 1987; Tamura and Tanaka 2001). One subregion encoding image value, for example, was identified bilaterally in anterior ventral IT. This patch was below the STS on the ventrolateral surface of the temporal lobe (Fig. 8). The BOLD time course of high and low value images is shown in Figure 8G,H for the clusters highlighted in black in the left hemisphere (Fig. 8A,C) and in the right hemisphere (Fig. 8B,D), respectively. Voxels within the lower bank of the STS also significantly encoded image value (note voxels near the sts label in Fig. 8C–F). Whole brain value effects, including visual areas V1–V4 and TEO, are shown in Supplementary Figures 2B,C and 3A–C.

Figure 8.

Image value effects in inferior temporal cortex. (A,B) Significant effects of image value from view trials were found below and within the superior temporal sulcus (STS) in IT cortex, shown in the coronal plane for both the left (A) and right (B) hemispheres. (C,D) Sagittal sections showing the same regions responsive to image value as in A,B. The white line in panels C–F indicates the coronal plane shown in A and B. (E,F) Reconstruction of the lateral surface of the left (E) and right (F) temporal lobe. The arrows in panels A, C, and E indicate the same significant voxel (t = 5.8) in TEad in the left hemisphere. Arrows in D and F indicate the same significant voxel (t = 5.2), also in TEad, in the right hemisphere. Value effects are also apparent within the STS in areas TEa and TEm in both hemispheres. t-Maps range from 3.8 (P = 0.001) to 7.0 (P < 6.5 × 10⁻⁷) and above. Corrected at FDR = 0.028. (G) Plot of BOLD responses to high and low value images in the clusters indicated in A and C. (H) BOLD responses to high and low probability images in the clusters indicated in B and D. Error bars are ±1 SEM of 24 sessions. Scale bar is 5 mm. amts, anterior middle temporal sulcus; ls, lateral sulcus; rs, rhinal sulcus; sts, superior temporal sulcus.

The right anterior temporal pole contained a large cluster of voxels that encoded image value (Fig. 9). The significant voxels extended caudally from area TG to area TEa and perirhinal area 36 (see Fig. 9E,G). The left anterior temporal pole also exhibited significant image value effects, but clusters there were smaller than on the right.

Figure 9.

Image value represented in temporal polar cortex. (A) A sagittal section showing the location of area TG, surrounded by a white box. Significant effects of image value from view trials were found within the anterior temporal pole, in ventral area TG (B,C), continuing caudally to TEa/36 (E,G). (C,E,G) show the anterior to posterior extent of value effects in TG and TEa in the coronal plane, with their corresponding sagittal sections shown in B, D, and F. White lines in B, D, and F indicate the coronal planes shown in C, E, and G; black lines in C, E, and G indicate the sagittal planes shown in B, D, and F. The significant voxels shown in the upper left of panels D and F involve the right anterior insula. t-Maps range from 3.8 (P = 0.001) to 7.0 (P < 6.5 × 10⁻⁷) and above. Corrected at FDR = 0.028. Scale bar (5 mm) applies to panels B–G. am, amygdala; arc, arcuate sulcus; ca, calcarine sulcus; cd, caudate head; cts, central sulcus; H, hippocampus; ls, lateral sulcus; lu, lunate sulcus; ps, principal sulcus; rs, rhinal sulcus; sts, superior temporal sulcus.

Task-Related Visual Activity in the Amygdala

Given that amygdala neurons in macaques respond to cues that predict reward (Paton et al. 2006), we expected that the amygdala would signal the expected values associated with high and low probability images. However, we did not identify significant image value effects (Fig. 10A) or reward receipt effects in this structure (Fig. 10B). Absence of image value or reward receipt effects were not due to low tSNR, because 1) we were able to recover signal from the lateral amygdala in a contrast of activations evoked by all visual images greater than baseline (Fig. 10B) and 2) our measured tSNR values from most of the amygdala (with the exception of the most ventral and medial part) range from 50 to 150, well within an acceptable range (Murphy et al. 2007; Simmons et al. 2010). tSNR values are shown for the whole brain in Supplementary Figure 4.

Figure 10.

Image value and visual responses in the amygdala. An anatomical mask was applied (not shown) to analyze data only in the amygdala; no significant effects of image value are present in the amygdala (Panel A). Visual responses are evident in the dorsal and lateral portions of the amygdala (t-statistic, P = 0.005, FDR corrected 0.008). Scale bar is 5 mm.

Dorsal and Ventral Striatum

We identified BOLD responses signaling image value in the head of the caudate nucleus and putamen, and responses to reward receipt in the ventral striatum. In the more anterior portions of the striatum, including the caudate head and putamen, voxels encoding image value were situated dorsally and displaced laterally from the midline (Fig. 11A,B). The location of voxels encoding image value was distinct from that encoding reward receipt, which was situated in the ventral striatum and ventral putamen (Fig. 11C,D). Examination of the right ventral putamen shown in Figures 11A (image value) and Figure 11C (reward receipt) shows that the same voxels were responsive to image value and reward receipt (see also conjunction analysis, below).

Figure 11.

Image value and reward receipt in the striatum. (A,B) Significant image value effects from view trials are shown at 2 anterior–posterior levels in the coronal plane. t-Maps range from 3.125 (P = 0.005) to 7.0 (P < 6.5 × 10⁻⁷) and above; corrected at FDR = 0.008. (C,D) Significant reward receipt effects are shown at equivalent anterior–posterior levels as in A,B. t-Maps range from 3.125 (P = 0.005) to 7.0 (P < 6.5 × 10⁻⁷) and above; corrected at FDR = 0.004. Dorsal striatum, including caudate head and putamen, encode value, but are not responsive to reward receipt. The ventral striatum, ventral putamen, and insula are responsive to reward receipt. Scale bars are 5 mm. am, amygdala; arc, arcuate sulcus; cd, caudate; arc.i, inferior limb of the arcuate; ps, principal sulcus; put, putamen; rs, rhinal sulcus; asc.s, superior limb of the arcuate; sts, superior temporal sulcus.

A Comparison of Choice and View Trials

Using the reinforcement learning model, we calculated regression coefficients from choice trials in addition to those from view trials and carried out a mixed-effects ANOVA, with monkey nested in session, and session treated as a random factor, to compare BOLD responses from image value in choice trials to image value in view trials. No voxels survived a corrected P value threshold of 0.05; no significant differences between chosen image value (because only the chosen image remains on the screen after choice) and value from singly viewed images were identified. Table 1 shows t-values from choice trials and view trials from a number of regions identified as encoding image value.

Areas Encoding Both Image Value and Reward Receipt

The insula, particularly its anterior portions, exhibited robust responses to both image value and reward receipt. Effects were bilateral, but t-values and cluster sizes were larger in the right hemisphere (Fig. 12A,B). We identified another portion of the insula, situated more caudally, that also responded to reward-predictive images (white arrows in Fig. 12A,C). This is likely the granular portion of the insula. The locations of voxels showing significant activation to reward receipt are shown in Figure 12C,D. Reward receipt also activated the precentral opercular area (PrCO, not shown) and somatosensory areas on the frontal operculum including areas 3a and 3b, and extending into the lateral sulcus to include areas 1, 2, and SII. Figure 12E shows the BOLD time courses in response to high and low value images and reward receipt of the cluster highlighted in Figure 12A,B.

Figure 12.

Image value and reward receipt effects in the insula. (A,B) Sagittal (A) and coronal (B) sections showing image value effects from view trials with the anterior insula circled in black. t-Maps range from 3.8 (P = 0.001) to 7.0 (P < 6.5 × 10⁻⁷) and above; corrected at FDR = 0.012. (C,D) The same region of the insula responsive to image value (A,B) is responsive to reward receipt. t-Maps range from 3.8 (P = 0.001) to 7.0 (P < 6.5 × 10⁻⁷) and above; corrected at FDR = 0.0078. The arrows in panels A and C indicate a portion of the posterior insula encoding only image value and not reward receipt. The vertical lines in B and D indicate the location of sagittal sections shown in panels A and C. (E) Time courses of BOLD responses to high and low value images in the anterior insula (denoted by black circle) and to reward receipt in same region. Error bars are ±1 SEM of 24 sessions. Scale bars are 5 mm. am, amygdala; arc, arcuate sulcus; arc.i, inferior limb of the arcuate; asc.s, superior limb of the arcuate; cd, caudate head; cts, central sulcus; ls, lateral sulcus; lu, lunate sulcus; put, putamen; rs, rhinal sulcus; sts, superior temporal sulcus.

We formally identified areas responsive to both image value and reward receipt using a conjunction analysis. The analysis revealed voxels in VLPFC area 12o (Fig. 13A) and the insula, including anterior portions as well as a centrally located dorsal portion of the dysgranular insula (Fig. 13B). This site of activation included adjacent gustatory cortex (G, not shown), PrCO (not shown), and somatosensory areas that included the face and oral cavity representation of areas 3a, 3b, 1, 2, and SII, and associated ventral posterolateral and ventral posteromedial thalamic nuclei. Limited regions of the striatum encoded both image value and reward receipt, including portions of the ventral putamen and a portion of the putamen situated more caudally (see Fig. 13C,D).

Figure 13.

Areas encoding both image value and reward receipt. A conjunction analysis revealed a limited number of areas that encode both image value (from view trials) and reward receipt (conjunction of t-Maps at P = 0.01 in each map). am, amygdala; cd, caudate head; los, lateral orbital sulcus; ls, lateral sulcus; mos, medial orbital sulcus; ps, principal sulcus; put, putamen; rs, rhinal sulcus; sts, superior temporal sulcus.

Discussion

Macaque monkeys performed a behavioral task with probabilistic reward that was optimized for event-related fMRI. During 2 days of training, and a follow-up Test scan, we monitored choice behavior to assess learning and monkeys' maintenance of learned values. A reinforcement learning model was used to estimate trial-by-trial values of images for fMRI analysis. Because monkeys required reinforcement to complete trials, we also modeled reward receipt, allowing us to separately assess learned image value and reward receipt.

We identified several cortical and subcortical sites that encoded image value. Based on evidence from human and monkey studies, we predicted and found image value encoding in MFC area 10 m/32, VLPFC area 12, as well as portions of the visual cortex (IT and visual areas V1 to V4), insula, and the striatum. In addition, a handful of regions exhibited responses to both image value and reward receipt, including the insula, striatum, and VLPFC area 12.

Image Value Signals Within Frontal Cortex

Neurons in OFC of monkeys signal the expected reward type and reward magnitude during presentation of reward-predictive images (Thorpe et al. 1983; Tremblay and Schultz 1999; Hikosaka and Watanabe 2000; Wallis and Miller 2003; Padoa-Schioppa and Assad 2006; Kennerley et al. 2009; Kennerley and Wallis 2009; Morrison and Salzman 2009; Rudebeck, Mitz, et al. 2013). In addition, neurons in monkey OFC are active in relation to reward probability (Kennerley et al. 2009; Kennerley and Wallis 2009). We therefore predicted that OFC would encode learned image value, though recent lesion studies have called into question the behavioral significance of value signals observed there (Rudebeck, Saunders, et al. 2013). One striking result of the present study, however, was the lack of robust coding of image value in OFC, with the exception of a small cluster of voxels in area 13b in the right hemisphere. This unexpected result might reflect the nature of value coding we examined. Other studies have used multiple flavors or quantities of juice, both of which might convey a sensory quality to the rewards. OFC may be specialized for associating sensory properties of outcomes (e.g., flavors, amounts) with reward-predictive visual images or actions (Noonan et al. 2011; Schoenbaum et al. 2011; Rudebeck and Murray 2014). Our task used the probability of water reward as a means of manipulating value, and therefore, all images in our task predicted the same fluid outcome and amount of reward (1 small drop, 0.3 mL). Accordingly, neurons signaling reward type or magnitude would not be expected to be differentially active on trials with high versus low probability images in our task. On this view, an fMRI design with value manipulations relating to palatability or magnitude of reward might activate OFC (Simmons et al. 2014). Another possible explanation for the lack of robust image value coding in OFC could be related to when we assessed image value effects. In our design, we measured image value effects for recently learned image–reward associations; the OFC may be more involved in learning than in maintaining learned representations (Gottfried et al. 2002; c.f. Rudebeck, Mitz, et al. 2013). Consistent with this idea, Nelissen et al. (2012) have demonstrated that orbitofrontal areas in monkeys (including area 12) are more active to cues predicting cocaine relative to a control cue during the Pavlovian conditioning process, but show little differential response to images prior to, or after, conditioning.

We identified 2 regions within frontal cortex with large clusters of voxels signaling image value: one in MFC and the other in VLPFC. In MFC, the cluster of voxels responsive to image value was situated on the medial wall above the rostral sulcus, in area 10 m/32. This finding is in agreement with a large body of literature identifying MFC in humans as encoding the value of stimuli or involved in computing values for choices (Knutson et al. 2005; Kable and Glimcher 2007; Rolls and McCabe 2007; FitzGerald et al. 2009; Kahnt et al. 2010; Kim et al. 2011; Levy and Glimcher 2011; Chikazoe et al. 2014; Metereau and Dreher 2015). Meta-analyses incorporating data from paradigms including choice tasks, gambling, passive viewing, and reversal learning with both primary and secondary reinforcers support the notion that MFC plays a central role in the computation of value (Levy and Glimcher 2012; Sescousse et al. 2013; Clithero and Rangel 2014), the representation of pleasant visual art or music (Ishizu and Zeki 2011), rewarded visual cues (Cox et al. 2005; Serences 2008; Weil et al. 2010), or pleasantness in general (Rolls and McCabe 2007; Costa et al. 2010; Kuhn and Gallinat, 2012). Despite the abundance of fMRI evidence indicating that human MFC is involved in value representation and its computation, little such evidence is available in monkeys, due in large part to the relative inaccessibility of MFC to neuronal recordings. Commensurate with our findings, 1 study with recordings above the rostral sulcus in area 32 of monkeys has shown that this area encodes information about stimulus value (Kaping et al. 2011). Other studies have targeted OFC area 14, sometimes considered part of MFC, and have reported a mixture of value and reward receipt responses (Bouret and Richmond 2010; Monosov and Hikosaka 2012; Strait et al. 2014). Yet other studies in monkeys have examined more dorsal regions of MFC, in the dorsal bank of the cingulate sulcus, where neurons also respond to both image value and reward receipt (Kennerley et al. 2009; Kennerley and Wallis 2009; Rudebeck, Mitz, et al. 2013). In the present study, voxels responsive to image value were observed in this region (Fig. 6B).

A second region showing robust coding of image value was located in VLPFC area 12. These findings are consistent with a number of other studies. For example, Rich and Wallis (2014) recorded from areas 12 and neighboring OFC areas 11, 13, and 14 in a task that manipulated image value with reward gains or losses. Though neurons that encoded image value were found in all areas, area 12 was found to have the largest proportion. Reward expectation responses in area 12 of monkeys have been reported in neurophysiological recording studies (Ifuku et al. 2003, 2006; Ohgushi et al. 2005). Using fMRI, Chau et al. (2015) focused on identifying signals that tracked monkeys' adaptive win-stay/lose-shift behavior and identified such signals in area 12. The image value signals we have identified in area 12 are consistent with their findings and may help animals update choices when reward contingencies change.

Lesion experiments in monkeys implicate VLPFC in mediating flexible stimulus–reward association. VLPFC lesions in monkeys yield impairments on object reversal learning (Butter 1969; Iversen and Mishkin 1970), on tasks requiring the flexible application of rules (Bussey et al. 2001; Baxter et al. 2009; Buckley et al. 2009) and in attentional selection (Rushworth et al. 2005). Human fMRI findings support the notion that VLPFC is involved in reversal learning (Cools et al. 2002). These findings are consistent with the present results. Taken together, VLPFC likely represents image value as part of its contribution to the selection of adaptive choices, especially object-based choices and perhaps also the rules or contexts that guide them.

Image Value Coding in Visual Areas and a Comment on the Amygdala

Our finding that IT encodes recently learned image value is consistent with those from fMRI work in humans demonstrating that BOLD responses to visual cues in early visual cortex are affected by reward probability (Serences 2008), reward magnitude (Small et al. 2005; Weil et al. 2010), and the valence of reward-predictive cues (Metereau and Dreher 2013). Reward magnitude also affects BOLD responses in scene-selective and face-selective visual cortex in the context of a working memory task (Krawczyk et al. 2007) and in object selective visual cortex in the context of a naturalistic visual search task (Hickey and Peelen 2015).

Our results showing that IT encodes the newly learned values of complex images complements that of Mogami and Tanaka (2006) and more recently Eradath et al. (2015) who investigated object–reward associations using single-unit recording methods in monkeys. It can be difficult to assess the spatial distribution of effects using neurophysiological recordings, as the recordings may not sample the full extent of the area, unlike the whole-brain coverage achieved with fMRI. Although the sites we identified as encoding image value below the STS are remarkably similar in location to those reported in Mogami and Tanaka (2006), our work using fMRI extends their findings by identifying additional image value encoding sites in the lower bank of the STS. Our demonstration that IT encodes recently learned image value is also consistent with neurophysiological recordings from IT showing that responses in anterior IT come to encode behaviorally meaningful images (Jagadeesh et al. 2001). In addition, we found that image value effects in IT appear to be organized in patches. Patches in the ventral visual stream are commonly associated with a face processing system (Wang et al. 1998; Tsao et al. 2003, Pinsk et al. 2005). The patch-like effects we identified in IT could be related to object selective cortex, given that monkeys learned associations between objects and the probability of reward (Fujita et al. 1992; Bell et al. 2009). This idea awaits further empirical investigation.

The amygdala has been implicated in processing images associated with value in monkeys (Paton et al. 2006; Bermudez and Schultz 2010), and we expected to identify BOLD responses that encoded image value. Nevertheless, we did not identify significant image value effects in the amygdala, despite the amygdala being responsive to visual stimuli in our task, and adequate tSNR. Notably, the amygdala and IT in both humans and macaques are responsive to facial expressions of emotion (Whalen et al. 1998; Vuilleumier et al. 2001; Pessoa et al. 2002; Hadj-Bouziane et al. 2008; Liu et al. 2015) and the human amygdala is likewise responsive to arousing pictures (Sabatinelli et al. 2009). In addition, it has been found that loss of amygdala input reduces modulation of the visual cortex to facial expressions of emotion relative to neutral faces (Vuilleumier et al. 2004; Hadj-Bouziane et al. 2012; c.f. Edmiston et al. 2013), an effect often attributed to direct amygdalo-cortical connections (Amaral et al. 2003; Freese and Amaral 2005). The lack of learned value effects in the amygdala suggests that rewarded images are processed differently from emotional facial expressions. If so, perhaps the amygdala relays information to IT during learning, when image and reward contingencies change, but not after learning. This fits with our interpretation that the effects we document in IT are likely due to plasticity that occurred during learning.

Notably, neither the frontal eye field (FEF) nor the lateral intraparietal area (LIP) encode image value in our task (see Supplementary Figs 2 and 3). To the extent that LIP and FEF are part of an attention network (Schall 2004; Bisley and Goldberg 2010), our value-coding signals may not be due to heightened attention to the high versus low probability images. Instead, perhaps image representations themselves are altered as a result of associative learning (i.e., plasticity), as suggested by Arsenault et al. (2013). If so, this interpretation may well apply to other areas where we identified image value effects. Nevertheless, rewarding stimuli attract attention and affect responses in visual areas (e.g., V4, Baruni et al. 2015; IT, Mogami and Tanaka 2006), and it is notoriously difficult to dissociate reward effects from attentional effects (Maunsell 2004). Accordingly, we cannot rule out entirely an attentional interpretation of the value effects we report.

Striatum and Insula Encode Image Value

We found robust value coding in the caudate head and putamen, as expected, consistent with the proposal that these value responses correlate with motivation to acquire rewarding outcomes (O'Doherty, 2004; Zink et al. 2004; Miller et al. 2014) and with results from neurophysiological recordings in monkeys (Hassani et al. 2001) and fMRI data in humans (Delgado et al. 2000, 2003; Delgado 2007; Knutson, Adams, et al. 2001; Knutson, Fong, et al. 2001; Miller et al. 2014; Sescousse et al. 2014). Our finding of image value effects in the putamen is consistent with results from both monkeys (Hassani et al. 2001) and humans (Knutson, Adams, et al. 2001; Knutson, Fong, et al. 2001) during delays as subjects anticipate reward. Our finding that the insula encodes the values of visual images fits with neurophysiological recordings in monkeys; the site of activation we report is similar in location to sites recorded by Asahi et al. (2006) and Mizuhiki et al. (2012).

Responses to Reward Receipt

We also surveyed the brain for areas simply responding to reward receipt, which our event-related design allowed us to isolate. Unlike the responses to image value, which changed as a result of training, responses to reward receipt were equally evident before and after training; reward receipt encoding did not change as a result of experience. Significant responses to reward receipt were widely distributed throughout OFC, including areas 11, 13, and 14. This finding is in agreement with recordings from monkeys demonstrating that neurons in OFC areas 11 and 13 are responsive to reward receipt (Thorpe et al. 1983; Schultz et al. 2000; Tremblay and Schultz 2000; Pritchard et al. 2005; Padoa-Schioppa and Assad 2006; Simmons and Richmond 2008; Lara et al. 2009), as are neurons in area 14 (Bouret and Richmond 2010; Monosov and Hikosaka 2012; Rudebeck, Mitz, et al. 2013; Strait et al. 2014). Our finding that OFC responds to reward receipt is also in agreement with human fMRI studies demonstrating BOLD responses to the delivery of rewards (tastes) in OFC (Zald et al. 1998; reviewed in Small et al. 1999; O'Doherty et al. 2001, 2002; Rolls and McCabe 2007). We also determined that VLPFC area 12 encoded reward receipt, in accord with several studies in monkeys (Rolls et al. 1990; Ifuku et al. 2003, 2006; Lara et al. 2009) and humans (Small et al. 1999; O'Doherty et al. 2001, 2002). Interestingly, and in agreement with results from Arsenault et al. (2013), visual areas, including inferior temporal cortex, responded with a negative trending BOLD time course to reward delivery.

The reward receipt effects found in the putamen are consistent with some human fMRI data demonstrating that the putamen encodes the value of received reward in a paradigm in which subjects received fluid rewards in the scanner (Metereau and Dreher 2013). The sites we identified in the putamen that were responsive to reward receipt are similar to those reported in monkeys (Apicella et al. 1991), which provides some validation for our fMRI paradigm in monkeys.

Responses to reward receipt have previously been reported in the dysgranular insula and gustatory cortex of the insula (G) (Yaxley et al. 1990; Ito and Ogawa 1994; Scott et al. 1994; Asahi et al. 2006; Lara et al. 2009; Mizuhiki et al. 2012), so our identification of this reward responsive site is to be expected. Additionally, our fMRI finding that the insula encodes reward receipt is in agreement with several human fMRI studies in which subjects received rewards or gustatory stimuli in the scanner (Zald et al. 1998; Small et al. 1999; O'Doherty et al. 2001; Small 2010; Metereau and Dreher 2013; Rolls et al. 2015). The cluster of voxels we identified as responsive to reward receipt extended beyond gustatory cortex (G), including dysgranular insula, area 12o, and somatosensory areas 3a, 3b, 1, and 2. This fits with findings of Ifuku et al. (2003) and Ohgushi et al. (2005) who reported that the majority of reward responsive neurons were in areas 12o, precentral opercular area (PrCO), and somatosensory areas 3a, 3b, 1, 2, and SII (see Cerkevich et al. 2014 for a definition of the oral cavity representation in macaque monkeys). The responses we documented in the insula, area 12o, and somatosensory areas 3a, 3b, 1 and 2 were associated with reward receipt and therefore could be related to licking that occurs while the monkeys consume water. These same areas, however, have been shown to respond to electrical microstimulation of the ventral tegmental area (VTA; Arsenault et al. 2014), a finding that argues against a pure motor or somatosensory interpretation of the BOLD responses to reward receipt. The identification of areas responsive to gustatory stimuli outside of what is typically considered primary gustatory cortex (G) has also been reported in some human fMRI studies (O'Doherty et al. 2001; Sescousse et al. 2013).

Conclusions

We used whole-brain fMRI in macaque monkeys to identify regions responsive to image value, reward receipt, or both. Based on the literature we expected that OFC and amygdala would both be responsive to valuable visual images. Following the learning of novel reward-predictive images, however, activations in OFC were largely driven by reward receipt, not image value, and the amygdala likewise did not encode image value. As outlined in the Discussion, the negative results could be due to any of several possible reasons. Our fMRI findings identified one region in VLPFC and another in MFC, both of which have activations correlated with image value and could serve to guide advantageous choices. The main area in MFC, lying dorsal to the rostral sulcus, is in a similar location to the one described in multiple fMRI studies in humans. This area could be a homologue of a medial agranular area found in all mammalian brains, often called the prelimbic cortex. Although this area has sometimes been labeled as anterior cingulate cortex, it is important to recognize that it differs from the more dorsal and posterior cingulate areas in several respects.

Finally, IT and other visual areas were found to develop responses to learned behaviorally relevant, valuable images. Two interesting questions are apropos to this finding: 1) what is the source of such value-related signals in the visual system and 2) what are the neural mechanisms that underlie such experience-related plasticity? The present study serves as a platform for investigation of these and other questions concerning the causal contributions of different brain areas to reward-based decision making in macaque monkeys.

Supplementary Material

Supplementary material can be found at: http://www.cercor.oxfordjournals.org/.

Funding

This research was supported by the NIMH Intramural Research Program (MH002886) and a NARSAD Young Investigator Award from the Brain and Behavior Research Foundation to P.M.K.