Multiple Mechanisms for Processing Reward Uncertainty in the Primate Basal Forebrain

Abstract

The ability to use information about the uncertainty of future outcomes is critical for adaptive behavior in an uncertain world. We show that the basal forebrain (BF) contains at least two distinct neural-coding strategies to support this capacity. The dorsal-lateral BF, including the ventral pallidum (VP), contains reward-sensitive neurons, some of which are selectively suppressed by uncertain-reward predictions (U⁻). In contrast, the medial BF (mBF) contains reward-sensitive neurons, some of which are selectively enhanced (U⁺) by uncertain-reward predictions. In a two-alternative choice-task, U⁻ neurons were selectively suppressed while monkeys chose uncertain options over certain options. During the same choice-epoch, U⁺ neurons signaled the subjective reward value of the choice options. Additionally, after the choice was reported, U⁺ neurons signaled reward uncertainty until the choice outcome. We suggest that uncertainty-related suppression of VP may participate in the mediation of uncertainty-seeking actions, whereas uncertainty-related enhancement of the mBF may direct cognitive resources to monitor and learn from uncertain-outcomes.

SIGNIFICANCE STATEMENT To survive in an uncertain world, we must approach uncertainty and learn from it. Here we provide evidence for two mostly distinct mechanisms for processing uncertainty about rewards within different subregions of the primate basal forebrain (BF). We found that uncertainty suppressed the representation of certain (or safe) reward values by some neurons in the dorsal-lateral BF, in regions occupied by the ventral pallidum. This uncertainty-related suppression was evident as monkeys made risky choices. We also found that uncertainty-enhanced the activity of many medial BF neurons, most prominently after the monkeys' choices were completed (as they awaited uncertain outcomes). Based on these findings, we propose that different subregions of the BF could support action and learning under uncertainty in distinct but complimentary manners.

Introduction

Uncertainty about important future events, such as rewards, influences a wide range of behavioral states (Loewenstein et al., 2001; Bach and Dolan, 2012). First, to obtain rewards, it is often necessary to choose an uncertain or risky option. To mediate such actions, the brain must integrate information about uncertainty and value. Second, after performing actions leading to uncertain outcomes, it is important to monitor those outcomes and learn from them. These observations suggest that there may be distinct mechanisms and circuits that integrate information about value and uncertainty for action and learning, and that these mechanisms may operate on different time scales.

We hypothesized that these multiple uncertainty-related functions may in part be mediated by the basal forebrain (BF). Indeed, a recent study identified a population of medial BF neurons that integrate information about reward uncertainty and reward value during a Pavlovian conditioning procedure (Monosov et al., 2015). However, if and how these neurons participate in processing reward uncertainty and reward value during uncertainty-seeking decisions was not assessed. Moreover, if and how other regions of the BF process information about uncertain or variable rewards remains unknown.

Answering these questions is important because different BF subregions are believed to play complementary but partly distinct roles in behavioral, emotional, and cognitive control (Haber et al., 1985; Everitt and Robbins, 1997; Baxter and Chiba, 1999; Lin et al., 2006; Smith et al., 2009; Baxter and Bucci, 2013; Avila and Lin, 2014). The medial portion of the BF, comprising the medial septal-diagonal band complex (Mesulam et al., 1983; Everitt and Robbins, 1997), strongly projects to cingulate and hippocampal cortices (Mesulam et al., 1983; Koliatsos et al., 1988), and has been implicated in learning, memory, and outcome monitoring (Damasio et al., 1985; Everitt et al., 1988; Morris et al., 1992; Voytko, 1996; Everitt and Robbins, 1997; Semba, 2000; Baxter and Bucci, 2013; Monosov et al., 2015). The nucleus basalis of Meynert, located in ventrolateral BF, projects to sensory and/or attention-related cortical areas (Mesulam et al., 1983) and the amygdala (Koliatsos et al., 1988), and has been implicated in salience-driven behavioral control (Everitt and Robbins, 1997; Lin and Nicolelis, 2008; Avila and Lin, 2014; Peck and Salzman, 2014; Hangya et al., 2015). Finally, the dorsolateral part of the BF includes the ventral pallidum (VP), which receives strong projections from the nucleus accumbens and other regions of the striatum, and contains many GABAergic neurons (Parent et al., 1988; Richardson and DeLong, 1991; Baxter and Chiba, 1999; Avila and Lin, 2014; Root et al., 2015) that serve as a key output node of the limbic basal ganglia for the control of reward motivated behavior (Spooren et al., 1996; Smith et al., 2009; Haber and Knutson, 2010). Testing if and how single neurons in different BF subregions encode information about reward value and uncertainty during reward-seeking decisions and actions could therefore provide important information about how the BF supports behavior, particularly in naturalistic environments, in which outcomes are often variable or uncertain.

Here, we examined how BF neurons process reward uncertainty and value. We found that a subset of BF neurons most often found in the dorsal-lateral BF (in regions occupied by VP) were suppressed by reward uncertainty as monkeys made uncertainty-seeking (risky) choices. Another subset of BF neurons, most often found in the medial BF, signaled the subjective values of reward options during choices, and were selectively enhanced by reward uncertainty after the risky choices (during the anticipation of the delivery of uncertain outcomes). Our findings support the notion that different subregions of the BF contribute differentially to behavior under uncertainty, on different timescales, to help us approach or avoid uncertainty and learn from it.

Materials and Methods

General procedures.

Three adult male rhesus monkeys (Macaca mulatta) were used for the experiments (Monkeys B, R, and W). All procedures conformed to the Guide for the care and use of laboratory animals and were approved by the Washington University Institutional Animal Care and Use Committee. A plastic head holder and plastic recording chamber were fixed to the skull under general anesthesia and sterile surgical conditions. The chambers were tilted laterally by 35° and aimed at the BF and the anterior portion of the caudate nucleus. After the monkeys recovered from surgery, they participated in the reward-probability and reward-amount behavioral procedure.

Data acquisition.

While the monkeys participated in the reward-probability and reward-amount behavioral procedure, we recorded the activity of 298 single neurons in the BF. Monkey B yielded 44 neurons, Monkey R yielded 95 neurons, and Monkey W yielded 159 neurons. The recording sites were determined with 1-mm-spacing grid system, with the aid of MR images (3T) obtained along the direction of the recording chamber. This MRI-based estimation of neuron recording locations was aided by custom-built software (Daye et al., 2013) and was described in detail previously (Daye et al., 2013; Monosov and Hikosaka, 2013; Monosov et al., 2015). These methods were verified previously using histology (Monosov and Hikosaka, 2012, 2013; Monosov et al., 2015), and by in vivo MR imaging of the electrode at the site of the neuronal recordings in this and one previous study (Monosov et al., 2011).

Single-unit recording was performed using glass-coated tungsten electrodes (Alpha Omega). The electrode was inserted into the brain through a stainless-steel guide tube and advanced by an oil-driven micromanipulator (MO-97A, Narishige). Signal acquisition (including amplification and filtering) was performed using Alpha Omega 44 kHz SNR system. Action potential waveforms were identified online by multiple time-amplitude windows with an additional template matching algorithm (Alpha Omega). Neuronal recording was restricted to single neurons that were isolated online. Neuronal and behavioral analyses were conducted offline in MATLAB (The MathWorks).

Eye position was obtained with an infrared video camera (Eyelink, SR Research). Behavioral events and visual stimuli were controlled by MATLAB with Psychophysics Toolbox extensions (Brainard, 1997; Eastman and Huk, 2012; Herman et al., 2015). Juice, used as reward, was delivered with a solenoid delivery reward system (CRIST Instruments).

Reward-probability and reward-amount behavioral procedure.

The reward-probability and reward-amount behavioral procedure consisted of two distinct alternating trial blocks: a reward-probability block and a reward-amount block, with each block consisting of 18 trials (Monosov et al., 2015). Monkeys often performed >20 blocks on any given experimental session.

In the reward-probability block, three visual fractal conditioned stimuli (CSs) were followed by a liquid reward (0.25 ml of juice) with 100%, 50%, and 0% chance. In the reward-amount block, three CSs were followed by a liquid reward of 0.25, 0.125, and 0 ml. Thus, the expected values of the three CSs matched between the probability and amount blocks. To control for neuronal object preference, we used two fractal sets (i.e., for every CS, there were two different visual fractals).

Each trial started with the presentation of a green trial-start cue at the center of the screen. The monkeys had to maintain fixation on the trial-start cue for 1 s; then the trial-start cue disappeared and one of the three CSs was presented pseudorandomly. After 2.5 s, the CS disappeared, and juice (if scheduled for that trial) was delivered. In each trial, the CS could appear in three locations: 10 degrees to the left or to the right of the trial-start cue, or in the center. One block consisted of 18 trials with fixed proportions of trial types (each of the three CSs appears 3 times each block, 9 of 18 trials total).

In the remainder of the trials in each block (9 of 18), the monkeys chose among the CSs, with each trial as a two-alternative forced choice. Their choice preference was tested among the probabilistic CSs (used in the reward-probability block), among the amount CSs (used in the reward-amount block), and across the probabilistic and amount CSs. Therefore, choice trials tested monkeys' preferences among all reward CSs in this study.

Each trial started with the presentation of a purple trial-start cue at the center, and the monkeys had to fixate it for 0.5 s. After the monkey fixated the trial start cue for 0.5 s, a choice array was presented consisting of two CS fractals used in the Pavlovian procedure. The monkey had to continue to fixate until the trial start cue disappeared (0.5 s). Monkeys then made saccadic eye movements to their preferred CS fractals and fixated them for 0.75 s to indicate their choices. Then, the unchosen CS disappeared, and the monkeys waited for 1 s to receive the scheduled outcome (associated with their chosen fractal).

Throughout this study, the intertrial intervals ranged from 3 to 6 s. Approximately 1 in 5 intertrial intervals contained uncued events (chosen randomly). These could be either a juice reward alone (0.25 ml) or a ∼70 dB 0.15 s auditory white-noise burst paired with a brief screen flash of equal duration.

The methods of creating the visual fractals used in our experiments were as previously described (Miyashita et al., 1991; Yamamoto et al., 2012). The variability in luminance across the (1) visual fractals, (2) the trial start center spot, and (3) the gray background of the screen was minimized.

Data processing and statistics.

Spike-density functions were generated by convolving spike times with a 100 ms Gaussian filter. For display purposes, spike-density functions of single neurons in the figures were generated by convolving spike times with a 100 ms Gaussian filter. CS-modulated neurons were defined as those that varied their responses across the 4 possible reward predictions (100% 0.25, 50% 0.25, 100% 0.125, and 0 ml of juice) (Kruskal–Wallis test, p < 0.05; analysis window: last 750 ms of the single CS trial).

A neuron was further defined as uncertainty-enhanced if its response to the uncertain CS (50%) was significantly stronger than its responses to both certain CSs (100% and 0% CSs). Likewise, a neuron was further defined as uncertainty-suppressed if its response to the uncertain CS was significantly weaker than its responses to both certain CSs (100% and 0% CSs). These tests were performed with 2 separate two-tailed rank-sum tests (i.e., 50% vs 100% and 50% vs 0%; p < 0.05 with Bonferroni correction for multiple comparisons). Performing the initial Kruskal–Wallis test on CSs in the reward-probability block only did not change the results in this study. Our uncertainty selectivity identification procedure allowed us to identify neurons whose responses were selective for reward uncertainty in the reward-probability block and to further study their activity in the reward-amount block and during choice trials (Monosov and Hikosaka, 2013; Monosov et al., 2015).

To normalize single CS task-event related responses, we subtracted baseline activity (the last 750 ms of the intertrial interval) from the activity during the task-event related measurement epoch. To normalize choice task-event related responses, we subtracted the trial-start cue fixation epoch activity from the activity during the task-event related measurement epoch. To study the reward-prediction related responses during single CS presentation trials, we used the time window of 100 ms after CS presentation until the outcome delivery.

All statistical tests were two-tailed. For comparisons between two task conditions for each neuron, we used the rank-sum test, unless otherwise noted. For comparisons between two task conditions across the population average, we used a paired signed-rank test, unless otherwise noted. The statistical significance of all correlations (p < 0.05) was tested using a permutation test (null hypothesis: neuronal activity was independent of reward value) by shuffling the neural firing rates across the conditions (e.g., reward sizes) 10,000 times (Monosov et al., 2015).

To assess whether single neurons' uncertainty responses contained information about spatial location or object identity, we computed response indices (difference between neuronal responses to two conditions divided by their sum). Specifically, to test for CS spatial sensitivity, we compared responses for 50% CS when it was shown 10 degrees to the right versus 10 degrees to the left of center. To test for object-feature sensitivity, we compared responses to two distinct 50% CS fractal objects.

To display the choice-related dynamics of uncertainty signals among different groups of uncertainty-sensitive neurons, we sorted the activity into three trial types: (1) trials in which monkeys chose 0.25 ml of juice, (2) trials in which the monkeys chose 50% of 0.25 ml of juice, and (3) trials in which the monkeys chose 0.125 ml of juice. We then performed a running test of uncertainty sensitivity in 100 ms time bins, stepping in 1 ms steps. To confirm the tests, uncertainty sensitivity was also tested in nonoverlapping time windows. Uncertainty sensitivity was defined as when there was significant variance across the three choice trial types (Kruskal–Wallis test, p < 0.01) and activity in uncertain choice trials was significantly stronger or weaker than during each of the other two certain choice trials (e.g., 0.25 ml and 0.125 ml of juice chosen; two-tailed rank-sum tests; p < 0.05).

To quantify irregularity of spiking activity, we used the irregularity index developed by Davies et al. (Davies et al., 2006; Nakamura et al., 2008; Matsumoto and Hikosaka, 2009). This measurement does not require constant firing during the measurement period and is therefore useful for analyzing neurons with different firing properties (Davies et al., 2006; Monosov et al., 2015). Baseline period for irregularity and firing rate measures was defined as the last second of an intertrial interval.

Results

We recorded single BF neurons while Monkeys B, R, and W participated in a behavioral procedure having two distinct contexts or blocks: (1) a reward-probability block in which three visual fractals predicted 0.25 ml of juice with 100%, 50%, and 0% chance; and (2) a reward amount block in which a different set of three visual fractals predicted 0.25, 0.125, and 0 ml of juice, with no uncertainty (Fig. 1A). The two contexts had the same expected value across an entire block, but reward uncertainty was only present in the reward probability block. During neuronal recordings, to verify that the monkeys understood the behavioral procedure, we included (1) choice trials and (2) measured the monkeys' licking behavior. First, monkey's choice was largely guided by reward value (Fig. 1B). Second, as in our previous reports (Monosov and Hikosaka, 2013; Monosov et al., 2015), the magnitude of the monkeys' licking behavior was correlated to the value of the fractals in the reward-probability block (p < 0.001; Spearman's rank correlation) and the reward-amount blocks (p < 0.001; Spearman's rank correlation).

Figure 1.

Responses of neurons in the BF to certain and uncertain reward predictions. A, Monkeys experienced two distinct blocks: a reward-probability block (left) in which three visual fractal CSs predicted juice 0.25 ml of juice with 100%, 50%, and 0% chance; and a reward amount block (middle) in which three fractals predicted three “certain” amounts of juice (0.25, 0.125, and 0 ml with 100% chance). Behavioral procedure is shown on the right. TS, Trial start cue. B, Choice percentage of a single reward probability CS versus all the other reward-probability CSs (red). Choice percentage of a single reward amount CS versus all the other reward-amount CSs (black). Data are compiled from a dataset of 18,287 trials. C, Average responses of three BF neurons in the two blocks to the certain and uncertain reward predictions (time window: 100 ms following CS presentation until the outcome). D, Task dynamics of an uncertainty-suppressed neuron (top; same neuron as C, left) and an uncertainty-enhanced neuron (bottom; same as C, right). Spike activity is shown by raster plots (top) and spike density function (bottom). Activity is separated by the 4 possible reward predictions. Dark blue raster plots represent the activity in 50% CS trials in which reward was omitted. TSf, Monkey fixated trial start cue; US, unconditioned stimulus (trial outcome).

We found that many BF neurons were sensitive to reward uncertainty (also often called “reward risk”) (O'Neill and Schultz, 2010; Burke and Tobler, 2011b; Monosov and Hikosaka, 2013) and reward value in different manners. The average visual fractal CS responses of three reward uncertainty-sensitive BF neurons to certain (black) and uncertain (red) reward predictions are shown in Figure 1C. In the reward probability block, the first two neurons (Fig. 1C, left and middle) had the lowest firing rate when the delivery of reward was predicted with 50% chance (p < 0.01, Wilcoxon rank-sum test). They were therefore selectively suppressed by uncertainty. The third neuron (Fig. 1B, right) displayed the highest firing rate when the delivery of 0.25 ml of the reward was predicted with 50% chance (p < 0.01, Wilcoxon rank-sum test). It was selectively enhanced by uncertainty. In contrast to the results obtained in the reward-probability block, in the reward-amount block (black), the neurons' activity was correlated with the monkeys' reward preference and reward value (Fig. 1B). The magnitude of the first neuron's activation was negatively correlated with the amount of predicted reward (p < 0.05; Spearman's rank correlation). Its activity scaled with reward amount (i.e., no reward > small reward > large reward). The second and third neuron showed a similar pattern of activation, but their activity was positively correlated with the amount of predicted reward (p < 0.05; Spearman's rank correlation). The three example neurons illustrate that the BF contains positive and negative reward value-coding neurons that are selectively enhanced or suppressed by reward uncertainty.

Beyond the difference in the direction (suppression or enhancement) of uncertainty modulation of value-coding neurons in BF, we observed that the dynamics of the uncertainty selective signals differed across the uncertainty-suppressed and uncertainty-enhanced neurons. This can be observed in Figure 1D (activity is shown for the same neurons as in Figure 1C, left, and Figure 1C, right). After the uncertain-reward prediction, the uncertainty-suppressed neuron became suppressed relatively quickly, and this uncertainty suppression persisted until the end of the trial. In contrast, the uncertainty-enhanced neuron continued to increase its uncertainty-related response in a ramp-like manner until the time of the trial outcome (reward delivery or omission). These different dynamics of the uncertainty-enhanced and uncertainty-suppressed example neurons may suggest that they contribute to reward value processing and outcome-anticipation in distinct manners.

To assess the possibility that the uncertainty-suppressed and uncertainty-enhanced BF neurons belong to functionally and anatomically distinct neuronal populations, we (1) examined their anatomical locations and intrinsic neuronal characteristics, (2) assessed the similarities and differences in their representation of reward value, and (3) studied the dynamics of their uncertainty signals while monkeys chose between certain and uncertain rewards.

We explored widely in the BF (anterior–posterior range relative to the center of the anterior commissure was as follows: −2 to 3 mm; mediolateral range relative to the midline was as follows: 0–9 mm). Of the 298 recorded BF neurons, 80 were selectively modulated by reward uncertainty (see Materials and Methods); 39 of 80 BF neurons were enhanced and 41 of 80 were suppressed by reward uncertainty. Their locations are shown in Figure 2A. Uncertainty-enhanced neurons were concentrated in the ventral-medial portion of the BF that projects to the cingulate and hippocampal cortices, and plays important roles in learning, memory, and outcome monitoring (Damasio et al., 1985; Everitt et al., 1988; Morris et al., 1992; Voytko, 1996; Everitt and Robbins, 1997; Baxter and Chiba, 1999; Semba, 2000; Baxter and Bucci, 2013; Monosov et al., 2015). Uncertainty-suppressed neurons were concentrated in a region slightly posterior, in the dorsal-lateral portion of the BF occupied mostly by the ventral pallidum, an output nucleus of the limbic striatal-pallidal system (Haber et al., 1985, 1995; Spooren et al., 1996). This region may be distinct from other BF subregions due to its neuronal composition (e.g., like other pallidal structures, it is mostly GABAergic), anatomical connectivity (Spooren et al., 1996; Smith et al., 2009; Haber and Knutson, 2010), and its role in adjusting the motivational drive relative to reward (Smith and Berridge, 2005; Smith et al., 2011; Tachibana and Hikosaka, 2012; Root et al., 2015).

Figure 2.

Differences between uncertainty-enhanced and uncertainty-suppressed BF neurons. A, Reconstructed locations of uncertainty-enhanced (red dots) and uncertainty-suppressed (blue dots) BF neurons plotted on two coronal sections where they were most often found. The location of the coronal sections along the anterior–posterior axis relative to the center of anterior commissure (AC) is indicated in italics. The locations of the uncertainty-enhanced and uncertainty-suppressed neurons differed along the anterior–posterior, medial-lateral, and depth dimensions (p < 0.01 by Wilcoxon rank sum tests). Estimated locations of medial BF (mBF), lateral BF (lBF), caudate (cd), putamen (put), dorsal pallidum (dp), septum (s), and ventral pallidum (vp) are shown on the diagrams. Scale bar, 6 mm. Inset, Histogram represents the anterior–posterior locations of uncertainty-enhanced (red) and uncertainty-suppressed (blue) neurons relative to the center of the AC. Below the reconstructions are coronal T1 MR images taken with electrodes at the location of identified uncertainty-enhanced (left) and uncertainty-suppressed (right) neurons. After identifying a single neuron, the electrode was raised 300 μm and was fixed to the recording grid. Then, MR images were acquired under gas anesthesia. The electrode's shadow in each MR image (indicated with a yellow arrow) is the black line whose tip is further highlighted with a red arrow. The locations of coronal images relative to the center of the AC are indicated below. Scale bar, 1.01 cm. For visibility, MRI slices were aligned along the electrode track. B–D, Electrophysiological properties of uncertainty-enhanced and uncertainty-suppressed neurons. Error bars indicate SE. Individual neuron's properties are shown as scatters. B, Average spike shapes of uncertainty-enhanced and uncertainty-suppressed neurons (red and blue, respectively). *p < 0.01, the duration, from trough to peak, was statistically different (Wilcoxon rank sum test). Single neuron spike widths are shown on the right. Average baseline irregularity (C) and firing rate (D) for the two groups of uncertainty-sensitive neurons. *p < 0.01, significant differences (Wilcoxon rank sum test).

The differences in the locations of the uncertainty-enhanced and uncertainty-suppressed neurons' were significant along the medial-lateral axis (p < 0.05; Wilcoxon rank-sum test), anterior–posterior axis (p < 0.05; Wilcoxon rank-sum test), and depth (p < 0.05; Wilcoxon rank-sum test). Lastly, among the uncertainty-suppressed neurons, some signaled positive or negative reward value. We did not find a difference in their locations. This was consistent with previous observations that the primate ventral pallidum contained a mixture of positive and negative coding neurons (Tachibana and Hikosaka, 2012).

Beyond anatomical differences in their locations, we found statistical differences in electrophysiological properties of uncertainty-suppressed and enhanced neurons. First, on average, uncertainty-enhanced neurons displayed wider action potential spike duration than uncertainty-suppressed neurons (Fig. 2B; p < 0.01; Wilcoxon rank sum test). Second, consistent with previous observations, uncertainty-enhanced neurons often exhibited regular firing relative to the uncertainty-suppressed neurons (Fig. 2C; p < 0.01; Wilcoxon rank sum test). Third, uncertainty-enhanced neurons on average had lower baseline firing rates than uncertainty-suppressed neurons (Fig. 2D; p < 0.01; Wilcoxon rank sum test). These differences seemed to be related to the locations of the uncertainty-enhanced and uncertainty-suppressed neurons. This was found when we studied the firing characteristics of the uncertainty-insensitive neurons (n = 218). We defined the ventral pallidum region using previously reported criteria (Tachibana and Hikosaka, 2012) and compared electrophysiological properties of putative ventral pallidal neurons (n = 85) with other BF neurons (n = 133). The same patterns of results were observed: on average, VP neurons displayed narrower spike shapes, more irregular firing, and higher firing rates (spike shape: p < 0.01, firing irregularity; p < 0.05, firing rate; p < 0.01).

Together with the difference in the anatomical location of the uncertainty-enhanced and uncertainty-suppressed neurons, these data support the notion that the two types of uncertainty-sensitive-neurons belong to mostly distinct neuronal populations.

BF is known to exert control over reward-modulated behaviors. To understand how uncertainty processing by uncertainty-enhanced and uncertainty-suppressed neurons could contribute to this BF function; we next asked how they represent information about reward amount.

To this end, we assessed the relationship between the size of expected rewards in the reward amount block and neuronal activity (see Materials and Methods); 33 of 39 of uncertainty-enhanced neurons' CS responses in the reward amount block were positively correlated with the size of the predicted reward (Fig. 3A; we term them positive reward value-coding neurons). No uncertainty-excited neurons displayed significant negative correlations with reward amount, whereas 16 of 41 of uncertainty-suppressed neurons were positively correlated with reward amount and 18 of 41 of uncertainty-suppressed neurons were negatively correlated with reward amount (we term them negative reward value-coding neurons). Thus, this analysis revealed three groups of neurons: uncertainty-suppressed positive reward value-coding neurons, uncertainty-suppressed negative reward value-coding neurons, and uncertainty-enhanced positive value-coding neurons. The average activity of uncertainty-enhanced reward value neurons and uncertainty-suppressed reward value neurons (for positive value and negative value separately) is shown in Figure 3B–D. Reward size was encoded by all three groups of uncertainty-sensitive neurons in a graded manner. The neurons discriminated between no reward versus small reward (0.125 ml of juice) and small reward versus large reward (0.25 ml of juice) (p < 0.05, Wilcoxon rank-sum test).

Figure 3.

Representation of reward value and uncertainty among uncertainty-enhanced and uncertainty-suppressed BF neurons. A, Histograms of correlation coefficients for uncertainty-enhanced and uncertainty-suppressed neurons (left and right, respectively) assessing the relationship of neuronal activity and the size of predicted reward in the reward amount block. Gray represents significant correlations. Numbers of positive or negative value-coding neurons (e.g., those with significant negative or positive correlation coefficients) are indicated in italics. B–D, Average normalized responses of value coding uncertainty neurons for reward probability (red) and reward amount predictions (black). Value coding was defined by each neuron's significant correlation coefficient (negative or positive). B, CS responses of the 33 uncertainty-enhanced positive value-coding neurons. C, CS responses of the 16 uncertainty-suppressed positive value-coding neurons. D, CS responses of the 18 uncertainty-suppressed negative value-coding neurons. Insets, Reward-amount population CS responses subtracted from the reward-probability population CS responses of equal expected value *p < 0.05, significant difference between conditions (paired sign-rank test; time window: 100 ms following CS presentation until the outcome). E–G, Task dynamics of uncertainty-enhanced positive value-coding neurons (E), uncertainty-suppressed positive value-coding neurons (F), and uncertainty-suppressed negative value-coding neurons (G). Activity is separated by the 4 possible reward predictions.

However, the dynamics of reward value and reward uncertainty coding was different between uncertainty-enhanced and uncertainty-suppressed neurons (Fig. 3E–G). Uncertainty-enhanced neurons rapidly encoded reward value in a phasic manner after the presentation of the reward prediction (Fig. 3E), while uncertainty-suppressed neurons differentiated between big and small rewards continuously (in a approximately tonic manner) until the time of the trial outcome (compare green vs black traces in Fig. 3F,G; and see additional example neurons in Fig. 4).

Figure 4.

Task dynamics of two (A, B) uncertainty-suppressed positive value-coding neurons (top and bottom). Conventions are the same as in Figure 1. The bottom neuron's reward and uncertainty preference is also shown in Figure 1B, middle.

Another difference between uncertainty-enhanced and uncertainty-suppressed neurons was observed during the trial start epoch (Fig. 3E–G). On average, uncertainty-enhanced neurons were significantly excited during the trial start fixation epoch (p < 0.01; sign rank test; comparison was between baseline activity and activity during the TS-fixation epoch), whereas uncertainty-suppressed neurons were suppressed (p < 0.01). This last finding is consistent with observations of Avila and Lin (2014) who reported that a subset of putative ventral pallidal neurons were inhibited during trial start epochs in rodents performing reward-seeking actions.

We verified that uncertainty responses in the BF were not due to object and spatial selectivity. To this end, we derived spatial- and object-sensitivity indices for each uncertainty-sensitive neuron (Fig. 5); 0 of 39 uncertainty-enhanced neurons displayed object or spatial sensitivity. Among the uncertainty-suppressed neurons, 4 of 41 displayed object sensitivity, and 1 displayed spatial sensitivity. These results replicate our and others' observations (Tachibana and Hikosaka, 2012; Monosov et al., 2015) that most neurons in the medial BF and ventral pallidum do not carry spatial or object signals.

Figure 5.

Lack of spatial and object sensitivity in BF uncertainty-enhanced and uncertainty-suppressed neurons. Object (left) and spatial (right) sensitivity indices shown separately for uncertainty-enhanced (A) and for uncertainty-suppressed (B) BF neurons. Neurons with significant sensitivity indices (Wilcoxon rank sum test; p < 0.05) are shown in gray. The count of significant neurons (gray) is indicated next to the histograms.

Next, to further examine the functional differences between uncertainty-suppressed and uncertainty-enhanced neurons, we analyzed the time course of uncertainty selectivity in the BF while monkeys chose between certain and uncertain CS fractals (see Materials and Methods; Fig. 6A). Briefly, monkeys fixated a center spot for 500 ms. Then, the two CS stimuli appeared. The CSs were the visual fractals associated with certain and uncertain rewards used in the single stimulus trials (Fig. 1). Monkeys continued to fixate the center spot for 500 ms, until the fixation spot disappeared. To choose among the two reward options, monkeys made a saccade to one of the two fractal CSs and maintained fixation for at least 750 ms. Then the unchosen stimulus disappeared; and after 1 s, the outcome (associated with the chosen CS) was delivered. As observed in previous studies, the monkeys' choices were largely guided by the expected reward value when they chose between CSs associated with different reward sizes or between CSs associated with 100% versus 50% rewards of the same size (Fig. 6B) (Padoa-Schioppa and Cai, 2011; Monosov and Hikosaka, 2013; Lak et al., 2014; Monosov et al., 2015). Also, in agreement with previous findings (Platt and Huettel, 2008; O'Neill and Schultz, 2010; Monosov and Hikosaka, 2013; Monosov et al., 2015), uncertainty or risk preference was evident because the monkeys preferred a 50% chance of 0.25 ml of juice >0.125 ml of juice on >93% of trials, even though the expected value of these options is the same (Fig. 6B).

Figure 6.

Choice task, monkeys' choice preference, and single neuron examples. A, Choice trial structure (left) and the monkeys choice behavior (right). Monkeys made a choice between two CSs among the well-learned CSs (3 indicating reward amounts, and 3 indicating reward probabilities). B, Percentage of trials the monkeys chose 0, 0.125, 50% 0.25, and 0.25 CSs. Percentage of choice for every type of choice trial is shown in the inset matrix. Each number in the matrix indicates the choice percentage of the CSs on the x-axis versus the y-axis. Data are compiled from a dataset of 18,287 trials. C–F, Activity of example neurons (same neurons as in Fig. 1D shown in C, D; same neurons as in Fig. 4, shown in E, F) aligned on choice array presentation and choice.

To examine the time course of uncertainty selectivity during choice behavior, we sorted the neuronal activity of uncertainty-enhanced and uncertainty-suppressed reward value correlated cells by the monkeys' choices. The activity of four single neurons is shown in Figure 6C–F, and the population activity of uncertainty-enhanced and uncertainty-suppressed value-coding cells is shown in Figure 7A, B. This analysis revealed important differences in the dynamics of value and uncertainty signals in the populations of uncertainty-enhanced and uncertainty-suppressed BF neurons. First, before the go signal (as the monkeys continued fixating the fixation cue), uncertainty-enhanced neurons displayed highest firing for the 100% 0.25 ml option, lower for the 50% 0.25 ml option, and lowest for the 0.125 ml option (this rank ordering matches the early phase of the single CS responses observed in Fig. 3E and the monkeys' preferences). The uncertainty-suppressed neurons also displayed “certain” reward value coding before the go signal because they differentiated choices between certain rewards of different reward values (Fig. 7B, compare black with blue). However, these neurons also displayed selective uncertainty suppression before the go signal (Fig. 7B, red trace). Second, uncertainty-enhanced neurons selectively signaled reward uncertainty mostly after the unchosen stimulus disappeared as the monkeys expected the receipt of uncertain rewards (Fig. 7A, red trace). Third, activity differences between choices of 0.25 ml versus 0.125 ml juice reward options were evident before the outcome delivery in the uncertainty-suppressed neurons but not in the uncertainty-excited neurons (compare black with blue before the choice outcome).

Figure 7.

Uncertainty signals in uncertainty-enhanced and uncertainty-suppressed BF neurons during choice. A, B, Average normalized responses during the choice task for uncertainty-enhanced value coding (n = 33; A) and uncertainty-suppressed value coding (n = 34; B) neurons sorted by the monkeys' choices. Specifically, neuronal activity was sorted into trials in which the monkey chose the neurons' preferred reward value, (black), uncertain reward value (red), or nonpreferred reward value (blue). Preferred values were defined by the sign of the correlation coefficients in Figure 3A. Variability across the three trial types is tested by a Kruskal–Wallis test (p < 0.05) for each time window shown in gray near the x-axis. Next, for significant time windows, Wilcoxon rank sum tests were performed across different trial types. The colored asterisks indicate significant pairwise comparisons (p < 0.05). B, Insets, Choice array onset epoch activity separately for uncertainty-suppressed negative and positive value neurons. C, D, Percentage of neurons showing either selective uncertainty enhancement (green) or suppression (black) is shown for the populations of uncertainty-enhanced (C) and suppressed BF neurons (D) in time. To display the choice-related dynamics of uncertainty signals (C, D), we sorted the activity of each neuron into three trial types: (1) trials in which monkeys chose 0.25 ml of juice, (2) trials in which the monkeys chose 50% of 0.25 ml of juice, and (3) trials in which the monkeys chose 0.125 ml of juice. We then performed a running test of uncertainty sensitivity (see Materials and Methods). The percentage of neurons displaying selective uncertainty suppression or enhancement is then displayed in C, D. A thin black line indicates 5%.

To display the observed differences in the time course of uncertainty signals among populations of uncertainty-enhanced and uncertainty-suppressed neurons, we calculated the proportion of neurons showing selective uncertainty enhancement and suppression during choice. This analysis further supported the results in Figure 7A, B. Among the uncertainty-enhanced neurons, the population uncertainty signal increased steadily after choice, as the time of the uncertain outcome receipt neared (Fig. 7C). In contrast, among the uncertainty-suppressed neurons, the population uncertainty signal began before the choice behavior was completed and persisted until the uncertain outcome was delivered (Fig. 7D).

Among the uncertainty-sensitive BF neurons, only uncertainty-enhanced positive reward value neurons encoded the three chosen reward options before the choice was finalized (100% 0.25, 50% 0.25, and 0.125 ml of juice) in a manner that resembled subjective preference (100% 0.25 > 50% 0.25 > 0.125; compare behavior in Fig. 6B with neuronal activity in Fig. 7A). Therefore, we further asked whether these neurons were sensitive to the subjective values of both options in the choice array or were only driven by the value of the chosen option. We found that, before the overt choice, their firing rate was related to the subjective value of both choice array options (Fig. 8). Overall, the results in Figures 7 and 8 suggest that uncertainty-enhanced neurons may signal different task variables during the different epochs of the decision-action-outcome continuum.

Figure 8.

Sensitivity of uncertainty-enhanced positive value-coding neurons to the subjective value of the choice array options. Normalized neuronal responses during the choice epoch of uncertainty-enhanced positive reward value-coding neurons (n = 33; analyses done in the first time window in Fig. 7A). The results of Spearman's correlation across the choice array option combinations are shown in the top left corner. The results of Spearman's correlation across 3 choice-array combinations in which one of the options was the 0.25 ml of juice are shown in larger font on the right. *p < 0.01, variance across 3 choice-array combinations in which one of the options was the 0.25 ml of juice (Kruskal–Wallis test). Error bars indicate SE. In this figure, single neurons' responses to the 6 choice-array combinations (inset) were normalized to the maximum response (i.e., to the most preferred choice-array combination), from 0 to 1.

Discussion

The ability to control our actions and internal states relative to uncertainty about future rewards is critical for flexible and adaptive behaviors, such as for learning and exploration, and for hopeful or risky reward seeking. Our data suggest that different regions of the primate BF could support these functions in distinct manners. We found that uncertainty preferentially enhanced the activity of many positive value-coding neurons that were largely concentrated in the medial regions of the BF and that this enhancement increased as the time of the uncertain outcome neared. In contrast, uncertainty suppressed the activity of many value-coding neurons that were most often found in the dorsal-lateral BF, a brain region corresponding to VP. This suppression was observed in positive and negative value-coding neurons and occurred when monkeys made actions aimed at uncertain or risky rewards. Last, we found statistical differences in firing patterns and spike shapes of neurons in the medial BF and dorsal-lateral BF, an observation that may support the hypothesis that they constitute partly distinct anatomical systems (Spooren et al., 1996; Smith et al., 2009; Avila and Lin, 2014; Root et al., 2015).

VP is considered crucial for setting motivational gain of behaviors aimed at rewards (Smith et al., 2009; Haber and Knutson, 2010; Tachibana and Hikosaka, 2012; Berridge and Kringelbach, 2013). We hypothesize that the uncertainty-related modulation of some value-coding neurons in the dorsal-lateral BF region corresponding to VP could mediate risky or hopeful behavioral states. One previous study found, that when VP is inactivated, the speed of the monkeys' actions increased and the effect of reward value associations on behavior decreased (Tachibana and Hikosaka, 2012). Therefore, the suppression of value-coding neurons in VP may mediate the rate of risky or exploratory behavior. In support of this notion, we found that, during risky choices, uncertainty-related suppression emerged before the overt choice behavior and continued to increase until the choice was finalized (Fig. 7D; when animal could not change its mind after the unchosen stimulus disappeared). Because VP projects to midbrain dopaminergic neurons (Smith et al., 2009; Haber and Knutson, 2010; Humphries and Prescott, 2010; Watabe-Uchida et al., 2012), one attractive possibility is that VP uncertainty-related suppression mediates dopamine responses and decreases their reward value selectivity. This mechanism could then promote actions aiming at new or uncertain contexts. To test this hypothesis, future studies will need to inactivate VP and record neuronal activity from dopaminergic neurons during safe and risky choice tasks.

Learning and memory are enhanced by surprising events and uncertain contexts (Pearce and Hall, 1980; Yu and Dayan, 2005; Courville et al., 2006; Esber and Haselgrove, 2011; Le Pelley et al., 2011; Bach and Dolan, 2012) and are thought to be in part dependent on the BF (Damasio et al., 1985; Everitt et al., 1988; Morris et al., 1992; Voytko, 1996; Everitt and Robbins, 1997; Semba, 2000; Baxter and Bucci, 2013). We observed that uncertainty-enhanced neurons displayed a ramp-like increase in activity after risky choices that terminated when the outcome was delivered. This suggests that the medial BF uncertainty-enhanced neurons may contribute to outcome monitoring and learning functions commonly associated with the medial BF (Damasio et al., 1985; Everitt et al., 1988; Morris et al., 1992; Voytko, 1996; Everitt and Robbins, 1997; Semba, 2000; Baxter and Bucci, 2013; Monosov et al., 2015). Anatomical, clinical, and imaging studies support this conjecture. First, medial BF neurons project strongly to cingulate cortex and the hippocampal formation (Mesulam et al., 1983), regions involved in learning and memory (Suzuki and Clayton, 2000; Suzuki and Eichenbaum, 2000; Suzuki, 2007; Bryden et al., 2011; Hayden et al., 2011). And in both the cingulate and the hippocampal formation, reward uncertainty (or risk) modulates neural activity during reward-seeking behaviors (Vanni-Mercier et al., 2009; Burke and Tobler, 2011a; Kolling et al., 2012). Second, lesions of the medial BF disrupt learning from outcomes and episodic memory acquisition (Damasio et al., 1985; Everitt et al., 1988; Morris et al., 1992; Voytko, 1996; Everitt and Robbins, 1997; Semba, 2000; Baxter and Bucci, 2013).

The possibility that uncertainty-enhanced and uncertainty-suppressed neurons play overlapping but different roles in behavior under uncertainty is further supported by the many clear differences in how the two neuronal populations signaled information about the value of rewards. Reward uncertainty suppressed the activity of many VP neurons that increased (i.e., positive reward value-coding neurons) or decreased (i.e., negative reward value-coding neurons) their activity in relation to increases in expected rewards (Fig. 3A). As observed in previous studies (Smith et al., 2011; Tachibana and Hikosaka, 2012), these reward value signals persisted until the trial's outcome (Figs. 3F,G, 7B). The tonic reward value signals observed in VP reward-sensitive neurons may aim and motivate actions toward known or certain rewards by persistently encoding their expected value until the action is complete (Fig. 7B) and the outcome is delivered.

In contrast, with the tonic reward value coding by uncertainty-suppressed neurons in the dorsal-lateral BF, uncertainty-enhanced neurons' reward value signal was relatively phasic and discriminated between big and small reward trials mostly during the first ∼500 ms of the trial (Figs. 3E, 7A). Also, a previous study showed that uncertainty-enhanced medial BF neurons displayed anticipatory ramping that preceded the delivery of punishments (Monosov et al., 2015). Most of these same neurons also responded to the delivery of punishments with rapid phasic activations. Lastly, here we found that uncertainty-enhanced neurons displayed unstable selectivity for different reward options during the choice task (Fig. 7A). Before the overt choice was made, the neurons rank ordered the reward options in the order of the monkeys' preference (100% big reward >50% big reward > small reward; Fig. 7A, left) and were sensitive to the subjective value of both options in the choice array (Fig. 8). However, after the monkeys chose the uncertain option, the neurons selectively signaled reward uncertainty (50% big reward >100% big reward; 50% big reward > small reward), and the discrimination of small versus big reward options mostly disappeared (Fig. 7A, right; p > 0.05). In sum, these data suggest that the uncertainty-enhanced neurons are not exclusively dedicated to signaling selective information about reward uncertainty or reward value. Instead, their selectivity seems to shift depending on the task at hand (Fig. 7A).

We previously proposed that a source of uncertainty signals in the medial BF is the anterodorsal septum (ADS) (Monosov and Hikosaka, 2013; Monosov et al., 2015), where some neurons are specifically dedicated to signaling information about reward uncertainty (Monosov and Hikosaka, 2013). Because ADS is particularly rich in GABAergic inhibitory neurons (Sheehan et al., 2004), one might hypothesize that uncertainty suppression in VP plays an intermediate role in the ADS-medial BF circuit (e.g., ADS would inhibit VP, which would then release medial BF from VP-mediated inhibition). However, the present study makes this possibility unlikely. First, on average, uncertainty suppression signals emerged as the time of the choice neared, whereas uncertainty enhancements in the medial BF emerged much later, after the unchosen stimulus turned off and the monkeys awaited the uncertain outcome of their choices (Fig. 7). Second, the selective targeting of uncertainty to positive reward value coding cells in the medial BF is unlikely to be facilitated by uncertainty-suppressed neurons in VP that included both negative and positive reward value coding cells (Fig. 3A). Third, available anatomical data indicate that the ADS and the medial BF do not project or receive inputs from VP, which receives inhibitory projections most prominently from the striatum (Spooren et al., 1996; Smith et al., 2009; Root et al., 2015). These observations suggest that the BF contains multiple distinct mechanisms for signaling uncertainty and reward value: one in VP and one in the septal-medial BF complex.

One interesting possibility is that the different integration of value and uncertainty in BF subregions is supported by monoamine transmission (and other inputs) from the brainstem and hypothalamus (Fiorillo et al., 2003; Watabe-Uchida et al., 2012; Ogawa et al., 2014; Cohen et al., 2015), but more anatomical studies are required to support or exclude this hypothesis.

To survive in an uncertain context, humans and other animals sometimes approach and other times avoid uncertainty, and learn from the receipt of the uncertain outcomes. These processes require that the brain integrates information about uncertainty and value. Our data show that the BF contains at least two reward uncertainty-coding strategies that could play important roles in uncertainty-related behavioral control: one mostly in the dorsal-lateral BF, occupied by VP, to mediate and adjust our actions aiming at certain and uncertain rewards; and another in the medial BF to help us anticipate, learn, and monitor reward-uncertain outcomes. How these BF subregions interact with other brain areas that are modulated by reward uncertainty (Hsu et al., 2005; McCoy and Platt, 2005; Preuschoff et al., 2006, 2008; Preuschoff and Bossaerts, 2007; O'Neill and Schultz, 2010; Burke and Tobler, 2011a; Schultz et al., 2011; Monosov and Hikosaka, 2012, 2013; Monosov et al., 2015) will be the subject of our next experiments.