Selective and graded coding of reward-uncertainty by neurons in the primate anterodorsal septal region

Abstract

Natural environments are uncertain. Uncertainty of emotional outcomes can induce anxiety and raise vigilance, promote and signal the opportunity for learning, modulate economic choice, and regulate risk seeking. Here we demonstrate that a subset of neurons in the anterodorsal region of the primate septum (ADS) are primarily devoted to processing uncertainty in a highly specific manner. Those neurons were selectively activated by visual cues indicating probabilistic delivery of reward (e.g. 25%, 50%, 75% reward) and did not respond to cues indicating certain outcomes (0% and 100% reward). The average ADS uncertainty response was graded with the magnitude of reward uncertainty, and selectively signaled uncertainty about rewards rather than punishments. The selective and graded information about reward uncertainty encoded by many neurons in the ADS may underlie uncertainty-modulation of value- and sensorimotor- related areas to regulate goal-directed behavior.

INTRODUCTION

Emotionally salient outcomes, such as rewards and punishments, are often difficult to predict. It has been theorized that the brain tracks the level of uncertainty associated with particular cues or actions. This theory is consistent with the observations that uncertainty of emotional outcomes can induce anxiety and raise vigilance ^1,2, promote and signal the opportunity for learning ^2–8, modulate economic choice ^2,3,9–12, and regulate risk seeking ⁴. Therefore, knowing the neuronal mechanisms of uncertainty-related behavioral modulations is critical for understanding how we make value-based decisions, as well as for understanding how uncertainty can induce aberrant behavior in human subjects.

Indeed, data from monkey neurophysiology ^13–16 and human fMRI ^17–19, in concert, suggest that the brain monitors the level of outcome uncertainty. Single neuron studies demonstrated correlations of neuronal activity with uncertainty in brain areas, including orbitofrontal cortex (OFC) ¹⁶, cingulate cortex (CG) ¹⁵, and midbrain ¹⁴. But in those structures reward-uncertainty signals were often found in subsets of neurons which also signaled other task-related information, such as prediction error, stimulus location, saliency, and/or fully predictable stimulus values ^14–16,20. In some of these reports, the uncertainty-related neural activity appeared to reflect the animal’s behavior preferring risky cues ^15,16. In addition, whether those neurons also signal uncertainty about aversive events remains untested.

Consistent with the integration of uncertainty signals with other motivation-related signals suggested by the single neuron studies, fMRI studies in humans revealed BOLD correlations with uncertainty in brain regions strongly implicated in motivation and emotion, such as the CG ^3,18, OFC ²¹, insula ¹⁷, and in the thalamus and basal ganglia ¹⁹. Although uncertainty-related modulations have been widely demonstrated, it is unclear whether there is a brain region in which many neurons are devoted to specifically signaling information about reward uncertainty to other brain regions.

Previous lesion and anatomical studies in rodents have linked the septal nuclei with the processing of motivational-related signals ^22–24 and in regulating anxiety and a variety of emotional responses ^23,25,26. In a series of primate studies, Ono and colleagues showed that some neurons in the septum respond to reward predicting cues as well as to other task related parameters, such as object and spatial information ^27,28. But to date, no study has examined how septal neurons process probabilistic information about rewards or punishments.

We discovered that a subset of neurons clustered in the anterior dorsal part of the primate septum (ADS) selectively signaled graded information about reward uncertainty. This was shown in a series of 4 neurophysiological experiments in 5 monkeys. Experiment 1 showed that many neurons in the ADS signaled reward, not punishment, uncertainty. Experiment 2 showed that the reward uncertainty-selective response varied with reward probability, but was on average insensitive to equal but fully predictable reward values. Experiment 3 showed that the uncertainty-selective response was modulated by the expected reward value. Lastly, Experiment 4 showed that the uncertainty-selective response developed quickly as the subject learned cue-outcome relationships. Our study thus demonstrates the role of the ADS in processing and signaling of information selective for reward uncertainty.

RESULTS

A cluster of neurons in the anterodorsal septal (ADS) region transmits uncertainty about rewards but not punishments

In Experiment 1, we studied single neurons over a wide area of the anterior ventral midline (Fig. 2) while three monkeys (B, S, and T) participated in a Pavlovian procedure (Fig. 1A) that contained two distinct contexts ²⁹: an appetitive block in which three visual conditioned stimuli (CSs) predicted juice with 100, 50, and 0% probabilities, and an aversive block in which three cues predicted air-puffs with 100, 50, and 0% probabilities. During the recordings the monkeys understood the meanings of the CSs (Fig. S1). We recently reported that many neurons in ventromedial prefrontal cortex (vmPFC) in the anterior ventral midline were sensitive to the prediction and reception rewards and punishments ¹³.

Figure 2. Locations of neurons tested for uncertainty coding.

Top: Estimated locations of 458 neurons (monkeys B, P, S, and T) plotted on a parasagittal MR image from Monkey B. Red dots: reward uncertainty neurons (n=22 from Experiment 1 and n=34 from Experiment 2); no punishment uncertainty neurons were found; black dots: neurons that were not selective to outcome uncertainty (n=297; Experiment 1); small light blue dots: encountered neurons that were not judged to be sensitive to outcome uncertainty without full examination (n=105). VMPFC: ventromedial prefrontal cortex; cc: corpus callosum; ac: anterior commissure. Bottom: Estimated locations of reward uncertainty neurons in the ADS plotted on two coronal MR images from Monkey B. Their locations are indicated on the parasagittal image at top: 5 mm anterior (slice 1) and 2 mm anterior (slice 2) to the center of the ac. Slice 1 includes neurons recorded in a ~3mm area (6mm from ac to 4mm from ac). Slice 2 includes neurons recorded in a ~3mm area (3mm from ac to 1mm from the ac). CG: cingulate cortex; CD: caudate nucleus, PUT: putamen.

Figure 1. Responses of reward uncertainty neurons in the anterodorsal septal region (ADS) to certain and uncertain predictions of rewards and punishment (Experiment 1).

(A) Monkeys (B, S, and T) experienced 2 distinct blocks: an appetitive block in which three visual conditioned stimuli (CSs) predicted juice with 100, 50, and 0% probabilities, and an aversive block in which three cues predicted air-puffs with 100, 50, and 0% probabilities. Juice and airpuffs were used as the US in the appetitive (left) and aversive (right) blocks, respectively. ITI: inter-trial-interval, TS: trial start. (B) Rasters showing the activity of a single neuron in two blocks, for each CS separately. Gray rasters indicate the activity in 50% CS trials in which US was omitted. Spike density functions for the same neuron are shown below the rasters. (C) Average activity of uncertainty selective neurons in the ADS (red dots in Figure S4) in the appetitive (left) and aversive (right) block recorded in monkeys B, S, and T.

Adjacent to the vmPFC, we found a group of neurons (n=21; neuron classification detailed in Methods) that responded in a completely different manner: selectively encoding reward-uncertainty. An example is shown in Fig. 1B. In the appetitive block the neuron was strongly excited by the CS indicating that a reward would be delivered with 50% probability (uncertain outcome), but did not respond to the 0% or 100% reward CSs (certain outcomes). This excitation was selective for uncertainty about rewards: in the aversive block the neuron showed no response to any of the CSs, regardless of uncertainty.

All of the reward uncertainty neurons showed similar responses (Fig. 1C). They were strongly activated by the uncertainty cue in the appetitive block (50% reward CS activity) (p<0.0001 when compared with activity during ITI or TS; paired signed rank test), but were weakly inhibited by the reward certainty cues (0% and 100% reward CSs) (Fig. 1C left). The response to the reward uncertainty cue started abruptly after CS onset (population latency of uncertainty selectivity: 199ms), continued throughout the CS period, and terminated soon after the outcome (US) which was either a reward or no reward. In contrast, these neurons showed no response to any CS in the aversive block (p>0.05; paired signed rank test) (Fig. 1C right). A similar pattern of neuronal activity was observed when we recorded from uncertainty selective neurons using the same procedure but with different stimuli (Monkey Sm; n=7; Fig. S2).

Some of the reward uncertainty neurons (7/21; p<0.05; ranksum test) were also activated by the trial start cue (TS) in the appetitive block. On average, activity during TS was higher than activity during ITI in the appetitive block (p<0.05; paired signed rank test) (Fig. 1C), but less so in the aversive block (Fig. 1D; p=0.05). The average activity during ITI was higher in the appetitive block than aversive block (p<0.05; paired signed rank test).

Notably, these reward uncertainty neurons were clustered in a small area (red dots in Fig. 2). Based on MRI (Fig. 2) and histological examination (Fig. S3), we judged that this area was located in the anterodorsal region of the septum (ADS). To examine how such reward uncertainty neurons were distributed, we recorded from neurons widely in the anterior midline area, including vmPFC (mostly in areas 14 and 25) and anterior dorsal regions of the septum (Fig. 2; n=319). Among them, only 1 neuron outside of the ADS showed CS responses selective for uncertainty. These results indicate that within our recorded population the reward uncertainty neurons were localized within a small area of the anterodorsal septal region. Within the ADS (or elsewhere in the population; Fig. 2), we found no neurons that signaled punishment uncertainty.

The results of Experiment 1, however, do not indicate that the ADS contains only reward uncertainty neurons. To address this issue, we first defined the ADS by the spatial distribution of reward uncertainty neurons (Fig. S4). Within the thus defined region, 85 neurons were recorded non-selectively in Experiment 1. Among them, 38 neurons showed differential responses to the 3 reward CSs (Fig. S4A). Among the 38 reward CS-sensitive neurons, the reward uncertainty neurons composed a prominent group (n=21/38). Characteristics of the other CS-sensitive neurons (n=17/38) are described in Fig. S4B.

ADS reward-uncertainty neurons transmit graded levels of uncertainty

Experiment 1 raised two questions. First, did the ADS reward uncertainty neurons truly encode reward uncertainty, or did they respond to the 50% CS simply because it was intermediate in value between the 0% and 100% CSs? Second, did reward uncertainty neurons merely encode the presence of reward uncertainty, or were they sensitive to the level of uncertainty in a graded manner?

To answer these questions we performed Experiment 2 by selectively recording from reward uncertainty neurons in the ADS. We used a Pavlovian-procedure that contained two distinct contexts; a reward-probability block (Fig. 3A; top) and a reward-amount block (Fig. 3A; bottom). In the reward-probability block, five fractal CSs indicated five different probabilities of 0.4ml of water. In the reward-amount block, five other fractal CSs indicated five different amounts of water. Behavioral measures showed that the monkeys understood the meanings of the CSs (Fig. S5–6). Importantly, the expected values of the CSs were exactly matched between the two blocks, but reward uncertainty was only present for three CSs in the reward-probability block: 25, 50, and 75%. Among them the magnitude of uncertainty was highest for the 50% CS, while 25% and 75% CSs had the same degree of uncertainty ^4,12,30.

Figure 3. Responses of reward uncertainty neurons to information about reward uncertainty and reward amount (Experiment 2).

(A) Activity of a single neuron across two distinct blocks: a reward-probability block (top) and a reward-amount block (bottom). In the reward-probability block, five fractal CSs indicated five different probabilities of 0.4ml of water, and in the reward-amount block, five other fractal CSs indicated five different amounts of water. Expected values of the CSs in the two blocks were matched. The actual fractals used are shown. TSa: Time when the monkey started fixating the trial start cue after making a saccade to it (trial start cue acquisition time). Otherwise, the labeling conventions are the same as in Fig. 1. (B) Average activity of the population of 34 reward-uncertainty neurons (18 in Monkey B and 16 in Monkey P). Reward probability-block (shown in black) and the reward amount-block (gray). Asterisks indicate significant difference between the spiking activity (not normalized) in the two blocks (paired signed rank test; p<0.05; gray: greater activity in the amount block; black: greater activity in the probability block). (C) Average normalized CSs responses of the same neurons for probability (black) and amount (gray) CSs. Black asterisks indicate significance between CSs. (p<0.05; paired signrank test). (D) CS responses shown individually for the 34 neurons in the probability block (normalized to the maximum CS response; from 0 to 1). Most neurons (65%) had the highest response for the 50% CS. All error bars present standard error (SEM).

We used this task to record from 34 reward uncertainty neurons in the ADS (18 in Monkey B and 16 in Monkey P). Reward uncertainty neurons were identified during online screening as neurons that responded to the uncertain CSs (25, 50, or 75%). An example neuron is shown in Fig. 3A. The neuron encoded reward uncertainty in a graded manner: it responded most strongly to the 50% CS, less strongly to the 25% and 75% CSs, and showed little response to the certain 0% and 100% CSs. Furthermore, it showed no response to any of the amount CSs, even those that had equivalent values to the uncertain probability CSs.

The average responses of the reward uncertainty neurons (Fig. 3B) were very similar to those of the example neuron. In the reward probability block, they were strongly activated by the reward uncertain CSs (25, 50, and 75%), and were weakly phasically suppressed by the reward certain CSs (0, 100%) (compared with the responses of CSs of equal value in the reward amount block; Fig. 3B, gray asterisks; p<0.05; paired signed rank test). The responses to reward uncertain CSs were not affected by the spatial location of the CS (Fig. S7). In sharp contrast, the average activity of reward uncertainty neurons showed no clear systematic response to any of the reward amount CSs (Fig. 3B). These results indicate that these neurons indeed encode reward uncertainty, not intermediate rewards values.

The average response for 50% reward CS was higher than 25 and 75% CSs (Fig. 3C). Single neurons showed the same tendency (Fig. 3D). Different theoretical models define uncertainty based on probability in slightly different ways ^3,4,12,30, but they all agree that the level of uncertainty is higher for 50% than 25% or 75%. As shown in Fig. S8 the responses of ADS neurons changed in a graded manner as the two theoretical models predict. These results suggest that the reward uncertainty neurons in the ADS encode the quantitative level of reward uncertainty.

ADS reward-uncertainty neurons are modulated by uncertain reward sizes

In the reward probability block of Experiment 1 and 2, the amount of reward was fixed. It is thus possible that the reward uncertainty response is influenced by the amount of the expected reward. To test this possibility, we performed Experiment 3 by varying both reward uncertainty and amount (Fig. 4). We used a single block task with 5 CSs: 1) CS predicting no reward (0%), 2) CS predicting a small reward with 100% probability (small-100%), 3) CS predicting large-100%, 4) CS predicting small-50%, 5) CS predicting large-50%. We used this task to record from 12 reward uncertainty neurons in the ADS in Monkeys Sm (n=7) and P (n=5). Reward uncertainty neurons were identified using the appetitive/aversive Pavlovian procedure (Fig. S2).

Figure 4. Responses of reward uncertainty neurons to uncertainty about different valued outcomes (Experiment 3).

(A) The activity of a single reward-uncertainty selective neuron (same conventions as in Fig. 3) recorded while monkey Sm experienced 5 reward predicting fractal cues: 100% of 0.5ml, 100% of 0.25ml, 50% of 0.5ml, 50% of 0.25ml, and 0ml of juice. (B) The average CS responses (normalized to baseline) of 12 neurons (7 in Monkey Sm and 5 in Monkey P). Asterisks indicate significant differences (p<0.05; paired signed rank test). Error bars represent standard error (SEM). (C) Uncertain CS responses shown individually for 12 neurons in the probability block (normalized to baseline). Black lines indicate sessions with significant difference (p<0.05; ranksum test).

An example ADS neuron showed relatively strong excitatory responses to the two uncertain CSs (small-50%, large-50%), but not to the three certain CSs (0%, small-50%, large-50%) (Fig. 4A). On the average (n=12), the response to large-50% CS was significantly stronger than to small-50% CS (p<0.05; paired signed rank test) (Fig. 4B). Among the 12 reward uncertainty neurons, 5 neurons responded significantly more strongly to large-50% CS than to small-50% CS (Fig. 4C; p<0.05; ranksum test). These results suggest that the response to the reward uncertain CS is enhanced by increasing the amount of the expected reward.

ADS reward-uncertainty neurons quickly acquire representation of reward uncertainty

Behavioral experiments have shown that humans and animals are exquisitely sensitive to reward uncertainty, rapidly estimating the uncertainty of each CS after a small number of exposures and using this knowledge to adjust preference between CSs and the speed of learning ^{4–8,12,31,32}. However, our experiments so far left it unclear whether reward uncertainty neurons could support these functions, since the monkeys had already learned the meaning of the CSs in Experiments 1–3 through extensive training. How rapidly do reward uncertainty neurons acquire their estimates of reward uncertainty? Experiment 4 was designed to answer this question. We devised a Pavlovian-procedure in which three novel fractals were used as CSs associated with 100, 50, and 0% reward probabilities. For this experiment, reward uncertainty neurons were recorded in Monkeys B (n=9) and P (n=6). Reward uncertainty neurons were identified using the amount/probability procedure for Monkey B (Fig. 3) and the appetitive/aversive procedure for Monkey P (Fig. S2).

Fig. 5A shows activity of one reward uncertainty neuron during the learning. Initially the neuron showed a weak increase in activity during all CS periods (rasters near bottom of Fig. 4A). As the monkey repeatedly experienced the CSs, the neuron’s activity started increasing for the uncertainty CS (50%) and decreasing for the certainty CS (0, 100%). Similar changes were commonly observed in the population of reward uncertainty neurons (Figs. 5B–C), while the monkey learned the value of each CS (Fig. 5E–F; also see Fig. S9). Before the monkeys’ learning, the neurons increased their activity gradually and non-selectively after CS onset (p<0.05; paired signed rank test; Fig. 5B–C). After the learning, the neurons responded abruptly and selectively to the onset of the uncertainty CS (50%). In contrast, their activity in the presence of the certainty CSs (0, 100%) became lower after learning (Fig. 5C). As a population, the selective response to the reward-uncertain CS emerged rapidly (Fig. 5D). This result indicates that reward uncertainty neurons rapidly acquire representation of reward uncertainty and confirms that they are indifferent to the expected value of the CSs.

Figure 5. Responses of reward uncertainty neurons during learning of novel conditioned stimuli (Experiment 4).

(A) Rasters show the activity of a single reward uncertainty neuron in Pavlovian-procedure in which three novel fractals were used as CSs associated with 100, 50, and 0% reward probabilities. Presentation number refers to the order in which the monkeys experienced each of the 3 CSs. All figure labels are the same as in Fig. 3. (B) Average CS-responses of the population of 15 reward uncertainty neurons before learning (gray; the first 5 trials in which the CS was experienced) and after learning (black; last five trials in which CS was experienced). Asterisks indicate significant difference between the activity in the before learning and after learning conditions (paired signed rank test; p<0.05; gray: greater activity before learning; black: greater activity after learning). (C) Normalized CS responses (normalization relative to ITI baseline activity) before learning (left; for all CSs) and after learning (right; for 3 CSs separately). (D) Neuronal learning expressed as changes in differential responses between the uncertain (50%) CS and the certain CSs (100%: green; 0%: black). The differential response was measured by an ROC area. Data were based on 9 neurons (6 in Monkey B and 3 in P) for which more than 25 trials were tested for all CSs. (E) Magnitudes of anticipatory licking before learning (left) and after learning (right). The magnitudes of licking after learning were correlated to the values of the CSs (15 sessions; rho=0.32; p<0.0001; permutation test; SEM and correlations calculated from single trial z-scores). (F) Latencies of eye movements before learning (left) and after learning (right). Data was based on monkey B (n=9), because for that monkey on a third of the trials the CSs appeared 10 degrees to the left or the right of the fixation point (Methods). The latency of the eye movement to the CS was measured from the CS onset to the time when the monkey’s gaze was fixated on the CS (target acquisition time). The latencies after learning were inversely correlated to the values of the CSs (9 sessions; rho = −0.2; p < 0.05; permutation test; SEM and correlations calculated from single trial latencies). All error bars represent standard error (SEM).

DISCUSSION

In the primate septum we found neurons that selectively encoded uncertainty about rewarding outcomes. They were clustered in the anterodorsal part of the septum (ADS). Their responses were triggered by the appearance of visual cues (CSs) that predicted uncertain reward outcomes (USs). The magnitudes of the cue responses were correlated with the degrees of uncertainty. The cue responses persisted until the uncertainty about the outcome was resolved (reward or no reward).

However, the reward uncertainty neurons in the ADS do not encode uncertainty wholly or purely. First, they encoded uncertainty about rewarding outcomes, not aversive outcomes (Experiment 1). Uncertainty about aversive outcomes may be encoded by neurons in different brain areas. Second, although on average reward uncertainty neurons were not systematically modulated by changing certain reward amounts (Experiment 2), their responses to uncertain cues were clearly enhanced when the amount of the expected reward was larger (Experiment 3). This may reflect the elevated risk due to the increase in the variance of the reward outcome ³³ or the increase in the expected reward size. Through these modulations, ADS may provide a mechanism through which the level of uncertainty could gate reward size information. Third, the uncertainty encoded by ADS neurons was based on the monkey’s knowledge about the CS-US relationships. This was shown in Experiment 4: As the monkey acquired knowledge about the CS-US relationships, the responses of the ADS neurons to the uncertain cue grew larger (Experiment 4). Such knowledge-based uncertainty is often called ‘risk’ ^9,12,21. Another type of uncertainty, which is based on the lack of knowledge about the CS-US relationships (ambiguity), may be encoded by neurons in different brain areas ²¹.

Importantly, the uncertainty-selective response of ADS neurons was dissociated from the monkey’s choice behavior. When two well-learned CSs were presented, the monkeys almost always chose a more valuable CS by taking into account both reward amount and probability (Fig. S5–6). This observation is distinct from previous reports showing that neural activity displayed a preference for reward-uncertain cues in subjects that behaviorally also showed clear preference for reward-uncertain cues ^15,16 (and also see ²⁰ for related discussion). Our results may suggest that the reward uncertainty neurons in the ADS do not directly influence the animal’s choice behavior. Instead, the ADS may send the reward uncertainty-selective information to other brain areas so that behavior can be modified appropriately under uncertain conditions. This scheme and its implications are further discussed below.

It is known that information about reward-uncertainty is utilized to drive learning ^2,5–8, which may reduce the uncertainty^7,11,22. It has been hypothesized that this process is done by increasing the saliency of sensory stimuli associated with reward-uncertainty ^7,31 or by increasing or decreasing the stochasticity of actions to promote exploratory and/or self-initiated behaviors ³⁴. In addition, it is well known that reward uncertainty can suppress or drive risk seeking in a context dependent manner ⁴. To facilitate such flexible utilization of information about uncertainty, it would be ideal if the brain is equipped with a system that selectively transmits reward uncertainty information to other brain regions ^2,8, where it can be utilized in a flexible manner. Our data suggest that the ADS could serve this function. Reward uncertainty neurons there may transmit the reward uncertainty-selective information to other brain areas where the uncertainty information would be integrated with sensorimotor and cognitive information. Depending on how the uncertainty information is integrated, learning or risk-seeking behavior may be facilitated or inhibited.

Indeed, the integration of reward-uncertainty signals by multiple neural systems is supported by studies of single neurons in several other brain regions and fMRI in humans. Various forms of uncertainty-related sensitivity have been reported in subsets of single neurons in OFC ¹⁶, posterior cingulate cortex (CGp) ¹⁵, vmPFC ¹³, and putative midbrain dopamine neurons (DA) ¹⁴, mixed with other types of motivation related signals (such as value related signals; discussed below). fMRI studies have implicated the insula ¹⁷, OFC ²¹, and the anterior cingulate ^11,18 in processing uncertainty, but most often simultaneously with other signals (such as prediction errors). The ADS neurons may transmit reward uncertainty information to these structures indirectly through their connections with the hypothalamus ^22,35, thalamic nuclei ^35,36 (which project to OFC, vmPFC, and CGp ^37,38) or through other septal regions ^39,40.

How the reward-uncertainty signals from ADS are integrated by these brain areas may be varied. For example, the CGp contains neurons that are sensitive to spatial positions, as well as various action- and reward-related parameters^11,28. By modulating spatial signals within CGp, the reward uncertainty signals may bias behavior towards or away uncertain stimuli ¹⁵ in a spatially-specific manner ⁴¹. The OFC is thought to contribute to learning and decision-making based on stimulus-value association ^42,43. In a task in which subjects preferred uncertain cues, OFC neurons encoded reward uncertainty (or salience) ²⁰, but did so most often as modulations of other task-related signals ^16,44. The uncertainty-related signals in the OFC therefore might underlie the subjects’ preference for uncertain cues as well as promote stimulus-value learning. The vmPFC is thought to be related to the regulation of mood ⁴⁵ (also see discussion section in ref. 13). Reward-preferring neurons are found clustered in the ventral part of the vmPFC¹³. Many of them signal reward values in a highly non-linear manner, treating 100% and 50% reward probabilities equally. Such signals could be generated by integrating value-selective signals with uncertainty-selective signals. Clearly, further anatomical studies of the septum in primates are needed to verify or exclude the role of ADS reward uncertainty neurons in those schemes.

Uncertainty about appetitive or aversive outcomes often induces emotional reactions, such as anxiety ^1,2,46–48. Neurons controlling emotional states may also modulate autonomic and endocrine functions. Therefore, it is plausible that reward uncertainty-selectivity neurons in the ADS have strong access to the autonomic and endocrine systems. The efferent connections of the ADS to the hypothalamus ^22,35,49 or other septal regions ^39,40 may serve this hypothetical function. The uncertainty-selective information would be useful in adjusting emotional states in a selective manner. How this might be done remains for future studies.

The world is complex and uncertain. It is uncertain because we lack knowledge about future or because events in the world occur randomly (even though we have knowledge) ². To adapt to the uncertain world, the brain needs to encode and evaluate the uncertainty, either to actively explore the uncertain events or to avoid them. Our discovery of a localized brain area where many neurons selectively signal the knowledge-based uncertainty about future rewards opens many opportunities to investigate the neuronal mechanisms underlying flexible modulation of behavior by uncertainty.

Methods

Five adult male rhesus monkeys (Macaca mulatta) were used for the experiments. All procedures for animal care and experimentation were approved by the Animal Care and Use Committee of the National Eye Institute and complied with the Public Health Service Policy on the humane care and use of laboratory animals. A plastic head holder and plastic recording chamber were fixed to the skull under general anesthesia and sterile surgical conditions. The chambers were tilted laterally by 35°, and aimed at the ventromedial prefrontal cortex and the anterior portion of the caudate nucleus. Two search coils were surgically placed under the conjunctiva of the eyes. After the monkeys recovered from surgery, they were conditioned using Pavlovian procedures (Figs. 1, 3, 4, 5). During the Pavlovian procedures, we recorded the activity of single neurons over a wide area of the anterior-ventral midline (Fig. 2).

Details of the recording procedures, as well as the histological and MRI-based confirmation methods of recording locations, were previously described¹³. Here, MRI-based estimation of neuron recording locations was aided by custom built software ⁵⁰.

Tasks

Experiment 1: Appetitive/aversive Pavlovian procedure (Fig. 1)

The Pavlovian procedure used in Experiment 1 (Fig. 1) consisted of two blocks: reward-probability and punishment-probability block. Monkeys B, S, and T participated in this procedure. In the reward-probability block, three conditioned stimuli (CSs) were followed by an unconditioned stimulus (US, apple juice), with 100%, 50% and 0% probability, respectively. In the punishment-probability block, three CSs were followed by an air puff US directed at the monkey’s face with 100%, 50% and 0% probability, respectively. Apple juice was delivered through a spout that was positioned in front of the monkey’s mouth. Air-puff (~35 p.s.i) was delivered through a narrow tube placed 6–8 cm from the face. This task was originally designed to test whether single neurons encode the values of stimuli associated with probabilistic delivery of reward or punishment ²⁹. Each trial started with the presentation of a trial start cue (TS). After 1 s, the TS disappeared and one of the three CSs was presented pseudo-randomly. After 1.5 s, the CS disappeared, and the US (if scheduled for that trial) was delivered. The monkeys were not required to fixate the TS or CS. On a subset of sessions uncued trials were included in which a juice reward alone (free reward) was delivered during the appetitive block and an air puff alone (free air puff) was delivered during the aversive block. All trials were presented with a random inter-trial interval that averaged ~7.5 s (5–10 s). One block consisted of 22 trials with fixed proportions of trial types (100%, ~7 trials; 50%, ~7 trials; 0%, ~7 trials; or during sessions in which we included free US the proportions were: 100%, 6 trials; 50%, 6 trials; 0%, 6 trials, and 4 uncued trials). The block changed without any external cue.

Experiment 2: Reward-probability/reward-amount Pavlovian procedure (Fig. 3)

The Pavlovian procedure used in Experiment 2 (Fig. 3) consisted of two blocks, a reward-probability block and a reward-amount block. The two blocks had equal utility because the average amount of reward was the same in either block. Monkeys B and P participated in this procedure. In the reward-probability block, five CSs were followed by a liquid reward (0.4ml of water) with 100%, 75%, 50%, 25%, and 0% probability, respectively. In the reward-amount block, five CSs were followed by a liquid reward of 0.4ml, 0.3ml, 0.2ml, 0.1ml, and 0ml, respectively.

Each trial started with the presentation of a TS. As the TS, we used a small purple square in the probability block and a small yellow square in the amount block. The monkeys had to maintain fixation on the TS for 1s, then the TS disappeared and one of the five CSs was presented pseudo-randomly. After 1.5 s, the CS disappeared, and the US (if scheduled for that trial) was delivered. The monkeys were not required to fixate the CSs. On each trial the CS could appear in three locations: 10 degrees to the left or, right of the TS, or in the center. All trials were presented with a random inter-trial interval that averaged ~ 6 s (4 – 8 s). One block consisted of 40 trials with fixed proportions of trial types (each of the 5 CSs appears 8 times each block).

In a separate task, we tested the monkeys’ choice preference for the CSs used in the probability/amount Pavlovian procedure. This allowed us to evaluate the monkeys’ preference among the probabilistic CSs, among the amount CSs, and across the probabilistic and amount CSs. Monkeys B and P participated in this task.

Each trial started with the presentation of the TS at the center, and the monkeys had to fixate it for 1 second. Then two CSs appeared - each 10 degrees to the left or to the right of the TS. The CSs could be any of the 10 CSs shown in Fig. 3A. The monkey had to make a saccade to one of the two CSs within 5 seconds and fixate it for at least 500ms. Then, the other CS disappeared, and after 700ms the outcome (associated with the remaining CS) was delivered, as the remaining CS disappeared simultaneously. If the monkey failed to fixate one of the CSs, the trial was aborted and all stimuli disappeared. The ITI was 2–4 seconds. The trials were presented pseudo-randomly, so that a block of 180 trials contained all possible combinations of 10 CSs four times.

Experiment 3: Single block reward probability and reward amount procedure (Fig. 4)

The Pavlovian procedure used in Experiment 3 (Fig. 4) consisted of one block in which 5 fractals were presented as CSs. Monkeys Sm and P participated in this procedure. Two fractal CSs were followed by 0.5ml of liquid reward with 100% and 50% probability. And two other fractal CSs were followed by 0.25ml of liquid reward with 100% and 50% probability. The fifth fractal CS indicated no reward would be delivered.

Each trial started with the presentation of a TS (gray square). After 1 s, the TS disappeared and one of the five CSs was presented pseudo-randomly. After 1.5 s, the CS disappeared, and the US (if scheduled for that trial) was delivered. The monkeys were not required to fixate the TS or CS. On each trial the CS appeared in the center of the screen.

Experiment 4: Reward probability Pavlovian procedure with novel stimuli (Fig. 5)

In Experiment 4, instead of using previously conditioned fractals, monkeys B and P were exposed to three novel CSs, each associated with 100%, 50%, and 0% 0.4ml of water reward, respectively. Task trials consisted of sub-blocks of 21 trials, in which the three CSs were randomly presented 7 times (but note that when the block changed, the CSs did not change). This procedure was used ensure that the monkeys experienced each of the three novel CSs approximately an equal number of times.

Each trial started with the presentation of a TS. The monkeys had to maintain fixation on the TS for 1s, then the TS disappeared and one of the three CSs was presented pseudo-randomly. After 1.5 s, the CS disappeared, and the US (if scheduled for that trial) was delivered. The monkeys were not required to fixate the CSs. For monkey B, on each trial the CS could appear in one of three locations: 10 degrees to the left or right of the TS, or in the center. For monkey P, the CSs appeared in the center.

Data Analysis

We monitored and analyzed the magnitude of the anticipatory mouth movements (licking the juice spout) and anticipatory blinking (during the task that contained airpuffs) ¹³. To quantify the magnitude of anticipatory licking, during the CS presentation (a period of 1500 milliseconds), we counted the number of milliseconds during which the strain-gauge signal was > 3 standard deviations away from baseline. To quantify the magnitude of blinking during CS presentation, we counted the number of milliseconds during which the vertical eye lid signal was > 3 standard deviations away from baseline. Licking and blinking measures were normalized within each session by transforming them to z-scores.

For the analyses of neuronal activity, spike density functions (SDFs) were generated by convolving spikes times with a Gaussian filter (σ = 50 ms). TS responses were measured during the 1s TS period (Fig. 1A). In Experiments 2 and 4 where the monkeys were required to fixate the TS for 1s, the TS epoch was defined as 1s window in which the animal fixated the TS (TSa in Figs. 3 and 5). CS responses were measured during a window of 150ms after CS presentation until the end of the CS epoch. To normalize CS responses, we subtracted baseline activity from the activity during the CS epoch.

All statistical tests were two-tailed. To assess significant differences between neuronal activities across the CSs, we used the Kruskal-Wallis non-parametric ANOVA. For comparisons between two task conditions for each neuron we used ranksum test, unless otherwise noted. For comparisons between two task conditions across the population average we used a paired signed rank test, unless otherwise noted. All correlations were evaluated using Spearman’s rank correlations. The statistical significance of the correlation was tested using a permutation test (20,000 permutations).

Uncertainty-preferring neurons (in Experiment 1) were defined as those whose responses varied across the three CSs within either aversive or appetitive block (Kruskal-Wallis test, p<0.05) and that had significantly different responses for the uncertain CS (50%) compared with both of the certain CSs (100% and 0%) (post-hoc test: two-tailed ranksum test; p<0.01).

To quantify the population latency (n=21 from Experiment 1 and n=34 from Experiment 2; total=55 neurons) of CS-related reward uncertainty selectivity we first generated average SDFs for each neuron (sigma = 5ms) for all certain CS trials (Experiment 1 and 2: 100 and 0% CS responses) and uncertain CS trials (Experiment 1: 50% CS; Experiment 2: 25, 50, 75% CS responses). Thus, for each of the 55 neurons we obtained 2 SDFs (certain and uncertain trials). The latency of the population selectivity was determined by a paired signed rank test at each millisecond following the time of CS presentation. The running statistical test estimated the probability (P) at each millisecond that the observed difference between certain CS and uncertain CS average responses was due to chance. The uncertainty selectivity latency time was defined as the first millisecond the P dropped below 0.05 and remained below 0.05 for 20 milliseconds of the following 25 milliseconds. Latency reported is calculated for 55 neurons from Experiment 1 and 2.

To calculate ROC-based neuronal learning curves, the magnitude of the neuronal preference for uncertainty was measured by comparing CS responses for the certain CSs (100% and 0%) with 50% CS responses. The analysis was structured so that ROC area values > 0.5 indicate that the activity for the uncertainty-CS was greater than the CS activity for the certain CS; values < 0.5 indicate that the activity was higher for the certain-CS. For the analysis of neuronal-learning (Fig. 5), we first sorted the trials for each CS in the order of CS presentation. Therefore, the order of the trials for each CS represented the order in which the monkey experienced the CS (and the outcome probability associated with the CS). The ROC analysis was performed compared CS-responses (0% versus 50% CS and 100% versus 50% CS, separately) in independent trial bins of 5 trials in the order of presentation.