Neurons in Monkey Dorsal Raphe Nucleus Code Beginning and Progress of Step-by-Step Schedule, Reward Expectation, and Amount of Reward Outcome in the Reward Schedule Task

Abstract

The dorsal raphe nucleus is the major source of serotonin in the brain. It is connected to brain regions related to reward processing, and the neurons show activity related to predicted reward outcome. Clinical observations also suggest that it is important in maintaining alertness and its apparent role in addiction seems to be related to reward processing. Here, we examined whether the neurons in dorsal raphe carry signals about reward outcome and task progress during multitrial schedules. We recorded from 98 single neurons in dorsal raphe of two monkeys. The monkeys perform one, two, or three visual discrimination trials (schedule), obtaining one, two, or three drops of liquid. In the valid cue condition, the length and brightness of a visual cue indicated schedule progress and reward amount, respectively. In the random cue condition, the visual cue was randomly presented with respect to schedule length and reward amount. We found information encoded about (1) schedule onset, (2) reward expectation, (3) reward outcome, and (4) reward amount in the mean firing rates. Information theoretic analysis showed that the temporal variation of the neuronal responses contained additional information related to the progress of the schedule toward the reward rather than only discriminating schedule onset or reward/no reward. When considered in light of all that is known about the raphe in anatomy, physiology, and behavior, the rich encoding about both task progress and predicted reward outcome makes the raphe a strong candidate for providing signals throughout the brain to coordinate persistent goal-seeking behavior.

Introduction

Over the past decade there has been a great deal of work in trying to describe the brain systems that underlie reward expectation and motivation. A view strongly supported by experimental evidence is that the so-called neuromodulatory systems, dopamine, norepinephrine, and serotonin, play important roles in encoding and distributing information about reward expectation and reward value throughout the brain. The dorsal raphe nucleus is the major source of the brain's serotonin. Anatomical data show that it is widely and reciprocally connected to parts of the forebrain thought to be involved in reward processing, e.g., prefrontal and cingulate cortices, basal ganglia, and amygdala (Hornung, 2003; Hensler, 2006). Functional evidence suggests that serotonin is important for emotion (Merens et al., 2007), appetite (Garfield and Heisler, 2009), response to stress (Chaouloff et al., 1999), and aggressive behavior (Carrillo et al., 2009). It is only recently that physiologists have begun to investigate the dorsal raphe's possible role in reward processing. At this point it has been shown, first, in monkeys, that the activity of the dorsal raphe neurons are related to expected or received reward amount (Nakamura et al., 2008); second, in the rat dorsal raphe neurons are transiently modulated in response to reward delivery or reward omission (Ranade and Mainen, 2009); and third, dorsal raphe neurons are activated when the animal is waiting for conditioned or unconditioned stimuli, with the latter responses disappearing just before giving up in waiting for a reward (Miyazaki et al., 2011).

Here, we ask how the responses of dorsal raphe neurons are modulated by two factors that influence the value of rewards, one, workloads preceding reward, a type of temporal discounting, and, two, the amount of reward. We used a modified version of the previously used multitrial reward schedule task (Bowman et al., 1996). In this task to obtain a reward monkeys must complete schedules of one, two, or three sequential color discrimination (red-to-green) trials, with the number of trials remaining in the current schedule signaled by a visual cue. The cue gives the monkeys full information about the reward contingency, both how many trials need to be done and how big the reward will be. This design provides a means to determine whether knowledge of the projected outcome, that is, expectancy, influences the neuronal responses. We also used a condition in which the cue did not indicate the schedule progress or reward amount (a random cue condition).

We found neurons with responses related to schedule onset, reward expectation, and/or reward outcome; that is, whether the trial will be rewarded, and how much reward is delivered. We also found that the responses were temporally modulated with the information about progress through the task in the temporal pattern of the responses; this information about progress through the task was not available when the number of spikes was used without regard to their pattern over time.

Materials and Methods

Two male rhesus monkeys (Macaca mulatta, subject K: 6 kg, S: 6 kg) were used to collect behavioral and single-unit data. The experiments were all conducted following the guidelines for the Care and Use of Laboratory Animals as adopted by the University of Tsukuba.

Multitrial reward schedule task.

Experimental control and data acquisition were performed using the real-time experimental control system “REX” running under the QNX operating system. Visual stimuli were presented by “Presentation” (Neurobehavioral Systems) running on a Windows PC. All training and recordings were performed in a sound-attenuated room with a quiet auditory noise playing.

The monkey was initially trained to perform sequential red-to-green visual discrimination trials (Fig. 1A). The monkey sat in a primate chair in front of the computer video monitor (size: 40.5 × 30.5 cm, distance: 1.0 m, 22 × 17 degree of visual angle). When the monkey touched a bar mounted on the primate chair, a rectangular cue appeared at the top of the computer video monitor. Then, 800 ms later, a fixation square target (0.17 degree of visual angle) appeared at the center of the computer video monitor. The spot was replaced with the red square target (WAIT, 0.40 degree of visual angle) 400 ms after the fixation target appeared. While the red target was present, the monkey had to keep touching the bar. After a randomly chosen period (400, 600, 800, 1000, and 1200 ms), the red target changed to green (Go). If the monkey released the bar 150–1000 ms after the green target appeared (Bar-release), the green target changed to blue (OK, for 400–800 ms), and a liquid reward was dispensed by opening a solenoid-operated valve (200 ms). If the monkey released the bar too early (within 150 ms after Go signal), the trial was counted as an error. The intertrial interval (ITI) and the interschedule interval were both 1000 ms.

Figure 1.

Behavioral task and recording site. A, Sequential red-green visual discrimination trial. The monkey must release the bar within 1 s after red target (Wait signal) changed to green (Go signal). B, Multitrial reward schedule task. The monkeys were required to perform one, two, or three repeats of visual discriminations to earn one, two, or three drops of liquid reward. In the valid cue condition, the cue length indicates the number of required trials to obtain reward while the cue brightness indicates reward amount. In the random cue condition, cue was picked randomly from all cue sets. C, Coronal section of monkey brain at anteroposterior 0 by MR image.

After the performance in the sequential red-to-green visual discrimination trials reached 80% correct for 5 consecutive days, the multitrial reward schedule task was introduced. In this task, the monkey was required to perform one, two, or three repeats of visual discrimination trials (schedule) to earn one, two, or three drops of liquid reward (0.15 ml per a drop). The amount of reward was manipulated by changing whether the solenoid-operated valve was open. In correctly performed unrewarded trials, a sham reward apparatus was activated.

To begin a schedule, both the schedule length and the reward amount were picked randomly. During the task, a rectangular bar of light was presented as a cue at the top of the monitor. The cue length indicated the number of trials in the schedule and the cue brightness indicated the amount of reward (Fig. 1B). In “the valid cue condition,” the length of the cue was proportional to the number of successfully completed trials, becoming longer as the rewarded trial approached. The cue brightness became brighter as the amount of reward increased. In the “random cue condition,” the cue was randomly chosen from all cue sets. The schedule state was defined as “trial number/schedule length.” There were six schedule states (i.e., 1/1, 1/2, 2/2, 1/3, 2/3, and 3/3) and three levels of reward (i.e., one, two, or three drops).

Animal preparation and electrophysiology.

As part of the preparation for surgery, the location of the dorsal raphe nucleus was estimated by the 3.0 T magnetic resonance imaging (MRI) system (Signa Horizon; GE). The recording chamber and head holder were attached to the skull in an aseptic surgical procedure. Anesthesia was induced with ketamine (4 mg/kg) followed by sodium pentobarbital anesthesia (20 mg/kg). A recording chamber was fixed at an angle of 35° from the median line (Fig. 1C). The monkeys received antibiotics for 1 week after the surgery to reduce the risk of postoperative infections. A week after the surgical procedure, a tungsten microelectrode was inserted at anterior 0 mm with respect to Horsley–Clerk reference frame, and a new MR image was obtained to identify the actual location of the recording chamber with respect to the dorsal raphe nucleus (Snider and Lee, 1961). Single-unit activity was recorded using tungsten microelectrodes (1.4–1.6 MΩ; Microprobe). Unit activity was identified outside of the task, and then the task was started.

After the conclusion of the experiment, the monkeys were killed and the recording track locations were reconstructed from histological sections.

Behavioral data analyses.

We used error rate and reaction time averaged across all recording sessions as our measure of behavioral performance. We defined the error rate as the ratio of the failed number of bar-releases and the total number of trials in each schedule state. Reaction time was defined as the time to release the holding bar after the Go signal appeared. To evaluate the effect of the different reward amounts and schedule states on behavior, the error rates and reaction times were compared using χ² test and two-way ANOVA, respectively, within the R statistics programming environment (R Foundation for Statistical Computing, Team RDC, 2004).

Neuronal data analyses.

First, the neuronal activity before the task ran was recorded for at least 3 min to measure the baseline neuronal activity. To identify whether the neurons responded, the firing rate during each task event was compared with the baseline firing rate. There were six task events: Cue (cue onset), Wait (red target onset), Go (green target onset), Bar (bar-release), OK (blue correct target onset), and Reward. The analytic time window for Cue spanned 0–800 ms from the cue onset. Those for Wait, Go, Bar, or OK spanned 0–400 ms from the onset of each event. The Reward epoch was defined to include 200–1400 ms after sham or reward apparatus activation. The neuronal data with 10 or more trials in every schedule state were used in our analyses (thus, using data with 10–20 trials). We considered a neuron to be responsive in the task event when the firing rate in at least one schedule state (there are six schedule states in three reward levels) was significantly different from the base activity (t test, p < 0.00046, Bonferroni correction).

To learn whether the responses fell into “natural” groups we performed hierarchical clustering on the neuronal responses separately in each of the Cue, OK, and Reward periods. Since different neurons have different overall levels of activity, the firing rates (3 reward sizes × 6 schedule states giving 18 measurements per neuron) were normalized so that they were on a single scale: (firing rate—minimum firing rate)/(maximum firing rate—minimum firing rate). The population data matrix for each of the three epochs was subjected to the clustering (“hclust” function of R with agglomeration method; “ward”).

We also performed ANOVA on the following models. In the analysis window, we compared the firing rates between the first (1/1, 1/2, 1/3) versus non-first schedule states (2/2, 2/3, 3/3) (two-level, one-factor ANOVA, p < 0.01) (whether the ongoing trial was the schedule onset), and the unrewarded (1/2, 1/3, 2/3) versus rewarded schedule states (1/1, 2/2, 3/3) (two-level, one-factor ANOVA, p < 0.01). The firing rates in the rewarded schedule states (1/1, 2/2, 3/3) were compared among the trials with one, two, or three drops of reward (three-level, one-factor ANOVA, p < 0.01). The firing rates in the unrewarded schedule states (1/2, 1/3, 2/3) were also compared among the trials associated with future one, two, or three drops of reward (three-level, one-factor ANOVA, p < 0.01). At every task event, we calculated the percentage of neurons having significant response in any of the three different ANOVA models (first vs non-first, rewarded vs unrewarded, and among different reward amounts). In addition, using these ANOVA models, we analyzed the variance explained, which is defined as the percentage of variance explained by a given model divided by the total variance (SS, sum of square; SS_factor/[SS_factor + SS_residuals] × 100) (Simmons and Richmond, 2008). For examining the pre-cue activity of the schedule onset neurons, we used the analysis window from 400 to 0 ms before the cue onset.

To investigate how much information related to schedule neurons carried, we performed information theoretic analysis. The information can be calculated as follows:

where I(S;R) is the information transmitted about the conditions, S, given the responses R. S is the average over each condition, and s is the condition related to response r. P(s|r) is the probability of s given r, i.e., the conditional probability of the condition being selected on the basis of the response. P(s) is the a priori probability of the condition, which is known in the experiment. Obtaining an accurate estimate of the transmitted information, I(S;R), requires an accurate estimate of P(s|r). We used a neural network to carry out a nominal regression of the experimental condition on the neural response. Information theoretic analysis by the neural network was performed as previously described (Kjaer et al., 1994; Shidara and Richmond, 2004). We used response codes using the spike count alone, and the spike count with up to the first three principal components. To calculate the principal components, we first smeared the spikes in each by convolving them with a Gaussian kernel of SD 10 ms and sampling the resulting smooth signal at 10 ms intervals. Using the smoothed version of every response, we calculated the coefficients of the first three principal components by using “prcomp” function of R. This vector of three coefficients was used along with the spike count as a code representing temporal features of the response. These vectors from one neuron were used as inputs to the neural network, and the two, four, or six states of the schedule were used as target output in the analysis. Early stopping using cross-validation was used to prevent overfitting by the neural network. The two states were defined as first/non-first schedule states or unrewarded/rewarded schedule states. The four states were defined as 1/1, 1/2, 1/3, and the non-first schedule states, or 1/2, 1/3, 2/3, and the rewarded schedule states. The first code asks if there is information about whether the ongoing trial was the schedule onset (first vs non-first) or if there is information about whether the current trial will be rewarded or not. The four-part code asks whether there is information relative to the schedule length or schedule progress, and the six-part code asks whether there is information about the specific schedule state. We calculated the information by using spike counts alone, and an extended code using the first three principal components of the neuronal responses within Cue, OK, and Reward periods.

To examine the relationship between the baseline neuronal activity and the positions of recorded neurons, regression analysis was performed. We entered the baseline activity as a dependent variable, and the anteroposterior and mediolateral positions of recorded neurons as independent variables. The firing rate and IR, which quantifies irregularity of spike trains (Davies et al., 2006; Nakamura et al., 2008), outside the task, and duration of single unit extracted from unfiltered raw data (Mizuhiki et al., 2012) was used as the baseline activity. IR was quantified by the following formula:

where ISI_i is the ith interspike interval (ISI) and ISI_i+1 is i + 1th of the ISI.

To investigate whether these three properties of the baseline neuronal activity and recorded position differed in neuronal response types, they were compared between the response types (Wilcoxon signed rank test, p < 0.05).

Results

Task performances

In the valid cue condition the behavior of both monkeys was affected by both reward amount (Fig. 2A; colors, χ² test, monkey S: χ² = 136.18, df = 2, p < 0.05; monkey K: χ² = 101.10, df = 2, p < 0.05) and schedule state (Fig. 2A, solid lines, χ² test, monkey S: χ² = 180.07, df = 5, p < 0.05; monkey K: χ² = 124.94, df = 5, p < 0.05). Overall, the monkeys made the most errors when the reward would be small in the first trial of the longest schedule (Fig. 2A; detailed analysis in Table 1).

Figure 2.

Behavioral results. A, The error rates in two monkeys performing multitrial reward schedule tasks. The reward level is indicated in different color (blue, one drop; green, two drops; red: three drops). Circle and cross denote the performances in the valid and the random cue condition, respectively. Left, Shows the performance of monkey S; right, Shows the performance of monkey K. B, Reaction time to release the holding bar. The same convention as in A.

Table 1.

The statistical test results of task performance

Error rate (χ2 test)		Monkey S		Monkey K
	df	χ2	p	χ2	p
One drop versus two drops (1/3)	1	29.00	p < 0.05	9.56	p < 0.05
One drop versus three drops (1/3)	1	51.12	p < 0.05	36.88	p < 0.05
Two drops versus three drops (1/3)	1	4.03	p < 0.05	8.65	p < 0.05
First versus non-first (random)	1	45.05	p < 0.05	18.22	p < 0.05

In the random cue condition (Fig. 2A, dashed lines), the error rates were smaller than those in the valid cue condition (χ² test, monkey S: χ² = 29.47, df = 1, p < 0.05; monkey K: χ² = 19.56, df = 1, p < 0.05). Overall, in the random cue condition, the error rates in the first schedule states (1/1, 1/2, 1/3) were smaller than those in the non-first schedule states (2/2, 2/3, 3/3) in both monkeys (Fig. 2A; detailed comparisons in Table 1).

The reaction times in the valid cue condition showed the same patterns as were seen with the error rates (Fig. 2B; detailed comparisons in Table 2). Initially we had predicted that the monkeys would not learn the amount of reward in the random cue condition, but monkey S appears to have done so. The reaction times in the random cue condition differed across reward amount levels after the first trials, possibly because the duration of ITI after the unrewarded schedule states was proportionally longer due to the number of sham apparatus activations; this monkey could have become aware of this ITI elongation.

Table 2.

The statistical test results of reaction time

Reaction time (Two-way ANOVA)	Monkey S			Monkey K
	df	F	p	df	F	p
Schedule states (valid)	5, 16,126	91.25	p < 0.01	5, 8868	55.72	p < 0.01
Reward amount (valid)	2, 16,126	65.51	p < 0.01	2, 8868	30.18	p < 0.01
Schedule states (random)	5, 14,565	44.51	p < 0.01	5, 7636	1.76	p = 0.12
Reward amount (random)	2, 14,565	29.74	p < 0.01	2, 7636	18.15	p < 0.01
First schedule states (schedule)	2, 7305	0.34	p = 0.71	2, 3827	2.79	p = 0.061
First schedule states (amount)	2, 7305	0.42	p = 0.66	2, 3827	4.10	p = 0.017

Electrophysiological result

We recorded 98 single neurons from dorsal raphe nucleus in two rhesus monkeys (monkey S, 57 neurons; monkey K, 41 neurons). All 98 neurons were recorded in both valid and random cue conditions. We were unable to identify systematic differences in either recording site or neuronal response types between the two monkeys. Therefore, all recorded neurons were treated as a single population.

Historically electrophysiological characteristics of a single unit were said to be useful for identifying neurochemicals (Aghajanian and Haigler, 1974; Vandermaelen and Aghajanian, 1983). Recent reports, however, suggested that the electrophysiological characteristics of a single unit (waveform, firing rate, and variability of spike intervals) are not reliable indicators of transmitter types released by a neuron (Allers and Sharp, 2003; Kocsis et al., 2006; Mizuhiki et al., 2012), so we treated all of the neuronal data as coming from one population, recognizing that the underlying neuronal types are likely to be mixed.

At each task event, just over 50% of the neurons responded in that the firing changed significantly from the background (see Materials and Methods), with the number of responsive neurons at each task event shown in Figure 3. There was no significant difference in number of responsive neurons across task events (χ² test, χ² = 1.55, df = 5, p = 0.907).

Figure 3.

Number of neurons responding at every task event. More than 50 out of 98 neurons responded at each task event.

From inspection, it appeared that there were four main classes of neurons: neurons related to (1) schedule onset, (2) reward expectation, (3) reward outcome, and (4) amount of reward. To test whether our observation that there might be four classes of neurons was a reasonable interpretation of the data, we performed hierarchical clustering (hclust function in R) on the population of neurons in the valid cue condition. This clustering made no assumption about the number of major clusters or their relations. This clustering showed that early in trials the population was distinguishing between trials that were first and non-first (Cue period; Fig. 4A), and late in trials the population distinguished between trials with reward and those without (OK period; Fig. 4B, and Reward period; Fig. 4C). The distinction between reward and no-reward was established by the time that the OK signal (blue target) appeared, showing that at this point in the trial the population encoded reward expectation, that is, the information was provided by the cue. This distinction carried through the reward itself, at which time we consider it to have encoded reward outcome. Thus, overall the population encodes signals that distinguish first versus non-first at the beginnings of the trials, and by the end of the trials the population signal had been transformed into one encoding information about whether this is a reward trial or not (reward vs no-reward).

Figure 4.

Hierarchical clustering on the population of neuronal responses. Hierarchical clustering was applied to the population of neuronal responses in six schedule states in three reward amount levels (18 states in total) (hclust function in R, ward method). The “tree” represented the binary divisions, and the vertical distance represents the relative distances of the cluster centers. Cue period (A); OK period (B); Reward period (C).

To characterize the population in greater detail, below we analyzed the data for the individual neurons, providing a more complete description of their sensitivities.

Schedule onset neurons

Fifty-six neurons responded in Cue period. Figure 5A shows a neuron for which the neuronal activity increased after the cue onset in the first schedule states of both valid and random cue conditions (black and gray lines in 1/3, 1/2, and 1/1 panels, respectively). The firing rate was different between the first and non-first schedule states in both conditions (two-level, one-factor ANOVA, valid cue condition, F_(1,329) = 77.05, p < 0.01; random cue condition, F_(1,355) = 93.74, p < 0.01). Figure 5B shows a neuron for the activity increased after the cue onset in the first schedule states of the valid cue condition only. The firing rate was different between the first and non-first schedule states in the valid cue condition (two-level, one-factor ANOVA, F_(1,180) = 18.54, p < 0.01), but was not different in the random cue condition (two-level, one-factor ANOVA, F_(1,178) = 0.30, p = 0.59).

Figure 5.

Schedule onset neurons. Two examples are demonstrated (A, B). Top shows Raster plots and bottom shows their spike densities (σ = 20 ms). Horizontal axis indicates the time from cue onset (milliseconds). Vertical axis in bottom shows the firing rate (spikes/s). Black vertical line shows the cue onset. Black points and lines show the neuronal activities in the valid cue condition; gray points and lines show the neuronal activities in the random cue condition. The schedule state is shown at the top of each raster plot. The time window for statistical analysis is shown by shaded gray area in 1/3. A, Responses at the beginning of the first schedule states in both the conditions. B, Responses to the first schedule states only in the valid cue condition. C, The number of neurons showing significant difference in firing rate between the first and non-first schedule states in Cue period. Horizontal axis means responsive conditions. The bars with different colors mean different response strength between the first and non-first schedule states. D, The population activity of schedule onset neurons that showed larger activity in the first schedule states in Cue period. The firing rates were normalized throughout the trial, with the data aligned on the event of interest, where the events are presented in the order from left to right in which they occurred. Horizontal axis shows time from each task event. Vertical axis shows normalized firing rates. Normalization as in cluster analysis. Light gray lines show the mean normalized firing rate in 1/3, dark gray lines show that in 2/3, and black lines show that in 3/3. E, The population activity of schedule onset neurons that showed larger activity in the non-first schedule states in Cue period. Symbols, lines, and axes as in Figure 5D.

Figure 5C shows the numbers of neurons for which the responses distinguished between the first and non-first schedule states in the period following the cue. Neurons (17/56) discriminated whether the forthcoming trial was the first schedule state in the random cue condition Fig. 5C, 11 from the leftmost bar and 6 from the rightmost bar). The responses of these neurons were not affected by the visual cue. Even in the random cue condition, the neuronal activity distinguished whether the current trial was the first schedule state presumably according to whether the preceding trial was rewarded or not. For 15/17 neurons the firing rates across the first schedule states (1/1, 1/2, and 1/3) were indistinguishable as were the responses across the non-first schedule states (2/2, 2/3, and 3/3; Tukey's HSD test, p > 0.05).

We investigated if these neurons discriminated the first schedule states from the non-first schedule states before cue presentation. Neurons (14/17) showed differential activity in the pre-cue period (paired t test, p < 0.05), suggesting that these responses reflected whether the preceding trial was rewarded or not; that is, these neurons reflected the immediate reward history.

The mean of population activity of schedule onset neurons throughout the trials is shown in Figure 5D, “first > non-first” type, and in Figure 5E, “first < non-first” type, for the valid cue condition, where it can be seen that the average activity for the first trials is different from the average activity for non-first trials in the Cue period. In the first > non-first type the population activity is also different during Reward period.

The responses of 15/56 neurons showed differences between the first and non-first schedule states only in the valid cue condition (Fig. 5C, center bar). Only 1 of these 15 showed a significant difference in firing rate across the first schedule states (1/1, 1/2, and 1/3) or across the non-first schedule states (2/2, 2/3, and 3/3; Tukey's HSD test, p < 0.05). This is in line with the clustering results above. This differential activity arose only after the cue appeared for 10/15 of these neurons. It appears that these responses code for schedule onset, but not schedule length.

Reward expectation neurons

Sixty-two neurons changed activity either in the unrewarded or rewarded schedule states before reward event. Figure 6A shows a neuron whose activity in the unrewarded schedule states of the valid cue condition started to increase even before the OK signal (black line in 1/3, 1/2, and 2/3). The firing rate during OK period was significantly different between the unrewarded and rewarded schedule states in the valid cue condition (two-level, one-factor ANOVA, F_(1,205) = 99.31, p < 0.01). The neuronal activity was not different across the unrewarded and rewarded schedule states in the random cue condition (two-level, one-factor ANOVA, F_(1,181) = 0.068, p = 0.79). The activity in the random cue condition was lower than that in the responsive schedule states in the valid cue condition. Neurons (17/62) discriminated whether the current trial was rewarded in the valid cue condition, and failed to respond in any schedule states during the random cue condition (Fig. 6C, black bars). These neurons seem to encode reward expectation.

Figure 6.

Reward expectation neurons. Black vertical line in each plot indicates the onset of OK target. The same convention as in Figure 5. A, The activity appeared in the unrewarded schedule states in the valid cue condition. B, The activity appeared in the rewarded schedule states in the valid cue condition and all states in the random cue condition. C, The number of neurons showing significant difference in firing rate between the unrewarded and rewarded schedule states in OK period. The bars with different colors show whether there were responses in the random cue condition. D, The population activity of reward expectation neurons that showed larger activity in the unrewarded schedule states in OK period. Symbols, lines, and axes as in Figure 5D. E, The population activity of reward expectation neurons that showed larger activity in the rewarded schedule states in OK period. Symbols, lines, and axes as in Figure 5D.

Figure 6B illustrates another type of reward expectation. In this group, the neurons responded in the rewarded schedule states of the valid cue condition and in every schedule state of the random cue condition. The activity was significantly different in the unrewarded and rewarded schedule states in the valid cue condition (two-level, one-factor ANOVA, F_(1,210) = 27.20, p < 0.01), but there was no significant difference among schedule states in the random cue condition (two-level, one-factor ANOVA, F_(1,240) = 0.34, p = 0.56). Neurons (16/62) showed this type of activity pattern; that is, there was an activity change in the valid cue condition and there was a change in firing for all schedule states in the random cue condition (Fig. 6C, gray bars). These neurons seem to predict that a future reward is possible.

The mean of population activity for these neurons throughout the trials is shown in Figure 6, D and E (no-reward > reward type and no-reward < reward type, respectively), where the activity is different in reward versus no-reward trials. The number of neurons showing an increase in activity in the unrewarded schedule states was not significantly different from the number showing a decrease (χ² test, χ² = 1.48, df = 1, p = 0.22). However, in a paired t test, the mean firing amplitude of these neurons was significantly larger in the unrewarded schedule states than that in the rewarded schedule states (paired t test, p < 0.05).

Reward outcome related neurons

Sixty-two neurons showed responses in the Reward period. Figure 7A shows one example neuron that increased activity after the reward started in the unrewarded schedule states of both conditions. Figure 7B shows another type of example in which the neuron increased its activity after reward onset in the reward schedule states of both conditions. In both neurons, the firing rate was significantly different between the unrewarded and rewarded schedule states both in the valid and random cue conditions (two-level, one-factor ANOVA, Fig. 7A: valid cue condition, F_(1,250) = 108.81, p < 0.01; random cue condition, F_(1,180) = 62.89, p < 0.01; Fig. 7B: valid cue condition, F_(1,288) = 34.09, p < 0.01; random cue condition, F_(1,211) = 50.34, p < 0.01).

Figure 7.

Reward outcome neurons. Black vertical line in each plot shows the onset of reward apparatus activation. The same convention as in Figure 5. A, The activity increased in the unrewarded schedule states in both conditions. B, The activity increased in the rewarded schedule states in both conditions. C, The number of neurons showing significant difference in spike count between the unrewarded and rewarded schedule states. Horizontal axis means responsive conditions. The bars in different colors show different response strength between the rewarded and unrewarded schedule states. D, The population activity of reward outcome neurons that showed larger activity in the unrewarded schedule states in the Reward period. Symbols, lines, and axes as in Figure 5D. E, The population activity of reward outcome neurons that showed larger activity in the rewarded schedule states in Reward period. Symbols, lines, and axes as in Figure 5D.

Figure 7C shows the number of neurons that differed activity between the unrewarded and rewarded schedule states in the valid and random cue condition. Neurons (26/62) (Fig. 7C, leftmost bar) seem to encode no-reward or reward delivery because the responses were shown in both valid and random cue condition. The mean of population activity in the Reward period in the reward trials is different from that in the no-reward trials, as shown in Figure 7, D and E (no-reward > reward type and no-reward < reward type, respectively).

Relation with reward amount

There were 24 neurons that discriminated the amount of reward in the rewarded schedule states, and there are 2 neurons that discriminated the predictable amount of reward in the unrewarded schedule states (three-level, one-factor ANOVA, p < 0.01). Figure 8 shows an example of the former type. The neuron showed larger responses when two or three drops of reward were delivered (three-level, one-factor ANOVA, F_(2,250) = 26.36, p < 0.01). Table 3 shows the classification of the reward amount neuron. There were no neurons that discriminated every level of reward amount (zero, one, two, and three drops). Neurons (13/24) discriminated two levels, like the neurons illustrated in Figure 8. Neurons (5/24) discriminated three levels, and 6/24 neurons discriminated between no-reward and reward.

Figure 8.

Reward amount neurons. Example of reward amount coding neurons. The reward amount level is indicated by different colors (blue, one drop; green, two drops; red, three drops). The responses in the valid cue condition are sorted in the unrewarded schedule states (left) and in the rewarded schedule states (right). This neuron responded when two or three drops of reward were dispensed (right, green and red lines).

Table 3.

The classification of reward amount related neurons

Discrimination of reward amount	Number of neurons
3, 2 > 1, 0	9
0, 1 > 2, 3	2
3 > 2, 1, 0	1
3, 2, 0 > 1	1
Other	11

The firing rate was compared among each amount of reward (Tukey's HSD test, p < 0.05) and classified the neurons into five groups. Inequality sign means the difference of firing rate amplitude.

Percentage and variance explained around each task event period

We examined how the response profile changed through the course of trials. The percentage of neurons discriminating between the first and non-first schedule states was the greatest at the beginning of trials in both the valid and random cue conditions (Fig. 9A, top, solid and dashed black lines, valid cue condition, χ² = 21.25, df = 5, p < 0.05; random cue condition, χ² = 15.94, df = 5, p < 0.05). However, the amount of information available must be small because the variance explained by schedule was only a little higher in Cue period than later in the trial (Fig. 9A, bottom, solid and dashed black lines).

Figure 9.

Percentage of neurons and variance explained. Horizontal axis indicates task event. Top, Vertical axis indicates the percentage of responsive neurons. Bottom, Vertical axis indicates the percentage of variance explained by responsive neurons, which is the proportion of variance accounted by a given ANOVA model (SS_factor/[SS_factor + SS_residuals]). A, Black solid and dashed lines indicate the neurons that discriminate the beginning of schedule in the valid and the random cue conditions, respectively. Gray solid and dashed lines indicate the neurons that discriminate the reward in the valid and the random cue condition, respectively. B, Gray line shows the neurons that discriminate reward amount level in the unrewarded schedule states of the valid cue condition. Black line shows the neurons that discriminate reward level in the rewarded schedule states of the valid cue condition.

The percentage of the neurons distinguishing between the unrewarded and rewarded schedule states decreased from Cue to Wait period, then increased toward the end of trial in the valid cue condition (Fig. 9A, top, solid gray line). In the random cue condition, a small percentage of neurons coded whether there was reward before the reward event. This number increased substantially in the Reward period (Fig. 9A, top, dashed gray line). The value of variance explained by the reward/no-reward factor showed the similar trends (Fig. 9A, bottom, solid and dashed gray lines). This raises the possibility that these neurons are coding for the presence of a reward in the current (the activation before the reward) or immediately preceding (activation after the reward carrying through to the beginning of the next trial) trial.

Figure 9B shows that few neurons discriminated the amount of reward in the unrewarded schedule states (gray line). In the rewarded schedule states, 24% of neurons distinguished the amount of reward in Reward period, but not at other times (black line). The average of variance explained showed the same trends (Fig. 9B, bottom).

Information about each schedule state

From inspection, it seemed possible that the temporal variation in neuronal activity carried further information about the schedule. To investigate this possibility, we performed information theoretic analysis using a neural network to estimate the needed conditional probabilities using both spike counts and principal components as response codes (Kjaer et al., 1994). We analyzed how the condition-specific (schedule state) information changed assuming different response codes (Shidara and Richmond, 2004).

First we examined whether the neurons that responded in Cue, OK, and Reward periods carried information related to schedule other than first/non-first discrimination (schedule onset) by using only spike count as response code. Assuming a spike count code, the information in Cue period for two states (schedule onset information) was not significantly different from that for four states (schedule length information) (Fig. 10A, dark gray bar) (paired t test, p = 0.049), assuming six states there was more information than for two and four states (paired t test, p < 0.01). In the OK and Reward periods (Fig. 10B,C, dark gray bar), the information for two states carried in the spike count code was less than the estimation uncertainty of 0.025 bits (Kjaer et al., 1994). The value became significantly larger as the states were increased from 2 to 4, and to 6 (paired t test, p < 0.01). These results suggest that the spike count code in Cue period carried information related to schedule onset and some information about the schedule states, whereas the spike count code in OK and Reward periods carried little information about the schedule onset, but have some information about schedule states.

Figure 10.

Information theoretic analysis for schedule onset. Horizontal axis shows the number of states. Vertical axis shows the information using spike count or first three principal components in addition to spike count. Dark gray bar is the result of information using spike count. Light gray bar is the result of using first three principal components in addition to spike count. Bars indicate SE. Two states means schedule onset or not. Four states means each first schedule state and non-first schedule state. Six states means all schedule states. A, The information in Cue period. B, The information in OK period. C, The information in Reward period.

Next we examined whether the information by using first three principal components in addition to spike count increased, to check if there was further information in temporal response profile. The principal components were extracted from the set of all responses for all trials in all conditions triggered on the event of interest (e.g., cue appearance). Before principal component extraction with the R function prcomp, each response was convolved with a Gaussian pulse with σ = 10 ms. An example showing how principal components represent the temporal profile is shown in Figure 11A. As should be expected from the Scree plot (Fig. 11B), the first three principal components account for a large proportion of the response variance, doing an excellent job of representing the response. Since the first principal component is well correlated with the spike count (r = 0.77, p < 0.01, in this example), the second and third principal components can be taken to represent temporal variation. We superimpose the responses when reconstructed from the weights of the first three principal components (Fig. 11C, inset) according to R_n(t) = avg + β_lnφ₁ + β_2nφ₂ + β_3nφ₃, where R(t) is the reconstructed average response in condition n, avg is the average response taken all responses in all conditions, φ is a principal component, and β is the weight of the principal component as the average of that weight in condition n. Figure 11C shows that the reconstructed neuronal activity from the first three principal components and the average firing gave a good fit to the original time-dependent response.

Figure 11.

Temporal feature of the neuronal responses. A, Time-dependent average response (black line) across all trials in all conditions triggered on the reward apparatus activation, and the shapes of the first three principal components. Horizontal axis shows time from reward apparatus activation. Left vertical axis is the firing rate (spikes/s). The right vertical axis gives the relative value of the principal component. The first three principal components are indicated by different colors (blue, PC1; green, PC2; red, PC3). B, Percentage of variance in the first 20 principal components. Cumulative variance explained was >80% in the first 20 principal components. C, Reconstructed neuronal activity in each schedule state. Horizontal axis shows time from reward apparatus activation. Vertical axis shows firing rate. Black lines are reconstructed neuronal activity, which is the average firing rate across all schedule states plus the first three principal components weighted the coefficient in each schedule state. Gray lines show original firing rates. Inset shows the coefficient of the first three principal components for the schedule set indicated by the fraction at the top of the panel.

Figure 10 shows that, overall, the information using the first three principal components in addition to spike count was 1.9 times larger than the information calculated by using only spike count (paired t test, p < 0.01). The difference in information across various combinations of schedule states in each task event period showed the same trends as for the spike count code (paired t test, Cue period: 2 states vs 4 states, p = 0.052; 2 states vs 6 states, p < 0.01; 4 states vs 6 states, p < 0.01; OK period: p < 0.01; Reward period: p < 0.01) (Fig. 10, light gray bar).

Second, we examined whether the neurons that responded in Cue, OK, and Reward periods carried information related to schedule other than reward/no-reward discrimination (reward contingency) by using spike count as response code (Fig. 12, dark gray bar). When using spike count in Cue period, the value for two states (reward contingency) was very small (< 0.025 bit). The value for the schedule progress (four states) was larger than that for the reward contingency (two states) (paired t test, p < 0.01), and significantly larger still for all schedule states (six states) (paired t test, p < 0.01) (Fig. 12A, dark gray bar).

Figure 12.

Information theoretic analysis for schedule progress. The same convention as in Figure 10. Two states means reward contingency. Four states means schedule progress depending on reward delivery.

The neurons that responded in OK and Reward periods showed different trends. In the neurons that showed responses in OK period, the value for reward contingency (two states) was not significantly different from that for the schedule progress (four states) (paired t test, p = 0.14) and all schedule states (six states) (paired t test, p = 0.27) (Fig. 12B, dark gray bar). The value for the schedule progress (four states) was not significantly different from that for all schedule states (six states) (paired t test, p = 0.94), either. Thus, the spike count code immediately before reward event carried only information about whether the reward is forthcoming or not.

In the Reward period, the information about reward contingency (two states) did not significantly differ from that for the schedule progress (four states) (paired t test, p = 0.13), but it was slightly, but significantly lower than that for all schedule states (six states) (paired t test, p < 0.01) (Fig. 12C, dark gray bar). The information about schedule progress (four states) was not significantly different from that for all schedule states (paired t test, p = 0.038). These results suggest that the spike count code in Reward period carried information mostly related to reward contingency.

We also examined whether the amount of information increased when the first three principal components in addition to spike count. The information that was calculated by using spike count and the first three principal components was 1.7 times larger than the information calculated by using only spike count (paired t test, p < 0.01). In each task event period, the information carried in the temporal code became significantly larger as the states were increased from two to four, and to six (paired t test, p < 0.01) (Fig. 12, light gray bar). These results show that the information in temporal neuronal responses included relatively more information about schedule progress and schedule states than was available from the spike count alone in OK and Reward periods.

Relationship between neuronal activity and location of neuron

To examine the relationship between baseline activity and recorded positions, we performed regression analysis. The dependent variable was firing rate and IR outside the task, and duration of single unit extracted from the unfiltered raw data. The independent variables were the absolute mediolateral positions or anteroposterior positions of the recorded neurons. The results of p value in Table 4 show that the significant correlation was observed only between the firing rate of the baseline activity and the anteroposterior positions. The neurons showing higher firing rate were recorded from more anteriorly (β = −1.14).

Table 4.

The relationship between the baseline activity and recorded positions

	Mediolateral position	Anteroposterior position
Firing rate	0.65	0.0038*
IR	0.24	0.35
Duration	0.95	0.38

Regression analysis was applied to examine the relationship between the baseline activity and recorded positions. Number shows p value of the regression analysis. Asterisk means a significant correlation.

We compared electrophysiological characteristics among different neuronal response types. For example, we compared the firing rate and IR of the baseline activity, duration of unit waveform, and recorded positions between two types of the schedule onset neurons (depending on the cue or preceding reward), two types of reward expectation and reward outcome neurons (higher activity in unrewarded or rewarded schedule states), and reward amount-related neurons (amount related or not in reward outcome neurons) (Table 5, p value of Wilcoxon signed rank test). We found significant differences only in the firing rate and anteroposterior positions between two types of reward expectation neurons during the OK period. The neurons showing larger responses during the OK period in the unrewarded schedule states had a larger firing rate of the baseline activity and were recorded from the more posterior half, than the neurons showing larger responses in the rewarded schedule states. The distribution of the former type (circle in Fig. 13) was shown in the reconstructed map of the recorded position (Fig. 13).

Table 5.

The relationship between the baseline activity properties, recorded positions and neuronal response profiles

	Firing rate	IR	Duration	Mediolateral position	Anteroposterior position
Schedule onset	0.28	0.88	1.00	0.35	0.45
Reward expectation	0.019*	0.23	0.50	0.51	0.031*
Reward delivery	0.073	0.051	0.76	0.34	0.61
Reward amount	0.24	0.45	0.060	0.25	0.20

We compared the baseline activity properties and recorded area to the neuronal response profiles in each task event period (Wilcoxon test). In Schedule onset row, the firing rate and IR of the baseline activity, duration of unit waveform and recorded positions was compared between the cue-dependent and cue-independent type neurons. In Reward expectation and Reward delivery rows, those were compared between the neurons that showed higher activity in the unrewarded schedule states and the neurons that showed higher activity in the rewarded schedule states in OK and Reward periods, respectively. In Reward amount row, those were compared between the neurons that distinguished the reward amount and the neurons that responded at reward outcome without reward amount discrimination. Number shows p-value of Wilcoxon test. Asterisk means a significant difference.

Figure 13.

The relation between neuronal response type in OK period and recorded position. Symbols show the recorded position on the reconstructed brain map. Circle means the neuron that showed high activity in the unrewarded schedule states in OK period. Cross means the neurons that showed high activity in the rewarded schedule states in OK period.

In the primate, the lateral dorsal raphe nucleus, so-called “lateral wing” around the trochlear nucleus, is strongly developed (Jacobs and Azmitia, 1992; Nakamura et al., 2008). We compared between the neuronal responses recorded in the lateral part (the distance from midline was >1 mm, 22/98 neurons) and those in the medial part. There were no significant differences with respect to location in the numbers of neurons in the different response classes described above (χ² test p > 0.05) that we could identify.

Discussion

We have found neurons in the dorsal raphe nucleus showing responses related to schedule onset, reward expectation, reward outcome, and reward amount. The amount of information about the schedule increased as the reward approached, with this additional information being found almost exclusively in the temporal variation of the response. The neurons did not carry information about the reward amount before reward onset. Therefore, these dorsal raphe neurons seem to encode information about the start of trials and expectation of reward, and the stepwise progress or proximity to the reward. They did not seem to encode information about the relative values among predicted rewards, at least not before reward delivery itself. When considered in light of all that is known about the raphe in anatomy, physiology, and behavior, this rich encoding about both task progress and predicted reward outcome makes the raphe a strong candidate for providing signals throughout the brain to coordinate persistent goal-seeking behavior.

Recorded area

The dorsal raphe nucleus is composed of several subdivisions. The subdivisions are characterized according to their different chemotypes and projections, and, perhaps, not surprisingly, the neurons in the different subdivisions seem to be endowed with different functions (Vertes, 1991; Charara and Parent, 1998; Peyron et al., 1998; Sakai and Crochet, 2001; Monti, 2010). Here, the firing rates of the neurons during the OK period in the unrewarded schedule states were higher in the caudal part of dorsal raphe nucleus than in the rostral part. In rodent studies, the caudal part of dorsal raphe nucleus appears to be related to stress and anxiety (Hammack et al., 2002; Abrams et al., 2004; Hale and Lowry, 2011). Therefore, the neuronal activity we observed in the caudal part of the nucleus in unrewarded trials might be related to the stress or anxiety of having to work even knowing that no reward will be forthcoming.

The effect of schedule for the activity of dorsal raphe neurons

The dorsal raphe neurons we recorded had responses that were modulated through the schedules and trials. The activity at the time of Cue distinguished between the first and non-first trials, and the activity at the time of OK and Reward distinguished between trials that would be rewarded and those that would not be. The information distinguishing between the first and non-first trials did not necessarily depend on the cue; this difference is related to whether the preceding trial had been reward (end of a schedule) or not (in the middle of a schedule). The information related to whether there would be a reward at the end of the trial depended on the meaning of the cue, which the behavior shows that both monkeys had learned.

For many neurons the responses to the cue were related to whether the trial was at the beginning of schedule or not. Some dorsal raphe neurons carried information related to schedule onset at the start of trial, but did not show reward contingency or future reward information, whereas other neurons carried information related to reward around the end of trial.

Nakamura et al. (2008) reported that the activity of dorsal raphe neurons was modulated by the expected and received reward size with either large or small reward. Ranade and Mainen (2009) reported that dorsal raphe neurons in rat responded to reward delivery and reward omission. Our results are consistent with both of these reports, and extend their findings. We found that most of the neurons in this group distinguished among reward amounts in a quantized manner; that is, many of these neurons responded differently above and below a threshold, for example, responding strongly when the monkey received two or three drops, and did not respond when the monkey received one drop, or when there was no reward. Thus, few of these neurons could be seen as coding for “reward amount” in that their responses did not covary reward by reward with the reward amount, but many of the neurons represented a threshold such that for any reward larger than this threshold, the neuron would increase its firing.

Many dorsal raphe neurons that distinguished between no-reward and reward expectation increased their firing as the monkeys waited for reward delivery, a result similar to what Miyazaki et al. (2011) found, where it was reported that putative serotonergic neurons in dorsal raphe nucleus increased their firing as the rats waited for future reward.

Bromberg-Martin et al. (2010) showed that dorsal raphe neurons in monkey also encode information about potential for producing future rewards with this difference in response appearing at the beginning of trials. We also show that not only the responses in the dorsal raphe neurons carry information about forthcoming rewards, but they also carry information that would be useful for keeping track of the progress toward both where the monkey is in the current schedule and the ends of trials. When we extended our analysis to include the rate modulation of the responses we found that in the periods after cue presentation and around reward there was additional information related to the proximity to the reward. As part of this spectrum of information related to the progress or expectation about the progress in the task, there were neurons whose activity appeared as the monkeys were waiting for the next step (OK period response in the unrewarded schedule states) and unpredictable reward (OK period response in the random cue condition). These results suggest the possibility that the activity of dorsal raphe neurons was related to delayed reward with several steps, and code the delay/workload to reward using the temporal variation of the firing as a substantial part of the coding.

A recent report helps attach these results to the activity of serotonin neurons in the dorsal raphe nucleus (Miyazaki et al., 2012). When the 5-HT_1A receptor agonist, 8-OH-DPAT, was injected into rat dorsal raphe nucleus the tolerance for delayed reward decreased. This raises the possibility that a similar treatment in monkeys performing a more elaborate schedule task make it difficult for the monkeys to perform normally because the information about the step-by-step progress through the trials and schedules would be lost.

Dorsal raphe as a coordinating center for persistence in goal-seeking behaviors

Responses related to schedule onset, schedule progress, or reward proximity have also been observed in other brain areas. The ventral striatum, amygdala, and dopamine neurons in substantial nigra pars compacta signal whether the current trial is the first schedule state by the visual cue (Shidara et al., 1998; Sugase-Miyamoto and Richmond, 2005; Ravel and Richmond, 2006). Anterior cingulate cortex neurons show changes in response strength related to schedule progress (Shidara and Richmond, 2002; Toda et al., 2012). Although the signal is not constructed in the same way as in the cingulate, the information theoretic analysis of dorsal raphe neurons showed that neuronal responses included information about schedule progress and/or reward proximity in their temporal response profiles. The neurons in orbitofrontal cortex show responses depending on the preceding reward (Simmons and Richmond, 2008). Thus, none of these dorsal raphe neuronal signals are unique, although the richness of the combination might be.

Three sets of data about the dorsal raphe nucleus help in hypothesizing its role. First there are the widespread anatomical connections of the dorsal raphe nucleus throughout the forebrain including anterior cingulate cortex (Wilson and Molliver, 1991), orbitofrontal cortex (Cavada et al., 2000), and subcortically to ventral striatum (Lavoie and Parent, 1990), amygdala (Bauman and Amaral, 2005), and substantial nigra pars compacta (Gervais and Rouillard, 2000). Second there is the rich encoding of reward expectation described above and the information about ongoing progress to the outcome that we have shown here. Third, there is the experimental and clinical data suggesting the important role for the dorsal raphe nucleus in maintaining alertness and reward processing (Kirby et al., 2011; Monti, 2011). Considering our findings about the rich encoding of task progress and predicted reward outcome in light of these three lines of evidence, we suggest that dorsal raphe neurons might have an important role for coordinating or synchronizing activity for persistence and goal-seeking, especially in the forebrain.