Prospective Coding of Dorsal Raphe Reward Signals by the Orbitofrontal Cortex

Abstract

The orbitofrontal cortex (OFC) is important for the cognitive processes of learning and decision making. Previous recordings have revealed that OFC neurons encode predictions of reward outcomes. The OFC is interconnected with the dorsal raphe nucleus (DRN), which is a major serotonin (5-HT) center of the brain. Recent studies have provided increasing evidence that the DRN encodes reward signals. However, it remains unclear how the activity of DRN neurons affects the prospective reward coding of OFC neurons. By combining single-unit recordings from the OFC and optogenetic activation of the DRN in behaving mice, we found that DRN stimulation is sufficient to organize and modulate the anticipatory responses of OFC neurons. During pavlovian conditioning tasks for mice, odorant cues were associated with the delayed delivery of natural rewards of sucrose solution or DRN stimulation. After training, OFC neurons exhibited prospective responses to the sucrose solution. More importantly, the coupling of an odorant with delayed DRN stimulation resulted in tonic excitation or inhibition of OFC neurons during the delay period. The intensity of the prospective responses was affected by the frequency and duration of DRN stimulation. Additionally, DRN stimulation bidirectionally modulated the prospective responses to natural rewards. These experiments indicate that signals from the DRN are incorporated into the brain reward system to shape the cortical prospective coding of rewards.

Introduction

The orbitofrontal cortex (OFC) is involved in the learning and decision making (Bechara et al., 2000). Electrophysiological recordings from monkeys and rats have shown that OFC neurons encode the motivational significance of the expected reward (Schoenbaum et al., 1998; Tremblay and Schultz, 1999). Moreover, subpopulations of OFC neurons are exquisitely tuned to specific reward features such as magnitudes (Tremblay and Schultz, 1999), temporal delays (Roesch et al., 2006), spatial directions (Feierstein et al., 2006), and reward identities (Padoa-Schioppa and Assad, 2006, 2008). The OFC receives extensive inputs from limbic cortical and subcortical areas (Carmichael and Price, 1995; Ongür et al., 1998; Cavada et al., 2000), which suggests that these input areas have functional roles in the generation and modulation of the coding properties of OFC neurons.

The dorsal raphe nucleus (DRN) in the midbrain densely innervates the OFC and modulates its behavioral functions (Wilson and Molliver, 1991; Clarke et al., 2004). The DRN represents the major source of 5-HT in the forebrain (Vertes, 1991; Jacobs and Azmitia, 1992). Dysfunctions of 5-HT signaling are associated with disorders in emotion, cognition, and reward processing (Charney et al., 1987; Sellers et al., 1992; Graeff et al., 1996; Millan, 2000; Müller et al., 2007). A growing body of research has revealed that the activity of DRN neurons is related to reward processing. Recordings from primates and rodents have shown that some DRN neurons are sensitive to the expectations, sizes, and deliveries of rewards (Nakamura et al., 2008; Ranade and Mainen, 2009; Bromberg-Martin et al., 2010; Miyazaki et al., 2011a; Inaba et al., 2013; Liu et al., 2014). Imaging, microdialysis, and pharmacological studies suggest that 5-HT mediates certain aspects of reward (Phillips et al., 1976; Redgrave and Horrell, 1976; Schwartz et al., 1990; Lorrain et al., 1997; Tanaka et al., 2004; Miyazaki et al., 2011b). We recently reported (Liu et al., 2014) that mouse behaviors are strongly reinforced by optogenetic activation of DRN Pet-1 neurons, which are predominantly serotonergic. Another study (Miyazaki et al., 2014) reported that the optogenetic stimulation of DRN 5-HT neurons increases patience for future rewards. Although the DRN is implicated in reward processing, it remains unclear whether the output from DRN neurons is integrated in the OFC to modulate or even guide the coding of prospective rewards.

To examine the effects of DRN activation on the responses of OFC neurons, we combined multichannel single-unit recordings of OFC neurons with selective optogenetic stimulation of DRN Pet-1 neurons in behaving mice. In pavlovian conditioning tasks, the mice were presented with different odorant cues that were coupled to delayed DRN stimulation or sucrose solution delivery. Our experiments revealed that DRN stimulation produces reward signals that effectively guide the prospective responses of OFC neurons. Additionally, DRN stimulation can modulate the responses of OFC neurons to expected natural reward outcomes. These results suggest that the DRN broadcasts reward signals to the OFC and drives cortical remodeling to organize and modulate reward processing.

Materials and Methods

Mice.

The animal care and use followed the institutional guidelines of the National Institute of Biological Sciences (Beijing, People's Republic of China) and the governmental regulations of the People's Republic of China. All experiments were performed on adult (8–16 weeks of age) ePet1-Cre mice [Tg (Fev-cre) 1Esd, The Jackson Laboratory; Scott et al., 2005] of either sex. These mice expressed Cre recombinase under the control of the ePet1 promoter, which drives gene expression primarily in 5-HT neurons. The mice were housed at a temperature of 22–25°C on a reverse 12 h light/dark cycle (lights on at 9:00 P.M.) with a standard chow diet. Drinking water was freely accessible unless noted otherwise.

AAV vectors and animal surgery.

We used Cre-dependent adeno-associated virus (AAV) vectors with double-floxed inverted open reading frames carrying the ChR2-mCherry gene in the 3′-to-5′ direction (AAV-DIO-ChR2-mCherry). The vectors were packaged in AAV serotype 2/9 vectors, consisting of AAV2 ITR genomes pseudotyped with AAV9 serotype capsid proteins. The AAV vectors were replicated in HEK293 cells with the triple plasmid transfection system, purified by cesium chloride-density gradient centrifugation, and desalinated via dialysis against a physiological buffer. The final viral vector titers were >10⁹ particles/μl.

Before surgery, the mice were anesthetized with pentobarbital (80 mg/kg, i.p.). The mice were mounted in a stereotaxic holder and kept warm (37°C) with a heating pad. A small craniotomy was made for injections of the AAV vectors (1 μl of AAV-DIO-ChR2-mCherry or AAV-DIO-mCherry) into the DRN and the later placement of a guide cannula for insertion of the optical fiber (Liu et al., 2014). A custom-made titanium head plate was then affixed on the skulls of the ePet1-Cre mice with M1 stainless steel screws and dental cement. The mice were maintained in individual cages for 2–3 weeks for recovery and viral transgene expression.

Behavioral training.

The mice were head restrained with two horizontal bars and were able to maneuver on an air-supported free-floating Styrofoam ball (Zhan and Luo, 2010). The head-fixed mice were trained to discriminate different odorants with the delayed pavlovian conditioning paradigm. Three odorants, n-amyl acetate, carvone, and ethanol (1% saturated vapor, 2 L/min, hereafter termed Odor 1, Odor 2, and Odor 3, respectively) were presented for 1 s in pseudorandom order with an intertrial interval of at least 12 s. Each odorant was presented at least 30 times in each recording session. A 2 s delay was inserted between the odorant offset and the outcome onset. The outcomes for Odor 1 and Odor 2 included a 10% sucrose solution (0.5 s) and the delivery of 15 ms light pulses of various durations and frequencies into the DRN (2 s at 5 Hz, 2 s at 20 Hz, 1 s at 20 Hz, or 0.5 s at 20 Hz). Odor 3 led to no outcome and was used as the control of a negatively conditioned stimulus. The mice were water deprived for 36 h before the training and recordings if the delivery of the sucrose solution was coupled to an odorant.

In vivo electrophysiology.

Extracellular spiking signals were detected with a 16-channel probe (NeuroNexus) and amplified (1000×) with a custom-made 16-channel amplifier with a built-in bandpass filter (0.5–3.6 kHz). One channel that did not exhibit spike signals was selected as a reference ground to reduce movement artifacts. Analog signals were digitized at 25 kHz and sampled with a Power1401 digitizer and Spike2 software (CED). At the end of the recordings, we created electrolytic lesions via DC current injection through two of the electrodes (15–20 s; 100 μA). The animals were then deeply anesthetized with an overdose of pentobarbital and perfused with 4% formaldehyde. The mouse brains were cut into 50 μm coronal sections to verify the recording sites.

Data analysis.

The spikes were sorted off-line with the Spike2 program. Single units were isolated using principal component analysis (PCA) of the spike waveforms that had signal-to-noise ratios of at least 4:1. The spikes were aligned to the odor onset, and the trial initiation signal occurred at −1.5 s. We plotted the receiver operating characteristic (ROC) curve and calculated the area under the curve (AUC) for the neural activity throughout each trial by comparing the spike firing rates of a 200 ms test window (50 ms advance step) to those in a control time window (200 ms) that occurred 200 ms before the trial onset (−2 to −1.8 s). ROC values >0.5 indicate neural excitation, and values <0.5 indicate inhibition. Contrasts of the ROC values between odorants were calculated by comparing the spike numbers during the same time windows (200 ms width, 50 ms advance step) throughout the entire trial. ROC values of 1 indicate complete selectivity for the first odor stimulus, and ROC values of 0 indicate complete selectivity for the second odor stimulus. Permutation tests with 1000 permutations were used to determine the statistical significance of the response strength and selectivity. Kolmogorov–Smirnov tests were performed to examine the significant differences between the cumulative probability distributions.

Results

Mouse OFC neurons exhibit prospective coding for natural rewards

Previous recordings from behaving monkeys and rats have revealed sustained responses of OFC neurons to the delayed rewards and suggested that OFC neurons encode the motivational significance of expected outcomes (Schoenbaum et al., 1998; Wallis, 2007). To take advantage of optogenetic tools that are available for mice, we sought to test the generality of the prospective coding of mouse OFC neurons. Multichannel extracellular single-unit recordings were collected from mice while they were engaged in a delayed pavlovian cue–reward association task (Fig. 1A). The mice were head restrained and were allowed to behave on an air-supported spherical treadmill. The mice were presented with three odorant cues (1 s duration) that predicted a sucrose solution reward [Odors 1 and 2, positively conditioned stimulus (CS+)] or no reward [Odor 3, negatively conditioned stimulus (CS−)]. A brief click sound 1.5 s before the delivery of the odorants was used as a trial start signal. A delay time (2 s) was inserted between the odorant cues and the sucrose solution (0.5 s valve on). Reliable anticipatory licking in response to the CS+ but not the CS− odorant was taken to indicate successful pavlovian conditioning (Fig. 1A). A 16-channel electrode array was targeted at the OFC of the head-restrained mice that were performing the task, and single units were isolated using the standard PCA method (Fig. 1B).

Figure 1.

Neurons in the OFC encode reward expectation in behaving mice. A, After pavlovian conditioning, mice exhibited vigorous anticipatory licking responses to the CS+ odorants but not to the CS− odorant (n = 42 sessions from three mice). The same amount of sucrose solution (10%; 0.5 s valve on) was delivered to animals 3 s after the onset of Odors 1 and 2 (CS+, 1 s duration) but not after the onset of Odor 3 (CS−). The shaded traces indicate the SEM. B, The spiking activity of OFC neurons was recorded with 16-channel silicon probes in behaving mice (left). OFC neurons that were simultaneously recorded at a single site were isolated using PCA (right). The shadowed traces represent the averaged spike waveforms with SDs. C, D, Representative responses of two OFC neurons during the odor–sucrose association task; these neurons exhibited tonic activation (C) or inhibition (D) following the sucrose-predicting odors. The color bars above the raster plots indicate the timing of the actions. Raster plots and peristimulus time histograms (smoothed with a Gaussian kernel of σ = 100 ms) showing similar excitations (C) and inhibitions (D) of OFC neurons in response to the two CS+ odorants and much weaker responses to the CS− odorant. The shaded area indicates SEM. E, Distribution of the response strengths during the 2 s delay period following the three different odorant cues (n = 241 cells). The spiking activity of each neuron during the delay period (1–3 s) was compared with that in a control period (−4 to −2 s) and is expressed as the ROC AUC. AUC values range from 0 to 1, with 0 indicating inhibition, 0.5 indicating no response, and 1 indicating strong excitation. The red and green bars show the numbers of neurons with significantly higher (red) or lower (green) activity compared with the control period (permutation test, p < 0.01). F, Comparisons of the neuronal AUCs between each of the three odorant cues. The colored dots indicate significant activity differences during the delay period between the odorant pairs (permutation test, p < 0.01). G, Histograms showing the distributions of odor selectivity during the delay period between each pair of the three odorant trials. The colored bars represent the number of cells that exhibited significant response selectivity (permutation test, p < 0.01).

A substantial number of OFC neurons exhibited a strong activity change during the delay period before reward delivery. Typically, we observed stronger and longer-lasting excitations following odorant pulses that predicted the sucrose solution (Fig. 1C). In some cases, the OFC neurons were inhibited following odorant cues, but such inhibitions were stronger when the cues were coupled to reward delivery (Fig. 1D). We analyzed the response strength of each neuron by constructing ROC curves from the spike firing rates during the delay and control periods. We then measured ROC AUCs [AUC = 0.5 (indicates no response)] and performed permutation tests to assess the statistical significance of the responses. Of the 241 OFC cells that were tested, 158 neurons (66%) exhibited significant changes during CS+ trials (p < 0.01, permutation tests); of these neurons, 127 exhibited significant activation, and 31 exhibited significant inhibition (Fig. 1E, two left panels). In contrast, only 42 neurons (17%) exhibited significant responses during the delay periods of the CS− trials (Fig. 1E, right). The scatterplots in Figure 1F show that only weak correlations existed between the strengths of the responses to the CS+ and CS− cues. In contrast, the strengths of the responses to the two CS+ cues were highly correlated (Fig. 1F, right).

To further evaluate response selectivity, we plotted ROC curves based on the spike firing rates of the trials with different cues and calculated the ROC values using the AUCs (Fig. 1G). These analyses revealed that >40% of OFC neurons responded selectively to reward-predicting cues (higher firing rates: 37% and 36% for Odor 1 vs Odor 3 and Odor 2 vs Odor 3, respectively; lower firing rates: 7% for both of the CS+ vs the CS− comparisons). Consistent with earlier reports of a lack of cue selectivity in the OFCs of monkeys and rats (Tremblay and Schultz, 1999; Feierstein et al., 2006), only a very small fraction of neurons exhibited significantly different responses to the two CS+ cues (8% higher and 7% lower firing rates for Odor 1 vs Odor 2; Fig. 1G, right). Thus, the mouse OFC neurons were strongly modulated by reward prediction, which suggests that mice are suitable for investigating the prospective coding of reward signals in the OFC.

Modulation of spontaneous activity in the OFC elicited by the optogenetic stimulation of DRN Pet-1 neurons

We tested whether the optogenetic stimulation of DRN neurons could directly affect the spontaneous activity of OFC neurons. Two weeks after the infusion of the AAV-DIO-ChR2-mCherry viral vectors into the DRNs of the ePet1-Cre mice, we observed numerous mCherry-expressing neuropils in the DRN. Our recent study (Liu et al., 2014) demonstrated that this approach labels the vast majority of 5-HT neurons. Consistent with previous findings that DRN neurons project to the OFC (Roberts, 2011), we observed mCherry-expressing terminals in the OFC (Fig. 2A). An optical fiber was placed above the DRN for light stimulation, and an array of 16 electrodes was used to record the spiking activity in the OFCs of head-fixed behaving mice.

Figure 2.

Optogenetic activation of Pet-1 neurons in the DRN modulates the spiking activity of some OFC neurons. A, Schematic illustration of the stimulation and recording sites. ChR2-mCherry was expressed specifically in the DRN Pet-1 neurons of ePet1-Cre mice. The images below show the mCherry signals that were detected in cells in the DRN and the axonal fibers within the OFC. Scale bars, 50 μm. B, C, Raster plots and peristimulus time histograms showing the effects of DRN stimulation on the firing activities of two OFC neurons. One cell was activated (B), and the other was inhibited (C) by the delivery of light pulses into the DRN at a frequency of 5 or 20 Hz for 2 s. D, Percentage of OFC neurons that exhibited significant activity changes due to light stimulation of the DRN Pet-1 neurons. E, Raster plot of the modulatory effects of light stimulation at 5 vs 20 Hz (n = 136 neurons). The blue dots indicate the neurons that exhibited significant changes in firing activity in response to light stimulation (permutation test, p < 0.01). F, Effects on spiking activity for the neurons that exhibited significant changes in response to DRN stimulation at 5 or 20 Hz.

DRN stimulations at a frequency of 5 or 20 Hz (15 ms pulses for 2 s) induced rapid increases in firing rates in some of the OFC neurons (Fig. 2B). Some other cells exhibited more long-lasting inhibitory responses (Fig. 2C). Of the 136 OFC neurons tested, 43 (26%) and 56 neurons (31%) were directly activated by laser stimulation at 5 and 20 Hz, respectively (Fig. 2D). A smaller fraction of neurons (6% and 10% of cells) were inhibited by DRN light stimulation at these two frequencies (Fig. 2D). Stimulation at 20 Hz often produced stronger responses than those observed following 5 Hz stimulation (Fig. 2E,F). These results thus demonstrated a functional connection between the DRN and the OFC.

Stimulation of the DRN Pet-1 neurons was sufficient to condition the prospective activity OFC neurons

In light of our recent findings that the activation of DRN Pet-1 neurons strongly reinforces animal behaviors (Liu et al., 2014), we asked whether optogenetic stimulation of these neurons could condition the OFC neurons to generate prospective responses. The pavlovian conditioning task was modified by coupling an odorant cue (Odor 1) with delayed light stimulation of the DRN Pet-1 neurons (15 ms at 20 Hz for 2 s) and without the delivery of the sucrose solution (Fig. 3). Odor 3 was not associated with any outcome and served as a CS− control. Multichannel recordings from the OFC were collected after the mice were extensively trained (400 trials in 2 d).

Figure 3.

The activity of OFC neurons can be conditioned by optogenetic stimulation of DRN Pet-1 neurons. A, ePet1-DRN^ChR2 mice were trained by coupling Odor 1 but not Odor 2 with delayed light stimulation of DRN Pet-1 neurons at 20 Hz for 2 s. OFC neurons were recorded after training animals for 400 trials over 2 d. The raster plot and peristimulus time histogram illustrate one OFC neuron that exhibited much higher activity during the delay period following the CS+ odor. B, Another OFC neuron showed strong inhibition following the delivery of the CS+ odor. C, One OFC neuron from a control ePet1-DRN^mCherry mouse did not show response in either the CS+ or CS− trial.

A total of 868 OFC cells were recorded during the optogenetic pavlovian conditioning task from the ePet1 mice with AAV-DIO-ChR2-mCherry injections into the DRN (termed ePet1-DRN^ChR2 for simplicity). Many OFC neurons exhibited strong responses following the CS+ odorant cue that predicted the DRN stimulation. The responses included both strong excitations and inhibitions during the delay period (Fig. 3A,B). Some OFC neurons also responded to the CS− cue, but these responses tended to be smaller and of shorter duration. As a control of light delivery, we recorded 84 OFC neurons from the ePet1 mice that were injected with AAV-DIO-mCherry virus (termed ePet1-DRN^mCherry mice), and underwent exactly the same training and recording procedures. Figure 3C shows one representative neuron that lacked any responses in the ePet1-DRN^mCherry mice.

Figure 4A shows a color map representation of the population neuronal response strengths (Fig. 4A, two left panels) and the response selectivity (Fig. 4A, right panel) of ePet1-DRN^ChR2 mice. These neurons were classified into three types based on their activity patterns during the delay period. Approximately half of the neurons (n = 415/868 cells; Type 1) exhibited significant tonic excitation in anticipation of the DRN stimulation and weak or absent responses following the CS− cue. Approximately 30% of neurons (261 cells; Type 2) lacked significant response during the delay time of either the CS+ or CS− trials. Type 3 neurons consisted of the remaining 192 cells (22%), which were significantly inhibited during the delay time of the CS+ trials. Independent of cell type, brief excitations were often observed at the onset of both the CS+ and CS− cues. The lack of response selectivity during this early period was probably due to response generalization from the CS+ cues to the CS− ones. A similar phenomenon has been observed in the activity of dopamine neurons (Waelti et al., 2001). In both Type 1 and Type 3 neurons, the activity differences between the CS+ and CS− trials reached a maximum at the onset of the DRN stimulation (Fig. 4B). None of the 84 cells from ePet1-DRN^mCherry mice showed significant responses or response selectivity (Fig. 4C,D), demonstrating the requirement of effective activation of DRN Pet-1 neurons for conditioning OFC neurons.

Figure 4.

The activity patterns of the OFC neuronal population in mice that were conditioned with DRN stimulation. A, Heat maps showing the ROC representation of the peristimulus time histogram data from the OFC neurons that were tested with delayed light coupled to odor delivery (n = 868). Each row represents the firing activity of a single OFC neuron. The left and middle panels are the group activity of the OFC neurons in the CS+ and CS− trials aligned to the onset of odorant cues, respectively. The spiking frequency in each time bin (sliding window, 200 ms; step, 50 ms) was compared with that during the control time window (−2 to −1.8 s, 200 ms) to calculate AUC values, which are represented with colors that range from dark blue to dark red. The OFC neuronal activities (right) in the CS+ and CS− trials were compared by calculating the firing rates of every single neuron in each corresponding time bin. An AUC value of 0.5 indicates no selectivity. The color maps are segmented with dotted lines that classify all of the recorded neurons into three types based on response strength during the delay time period in the CS+ trials. The neurons of Types 1, 2, and 3 correspond to the cells that exhibited significant excitations, no response, and significant inhibitions, respectively (permutation test, p < 0.01; n = 868). B, Averaged response strengths of the three types of OFC neurons to the CS+ odors and CS− odors, and the contrast of the CS+ and CS− stimuli. C, Group data of the OFC neurons recorded from the control ePet1-DRN^mCherry mice. D, Averaged response strengths of the OFC neurons recorded from the control mice.

We further calculated the neuronal response strengths and the response selectivities to CS+ and CS− cues during the odorant cue, delay, and DRN stimulation phases (Fig. 5A,B). Immediately after the onset of the odorant pulses, both the CS+ and CS− cues activated a substantial number of neurons (421 cells during the CS+ and 355 cells during the CS−). A smaller number of cells were inhibited during this early phase (96 cells during the CS+ and 94 cells during the CS−). During the delay period, the numbers of CS+ excitatory cells remained relatively stable, but the numbers of CS+ inhibitory cells increased, and the numbers of CS− excitatory cells decreased (Fig. 5C). During the phase of DRN stimulation, the number of CS+ excitatory neurons decreased, whereas that of CS+ inhibitory neurons increased. Nearly two-thirds of the OFC neurons predicted the outcome during the delay period (∼46% CS+ selective and 20% CS− selective; Fig. 5D). Thus, similar to the delivery of the sucrose solution, light stimulation of the DRN Pet-1 neurons effectively guided the formation of prospective coding in the OFC neurons.

Figure 5.

Analysis of the response selectivity of OFC neurons during the different task phases. A, Scatterplots comparing the response strengths of the OFC neurons during the odor, delay, and outcome phases in the CS+ trials and CS− trials. Response strength was determined as the ROC value (AUC) that resulted from comparisons between the firing rates during each period (odor, 0–1 s; delay, 1–3 s; outcome, 3–5 s) and those during the control period (−4 to −2 s). The blue and gold dots represent the neurons with a significant firing preference for the CS+ and CS− trials, respectively. The bar graphs show the distributions of the neuronal activities of the OFC neurons in the CS+ and CS− trials. The red and green bars plot the numbers of neurons with significant activation and inhibition responses, respectively. B, The numbers of cell that exhibited different response selectivity during the CS+ or CS− trials. The neurons that exhibited significant selectivity for the CS+ are shown in blue, and those that exhibited selectivity for the CS− are shown in gold. C, Numbers of OFC neurons that exhibited significant response selectivity for the CS+ or CS− stimulus during the different task phases. D, Percentages of OFC neurons that were significantly recruited during the different task phases in the CS+ and CS− trials.

The effects of stimulation strength on the prospective coding of OFC neurons

Primate OFC neurons respond differentially to the expectations of natural rewards of different magnitudes (Tremblay and Schultz, 1999). We sought to test whether the activities of mouse OFC neurons differentially encoded predictions of DRN stimulations of varying intensities. Using the optogenetic pavlovian conditioning task, we associated two odorant cues (Odor 1 and Odor 2) with delayed light stimulations of various frequencies and durations. Odor 1 was always coupled to the strong stimulation of 20 Hz light pulses for 2 s, and Odor 2 was coupled to a weaker stimulation (i.e., 5 Hz for 2 s, 20 Hz for 0.5 s, or 20 Hz for 1 s; n = 149, 110, and 128 OFC neurons, respectively, for the three outcome pairs; Fig. 6A).

Figure 6.

The effects of DRN stimulation intensity on the conditioning of OFC neurons. A, Distributions of OFC neuronal selectivity for each pair of odorant cues that were followed by delayed DRN stimulation with different parameters. Selectivity was calculated as a ROC value (AUC) by comparing the neuronal firing rates during the delay period (1–3 s) to those during the control time window (−4 to −2 s). B, Raster plots and peristimulus time histograms showing three representative OFC neurons that exhibited stronger excitatory responses to the anticipation of DRN stimulations at higher intensities or longer durations (20 vs 5 Hz, 2 s durations; 2 vs 0.5 s at 20 Hz; 2 or 1 s at 20 Hz). The shaded areas indicate the SEMs. C, Three representative neurons that exhibited stronger inhibitory responses to the anticipation of more intense DRN stimulation. D, Bar plots showing the number of neurons that exhibited significant selectivity to either the CS+ (red) or CS− (green) stimulus.

Scatterplots revealed that the response profiles were significantly correlated with the different pairs of stimulation parameters (p < 0.0005, Pearson's correlation; Fig. 6A). The prospective response strengths to the strong DRN stimulations were larger than those to the weak stimulations (Fig. 6A, slopes <1). Further analysis revealed that many neurons exhibited differential firing activities during the delay period leading to the DRN stimulations of different intensities. Typically, neurons with excitatory responses became more activated following Odor 1, which was always associated with the strong DRN stimulation (Fig. 6B). Similarly, the neurons with inhibitory response exhibited lower spiking activities following Odor 1 (Fig. 6C). Overall, ∼40% of the neurons responded selectively to either the 20 or 5 Hz stimulation (Fig. 6D, left). Approximately half of neurons exhibited significantly discrepant responses to light stimulations of 2 and 0.5 s (Fig. 6D, middle), whereas ∼20% of neurons responded differentially to the DRN stimulations of 2 and 1 s (Fig. 6D, right). Thus, the subpopulations of OFC neurons exhibited selective responses that predicted the strength of the DRN stimulation, which suggests that reward magnitudes could be mimicked by adjusting the intensities of optogenetic stimulation of the DRN Pet-1 neurons.

Comparison of the natural rewards and DRN stimulations

Because the OFC neurons exhibited prospective coding for both the artificial reward of DRN stimulation and the natural reward of sucrose solution, we compared the response patterns of these neurons during the delay periods that led to two different rewards. In a pavlovian conditioning task, Odor 1 was coupled to the delayed light stimulation of the DRN neurons (20 Hz for 2 s), and Odor 2 was coupled to the delayed delivery of a sucrose solution.

Of the 288 neurons tested, 251 exhibited significant responses to at least one conditioned odorant. Approximately 36% of the cells (n = 105 of 288 cells) exhibited similar activations during the delay periods following both of the odorant cues (Fig. 7A), whereas ∼16% of cells (n = 46) responded with similar inhibitions (Fig. 7B). Interestingly, ∼23% of cells exhibited specific activations that followed only one cue. Figure 7C shows one laser-specific cell (n = 38), and Figure 7D shows one sucrose-specific cell (n = 28). Additionally, ∼10% of cells exhibited opposite response patterns (i.e., 23 cells, ∼8%) were activated before delivery of the sucrose solution and were inhibited before the laser stimulation (Fig. 7E), and 7 cells were excited before the laser and inhibited before delivery of the sucrose solution (Fig. 7F).

Figure 7.

Comparisons of prospective codes for natural rewards and DRN stimulation. A, B, The activity patterns of two OFC neurons that exhibited coherent activation (A) or inhibition (B) following the application of two odorants, one of which predicted DRN light stimulation and the other predicted sucrose solution. C, D, Two example cells that exhibited selective responses to the anticipation of light stimulation (C) or sucrose (D). E, F, Two cells that exhibited opposite responses to the outcomes of light stimulation and sucrose solution. G, Mouse licking responses to the two odorants that were coupled to the delivery of sucrose solution or DRN stimulation (n = 36 sessions from three mice). H, Scatterplots of the OFC neuronal responses to the different pairs of reward outcomes (n = 288 neurons). Regression analysis revealed that the OFC neurons responded similarly to the expectation of DRN stimulation of different durations or frequencies. However, some OFC neurons exhibited greater differences in their responses to the anticipation of DRN stimulation than the anticipation of sucrose solution delivery. I, Cumulative probability distribution (left) and bar plots (right) of the OFC neuronal response variance for different pairs of reward outcomes. Variance was calculated as the distance of each dot to its corresponding regression line as shown in G. n.s., Not significant. *p < 0.05; ***p < 0.001, Kolmogorov–Smirnov tests. Suc, Sucrose.

In this test paradigm, mice exhibited vigorous licking responses to the odorant cue predicting sucrose solution delivery but not to the cue predicting DRN light stimulation (Fig. 7G). Despite this drastic difference in behavioral responses, the predictive physiological responses of the OFC neurons for DRN stimulation and sucrose solution delivery were significantly correlated (p < 0.0001, Pearson's correlation; Fig. 7H). The slope of 0.58 indicates that, overall, the laser stimulation of DRN neurons evoked stronger prospective responses than did the sucrose solution. Nevertheless, the response variability between these two different reward signals appeared to be larger than the variabilities between the DRN light stimulations with different parameters (Figs. 6A, 7I). To quantify this variability, we computed the distance of each neuron to the regression line in the scatterplots to acquire a variance value for each neuron. The cumulative distributions of the variances revealed significant differences between the “laser versus sucrose” task and the other three “laser versus laser” tasks (i.e., 20 vs 5 Hz, 2 vs 1 s, and 2 vs 0.5 s; ***p < 10⁻⁷ for all comparisons between the laser vs sucrose and the laser vs laser tasks, Kolmogorov–Smirnov test; Fig. 7I, inset).

Bidirectional modulation of OFC reward coding by DRN stimulation

To test how DRN activation modulated the anticipatory responses of OFC neurons for natural rewards, we examined the effects of combining optogenetic stimulation of the DRN Pet1 neurons with the sucrose solution delivery on the coding properties of the OFC neurons. In a pavlovian conditioning task, Odor 1 was coupled to the delayed delivery of both sucrose solution and DRN stimulation (20 Hz for 2 s), and Odor 2 was coupled to the sucrose solution only.

Of the 245 OFC cells tested, 134 cells (∼55%) exhibited differential responses during the delay periods following the two odorant cues. For 44 cells (∼18%), we found that the addition of DRN stimulation amplified the excitatory responses that predicted the delivery of sucrose solution (Fig. 8A). In ∼14% cells, the inhibitory responses were enhanced (Fig. 8B). In contrast, DRN stimulation attenuated the excitatory or inhibitory responses that preceded the sucrose solution of ∼23% of cells (14 and 42 cells, respectively; Fig. 8C,D).

Figure 8.

DRN stimulation bidirectionally modulates the responses of OFC neurons to the expectation of natural reward. A–D, Raster plots and peristimulus time histograms of four representative OFC neurons that were modulated by optogenetic stimulation of the DRN Pet-1 neurons during the period of sucrose solution delivery. Odor 1 was coupled to sucrose solution and laser stimulation of the DRN Pet-1 neurons at 20 Hz for 2 s, whereas Odor 2 was coupled to sucrose solution alone. A, B, The examples show that DRN stimulation could potentiate excitatory responses (A) and inhibitory responses (B) to sucrose delivery. C, D, We also observed attenuation effects of DRN stimulation on excitatory responses (C) and inhibitory responses (D) to the expectation of sucrose solution delivery. E, Scatterplot showing the strengths of the OFC responses to the two conditioned odorants (n = 245 neurons). The AUC values were calculated by comparing the differences between the firing rates of the OFC neurons during the delay time window. The colored dots represent neurons with significant selectivity for the CS+ trials (red) and the CS− trials (permutation test, p < 0.01). F, The bar graph showing the distribution of OFC neuronal selectivity. G, Cumulative probability of the variances of the OFC neuronal responses to the two conditioned odorants. Variance was determined by measuring the Euclidean distance of each point from the nonselective line. The blue line was calculated from the data shown in E, and the black line was calculated from the control data shown in Figure 1E. The mean variances of both cases are plotted in the right panel, and statistical significance was determined with a Kolmogorov–Smirnov test (p = 7.4 × 10⁻⁷). H, Mouse licking responses during the recording sessions (n = 43 sessions from four mice).

Compared with the control experiment that coupled two different odorants with the same sucrose solution outcome (Fig. 1E, right), the addition of DRN stimulation to one outcome increased the variability between the responses to the two odorant cues (Fig. 8E). When the two different odorant cues were coupled to the same sucrose delivery outcome, the majority of neurons (85%) exhibited similar prospective responses regardless of odorant identity (Fig. 1). The addition of the DRN stimulation reduced the ratio of similar responses to 45% (Fig. 8F). We calculated the distance of each neuron from the nonselective line (Figs. 1E, right, 8E) and used this variance to quantify the modulatory effect of the DRN stimulation. The addition of the DRN stimulation significantly increased the variance of the prospective responses that followed the two odorant cues (Fig. 8G). Consistent with the modulation of physiological responses, the addition of DRN stimulation reduced licking intensity at the animal behavioral level (Fig. 8H).

Discussion

In this study, we investigated the effects of DRN activation on OFC neurons by combining optogenetic stimulation and single-unit recordings in behaving mice. Multichannel recordings were performed in head-restrained mice undergoing pavlovian cue–reward association tasks. The majority of the OFC neurons altered their spike firing rates during the anticipation of sucrose solution delivery and DRN stimulation. The responses of many OFC neurons were tuned to specific reward features, such as DRN stimulations of specific frequencies and durations. Furthermore, the addition of DRN stimulation modulated the neuronal responses to natural rewards. These results support the concept that DRN activation produces reward signals that can effectively organize and modulate reward processing in the OFC.

Our recordings confirmed that OFC neurons in mice, similar to those in primates and other rodents, exhibit tonic responses during waiting periods preceding natural rewards (Schoenbaum et al., 1998; Tremblay and Schultz, 1999). Consistent with previous reports (Feierstein et al., 2006), the mouse OFC neurons did not encode the identity of odorants but rather encoded the reward outcomes predicted by the odorant cues. Resembling the natural sucrose solution rewards, optogenetic stimulation effectively guided the prospective responses of the OFC neurons (Figs. 3, 4, 5). Similar to the magnitude effects of natural rewards, the frequencies and durations of the DRN light stimulations affected the response strengths of the OFC neurons (Fig. 6). More importantly, direct comparison revealed a significant positive correlation between the prospective responses to the sucrose solution and those to DRN stimulation (Fig. 7). In combination with our recent report that DRN stimulation reinforces animal behaviors and neuronal activity in the motor cortex (Liu et al., 2014), our observations from this study further support the notion that activating DRN neurons produces reward signals. Our recent behavioral assays revealed that light stimulation of DRN Pet-1 neurons competes favorably against sucrose solutions in two-bottle preference tests (Liu et al., 2014). Consistent with this observation, our recordings demonstrated that OFC neurons respond more strongly to the anticipation of DRN stimulation than to the anticipation of sucrose solution (Fig. 7). Thus, DRN activation can provide powerful artificial reward signals that guide neuronal responses in the OFC.

Our data do not necessarily exclude the possibility that the activation of DRN 5-HT neurons increased the animals' patience for future rewards (Miyazaki et al., 2014). Similar to OFC neurons, many DRN Pet-1 neurons also exhibit tonic activation during the delay period following an odorant cue that predicts a delayed delivery of a natural reward (Liu et al., 2014), which is suggestive of prospective coding of the expectation of reward outcome. However, the maximal activation of DRN neurons occurs following reward consumption, which implies that direct reward signaling underlies the behavioral reinforcement effect of the stimulation of DRN Pet-1 neurons (Liu et al., 2014). In addition to the connection from the DRN to the OFC, strong projections from the OFC back to the DRN also exist (Ogawa et al., 2014; Pollak Dorocic et al., 2014; Weissbourd et al., 2014). These strong interconnections suggest that the OFC and the DRN might function together to respond to cues that predict reward outcome to optimize behavioral responses during the process of decision making.

Our experiments suggest that OFC neurons distinguished the identities of the different reward signals. The responses of some OFC neurons were tuned to the outcomes of DRN stimulations at different intensities. Moreover, some of the neurons responded selectively to the outcomes of either the artificial reward of DRN stimulation or the natural reward of the sucrose solution. The addition of optogenetic activation of the DRN Pet-1 neurons bidirectionally modulated the responses of many OFC neurons to the natural rewards. These results suggest that DRN stimulations and natural rewards are encoded by overlapping but different subsets of OFC neurons. Moreover, individual OFC neurons were tuned to predict rewards with different properties. Our results are consistent with recent recordings from nonhuman primates demonstrating that individual OFC neurons are sensitive to specific reward variables, such as behavioral confidence (Kepecs et al., 2008), temporal discounting (Roesch et al., 2006), response requirements (Feierstein et al., 2006; Furuyashiki et al., 2008; Tsujimoto et al., 2009), relative reward outcomes (Tremblay and Schultz, 1999; Kobayashi et al., 2010), and specific sensory features of the reward such as taste (Padoa-Schioppa and Assad, 2006, 2008). Similarly, lesion studies have also shown that the OFC is necessary for learning about changes in specific rewards (Burke et al., 2008; McDannald et al., 2011). Several recent studies (Jones et al., 2012; McDannald et al., 2014; Stalnaker et al., 2014) reported that the activity of OFC neurons predicts value-neutral changes in rewards, suggesting a probable coding of valueless saliency by the OFC. Since light delivery into the DRN of ePet1-DRN^mCherry mice lacked the capability of conditioning the prospective responses of OFC neurons, it is unlikely that the potentially salient stimuli of laser delivery alone guides the OFC anticipatory responses to the optogenetic stimulation of DRN Pet-1 neurons in ePet1-DRN^ChR2 mice. Some human imaging studies have hypothesized that the OFC tracks subjective values in a common neural currency (Chib et al., 2009; FitzGerald et al., 2009; Smith et al., 2010; Kim et al., 2011; Levy and Glimcher, 2011, 2012; Lin et al., 2012). A substantial number of OFC neurons responded to both artificial and natural rewards in our experiments, which suggests that the OFC can also transform rewards with various properties into a common neural currency to mediate reinforcement learning (Schultz et al., 1997; Glimcher, 2011).

Our recordings provide evidence that DRN activation produces reinforcement signals that condition and modulate prospective responses to reward outcomes in the OFC and are suggestive of the importance of DRN neurons in reward processing. Further studies are required to dissect the neural circuits and neurotransmitters that underlie information exchange between the DRN and the OFC in greater detail. In addition to the OFC, the DRN also sends extensive axonal terminals to many brain areas that are known be associated with reward processing, such as the ventral tegmental area (VTA), the nucleus accumbens, and the lateral hypothalamus (Lewis, 1990; Wilson and Molliver, 1991). Reward signals from the DRN might shape the prospective responses of OFC neurons through the direct projection of the DRN to the OFC or through indirect connections via relays such as the VTA and other brain areas that projected to the OFC. Addressing the functions of the different relay circuits will require precise inactivation of different components along the neural pathways with optogenetic or chemogenetic inhibition methods (Armbruster et al., 2007; Gradinaru et al., 2008; Berndt et al., 2014). DRN neurons can also be inactivated to examine their contribution to the guidance of the prospective coding of OFC neurons for natural rewards in behaving states.

The DRN is the major brain 5-HT center, and neurons in the prefrontal cortex express several 5-HT receptors (Puig and Gulledge, 2011). Pharmacological studies in rodents and primates revealed important roles of 5-HT in the regulation of OFC functions such as reversal learning, response inhibition, devaluation, and aggressive behaviors (Clarke et al., 2004, 2007; De Almeida et al., 2006; Walker et al., 2006; Boulougouris and Robbins, 2010; Man et al., 2010; West et al., 2013). Although these previous studies emphasized the functions of the 5-HT signaling within the OFC, DRN Pet-1 neurons release 5-HT and glutamate, and both of these neurotransmitters contribute to reward signaling (Liu et al., 2014). In addition, two recent reports (McDevitt et al., 2014; Qi et al., 2014) focused on the role of glutamate transmission from DRN neurons in reward signaling. In the present study, we did not differentiate the possible distinct functions of the two neurotransmitters by separately manipulating their release. It will be interesting to use various mutant mouse lines to examine the precise contributions of 5-HT and glutamate from DRN neurons in the conditioning of the reward responses of OFC neurons.

Similar to other cortical areas, the OFC consists of many different cell types. We observed both excitatory and inhibitory responses to expected reward outcomes and classified the neurons into Type 1 and Type 3. Although the distributions of the basal firing rates and spike half-widths of these two types of neurons were significantly different, more rigorous ROC analyses did not support the use of these two physiological properties to classify cell types (data not shown). The optical tagging method has been used to facilitate the classification of recorded cells in several brain areas (Gradinaru et al., 2007; Lima et al., 2009; Cardin et al., 2010; Anikeeva et al., 2012; Cohen et al., 2012; Jennings et al., 2013; Kim et al., 2013; Kvitsiani et al., 2013; Liu et al., 2014). The exploitation of optical tagging to correlate physiological response patterns with specific anatomical cell types will be valuable for future studies.