Prefrontal Cortex Mediates Extinction of Responding by Two Distinct Neural Mechanisms in Accumbens Shell

Abstract

Suppression of ill-timed or competing actions optimizes goal-directed behaviors. Diminished inhibitory control over such actions is a central feature of such disorders as impulsivity, obesity, and drug addiction. The ventromedial prefrontal cortex (vmPFC) is involved in suppression of unreinforced actions. Using reversible inactivation in rats, we demonstrate that vmPFC activity is also required for inhibition of unreinforced actions extinguished during learning of a cued appetitive task and that behavioral disinhibition following vmPFC inactivation depends on dopamine signaling in nucleus accumbens shell (NAcS). Combining electrophysiological recording in NAcS with vmPFC inactivation in rats reveals two neural mechanisms by which vmPFC inhibits unreinforced actions. The first is by suppressing phasic excitations that promote behavioral cue responding. The second is by increasing the basal firing of NAcS neurons that tonically inhibit reward seeking. These results identify the vmPFC and the NAcS as critical elements of the circuits relevant to suppression of inappropriate actions.

Introduction

In animals with flexible behavioral repertoires, optimal action selection is critical for inclusive fitness. In adapting to novel environments, the following two major classes of behavioral changes are required: (1) increasing the probability of actions that promote positive outcomes; and (2) reducing the probability of interfering or unproductive actions. For animals with large repertoires of potential actions, such as mammals, more behaviors must be inhibited than are promoted as the animals learn to perform tasks. The present study addresses the question of how these unreinforced actions are inhibited after animals have learned an appetitive task.

A growing body of evidence specifically implicates the prefrontal cortex (PFC) in suppression of unproductive or ill-timed actions. Over the course of mammalian evolution, the PFC has undergone a marked increase in size and functionality in species with increasingly complex behavioral repertoires (Fuster, 1997). Furthermore, impaired PFC function is associated with loss of inhibitory control in animals and humans (Miller, 2000; Dalley et al., 2004; Bechara and Van der Linden, 2005). Previous studies in rodents found that the ventromedial part of the PFC (vmPFC; mostly the infralimbic area) is involved in suppression of extinguished behaviors such as cue-elicited conditioned fear or cocaine self-administration (Milad and Quirk, 2002; Peters et al., 2008). Based on these observations, we hypothesized that in tasks where reward availability is restricted to discrete well signaled or predictable times, the PFC should contribute to behavioral flexibility by limiting the execution of unrewarded actions. To test this possibility, we first studied the development of behavioral suppression during training as rats' performance efficiency improved in a learned appetitive behavior. We then inactivated the vmPFC after the rats had learned to suppress unrewarded actions.

The neural mechanism by which the vmPFC implements action suppression is unclear. The vmPFC has a major projection to the nucleus accumbens shell (NAcS), and this connection has been implicated in maintaining the extinction of cocaine seeking (Peters et al., 2008). Consistent with the idea that vmPFC inhibits unreinforced actions through connections to the NAcS, we recently showed that electrical stimulation of the NAcS immediately interrupts licking for sucrose (Krause et al., 2010) and that NAcS inactivation induces behavioral disinhibition (Ambroggi et al., 2011) (see also Basar et al., 2010). Here we performed simultaneous neural recording in the NAcS with vmPFC inactivation. By using reversible inactivation in this study, we were able to causally implicate the vmPFC in shaping the NAcS neuronal activity and the consequent behavioral output. This simultaneous inactivation and recording technique is novel in the context of response inhibition and extends previous correlational results obtained in our studies about the role of NAcS neurons in goal-directed behaviors.

Materials and Methods

Animals

The subjects were male Long–Evans rats (Harlan Sprague Dawley) weighing ∼250 g on arrival and individually housed on a 12 h reversed light/dark cycle. Experiments were conducted during the dark phase. After receipt, rats were allowed at least 1 week of ad libitum food and water, followed by 1 week of restricted food before training. Throughout all experiments, food restriction was adjusted daily at the end of experimental manipulations to maintain the rats at ∼90% of their initial body weight. Animal handling and experiments conformed to National Institutes of Health and Ernest Gallo Clinic and Research Center animal care and use policies.

Experimental groups

Two groups of rats were used in this study. The pharmacology group (n = 9) had cannulae in both vmPFC and NAcS. The electrophysiology group (n = 9) had cannulae in vmPFC and electrode arrays in NAcS.

Behavioral task

Operant chambers.

All studies were conducted in standard operant chambers (23.5 × 30.5 cm) equipped with two retractable levers and one reward receptacle located on one wall (one on each side of a reward receptacle) (see Fig. 1a, left). Two house lights, a white-noise speaker, and a tone speaker were also installed in the chambers (Med Associates). A syringe pump delivered 50 μl of 10% liquid sucrose into a well in the reward receptacle. Each delivery of sucrose was counted as one reward. An infrared beam installed across the reward receptacle detected receptacle entry and exits.

Figure 1.

Schematic of the DS task and role of vmPFC and NAcS in learned extinction of unreinforced actions. a, Two cue tones (up to 10 s for the rewarded DS; 10 s for the NS) were randomly presented on a variable-interval schedule with an average interval of 30 s. A lever press was required during DS presentation to terminate the DS and cause the delivery of a 10% sucrose reward into an adjacent receptacle. b, DS ratio and receptacle entry significantly increases during cued FR1 training (left). Upon being switched to the DS task, rats learn to suppress responding to NS. Unrewarded lever pressing and receptacle entry are decreased significantly without affecting the DS responding (right). One-sided arrows mark the first hour with significant reduction of responding compared with the first switch hour. Two-sided arrow marks the first hour in which NS ratio became significantly smaller than DS ratio. Lines indicate significant ANOVA effect with time (n = 9). c, vmPFC inactivation (BM) significantly increases measures of unrewarded responding without affecting the DS ratio. Dopamine antagonist injection (SR) in NAcS significantly reduces responding in the DS task. Furthermore, it blocks the behavioral disinhibition caused by vmPFC inactivation. Significant change compared with CSF is marked with an asterisk. Significant disinhibition blocking is marked with the number sign (#): *p < 0.05,**p < 0.01,***p < 0.001; ^#p < 0.05; ^##p < 0.01, ^###p < 0.001 in all figures.

Magazine and cued fixed ratio 1 training.

The first training session was designed to habituate the rats to the experimental chambers and to introduce them to the liquid sucrose reward delivered into the reward receptacle. After the session started, an entry into the receptacle resulted in delivery of sucrose. This was followed by a 10 s time-out, during which additional receptacle entries had no programmed consequences. The magazine session continued until each rat earned 100 rewards. The rats were then switched to a cued fixed ratio 1 (FR1) task on the same day or the next. During the first session, one lever was inserted into the chamber for 60 s and was retracted afterward for 10 s. The inserted lever was designated as the active lever and remained the active lever during later phases of training. Pressing the active lever resulted in reward, lever retraction, and initiation of a 10 s intertrial interval (ITI). No auditory cue was presented in the first session [Fig. 1b, left, no discriminative stimulus (DS) ratio for the first session]. From the second session, an auditory cue (DS) was presented for the duration of active lever extension and was shut off at lever press. The other event timings were the same as those for the first session. The auditory cue was randomly chosen for each rat to be either an intermittent 4 kHz tone (40 ms on and 50 ms off) or a siren tone (ramped from 4 to 8 kHz with a 400 ms period) and remained as the reward predictive cue during the later phases of training. From the third session of cued FR1 training onward, the active lever was no longer retracted upon press and remained available for the session duration (Fig. 1b, left, unrewarded lever presses are only available from the third session onward). A press on the lever during the cue resulted in cue termination and sucrose delivery. During the ITI following reward delivery, active lever presses had no programmed consequences. Over six sessions, the average ITI durations were gradually increased from 10 s to 30 s. After ITI expiration, the auditory cue was presented again, signaling sucrose availability following active lever presses. For four of the nine rats in the electrophysiology group, there was no maximal cue presentation length. For all other rats, cue duration was gradually shortened across six sessions from 60 s to 10 s. Furthermore, in these four electrophysiology rats auditory cues were used from the first session and no lever extension-retraction was used. The animals were trained in the cued FR1 task for 4 h in the pharmacology group and 2 h in the electrophysiology group per session. The inactive lever was not presented during cued FR1 training. For the pharmacology group, behavioral data were not recorded during the last hour (fourth hour) in a few sessions due to a code malfunction (Fig. 1b, left, missing points).

DS task training.

The DS task structure was similar to that reported in previous publications (Nicola et al., 2004; Yun et al., 2004b). After the rats finished training in the cued FR1 task, they were switched to the DS task. Subsequently, both active and inactive levers were inserted into the chamber. Also, both the siren and the intermittent tones were presented on a pseudorandom variable interval schedule with a mean interval of 30 s (Fig 1a, right). The tone initially used for each rat in the cued FR1 task continued to signal reward availability upon active lever press and is referred to as the DS. The novel auditory cue is referred to as the nonrewarded stimulus (NS). Pressing the active lever during the DS presentation resulted in the reward delivery and termination of the DS tone similar to cued FR1 task. Each DS lasted for up to 10 s if no active lever press occurred, and each NS lasted 10 s regardless of the rat's actions. Responding on either lever during the NS or in the absence of the DS was never rewarded in this study.

The animals were trained in the DS task every day for ∼4 h in the pharmacology group and ∼2 h in the electrophysiology group.

Surgeries.

In the pharmacology group, surgeries were done before the magazine training. In the electrophysiology group, surgeries were performed after rats reached criterion performance of >80% DS response ratio and <20% NS response ratio (defined as the percentage of all DSs or NSs in the session to which the animal responded).

For the electrophysiology group, bilateral guide cannulae (27 gauge, Plastics One) aimed at the vmPFC were implanted. A 14° rostral angle was used for the cannulae to allow enough space for the electrode array and the microdrive. The target cannulae coordinates were anteroposterior (AP) +4.2, mediolateral (ML) ±0.8, and dorsoventral (DV) −2.8 mm. The injectors were cut to extend 2.5 mm beyond the cannulae. Therefore, the target injection coordinates were AP +2.9, ML ±0.8, and DV −5.15 mm in the vmPFC. For the pharmacology group, bilateral guide cannulae were implanted similarly with a 14° rostral angle and with coordinates of AP +4.2, ML ±0.8, and DV −4 mm. Injectors were cut to extend 1 mm beyond cannulae, making the target injection coordinates AP +3.0, ML ±0.8, and DV −4.8 mm in the vmPFC. For the pharmacology group, rats were also bilaterally implanted with microinjection guide cannulae (27 gauge, Plastics One) in the NAcS (AP +1.6, ML ±0.8, and DV −6.0 mm; all coordinates are relative to the bregma).

For electrophysiological recordings, both bilateral (four rats) and unilateral (five rats) electrode arrays were used. In two animals, 32 electrodes (16 on each side) were used, while all other animals had 16 electrodes. The electrodes [50 μm stainless steel wires (NB Labs) in five animals; 50 μm tungsten wires (Innovative Neurophysiology) in 4 animals] were attached to a microdrive that allowed the entire array to be lowered by 80 μm increments. Target coordinates of the medioposterior electrode of each array for NAcS during surgery was AP +1.2, ML ±0.8, and DV −6.5 or −7 mm.

Animals were anesthetized with isoflurane (5%) and placed in a stereotaxic apparatus. Anesthesia was maintained with isoflurane (0.5–2.0%) during surgery. After opening the scalp, two holes were drilled for the guide cannulae. The guide cannulae were lowered to the target DV. Dental acrylic was then added to secure the cannulae, and wire obturators were inserted. The ends of the obturators were flush with the ends of the guide cannulae. Then a larger craniotomy was made for the insertion of recording electrodes. The electrodes were lowered into the brain and the microdrive was secured to the skull with bone screws and dental acrylic. After finishing the surgery, rats received an intramuscular injection of antibiotics and intraperitoneal injections of ketaprofen and xylazine (1 and 0.5 mg/kg, respectively) for analgesia and sedation. Rats were allowed at least 7 d of recovery before being retrained on the DS task and habituated to the handling procedures.

Microinjections.

To inactivate vmPFC, a mixture of baclofen and muscimol (BM; GABA_B and GABA_A receptor agonists, respectively) was used. To inject animals, the obturators were removed and 30 gauge injector cannulae were inserted into the guides. We injected 0.5 μl of artificial CSF (aCSF) or BM (12.5 ng of baclofen and 12.5 ng of muscimol dissolved in aCSF) for >2 min. After a 1 min postinjection period, the injectors were removed and the obturators replaced.

The same general procedure was used for injection of dopamine receptor antagonists in NAcS, except that the total volume injected was 0.3 μl to reduce diffusion to NAc core. In this case, a mixture of D1 and D2 receptor antagonist was used (1 μg of SCH23390 and 2 μg of raclopride in 0.3 μl of aCSF). To control for the coinjection of drugs in vmPFC and NAcS, sessions with matching volumes of aCSF in vmPFC and NAcS were done in the pharmacology group.

DS task

Pharmacology group.

Injections reported in this article started after the rats finished 12 sessions of training in the DS task. Six injection conditions were pseudorandomized among rats. Injections were performed in two consecutive sessions followed by one injection-free session. The first two injections were either BM or aCSF in vmPFC. The next four injections were randomly chosen as D1/D2 antagonist in NAcS, aCSF in NAcS, BM in vmPFC together with D1/D2 antagonists in NAcS, and aCSF in both vmPFC and NAcS. Only the first hour of the 4 h sessions, when drug effects were at maximum, was used for analysis. Because no difference was observed in results of various aCSF injection conditions, their results were combined for analysis (n = 3 × 9 = 27). Eight of nine rats had histologically confirmed vmPFC injection sites, and eight of nine rats had histologically confirmed NAcS injection sites. Seven rats had both injectors located correctly within vmPFC and NAcS. All the rats in this group also received injections in vmPFC earlier in training, but these data were not used in the current study.

Electrophysiology group.

Electrophysiological recording started after the rats regained the performance criteria (DS response ratio >80% and NS response ratio <20%) following surgery. In this phase, the rats were run in the DS task for 1–2 h. They were then removed from the chamber and microinjected in the vmPFC with aCSF or BM according to a randomized schedule for each animal. There was at least a day interval between two consecutive injections. Following the injection, the animals were immediately put back in the behavioral chamber and were run in the DS task for 1–2 more hours (see Fig. 2a). The behavioral data and electrophysiological recordings were obtained before and after injection during the task performance. Each rat received multiple injections of BM and aCSF across consecutive sessions. After each recording session, the electrodes were lowered to record a new set of neurons. The bilateral injection data were obtained from 27 aCSF and 32 BM sessions. Three sessions with aCSF injections and eight sessions with BM injections were excluded from our analysis due to insufficient behavioral responding following injections (DS response ratio <30%).

Figure 2.

Injection paradigm and behavioral results. a, Rats were run in the DS task for 1–2 h before injection (PRE), then were then taken out and received injection of aCSF or BM in the vmPFC, and then immediately put back for another 1–2 h following injection (POST). b, Depiction of epochs used for behavioral microanalysis. c, Rates of active lever presses during each epoch for preinjection and postinjection periods for both aCSF (left) and BM (right) conditions. Total sessions: n = 27 for aCSF and n = 32 for BM.

Unilateral injections of BM were also done in some sessions to investigate the laterality of the effects and to help establish the direction of causality between changes in neuronal firing and changes in behavior. Almost all of the unilateral injections were performed following sessions where NS emergence (vide infra) was observed after bilateral inactivations and before lowering the electrodes to a new recording site. This strategy was taken to increase chances of recording neurons with emergent NS responses. Four sessions with ipsilateral injections and three sessions with contralateral injections were recorded.

Electrophysiology

Electrophysiological recording was conducted as described previously (Nicola et al., 2004; Ambroggi et al., 2008; Ghazizadeh et al., 2010). Animals were connected to the recording apparatus (Plexon), which consisted of a head stage with operational amplifiers, cable, and a commutator to allow free movement within the chamber. The microdrive carrying the electrode arrays was lowered by ≥160 μm at the end of each session to get a new set of neurons every day. Histological analysis was done to verify the location of vmPFC cannulae and electrodes in the NAcS.

Spike sorting.

Isolation of individual units was performed off-line with Offline Sorter (Plexon) using principal component analysis. Only units with well defined waveforms with characteristics that were constant over the entire recording session were used for spike sorting. Interspike interval distribution, cross-correlograms, and autocorrelograms were used to ensure that single units were isolated.

Inclusion criteria.

A total of 178 neurons in BM conditions and 128 neurons in CSF conditions were histologically confirmed to be in the NAcS. Furthermore, we required the neuron to have a baseline firing rate of >0.03 Hz. Included in our analysis are 131 (80 left hemisphere) neurons in BM condition and 98 (55 left hemisphere) in the CSF condition that met this firing criterion. Meanwhile, 39 (32 left hemisphere) neurons in the unilateral condition met the histology and firing criteria (17/22 in ipsilateral/contralateral injections).

Determination of the optimal bin size.

The size of the bins significantly influences the analysis of perievent time histograms (PETHs), and potentially useful information can be lost at certain durations. We have developed an objective approach to determining optimal bin sizes for each PETH using Akaike information criterion (AIC) (Ambroggi et al., 2011). Because the optimal bin size is usually rather large for most neurons (∼0.1–1 s), and because the AIC commonly shows a fast reduction over small bin sizes followed by slow changes around the optimal bin size, we also made PETHs using the smallest bin size that showed <10% change from the optimal AIC value. This bin size is referred to as the deflection point bin size and was on average 60 ms across our neuronal population.

Response detection.

PETHs constructed around the behavioral events, with the optimal bin size, were used to detect excitations and inhibitions. The 10 s period before cue onset was used as the baseline period. Excitation and inhibition to each event was determined by the presence of at least one bin above (for excitations) or below (for inhibitions) a predefined confidence interval around the baseline (baseline_max + 0.5 × baseline_range for excitations, baseline_min − 0.25 × baseline_range for inhibitions) during the analysis window for each event using the optimal bin size. Onset detection was performed using a telescopic approach. We selected the first significant bin using the optimal size and searched within this time window for the first bin using the deflection point bin size that was beyond our confidence interval of the baseline made with the deflection point bin size. If we found this bin, we then searched within this deflection point bin for the first 10 ms bin that was beyond the confidence interval of the baseline made with the 10 ms bin size. With this method, a high resolution for onset is achieved for each PETH. Response duration was computed by finding the first bin after the detected onset that fell into the confidence interval of the baseline based on the optimal bin size only. For some neurons, the response types, onsets, and durations were also visually inspected afterward to ensure accuracy. For a neuron to be included as correlated with an event, it should have had an onset latency of no later than 0.5 s to that event. It should also have had at least one significant optimal bin of firing within −0.25–0.25 s for noncue responses (e.g., lever press responses) and within 0–0.5 s for cue responses. With the exception of consumption PETHs, there was no restriction on the duration of responses, and therefore no distinction is made between short and long responses to events. Consumption PETHs were required to be at least 1 s. This is based on our observation that consumption-related responses are often sustained from receptacle entry to receptacle exit (often >3 s).

Data transformation and plotting.

To average PETHs across neurons, the firing rate of each neuron during each bin was transformed to a z-score, as follows: (F_i − F_mean)/F_sd, where F_i is the firing rate of the ith bin of the PETH, and F_mean and F_sd are, respectively, the mean and the SD of the firing rate during the 10 s preceding cue onset.

Histology

Animals were deeply anesthetized with pentobarbital and perfused intracardially with saline and 4% formalin (plus 3% ferrocyanide for rats with stainless steel electrode arrays). Brains were removed, sectioned (50 μm), and stained with Nissl substance to locate injection or recording sites (labeled by passing a DC current through each electrode before perfusion).

Statistical analysis

ANOVAs or t tests followed by post hoc tests corrected for multiple comparisons by Bonferroni or Bonferroni–Holms methods were used when appropriate. Bootstraps with permutation were used to compare percentage changes. All results were considered significant at p < 0.05.

Results

Figure 1a shows a diagram of the chamber wall with levers (active and inactive) and receptacle, and the temporal sequence of events in the DS task. Before being switched to the DS task, all rats received training in a simpler cued FR1 task where only a single cue (DS) was presented. Only the active lever was inserted into the chamber at this stage. Therefore, upon later switching to DS task, NS and inactive lever represented a novel stimulus and a novel action, respectively. DS and NS ratios are defined as the percentage of stimuli presentations followed by active lever pressing.

Experiment 1

Extinction of unreinforced actions during training

Optimal performance in the DS task requires increasing the frequency of rewarded actions while suppressing unrewarded actions. The suppression reduces unproductive energy expenditure and interference with other ongoing behaviors. Indeed, in the early phase of training during the cued FR1 task, rats quickly learn to press the active lever in response to the DS at rates that increase steadily during training. Entering the reward receptacle also increased significantly, indicating increased reward-seeking behavior (one-way ANOVA F_(20,132) = 2.88, p < 0.05; DS ratio and F_(20,132) = 5.28, p < 0.001; unrewarded entry) (Fig. 1b, left). Following the switch to the DS task, rats continued to respond to the DS at high rates, even exhibiting slight improvements in the DS ratio (one-way ANOVA F_(47,347) = 2.88, p = 0.04, DS ratio). On the other hand, the rate of unrewarded active lever presses and unrewarded receptacle entries decreased significantly over trials (one-way ANOVAs; F_(47,347) = 7.8 and F_(47,347) = 3.74, p < 0.001, for unrewarded active lever and entry, respectively). Interestingly, the response ratio to the newly introduced (and never explicitly reinforced) NS was not initially different from the DS ratio (likely due to stimulus generalization) (two-way ANOVA cue × time F_(47,703) = 6.38, p < 0.001, p > 0.05 up to the seventh hour following switch to DS, post hoc test). However, NS responding decreased significantly over time (F_(47,347) = 14.89, p < 0.001, NS ratio, one-way ANOVA). Similarly, the rate of inactive lever presses was initially high, despite never being reinforced, and then it also dropped significantly over time (F_(47,347) = 4.42, p < 0.001, one-way ANOVA) (Fig. 1b, right).

These findings clearly demonstrate the inhibition of unreinforced actions that were initially frequent in the DS task. This rapid extinction proceeded concomitant with maintenance of previously acquired high DS ratios. This indicates that adding the NS cue and the inactive lever did not affect overall reinforcement rates for the previously learned DS task. Furthermore, the findings suggest that efficiency is a powerful driver of learning in this task and, in particular, for the suppression of task-irrelevant actions.

VmPFC to NAcS circuitry mediates response suppression

If vmPFC mediates the response suppression that underlies extinction of unreinforced actions, then its inactivation should result in disinhibition of those responses that the rats have learned to suppress during the training. Indeed, inactivation of vmPFC resulted in significant disinhibition of most unreinforced actions (post hoc test; p < 0.001 for NS ratio and unrewarded active lever, p < 0.05 for unrewarded receptacle entry, BM compared with CSF) (Fig. 1c). On the other hand, the DS ratio was not affected by vmPFC inactivation (p > 0.05, BM compared with CSF, post hoc test). This suggests that vmPFC is both critical to and selectively involved in suppression of unrewarded actions.

Given the known projections from vmPFC to NAcS (Sesack et al., 1989; Voorn et al., 2004), we investigated whether the behavioral suppression removed by vmPFC inactivation depends upon NAcS neurons. In fact, bilateral injection of dopamine D1- and D2-type receptor antagonists [SCH23390 and raclopride (SR)] in the NAcS blocked the enhanced responding that emerged following vmPFC inactivation (post hoc test; p < 0.001 for NS ratio and unrewarded entry and p < 0.05 for unrewarded active lever press, BM/SR compared with BM) (Fig. 1c). This finding directly implicates NAcS neurons in mediating the disinhibition of unreinforced actions revealed by inactivation of vmPFC. Injection of D1/D2 antagonists in NAcS also significantly reduced DS responding (p < 0.001, SR compared with CSF, post hoc test), as was recently reported in a similar task (Nicola, 2010). This indicates that, in contrast to the role of vmPFC in this task, intact dopamine signaling in NAcS is required for both reinforced and unreinforced responding.

To further explore the NAcS neuronal mechanisms affected by vmPFC inactivation, we trained a second group of rats in the DS task (following a similar training schedule) with chronically implanted electrodes in NAcS and microinjection cannulae in vmPFC. This setup enabled us to track NAcS neuronal activity before and after vmPFC inactivations while simultaneously recording behavioral changes.

Experiment 2

Behavioral disinhibition following vmPFC inactivation is temporally specific

Similar to the first experiment, vmPFC inactivation resulted in significant disinhibition of actions unreinforced during training. The injection paradigm was such that each rat received multiple injections in both CSF and BM condition (Fig. 2a). The general disinhibition of active lever presses and the increase in NS ratio raised the question about the temporal specificity of the disinhibition. Is the disinhibition of lever pressing uniform over time or is it correlated with temporally specific task events, such as cues or reward consumption? A temporally nonspecific increase in lever pressing could account for an increase in NS responding. Alternatively, the increased NS responding could result from removal of an inhibition that selectively targets a temporally specific cue-driven response. We divided the DS task into four epochs to quantify the temporal specificity of active lever press changes (Fig. 2b). These epochs include the DS, NS, and postreward periods (10 s period following rewarded receptacle exit) as well as the Spont period (time outside the other three epochs). There was a strong modulation of active lever presses over epochs and injections in both CSF and BM conditions (two-way CSF ANOVA injection: F_(1,182) = 8, p < 0.01; epoch F_(3,182) = 1780, p < 0.001; injection × epoch F_(3,182) = 10, p < 0.001; two-way BM ANOVA injection: F_(1,217) = 30, p < 0.001; epoch F_(3,217) = 398, p < 0.001; injection × epoch: F_(3,217) = 25, p < 0.001). Presentation of the DS induced a rate of active lever pressing higher than any other epoch in both CSF and BM conditions (p < 0.001, post hoc test) (Fig. 2c). However, after inactivation of vmPFC, there were significant increases in the lever pressing during NS and postreward epochs (p < 0.001 compared with preinjection period, post hoc test) (Fig. 2c, right). Inactivation of vmPFC also resulted in a nonsignificant trend toward an increase in active lever presses during the Spont epoch. The increase in active lever pressing during the NS and postreward epochs was significantly higher than that during the Spont epoch (p < 0.001, post hoc test). This indicates that the increased NS ratio after vmPFC inactivation reflects removal of a temporally specific inhibition following NS presentation and not a general increase in lever pressing. Although the active lever press rate during the DS period showed a moderate reduction, this reduction was observed in both CSF and BM conditions (p < 0.001 CSF and p < 0.01 BM, post hoc test).

Analyzing receptacle entries during the same epochs revealed overall similar effects, suggesting that the vmPFC controls both types of reward-seeking behavior (i.e., lever pressing and receptacle entry) in a temporally specific manner. Interestingly, inactive lever pressing did not show modulation by epochs in either injection condition (data not shown). Therefore, unlike the temporally specific suppression of active lever presses, the suppression of inactive lever pressing was uniform in time, pointing to differential suppression of active and inactive lever presses by vmPFC.

Gating model or impulse model?

There are at least two distinct behavioral models that could generate the learned pattern of optimal responding (i.e., active lever presses) observed in the DS task. The first model here, called the “gating model,” assumes a constant promotive process that drives responding throughout the task. However, this promotive process is actively opposed by a suppressive process that inhibits responding. The actual temporal pattern of responding in this model is produced by periods during which suppression strength is reduced. This pattern is similar to opening a gate that allows responding (Fig. 3a, left). The second model here, the impulse model, proposes that motivated actions are controlled by the dynamics of the promotive process overcoming an ongoing background of constant suppression (Fig. 3a, right). Both models assume that response rate is proportional to the magnitude of the difference between the processes that either promote or inhibit actions, and both assume that responding happens when this difference is positive.

Figure 3.

Models of motivated responding in the DS task. a, schematics of gating model (left) and impulse model (right) of response promotion and suppression. b, Estimation of promotive and suppressive processes in the DS task based on the average rate of active lever pressing during various task epochs in the CSF condition and BM condition. Response rate during the BM is assumed to be solely due to promotive component. Suppressive component can be inferred by subtracting the average response rate in CSF condition from BM condition (dashed trace).

Assuming that inactivation of vmPFC effectively (and selectively) eliminates the suppressive process, the temporal rate of active lever presses in this condition should reveal the shape of the opposing promotive process. Though we cannot at present directly measure the dynamics of the active lever press-suppressing process, its shape can be inferred by subtracting the rate of responding in the BM condition (which should reflect only the promotive process) from the CSF condition (which reflects the difference between promotive and suppressive processes) (Fig. 3b). Based on our behavioral results, the inferred temporal dynamics of promotion and suppression suggest that a combination of gating and impulse models governs responding in the DS task. This analysis shows that active lever press suppression was negligible during the DS but increased during postreward and NS. Therefore, these results predict that the reduction of suppressive processes following vmPFC inactivation should not affect the neural representation of events during the DS (reinforced) epoch in NAcS but should change neural representations during unreinforced epochs. We tested this hypothesis using our simultaneously recorded behavioral and neural data.

A total of 98 and 131 neurons recorded during CSF and BM conditions, respectively, are analyzed in the following sections.

No net change in phasic neural representation of the reward-related events

On the majority of trials after the DS was presented, rats pressed the active lever and entered the receptacle to consume the sucrose reward. Rats then exited the receptacle following reward consumption. Inactivation of vmPFC did not affect this aspect of the DS task behaviorally. Interestingly, at the neuronal level, activity during the DS epoch in the NAcS was also unaffected by vmPFC inactivation. Such inactivation did not produce significant changes in either the percentage of responsive neurons or the magnitude of their response to the DS, rewarded lever press, rewarded receptacle entry, consumption, or the receptacle exit that followed (post hoc test; p > 0.05 for z-score peaks of excitations and inhibition, permutation test for percentages p > 0.05) (Fig. 4). This result is consistent with our behavioral results, and our prediction that the contribution of vmPFC to NAcS neuronal firing in the DS epoch would be negligible.

Figure 4.

Encoding of sequence of actions in the rewarded trials following DS presentation before and after vmPFC inactivation. a–e, Changes in the magnitude of peak-firing z-score of neurons responsive to the rewarded DS, rewarded lever press (LP), rewarded receptacle entry (Entry), consumption (Cons), and rewarded receptacle exit (Exit) in either the preinjection or postinjection period (open and filled black bar) in the BM condition for inhibitions (INH) and excitations (EXC). Total neurons: n = 37, 14 for DS+, −; n = 35, 40 for LP+, −; n = 40, 56 for Entry+, −; n = 27, 44 for Cons+, −; n = 32, 47 for Exit+, −. f–j, Percentage of neurons responsive to rewarded DS, rewarded LP, rewarded receptacle entry, consumption, and rewarded receptacle exit in the preinjection and postinjection periods in CSF and BM conditions. Excitation and inhibition percentages are shown on top and bottom of each panel, respectively. Preinjection and postinjection results are shown with open and filled bars, respectively (gray, CSF Post; black, BM Post).

VmPFC inactivation enhances the phasic neural representation of unreinforced events

Unlike the neural activity related to rewarded events, we observed significant changes in NAcS neuronal encoding of unrewarded actions following inactivation of vmPFC. Importantly, the increase in lever pressing during the NS observed following vmPFC inactivation (Fig. 2c) was associated with a significant increase in NS-evoked firing in the NAcS. vmPFC inactivation increased the peak firing z-score of NS excitations (F_(1,27) = 7.74, p < 0.01 for injection, two-way ANOVA; p = 0.025 for excitation but p = 0.45 for inhibition, post hoc test) (Fig. 5a). With vmPFC intact, the proportion of NS responsive neurons was low. Inactivating the vmPFC recruited a new population of NS-excited neurons. Overall, the percentage of NS-excited neurons increased by 58% after inactivation (p = 0.019, permutation test) (Fig. 5d). Furthermore, inactivation of vmPFC also significantly increased the representation of unrewarded lever presses. Z-scores of the peaks of both phasic excitations and inhibitions increased (two-way ANOVA injection F_(1,61) = 5.37, p < 0.05, post hoc p < 0.001 for excitation and p < 0.01 for inhibition) (Fig. 5b). In addition, the percentage of unrewarded lever press-inhibited neurons showed a very large increase (+238%; p < 0.001, permutation test) (Fig. 5e). vmPFC inactivation did not significantly change the percentage or firing rate of neurons responsive to unrewarded receptacle entries (for excitation and inhibition: p > 0.05, post hoc test; p > 0.05, permutation test) (Fig. 5c,f). This might reflect the fact that the relative magnitude of the behavioral disinhibition of unrewarded entries following vmPFC inactivation was considerably smaller than that of unrewarded lever pressing (Fig. 1c).

Figure 5.

Encoding of unreinforced actions in NAcS neurons before and after vmPFC inactivation. a–c, Changes in the magnitude of peak firing z-score of neurons responsive to NS, unrewarded lever press, and unrewarded receptacle entry in either preinjection or postinjection period (open and filled black bar) in the BM condition for inhibitions (INH) and excitations (EXC). Total neurons: n = 21, 7 for NS+, −; n = 32, 29 for unrewarded lever press (Unrew LP)+, −; n = 45, 54 for unrewarded entry (Unrew Entry)+, −. d–f, Percentage of neurons responsive to NS, unrewarded lever press, and unrewarded receptacle entry in the preinjection and postinjection periods in CSF and BM conditions. Excitation and inhibition percentages are shown on top and bottom of each panel, respectively. Preinjection and postinjection results are shown with open and filled bars, respectively (gray, CSF Post; black, BM Post).

Figure 6a shows an example neuron with emergent phasic short latency NS excitation following vmPFC inactivation. Interestingly, all the neurons with emergent NS excitations were also excited by the DS. Given the evidence that DS excitations in these neurons promote responding (Yun et al., 2004a; Ambroggi et al., 2008; Ishikawa et al., 2008), it is highly likely that NS excitations also promote behavioral responding to the NS. However, despite the clear enhancement of NS excitations by vmPFC inactivation in neurons showing either emergent (n = 9) or enhanced (n = 3) NS excitations, the average DS response for this same population of neurons was not significantly changed by vmPFC inactivation (Fig. 6b). This is consistent with the observation that DS-related responding is unaffected by vmPFC inactivation. Furthermore, the lack of effect on DS responding indicates that the inhibitory effect of vmPFC on NS excitations in NAcS neurons is temporally specific. To further investigate the laterality of vmPFC projections, which supposedly suppress NS excitations in NAcS, we performed unilateral inactivation of vmPFC in some sessions (n = 7). Overall, 17 and 22 neurons were recorded ipsilateral and contralateral to the injection site. Consistent with the bilateral vmPFC projection to the NAcS (Sesack et al., 1989), emergent NS excitations were found both ipsilateral and contralateral to the side of inactivation (Fig. 6c). The increase in the total number of NS excitations following unilateral inactivations was significant compared with the CSF condition after combining the results of both ipsilateral and contralateral conditions. Again, similar to the bilateral inactivation, there was no significant change in the percentage of DS-encoding neurons following unilateral inactivations.

Figure 6.

Encoding of neutral stimuli in NAcS before and after vmPFC inactivation. a, Example neuron showing emergent NS excitation following bilateral vmPFC inactivation. Overlaid red line shows the PETH made from the NSs not followed with a behavioral response. b, Average z-scores of NS PETHs for neurons showing emergence or significant increase in excitations to NS before (gray) and after (black) vmPFC inactivation and average z-scores of DS PETHs for the same group of neurons. c, Example neurons showing emergent NS excitation following ipsilateral or contralateral vmPFC inactivations. Overlaid red line shows the PETH from NS not followed with a behavioral response.

Because the rate of lever pressing increased after vmPFC inactivation, the emergence of NS excitations could be coincidental or secondary to the behavioral response. However, three observations make this possibility very unlikely. First, NS excitations appeared at short latency to cue onset (<0.5 s compared with >2 s average lever press latency to NS). Second, emergent neural responses to NS remained even when we only included trials where no lever pressing occurred during the NS (Fig. 6c, overlaid red line). Third, we found, even in the unilateral vmPFC inactivations, emergent NS excitations that did not induce significant behavioral disinhibition.

NS-responsive neurons in our task were distributed throughout the medial NAcS, but there was no relationship between the magnitude of the firing changes produced by vmPFC inactivation and the rostrocaudal or dorsomedial axes or their interactions (R² = 0.2, p = 0.26).

Reduction of tonic firing in phasically inhibited but not excited neurons

Inactivation of vmPFC eliminates a source of excitatory glutamatergic input to NAcS neurons, which can lower their firing rate. In fact, basal firing rate was reduced primarily in neurons that exhibited phasic inhibitions to task events (post hoc-rewarded lever press inhibition, p = 0.02; rewarded entry, p = 0.008; rewarded exit, p = 0.003; unrewarded entry, p < 0.001) (for examples, see Fig. 7). In striking contrast, vmPFC inactivation did not on average reduce firing in phasically excited NAcS neurons (Fig. 8). This observation is particularly instructive in view of our recent evidence that phasic inhibitions permit reward-seeking behaviors (Taha and Fields, 2006; Krause et al., 2010). A reduction of tonic firing in such gating neurons could account for the generalized disinhibition of responding following vmPFC inactivation, for example, during the Spont epoch (Fig. 2c). Furthermore, neurons with significant tonic firing decreases with vmPFC inactivation were on average inhibited to unrewarded entry and did not respond to the NS. This is consistent with results illustrated in Figure 8, emphasizing the overlap between neurons that show tonic firing reduction and phasic inhibitions.

Figure 7.

Tonic firing of NAcS phasically inhibited neurons before and after vmPFC inactivation. Three example neurons (with firing raster on top and PETH below) that show phasic inhibitions to rewarded lever press, rewarded entry, and unrewarded entry before and after vmPFC inactivation. Basal firings of all three neurons show drastic reductions following vmPFC inactivation. These reductions are accompanied by a marked disinhibition of unrewarded lever pressing (red dots) compared with preinactivation period.

Figure 8.

Changes in tonic firing of neurons with event-related phasic excitations and inhibitions. Changes in the basal firing of neurons responsive during the rewarded events (top row) and unrewarded events (bottom row) with phasic inhibitions (INH) and phasic excitations (EXC). Preinjection and postinjection results are shown with open and filled black bars, respectively. Total neurons: n = 37, 14 for DS+, −; n = 35, 40 for lever press (LP)+, −; n = 40, 56 for entry (Entry)+, −; n = 27, 44 for consumption (Cons)+, −; n = 32, 47 for exit (Exit)+, −; n = 21, 7 for NS+, −; n = 32, 29 for unrewarded lever press (Unrew LP)+, −; n = 45, 54 for unrewarded entry (Unrew Entry)+, −.

In contrast, neurons with significant increases in their tonic (basal) firing were on average phasically excited to unrewarded events, such as NS or unrewarded entry. This suggests that at least some of the phasically excited neurons are inhibited by vmPFC inputs. Such inhibition would likely be indirect because the direct projections from vmPFC to NAcS medium spiny projection neurons (MSNs) are via excitatory glutamatergic neurons.

Corticostriatal circuit model mediating behavioral disinhibition

We propose a circuit model that incorporates observed phasic and tonic changes in the NAcS following vmPFC inactivation and that explains the concomitant behavioral disinhibition. In this model, phasically inhibited NAcS neurons receive direct excitatory input from vmPFC and suppress behavioral responding outside the DS. In addition, phasically excited neurons that promote responding are indirectly inhibited by vmPFC (except when the DS is presented) (Fig. 9a). This indirect inhibition could arise from feedforward connections within or outside NAcS or via local axon collaterals from phasically inhibited NAcS MSNs.

Figure 9.

Proposed vmPFC to NAcS circuit model for suppression and promotion of responding. a, vmPFC input excites phasically inhibited neurons (black) for generalized response suppression. It also indirectly inhibits phasically excited neurons (gray) for a temporally selective suppression of responding. b, Following vmPFC inactivation, firing of phasically inhibited neurons is reduced, resulting in generalized disinhibition. Phasically excited neurons are disinhibited and can be driven by other inputs to promote temporally specific responding. Solid and dashed lines indicate strong and weak connections in each condition, respectively. Neural connections are in black and behavioral effects are in gray. Plus and minus signs respectively indicate excitatory and inhibitory influences.

Removal of this vmPFC-derived inhibition allows excitatory inputs from other structures [e.g., basolateral amygdala (BLA) and prelimbic cortex] to drive these phasically excited neurons and promote NS responding (Fig. 9b). We previously found that dopaminergic input from the ventral tegmental area (VTA) enhances these glutamate-dependent phasic excitations in NAc neurons in response to cues (Yun et al., 2004a; Ambroggi et al., 2008). Consistent with this hypothesis, we see blockade of behavioral responding (both DS and NS) if dopamine antagonists are injected in NAcS following vmPFC inactivation.

In addition to this temporally specific effect on firing, vmPFC inactivation reduces the tonic firing of phasically inhibited neurons, thereby reducing tonic behavioral inhibition, lowering the action initiation threshold, and increasing the probability of normally inhibited unreinforced actions (Fig. 9b).

A simple interpretation of our model is that the opposing behavioral effects of the phasically excited and inhibited NAcS neurons sum to produce changes in responding. In fact, plotting the arithmetic difference in firing between phasically excited (∼30%) and phasically inhibited (∼50%) neurons over the four previously discussed behavioral epochs revealed a pattern similar to the behavioral response data in Figure 2c in both injection conditions (Fig. 10). In the CSF condition, this difference was significantly higher than zero only during the presentation of the DS in both preinjection and postinjection periods (p < 0.001, post hoc test). This was consistent with a high rate of lever pressing during the DS. The same pattern was observed in the preinjection period in the BM condition. However, following vmPFC inactivation, there was a significant increase in the magnitude of the difference between firing of phasically excited and phasically inhibited neurons in parallel with the increased rate of active lever pressing.

Figure 10.

Firing difference between phasically excited and phasically inhibited neurons during task epochs. Modulation of the firing difference between phasically excited and phasically inhibited neurons across epochs before and after injection of aCSF or BM (left and right, respectively). Preinjection and postinjection results are shown with open and filled bars, respectively. Gray and black bars show the results for aCSF and BM injections, respectively. Two-way CSF ANOVA injection: F_(1,462) = 1.08, p = 0.29; epoch: F_(3,462) = 18, p < 0.001; injection × epoch: F_(3,462) = 0.13, p = 0.9; two-way BM ANOVA injection: F_(1,749) = 21, p < 0.001; epoch: F_(3,749) = 7.3, p < 0.001; injection × epoch: F_(3,749) = 0.2, p = 0.9.

Discussion

We previously demonstrated that rewarded actions in the DS task require projections to the NAc core from the VTA, BLA (Yun et al., 2004a; Ambroggi et al., 2008), and dorsomedial prefrontal cortex, including the prelimbic area (Ishikawa et al., 2008). In contrast, we found that the vmPFC, largely the infralimbic area (IL), suppresses responding to the unreinforced cue NS in the DS task (Ishikawa et al., 2008). Consistent with this finding, vmPFC also inhibits premature and perseverative errors in a five-choice serial reaction time task (Muir et al., 1996; Chudasama et al., 2004). Furthermore, in the context of fear conditioning, phasic activation of vmPFC neurons (including IL) mediates extinction of a tone previously paired with foot shock (Milad and Quirk, 2002). Extinction is a learning process that improves performance efficiency by inhibiting actions that do not contribute to an optimal outcome. Following extinction training, such inhibited actions reappear when vmPFC is inactivated. The current study extends understanding of extinction by showing that vmPFC controls two distinct population of neurons in NAcS: one that inhibits and another that promotes actions during the task.

Following vmPFC inactivation, disinhibition of responding was observed in all epochs of the DS task except during the DS period, which includes a sequence of rewarded cues and actions. The disinhibition of unreinforced active lever pressing is not uniform over time. We find a much greater increase in responding during NS compared with the Spont epoch (Fig. 2c), indicating that there is a temporally specific component to the inhibition of NS mediated by vmPFC. Our analysis of the temporal dynamics of response-promoting and response-suppressing processes during the DS task also reveals that the suppressive process is transiently reduced during the DS while, during other epochs, it increases to oppose the promotive process that drives unreinforced responding. In addition, this analysis indicates that both promoting tendencies (impulses) and temporally specific reduction of suppression (gating) sum to enhance appropriate responding during the reward approach and consummatory period (Fig. 3).

Striatal projections arising from the vmPFC mainly target the medial NAcS (Voorn et al., 2004) and are required for suppression of unrewarded responding in other behavioral paradigms (Christakou et al., 2004; Voorn et al., 2004; Peters et al., 2008). We find that injection of dopamine receptor antagonists in NAcS reverses the behavioral disinhibition following vmPFC inactivation, directly implicating vmPFC–NAcS circuitry in mediating learned behavioral suppression (Fig. 1c).

Our recently published study shows that NAc and, particularly, NAcS are critically involved in response suppression (Ambroggi et al., 2011). Furthermore, consistent with the hypothesis that phasic inhibitions observed in the NAc have a permissive role in behavioral responding (Taha and Fields, 2006), electrical stimulation of NAc potently inhibits consummatory behavior (Krause et al., 2010). Importantly, we find that the “spontaneous” firing of neurons that are phasically inhibited during various task-related actions is reduced following vmPFC inactivation. We propose that this reduction of firing, which presumably arises from elimination of excitatory input to the phasically inhibited neurons, mediates the general (temporally nonspecific) disinhibition of unreinforced lever pressing and receptacle entry (Figs. 1c, 2c).

A second and distinct aspect of the behavioral disinhibition following vmPFC inactivation was revealed by the temporally specific increase in behavioral responding to the NS. This increase in behavioral responding is accompanied by an increase in both the proportion and the magnitude of phasic NS excitations in a subset of NAcS neurons (Figs. 5, 6). This suggests that these NS excitations promote behavioral responding to the NS. The fact that dopamine antagonist injections in NAcS block this disinhibited NS responding strengthens this conclusion because cue excitations in NAc depend on inputs from the VTA and behavioral responses to both NS and DS are blocked by dopamine antagonists microinjected into the NAc (Yun et al., 2004a). Furthermore, the observed emergence of NS excitations following both ipsilateral and contralateral inactivations (Fig. 6c) is consistent with bilateral projections from vmPFC to NAc (Sesack et al., 1989).

Because the direct vmPFC projection to the NAcS is glutamatergic, the emergence of excitations to NS and unrewarded lever press after inactivation implies the existence of at least one layer of intervening neurons that receive vmPFC input and inhibit the emergent responses under normal conditions. Within the striatum, this could be achieved through fast-spiking interneurons by fast feedforward inhibitions (Mallet et al., 2005) or by lateral inhibitions from the axon collaterals of other MSNs (Taverna et al., 2004; Chuhma et al., 2011). For instance, what we observed as a significant reduction of baseline firing in action-inhibited neurons might be responsible for the emergence of excitatory responses to unrewarded cue and actions. Of course, this indirect inhibition can be mediated by an intervening structure outside the NAcS. Here we proposed a circuit model in which phasically inhibited neurons receive direct projections from vmPFC and suppress responding, while phasically excited neurons that promote responding are under indirect inhibition from vmPFC via an intervening GABAergic influence. Further studies will be needed to test this model. For instance, it is unclear whether the same pyramidal neuron may mediate both types of controls on the NAcS neurons. There is some evidence that medial PFC projections to matrix and patch-like regions in ventral striatum may arise from separate groups of neurons in deep and superficial layer V (Berendse et al., 1992). A future study using optogenetic silencing or stimulation of the NAcS projecting vmPFC neurons could inform this issue.

Conclusion

Together, our results suggest a dual neural mechanism by which vmPFC inhibits unreinforced actions via NAcS neurons. The first mechanism is by temporally specific inhibition of phasic excitations encoding unrewarded actions and stimuli that promote unrewarded responding. A phenomenon such as resistance to distracters could be explained by this mechanism. The second mechanism is by increasing the tonic firing of gating neurons that inhibit unreinforced, task-irrelevant actions. In contrast to the first mechanism, this inhibitory mechanism is not temporally specific and may nonselectively control a variety of potentially competing and irrelevant actions in the context of a learned, goal-directed appetitive behavior.

Notes

Supplemental material for this article is available at http://www.galloresearch.org/ambroggi/ghazizadeh-2011-supp.pdf. Supplemental figures include (1) training data for electrophysiology group (supporting Fig. 1), (2) histological locations of injections and recorded neurons, (3) epoch analysis of receptacle entry (supporting Fig. 2), (4) neuronal and behavioral results of unilateral inactivations (supporting Fig. 6), (5) location of NS responsive neurons, and (6) differential modulation of phasically excited and inhibited neurons (supporting Fig. 8). This material has not been peer reviewed.