Nucleus Accumbens Core Neurons Encode Value-Independent Associations Necessary for Sensory Preconditioning

Abstract

Reinforcement-based learning models predict that the strength of association between cues and outcomes is driven by aspects of outcome value. However, animals routinely make associations between contingent stimuli in the world, even if those associations hold no value to the organism. At the neural level, the nucleus accumbens (NAc) is known to encode associative information, but it is not known whether this encoding is specific for value-based information (consistent with reinforcement-based models), or if the NAc additionally plays a more general role in forming predictive associations, independent of outcome value. To test this, we employed a sensory preconditioning (SPC) task where rats initially (Preconditioning) received either contingent pairings of two neutral stimuli (e.g., tone [A] and light [X]; ‘Paired’), or random non-contingent presentations (‘Unpaired’). After cue X was subsequently conditioned with food (First-Order Conditioning), the effect of preconditioning was assessed in phase 3 (Test) by presentations of cue A alone. Electrophysiological recordings from the NAc core showed significant increases in phasic encoding for the stimuli in the Paired (but not Unpaired) condition as well as during Test. Further, these effects were only seen in Paired rats that showed successful behavior during Test (Good Learners), but not those who did not (Poor Learners) or Unpaired controls. These findings reveal a role for the NAc in the encoding of associative contingencies independent of value, and suggest that this structure also plays a more general role in forming associations necessary for predictive behavior.

Introduction

In Pavlovian conditioning, associations are formed between stimuli and outcomes when they are observably contingent upon one another (Rescorla, 1988). In a typical experiment, an otherwise neutral stimulus is contingently paired with a salient US (e.g., food, footshock), and through learning, the cue CS comes to reliably predict the reinforcer. A host of reinforcer-based learning models, the most influential being the Rescorla Wagner (RW) model (Rescorla & Wagner, 1972), posit that the discrepancy between the predicted outcome value during the cue and the actual outcome value received generates a prediction error. This difference augments the strength of association of subsequent trials, guiding new learning across trials. Thus, this type of learning requires that the outcome have a non-zero value, or else animals will be unable to generate prediction errors necessary for the production of associative behavior.

The neural basis of this learning has been a focus of intense study, as activity in regions of the limbic system and its dopaminergic input encode associative information in a way strongly predicted by reinforcement-based learning models. One of these regions, the nucleus accumbens (NAc), is well known to be involved in multiple aspects of associative learning. For example, in first-order conditioning (FOC), NAc core lesions selectively impair the ability of food-paired CSs to elicit cue-oriented, but not food-oriented, responses (Cardinal et al., 2002; Chang et al., 2012). NAc core lesions also impair the ability for CSs to enhance instrumental responding in a Pavlovian-to-instrumental transfer task (Hall et al., 2001). Further, lesions of the NAc, even if made after FOC, disrupt the ability of animals to acquire and express associations between a new stimulus and a CS with acquired value in second-order conditioning (SOC) (McDannald et al., 2013). Similar results have been demonstrated with lesions of the basolateral amygdala (BLA), a major limbic input to the NAc. Like NAc lesions, disconnection lesions of the NAc and BLA also impair cue-oriented responses (Chang et al., 2012), and SOC (Setlow et al., 2002b). Unlike lesions of the NAc however, BLA lesions made after FOC do not disrupt SOC (Setlow et al., 2002a). Thus, it appears that BLA-NAc connectivity is necessary for the acquisition, while the NAc alone is sufficient for the expression of acquired motivational value.

Further, previous work from our lab has shown that, as animals learn FOC, NAc core neurons develop differential encoding to cues that predict intrinsically valuable outcomes (such as food) versus cues that do not (Day et al., 2006), as well as to cues that predict outcomes of acquired value in SOC (Saddoris & Carelli, 2014). Indeed, encoding of associations with valuable outcomes in the NAc is consistent with reinforcer-based prediction error models (Cromwell & Schultz, 2003; Roesch et al., 2009). However, in these experiments, learning about associations between stimuli (e.g., cue and food) is confounded with learning based on reinforcer value (e.g., cues elicit approach because it has acquired associative value). As such, it is unclear whether the NAc is important for predictions of value, or if in addition, NAc is recruited to encode expected events in general, even if those events have no intrinsic worth.

One mechanism to dissociate these functions is by using a task in which animals must learn to form associations that are independent of value, such as sensory preconditioning (SPC) (Brogden, 1939; Rizley & Rescorla, 1972). In this task, animals initially make associations between two neutral stimuli (AX: Preconditioning), later learn that one of the stimuli has acquired value (X→food: FOC), and then transfer that new information to the paired associate (A) when presented alone (SPC Test). Thus, in contrast to value-based associative learning paradigms (e.g., FOC or SOC), SPC is reliant upon inferred value from associations between value-neutral stimuli, thereby requiring associative predictions in the absence of value. While it appears different associative regions such as perirhinal cortex (Nicholson & Freeman, 2000) and orbitofrontal cortex (Jones et al., 2012) contribute to the successful performance of SPC, the BLA appears to have much less of a role (Blundell et al., 2003; Dwyer & Killcross, 2006). Despite these findings, and evidence that the NAc is necessary when learning is dependent on the identity of outcomes (McDannald et al., 2011) or even contextual configurations (Westbrook et al., 1997), the role of the NAc core in a more general associative task like SPC remains unknown.

The present study was designed to directly examine the activity of NAc core neurons during all phases of an SPC task. If the NAc core is important for learning predictive relationships even in the absence of value, then differential encoding should develop to repeated presentations of stimuli during Preconditioning. In contrast, if NAc encoding specifically reflects information about predictions of value, then there should be no differential encoding of cues during Preconditioning; instead, the NAc should be biased towards encoding during first-order conditioning when the cues are explicitly paired with an outcome of intrinsic value.

Methods

Subjects

Experimentally naive male Sprague-Dawley rats (n = 28; Charles River Laboratories), aged 8-12 weeks and weighing approximately 300 g at the time of arrival were used. The individually-housed rats habituated to their homecages for approximately 1 wk, during which time they had ad-libitum access to food and water and were maintained on a 12 h light / dark schedule. Following habituation, rats were implanted with indwelling electrophysiological recording arrays in the core of the NAc (see below). After 2 wks, rats were food restricted (unlimited water access) to 15 g chow/d to maintain their weight. Rats remained on this restricted diet for the duration of all behavioral procedures. Animal procedures were approved by the University of North Carolina, Chapel Hill Institutional Animal Care and Use Committee (IACUC).

Surgical Methods

Prior to all behavioral testing, rats were anesthetized with ketamine (100 mg / kg) and xylazine (10 mg / kg), and then secured in a stereotaxic apparatus (Kopf Instruments, Tijunga, CA, USA). The scalp was incised and retracted, and the skull was adjusted to level in all planes. Holes were drilled in the skull above the NAc core (AP: +1.8 mm, ML: ± 1.4 mm, relative to Bregma) in both hemispheres. An eight-microwire recording array (NB Labs, Denison, TX, USA) was slowly lowered into the NAc core at a depth of 6.2 mm from the brain surface. The arrays consisted of two parallel rows of four stainless steel Teflon-coated, 50 um-diameter wires, tips spaced evenly 0.5 mm apart. A ground wire for each array was placed in the brain distal to the recording location in the same hemisphere. The apparatus was permanently secured with dental acrylic attached to screws embedded in the skull surface. Animals were given an oral dose of 1.0 mg / kg meloxicam (Metacam, Boehringer Ingelheim Vetmedica, St Joseph, MO, USA) as a post-operative analgesic for 2 d, and at least 1 wk to recover from surgery before beginning food restriction and behavioral training.

Apparatus

All training and testing took place in a custom-built behavioral chamber (43 x 43 x 53 cm; MED associates, St Albans, VT, USA) housed in a sound-attenuating cabinet. The interior walls of the cabinet were covered in metal mesh to provide insulation from external electrical signals. Chambers were illuminated by a houselight located on the ceiling. Masking noise and ventilation were provided by a wall mounted fan. A centrally-located foodcup (4 cm above the floor), equipped with photobeams to automatically detect head entries, was mounted on the right wall of the chamber. Auditory stimuli were delivered by a speaker 18 cm above the floor, and consisted of either a tone (800 Hz) or white noise, calibrated to 65 dB. Visual stimuli were presented at a pair of 2.5 cm-diameter cue lights (flanking the food cup 22 cm apart and 12 cm above the floor). Visual stimuli consisted of a solid light and a flashing light (2 Hz), delivered at the right and left cue lights, respectively.

Electrophysiological recordings were taken during all behavioral sessions. Details on electrophysiological recording procedures have been reported previously (Saddoris et al., 2011). Briefly, rats were connected to a recording cable that terminated in a headstage (Plexon Inc., Dallas, TX, USA). The cable was connected at the other end to a commutator (MED Associates and Crist Instruments) allowing free movement throughout the chamber during sessions. Amplified neural signals were then passed to a Multichannel Acquisition Processor (MAP) system (Plexon Inc.) where they were captured by a neural analysis program (Sort Client, Plexon Inc.). A separate computer controlled external stimuli and captured behavioral events (TRANS IV, MED Associates). Neural data analysis was performed using software (Offline Sorter, Plexon Inc.) to sort neural waveforms by principal components analysis. Finally, the resulting timestamps for valid waveforms were further analyzed in relation to behavioral markers and events of interest using NeuroExplorer software (NEX Technologies, Littleton, MA, USA).

Behavioral Task

Before training, rats were given a brief session in which they received 8 noncontingent, pseudorandomly delivered 45 mg sucrose pellets (Purina, Richmond, IN, USA) in order to familiarize them with reward delivery and the food cup. Rats with recording arrays were also connected to the recording apparatus during this session to habituate them to the tether. An overview of the sensory preconditioning task is shown in Table 1. Before training, rats were divided into ‘Paired’ (n = 20) and ‘Unpaired’ (n = 8) groups. Four stimuli (described above), white-noise (N), tone (T), flashing right panel light (F) and solid left panel light (S), were used as cues which related to four different cue types (denoted A, B, X, and Y) during this task. To mitigate non-associative aspects of cue salience on learning, subjects within the Paired group received different associations between cues and specific stimuli. For one group (n = 11), cues A, B, X, and Y corresponded to stimuli N, T, F, and S, respectively. Likewise, for other Paired animals, the respective stimuli used for A, B, X, and Y were either T, F, N, and S (n = 3), F, N, S, and T (n = 4), or F, T, S, and N (n = 2). For Unpaired rats, cues A, B, X, and Y corresponded to stimuli N, T, F, and S, respectively. All cues were presented for 10 s. For purposes of behavioral and neural analysis (see below), counterbalanced subjects were grouped pooled by cue type (e.g., Cue A) regardless of the specific stimulus type (e.g., tone, noise etc).

Table 1.

Behavioral design for SPC training.

	Preconditioning	First-Order Conditioning	Test
	Days 1-2	Days 3-5	Day 6
Paired	A→X	X→US	A
	B→Y	Y	B
Unpaired	A
	B	X→US	A
	X	Y	B
	Y

Note. US: 3 sucrose pellets. On Test, A is also referred to as SPC+, and B as SPC−.

Preconditioning

Rats were assigned into either the ‘Paired’ and ‘Unpaired’ group. Rats in each group received two consecutive days of Preconditioning (days 1 and 2). Animals in the Paired group received two blocks of 12 cue pairings per day; in one, presentation of A co-terminated with the onset of X, while in the other, B co-terminated with the onset of Y. A variable inter-trial interval (ITI) of 90-270 s separated each set of cue pairings. Blocks were separated by a 10 m timeout without fan or houselight, and block order was counterbalanced between animals and reversed on the second day of training. Animals in the Unpaired group received 12 trials of each of the 4 cues, presented in isolation, on both days of Preconditioning. Cues were presented in pseudorandom order, following a variable ITI of 50-140 s. Note that animals in the Paired and Unpaired group only differed in whether or not preconditioned stimuli were paired, and received identical behavioral manipulations on all subsequent days of the SPC task.

First-order conditioning

After Preconditioning, all animals received three consecutive days of Pavlovian FOC (days 3-5). Each day, the preconditioned cues X and Y were presented in pseudorandom order to serve as CSs. Trials were separated by a variable ITI of 90-270 s. Three 45 mg sucrose pellets were delivered immediately after each termination of X (the CS+), while Y (the CS-) was never followed by reinforcement during these sessions. On the first two days of FOC, 21 trials of X and 20 of Y were given, with the unconditioned stimulus (sucrose pellets; US) omitted on 3 and 4 of the total cue X trials, respectively, in order to increase resistance to extinction effects during later testing. On the third day, 20 trials of X and 21 of Y were given, with the US omitted on 2 of the trials with X.

Test session

Once FOC training was completed, the SPC effect was assessed during a final Test session (day 6). Rats in both the Paired and Unpaired groups were pseudorandomly presented with preconditioned cue A, for 19 trials, and B, for 21 trials, featuring each cue in isolation. In addition, 3 reminder trials for both of the CSs used during FOC (X followed by the US, and Y without the US) were interspersed into the session to prolong behavior under extinction conditions. Again, each cue trial was separated by a variable 90-270 s ITI.

Histology

Histological verification of electrode placements was accomplished using established procedures (Saddoris et al., 2011). Briefly, after the experiments, animals were heavily anesthetized with ketamine (100 mg / kg) and xylazine (10 mg / kg), a 14.4 μA current was then passed through each stainless-steel microwire for 5 seconds to leave an iron deposit in the tissue. To identify the wire tips, rats were perfused transcardially with saline (10 m, 20 mL / m), followed by a 3% potassium ferricyanide in 10% formalin solution. The brain was removed, frozen to -20 °C and coronally sliced (40 um thick) throughout the extent of the NAc. Slices were mounted on slides, documented with high-resolution photomicrographs, and electrode placement was confirmed within the NAc using a standard atlas (Paxinos & Watson, 1997).

Analysis of Behavior

Behavioral conditioning was assessed during each session as the mean number of ‘cue’ head entries into the foodcup made during 10 s presentations of a particular stimulus across trials minus the ‘baseline’ mean foodcup entries made during the 10 s period preceding all stimuli, for each animal.

To compare individual differences in behavior and neural encoding, we examined on Test day how well subjects demonstrated successful transfer of information from initial Preconditioning. For each subject, average head entries across all trials during the 10 s preceding cue onset (baseline, BL) were compared to entries made during the SPC+ (cue A) and the SPC-(cue B) using a one-way ANOVA. For significant main effects of Event (BL, SPC+, SPC-), post-hoc analysis assessed whether responding was greater during the different cues and the baseline period. Rats in the Paired group with significantly greater responding for the SPC+ compared to both the SPC- and BL were “Good Learners” (n = 9), while rats that failed to discriminate between Events (as indicated by no main effect for the ANOVA) were “Poor Learners” (n = 11). All rats in the Unpaired group were analyzed together, regardless of behavior during test. Using this categorical factor, behavior and neural activity during Preconditioning and Test days were assessed relative to the quality of those subjects’ learning.

Analysis of Neural Activity

Neural firing was assessed using similar methods as described previously from our lab (Sugam et al., 2014). For each cue (A, B, X and Y), we calculated the expected (i.e., average) firing rate in each 250ms bin averaged across all the trials for that cue in that session. Next, the average firing rate in the 2s following cue onset was divided by the average of the 10s baseline firing rate. During first-order conditioning and test, the baseline was the 10s immediately preceding each cue. During preconditioning, because some cues were presented sequentially (Paired group) and others presented in isolation (Unpaired), we used as the baseline period the 10s prior to the first cue presented. That is, for the Paired group subjects, the 10s period prior to cue A served as the baseline for both A and X, and likewise for B and Y. Baseline for the Unpaired group was the 10s prior to each cue. Cells that showed firing during the cue that was at least 10% greater than baseline were considered “Excitatory” and those that were at least 10% below baseline were “Inhibitory.” Next, cells were z-normalized relative to the 10s baseline prior to each cue. For each bin, the mean baseline firing rate was subtracted from the firing rate in the current bin, and then divided by the standard deviation of the baseline firing rate. To assess associative changes in firing for these populations, peak firing rate (i.e., maximum deflection from baseline within 1s of cue onset) for a given cue was averaged by group (Good Learners, Poor Learners and Unpaired controls), and differences in normalized peak were assessed using a one-way ANOVA by group, or a 2-way ANOVA if multiple cues were analyzed concurrently. Because very low firing cells (<0.3Hz) were likely to induce artifacts in normalization due to random noise, these cells (n=81 [13%] during SPC; n=86 [11%] during FOC; n=17 [6%] during Test) in were excluded from all magnitude analyses.

Statistical Analysis

All statistical analyses were conducted using STATISTICA version 10 (StatSoft, Inc., 2011). Repeated measures Analysis of Variance (ANOVA) were used to compare conditioned behavior (i.e. head entries) or neural encoding (i.e. % selective data) for all animals, with cue and day (where applicable) as within-subjects factors and group (Paired vs. Unpaired) as a between subjects factor. Significant main effects and interactions were further investigated using Tukey's HSD pairwise comparisons. Finally, linear regressions were used to determine whether or not neural activity was correlated with behavior. The critical value for each comparison was determined at α = 0.05.

Results

Behavior

Preconditioning

Since no food was delivered during this phase, animals were not expected to show significant behavioral conditioning. Consistent with this, subjects showed virtually no head entries during either the baseline or cue periods (avg: <0.1 entry per bin; data not shown). All rats, regardless of how well they later performed during the Test session (i.e. Good vs Poor Learners), showed similarly low rates of foodcup approaches, as revealed by a repeated measures three-way ANOVA on BL-subtracted head entries. There were no significant main effects of Group (Good Learners vs Poor Learners vs Unpaired), Day (Days 1-2), or Cue (A vs B vs X vs Y), or any significant interactions therein (all P values > 0.10).

First-order conditioning

All subjects, regardless of background training from Preconditioning (i.e., Paired or Unpaired), or future performance ability during test (i.e, Good Learners versus Poor Learners), learned the first-order discrimination as indicated by head entry rate during the cues (Figure 1A). A three-way ANOVA that examined group performance (Good Learners, Poor Learners and Unpaired controls), Cue (CS+ vs CS-) and Day (Days1-3) on this task showed that there was a main effect of Day (F_{2, 50} = 62.37, P < 0.0001), a main effect of Cue (F_{1, 25} = 17.7, P < 0.001), and interaction of Day x Cue (F_{2, 50} = 33.5, P < 0.0001). Here, baseline-subtracted foodcup responding during the CS+ (Cue X) increased between Day 1 and Day 2 (P = 0.0001), and again from Day 2 to Day 3 (P = 0.0001). There was no difference between responding for Cue X and Cue Y on either Day 1 (P = 0.99) or on Day 2 (P = 0.84), but a significant increase in responding for Cue X relative to Cue Y on Day 3 (P = 0.0001). However, at no point were there any differences between groups, or any interaction of Group by any other factor.

Figure 1.

Behavioral responding during First-Order Conditioning (left) and sensory preconditioning Test (right). Rats were assessed during Test (Day 6) on their ability to show evidence of successful preconditioning (see Methods). (A) All rats, regardless of grouping, showed accurate discrimination across days between the X cue (CS+) and the Y cue (CS-) during first-order conditioning. †p < 0.05, Cue X D2 vs D1; ‡p < 0.05, Cue X Day 3 vs Day 2; *p < 0.05, Cue X vs Cue Y. (B). In the paired group, Good Learners showed selective increases in responding to the A cue, but not B cue, while Poor Learners failed to acqiure this discrimination. *p < 0.05, A vs B Good Learners; †p < 0.05, Good vs Poor Learners, A cue.

Test

Rats in the Paired group showed a differential ability to successfully learn the SPC task, with some that displayed reliable discrimination between cues during test (Good Learners) and others who showed poor discrimination (Poor Learners). Comparing these subjects versus the Unpaired controls with a 2-way ANOVA, we found that there was a nearly significant trend of group, F_{2 25}= 2.91, p=0.073, and an interaction of group X cue, F_{2, 25}=13.35, p<0.0001 (Figure 1B). Specifically, Good Learners showed elevated BL-subtracted head entries during test for the A compared to the B cue, but Poor Learners and Unpaired subjects showed behavior that was similar for both cues (both p>0.40). Similarly, Good Learners showed head entries during the A cue that was significantly greater than either the Poor Learners (p<0.001) and the Unpaired controls (p<0.001), though head entries rates for the control B cue did not differ between groups (all p > 0.50).

Neural Data

Preconditioning

Next, we tested whether the magnitude of the neural encoding during preconditioning was related to associative encoding of the stimuli, grouped by whether the rats later in Test showed evidence of having learned the SPC task (“Good Learners”) and those who failed to learn the task (“Poor Learners”). We recorded the activity of 593 neurons in the NAc core from Paired (day 1, n = 299; day 2, n = 294), of which there were approximately similar numbers of cells from Good Learners (day 1 n = 111; day 2, n = 121) and Poor Learners (day 1, n = 188; day 2, n = 173). Likewise, we recorded 235 neurons from Unpaired animals during Preconditioning (day 1, n = 116; day 2, n = 119). Using this population, we found that, for all cues (A, B, X and Y), excitatory neurons in the Good Learners group showed greater changes in the peak magnitude of neural activity to cues relative to those in either the Poor Learners or the Unpaired controls. For A (Figure 2A), a one-way ANOVA, F_{2, 180} = 4.45, P < 0.02, showed selective increases in firing following cue presentation in the Good Learners compared to the Poor Learners (P < 0.01), while there were no differences in A encoding between Poor Learners and Unpaired controls (P = 0.65). A planned contrast between Good Learners versus Poor Learners and Unpaired controls was likewise significant (F_{1, 180} = 7.83, P < 0.01). These same effects were found for X (Figure 2B-C: F_{2, 170} = 4.55, P < 0.02; planned contrast, Good vs Poor and Unpaired: F_{1, 170} = 8.29, P < 0.005) and B (Figure 2D: F_{2, 152} = 3.65, P < 0.03; planned contrast, Good vs Poor and Unpaired: F_{1, 152} = 6.77, P < 0.02), while the effect was more marginal for Cue Y (Figure 2E-F: F_{2, 174} = 2.60, P = 0.08; planned contrast: F_{1, 170} = 3.92, P < 0.05).

Figure 2.

Normalized excitatory neural activity during preconditioning. (A-B) Normalized excitatory populations of neurons during both days of preconditioning (Days 1-2) aligned to the presentation of either cue A or cue X. (C) Peak z-normalized firing rates aligned to the presentations of the A and X cues during preconditioning. (D-E) Similarly, normalized excitatory activity aligned to cue B and cue Y, with peak firing to the cues in (F). *p < 0.05, **p < 0.01, Paired (Good) vs Paired (Poor) and Unpaired.

In stark contrast, normalized trough inhibitory encoding for the cues during Preconditioning had no detectable relationship with subsequent Test performance (Figures 3A-F). Of the four cues, none showed a significant main effect of Group on trough inhibitory encoding (A: F_{2, 231} = 1.07, P = 0.34; B: F_{2, 241} = 0.26, P = 0.77; Y: F_{2, 204} = 1.00, P = 0.37) except Cue X, which showed a marginal effect (F_{2, 228} = 3.11, P = 0.047), but no effect of the planned contrast between Good Learners versus Poor Learners and Unpaired controls (ANOVA: F_{2, 228} = 0.16, P = 0.68). Thus, differential magnitude encoding of the stimuli during learning was predictive of later success during test, but only for excitatory encoding cells.

Figure 3.

Normalized inhibitory neural activity during preconditioning. (A-B) Normalized inhibitory populations of neurons during both days of preconditioning (Days 1-2) aligned to the presentation of either cue A or cue X. (C) Trough z-normalized inhibitory firing rates aligned to the presentations of the A and X cues during preconditioning. (D-E) Similarly, normalized excitatory activity aligned to cue B and cue Y, with minimum inhibitory firing to the cues in (F). There were no effects of inhibitory encoding in the NAc core during preconditioning.

These magnitude effects were modestly mirrored in the relative proportions of cells that encoded this information as an excitation or inhibition. For example, while the majority of Good Learners showed excitatory compared to inhibitory encoding (62% vs 42%), Poor Learners showed instead a bias towards inhibitory encoding (46% vs 49%), a difference that was nearly significant by chi-square analysis (χ² = 3.46, p = 0.06). There was no such difference between Poor Learners and Unpaired controls (χ² = 1.72, p = 0.19).

First-order conditioning

Across the three days of FOC, we collected a total of 246 recordings from NAc core neurons from Good Learners (day 1, n = 93; day 2, n = 76; day 3, n = 77), 329 recordings from Poor Learners (day 1, n = 124; day 2, n = 99; day 3, n = 106), and 201 neurons from Unpaired controls (day 1, n = 55; day 2, n = 62; day 3, n = 84). Similar to above, NAc neurons displayed changes in normalized firing rats relative to X and/or Y cues during FOC. Across these populations of cells, we found that neither preconditioning history with the X and Y stimuli (i.e., Paired vs Unpaired; Figures 4A-H), nor the rats’ subsequent performance on Test (i.e., Good Learners vs Poor Learners) had any effect on the rats’ ability to successfully encode first-order conditioning information across days. This was true regardless of whether encoding for the stimuli was excitatory or inhibitory. For excitatory neurons, encoding for the stimuli increased across session commensurate with behavior. A 3-way ANOVA that used Group (Good, Poor, Unpaired), Day (D1-3) and Cue (X vs Y) showed only a main effect of Cue, F_{1, 360} = 25.44, P < 0.0001, with greater encoding for the X cue than the Y cue, and a main effect of Day, F_{2, 360} = 5.5, P < 0.02, which was supported by increases in encoding between Day 1 and Day 2 (P = 0.02), and between Day 1 and Day 3 (P = 0.004), but not between Day 2 and Day 3 (P = 0.91), Figure 4D. However, there was no main effect of Group or interactions of Cue with Day and Cue. Similarly, a 3-way ANOVA that examined inhibitory encoding found a main effect of Cue, F_{1, 156} = 8.29, P < 0.005, with greater inhibitory deflection for the X cue, and a main effect of Day, F_{2, 156} = 4.95, P < 0.01, which was supported by a trend towards an inhibitory decrease between Day 1 and Day 2 (P = 0.08) and a significant decrease between Day 1 and Day 3 (P = 0.002), Figure 4H. As with the excitations, there were no main effects of Group, or any interactions between Group with other factors for inhibitory encoding.

Figure 4.

Neural activity during first order conditioning. (A-C) Normalized excitatory populations of neurons during each day of first-order conditioning, where cue X was reinforced with food delivery. (D) Peak firing magnitude to the food-predictive cue increased over sessions. (E-G) Normalize inhibitory populations of neurons displaying changes in firing relative to the food-paired cue X. (H) As with excitions, peak deflections from baseline increased with learning. At no point were there any reliable differences in encoding between groups. #p < 0.08; *p < 0.05; **p < 0.01 vs D1. Note: ‘Unpaired’ refers to the prior preconditioning phase.

Consistent with the minimal effects of learning on magnitude encoding, we also found that the proportions of cells that showed either excitatory or inhibitory encoding of the stimuli were not different between groups on any of the days, as tested with chi-square analysis (all p > 0.15).

Test

We recorded the activity of 81 neurons in the Good Learners, 111 in the Poor Learners, and 87 in the Unpaired group of animals during SPC Test. Average populations of neural encoding of excitations showed increased magnitude of firing immediately following cue onset for the A cue (Figure 5A), but not for the B cue (Figure 5B). Comparing firing at the time of peak, there was a selective enhancement of firing in the Good Learners compared to the Poor Learners and Unpaired controls. Specifically, a 2-way ANOVA that looked at peak normalized firing found a significant interaction between Group x Cue , F_{2, 100} = 3.56, p = 0.032. Post-hoc examination of this effect showed that this was due to increased peak firing for A in the Good Learners compared to both the Poor Learners (p = 0.027) and to the Unpaired Controls (p = 0.013), though there was no difference between Poor Learners and Unpaired Controls (p = 1.0). However, this was not due to a nonspecific increase in firing of all cues, as there were no differences in peak firing between groups for the B cue (all p > 0.90). Relatedly, only in the Good Learners was there evidence of discriminative firing between the different cues. Neurons in rats from this group displayed significantly greater firing for A than B (p = 0.023), but there were no such differences in the Poor Learners or Unpaired controls (both p > 0.95).

Figure 5.

Neural activity during Sensory preconditioning (Test). (A-B) Normalized excitatory populations of neurons during Test sessions for either SPC+ (cue A) or SPC- (cue B). (C) Peak excitatory firing magnitude to Cue A was selectively enhanced in rats the successfully perfomed the task (”Good Learners”) relative to those who faile to accurately discriminat (”Poor Learners”) or Unpaired controls. (D-E) Normalize inhibitory populations of neurons displaying changes in firing relative to the same cues did not show group-wise changes in activity following cue onset. (F) Peak inhibitory firing was similar across all groups. *p < 0.05; Paired (Good) vs Paired (Poor); †p < 0.05. Paired (Good) vs Unpaired.

However, as during Preconditioning, performance-based differences in encoding had little effect on the encoding of inhibitions (Figures 5C-D). Unlike excitations, an ANOVA found no main effects of Group, Cue or Cue X Day interaction (all F < 1). Planned pairwise comparisons likewise showed that there were no differences in trough inhibition between groups for either A or B. This effect was captured in the relative proportions of cells that encoded excitations compared to inhibitions. Chi-square analysis of the populations showed that Good Learners had a greater than expected distribution of excitations (53%) and inhibitions (30%) relative to Poor Learners (24% vs 34%, χ² = 5.91, p = 0.02) and to Unpaired controls (39% vs 47%, χ² = 4.33, p = 0.04). However, there were no differences in excitation and inhibition distribution between Poor Learners and Unpaired controls, χ² = 0.08, p = 0.78.

Histology

Histological reconstruction of electrode positions revealed that the neurons recorded during the SPC task sessions were located in the core subregion of the NAc . Electrode placements spanned a rostral–caudal distance of ~2.5 mm, ranging from 3.1 to 0.7 mm rostral to bregma, with a medial-lateral range from 0.8 to 3.8 mm lateral to midline, and a dorsal-ventral range of 6.0 to 8.2 mm ventral to the skull surface at bregma. Cases in which wires were not positioned in the NAc core were excluded from the data analysis.

Discussion

Associative learning in a process by which animals acquire the ability to predict future events based on past contingent relationships between stimuli. While reinforcement-based learning demands that learning optimally occurs when the predicted US is of value, animals routinely form associations between otherwise neutral stimuli. However it was unknown the extent to which traditional limbic brain systems are essential for this kind of learning. Here, rats received either predictive pairings or unpaired presentations of neutral stimuli during Preconditioning, and were given the opportunity to direct their behavior towards a preconditioned cue at Test once its paired associate had acquired value in FOC. Under this schedule, a subset of rats in the Paired group (Good Learners) successfully showed selective and enhanced head entries for the SPC+ cue during Test, while the other set of Paired subjects (Poor Learners) failed to demonstrate evidence of preconditioning, and were similar to Unpaired controls. Based on this, we found that subsets of neurons in the NAc core differentially tracked cue information during initial learning (Preconditioning) and subsequent performance (Test) that reflected the subjects’ individual differences in their ability to perform accurately on the task. Specifically, rats in the Good Learner group displayed greater excitatory activity for all stimuli during both Preconditioning and Test relative to Poor Learners and Unpaired controls. These findings demonstrate that NAc core neurons track associative relationships in the absence of value, and further, that the strength of this encoding during learning was predictive of subsequent performance on the Test portion of the sensory preconditioning task.

Reinforcement-based learning models such as RW have been a powerful mechanism to understand the essential components of Pavlovian conditioning and other associative behaviors (Rescorla, 1980a) and have been highly influential in understanding the mechanism of dopamine function during learning (Schultz et al., 1997). However, these and similar predictive models require that animals use the value of the expected reinforcer to support changes in learning. Sensory preconditioning is a task that requires animals to learn about stimuli in a value-free state, and thus, reinforcement-based models should predict that animals will be unable to successfully learn about these neutral associations. However, in accordance with early findings (Rizley & Rescorla, 1972; Rescorla, 1980b), rats in the present study showed evidence of learning SPC despite a value-free initial learning state. Contemporary models suggest that animals may employ either “model-free” representations of the world (e.g., RW), that preferentially use value estimates to generate associative behaviors, and/or “model-based” representations, which would make more use of associations among different related stimuli (McDannald et al., 2012). Given the present findings, and previous work along these lines (McDannald et al., 2011), the NAc appears to be important for at least some aspects of model-based encoding for supporting behavior (but see McLaren et al. (2012) for a possible alternative interpretation).

Neural encoding in this task supports a role for the NAc in the encoding of more model-based value-neutral associations. Here, we demonstrate that subjects who received contingent pairings of stimuli during preconditioning and were able to successfully transfer this information to support behavior in Test (Good Learners) showed significantly greater firing rates during cues during Preconditioning sessions. This differential encoding was found for all preconditioned cues, and was greater than that displayed by Poor Learners and Unpaired controls. Because all paired subjects received the same number of contingent pairings of stimuli, the only measurable difference during Preconditioning between Good and Poor Learners was the magnitude of cue encoding in the NAc. This suggests that this differential encoding was functional, allowing rats to attend to the relevant stimuli and their associations, and permitting the transfer of value between the stimuli after FOC training. However, it is unknown whether this encoding is necessary for SPC encoding, and future studies will investigate this possibility.

One factor that appears to have contributed to differential encoding properties between different stimuli was an unanticipated difference in cue salience for the different stimuli employed. Specifically, excitatory encoding for the white noise stimulus during both preconditioning and test was reliably greater than for the other auditory and visual stimuli in the same sessions. Indeed, due to artifacts associated with initial experimental design, the noise stimulus was more often used as Cue A than for other cues, giving rise to greater excitatory encoding of that cue than others. It is not known specifically why this is the case, as previous experiments using these stimuli in a similar task failed to show any reliable differences in stimulus encoding (e.g., Saddoris & Carelli, 2014). However, one possible difference between SPC and previous tasks is that preconditioning occurs in the absence of any salient reinforcing outcomes such as food. As such, in more standard conditioning tasks, the salience of the food US may effectively overshadow any intrinsic differences in cue salience, while in the present SPC task, any intrinsic differences between stimuli may be enhanced due to the relative paucity of any other competing stimuli.

Regardless, intrinsic differences in cue salience cannot explain the set of findings presented here. First, rats in the Good Learner group by definition showed enhanced behavioral transfer for SPC+ cue during test, while none of the other subjects in the Poor Learner or Unpaired controls did so. Thus, salience of the stimuli alone was insufficient to explain the differential performance in behavior. Second, while excitatory encoding for Cue A during preconditioning was overall higher than for the other cues due to the high proportion of the noise stimulus in that condition, this encoding was nonetheless strongly enhanced when it was associatively predictive of X in the Good Learners, showing roughly a doubling in excitatory peak relative to Poor Learners and Unpaired controls. Likewise, though peak excitatory encoding was somewhat lower for the other cues B, X and Y compared to A, we again saw the enhancement in peak excitatory firing for those stimuli in the Good Learners that was roughly double that in the Poor Learners and Unpaired controls. Thus, cue salience was encoded in these neurons, but it was strongly and significantly modulated by associative learning during preconditioning and test based on task performance.

A robust finding in the neural signal was the selective modulation of excitatory encoding for the preconditioned cues based on task performance, while inhibitions failed to differentially encode learning-based differences in behavior. One possible explanation for this observation is a lack of sensitivity to detect inhibitions due to the population averaging method employed here, particularly given that the noted low resting baseline activity of medium spiny neurons (MSNs) in the NAc. However, we found approximately equal proportions of inhibitions and excitations during Preconditioning, and in previous work, we have shown the ability to detect differences in inhibitory encoding based on task demands in the NAc (Sugam et al., 2014). An alternative interpretation is that excitatory encoding displayed here during Preconditioning and Test reflects important processing components for NAc and its limbic inputs. For example, in response to convergent bursts of afferent excitation, MSNs in the NAc typically increase their firing rate (Pennartz et al., 1991; Lape & Dani, 2004). One possibility, then, is that excitatory information encoded by MSNs in the NAc core during SPC reflects the input of glutamateric limbic afferents. However, as bilateral lesions of the BLA are without effect on SPC (Blundell et al., 2003; Dwyer & Killcross, 2006), possible alternative sources of excitatory activity observed in this study are the prefrontal regions, which have been shown to be necessary for cue-evoked phasic activity in the NAc core and behavioral responses to reward predictive cues (Ishikawa et al., 2008). In support, Jones et al. (2012) provide compelling evidence that inactivation of the orbitofrontal subregion of the prefrontal cortex (OFC) disrupts SPC by preventing the transfer of motivational value to once neutral paired stimuli at Test. However, the findings of the present study indicate that NAc neurons encode information significantly in advance of the time of Test performance, displaying differential activity for the cues during initial learning. It is unknown whether OFC plays a similar learning role during preconditioning, or if instead, the NAc coordinates this activity with other components of mesolimbic circuitry.

As such, dopaminergic (DA) input to the NAc from midbrain regions may be one important mechanism in this process (Saddoris et al., 2013). In support of a DAergic role in SPC, Young and colleagues examined DA levels in the NAc in SPC using microdialysis (Young et al., 1998). In that study, DA concentrations were higher for two neutral stimuli presented simultaneously (paired) versus when presented in an unpaired fashion. These results are in accordance with the heightened excitatory encoding of stimuli by Paired animals observed in the present study during Preconditioning. Further, they found that during Test, DA increased for the previously paired, but not unpaired preconditioned cue, consistent with the differential neural encoding by NAc neurons observed here. Despite the slower timescale used in microdialysis, these findings suggest the intriguing possibility that phasic DA release during both SPC learning and test could contribute to the neural encoding seen here.

More generally, the DA system has long been thought to provide a neural signal consistent with reinforcement-based predication error models (Schultz et al., 1997). For example, DA release within the NAc shows phasic release when an unexpected reward is delivered, signaling a positive prediction error, but shifts to the predictive cue with learning (Day et al., 2007). Importantly, DA release at the point of reward decreases as it becomes better predicted, indicating a minimization of the error between the prediction value and received outcome. However, because in these experiments prediction is conflated with value, it is unknown whether DA is signaling the expectation of another event, or if instead it is tracking the expected value of the reinforcer. Recordings of putative DA neurons indicate that encoding of a short chain of cues (A → B → food) tracks information about both cues well after the task is well learned (Pan et al., 2005), suggesting a possible role of DA signaling in polysensory associative encoding.

This possible role for DA may relate to the neural responses described here. Findings from our lab suggest that bursts of DAergic transmission to the NAc core from the ventral tegmental area (VTA) is phasically released in the same regions where MSNs develop phasic neural responses to associative stimuli (Cheer et al., 2005; Cheer et al., 2007; Owesson-White et al., 2009; Cacciapaglia et al., 2011). Further, we have recently shown that phasic DAergic signaling is important for selectively supporting excitatory but not inhibitory responses of NAc neurons during reward-seeking behavior (Cacciapaglia et al., 2011). In that study, pharmacological blockade of burst firing of the VTA with AP5 resulted in the loss of both phasic DA release and excitatory encoding of MSNs for task-related stimuli. However, inhibitory encoding of stimuli were unaffected by the loss of the DA signal. This disparity may reflect differences in the relative expression of D1 and D2 receptors on neurons in the NAc and their related output (direct versus indirect) pathways (Bertran-Gonzalez et al., 2008; Hasbi et al., 2010), as phasic but not sustained tonic DA release activates low affinity D1 receptors (Richfield et al., 1989). In contrast, inhibitions in NAc cell firing by contrast are likely mediated by high affinity D2 receptor activation (Pennartz et al., 1994; Dreyer et al., 2010). As such, the changes in excitatory encoding seen here during SPC may reflect DAergic input acting though D1R-mediated mechanisms, and possibly coordinating this activity with excitatory glutamatergic input from other limbic regions such as the prefrontal cortical areas.

In conclusion, we have demonstrated that the NAc core participates in the encoding of associations that do not necessarily reflect the intrinsic value of the reinforcer or prediction errors based on value discrepancies across trials. Indeed, this differential encoding during learning was predictive of future success in transferring associative value during test, and suggests a possible functional role for NAc in this type of more model-based form of associative learning. Given the well-established role of NAc in value-based associative encoding, these findings further our understanding of NAc function as playing a role in both reinforcement-based and more general associative processes.